【应急响应】帮哥们应急记录20170410

背景


一哥们反应自己的测试机器总是关机,联系云主机客服得知服务器总是大量发包,导致技术关停该机器。

排查过程


Last查看登录记录

查找到一个可疑IP

[root@i-9kp9tipm dpkgd]# grep "1.180.212.21" /var/log/secure*
/var/log/secure-20170409:Apr  4 22:20:43 i-9kp9tipm sshd[25921]: Accepted password for root from 1.180.212.21 port 5777 ssh2
/var/log/secure-20170409:Apr  4 23:21:16 i-9kp9tipm sshd[29214]: Accepted password for root from 1.180.212.21 port 50625 ssh2
/var/log/secure-20170409:Apr  5 00:13:12 i-9kp9tipm sshd[31509]: Accepted password for root from 1.180.212.21 port 12305 ssh2
/var/log/secure-20170409:Apr  5 00:24:56 i-9kp9tipm sshd[32029]: Accepted password for root from 1.180.212.21 port 12334 ssh2

查看所有登录成功的记录

[root@i-9kp9tipm log]# grep "Accept" /var/log/secure* | awk '{print $11}' | sort | uniq
1.180.212.21
119.254.100.106
124.207.112.10

确认后得知119.254.100.106和124.207.112.10为正常IP,但是这哥们的Root密码也算是复杂。
既然Root被拿了,那么看一下常用命令呢

[root@i-9kp9tipm log]# ls -alt /bin/ | head -n 10
total 8324
-rwxr-xr-x.  1 root root 1135000 Apr 10 10:05 ps
-rwxr-xr-x.  1 root root 1135000 Apr 10 10:05 netstat

发现ps和netstat被替换

[root@i-9kp9tipm log]# ls -alt /usr/sbin/ | head -n 10
total 11108
-rwxr-xr-x.  1 root root     1223123 Apr 10 10:05 ss

发现ss被替换

[root@i-9kp9tipm log]# ls -alt /usr/bin/ | head -n 10
total 25908
-rwxr-xr-x   1 root root   1135000 Apr 10 10:05 pythno
-rwxr-xr-x   1 root root   1223123 Apr 10 10:05 .sshd
drwxr-xr-x   2 root root      4096 Apr  7 21:29 bsd-port
dr-xr-xr-x.  4 root root     16384 Apr  7 21:29 .
drwxr-xr-x   2 root root      4096 Apr  7 06:41 dpkgd

发现2进制程序pythno和.sshd,并且在bsd-port下也有两个可执行程序
在dpkgd目录下发现替换之前的命令

[root@i-9kp9tipm dpkgd]# ll
total 280
-rwxr-xr-x 1 root root 123360 Apr  7 06:41 netstat
-rwxr-xr-x 1 root root  85304 Apr  7 06:41 ps
-rwxr-xr-x 1 root root  71024 Apr  7 06:41 ss

看一下.sshd文件的时间点。

 [root@i-9kp9tipm log]# stat /usr/bin/.sshd
  File: `/usr/bin/.sshd'
  Size: 1223123       Blocks: 2392       IO Block: 4096   regular file
Device: 801h/2049d    Inode: 794702      Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2017-04-10 10:05:45.425551007 +0800
Modify: 2017-04-10 10:05:44.419550999 +0800
Change: 2017-04-10 10:05:44.419550999 +0800

用Strings查看一下

[root@i-9kp9tipm log]# strings /usr/bin/.sshd | egrep '[1-9]{1,3}\.[1-9]{1,3}\.'
8.8.8.8
8.8.4.4
8.8.8.8
61.132.163.68
202.102.192.68
202.102.213.68
58.242.2.2
202.38.64.1
211.91.88.129
211.138.180.2
218.104.78.2
202.102.199.68
202.175.3.3

发现里面有很多IP地址。

查看下当前的连接:

[root@i-9kp9tipm tmp]# netstat -antlp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LISTEN      1273/sshd           
tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      1655/master         
tcp        0     64 192.168.100.3:22            218.247.17.100:29737        ESTABLISHED 2007/sshd           
tcp        0      0 192.168.100.3:22            124.207.112.10:54772        ESTABLISHED 1919/sshd           
tcp        0      1 192.168.100.3:35806         43.241.157.58:6001          SYN_SENT    1742/getty          
tcp        0      1 192.168.100.3:48358         211.149.149.191:45693       SYN_SENT    1677/abcfg          
tcp        0      1 192.168.100.3:47268         61.147.73.76:1233           SYN_SENT    1683/VI             
tcp        0      0 :::3306                     :::*                        LISTEN      1508/mysqld         
tcp        0      0 :::22                       :::*                        LISTEN      1273/sshd           
tcp        0      0 ::1:25                      :::*                        LISTEN      1655/master

发现了3个可以进程1742、1677、1683
看一下这些可执行程序在什么地方

[root@i-9kp9tipm tmp]# ps axu | grep 1677
root      1677  0.0  0.0  93636   892 ?        Ssl  10:05   0:01 /usr/local/tomcat/abcfg
[root@i-9kp9tipm tmp]# ps axu | grep 1683
root      1683  0.0  0.0  73088   824 ?        Ssl  10:05   0:01 /usr/local/tomcat/VI
[root@i-9kp9tipm init.d]# ps aux | grep 1742 | grep -v grep
[root@i-9kp9tipm init.d]#

发现这个ps看不到1742,然后我从自己主机上拷贝了一个ps过来,再执行下

[root@i-9kp9tipm init.d]# /tmp/ps aux | grep 1742 | grep -v grep
root      1742  0.0  0.0  93636   896 ?        Ssl  10:05   0:00 /usr/bin/bsd-port/getty
可以看到了
[root@i-9kp9tipm tomcat]# ls /usr/local/tomcat
abcfg  conf    idus.log  LICENSE  logs    RELEASE-NOTES  temp      VI       work
bin    conf.n  lib       linux    NOTICE  RUNNING.txt    vga.conf  webapps

这三个2进制程序

-rwxrwxrwx   1 root root 1135000 Apr  7 21:24 VI
-rwxrwxrwx   1 root root  826008 Apr  6 20:22 linux
-rwxrwxrwx   1 root root 1223123 Apr  6 19:54 abcfg

然后哥们反馈的是开机就会大量发包,来看下crontab,没有异常。查看rc.local没有异常。查看/etc/cron*没有异常。检查/etc/init.d的时候发现

[root@i-9kp9tipm init.d]# ls -alt /etc/init.d/
total 240
drwxr-xr-x.  2 root root  4096 Apr 10 10:43 .
-rwxr-xr-x   1 root root    36 Apr 10 10:05 selinux
-rwxr-xr-x   1 root root    33 Apr 10 10:05 VsystemsshMdt
-rwxr-xr-x   1 root root    36 Apr 10 10:05 DbSecuritySpt

[root@i-9kp9tipm init.d]# cat /etc/init.d/VsystemsshMdt
#!/bin/bash
/usr/local/tomcat/VI
[root@i-9kp9tipm init.d]# cat /etc/init.d/DbSecuritySpt
#!/bin/bash
/usr/local/tomcat/abcfg
[root@i-9kp9tipm init.d]# cat /etc/init.d/selinux
#!/bin/bash
/usr/bin/bsd-port/knerl

/tmp下发现DDOS客户端文件

[root@i-9kp9tipm tmp]# ll
total 3132
-rwxrwxrwx  1 root  root     6800 Apr  9 16:57 dbuspm-session
-rwxrwxrwx  1 root  root  1584675 Apr  9 16:44 DDosClient
-rwxrwxrwx  1 root  root  1584675 Apr  9 16:40 dllhest
-rwxr-xr-x  1 root  root        4 Apr 10 10:50 gates.lod
drwxr-xr-x  2 root  root     4096 Apr 10 10:55 hsperfdata_root
-rwxr-xr-x  1 root  root        4 Apr 10 10:50 moni.lod
srwxrwxrwx  1 mysql mysql       0 Apr 10 10:05 mysql.sock

检查history没有发现操作记录。
检查mysql,发现密码简单123456,启动权限为mysql,plugin目录没有写入权限。
history中看到哥们之前有安装redis,且redis.conf中没有配置pass,检查/root/.ssh下没有发现异常,查看redis键未发现异常。
中间件是tomcat,发现/manager/html,没有账户。
项目为java项目,用到了struts2,检查struts2的版本

[root@i-9kp9tipm lib]# ll struts2-core*
-rw-r--r-- 1 root root 802044 Apr  6 16:05 struts2-core-2.3.15.1.jar

查看access log

cat localhost_access_log* | grep -v '127.0.0.1' | grep -v '124.207.112.10'

没有发现异常

处理方式


1)加强SSH和Mysql密码,SSH限制IP访问,redis后期如果使用需配置认证。如无需对外提供服务,仅监听本地。
2)升级struts2到2.3.32版本
3)杀死进程

[root@i-9kp9tipm init.d]# kill -9 1742
[root@i-9kp9tipm init.d]# kill -9 1677
[root@i-9kp9tipm init.d]# kill -9 1683

4)删除木马文件,拷贝正常命令

[root@i-9kp9tipm manager]# rm -f /usr/bin/.sshd
[root@i-9kp9tipm manager]# rm -f /bin/netstat
[root@i-9kp9tipm manager]# rm -f /bin/ps
[root@i-9kp9tipm manager]# rm -f /usr/bin/pythno
[root@i-9kp9tipm bin]# rm -f /usr/bin/bsd-port/getty*
[root@i-9kp9tipm bin]# rm -f /usr/bin/bsd-port/knerl*
[root@i-9kp9tipm bin]# rm -f /usr/sbin/ss
[root@i-9kp9tipm tmp]# rm -rf /tmp/*

[root@i-9kp9tipm tomcat]# rm -f abcfg
rm: cannot remove `abcfg': Operation not permitted
[root@i-9kp9tipm tomcat]# lsattr abcfg
----i--------e- abcfg
[root@i-9kp9tipm tomcat]# chattr -i abcfg
[root@i-9kp9tipm tomcat]# rm -f abcfg
[root@i-9kp9tipm tomcat]# rm -f linux
rm: cannot remove `linux': Operation not permitted
[root@i-9kp9tipm tomcat]# chattr -i linux
[root@i-9kp9tipm tomcat]# rm -f linux
[root@i-9kp9tipm tomcat]# rm -f VI
[root@i-9kp9tipm tomcat]# rm -f idus.log
[root@i-9kp9tipm tomcat]# rm -f vga.conf

5)删除开机启动项

[root@i-9kp9tipm init.d]# rm -f /etc/init.d/selinux
[root@i-9kp9tipm init.d]# rm -f /etc/init.d/DbSecuritySpt
[root@i-9kp9tipm init.d]# rm -f /etc/init.d/VsystemsshMdt

6)重启