Postfix: Deal with a cracked email account

This is a tipical friday story on sysadmin world, starting with a common Munin alert email and ending fighting against a cracked email account due a weak password. The alert referred to a weird problem with inodes outside range in / partition:

servers :: our.server :: Inode usage in percent
        CRITICALs: / is 100.00 (outside range [:98]).

Inode usage in percent was outside normal range, but df gave me no clue about that strange situation. So after a couple of searches about how to check a directories' list order by inodes usage, I found an interesting resource (stackoverflow, of course):

# find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n
      9 tmp
     23 lib32
     31 root
     91 bin
    129 sbin
    212 boot
    888 etc
   2929 lib
  27393 usr
2327393 var
# cd /var
# find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n
    245 log
    912 cache
2325393 spool

Round one, MTA (Postfix in this case) was the main suspect. After playing a bit with mailq I confirmed that something was rotting there, so before restart the service and clear the queue, should find from where the disaster came: external IP or local script.

Dropping port 25 vía iptables should resolve the question (iptables -A INPUT -p tcp --dport 25 -j DROP) and there it was, attack stopped. So we configure iptables again to accept pettions -iptables -A INPUT -p tcp --dport 25 -j DROP- and let's see what netstat can show:

# netstat -nap | grep ":25" | more
tcp        0      0 192.168.1.25:25         0.0.0.0:*               LISTEN
                2239/master
tcp        0     14 192.168.1.25:25         94.251.111.121:56253    ESTABLISHED
                2310/smtpd

After a docen of tries, the fact of ever seeing that IP (94.251.111.121) was really annoying, even further in case of a .ru IP so we proceeded with a ban to really confirm the source of the attack.

# iptables -A INPUT -s 94.251.111.121 -p tcp --dport 25 -j DROP

Attack stopped again, in fact we have found the real offender but... how that machine have obtained that right?. That's the other unknown we must resolve. As we still have the mailq information (remember, we didn't stop the service nor delete the queue), we could proceed with some commands that help us in searching. With mailq we can focus the mail ID, and withpostcat we can show the mail details:

# mailq
840A4C0FF0  1145 Fri May 23 13:56:34 non-existent@ourdomain.com
(host alt1.gmail-smtp-in.l.googl...
# postcat -q 840A4C0FF0
# postcat -q 840A4C0FF0 | grep sasl_username
named_attribute: sasl_username=email@ourdomain.com

We can figure that someone is sending spam with the email@ourdomain.com local account, but the best approach is to see more information than an unique random ID/email:

# for i in $(mailq | cut -f1 -d" " | grep -v "(" | grep -v "-" | sort | uniq); do postcat -q $i | grep sasl_username; done
named_attribute: sasl_username=email@ourdomain.com
named_attribute: sasl_username=email@ourdomain.com
...

Suspicion confirmed, a weak password was the cause of all that weird situation. So let's change the password, clear the queue and restart the servers. Last task, but not least, is to have a serious talk with the proper user.