The feeling of getting your home server hacked

[slaeshjag]

I was just about to go to bed, when I just randomly felt I needed to visualize for myself how it would look for users who logged in on my home server (a few friends have a shell account on it.) After logging in, I was just about to close the lid on my laptop when I noticed something strange.


Last login: Fri Mar 11 10:38:21 2011 from 64.206.96.xx


That's odd, I've been home all day, and I didn't wake up until 1pm. So I did a quick lookup on the IP. It's registered to a company in NY, US. That can't be good.


What was he up to on my server? And what was he doing with my account? Well, since I know what I typically do and don't, I was able to extract it from my .bash_history

Spoiler

w


ls


cat sl.py


ls


cat isbigger.c


cat config


ls -la


cat cowsay


cat .bash_history


nano .bash_history


sudo


ls


cat fortune.txt


ls


cat stream.sh


cat xorg.conf


ls -la


cat .nano_history


cd code


ls


cat socket-hidden-server.c


ls


./socket-hidden-server


cd pub/


ls


cat posix_tcp_ip_sockets.txt


w


strings /usr/sbin/sshd|grep %s:%s -B2|head -n 1|grep -v port


ps ax


cat /proc/cpuinfo


w


tmux


tmux --help


tmux --morte


tmux --more


socklist


netstat -tan


su


tmux attach


tmux attach


w


uname 0a


uname -a


ls


pwd


nc


nc -vvlp 2121

Now, this is just rude. Not only does he poke around in my files, HE ATTACHED MY TMUX! This is the point where it all became personal. I did the bad practice of having a root terminal up, but it was following a log, so he probably didn't notice it. But it also had my irssi etc.. That's not very nice. This is the point where I'm thankful for my ISPs bad peering to Germany, my session at a friends server had already been dropped by the point he logged in.


but how did he get in? I grepped his IP in auth.log, but there was only one login attempt, the one he got in on. Grepped on my username that day, only one login before I woke up.


Mar 11 10:38:21 Choksondik sshd[25726]: Accepted publickey for steven from 64.206.96.xx port 45381 ssh2


Publickey? WHAT? And it doesn't even seem like he had to guess the username, I'm pretty sure I didn't leave my public/private key pair on the internetz


Is this a hole in the ssh server that debian hasn't patched? What the hell?


Mar 11 10:38:21 Choksondik sshd[25726]: pam_unix(sshd:session): session opened for user steven by (uid=0)


Mar 11 10:39:35 Choksondik sshd[25726]: pam_unix(sshd:session): session closed for user steven


He didn't stay very long either, is this a scripted bot, or is he some sort of super human? Just look at everything he did during that minute fourteen.


Needless to say, I've closed down external access to my SSH server until I know what the hell is going on.


Now I'm just feeling... Dirty, violated... The though of somebody accessing my private shell/fileserver... It's hard to describe, I just feel like I need revenge.


What would you do in a situation like this?

 

[levi1123]

A tad above my head, as far as how this happened. I can at least relate with being on these very boards I was brewing coffee. The Keurig finished, so I got up to prepare it I returned to having 50 IE windows open. All with various pornos/ads playing ( I have my sound muted most of the time, which is probably why I didn't notice noise ) Anti Spyware Removal Pro 2010 or something behind them. My public IP was displayed on my background as well as my address somehow, and my log on password changed. There was also a password put on the root of my drive, so I couldn't recover any data by slaving the drive via USB without it. All must have been done in a matter of 3-5 minutes with no hope of me seeing who/what/how this happened.


Logging on my router, which I had enabled at the time, was disabled as well. Running the latest DD-Wrt. I knew enough after that to change the MAC on my router, to get a different IP from my ISP and to wipe the Router, to reinstall the firmware on it. I then had to completely wipe out my hard drive losing all data I hadn't gotten to backup for the past few months and reinstall Windows.


After that happened I felt as if someone came to my house while I was on the john, decided to clean out my fridge, burn all my clothing, then tear my bed sheets to bits without me noticing. It sucked to have to deal with, but luckily my hardware wasn't damaged so that all I had to do was reformat the computer. Still, I am trying to get back things I've lost and had to re-download or even repurchase due to a one time license. I feel your pain.

 

[bzar]

I know the feeling all too well. Our server was hacked some time ago (posted about it in the wars thread), and even after I purged the LAN going from computer to computer looking for evidence of unauthorized logins and taking all the necessary precautions and actions (like reinstalling the OS on the rootkitted media server ) I felt insecure. I knew exactly where the attacker had gotten and had no doubt s/he'd never get in again from the same hole, but the feeling of uneasiness stayed for the few days. Now trying the same trick again will only get the attacker to my virtual sandbox, where I can monitor breaches before they actually reach anything important.

 

[lulzfish]

This thread freaked me out a little, so I checked /var/log/auth.log on my server.


Despite my constant use of SSH, auth.log and auth.log.1 are blank. auth.log.2 starts at February 18th.


What the hell.


I think more files were blanked. "who" reports 4 users, there should only be two, my SSH and my local terminal. I just unplugged my server's LAN cable, going to check things on local. This is fucked up.


I think the other two users were my screen terminals, and it's possible that SSH somehow forgot to log after it was upgraded, but that would be weird.

 

[Granitehead]

This thread freaked me out a little, so I checked /var/log/auth.log on my server.

Just did the same.


Most of it looked like this:

Mar 12 05:16:38 t4bserver sshd[7884]: reverse mapping checking getaddrinfo for ge-3-3-0-core-as12455.orange.co.ke [196.202.196.213] failed - POSSIBLE BREAK-IN ATTEMPT!

Mar 12 05:16:42 t4bserver sshd[7886]: reverse mapping checking getaddrinfo for ge-3-3-0-core-as12455.orange.co.ke [196.202.196.213] failed - POSSIBLE BREAK-IN ATTEMPT!

Mar 12 05:16:46 t4bserver sshd[7888]: reverse mapping checking getaddrinfo for ge-3-3-0-core-as12455.orange.co.ke [196.202.196.213] failed - POSSIBLE BREAK-IN ATTEMPT!

Mar 12 05:16:50 t4bserver sshd[7890]: reverse mapping checking getaddrinfo for ge-3-3-0-core-as12455.orange.co.ke [196.202.196.213] failed - POSSIBLE BREAK-IN ATTEMPT!

Mar 12 05:16:54 t4bserver sshd[7892]: reverse mapping checking getaddrinfo for ge-3-3-0-core-as12455.orange.co.ke [196.202.196.213] failed - POSSIBLE BREAK-IN ATTEMPT!

Mar 12 05:16:58 t4bserver sshd[7894]: reverse mapping checking getaddrinfo for ge-3-3-0-core-as12455.orange.co.ke [196.202.196.213] failed - POSSIBLE BREAK-IN ATTEMPT!

Mar 12 05:17:02 t4bserver sshd[7896]: reverse mapping checking getaddrinfo for ge-3-3-0-core-as12455.orange.co.ke [196.202.196.213] failed - POSSIBLE BREAK-IN ATTEMPT!

Mar 12 05:17:05 t4bserver sshd[7898]: reverse mapping checking getaddrinfo for ge-3-3-0-core-as12455.orange.co.ke [196.202.196.213] failed - POSSIBLE BREAK-IN ATTEMPT!

Mar 12 05:17:09 t4bserver sshd[7900]: reverse mapping checking getaddrinfo for ge-3-3-0-core-as12455.orange.co.ke [196.202.196.213] failed - POSSIBLE BREAK-IN ATTEMPT!

Mar 12 05:17:13 t4bserver sshd[7902]: reverse mapping checking getaddrinfo for ge-3-3-0-core-as12455.orange.co.ke [196.202.196.213] failed - POSSIBLE BREAK-IN ATTEMPT!

Mar 12 05:17:16 t4bserver sshd[7904]: reverse mapping checking getaddrinfo for ge-3-3-0-core-as12455.orange.co.ke [196.202.196.213] failed - POSSIBLE BREAK-IN ATTEMPT!

Mar 12 05:17:20 t4bserver sshd[7906]: reverse mapping checking getaddrinfo for ge-3-3-0-core-as12455.orange.co.ke [196.202.196.213] failed - POSSIBLE BREAK-IN ATTEMPT!

or this

Mar  6 07:29:45 t4bserver sshd[30439]: reverse mapping checking getaddrinfo for 122-146-120-139.static.sparqnet.net [122.146.120.139] failed - POSSIBLE BREAK-IN ATTEMPT!

Mar  6 07:29:45 t4bserver sshd[30439]: Invalid user ts from 122.146.120.139

Mar  6 07:29:45 t4bserver sshd[30439]: pam_unix(ssh:auth): check pass; user unknown

Mar  6 07:29:45 t4bserver sshd[30439]: pam_unix(ssh:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=122.146.120.139 

Mar  6 07:29:46 t4bserver sshd[30439]: Failed password for invalid user ts from 122.146.120.139 port 46328 ssh2

Mar  6 07:29:49 t4bserver sshd[30441]: reverse mapping checking getaddrinfo for 122-146-120-139.static.sparqnet.net [122.146.120.139] failed - POSSIBLE BREAK-IN ATTEMPT!

Mar  6 07:29:49 t4bserver sshd[30441]: Invalid user TeamSpeak from 122.146.120.139

Mar  6 07:29:49 t4bserver sshd[30441]: pam_unix(ssh:auth): check pass; user unknown

Mar  6 07:29:49 t4bserver sshd[30441]: pam_unix(ssh:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=122.146.120.139 

Mar  6 07:29:51 t4bserver sshd[30441]: Failed password for invalid user TeamSpeak from 122.146.120.139 port 46623 ssh2

Mar  6 07:29:54 t4bserver sshd[30443]: reverse mapping checking getaddrinfo for 122-146-120-139.static.sparqnet.net [122.146.120.139] failed - POSSIBLE BREAK-IN ATTEMPT!

Mar  6 07:29:54 t4bserver sshd[30443]: Invalid user cisco from 122.146.120.139

Mar  6 07:29:54 t4bserver sshd[30443]: pam_unix(ssh:auth): check pass; user unknown

Mar  6 07:29:54 t4bserver sshd[30443]: pam_unix(ssh:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=122.146.120.139

More than every eighth line contains "POSSIBLE BREAK-IN ATTEMPT!"


But, as far as I can tell, nobody got in.


Just changed my password to something even more paranoid.

 

[bzar]

^ I recommend installing an automatic blacklister that automatically adds frequent failers to hosts.deny. They get a few tries with their common user names and get blacklisted before getting to any working ones. Just don't use any common user names and you'll evade a great portion of any break-in attempts.


EDIT: I use denyhosts. I have 265 hosts on blacklist so far.

 

[Granitehead]

^ I recommend installing an automatic blacklister that automatically adds frequent failers to hosts.deny. They get a few tries with their common user names and get blacklisted before getting to any working ones. Just don't use any common user names and you'll evade a great portion of any break-in attempts.


EDIT: I use denyhosts. I have 265 hosts on blacklist so far.

Yeah, I'll do so.

 

[I8NY]

same here but i thought linux was more secure than this

 

[bzar]

^ Linux is, but some stuff you can put on it are not.

 

[dreamer]

Mar 11 10:38:21 Choksondik sshd[25726]: Accepted publickey for steven from 64.206.96.xx port 45381 ssh2

Publickey? WHAT? And it doesn't even seem like he had to guess the username, I'm pretty sure I didn't leave my public/private key pair on the internetz


Is this a hole in the ssh server that debian hasn't patched? What the hell?

Ouch! That is really nasty!


I hope this is not some gaping hole in debian's openssh-server

 

[Alec]

Mar 11 10:38:21 Choksondik sshd[25726]: Accepted publickey for steven from 64.206.96.xx port 45381 ssh2

Publickey? WHAT? And it doesn't even seem like he had to guess the username, I'm pretty sure I didn't leave my public/private key pair on the internetz


Is this a hole in the ssh server that debian hasn't patched? What the hell?

Ouch! That is really nasty!


I hope this is not some gaping hole in debian's openssh-server

Sounds to me like the intruder managed to write his key into ~/.ssh/authorized_keys

 

[WizardStan]

I hope this is not some gaping hole in debian's openssh-server

About a year back, I think it was, it was discovered that someone had accidentally on purpose removed some of the entropy from the ssh key gen code. The result was that there were a lot of keys which were more common than others, which opened a lot of potential for attacks.


If you've been keeping your system up to date, this shouldn't be it, though. It was patched and the keys forced deleted as soon as it was discovered. That's the only bug that I'm aware of that would behave in this way, assuming I understood everything correctly.

 

[slaeshjag]

Mar 11 10:38:21 Choksondik sshd[25726]: Accepted publickey for steven from 64.206.96.xx port 45381 ssh2

Publickey? WHAT? And it doesn't even seem like he had to guess the username, I'm pretty sure I didn't leave my public/private key pair on the internetz


Is this a hole in the ssh server that debian hasn't patched? What the hell?

Ouch! That is really nasty!


I hope this is not some gaping hole in debian's openssh-server

Sounds to me like the intruder managed to write his key into ~/.ssh/authorized_keys

There was only my key in there.

 

[metallurge]

I'll second the denyhosts solution for securing an Internet-facing ssh server. I've been using it for years with very paranoid settings, as in more paranoid than the defaults.

 

[XxionxX]

This entire thread is why I want to learn how to secure my network and learn how to code. It really bothers me that I only have a vague idea about what is going on, but I totally understand of having someone violate your space :angry: . Its too bad that it is so hard to find the guy. <_<

 

[lulzfish]

Learn to code?


That's not a problem here. It's the fucking obtuse configuration files. I just copied my SSH configs from the Arch Linux wiki and have no idea what it means, and it's probably all wrong. Apache is the same way, I just fucked with things until they sort of worked and then ignored security.


It's always the configuration files. >

 

[XxionxX]

I mean that if there was a security hole in the existing software you could fix it if you knew how to code (speaking for myself).

 

[slaeshjag]

And now it happened again, through an outdated version of nginx. So I got a rootkit brute forcing italian VoIP servers. Fun. On top of that, my servers sysdisk had a borken boot secor, so I couldn't get the new system to boot. I've now spent 8 hours on this, and from now on, I'll keep an eye out on every f-ing package installed and every exploit found. This sucks ;_;


Webserver, storage etc. will have to wait until tomorrow (later today, 3.20am <.<)

 

[doublebeta]

I was just about to go to bed, when I just randomly felt I needed to visualize for myself how it would look for users who logged in on my home server (a few friends have a shell account on it.) After logging in, I was just about to close the lid on my laptop when I noticed something strange.


Last login: Fri Mar 11 10:38:21 2011 from 64.206.96.xx


That's odd, I've been home all day, and I didn't wake up until 1pm. So I did a quick lookup on the IP. It's registered to a company in NY, US. That can't be good.


What was he up to on my server? And what was he doing with my account? Well, since I know what I typically do and don't, I was able to extract it from my .bash_history

Spoiler

w


ls


cat sl.py


ls


cat isbigger.c


cat config


ls -la


cat cowsay


cat .bash_history


nano .bash_history


sudo


ls


cat fortune.txt


ls


cat stream.sh


cat xorg.conf


ls -la


cat .nano_history


cd code


ls


cat socket-hidden-server.c


ls


./socket-hidden-server


cd pub/


ls


cat posix_tcp_ip_sockets.txt


w


strings /usr/sbin/sshd|grep %s:%s -B2|head -n 1|grep -v port


ps ax


cat /proc/cpuinfo


w


tmux


tmux --help


tmux --morte


tmux --more


socklist


netstat -tan


su


tmux attach


tmux attach


w


uname 0a


uname -a


ls


pwd


nc


nc -vvlp 2121

Spoiler

Now, this is just rude. Not only does he poke around in my files, HE ATTACHED MY TMUX! This is the point where it all became personal. I did the bad practice of having a root terminal up, but it was following a log, so he probably didn't notice it. But it also had my irssi etc.. That's not very nice. This is the point where I'm thankful for my ISPs bad peering to Germany, my session at a friends server had already been dropped by the point he logged in.


but how did he get in? I grepped his IP in auth.log, but there was only one login attempt, the one he got in on. Grepped on my username that day, only one login before I woke up.


Mar 11 10:38:21 Choksondik sshd[25726]: Accepted publickey for steven from 64.206.96.xx port 45381 ssh2


Publickey? WHAT? And it doesn't even seem like he had to guess the username, I'm pretty sure I didn't leave my public/private key pair on the internetz


Is this a hole in the ssh server that debian hasn't patched? What the hell?


Mar 11 10:38:21 Choksondik sshd[25726]: pam_unix(sshd:session): session opened for user steven by (uid=0)


Mar 11 10:39:35 Choksondik sshd[25726]: pam_unix(sshd:session): session closed for user steven


He didn't stay very long either, is this a scripted bot, or is he some sort of super human? Just look at everything he did during that minute fourteen.


Needless to say, I've closed down external access to my SSH server until I know what the hell is going on.


Now I'm just feeling... Dirty, violated... The though of somebody accessing my private shell/fileserver... It's hard to describe, I just feel like I need revenge.


What would you do in a situation like this?

Why someone would deliberately target YOU, I don't know. You've done nothing wrong, which leaves me to believe you're being used as a bounce. Back up your personal stuff and reinstall.


Hmm, either you slurred out your keys in your sleep "7F...E8...D3...20", or _something_ got owned. It needn't be SSH. Your webserver, or something, could have been compromised - if the intruder had gained root, it's easy to get your keys, by that point. Remote shells spawned via shellcode are hard to use, since a pseudo-terminal isn't made. So the keys were probably copied out and then used to login normally.


(From Alec):


>Sounds to me like the intruder managed to write his key into ~/.ssh/authorized_keys


Or that, yeah.


Personally I have login via keys denied. Because I don't use them, I use passwords. Because I don't trust myself noit to lose, like, my laptop or something, and then let whoever finds it have a shell. BUt I'm just super paranoid.


My sudo-group pass is about as long as a key anyway .


I use denyhosts, too. Denyhosts, rkhunter, snort. Don't go without them.


How would I feel in your situation? Somewhat...pleased. Happy to know that there are still exploits that aren't PHP-related. Means I might still have game . I'd be annoyed that people are going through my crap - that's why I encrypt my important stuff, and just DON'T keep stuff on an internet facing server, and concerned for others' safety, hoping that my box wasn't used to launch exploits, D(D)oS, brute force, or distribute contraband.


But I would enjoy the excitement of seeing an unauthorised login, but that's only because I'm kinda messed up in the head.


That's not to say that I want to be owned.


BTW, Anyone else been getting loads of connections from no-ip.com? Denyhosts blocked it in a hurry.


Zero For 0wned 'hint' at having an ssh 0day. In zf03(rather old though), during a dump of IRC logs:


"16:18 <&SHAGGSTaRR> ssh vuln? #YES 0DAY LOL"


So SSH _may_ be vulnerable. But so you know, as long as you don't script kiddie or whitehat by night, it's probably NOT zf0 themselves. But who knows? Grep zf06 when it comes out.


What would I do in your situation? Reinstall, ASAP.


EDIT: Oops. you made a new post. Yeah. Reinstall, asap. When you reinstall, install denyhosts, rkhunter, snort, and put a script to mail you a 'ps -aux --forest', 'w', and 'netstat' twice daily in Cron. It's what I do. And for god's sake, never use root! Move to using a user account and sudo. Use keys for your login, if you will, but use a password to become root. If you really really need to actually be root, do a 'sudo su'. Of course, _disable_ the root account, and deny root logins in your sshd.


How old is your nginx? I saw in an advisory that the version I used to run, which was latest in the ubuntu repos, is vulnerable to buffer overflow - IIRC it was in the non-root process, not the root process which binds port 80, so the worst an attacker could do without a local privilage escalation is deface the site. On my server, not even that would work, since all the pages belong to root. I replaced the binary with one built from the latest stable sources.


I'm not a security expert. These are just a few tips. Good Luck!


EDIT: If you are near them IRL, call your friends with shell into a darkened room and question them. Btw, you didn't give any of them root, right??

 

[dreamer]

I use passwords. Because I don't trust myself noit to lose, like, my laptop or something, and then let whoever finds it have a shell. BUt I'm just super paranoid.
My sudo-group pass is about as long as a key anyway .

The problem with this are MITM-attacks. You will never know if someone is grabbing your PW. At least with ssh-keys any PW-prompt means there is a MITM going on.