Rmmod nf_conntrack high usage

1

image
image1316×86 103 KB

What is that and why would it be using too much CPU?

It was fine a couple of hours ago. This is a MySQL server.

2

My first thought is compromised server and that isn’t doing what it wants you to think it’s doing.

3

Oh shit. Seeing the TIME columns, it’s weird.

4

strace -s 1024 -f -p 5057 to see if there’s something anomalous. With @Jarland you got rooted. An strace for sure will give you an idea what type of system calls it’s making. Otherwise check dmesg for a potential kernel panic.

Edit: lsof -p 5057 might give you a better idea what files it has open, where it’s cwd is, and so on as well.

4 Likes
5

yep, lsof is the way

and after identifying the filenames, use fuser to get some more info

1 Like
6

Thank you both. Yes, learned about lsof here at HostBalls :smiley:

Output:

[root@dal03 ~]# lsof -p 5057
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF      NODE NAME
rmmod   5057 root  cwd    DIR  253,2     4096         2 /
rmmod   5057 root  rtd    DIR  253,2     4096         2 /
rmmod   5057 root  txt    REG  253,2   146680     11217 /usr/bin/kmod
rmmod   5057 root  mem    REG  253,2   142144      4153 /usr/lib64/libpthread-2.17.so
rmmod   5057 root  mem    REG  253,2  2156240      4125 /usr/lib64/libc-2.17.so
rmmod   5057 root  mem    REG  253,2    88776     17016 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
rmmod   5057 root  mem    REG  253,2    90248      4451 /usr/lib64/libz.so.1.2.7
rmmod   5057 root  mem    REG  253,2   157424      5297 /usr/lib64/liblzma.so.5.2.2
rmmod   5057 root  mem    REG  253,2   163312      2307 /usr/lib64/ld-2.17.so
rmmod   5057 root    0r  FIFO    0,9      0t0 315805920 pipe
rmmod   5057 root    1w  FIFO    0,9      0t0 315805921 pipe
rmmod   5057 root    2w  FIFO    0,9      0t0 315805921 pipe

/usr/bin/kmod is an old file according to ls.

I found this binary at /root/ created today.

-rw------- 1 root root 1024 Jun 6 10:26 .rnd

HOOOWWWW?
WHYYYYYYYY?

7

I see the .rnd was generated by OpenSSL. I run openssl dhparam today.

I didn’t see anything interesting with fuser .

Rebooted the server and now it’s idling.

Fuck.

I’m going to rebuild it anyway and put more restrictions. If it got hacked, I don’t have any idea how it happened, it was running mariadb service only.

8

@imok didn’t you write something about trying to install firewalld on ovz or lxc or the likes? is that the same system?
rmmod should do exactly what the name suggests, removing kernel modules. in this case nf_conntrack …
in context with you maybe fiddling around with different firewall software that might make sense and it probably just got stuck along the way?
so while I agree it’s always good to be cautious this doesn’t look that worrysome to me :wink:

4 Likes
9

I was about to reply … but @Falzo beat me to it … :slight_smile:

3 Likes
10

No, it’s another machine. This is KVM.

But yes, I was playing with firewalld around the hour problems started.

I think something just got stuck. I reinstalled the server anyway.

1 Like