a suspicious process named 'kdevtmpfsi'，likely related to redis offical image 'redis:4-alpine' in docker hub

[biello]

I notice that there is a kdevtmpfsi executable file under my redis:4-alpine container fs /tmp and a process named 'kdevtmpfsi' cost about 400% cpu, when i remove redis container and run again the file is gone. is this normal? seems like being hijacked to mine cryptocurrency or something.

[iamareebjamal]

Remove the added cron and /tmp/zzz.sh kdevtmpfsi and search kinsing and delete every folder containing those processes. Now kill the process and restart

[wglambert]

Yeah that file isn't supposed to be there
A search query for that filename says it's malicious

#140 (comment)

As noted by the creator of redis; it is very easy to "hack" a redis server and change files: http://antirez.com/news/96.

I successfully gained access as the Redis user, with a proper shell, in like five seconds. Courtesy of a Redis instance unprotected being, basically, an on-demand-write-this-file server, and in this case, by ssh not being conservative enough to deny access to a file which is all composed of corrupted keys but for one single entry. However ssh is not the problem here, once you can write files, even with binary garbage inside, it’s a matter of time and you’ll gain access to the system in one way or the other.

We do not build images with malware installed (I did just pull and check them in case there was some sort of exploit of Docker Hub). The affected users are likely exposing their redis to the public internet where it is trivial to "hack".

Edit: previous users with problems by "hackers", #44 (comment), #44 (comment), #44 (comment)

[iamareebjamal]

This is not a redis issue. This is a known crypto mining malware unrelated to redis

[tianon]

See also docker-library/docs#1571 where this was officially documented.

Also, just to be fully explicit:

$ docker pull redis:4-alpine
4-alpine: Pulling from library/redis
Digest: sha256:cc33d01d0c96e498284886d2afc0a5d09250334efd61dd430e5edce9b89a9a3c
Status: Image is up to date for redis:4-alpine
docker.io/library/redis:4-alpine

$ docker run --rm redis:4-alpine find / -iname kdevtmpfsi
$ 

[andrerfneves]

@iamareebjamal happen to have any more info/links on that? Can't seem to find anything

[iamareebjamal]

https://r.virscan.org/report/07256aabf45b690a8a3989d0efa38e18
https://stackoverflow.com/questions/59487096/kdevtmpfsi-how-to-find-and-delete-that-miner
https://forums.unraid.net/topic/86241-help-cpu-goes-to-100-all-cores/
linuxserver/docker-nextcloud#125
laradock/laradock#2451
Gerapy/Gerapy#130

[andrerfneves]

thanks!

[derakhshanfar]

Guys the same issue for me. do you have any workaround?

[iamareebjamal]

Remove /tmp/kdevtmpfsi, /tmp/zzz and /var/tmp/ executables and replace with blank files with no permissions, then the miner cannot readd the files, then kill the running process

[iamareebjamal]

Or follow this - laradock/laradock#2451 (comment)

[derakhshanfar]

@iamareebjamal A big thank you to you . I did something like this:

touch /tmp/kdevtmpfsi && touch /var/tmp/kinsing
echo "everything is good here" > /tmp/kdevtmpfsi
echo "everything is good here" > /var/tmp/kinsing
cat /tmp/kdevtmpfsi && cat /var/tmp/kinsing

chmod go-rwx /var/tmp
chmod 1777 /tmp

but after a couple of mitues it started again !

[iamareebjamal]

/tmp/zzz is the bootstrap file. Do the same with it as well

[derakhshanfar]

@iamareebjamal do you mean something like this?

touch /tmp/zzz
echo "everything is good here" > /tmp/zzz
chmod go-rwx /var/tmp
chmod 1777 /tmp

[tianon]

You also should change your container to not expose the database port publicly.

[itamarhaber]

And, perhaps above all, require a strong password to connect.

[paulosouzainfo]

this solve for me.
docker stop _your-redis-container_
find / -iname kdevtmpfsi --exec rm -fv {} \;
docker start _your-redis-container_

[StevenDiz]

So, if you aren't even using Docker, and this thing has affected you, it might be due to an unpatched PHP Remote Code Execution vulnerability. (We don't use Docker, got hit with this thing and couldn't get it to stop trying to run the script that reinstalls the CRON job, even after we locked it down otherwise.) In any case, this is the vulnerability you need to check. Hope this helps someone!

https://www.sourceclear.com/vulnerability-database/security/remote-code-execution-rce/php/sid-4487

[jzdayz]

this solve for me.
docker stop _your-redis-container_
find / -iname kdevtmpfsi --exec rm -fv {} \;
docker start _your-redis-container_

It's work for me !

[16g]

Do not expose Redis ports to host without Iptables rules!

kill - 9 miner_PIDs_here
cd /tmp
rm -f kdevtmpfsi
rm -f kinsing
crontab -l
find / -iname kdevtmpfsi
find / -iname kinsing
chmod -R 700 /tmp
remove output files

[CKmaenn]

maybee this works...

`#!/bin/bash

kill the tmp dirs

chattr -iR /tmp/ 2>/dev/null
tntrecht -iR /tmp/ 2>/dev/null
rm -fr /tmp/* 2>/dev/null
rm -fr /tmp/.* 2>/dev/null
chattr -iR /var/tmp/ 2>/dev/null
tntrecht -iR /var/tmp/ 2>/dev/null
rm -fr /var/tmp/* 2>/dev/null
rm -fr /var/tmp/.* 2>/dev/null

set recursive chattr -i

chattr -iR /etc/ 2>/dev/null
tntrecht -iR /etc/ 2>/dev/null
chattr -iR /root/ 2>/dev/null
tntrecht -iR /root/ 2>/dev/null

find & kill kinsing* binarys

find / -name 'kinsing*' -exec rm -rv {} ;
find / -name 'kdevtmpfsi' -exec rm -rv {} ;

kill kinsing procs

pkill -f kinsing
pkill -f kdevtmpfsi

clean crontabs/jobs

CRONCLEAN=("/var/spool/cron/" "/var/spool/cron/crontabs/" "/etc/cron.d/" "/etc/cron.daily/" "/etc/cron.hourly/" "/etc/cron.monthly/" "/etc/cron.weekly/")
tntrecht -i /etc/crontab 2>/dev/null
chattr -i /etc/crontab 2>/dev/null
crontab -r 2>/dev/null 2>/dev/null
echo " " > /etc/crontab 2>/dev/null
for crondir in ${CRONCLEAN[@]}; do
chattr -iR $crondir
tntrecht -iR $crondir
find $crondir -type f -exec grep -q -i ‘curl’ ‘{}’ ; -delete
find $crondir -type f -exec grep -q -i ‘wget’ ‘{}’ ; -delete
find $crondir -type f -exec grep -q -i ‘cdl’ ‘{}’ ; -delete
find $crondir -type f -exec grep -q -i ‘wdl’ ‘{}’ ; -delete
done`

[amulcse]

I found all the above solution helpful but all of them are seems the temporary solution as we need to monitor the instance and if we notice any malicious activity then do the same process again.

I have come across this virus around 1 month back and applied all the solution above which works fine for the limited period after that, it will come again.

Even I didn't install docker in the system so docker open API port was not an issue.

But there are some open source software which are the open gate for kinsing.

PhpMailer and Solr have some Remote Code Exec vulnerability cause the whole issue.

Easy solution is upgrade your Solr version to 8.5.1 and there one more thing you can set as security which will 100% remove the virus and it will be the permanent.

Here is the full explanation: https://github.com/amulcse/solr-kinsing-malware

[CKmaenn]

I found all the above solution helpful but all of them are seems the temporary solution as we need to monitor the instance and if we notice any malicious activity then do the same process again.

I have come across this virus around 1 month back and applied all the solution above which works fine for the limited period after that, it will come again.

Even I didn't install docker in the system so docker open API port was not an issue.

But there are some open source software which are the open gate for kinsing.

PhpMailer and Solr have some Remote Code Exec vulnerability cause the whole issue.

Easy solution is upgrade your Solr version to 8.5.1 and there one more thing you can set as security which will 100% remove the virus and it will be the permanent.

Here is the full explanation: https://github.com/amulcse/solr-kinsing-malware

kinsing is not only distributed via docker api, it penetrates massively via redis, docker api, kubernet, rfi e.t.c .... into the systems! to avoid reinfect you could try the following (yes not the cleanest way but it works) :::

• Deactivate cron (d)
• set file dumys for kinsing, kdevtmpfsi
• don't run your damn applications under root !!!!!

if in doubt contact me and let me check your system from the outside. can you hack it then I hack it and record my procedure / the vulnerable. (I am a private person not a company !!!)

PS: amulcse did it come back under the same name? were the files in the same places? if so then they have not considered the variant with the file dumys. I would be happy to receive further information. please take a look at these candidates here, not as common as kinsing but also current and dangerous: https://github.com/CKmaenn/CryptoHijacking_Binarys.and.InstallScripts_sorted

have a nice day ;)

[amulcse]

I found all the above solution helpful but all of them are seems the temporary solution as we need to monitor the instance and if we notice any malicious activity then do the same process again.
I have come across this virus around 1 month back and applied all the solution above which works fine for the limited period after that, it will come again.
Even I didn't install docker in the system so docker open API port was not an issue.
But there are some open source software which are the open gate for kinsing.
PhpMailer and Solr have some Remote Code Exec vulnerability cause the whole issue.
Easy solution is upgrade your Solr version to 8.5.1 and there one more thing you can set as security which will 100% remove the virus and it will be the permanent.
Here is the full explanation: https://github.com/amulcse/solr-kinsing-malware

kinsing is not only distributed via docker api, it penetrates massively via redis, docker api, kubernet, rfi e.t.c .... into the systems! to avoid reinfect you could try the following (yes not the cleanest way but it works) :::

• Deactivate cron (d)
• set file dumys for kinsing, kdevtmpfsi
• don't run your damn applications under root !!!!!

if in doubt contact me and let me check your system from the outside. can you hack it then I hack it and record my procedure / the vulnerable. (I am a private person not a company !!!)

PS: amulcse did it come back under the same name? were the files in the same places? if so then they have not considered the variant with the file dumys. I would be happy to receive further information. please take a look at these candidates here, not as common as kinsing but also current and dangerous: https://github.com/CKmaenn/CryptoHijacking_Binarys.and.InstallScripts_sorted

have a nice day ;)

Hey CKmaenn.

As your suggested solution might work but they will try always the different directory and anonymous name of the folder. So basically you should identify the entry gate instead of changing the permission of the folder and files.

One more thing, even it will not harm by changing the permission still it will annoyingly try to access the system in different ways.

Yes, It did not come in my system again as I knew the entry point.

Thanks

[mooleshacat]

I noticed a bunch of commands spread out amonst many different comments... Here is what I have found helpful and working:

• service redis stop
• kill -9 $(pidof kdevtmpfsi)
• kill -9 $(pidof kinsing)
• rm -f /tmp/kdevtmpfsi ; echo "ALL_GOOD_HERE" > /tmp/kdevtmpfsi
• rm -f /var/tmp/kinsing ; echo "ALL_GOOD_HERE" > /var/tmp/kinsing
• rm -f /tmp/zzz ; echo "ALL_GOOD_HERE" > /tmp/zzz
• rm -f /tmp/zzz.sh ; echo "ALL_GOOD_HERE" > /tmp/zzz.sh
• chmod -R go-x /tmp
• chmod -R 0777 /tmp/
• chmod -R go-x /var/tmp
• chmod -R 0777 /var/tmp/
• optionally complete steps below to install/configure/enable UFW
• service redis start

Immediately after performing the above steps, I see the process disappear and usage return to normal. I have been running now 6 hours with no more usage issues.

If you MUST leave open your redis without password - say you use an application that does not support passwords - you can clean up the miners as described above and do the following:

• ensure your router port is forwarding 6379 to your local PC if you have a router involved
• apt install -y ufw
• ufw reset if you need to wipe the rules
• /usr/sbin/ufw default deny incoming ;
• /usr/sbin/ufw default allow outgoing ;
• /usr/sbin/ufw allow from 192.168.1.0/24 ; to allow local IP's (change if you are different from 192.168.1.1 - 192.168.1.254)
• ufw allow from <insert your home IPV4 address here>
• ufw allow from <insert your home IPV6 address here>
• repeat last two steps for every server IP that needs to access ssh/redis
• ufw enable press y to confirm

You can also allow/deny other common services to the world:

• ufw allow 53 for DNS
• ufw allow 80 for HTTP
• ufw allow 443 for HTTPS

Note: if you allow your IP you will want to remove the allow 22 or deny 22 rule with ufw delete allow 22 and ufw delete deny 22 otherwise it still processes those rules and either allows world or blocks world (including you)

This is by no means considered a 'secure' way to fix the issue, the secure way is to recode the program to work with a redis password. This is because if the firewall is disabled for any reason, the server is immediately insecure again and can be breached.

This is however a temporary fix for those who are not programmers.

[hrasekh]

As I have Redis Docker, first, i restart Redis docker container, and find out the _kdevtmpfsi_ stop from working but after some minutes it start again.

I went to /var/lib/docker/ and:

sudo find / -iname kdevtmpfsi

I get the following result:

/var/lib/docker/aufs/mnt/ab584efd38b6a47a8c99a0cab29f98b83d38c45aa8e458bd65ddde80337fb744/tmp/kdevtmpfsi
/var/lib/docker/aufs/diff/ab584efd38b6a47a8c99a0cab29f98b83d38c45aa8e458bd65ddde80337fb744/tmp/kdevtmpfsi

So my kdevtmpfsi was in docker tmp instead of system local /tmp
To solve my problem, I run the following commands:

cd /var/lib/docker/aufs/mnt/ab584efd38b6a47a8c99a0cab29f98b83d38c45aa8e458bd65ddde80337fb744/
rm tmp/kdevtmpfsi*
rm var/tmp/kdevtmpfsi*
chmod 000 tmp
chmod 000 var/tmp
sudo kill -9 $(pidof kdevtmpfsi)

[tianon]

That typically means your Redis server is exposed to the public internet, and you really should fix that problem ASAP.

[Vendenisko]

If other solutions not working. Use it.

<?php

while(true) {
system("kill $(ps aux | grep 'kinsing'  | awk '{print $2}')");
system("kill $(ps aux | grep 'kdevtmpfsi' | awk '{print $2}')");
sleep(1);
}

?>

And add it to screen by command:
screen -AdmS miner_killer php <filename>
I know better idea is remove problem completely but if other ideas not working.
You can use my solution.

[ammarfaizi2]

@Vendenisko your script will consume high CPU usage too, it may be better to put sleep(1) inside the "while loop" for CPU relax.

[Vendenisko]

I'm sorry, my mistake. I will edit my comment and add sleep to loop.

[yosifkit]

You really, really ought to fix it by stopping the remote access from unauthorized users. Just stopping their mining process means that they still have access to your redis database and can do anything with the data it contains.

The two simplest ways should be to not expose the port to the world (no -p on docker run) or add a password/new ACL support.

[Vendenisko]

Yes you are right, I said better is remove the problem completely. Not use my solution all time. This is a temporary solution. Or if you don't have better ideas, or you can't use another solution.

[Vendenisko]

My friend has this problem, temporary used my simple PHP script, then I logged to his machine I removed it completely.

[M4T14SG4M3R]

My friend has this problem, temporary used my simple PHP script, then I logged to his machine I removed it completely.

How did you removed it completely?

[JEAholding]

OK, so I would love for someone to explain what is happening in my case...

No docker image, no Redis installed, no Apache Solr installed, no PHPMailer installed and PHP is installed but is latest 7.4... Where is the vulnerability in this case?

Miner gets reinstalled on average every 2 days, NO CRONJOBS gets installed (from what I can see) and I have authorized keys to enter the SSHD...

Saw a couple of strange PORT open with ss -plntu = I kill the process that had them open.

How is the miner getting installed in my case I only have 80,443 and 25 ports open. Must be an apache vulnerability or could it be Centos 7?

[StevenDiz]

OK, so I would love for someone to explain what is happening in my case...

JEAholding,

Your situation sounds similar to ours. In our case it was a Remote Code Execution vulnerability in PHP Unit. Basically, it (PHP Unit) included a script that would accept a php file upload and execute it. That was how the miner was getting reinstalled on our box. We removed PHP Unit and the problem went away.

(PHP Unit can be part of other 3rd party utilities and can get installed without you realizing it. That's what happened to us.)

Hope this helps!

[JEAholding]

OK, so I would love for someone to explain what is happening in my case...

JEAholding,

Your situation sounds similar to ours. In our case it was a Remote Code Execution vulnerability in PHP Unit. Basically, it (PHP Unit) included a script that would accept a php file upload and execute it. That was how the miner was getting reinstalled on our box. We removed PHP Unit and the problem went away.

(PHP Unit can be part of other 3rd party utilities and can get installed without you realizing it. That's what happened to us.)

Hope this helps!

I have the latest PHPunit 8.5^ wasn't this fix on later versions? Or phpunit is vulnerable even on latest updated version ? What version of phpunit did you have on your box ?

[StevenDiz]

@JEAholding - Honestly, I don't remember. (We resolved this about a year ago.) I would hope that it had been resolved in PHPUnit, but it could always be a similar issue in another library. :/ Sorry I can't be more help there.

[yosifkit]

Apologies for locking the conversation. This is not a general discussion board for issues of being hacked and kdevtmpfsi being installed. Please use a dedicated support forum.

If you have real bugs with the redis Docker image, then please file an issue.