Initial Notification: 13th May 2022, 7:48 PM.
Attacker IP: 194.145.227.21
Attacker Port: 5443
On careful Observation, an unknown process is utilising a lot of CPU, which is more than 70%. The process is named kthreaddk, which is invoked by git user. We try to find the path using its CURRENT PID and the executable is already deleted. We found the remote server which our instance was connected to, with netstat utility. It has established a TCP connection to 194.145.227.21 as Destination IP and 5443 as Destination Port.
VirusTotal Report on IP: https://www.virustotal.com/gui/ip-address/194.145.227.21/details
This leads to confirmation that the instance was a part of Crypto Mining Botnet named after Sysrv-Hello. After thorough research, we came to an understanding that Gitlab, which was deployed in that instance, has a very bad and popular Vulnerability, which helps to gain RCE. Link to official CVE. The kthreaddk process, which is invoked by git is executing a binary file, which is throttling the max CPU usage. Even killing the process spawns it again. This actually means, when you delete the file currently in use, it gets marked deleted and is no longer listed but can still be used until the process currently having it open closes it.
We can find the executable crash info in crash logs. What are the files located in `/var/crash/`? – Unix & Linux Stack Exchange. The malware runs cron jobs for the git user every second at different paths, unable to trace and kill the process.
As suggested by official gitlab,if this RCE vulnerability was exploited on an instance, it’s possible that abuse or malicious user access to the system may persist even after upgrading or patching GitLab. The best and efficient way to mitigate this is to start a new instance from scratch and restore the gitlab from the most recent version of backup before the attack.
Guides: Accurate One : https://notes.netbytesec.com/2021/11/analysis-of-compromised-for-gitlabs-cve.html
AWS Suggested: https://cujo.com/the-sysrv-botnet-and-how-it-evolved/ Reddit: Sysrv Botnet Mining Malware Analysis (kthreaddk) : r/MalwareAnalysis How to to trace malware orignation on ubuntu server and stop it – Stack Overflow kdevtmpfsi using the entire CPU – Stack Overflow CVE-2021-22205: How to determine if a self-managed instance has been impacted – How to Use GitLab high CPU usage (#345091) · Issues – GitLab.org How to to trace malware orignation on ubuntu server and stop it – Stack Overflow Sysrv Botnet Mining Malware Analysis (kthreaddk) : r/MalwareAnalysis Kinsing malware (kdevtmpfsi)- how to kill kdevtmpfsi using the entire CPU – Stack Overflow Sysrv-Hello Expands Infrastructure.