The following post by 0xb0b is licensed under CC BY 4.0
The task provides us with the following information and a zip containing the memory dump of a Linux server:
The incident response team has alerted you that there was some suspicious activity on one of the Linux database servers.
A memory dump of the server was taken and provided to you for analysis. You advise the team that you are missing crucial information from the server, but it has already been taken offline. They just made your job a little harder, but not impossible.
Click on the Download Task Files button at the top of this task. You will be provided with an evidence.zip file.
Extract the zip file's contents and begin your analysis in order to answer the questions.
Note: The challenge is best done using your own environment. I recommend using Volatility 2.6.1 to handle this task and strongly advise using this article by Sean Whalen to aid you with the Volatility installation.
Volatility 3 was already installed on one of my instances, with which I initially solved the tasks, but was not able to do so fully, because, as far as I know, this extraction of files is not yet supported in Volatility 3.
When analyzing the dump with imageinfo for the first time, there were no results at first. By issuing the command vol.py --info, I notice that I do not have any profiles for Linux. And that's the entire point of the challenge—getting a profile on your own.
Creating A Profile
We need a profile to analyze a memory dump because it contains essential debugging symbols and system-specific information that guide Volatility 2 in correctly interpreting the unique structures and data within that specific operating system’s memory. Without the right profile, the tool cannot accurately map and decode the raw binary data from the memory dump, leading to potential misanalysis or incomplete forensic results.
Without needing a profile, we can run banners with Volatility Version 2 or 3, to get the necessary information: the Linux kernel in use and the operating system. This is necessary since we need a System running on those specific aspects to build the profile.
We are dealing with Ubuntu 20.04.2 running on the 5.4.0-166-generic kernel.
python3vol.py-f../linux.membannersVolatility3Framework2.7.0Progress:100.00PDBscanningfinishedOffsetBanner0x2f9c4c88 Linux version 5.4.0-166-generic (buildd@lcy02-amd64-011) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)) #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 (Ubuntu 5.4.0-166.183-generic 5.4.252)
0xa707c8c8 Linux version 5.4.0-166-generic (buildd@lcy02-amd64-011) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)) #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 (Ubuntu 5.4.0-166.183-generic 5.4.252)
0xd46001a0 Linux version 5.4.0-166-generic (buildd@lcy02-amd64-011) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)) #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 (Ubuntu 5.4.0-166.183-generic 5.4.252)
0xd619de54 Linux version 5.4.0-166-generic (buildd@lcy02-amd64-011) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)) #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 (Ubuntu 5.4.0-166.183-generic 5.4.252)
0x106d64c88 Linux version 5.4.0-166-generic (buildd@lcy02-amd64-011) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)) #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 (Ubuntu 5.4.0-166.183-generic 5.4.252)
Let's download an ISO of Ubuntu 20.04 and set up a VM.
After having the VM setup, we see we have the wrong kernel installed. No problem.
But first, let's set up Volatility 2 on that VM, since we want to have a fresh installation of Volatility 2 that won't mess up our dependencies on our Workstation. Furthermore, Volatility 2 is required on the VM anyway to create the profile.
$uname-r5.15.0-105-generic
Volatility 2
The following section describes the installation of Volatility 2 by Sean Whalen.
Installing Volatility 2
We stay on our new setup vm and install Volatility 2
We need to build a kernel module that Volatility will use to extract the necessary symbols. This requires us to have the correct Linux headers for the Linux kernel 5.4.0-166-generic. We install it with the other requirements, like dwarfdump. The Makefile to craft such a module can be found inside Volatility's Linux tools directory.
Next, we just edit the Makefile in volatility/tools/linux which builds that said module. The original one will build it for the current running kernel, but we want to avoid the hassle of downgrading the kernel and just hard-code the path to /lib/modules/5.4.0-166-generic/build.
Next, we actually have to install the kernel, but it is not required to have it running. We need the System.map file that corresponds to the kernel. This file contains all the symbols and their addresses in the kernel, and it's used alongside the DWARF information. Typically, this file can be found in /boot:
After installing the kernel, we can copy /boot/System.map-5.4.0-166-generic to our current directory. Next, we zip the System.map file and the compiled module.dwarf and place the zip archive at volatility/plugins/overlays/linux/.
The profile creation of Volatily 3 is a bit different, our target VM ist still required but a bit more approachable with the tool dwarf2json. To solve this Challenge Volatilty 2 is required; if a solution with Volatilty 3 is possible, it can also be found here as an alternative command.
The following resource gives a detailed description of how to create a profile for Volatility 3:
A comprehensive guide on how to use Volatility to analyze a Linux memory dump can be found here:
What is the exposed root password?
The first question is about an exposed root password. There are several places you could look, such as environment variables, files with passwords or the bash history. We find what we are looking for in the bash history, which we can restore using linux_bash.
linux_bash - Recover bash history from bash process memory
In addition to the root password, we can also directly answer the second question of when the users.db was accessed.
But there is more to discover. Furthermore, a reverse shell in c was possibly obtained from the source IP 10.0.2.72, which was subsequently compiled to pkexecc. This will be useful in answering further questions.
The next question asks for a suspicious file, which will probably be the pkexecc. We need to determine the MD5 hash for this file; this requires that we extract the file. This is probably only possible using Volatility 2. If it does work with Volatilty 3, please let me know.
To be able to extract the file, we first need to know exactly where it is located. We can find this out using linux_enumerate_files and look at all entries using the grep filter with the name pkexecc. We find the file pkexecc located in /home/paco/.
linux_enumerate_files - Lists files referenced by the filesystem cache
The entry also contains the offset where this can be found exactly. We now need this to recover the file using linux_find_file. After we have restored the file, we determine the MD5 hash using md5sum.
linux_find_file - Lists and recovers files from memory
What is the IP address and port of the malicious actor? Format is IP:Port
We already got a hint about the IP of the malicious actor from the bash history: 10.0.2.72. The malicious file has already been confirmed by answering the previous questions. With linux_netstat we can view all open connections. We filter the output directly based on the IP we already know and find what we are looking for.
linux_netstat - Lists open sockets
~/volatility$ python2 vol.py -f ../linux.mem --profile=Linux5_4_0x64 linux_netstat |grep10.0.2.72VolatilityFoundationVolatilityFramework2.6.1TCP10.0.2.73:2210.0.2.72:53888ESTABLISHEDsshd/1002TCP10.0.2.73:2210.0.2.72:53888ESTABLISHEDsshd/1075TCP10.0.2.73:4101210.0.2.72: [REDACTED] ESTABLISHED sh/1093 TCP10.0.2.73:4101210.0.2.72: [REDACTED] ESTABLISHED sh/1093 TCP10.0.2.73:4101210.0.2.72: [REDACTED] ESTABLISHED sh/1093 TCP10.0.2.73:4101210.0.2.72: [REDACTED] ESTABLISHED sh/1093 TCP10.0.2.73:4101210.0.2.72: [REDACTED] ESTABLISHED su/1095 TCP10.0.2.73:4101210.0.2.72: [REDACTED] ESTABLISHED su/1095 TCP10.0.2.73:4101210.0.2.72: [REDACTED] ESTABLISHED su/1095 TCP10.0.2.73:4101210.0.2.72: [REDACTED] ESTABLISHED su/1095 TCP10.0.2.73:4101210.0.2.72: [REDACTED] ESTABLISHED bash/1097 TCP10.0.2.73:4101210.0.2.72: [REDACTED] ESTABLISHED bash/1097 TCP10.0.2.73:4101210.0.2.72: [REDACTED] ESTABLISHED bash/1097 TCP10.0.2.73:4101210.0.2.72: [REDACTED] ESTABLISHED bash/1097 TCP10.0.2.73:2210.0.2.72:57690ESTABLISHEDsshd/1099
What is the full path of the cronjob file and its inode number? Format is filename:inode number
That was a bit tricky, with linux_enumerate_files we can localize the cronjob, just like with the file pkexecc, but we also have to know its inode number, unfortunatley the one listed here is not the correct one.
But we can list all file descriptors and their paths, which also displays the inode number. One path stands out here very often, the one with the anon_inode, which is referring to the cronjob.
linux_lsof - Lists file descriptors and their path
