Table of Contents
Citrix issued an alert (10/10/2023) about a critical vulnerability (CVE-2023-4966) in all NetScaler (Citrix ADC) & Gateway systems. Several working exploits have been published.
Please note that simply updating the systems is not enough. The connection tokens must also be reset.
Important ! There are no patches for NetScaler (Citrix ADC) version 12.1 or older. These systems have reached their EOL and will therefore no longer be equipped with the necessary fix. In this case please update to the latest 13.0, 13.1 or 14.1 version.
The vulnerability allows anonymous remote code execution and thus unauthenticated attackers to take over various machines with root privileges.
It can be assumed that lists created by various institutions, about NetScaler (Citrix ADC) machines on the Internet, are now being attacked.
The output of a script on a system that is not yet patched and/or would be susceptible to Citrix Bleed.
Firmware Updates
Citrix has made a patch available for all supported versions.
| NetScaler (Citrix ADC) und NetScaler (Citrix) Gateway | ||
|---|---|---|
| Version | Refresh Build | Expected Release Date |
| 13.0 | 13.0-92.19 | Patch is here |
| 13.1 | 13.1.-50.23 | Patch is here |
| 14.1 | 14.1.-12.30 | Patch is here |
Instructions for updating the appliances can be found here.
Work after the update
After updating to a version that is no longer affected, all active and persistent sessions must be terminated with the following commands.
|
1 2 3 4 5 |
kill icaconnection -all kill rdp connection -all kill pcoipConnection -all kill aaa session -all clear lb persistentSessions |
Review of the systems
Even systems with the protections in place should be subjected to a thorough review, as it is never possible to say with certainty when attacks may have begun before they are widely deployed.
Here is a summary of articles, Twitter entries and our own experiences regarding the wave of attacks.
Find out the time of the last update
At the moment, an attack can only be detected by overwritten timestamps in certain files. These timestamps usually only change when updating the NetScaler and therefore it is important to know this date. A good indicator is to look at the date of the unpacked installation packages.
All commands should be executed via Putty or CLI.
First, of course, you should log in with nsroot or another administrative account. After that you can view your extracted installation data with the following command.
|
1 |
shell ls -ll /var/nsinstall |
Here you can see the date of the last NetScaler update package (19.07.2023). We should note this, because we need this for the next commands. It is important, that we do not enter the determined date in the following commands, but the determined date in the form YYYYMMDD +1. So in my case 20230720.
Edited files
Now we can check whether certain files have been adjusted since the last update. This would not be proof yet, but a first indication and should be taken seriously.
|
1 2 3 4 |
shell find /var/vpn/ -type f -newermt {Timestamp der Installer Files +1} -exec ls -l {} \; find /var/netscaler/logon/ -type f -newermt {Timestamp der Installer Files +1} -exec ls -l {} \; find /var/python/ -type f -newermt {Timestamp der Installer Files +1} -exec ls -l {} \; |
A common webshell I see in the wild is identified by the file a.php under /var/netscaler/logon. Please also check if this file is present, this is a sign that an attack has taken place.
The files under ns_gui change the timestamp not only when the files are installed, but at every reboot. Therefore, this should be taken into account when checking the timestamp.
|
1 2 |
shell find /netscaler/ns_gui/ -type f -name *.php -newermt {Timestamp der Installer Files +1} -exec ls -l {} \; |
Or with the following Python command.
|
1 2 |
shell python -c "import os, glob, time; newest_timestamp = max(os.stat(f).st_mtime for f in glob.glob('/var/nsinstall/*')); print('\n'.join(os.path.join(dirpath, f) for dir in ['/var/netscaler/logon/', '/var/python/', '/var/vpn/'] for dirpath, _, files in os.walk(dir) for f in files if os.stat(os.path.join(dirpath, f)).st_mtime > newest_timestamp))" |
If nothing has been edited since the last update this is a good sign.
New files (webshell)
We check various directories for new files as well, this would indicate a webshell.
|
1 2 3 4 |
shell ls -ll /var/nsproflog/*.php ls -ll /var/netscaler/gui/*.php ls -ll /netscaler/portal/*.php |
HTTP error log files
There may also be entries in the http error logs during attacks. We are looking for anomalies regarding .sh and .php here.
|
1 2 3 |
zgrep '\.sh' /var/log/httperror.log* zgrep '\.php' /var/log/httperror.log* zgrep '\.pl' /var/log/httperror.log* |
The picture shows that the search for .sh was successful. You can check this directly by looking into the file with an editor of your choice (e.g. vi).
In this case, I just added .sh in the affected file to check my command.
Shell / Bash log files
In addition, attacks leave entries in the shell or bash log files.
|
1 2 |
shell zgrep -E 'database.php|/flash/nsconfig/keys/updated|/flash/nsconfig/keys|/ns_gui/vpn|LDAPTLS_REQCERT|ldapsearch|openssl|/nsconfig/ns.conf|del /etc/auth.conf|cp /usr/bin/bash|.F1.key|.F2.key|nobody' /var/log/sh.log* shell zgrep -E 'database.php|/flash/nsconfig/keys/updated|/flash/nsconfig/keys|/ns_gui/vpn|LDAPTLS_REQCERT|ldapsearch|openssl|/nsconfig/ns.conf|del /etc/auth.conf|cp /usr/bin/bash|.F1.key|.F2.key|nobody' /var/log/bash.log* |
Here in the screenshot you can see only my tests and no attack.
Log files
Check log files for known IOCs.
|
1 |
grep -v '127\.0\.0' /var/log/*.log | grep 'nc -l\|/etc/passwd\|/etc/shadow\|python -c\|curl\|\.php' |
Edited files with the setuid bit
|
1 |
find /var -perm -4000 -user root -not -path "/var/nslog/*" -newermt {Timestamp der Installer Files +1} -exec ls -l {} \; |
Nobody processes
In the context of the Nobody user, new processes were also started during attacks. Here is an example of the allowed processes.
|
1 |
shell ps aux | grep nobody | grep -v '/bin/httpd' |
And an attacked server. The top line is malware.
Check Crontab for new entries
With the following command we can examine the current crontab file. It should look like in the displayed image.
|
1 |
shell grep '' /etc/crontab |
There should be no crontab for nobody. This can be checked with the following command.
|
1 |
shell crontab -l -u nobody |
I have received several community comments that the following entries are also normal:
- /netscaler/adss-licexp.sh –> For trial licenses
- 0/5 * * * * root /netscaler/iprep –> When IP Reputation is in use
- */5 * * * * root /var/python/bin/python /netscaler/aslearn_health_monitor.py –> When app firewall is used
- */5 * * * * root /bin/pgrep -f /netscaler/appfw_dynamic_profiles/appfw_dynamic_profiles.py; [ $? != 0 ] && /var/python/bin/python /netscaler/appfw_dynamic_profiles/appfw_dynamic_profiles.py –> When app firewall is used
NetScaler HA Systems
During various attacks, the process responsible for HA (nsfsyncd) was disabled on NetScaler HA systems.
|
1 |
shell ps aux | grep nsfsyncd |
On standalone machines, the process should not appear either. Shown is the case how it should look like on a working system.
httpaccess-vpn log files
Check log files for successful web accesses to unknown resources.
|
1 |
shell zgrep -E -v 'CitrixReceiver' /var/log/httpaccess-vpn.log* | grep ' 200 ' |
As well as web accesses from HeadlessChrome.
|
1 |
shell zgrep 'HeadlessChrome' /var/log/httpaccess-vpn.log* |
NPPE Core Dumps
In some attacks, a core dump is created via NPPE (NetScaler Packet Processing Engine) after a critical failure in NetScaler. Normally, the following directory should be empty.
|
1 |
shell ls -ll /var/core/1 |
This is what it looks like for an attacked system.
Perl & Python Scripte
Adding custom Perl or Python scripts can also open a backdoor.
|
1 2 |
shell ps -aux | grep python shell ps -aux | grep perl |
If more than the grep queries shown are seen, the further scripts, should be checked by a Google search.
If something appears, run the command again after a few seconds. Some scheduled tasks on the NetScaler also use python or perl. Only if it still runs on the second call, this should be checked.
These are normal scripts for an NetScaler instance hosted in Azure.
When the StoreFront monitor is used, you will permanently see the following Perl script.
Crypto Miners
I have seen several NetScaler instances used as crypto miners after the attack.
With the following command you can see all running processes.
Here, no other process besides NSPPE-xx should be permanently at 100%. A normal state is shown.
|
1 |
shell top -n 10 |
Check network and firewall logs
Checks the network and firewall logs for the following events:
- Scans of the subnets sent by NetScaler for the protocols HTTP / HTTPS / SMB (Port 80 / 443 / 445)
- Spikes in the queries from NetScaler regarding LDAP / LDAPS / DNS / AD (Port 389 / 636 / 53 / 88 / 135 / 137-138 / 464 / 3268-3269) protocols
Countermeasures for affected systems
My recommendation regarding action when the system has been corrupted.
- Remove the NetScaler instance from the network
- Changes the password of all LDAP or other AD / network accounts stored on the NetScaler.
- Issues a new SSL certificate and key file for all client SSL files on the device (the keys are stored in files on the NetScaler, which could theoretically be read by the attacker)
- If it is a VPX appliance and there are snapshots of the appliance (older than 3 months), it may be worth restoring them first, but this is NO GUARANTEE of security
- To be on the safe side, export the configuration file of the affected system and copy / restore it to a new fresh VPX appliance
- Note that the same hardware address is used, otherwise the license is invalid and must be requested / imported again
- Disconnects the network before startup
- Starts the appliance and verifies via the console that the VPX is not compromised
- Changes the nsroot password
- Connect only the internal network
- Connect the external network
- Keeps an eye on the logs and responder hits
- Replaces the SSL certificates on the appliance at the earliest possible time
- Revokes the compromised SSL certificates and SSL keys