Table of Contents
Citrix issued an alert yesterday (07/18/2023) about a critical vulnerability (CVE-2023-3519) in all NetScaler (Citrix ADC) & Gateway systems. To date, no working exploits have been published.
Important ! There are no patches for NetScaler (Citrix ADC) version 12.1 or older. These systems have reached their EOL and will therefore no longer be equipped with the necessary fix. In this case please update to the latest 13.0 or 13.1 version.
The vulnerability allows anonymous remote code execution and thus unauthenticated attackers to take over various machines with root privileges.
As we hear from the Citrix community, more and more attacked systems are being found. The first exploits have also been available for purchase on the dark web for some time.
Thanks to Sander Horsman of Conda Security for the further insights and information.
Assetnote has now reverse engineered a script to detect open systems. In the first step, this only affects machines where SAML is enabled.
The first exploits and scanners are now publicly available via GitHub. Attached is the output of the script from Bryan Smith (@securekomodo).
The output on a patched clean system.
And on a system that is not yet patched.
Firmware Updates
Citrix has made a patch available for all supported versions.
| NetScaler (Citrix ADC) and NetScaler (Citrix) Gateway | ||
|---|---|---|
| Version | Refresh Build | Expected Release Date |
| 13.0 | 13.0-91.13 | Patch is here |
| 13.1 | 13.1.-49.13 | Patch is here |
Instructions for updating the appliances can be found here.
Review of the systems
Even systems with the protections in place should be subjected to a thorough review, as it is never possible to say with certainty when attacks may have begun before they are widely deployed.
Here is a summary of articles, Twitter entries and our own experiences regarding the wave of attacks.
Find out the time of the last update
At the moment, an attack can only be detected by overwritten timestamps in certain files. These timestamps usually only change when updating the NetScaler and therefore it is important to know this date. A good indicator is to look at the date of the unpacked installation packages.
All commands should be executed via Putty or CLI.
First, of course, you should log in with nsroot or another administrative account. After that you can view your extracted installation data with the following command.
|
1 |
shell ls -ll /var/nsinstall |
Here you can see the date of the last NetScaler update package (19.07.2023). We should note this, because we need this for the next commands. It is important, that we do not enter the determined date in the following commands, but the determined date in the form YYYYMMDD +1. So in my case 20230720.
Edited files
Now we can check whether certain files have been adjusted since the last update. This would not be proof yet, but a first indication and should be taken seriously.
|
1 2 3 4 |
shell find /var/vpn/ -type f -newermt {Timestamp der Installer Files +1} -exec ls -l {} \; find /var/netscaler/logon/ -type f -newermt {Timestamp der Installer Files +1} -exec ls -l {} \; find /var/python/ -type f -newermt {Timestamp der Installer Files +1} -exec ls -l {} \; |
A common webshell I see in the wild is identified by the file a.php under /var/netscaler/logon. Please also check if this file is present, this is a sign that an attack has taken place.
The files under ns_gui change the timestamp not only when the files are installed, but at every reboot. Therefore, this should be taken into account when checking the timestamp.
|
1 2 |
shell find /netscaler/ns_gui/ -type f -name *.php -newermt {Timestamp der Installer Files +1} -exec ls -l {} \; |
Or with the following Python command.
|
1 2 |
shell python -c "import os, glob, time; newest_timestamp = max(os.stat(f).st_mtime for f in glob.glob('/var/nsinstall/*')); print('\n'.join(os.path.join(dirpath, f) for dir in ['/var/netscaler/logon/', '/var/python/', '/var/vpn/'] for dirpath, _, files in os.walk(dir) for f in files if os.stat(os.path.join(dirpath, f)).st_mtime > newest_timestamp))" |
If nothing has been edited since the last update this is a good sign.
New files (webshell)
We check various directories for new files as well, this would indicate a webshell.
|
1 2 3 4 |
shell ls -ll /var/nsproflog/*.php ls -ll /var/netscaler/gui/*.php ls -ll /netscaler/portal/*.php |
HTTP error log files
There may also be entries in the http error logs during attacks. We are looking for anomalies regarding .sh and .php here.
|
1 2 3 |
zgrep '\.sh' /var/log/httperror.log* zgrep '\.php' /var/log/httperror.log* zgrep '\.pl' /var/log/httperror.log* |
The picture shows that the search for .sh was successful. You can check this directly by looking into the file with an editor of your choice (e.g. vi).
In this case, I just added .sh in the affected file to check my command.
Shell / Bash log files
In addition, attacks leave entries in the shell or bash log files.
|
1 2 |
shell zgrep -E 'database.php|/flash/nsconfig/keys/updated|/flash/nsconfig/keys|/ns_gui/vpn|LDAPTLS_REQCERT|ldapsearch|openssl|/nsconfig/ns.conf|del /etc/auth.conf|cp /usr/bin/bash|.F1.key|.F2.key|nobody' /var/log/sh.log* shell zgrep -E 'database.php|/flash/nsconfig/keys/updated|/flash/nsconfig/keys|/ns_gui/vpn|LDAPTLS_REQCERT|ldapsearch|openssl|/nsconfig/ns.conf|del /etc/auth.conf|cp /usr/bin/bash|.F1.key|.F2.key|nobody' /var/log/bash.log* |
Here in the screenshot you can see only my tests and no attack.
Log files
Check log files for known IOCs.
|
1 |
grep -v '127\.0\.0' /var/log/*.log | grep 'nc -l\|/etc/passwd\|/etc/shadow\|python -c\|curl\|\.php' |
Edited files with the setuid bit
|
1 |
find /var -perm -4000 -user root -not -path "/var/nslog/*" -newermt {Timestamp der Installer Files +1} -exec ls -l {} \; |
Nobody processes
In the context of the Nobody user, new processes were also started during attacks. Here is an example of the allowed processes.
|
1 |
shell ps aux | grep nobody | grep -v '/bin/httpd' |
And an attacked server. The top line is malware.
Check Crontab for new entries
With the following command we can examine the current crontab file. It should look like in the displayed image.
|
1 |
shell grep '' /etc/crontab |
There should be no crontab for nobody. This can be checked with the following command.
|
1 |
shell crontab -l -u nobody |
I have received several community comments that the following entries are also normal:
- /netscaler/adss-licexp.sh –> For trial licenses
- 0/5 * * * * root /netscaler/iprep –> When IP Reputation is in use
- */5 * * * * root /var/python/bin/python /netscaler/aslearn_health_monitor.py –> When app firewall is used
- */5 * * * * root /bin/pgrep -f /netscaler/appfw_dynamic_profiles/appfw_dynamic_profiles.py; [ $? != 0 ] && /var/python/bin/python /netscaler/appfw_dynamic_profiles/appfw_dynamic_profiles.py –> When app firewall is used
NetScaler HA Systems
During various attacks, the process responsible for HA (nsfsyncd) was disabled on NetScaler HA systems.
|
1 |
shell ps aux | grep nsfsyncd |
On standalone machines, the process should not appear either. Shown is the case how it should look like on a working system.
httpaccess-vpn log files
Check log files for successful web accesses to unknown resources.
|
1 |
shell zgrep -E -v 'CitrixReceiver' /var/log/httpaccess-vpn.log* | grep ' 200 ' |
As well as web accesses from HeadlessChrome.
|
1 |
shell zgrep 'HeadlessChrome' /var/log/httpaccess-vpn.log* |
NPPE Core Dumps
In some attacks, a core dump is created via NPPE (NetScaler Packet Processing Engine) after a critical failure in NetScaler. Normally, the following directory should be empty.
|
1 |
shell ls -ll /var/core/1 |
This is what it looks like for an attacked system.
Perl & Python Scripte
Adding custom Perl or Python scripts can also open a backdoor.
|
1 2 |
shell ps -aux | grep python shell ps -aux | grep perl |
If more than the grep queries shown are seen, the further scripts, should be checked by a Google search.
If something appears, run the command again after a few seconds. Some scheduled tasks on the NetScaler also use python or perl. Only if it still runs on the second call, this should be checked.
These are normal scripts for an NetScaler instance hosted in Azure.
When the StoreFront monitor is used, you will permanently see the following Perl script.
Crypto Miners
I have seen several NetScaler instances used as crypto miners after the attack.
With the following command you can see all running processes.
Here, no other process besides NSPPE-xx should be permanently at 100%. A normal state is shown.
|
1 |
shell top -n 10 |
Check network and firewall logs
Checks the network and firewall logs for the following events:
- Scans of the subnets sent by NetScaler for the protocols HTTP / HTTPS / SMB (Port 80 / 443 / 445)
- Spikes in the queries from NetScaler regarding LDAP / LDAPS / DNS / AD (Port 389 / 636 / 53 / 88 / 135 / 137-138 / 464 / 3268-3269) protocols
Countermeasures for affected systems
My recommendation regarding action when the system has been corrupted.
- Remove the NetScaler instance from the network
- Changes the password of all LDAP or other AD / network accounts stored on the NetScaler.
- Issues a new SSL certificate and key file for all client SSL files on the device (the keys are stored in files on the NetScaler, which could theoretically be read by the attacker)
- If it is a VPX appliance and there are snapshots of the appliance (older than 3 months), it may be worth restoring them first, but this is NO GUARANTEE of security
- To be on the safe side, export the configuration file of the affected system and copy / restore it to a new fresh VPX appliance
- Note that the same hardware address is used, otherwise the license is invalid and must be requested / imported again
- Disconnects the network before startup
- Starts the appliance and verifies via the console that the VPX is not compromised
- Changes the nsroot password
- Connect only the internal network
- Connect the external network
- Keeps an eye on the logs and responder hits
- Replaces the SSL certificates on the appliance at the earliest possible time
- Revokes the compromised SSL certificates and SSL keys
Thanks!
Thanks for keeping this updated – FYSA – change your grep on those httperror logs to ‘zgrep’
Thx
Thanks, helps big time. The command to find nobody http processes needs quotes around ‘/bin/httpd’ to match.
Thx
Thanks for sharing .
But I have a problem with the shell ps aux comand, my netscaler does not seem to know the comand.
as output only appears: -bash: shell: command not found
Plz try the following..
First enter shell and then the rest of the command
The “nobody” process is very interesting. Thank you very much!
When i check the timestamps of files and directories in /netscaler/ns_gui/ they all (including the .php) seem to have the last startup time as timestamp set. Is this normal behaviour?
I see the same behavior.
Yes this is normal behavior, I’ll correct the entry.
About the mentioned crontab entries: I have one additional: /netscaler/adss-licexp.sh
But it seems to me that this indeed only does some kind of evaluation license check (as the name already suggests) and actually reboots the device if some grace period expired. Anyone else seeing this on their appliance?
We have enabled the IP Repution filter on some NetScaler instances. Therefore we have additional the follwing entry in our crontab file:
0/5 * * * * root /netscaler/iprep
(The iprep process connects with Webroot and updates the database every 5 minutes.)
With App Firewall enabled another additional entries appears in the the crontab file:
*/5 * * * * root /var/python/bin/python /netscaler/aslearn_health_monitor.py
*/5 * * * * root /bin/pgrep -f /netscaler/appfw_dynamic_profiles/appfw_dynamic_profiles.py; [ $? != 0 ] && /var/python/bin/python /netscaler/appfw_dynamic_profiles/appfw_dynamic_profiles.py
thx i will implement this in the article
We checked the Perl scripts and the processes and got the following output.
ps -aux | grep perl
“nsmonitor 943 0.0 0.8 33532 12668 – S 17:16 3:31.94 /usr/bin/perl -w /netscaler/monitors/nssf.pl”
top -n 10
“943 nsmonitor 20 0 33532K 12648K sbwait 0 3:29 0.00% perl”
I checked some other ADCs and could not find any user nsmonitor there.
The file nssf.pl is identical from the timestamp to the others.
Is this a normal process?
Thanks a lot
Du u rerun the test after a minute…some perl monitor are normal..but they should run only several seconds, not the whole time
Yes I have checked it over and over again for several hours now.
The process remains.
I am also a bit surprised by the username nsmonitor.
I have never seen this before.
In the Config this is also not to be found.
OK i test this also on my machines and doesn’t find the same…
I would recommend to rebuild the machine.
Can u send me a screenshot for my article of this ? I can send it to consulting@www.deyda.net
This is a normal behavior when u have configured a storefront monitor.
Yep if the ADM Connector is enabled, it will show that.
I find 2 python scripts that seem to run all the time on our instance:
/var/mastools/scripts/admautoreg/adm_auto_reg_daemon.py
/var/mastools/scripts/mastools_upgrade.py
Now i’m not a python expert at all but at the first look they don’t really look suspicious to. Searching for mastools on Google leads to some info about Citrix “ADM agent”. Does anyone else see this on their Netscaler installations?
hi, i have the same files on a not exposed ADC systems, i believe are the ADM configuration scripts
Hello,
Thanks for the article.
I have questions concerning the timestamp to use for commands under “Edited files”
Do I have to use the timestamp of the very last update (on July 18) or the timestamp of the previous update (on May 9) ?
I have a doubt because when I run the following 2 commands using the timestamp of previous update, I can see results on July 7
find /var/netscaler/logon/ -type f -newermt {Timestamp der Installer Files +1} -exec ls -l {} \;
find /var/python/ -type f -newermt {Timestamp der Installer Files +1} -exec ls -l {} \;
At the same time Citrix has released his own script “IoC (Indicator of Compromise) script v5” which returns “Scan OK” for my ADC which means that they have not been compromised.
I don’t know which result to believe.
regards,
Hello Richard,
as described in this article “The files under ns_gui change the timestamp not only when the files are installed, but at every reboot.”
Therefore you need to enter enter one day after the last reboot to these commands to check whether something was changed by a malicious compromise.
By the way: Where can we find this “IoC (Indicator of Compromise) script v5”?
Best regards, Richard K. 😉
Hi,
The commands I have indicated are not under “ns_gui”.
You must open a case on Citrix Support, they will provide you the script.
regards,
PLZ make a support case at citrix to get the ioc.
If you don’t get minimum version 5 from the IOC, you have to ask again. Because currently only with version 5 all current things are found.
If you have checked with an earlier version, you have to do the test again with version 5.
I’ve asked Citrix support and they told me that v4 is the only version that is available.
You must use the date of last update. If u want to check if you are compromised, you must do it, before u update your NetScaler. Because after the update all of the timestamps changed to the newest. Some are at the date of the update, some are before.
Ok good news.
So just to be sure, I could restore a backup of my ADC juste before the last update and retry the commands in order to see if I see the same timestamp or not.
regards,
Hello,
Ok commands returned nothing but I also ran the IOC script v5 on my restored ADC (with version before the last update) and this time the result is not ok. The result is ok when ADC’s are patched.
So, are they compromised because of the result of the scan made on ADC with “old” version ?
Regards
I see several positive false with the ioc. But if u have a not ok plz investigate why this is so.
No the script should also say on old version that the machine is not compromised
Hello,
With this command I have:
root@NSX01# crontab -l -u nobody
* * * * * /var/nstmp/.nscache/httpd
and :
root@NSX01# ls -ll /var/core/1 I have a lot of entries:
-rw——- 1 root nobody 32835333 Aug 3 14:03 NSPPE-00-13183.gz
-rw——- 1 root nobody 32138509 Aug 11 02:16 NSPPE-00-15890.gz
-rw——- 1 root nobody 47841571 Aug 8 20:12 NSPPE-00-20047.gz
-rw——- 1 root nobody 26907184 Aug 3 14:14 NSPPE-00-34098.gz
-rw——- 1 root nobody 30776903 Aug 3 16:50 NSPPE-00-35445.gz
-rw——- 1 root nobody 29625372 Aug 3 18:35 NSPPE-00-40404.gz
-rw——- 1 root nobody 32530277 Aug 11 15:26 NSPPE-00-43536.gz
-rw——- 1 root nobody 27739676 Aug 3 19:37 NSPPE-00-43841.gz
-rw——- 1 root nobody 26559968 Aug 3 20:14 NSPPE-00-46150.gz
-rw——- 1 root nobody 26244527 Aug 3 20:44 NSPPE-00-47797.gz
-rw——- 1 root nobody 27378152 Aug 3 21:51 NSPPE-00-49209.gz
-rw——- 1 root nobody 27098505 Aug 3 22:26 NSPPE-00-51719.gz
-rw——- 1 root nobody 32008591 Aug 9 11:04 NSPPE-00-53033.gz
-rw——- 1 root nobody 29146796 Aug 4 03:20 NSPPE-00-53234.gz
-rw——- 1 root nobody 31990392 Aug 4 11:28 NSPPE-00-61904.gz
-rw——- 1 root nobody 30845123 Aug 12 10:26 NSPPE-00-65524.gz
-rw——- 1 root nobody 29149989 Aug 7 04:48 NSPPE-00-74513.gz
-rw——- 1 root nobody 30510454 Aug 4 13:49 NSPPE-00-75891.gz
My Netscaler is attacked? How I can fix it? firts with update 13.1 but if my system is compromised what I can do ?
Thanks
Yes this looks like your sytem is compromised.
Follow the recommendation at the end of my article.