Table of Contents
Critical vulnerabilities in NetScaler ADC and NetScaler Gateway require a structured response: assess exposure, select a secure target build, update the systems and then check for indicators of compromise.
This checklist summarizes the key steps for new NetScaler CVEs. The current vendor security bulletin, supported firmware releases and the specifics of the environment always remain authoritative.
Assess the CVE and prepare the update
- Review the security bulletin and identify affected product versions.
- Document the current build, HA status, partitions and enabled features.
- Check the supported target release, known issues and upgrade paths.
- Back up the configuration and system files and define a reliable rollback plan.
- Implement and document vendor-provided workarounds until the update is completed.
Update the firmware
Update every affected NetScaler system to a Citrix-approved build. For HA pairs, perform the update in a controlled sequence while validating synchronization, failover and the availability of published services. Obsolete or unsupported versions must be migrated to a supported release branch.
A firmware update alone does not prove that the system was unaffected before remediation. For critical vulnerabilities, a subsequent security assessment is therefore part of the response process.
Review of the systems
Even systems with the protections in place should be subjected to a thorough review, as it is never possible to say with certainty when attacks may have begun before they are widely deployed.
Here is a summary of articles, Twitter entries and our own experiences regarding the wave of attacks.
When compromise is suspected, preserve at least a VPX snapshot or forensic image, the system time, timezone and NTP configuration, a Technical Support Bundle, and local and centralized logs before rebooting, updating or cleaning the system. Depending on the incident, an NSPPE core dump may also be useful. Citrix documents the process in CTX694799.
A single finding is not automatically proof of a successful attack. Conversely, an inconclusive review cannot rule out compromise because logs rotate and attackers may alter evidence. Always assess anomalies in the technical and chronological context of the relevant security bulletin.
Find out the time of the last update
At the moment, an attack can only be detected by overwritten timestamps in certain files. These timestamps usually only change when updating the NetScaler and therefore it is important to know this date. A good indicator is to look at the date of the unpacked installation packages.
All commands should be executed via Putty or CLI.
First, of course, you should log in with nsroot or another administrative account. After that you can view your extracted installation data with the following command.
|
1 |
shell ls -ll /var/nsinstall |
Here you can see the date of the last NetScaler update package (19.07.2023). We should note this, because we need this for the next commands. It is important, that we do not enter the determined date in the following commands, but the determined date in the form YYYYMMDD +1. So in my case 20230720.
Edited files
Now we can check whether certain files have been adjusted since the last update. This would not be proof yet, but a first indication and should be taken seriously.
|
1 2 3 4 |
shell find /var/vpn/ -type f -newermt {Timestamp der Installer Files +1} -exec ls -l {} ; find /var/netscaler/logon/ -type f -newermt {Timestamp der Installer Files +1} -exec ls -l {} ; find /var/python/ -type f -newermt {Timestamp der Installer Files +1} -exec ls -l {} ; |
A common webshell I see in the wild is identified by the file a.php under /var/netscaler/logon. Please also check if this file is present, this is a sign that an attack has taken place.
The files under ns_gui change the timestamp not only when the files are installed, but at every reboot. Therefore, this should be taken into account when checking the timestamp.
|
1 2 |
shell find /netscaler/ns_gui/ -type f -name *.php -newermt {Timestamp der Installer Files +1} -exec ls -l {} ; |
Or with the following Python command.
|
1 2 |
shell python -c "import os, glob, time; newest_timestamp = max(os.stat(f).st_mtime for f in glob.glob('/var/nsinstall/*')); print('n'.join(os.path.join(dirpath, f) for dir in ['/var/netscaler/logon/', '/var/python/', '/var/vpn/'] for dirpath, _, files in os.walk(dir) for f in files if os.stat(os.path.join(dirpath, f)).st_mtime > newest_timestamp))" |
If nothing has been edited since the last update this is a good sign.
New files (webshell)
Also check for unknown PHP, Perl, Python, JavaScript or ELF files in /var/vpn/, /var/netscaler/logon/, /var/python/, /netscaler/ns_gui/, /tmp/ and /var/tmp/.
We check various directories for new files as well, this would indicate a webshell.
|
1 2 3 4 |
shell ls -ll /var/nsproflog/*.php ls -ll /var/netscaler/gui/*.php ls -ll /netscaler/portal/*.php |
HTTP error log files
Indicators include unexpected successful requests to unknown resources, unusual POST requests, user agents that do not match the normal profile, and crashes or core dumps close to the suspected exploitation window.
There may also be entries in the http error logs during attacks. We are looking for anomalies regarding .sh and .php here.
|
1 2 3 |
zgrep '.sh' /var/log/httperror.log* zgrep '.php' /var/log/httperror.log* zgrep '.pl' /var/log/httperror.log* |
The picture shows that the search for .sh was successful. You can check this directly by looking into the file with an editor of your choice (e.g. vi).
In this case, I just added .sh in the affected file to check my command.
Shell / Bash log files
In addition, attacks leave entries in the shell or bash log files.
|
1 2 |
shell zgrep -E 'database.php|/flash/nsconfig/keys/updated|/flash/nsconfig/keys|/ns_gui/vpn|LDAPTLS_REQCERT|ldapsearch|openssl|/nsconfig/ns.conf|del /etc/auth.conf|cp /usr/bin/bash|.F1.key|.F2.key|nobody' /var/log/sh.log* shell zgrep -E 'database.php|/flash/nsconfig/keys/updated|/flash/nsconfig/keys|/ns_gui/vpn|LDAPTLS_REQCERT|ldapsearch|openssl|/nsconfig/ns.conf|del /etc/auth.conf|cp /usr/bin/bash|.F1.key|.F2.key|nobody' /var/log/bash.log* |
Here in the screenshot you can see only my tests and no attack.
Log files
Also look for gaps in logging, unexpectedly early log rotation, and timestamps that do not align with the system time, timezone, NTP configuration or the surrounding sequence of events.
Check log files for known IOCs.
|
1 |
grep -v '127.0.0' /var/log/*.log | grep 'nc -l|/etc/passwd|/etc/shadow|python -c|curl|.php' |
Edited files with the setuid bit
Copied shells, new setuid files, and unusual ownership or permissions are particularly significant and should be compared with a known-good system state.
|
1 |
find /var -perm -4000 -user root -not -path "/var/nslog/*" -newermt {Timestamp der Installer Files +1} -exec ls -l {} ; |
Nobody processes
In the context of the Nobody user, new processes were also started during attacks. Here is an example of the allowed processes.
|
1 |
shell ps aux | grep nobody | grep -v '/bin/httpd' |
And an attacked server. The top line is malware.
Check Crontab for new entries
In addition to the crontabs, review /flash/nsconfig/rc.netscaler and /etc/monitrc. Suspicious entries may recreate web shells after a reboot, set the setuid bit, or hide and terminate monitored processes.
With the following command we can examine the current crontab file. It should look like in the displayed image.
|
1 |
shell grep '' /etc/crontab |
There should be no crontab for nobody. This can be checked with the following command.
|
1 |
shell crontab -l -u nobody |
I have received several community comments that the following entries are also normal:
- /netscaler/adss-licexp.sh –> For trial licenses
- 0/5 * * * * root /netscaler/iprep –> When IP Reputation is in use
- */5 * * * * root /var/python/bin/python /netscaler/aslearn_health_monitor.py –> When app firewall is used
- */5 * * * * root /bin/pgrep -f /netscaler/appfw_dynamic_profiles/appfw_dynamic_profiles.py; [ $? != 0 ] && /var/python/bin/python /netscaler/appfw_dynamic_profiles/appfw_dynamic_profiles.py –> When app firewall is used
NetScaler HA Systems
Unexplained HA synchronization failures and stopped or modified monitoring and synchronization processes may also indicate manipulation.
During various attacks, the process responsible for HA (nsfsyncd) was disabled on NetScaler HA systems.
|
1 |
shell ps aux | grep nsfsyncd |
On standalone machines, the process should not appear either. Shown is the case how it should look like on a working system.
httpaccess-vpn log files
Also review Gateway and AAA sessions for changing source IP addresses, countries or user agents within the same session, and for sessions that remain usable after password changes or MFA actions.
Check log files for successful web accesses to unknown resources.
|
1 |
shell zgrep -E -v 'CitrixReceiver' /var/log/httpaccess-vpn.log* | grep ' 200 ' |
As well as web accesses from HeadlessChrome.
|
1 |
shell zgrep 'HeadlessChrome' /var/log/httpaccess-vpn.log* |
NPPE Core Dumps
Search core dumps and related artifacts for evidence that ns.conf, .F1.key, .F2.key, certificates or private keys were copied into web, theme, template or temporary directories.
In some attacks, a core dump is created via NPPE (NetScaler Packet Processing Engine) after a critical failure in NetScaler. Normally, the following directory should be empty.
|
1 |
shell ls -ll /var/core/1 |
This is what it looks like for an attacked system.
Perl & Python Scripte
Adding custom Perl or Python scripts can also open a backdoor.
|
1 2 |
shell ps -aux | grep python shell ps -aux | grep perl |
If more than the grep queries shown are seen, the further scripts, should be checked by a Google search.
If something appears, run the command again after a few seconds. Some scheduled tasks on the NetScaler also use python or perl. Only if it still runs on the second call, this should be checked.
These are normal scripts for an NetScaler instance hosted in Azure.
When the StoreFront monitor is used, you will permanently see the following Perl script.
Crypto Miners
I have seen several NetScaler instances used as crypto miners after the attack.
With the following command you can see all running processes.
Here, no other process besides NSPPE-xx should be permanently at 100%. A normal state is shown.
|
1 |
shell top -n 10 |
Check network and firewall logs
Investigate unusual traffic from the NSIP: internal scanning on ports 80, 443 or 445, spikes in DNS or LDAP/LDAPS activity, RDP and network logons, excessive connections from one source address, and large outbound transfers. Include AD events 4624 and 4625 as well as unusual Kerberos and LDAP queries.
Checks the network and firewall logs for the following events:
- Scans of the subnets sent by NetScaler for the protocols HTTP / HTTPS / SMB (Port 80 / 443 / 445)
- Spikes in the queries from NetScaler regarding LDAP / LDAPS / DNS / AD (Port 389 / 636 / 53 / 88 / 135 / 137-138 / 464 / 3268-3269) protocols
Countermeasures for affected systems
My recommendation regarding action when the system has been corrupted.
- Remove the NetScaler instance from the network
- Changes the password of all LDAP or other AD / network accounts stored on the NetScaler.
- Issues a new SSL certificate and key file for all client SSL files on the device (the keys are stored in files on the NetScaler, which could theoretically be read by the attacker)
- If it is a VPX appliance and there are snapshots of the appliance (older than 3 months), it may be worth restoring them first, but this is NO GUARANTEE of security
- Replaces the SSL certificates on the appliance at the earliest possible time
- Revokes the compromised SSL certificates and SSL keys
Get your NetScaler security professionally assessed
Do you need support with firmware updates, assessing critical vulnerabilities or checking for indicators of security incidents? Deyda Consulting supports you with NetScaler Security Readiness – structured, transparent and focused on real-world operations.