Table of Contents
Citrix has released a critical vulnerability warning (CVE-2019-19781) in all Citrix ADC & Gateway systems one week before Christmas. Several working exploits have been released since Jan. 10, 2020 and are available to everyone.
Important ! The fix from Citrix with the Responder Policy does not work on systems with version 12.1.51.16/51.19, 50.31 and older. If this version is in use, please update to the latest 12.1 version.
The exploits allow remote code to be executed anonymously, allowing unauthenticated attackers to take over the various machines with root privileges.
For quite some time now, massive scanning for Citrix ADC machines has been observed by various institutions. This compiled list is now under attack.
For pure Microsoft Admins there are now also Powershell Scripts to check if the system is corrupted or still open.
Firmware Updates
Citrix is currently unable to provide a patch. However, there are already dates set when these will be released:
| Citrix ADC and Citrix Gateway | ||
|---|---|---|
| Version | Refresh Build | Expected Release Date |
| 10.5 | 10.5.70.12 | Patch is here |
| 11.1 | 11.1.63.15 | Patch is here |
| 12.0 | 12.0.63.13 | Patch is here |
| 12.1 | 12.1.55.18 | Patch is here |
| 13.0 | 13.0.47.24 | Patch is here |
| Citrix SD-WAN WANOP | ||
| Release | Citrix ADC Release | Expected Release Date |
| 10.2.6 | 10.2.6b | Patch is here |
| 11.0.3 | 11.0.3b | Patch is here |
Instructions for updating the appliances can be found <<here>>
Workaround
Executes the following commands via the GUI command line interface or directly on the appliance (via Putty).
Command line interface
- Log on to the affected system and go to the navigation points System > Diagnostics
- In the menu item Diagnostics on the right side go to Command line interface
- In the CLI you can enter the following commands and execute them one by one by Go
Directly per Putty
- Starts Putty and enters the management IP of the affected system
- Connect to the system and enter your username and password
- Now you can insert and execute the commands listed below
Standalone System
The following commands are used to create the Responder Actions & Policy.
|
1 2 3 4 5 |
enable ns feature responder add responder action respondwith403 respondwith "\"HTTP/1.1 403 Forbidden\r\n\r\n\"" add responder policy ctx267027 "HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\")) || HTTP.REQ.HEADER(\"NSC_USER\").CONTAINS(\"/../\") || HTTP.REQ.HEADER(\"NSC_NONCE\").CONTAINS(\".pl\")" respondwith403 bind responder global ctx267027 1 END -type REQ_OVERRIDE save config |
Now we make sure that the changes also apply to the management interface.
|
1 2 3 |
shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0 shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler" reboot |
HA Pair
On the Primary
Enter the following by Putty or CLI.
|
1 2 3 4 5 |
enable ns feature responder add responder action respondwith403 respondwith "\"HTTP/1.1 403 Forbidden\r\n\r\n\"" add responder policy ctx267027 "HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\")) || HTTP.REQ.HEADER(\"NSC_USER\").CONTAINS(\"/../\") || HTTP.REQ.HEADER(\"NSC_NONCE\").CONTAINS(\".pl\")" respondwith403 bind responder global ctx267027 1 END -type REQ_OVERRIDE save config |
Afterwards, switch the HA state of the machine to Secondary via GUI or Shell Command.
- Navigate to System > High Availability
- In the Nodes tab, select a node and click Force Failover in the Action list.
Or enter the following command via Putty or CLI:
|
1 |
force HA failover |
Now we make sure that the changes also apply to the management interface. Again using Putty or CLI, enter the following.
|
1 2 3 |
shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0 shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler" reboot |
ON THE SECONDARY, AFTER THE PRIMARY IS STARTED
Switch the HA state of the machine back to secondary via GUI or shell command and then enters the following via Putty or CLI
|
1 2 3 |
shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0 shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler" reboot |
Cluster
CLIP
Enter the following by Putty or CLI.
|
1 2 3 4 5 |
enable ns feature responder add responder action respondwith403 respondwith "\"HTTP/1.1 403 Forbidden\r\n\r\n\"" add responder policy ctx267027 "HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\")) || HTTP.REQ.HEADER(\"NSC_USER\").CONTAINS(\"/../\") || HTTP.REQ.HEADER(\"NSC_NONCE\").CONTAINS(\".pl\")" respondwith403 bind responder global ctx267027 1 END -type REQ_OVERRIDE save config |
Now we make sure that the changes also apply to the management interface and enter the following via Putty or CLI.
|
1 2 3 |
shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0 shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler" reboot |
ON EACH CLUSTER NODE
Enter the following by Putty or CLI.
|
1 2 3 |
shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0 shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler" reboot |
IN THE ADMIN PARTITION
Enter the following by Putty or CLI.
|
1 2 3 4 5 6 7 8 9 |
switch ns partition default enable ns feature responder add responder action respondwith403 respondwith "\"HTTP/1.1 403 Forbidden\r\n\r\n\"" add responder policy ctx267027 "HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/vpns/\") && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\"/../\")) || HTTP.REQ.HEADER(\"NSC_USER\").CONTAINS(\"/../\") || HTTP.REQ.HEADER(\"NSC_NONCE\").CONTAINS(\".pl\")" respondwith403 bind responder global ctx267027 1 END -type REQ_OVERRIDE save config shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0 shell "echo 'nsapimgr_wr.sh -ys skip_systemaccess_policyeval=0' >> /nsconfig/rc.netscaler" reboot |
Review of the systems
Even systems with the protection methods in place should be subjected to a thorough review, as it is never possible to say when attacks may have started before they have become broadly based.
Here is a summary of the articles, Twitter entries and own experiences regarding the attack wave.
Bash script to check the system
Daniel Weppeler has created a bash script to check Citrix ADC / Netscaler for corruption.
Download the script in Git under
Afterwards you can start the script as follows.
|
1 |
bash CVE-NetScalerFileSystemCheck.sh |
Citrix has also released a script. Download it here and then copy it to the Netscaler / Citrix ADC Then start it with the following command (I have it in /tmp).
|
1 |
shell bash ./ioc-scanner-CVE-2019-19781-v1.2.sh > "/tmp/results-$(date).txt" |
Then you can check the log file with the following commands.
|
1 2 3 |
shell cd /tmp vi results(Datum der Datei).txt |
XML files
The exploits write all files into several different directories. Therefore, check the following directories to see if unknown user names appear there. Meanwhile, hackers delete these files after they have been opened. All commands should be executed by Putty or CLI.
|
1 |
shell ls -latr /netscaler/portal/templates/*.xml |
This directory normally does not contain XML files and therefore the following display should follow.
If files have been found, the first access to the system has already happened.
|
1 |
shell ls -latr /var/tmp/netscaler/portal/templates |
This directory normally does not exist.
But if no error occurs or there are even files in this folder, the system has been successfully attacked.
Here you see an attack with a .xml.ttc2 file.
The content of the .xml.ttc2 file.
|
1 |
shell ls -latr /var/vpn/bookmark/*.xml |
This directory should not exist or only contain XML files from users you know.
If there are XML files in this folder that are unknown, the system was successfully attacked.
If files are present, delete them immediately to detect further attacks directly.
New Folder
The following folders should not exist (NOTROBIN-Backdoor).
|
1 |
shell ls -latr /tmp/.init/* |
|
1 |
shell ls -latr /var/nstmp/.nscache/* |
UDP Listener
During an attack, where the attacker closes all other backdoors, he then builds himself a backdoor via UDP port 18634. Therefore scans exactly for this port.
|
1 |
shell netstat | grep 18634 |
Bash logs
There may also be entries in the bash.logs when attacks occur. We are looking for the user nobody here, as these are his rights when he comes through Apache and a path that is created when attacks occur.
|
1 2 3 4 |
shell cat /var/log/bash.log | grep nobody shell gzcat /var/log/bash.*.gz | grep nobody shell cat /var/log/bash.log | grep /tmp/.init/httpd shell gzcat /var/log/bash.*.gz | grep /tmp/.init/httpd |
Shown are more attacks.
Apache log files
Furthermore, attacks leave entries in the Apache httpaccess log files.
|
1 |
shell cat /var/log/httpaccess.log | grep vpns | grep xml |
Attacks…
|
1 |
shell cat /var/log/httpaccess.log | grep "/\.\./" |
Shown are more attacks.
|
1 |
shell gzcat /var/log/httpaccess.log.*.gz | grep vpns | grep xml |
And more attacks..
|
1 |
shell gzcat /var/log/httpaccess.log.*.gz | grep "/\.\./" |
Above is an attack with a nice reference to the open vulnerable under CVE-2019-19781 (11 January 2020 07:39:59) and more attacks in the evening (19:18:11 etc.)
Cron Jobs
Attackers could have installed a backdoor via the cron jobs and thus gain persistent access, even if the vulnerability is patched. Check your crontab file for anomalies.
|
1 |
shell cat /etc/crontab |
Listed below is an uncorrupted crontab file.
Now check if new users have been added. Listed below are the normal users in the passwd file for versions before 13.0.
|
1 |
shell cat /etc/passwd |
Here a list for version 13.0.
Now check if one of the existing users has assigned a cron job (here e.g. the user nobody).
|
1 |
crontab -u nobody -l |
Here you can see the crontab file on a corrupted server.
Content of this file.
Delete the crontab file in the path:
|
1 |
/var/cron/tabs |
The ERROR response just means that these users have no cron jobs assigned, which is normal behavior.
Nobody processes
In the context of the nobody user, new processes were also started during attacks. Here is an example of the permitted processes.
|
1 |
shell ps auxd | grep nobody |
And an attacked server. The top line is malware.
Perl & Python Scripts
By adding your own Perl or Python scripts you can also open a backdoor.
|
1 2 |
shell ps -aux | grep python shell ps -aux | grep perl |
If there are more than the grep queries shown, the other scripts should be checked by a Google search.
These are normal scripts for an ADC instance hosted in Azure.
Crypto Miners
I have seen several ADC instances that were used as Crypto Miners after the attack. With the following command you can see all running processes. Here no other process, besides NSPPE-xx, should be permanently at 100%. Shown is a normal state.
|
1 |
shell top -n 10 |
Procedure after update by Citrix (Standalone,CLIP, HA Primary)
Executes the following commands in Putty or via the CLI after the patch from Citrix (end of January) is installed
|
1 2 3 4 |
unbind responder global ctx267027 rm responder policy ctx267027 rm responder action respondwith403 save config |
Remove the nsapi command from the rc.netscaler file.
|
1 2 3 |
shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval=1 shell "sed -i '' '/skip_systemaccess_policyeval=0/d' /nsconfig/rc.netscaler" reboot |
A restart of the instances is not necessary to apply the directive. It is a preventative step to ensure that all open sessions received via the vulnerability are deleted.
Countermeasures for affected systems
As DJ Eshelmann writes in his Blog, if the system has been corrupted, you should do the following
- Take the ADC instance from the network
- Changes the password of all LDAP or other AD / network accounts stored on the ADC
- Issues a new SSL certificate and a new key file for all client SSL files on the device (The keys are stored in files on the ADC that could theoretically be read by the attacker)
- If it is a VPX appliance and snapshots of the appliance (older than January 9, 2020) are available, it may be worthwhile to restore it first, but this is NOT A WARRANTY for security
- To be on the safe side, export the configuration file of the affected system and copy / restore it to a new fresh VPX appliance
- Make sure that the same hardware address is used, otherwise the license is invalid and must be requested / imported again
- Disconnect the network before starting
- Launch the appliance and check via the console that the VPX is not compromised
- Change the nsroot password
- Connect the internal network only
- Perform the fix listed above
- Connecting the external network
- Keep an eye on the protocols and responder hits
- Replace the SSL certificates on the appliance at the earliest possible time
- Revokes the compromised SSL certificates and SSL keys
The command shell ps -aux | grep perl will show perl based monitors for LDAP or Storefront. Nothing to worry about
Yes, I know, therefore my comment that these scripts must then be checked. The first results tell you directly that you don’t have to worry. If you have a screenshot about such a request, I can deposit it gladly. Unfortunately I only had one about the scripts that run by default on a system hosted in Azure.
In remediation steps for a compromised Netscaler/ADC you also have to make sure to actively revoke the old SSL certs for protecting from MITM attacks after having recreated and redeployed the new certs (with new private keys ) and installed theses everywhere they have to be used (i.e other systems than the Netscaler that was compromised if certs are shared between systems in such a way, think about wildcard certs sprawl generally found in so many IT infrastructures)