Checklist for Citrix ADC CVE-2019-19781

Citrix has released a critical vulnerability warning (CVE-2019-19781) in all Citrix ADC & Gateway systems one week before Christmas. Several working exploits have been released since Jan. 10, 2020 and are available to everyone.

Important ! The fix from Citrix with the Responder Policy does not work on systems with version 12.1.51.16/51.19, 50.31 and older. If this version is in use, please update to the latest 12.1 version.

The exploits allow remote code to be executed anonymously, allowing unauthenticated attackers to take over the various machines with root privileges.

For quite some time now, massive scanning for Citrix ADC machines has been observed by various institutions. This compiled list is now under attack.

There is now an online checker for the exploit.

cve-2019-19781.azurewebsites.net

For pure Microsoft Admins there are now also Powershell Scripts to check if the system is corrupted or still open.

https://github.com/j81blog/ADC-19781

Firmware Updates

Citrix is currently unable to provide a patch. However, there are already dates set when these will be released:

Citrix ADC and Citrix Gateway
VersionRefresh BuildExpected Release Date
10.510.5.70.12Patch is here
11.111.1.63.15Patch is here
12.012.0.63.13Patch is here
12.112.1.55.18Patch is here
13.013.0.47.24Patch is here
Citrix SD-WAN WANOP  
ReleaseCitrix ADC ReleaseExpected Release Date
10.2.610.2.6bPatch is here
11.0.311.0.3bPatch is here

Instructions for updating the appliances can be found <<here>>

Workaround

Executes the following commands via the GUI command line interface or directly on the appliance (via Putty).

Command line interface

  • Log on to the affected system and go to the navigation points System > Diagnostics
  • In the menu item Diagnostics on the right side go to Command line interface
  • In the CLI you can enter the following commands and execute them one by one by Go

Directly per Putty

  • Starts Putty and enters the management IP of the affected system
  • Connect to the system and enter your username and password
  • Now you can insert and execute the commands listed below

Standalone System

The following commands are used to create the Responder Actions & Policy.

Now we make sure that the changes also apply to the management interface.

HA Pair

On the Primary

Enter the following by Putty or CLI.

Afterwards, switch the HA state of the machine to Secondary via GUI or Shell Command.

  • Navigate to System > High Availability
  • In the Nodes tab, select a node and click Force Failover in the Action list.

Or enter the following command via Putty or CLI:

Now we make sure that the changes also apply to the management interface. Again using Putty or CLI, enter the following.

ON THE SECONDARY, AFTER THE PRIMARY IS STARTED

Switch the HA state of the machine back to secondary via GUI or shell command and then enters the following via Putty or CLI

Cluster

CLIP

Enter the following by Putty or CLI.

Now we make sure that the changes also apply to the management interface and enter the following via Putty or CLI.

ON EACH CLUSTER NODE

Enter the following by Putty or CLI.

IN THE ADMIN PARTITION

Enter the following by Putty or CLI.

Review of the systems

Even systems with the protection methods in place should be subjected to a thorough review, as it is never possible to say when attacks may have started before they have become broadly based.

Here is a summary of the articles, Twitter entries and own experiences regarding the attack wave.

Bash script to check the system

Daniel Weppeler has created a bash script to check Citrix ADC / Netscaler for corruption.

Download the script in Git under

https://github.com/DanielWep/CVE-NetScalerFileSystemCheck

Afterwards you can start the script as follows.

Citrix has also released a script. Download it here and then copy it to the Netscaler / Citrix ADC Then start it with the following command (I have it in /tmp).

Then you can check the log file with the following commands.

XML files

The exploits write all files into several different directories. Therefore, check the following directories to see if unknown user names appear there. Meanwhile, hackers delete these files after they have been opened. All commands should be executed by Putty or CLI.

This directory normally does not contain XML files and therefore the following display should follow.

If files have been found, the first access to the system has already happened.

This directory normally does not exist.

But if no error occurs or there are even files in this folder, the system has been successfully attacked.

Here you see an attack with a .xml.ttc2 file.

The content of the .xml.ttc2 file.

This directory should not exist or only contain XML files from users you know.

If there are XML files in this folder that are unknown, the system was successfully attacked.

If files are present, delete them immediately to detect further attacks directly.

New Folder

The following folders should not exist (NOTROBIN-Backdoor).

UDP Listener

During an attack, where the attacker closes all other backdoors, he then builds himself a backdoor via UDP port 18634. Therefore scans exactly for this port.

Bash logs

There may also be entries in the bash.logs when attacks occur. We are looking for the user nobody here, as these are his rights when he comes through Apache and a path that is created when attacks occur.

Shown are more attacks.

Apache log files

Furthermore, attacks leave entries in the Apache httpaccess log files.

Attacks…

Shown are more attacks.

And more attacks..

Above is an attack with a nice reference to the open vulnerable under CVE-2019-19781 (11 January 2020 07:39:59) and more attacks in the evening (19:18:11 etc.)

Cron Jobs

Attackers could have installed a backdoor via the cron jobs and thus gain persistent access, even if the vulnerability is patched. Check your crontab file for anomalies.

Listed below is an uncorrupted crontab file.

No alt text provided for this image

Now check if new users have been added. Listed below are the normal users in the passwd file for versions before 13.0.

Here a list for version 13.0.

Now check if one of the existing users has assigned a cron job (here e.g. the user nobody).

Here you can see the crontab file on a corrupted server.

Content of this file.

Delete the crontab file in the path:

The ERROR response just means that these users have no cron jobs assigned, which is normal behavior.

Nobody processes

In the context of the nobody user, new processes were also started during attacks. Here is an example of the permitted processes.

And an attacked server. The top line is malware.

Perl & Python Scripts

By adding your own Perl or Python scripts you can also open a backdoor.

If there are more than the grep queries shown, the other scripts should be checked by a Google search.

These are normal scripts for an ADC instance hosted in Azure.

Crypto Miners

I have seen several ADC instances that were used as Crypto Miners after the attack. With the following command you can see all running processes. Here no other process, besides NSPPE-xx, should be permanently at 100%. Shown is a normal state.

Procedure after update by Citrix (Standalone,CLIP, HA Primary)

Executes the following commands in Putty or via the CLI after the patch from Citrix (end of January) is installed

Remove the nsapi command from the rc.netscaler file.

A restart of the instances is not necessary to apply the directive. It is a preventative step to ensure that all open sessions received via the vulnerability are deleted.

Countermeasures for affected systems

As DJ Eshelmann writes in his Blog, if the system has been corrupted, you should do the following

  • Take the ADC instance from the network
  • Changes the password of all LDAP or other AD / network accounts stored on the ADC
  • Issues a new SSL certificate and a new key file for all client SSL files on the device (The keys are stored in files on the ADC that could theoretically be read by the attacker)
  • If it is a VPX appliance and snapshots of the appliance (older than January 9, 2020) are available, it may be worthwhile to restore it first, but this is NOT A WARRANTY for security
    • To be on the safe side, export the configuration file of the affected system and copy / restore it to a new fresh VPX appliance
    • Make sure that the same hardware address is used, otherwise the license is invalid and must be requested / imported again
    • Disconnect the network before starting
    • Launch the appliance and check via the console that the VPX is not compromised
    • Change the nsroot password
    • Connect the internal network only
    • Perform the fix listed above
    • Connecting the external network
    • Keep an eye on the protocols and responder hits
  • Replace the SSL certificates on the appliance at the earliest possible time
  • Revokes the compromised SSL certificates and SSL keys

3 thoughts on “Checklist for Citrix ADC CVE-2019-19781”

    1. Yes, I know, therefore my comment that these scripts must then be checked. The first results tell you directly that you don’t have to worry. If you have a screenshot about such a request, I can deposit it gladly. Unfortunately I only had one about the scripts that run by default on a system hosted in Azure.

  1. In remediation steps for a compromised Netscaler/ADC you also have to make sure to actively revoke the old SSL certs for protecting from MITM attacks after having recreated and redeployed the new certs (with new private keys ) and installed theses everywhere they have to be used (i.e other systems than the Netscaler that was compromised if certs are shared between systems in such a way, think about wildcard certs sprawl generally found in so many IT infrastructures)

Leave a Reply

Your email address will not be published. Required fields are marked *

*