Citrix ADC Exploit • Solution Architect · Trainer · Project Manager • Roland Geldner

Citrix ADC Exploit “CVE-2019-19781”-en

by
Last revised on January 27th, 2020

What’s happened:

Citrix Systems received a warning regarding a security vulnerability on December 17, 2019 published in all Citrix ADC and gateway platforms.
The vulnerability has since spread through the media and television.

Several working exploits were released on Friday, January 10th, 2020. The number of attacks registered has increased dramatically since they were released.
Most of these attacks are carried out with tools that are able to upload files to the ADC, but otherwise do no harm.
Even if most of the attacks that have occurred since the vulnerability was published do no relevant damage:
The security vulnerability has existed for some time and it is imperative that all Citrix ADCs are checked to determine the damage that has occurred, to analyze it and to take the correct steps.

The security vulnerability affects only virtual IP addresses on which authentication takes place via, for example, SAML or Form Based Authentication (logon side).
Classic “load balancing” and “reverse proxy VIPs” without authentication are NOT affected.
Just a few hours later, the number of attacks against ADCs increased dramatically.


Source: ISC

Which Citrix ADC versions are affected?

The vulnerability affects all supported product versions and all supported platforms:

Simple Test Methods – Is My Citrix ADC Vulnerable?

Since many systems that have not yet been secured have actually been hacked, there is an urgent need for all operators of a Citrix ADC to analyse whether they are affected and take countermeasures.

You can determine whether your ADC is vulnerable using a simple CURL command (my will ADC as an example):
curl -v https://unigate.geldner.co.at/vpn/../vpns/cfg/smb.conf –path-as-is

The following test methods were also described and published:

In any case, systems with the existing countermeasures should also be subjected to an in-depth review.
Since the vulnerability has been known for a long time, the environment could have been compromised some time ago.

Has my system been hacked already ?

Citrix has published a Python script together with FireEye, which can be used to automatically test whether an ADC could has been attacked and or hacked.
How the tool works and where it can be downloaded can be found here.

Detailed Testing  Methods – Has My System Been Attacked?

Attention: If none of the checks described here give a positive result, this is a good sign but NO WARRANTY that your system was not successfully attacked!

Have CronTab jobs been stored by the Attacker?

Check whether attackers have been granted access to your cron jobs in the BSD in order to continue to have access after the hole has been remedied. Check your crontab file:

cat / etc / crontab
crontab -l -u nobody

If you find crontab jobs that run under the user “nobody” or do not know each other and do not belong there, your system has been hacked and countermeasures have to be taken.
Attention: Your httpd services are running under “nobody” privileges.

Check your HTTP access logs for suspicious access:

You can check whether your system has been compromised using the following commands:

gzcat /var/log/httpaccess.log.*.gz | grep vpns | grep xml
gzcat /var/log/httpaccess.log.*.gz | grep “/\.\./”
cat /var/log/httpaccess.log | grep vpns | grep xml
cat /var/log/httpaccess.log | grep “/\.\./”

If you see suspicious .xml uploads in the logs or you will see traffic that is contained in the URLs /../,
your system has been attacked and countermeasures urgently need to be taken.

Check the template files:

The most frequently seen attack can be proven by checking whether suspicious .xml files have been uploaded to your ADC.

ls -lh /var/vpn/bookmark/*.xml
ls -lh /netscaler/portal/templates/*.xml
ls -lh / var / tmp / netscaler / portal / templates

If you find suspicious .html, .html.ttc2 or other files in the directories that should not be there
your system has been attacked and countermeasures urgently need to be taken.

Check whether backdoor scripts have been implemented:

Backdoor scripts or other malicious tasks can infect your system as Perl or Python scripts.
To check if any Perl or Python tasks are running:

ps -aux | grep python
ps -aux | grep pearl

If you see more than the “grep” commands yourself, check these running scripts whether they should also run there

Attention:
Multiple native Citrix ADC tasks may appear.
Some of these scripts are scheduled to run.
Run the query again a few seconds later.

In case you see running  foreign scripts. your system has been attacked / infected and countermeasures urgently need to be taken.

Check whether third-party services or crypto miners have been implemented:

In some cases an attempt was made to install Crypto Miner and other services.
You can identify them by looking at and evaluating the CPU-intensive processes:

top -n 10

If you see processes other than NSPPE-00, NSPPE-01, NSPPE-002 NSPPE-03, NSPPE-04, NSPPE-05 that have high CPU usage,
so you may have found a crypto miner!

What to do if you suspect a compromise?

Ideally, contact the Citrix ADC expert you trust to check what has happened and to repair any damage.
If you suspect that your infrastructure has been compromised, measures should definitely be taken.

From today’s perspective, the possible measures to be taken can be divided into 3 steps:

  • Step 1; Could I be affected?
    • Do I use AAA on Citrix ADC Gateway or via the AAA module? If not, I cannot be affected.
    • Use the “Simple Test Methods” and “Simple Cleanup Methods” published in this article to ensure that the attacker cannot gain further access to your ADCs using the known attack or via the implemented malicious code.
  • Step 2; I am not sure or am affected! Steps that should definitely be taken:
    • Close the security gap How we do this is described here under “Workaround to close the security gap”
    • Remove any kind of third-party code that has been stored on the ADC. You can find initial information here under “Simple Test and Cleanup Methods”
    • Prevent your ADC (NSIP, SNIP ..) from accessing the Internet on your firewall.
    • Check whether the ns.conf has been changed. If so, restore a clean backup or remove unwanted entries in ns.conf
    • Reboot your ADC`s. As a result, the ramdisk in the ADC is running new and clean.
    • Before the reboot, check the rc.netscaler and the crontab for malicious code and remove it.
    • There is a risk that your SSL certificates have been compromised. Have all certificates with “new private key files” reissued and exchanged at the ADC. Have all old certificates revoked so that they cannot be misused.
    • We are at risk that all passwords stored at the ADC have been compromised. Exchange all stored passwords.
  • Step 3; I am not sure if am affected! Further steps to be sure:
    • In certain cases, it may be necessary to temporarily deactivate the affected VIPs or all IPs until all dangers have been eliminated
    • In certain cases, reinstalling the ADCs can be helpful or must be carried out
    • In some environments where security is a very critical issue, it may be necessary to replace the hard drives of the ADCs or to replace the entire hardware.
      Please contact Citrix Support.

When will the vulnerability be fixed?

Citrix has released firmware versions regarding the vulnerability CVE-2019-19781 that close the vulnerability.
These versions should  be downloaded and installed from the Citrix download website.
We are happy to help with analysis, assessment and correction.

Workaround to close the security gap:

You are not ready to upgrade yet?
With the help of this responder policy (the feature must be available and activated at the ADC), the security gap can be temporarily closed.
If your system has already been compromised, it is important to clean up the system.

This responder policy closes the gap at the AAA and ADC gateway levels:

add responder action respondwith403 respondwith “\”HTTP/1.1 403 Forbidden\r\n\r\n\””
add responder policy ctx267027 “HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\”/vpns/\”) && (!CLIENT.SSLVPN.IS_SSLVPN || HTTP.REQ.URL.DECODE_USING_TEXT_MODE.CONTAINS(\”/../\”))” respondwith403
add responder policy ctx267027_a “http.REQ.HEADER(\”NSC_USER\”).CONTAINS(\”/../\”)||http.REQ.HEADER(\”NSC_NONCE\”).CONTAINS(\”.pl\”)” respondwith403
bind responder global ctx267027 1 END -type REQ_OVERRIDE
bind responder global ctx267027_a 2 END -type REQ_OVERRIDE
save config

Caution:
Citrix ADC Release 12.1 builds before 51.16 a bug exists that affects responder and rewrite policies bound to VPN virtual servers.
That bug causing causes ADC not to process the packets that matched policy rules.
Wes strongly recommend update to latest 12.1 build for the mitigation steps to take effect.

With the help of these shell commands, the gap on Management GUI is closed:
shell nsapimgr_wr.sh -ys skip_systemaccess_policyeval = 0
shell “echo’ nsapimgr_wr.sh -ys skip_systemaccess_policyeval = 0 ’>> /nsconfig/rc.netscaler”
reboot

I will gladly  support you in further analysis and troubleshooting.