Detect and prevent DDoS attacks on Citrix ADC via DTLS

by

Detect and prevent DDoS attacks on Citrix ADC via DTLS                                 ==> read german version 

Last revised on January 5th, 2021
!! Permanent solution available since January 4th, 2021

On December 24th, Citrix announced in a “Security Bulletin” a DDoS attack pattern that can affect Citrix ADCs. As part of this attack, attackers can overload the Citrix ADC DTLS network throughput (ADC Out).
This can deplete the available outbound bandwidth and therefore negatively effect SLA.
Since January 4, 2021, Citrix has released firmware with which DTLS / Adaptive Transport can be reactivated and secured against the theoretical DDOS attack using a DTLS profile.
This Post is about how to detect and prevent DDoS attacks on Citrix ADC via DTLS
==> To the permanent solution

The actual situation

In the week before Christmas 2020, various cases were reported in which Citrix Gateway installations had reached their license limits with regard to throughput limit or  WAN lines (ADC – Out) had been overloaded.
Currently the area of ​​attack is limited to a small number of customers and Citrix VPX.

Citrix vulnerabilities are not known in connection with  those events, but depending on the dimensioning of the WAN line, available license and permitted hardware resources, production failures or performance restrictions might occur.

What is  being attacked

The attacks take place on the DTLS interface of the “Citrix Gateways” on Citrix ADC devices.
The Citrix Common Gateway protocol can be encrypted via TCP or UDP depending on the requirements:

Common Gateway Protocol (CGP) Adaptive Transport
Common Gateway Protocol (CGP) Adaptive Transport

DTLS is a version of the Transport Layer Security Protocol, which in this case is implemented on the streaming-friendlier UDP transmission protocol. DTLS is also “spoofable” and can therefore be used for DDoS attacks.

This means that attackers send small DTLS packets to the DTLS interface of the Citrix ADC and a response in the form of a much larger packet is sent back to the (spoofed) source IP address (the actual victim of the DDoS attack).
In this way, both the outgoing bandwidth of the ADC operator and the incoming bandwidth width of the spoofing victim are overloaded.

Analysis – Am I a target of these attacks?

To determine if a Citrix ADC is being attacked by this , examine the outbound traffic for anomalies or spikes.

Monitor the following indicators to see if you are being targeted by this DTLS DDoS:

  • High volume of DTLS outbound traffic
  • Increased CPU usage
  • Increased memory consumption

If the “Out” value in the “Throughput (Mbps)” metric is many times higher than the “In” value, further investigations should be carried out.

Throughput under normal conditions
Throughput under normal conditions

Create an “nstrace” and analyze it in e.g. Wireshark:

  • If you do not use “Citrix Virtual Apps and Desktops” in your company and you see DTLS traffic between clients outside your network and your Citrix ADC Gateway VIP, you are affected by the attack.
  • If you do not use the “Adaptive Transport” function on your “Citrix Virtual Apps and Desktops” installation and you see DTLS traffic between clients outside your network and your Citrix ADC Gateway VIP, you are affected by the attack.
  • If you use the “Adaptive Transport” function on your “Citrix Virtual Apps and Desktops” installation and you see a large number of DTLS packets between certain clients outside your network and your Citrix ADC Gateway VIP consisting of a recurring DTLS handshake “Client Hello” so you are affected by the attack.
  • Filter your network trace on the Citrix ADC
    dtls.alert_message.desc == “Internal Error”
    to see whether you are affected by the attack and which IP addresses the attacks come from.

The preferred solution:

With the following command you may deactivate DTLS on your Gateway V-server:

set vpn vserver <vpn_vserver_name> -dtls OFF

Disabling the DTLS protocol m,ight degrade the performance of those HDX real-time applications that use DTLS in your environment.

HDX with EDT / Adaptive Transport

NOTE:
If DTLS is disabled, the HDX connections will be frozen for a few seconds while the DTLS traffic falls back to TLS (TCP).
If you are running applications that need to use EDT, or if you cannot disable DTLS, contact Citrix Technical Support.

Another solution:

I case you use EDT / AT you can try to block the source IPs of the attacks in the company firewall. This method usually only provides relief for a short time, is associated with a lot of effort and not always very effective.

If you have the option of completely blocking UDP: 443 for the customer’s VIP gateway on the company’s firewall level, you might go for that.
In case you had used Citrix EDT bevore that, the HDX data stream will switch over  to  transport via TCP.

Another (not recommended) solution:

One solution variant discussed in various blogs for the EDT / UDP-DTLS connection on port 443 is to have the DTLS requirements properly validated by Citrix ADC using an SSL profile.

set ssl dtlsProfile nsdtls_default_profile -helloVerifyRequest ENABLED

There are now several confirmed cases that the -helloVerifiyRequest option does not work reliably on all Citrix ADC firmware versions. This may lead to a crash of the Citrix ADC after a few hours and should therefore not be used untested in productive environments.

The announced solution:

Citrix is ​​working on adding functionality to DTLS to remove the vulnerability to this attack.
Citrix expects this extension to be available on the Citrix downloads page for all supported versions on January 12, 2021.

Permanent solution available from manufacturer:

Since January 4, 2021, Citrix has released firmware with which DTLS / Adaptive Transport can be reactivated and secured against the theoretical DDOS attack using a DTLS profile.
The firmware is available for downolad on the Citrix download page in the form of the versions

to disposal.

After updating the firmware, anyone who wants to use DTLS can reactivate DTLS:

set vpn vserver <vpn_vserver_name> -dtls ON

A list of all available DTLS profiles can then be called up

show dtlsProfile   

and secured via these DTLS profiles with the “HelloVerifyRequest” function

set dtlsProfile <dtls_Profile_Name> -HelloVerifyRequest ENABLED

Citrix provides a detailed description of what has to be done in order to obtain a permanent solution to the DTLS problem in the “Security Bulletin” CTX289674.