Citrix ADC 101 – Fundamentals

The following is a collection of basic information about Citrix ADC. From licensing, to the most important commands, to the update procedures that can be performed.

General Information

Here is some basic information about Citrix ADC.

Operating System and Architecture

Citrix ADC is based on the open source operating system FreeBSD. Unlike the very similar Linux, FreeBSD has a modular kernel and Citrix has taken advantage of this to modify FreeBSD’s Bash shell by removing the networking subsystem and replacing it with its own. The modifications were placed in a custom kernel module called NetScaler Core Packet Processing Engine (PPE).

Citrix ADC FreeBSD

So the Citrix ADC consists of two shells: the BSD kernel and the NetScaler kernel. Both work as a cohesive unit thanks to the strict delineation of roles. The BSD kernel manages the boot process, file system access and long-term logging. The NetScaler kernel controls time slicing for BSD, network access, SSL offloading, SNMP and syslog processing.

NetScaler Kernel Architecture

The PPE (alternatively referred to as the Packet Engine (PE)) is designed to take advantage of the performance gains that can be achieved through parallelization. Each PPE process is assigned to a core and operates as follows:

  • Monitor incoming packets
  • pull them off the package queue
  • handle them accordingly for content switching, frontend optimization, caching, etc.
  • put the packets back into the packet queue
  • wait for more packages

So the process is either working on a packet or waiting for packets at any time. With multi-core CPUs this can be done in parallel. Certain cores are entrusted with certain functions. For example, core 1 might be responsible for managing network traffic, core 2 for processing TCP/IP, core 3 for processing Layer 7 (e.g. HTTP), and so on. This is possible because each process is a mini ADC that can perform all application optimization tasks supported by ADC.

The upper limit of how much parallel processing can take place at any given time is determined by the number of cores in the CPU. For example, for a CPU with 4 cores, 3 cores are assigned to 3 separate PSAs, with 1 core reserved for management functions, such as SNMP. Note that one core is always reserved for management.

When the ADC is powered on, FreeBSD boots and loads the NetScaler kernel. It lets the NetScaler kernel take over all CPUs except the management core, and then passes the reins to the ADC to complete the boot.

Platforms

Citrix ADC is available in 4 platform versions. The two virtual versions VPX and CPX. VPX for the well-known hypervisors and CPX for Docker hosts. As well as the three physical versions MPX, SDX and BLX. MPX and SDX comes directly as hardware from Citrix, where SDX is a Citrix hypervisor that can include up to 115 independent VPX (Depends on the hardware). BLX is a bare metal software version that can run on its own hardware. Not all hardware is supported here!

BLX SDX MPX CPX VPX Citrix ADC Plattform

Each of the above mentioned platforms has a bandwidth limit stored in the license. This can be adjusted by importing a new, higher license (pay-as-you-grow). The naming scheme of the licenses directly reveals the maximum bandwidth, e.g. a VPX50 has a maximum incoming bandwidth of 50 Mbps. The outgoing traffic is not included in the Citrix bandwidth limitation.

A machine without a license installed is called Citrix ADC Express and has the following limitations:

  • 20 Mbps bandwidth
  • All ADC standard license features, except Citrix Gateway and L4 and L7 defenses
  • Maximum 250 SSL Sessions
  • 20 Mbps SSL throughput

Licensing

Citrix ADC is available in four different license models. Three different Citrix ADC models and the Citrix Gateway license. The different supported features are shown in the following list.

FeaturePremiumAdvancedStandardGateway
Load BalancingYESYESYES
Content SwitchingYESYESYES
AppExpert Rate ControlsYESYESYES
IPv6 SupportYESYESYES
Traffic DomainsYESYESYES
Subscriber-Aware Traffic SteeringYESYESYES
Global Server Load Balancing (GSLB)YESYESOptional
Carrier-Grade Network Address Translation (CGNAT)YESYES
Dynamic Routing ProtocolsYESYES
Surge ProtectionYESYES
Priority QueuingYESYES
TriScale ClusteringYESYES
TCP OptimizationsYESYESYES
AppCompressYESYESOptional
AppCacheYESOptional
DoS DefensesYESYESYES
Rewrite and ResponderYESYESYES
AAA for Traffic ManagementYESYES
Citrix Web AppFirewall (WAF)YESOptional
IP ReputationYESOptional
nFactor AuthenticationYESYES
Cloud ConnectorYES
Insight Center-Web InsightYESYESYES
AppExpertYESYESYES
ActionAnalyticsYESYESYES
Configuration Wizards YESYESYES
Native Citrix Web InterfaceYESYES
Citrix Command CenterYESYESYES
Federated IdentityYESYES
One URL/SSO using SAML 2.0YESYES
Cluster for ICA Proxy (Striped)YESYES
Monitoring of Citrix Apps and Desktops Traffic (Real Time)YESYES
Monitoring of Citrix Apps and Desktops Traffic (Historical)YES
Monitoring of Gateway Traffic (Real Time)YESYES
Monitoring of Gateway Traffic (Historical)YES
Customizable Web PortalYESYESYESYES
SSL VPN Remote AccessYESYESYESYES
ICA Proxy to Citrix Virtual Apps and DesktopsYESYESYESYES
Contextual Policies for Citrix Apps and DesktopsYESYESYESYES
End Point AnalysisYESYESYESYES
Secure Browser-Only Access (CVPN)YESYESYESYES
Always-OnYESYESYES
Integration with StoreFrontYESYESYES
System >License > ADC License

Troubleshooting

Useful information and commands for troubleshooting.

Directories & Files

A list of the most important directories and files on the Citrix ADC machine.

ExplanationDirectory / File
System Syslog File/var/log/ns.log
Alle logged entries/var/log/messages
Authentication /Authorization Logs/var/log/auth.log
Hardware Error & Boot Sequence Error Log/var/log/dmesg.*
Main Log File in NS Data Format. Older files are archived in the same folder but in GZ format./var/nslog/newnslog
Core Crash Dump Files/var/crash/vmcore.*.gz
/var/core/NSPPE-**-*.gz
Kernel Crash Dump Files/var/crash/kernel.*
Core Dump Log File/tmp/savecore.log
Symbolic link to /flash/nsconfig/nsconfig
Location of Citrix License Files/flash/nsconfig/license/*.lic
Current configuration file. Older configurations are stored in the same folder as ns.conf.*./flash/nsconfig/ns.conf
SSL certificates location/flash/nsconfig/ssl
Location of the custom monitors/flash/nsconfig/monitors
Location of the firmware update files/var/nsinstall
/flash
/var/log/

Processes

List of the most important processes that can be found on the Citrix ADC machine.

ExplanationProcess
NetScaler Packet Enginensppe
RBA and SSL VPN External Authnsaaad
Write the ns.conf filensconf
Controls the logging for newnslognslog.sh
HA Syncnssync
Reads SSL Cert filesnsreadfile
SSL CRL List Updatenscrlrefresh
Synchronizes bookmarks and SSL certificatesnsfsyncd
Configuration changes through the GUInsnetsvc
Runs the monitors with scriptnsumond
Controls the writing of the newnslognsconmsg
Collects statistics data for the Historical Reportingnscollect
Routing processesimi / ripd / ospfd / bgpd
nsppe nsaaad

Command Line Interface (CLI) commands

The CLI is part of the NetScaler kernel and is the first thing you see when you connect to the machine.

General Commands

ExplanationCommand
Enables CLI Color Modeset cli mode -color ON
Adding current user, hostname, time and node status to the CLIset cli prompt %u@%h-%T-%s
Increase timeout for CLI session (here to 30 minutes (1800 seconds))set cli mode -timeout 1800
History of executed commandshistory | more
Help display for specific commandhelp <Command>
Display MAN page for specific commandman <Command>
Configuration menuconfig ns
Creates backup of configuration files (/nsconfig/, /var/, /netscaler/, ns.conf) in folder /var/ns_sys_backupcreate system backup <Backup Name> -level basic
Creates extended backup (/nsconfig/, /var/, /netscaler/, ns.conf, Certificates, License Files)in the folder /var/ns_sys_backupcreate system backup <Backup Name> -level full
Displays existing backupsshow system backup
Restore from existing backuprestore system backup <Backup Name>
Configuration modeshell
Features (Available & Configured)show feature
Enables certain feature (if it is supported by the installed license)enable feature <Acronym>
Disables specific featuredisable feature <Acronym>
Mode (Available & Configured)show ns mode
Enables specific modeenable ns mode <Acronym>
Disables specific modedisable ns mode <Acronym>
Saved configurationshow savedConfig | more
Running configurationshow run | more
Differences between the running configuration with the saved configurationdiff ns config -outtype CLI
Save running configurationsave config
Creates file under /var/tmp/support/ for manual upload to cis.citrix.com (health check of Citrix ADC)show techsupport
Creates file and uploads it automatically to cis.citrix.com. The login is done via the supplied credentials.show techsupport -upload -username <Citrix Username> -password <Citrix Password>
HA Node statusshow ha node
Set the current HA node to Stayprimary. (For Staysecondary just adapt the command)set ha node -hastatus stayprimary
Perform HA synchronization (parameters for single synchronization instead of all are: bookmarks, ssl, htmlinjection, imports, misc, all_plus_misc).sync ha files all
Disable HA Syncset ha node -hasync disabled
HA Failoverforce ha failover
Routing tableshow route
Add static routeadd route <Network> <Netmask> <Gateway>
Remove static routerm route <Network> <Netmask> <Gateway>
Network Interfaces Detailedshow interface
Network Interfaces Compactshow interface -summary
Detailed information network interfaceshow interface <Interface Number>
Enables network interfaceenable interface <Interface Number>
Disables network interfacedisable interface <Interface Number>
show techsupport -upload -username Citrix Username -password Citrix Password

System Information

ExplanationCommand
Collection of information (e.g. firmware, host names, etc.)show ns info
Firmware versionshow version
Hostnameshow hostname
License detailsshow license
Hardware Details & Serial Numbershow hardware
HA Node configurationshow node
IP addresses (NSIP, SNIP,VIP, MIP)show ip
ARP tableshow arp
VLANsshow vlan
DNS Servershow dns addrec -type proxy
RPC Node Informationshow ns rpcnode
All current connectionsshow connectiontable
All current connections, filtered on defined IP addressshow connectiontable | grep <IP Address>
Current AAA Sessionsshow aaa session
Current Persistence Sessionsshow persistentsessions
Cached http objectsshow cache object
Cached http objects limited to specific ContentGroupshow cache object | grep -i “<ContentGroup>”
Detailed display of cached http objects (locator can be retrieved via previous command)show cache object -locator <locator>
show ns info

Load Balancing

ExplanationCommand
Load Balancing vServer List & Configurationshow lb vserver | more
Detailed Load Balancing vServer configurationshow lb vserver <LB vServer Name>
Enables Load Balancing vServerenable lb vserver <LB vServer Name>
Disables Load Balancing vServerdisable lb vserver <LB vServer Name>
Load Balancing Service List & Configurationshow service | more
Detaillierte Load Balancing Service Konfigurationshow service <LB Service Name>
Enables Load Balancing Serviceenable service <LB Service Name>
Disables Load Balancing Servicedisable service <LB Service Name>
Load Balancing Service Group List & Configurationshow servicegroup | more
Detailed Load Balancing Service Group Configurationshow servicegroup <LB Servicegroup Name>
Enables Load Balancing Service Groupenable servicegroup <LB Service Group Name>
Disables Load Balancing Service Group (Delay in seconds)disable servicegroup <LB Service Group Name> -delay <Seconds>
Load Balancing Server List & Configurationshow server | more
Detailed Load Balancing Server Configurationshow server <LB Server Name>
Enables Load Balancing Serverenable server <LB Server Name>
Disables Load Balancing Server (Delay in seconds)disable server <LB Server Name> -delay <Seconds>
Load Balancing Monitor List & Configurationshow monitor | more
Detailed Load Balancing Monitor Configurationshow monitor <LB Monitor Name>
Enables Load Balancing Monitorenable monitor <LB Monitor Name>
Disables Load Balancing Monitordisable monitor <LB Monitor Name>
CLI configuration for a specific Citrix ADC object (here Load Balancer vServer)sh run | grep -i “<LB vServer Name>”
show lb vserver | more

Content Switching

ExplanationCommand
Content Switch vServer List & Configurationshow cs vserver | more
Detailed Content Switch vServer Configurationshow cs vserver <CS vServer Name>
Enables Content Switch vServerenable cs vserver <CS vServer Name>
Disables Content Switch vServerdisable cs vserver <CS vServer Name>
Content Switch Action List & Configurationshow cs action | more
Content Switch Policy List & Configurationshow cs policy | more
Detailed Content Switch Policy Configurationshow cs policy <CS Policy Name>
CLI configuration for a specific Citrix ADC object (here Content Switch Action)sh run | grep -i “<CS Action Name>”
show cs policy | more

VPN / Gateway

ExplanationCommand
VPN / Gateway vServer List & Configurationshow vpn vserver | more
Detailed VPN / Gateway vServer Configurationshow vpn vserver <VPN / Gateway vServer Name>
Enables VPN vServerenable vpn vserver <VPN / Gateway vServer Name>
Disables VPN vServerdisable vpn vserver <VPN / Gateway vServer Name>
CLI configuration for a specific Citrix ADC object (here VPN / Gateway vServer)sh run | grep -i “<VPN / Gateway vServer Name>”
sh run | grep -i "VPN"

AAA

ExplanationCommand
AAA vServer List & Configurationshow authentication vserver | more
Detailed AAA vServer Configurationshow authentication vserver <AAA vServer Name>
Enables AAA vServerenable authentication vserver <AAA vServer Name>
Disables AAA vServerdisable authentication vserver <AAA vServer Name>
AAA Policy List & Configurationshow authentication policy | more
Detailed AAA Policy Configurationshow authentication policy <AAA Policy Name>
AAA LDAP Action List & Configurationshow authentication ldapaction | more
Detailed AAA LDAP Action Configurationshow authentication ldapaction <AAA LDAP Action Name>
AAA LDAP Policy List & Configurationshow authentication ldappolicy | more
Detailed AAA LDAP Policy Configurationshow authentication ldappolicy <AAA LDAP Policy Name>
AAA SAML Policy List & Configurationshow authentication samlpolicy | more
Detailed AAA SAML Policy Configurationshow authentication samlpolicy <AAA SAML Policy Name>
AAA SAML Action List & Configurationshow authentication samlaction | more
Detailed AAA SAML Action Configurationshow authentication samlaction <AAA SAML Action Name>
AAA SAML IdP Policy List & Configurationshow authentication samlIdPpolicy | more
Detailed AAA SAML IdP Policy Configurationshow authentication samlIdPpolicy <AAA samlIdPpolicy Name>
AAA SAML IdP Profile List & Configurationshow authentication samlIdPprofile | more
Detailed AAA SAML IdP Profile Configurationshow authentication samlIdPprofile <AAA SAML IdP Profile Name>
AAA Radius Action List & Configurationshow authentication radiusaction | more
Detailed AAA Radius Action Configurationshow authentication radiusaction <AAA Radius Action Name>
AAA Radius Policy List & Configurationshow authentication radiuspolicy | more
Detailed AAA Radius Policy Configurationshow authentication radiuspolicy <AAA Radius Policy Name>
CLI configuration for a specific Citrix ADC object (here AAA vServer)sh run | grep -i “<AAA vServer Name>”
show aaa session

SSL

ExplanationCommand
Advanced SSL parametersshow ssl parameter
SSL vServer List & Configurationshow ssl vserver | more
Detailed SSL vServer Configurationshow ssl vserver <SSL vServer Name>
SSL Policy List & Configurationshow ssl policy | more
Detailed SSL Policy Configurationshow ssl policy <SSL Policy Name>
SSL Action List & Configurationshow ssl action | more
Detailed SSL Action Configurationshow ssl action <SSL Action Name>
SSL Profile List & Configurationshow ssl profile | more
Detailed SSL Profile Configurationshow ssl profile <SSL Profile Name>
SSL Service List & Configurationshow ssl service | more
Detailed SSL Policy Configurationshow ssl service <SSL Service Name>
SSL Service Group List & Configurationshow ssl servicegroup | more
Detailed SSL Service Group Configurationshow ssl servicegroup <SSL Service Group Name>
SSL Certificates / CA List & Configurationshow ssl certkey | more
Detailed SSL Certificate / CA Configurationshow ssl certkey <SSL Certificate / CA Name>
Certificates linkingshow ssl certlink
CLI configuration for a specific Citrix ADC object (here SSL vServer)sh run | grep -i “<SSL vServer Name>”
show ssl parameter

Statistics

ExplanationCommand
Citrix ADC statisticsstat ns
SSL statisticsstat ssl
Interface statisticsstat interface
Detailed interface statisticsstat interface <Interface Name>
CPU statisticsstat cpu
RAM consumptionstat cache -detail | grep -i “Utilized memory”
AAA statisticsshow aaa stats
Statistics of all LB vServersstat lb vserver -full
Load Balancing vServer statisticsstat lb vserver <LB vServer Name>
Statistics of all LB Servicesstat service -full
Load Balancing Service statisticsstat service <LB Service Name>
Statistics of all LB Service Groupsstat servicegroup -full
Load Balancing Service Group statisticsstat servicegroup <LB Service Group Name>
Statistics of all LB Serversstat server -full
Load Balancing Server statisticsstat server <LB Server Name>
Statistics of all CS vServer stat cs vserver -full
Content Switching vServer statisticsstat cs vserver <CS vServer Name>
Statistics of all VPN / Gateway vServersstat vpn vserver -full
VPN / Gateway vServer statisticsstat vpn vserver <VPN / Gateway vServer Name>
Statistics of all AAA vServersstat authentication vserver -full
AAA vServer statisticsstat authentication vserver <AAA vServer Name>
Statistics of all AAA Policystat authentication policy -full
AAA Policy statisticsstat authentication policy <AAA Policy Name>
Statistics of all AAA SAML IdP Policystat authentication samlIdPpolicy -full
AAA SAML IdP Policy statisticsstat authentication samlIdPpolicy <AAA SAML IdP Policy Name>
Statistics of all SSL vServersstat ssl vserver -full
SSL vServer statisticsstat ssl vserver <SSL vServer Name>
stat ns

Configuration mode (shell) commands

The configuration mode belongs to the BSD kernel and is accessible via the CLI. In the CLI you have to execute the command shell to get into the configuration mode.

General Commands

ExplanationCommand
Exit configuration modeexit (Ctrl + D)
Traceroutetraceroute <IP or DNS Name>
Pingping <IP or DNS Name>
Telnettelnet <IP or DNS Name>
Dig (DNS Utility) [until BSD kernel 10.x]dig <IP or DNS Name>
Drill (DNS Utility) [from BSD kernel 10.x]drill <IP or DNS Name>
List of running processesps -ax
ADC “Task Manager”top
Unpacking of .tar.gz files (Here e.g. Historical newnslog file for later analysis)tar xvfz /var/nslog/newnslog.99.tar.gz
ps -aux

System Information

ExplanationCommand
Current operating time ADCuptime
Detailed ADC info (description, model, platform, CPU, etc.)sysctl -a netscaler | more
Disk spacedf -h
View the integrated cachenscachemgr -a
uptime

Logging

ExplanationCommand
LDAP Authentication Log Outputcat /tmp/aaad.debug
Delete Kerberos tickets (Important for troubleshooting of the Kerberos auth)nsapimgr_wr.sh -ys call=ns_aaa_flush_kerberos_tickets
Kerberos Authentication Log Outputcat /tmp/nskrb.debug
Current real-time info from ns.logtail -f /var/log/ns.log
Current real-time info regarding SNMP from ns.logtail -f /var/log/ns.log | grep -i “snmp”
Current Hardware Error & Boot Sequence Error Logdmesg
Displays real-time packets from / to <IP Address>nstcpdump.sh host <IP Address>
Displays real-time packets between <IP Address> and <IP Address>nstcpdump.sh host <IP Address> and host <IP Address>
Displays real-time packets on port <Port Number>nstcpdump.sh port <Port Number>
Displays real-time packets from / to <IP Address> on port <Port Number>nstcpdump.sh host <IP Address> and port <Port Number>
Displays real-time packets from / to <Network Address> with <Subnet Mask>nstcpdump.sh net <Network Address> mask <Subnet Mask>
Displays real-time packets from / to <IP Address> between port <Start Port> and port <End Port>nstcpdump.sh host <IP Address> and portrange <Start Port-End Port>
Displays <Packet Count> of real-time packets from / to <IP Address>nstcpdump.sh -c <Packet Count> host <IP Address>
Displays real-time packets from / to <IP Address> or <IP Address>nstcpdump.sh host <IP Address> or host <IP Address>
Displays real-time tcp packets from / to <IP Address>nstcpdump.sh host <IP Address> and tcp
Displays real-time udp packets from / to <IP Address>nstcpdump.sh host <IP Address> and udp
Displays real-time arp packets from / to <IP Address>nstcpdump.sh host <IP Address> and arp
Displays real-time icmp packets from / to <IP Address>nstcpdump.sh host <IP Address> and icmp
Capture of real-time packets in Wireshark capture formatnstcpdump.sh port <Port Number> -w /var/tmp/test.pcap
Capture the real-time packets of the specified interfaces in Wireshark capture format (Important -i keyword works only in Wireshark capture format)nstcpdump.sh -w /var/tmp/test.pcap -i <Interface Number> -i <Interface Number>
dmesg

nsconmsg

The most important tool for troubleshooting in configuration mode is nsconmsg. A small briefing follows, how this tool is to be served. Later still special commands follow, which are more understandable thereby.

General nsconmsg parameters are:

-d <Operation>

-dCurrentCurrent performance data
-dStatsCurrent statistics counter
-dMemstatsCurrent memory statistics
nsconmsg -d current

-K <File Name>

-KnewnslogPerformance information from this log file

-s <name=value>

-sConBL=2Load Balancing performance data
-sConCSW=2Content Switch performance data
-sConSSL=3SSL performance data (1 = Front End Connections / 2 = Back End Connections / 3 = Front & Back End Connections)

-g <Match String>

-gnic_errFilters to only the information that matches the string
nsconmsg -K /var/nslog/newnslog -g cc_cpu_use -s disptime=1 -d current | more

ExplanationCommand
Analyze the unpacked newnslog.99 file. Here on historical memory usage.nsconmsg -K /var/nslog/newnslog.99 -s ConMEM=2 -d oldconmsg | more
Check if network packets were dropped by ADC due to a bandwidth limitationnsconmsg -K /var/nslog/newnslog -g nic_err_rl -d current -s disptime=1 | more
Policy Hits for Session Policiesnsconmsg -d current  -g _hits
Policy Hits for Rewritesnsconmsg –d current | egrep –i rewrite
Policy Hits for Respondernsconmsg –d current | egrep –i responder
Current memory statisticsnsconmsg -K /var/nslog/newnslog -d memstats
Current memory errorsnsconmsg -K /var/nslog/newnslog -g mem_err -d statswt0
Data file start and end timensconmsg -K /var/nslog/newnslog -d setime
Archive file start and end timezcat /var/nslog/newnslog.99.gz | nsconmsg -K pipe -d setime
Restricting the log file to a specific time rangensconmsg -K /var/log/newnslog -s time=12Aug2021:00:00 -k short_log.nsl -T 1200 -d copy
Current statistics counternsconmsg -K /var/nslog/newnslog -d stats | more
Statistics of the specific counter, here ssl_err & nic_errnsconmsg -K /var/nslog/newnslog -g nic_err -g ssl_err –s disptime=1 -d current
Current statistics SAML Auth.nsconmsg -d current -g saml
Historical statistics SAML Auth.nsconmsg -d stats -g saml
Network statistics of the specified Load Balancer vServer. Via ConLb the level of detail of the output can be defined (1 or 2)nsconmsg -K /var/nslog/newnslog -j <LB vServer Name> -T 7 -s ConLb=2 -d oldconmsg
Current CPU utilization (Pay attention to totalcount-val, 463 would be e.g. 46,3 %)nsconmsg -K /var/nslog/newnslog -g cpu_use -s disptime=1 -d current | more
Current Packet Engine (PE) CPU utilization (pay attention to totalcount-val, 463 would be e.g. 46,3 %)nsconmsg -K /var/nslog/newnslog -g cc_cpu_use -s disptime=1 -d current | more
Current management CPU utilization (pay attention to totalcount-val, 463 would be e.g. 46,3 %)nsconmsg -K /var/nslog/newnslog -g mgmt_cpu_use -s disptime=1 -d current | more
Time span covered by a given newnslog file.nsconmsg -K /var/nslog/newnslog -d setime
Current eventsnsconmsg -d current -d event
All ADC monitors currently marked as DOWN and the reason whynsconmsg -K /var/nslog/newnslog -d event | grep -i “DOWN;”
Checks the HA packets (pay attention to the delta column. If the number here changes upwards, there are network problems between the ADC nodes).nsconmsg -K /var/nslog/newnslog -s disptime=1 -d current -g ha_tot_pkt_rx | more
Consoles messagesnsconmsg -K /var/nslog/newnslog -d consmsg
Checks if IP conflicts have been detected in a subnet used by the Citrix ADCnsconmsg -K /var/nslog/newnslog -d consmsg | grep -i conflict
nsconmsg -d current  -g _hits

Citrix ADC Update

The Citrix ADC update comes at a regular interval. It is important to note here that an update always affects users and should therefore not be carried out carelessly.

Procedure

  1. Create snapshot of the machine (if VPX)
  2. Save current configuration
  3. Create system backup (CLI: create system backup -level full or GUI: System > Backup and Restore) and download from system (CLI: /var/ns_sys_backup/ or GUI)
  4. Check used features, as far as possible, before update (gateway access, load balancer, LDAP access)
  5. Only update in hours of low operation, because even in the HA cluster short-term connection problems occur (user receives a message that he must reconnect for approx. 3 seconds)
  6. Update in HA Cluster the Secondary Node
  7. Check Secondary Node configuration for completeness
  8. Switch the Secondary Node to Primary Node
  9. Functional test of used features (gateway access, load balancer, LDAP access)
  10. Update of the former Primary Node
  11. Check HA status and synchronization
  12. Switch the HA Nodes
  13. Functional test of used features (gateway access, load balancer, LDAP access)

Using the CLI

Since the update via the GUI sometimes hangs, it actually always makes sense to perform this via the CLI. To do this, connect to the Citrix ADC via putty (NetScaler IP).

putty

Then enters its credentials in the following window.

Citrix ADC NSIP Connect

To be on the safe side, we first save the running configuration using the command.

save config

Now you have to switch to configuration mode and create a folder for the new image.

shell
mkdir /var/nsinstall/Version

The update can be copied into this folder via WinSCP or similar. After this is done, the file must be unpacked.

cd /var/nsinstall/Version/
tar xzvf Update File.tgz

Starts the update after unpacking with the command.

./installns

Then restart the system and check if everything is working.

Using the GUI

Of course, you can also start the update via GUI. This helps you to avoid uploading and unpacking the new firmware in case of errors. First, log on to the Citrix ADC machine (NetScaler IP).

Logon mask Citrix ADC

If it is an HA cluster, the following message should appear. With this we know that we can safely perform the update without restricting the users.

You are connected to a secondary node

First we save the running configuration for safety’s sake, for this we click on the disk in the upper right corner. Under HA status we also see that we are on the secondary node.

Save Config

After that click on System Upgrade.

Citrix ADC System Upgrade

In the following window we check if there is still enough space (used >55%) available on the /var directory for the update.

System Upgrade

If there is enough free space, click Choose File and click Local.

Choose File

Selects the downloaded firmware file there.

Firmware

Check the settings under Upgrade Options and Citrix ADM Service Connect. If no Citric ADM is available, the option under there can be disabled.

Citrix ADM Service Connect

Important is under Upgrade Options. If Reboot after successful installation is selected there, it does not get a clean message that the system is rebooting.

Reboot after successful installation

It just seems to hang in the installation step. After refreshing the browser, you see the new firmware and that the Citrix ADC is already booted.

Stuck Upgrade

Start the update by clicking on Upgrade.

Upgrade Citrix ADC

A window opens and you can see that the firmware data is being uploaded.

Uploading Firmware

After that the firmware update will be installed and you will see the following message at the end.

Update done

As indicated, simply restart the machine.

Free Disk Space

If one of the following messages appears during the update:

Error: No space left on /flash/ filesystem, aborting installatio

Then space must be freed on the respective drive of the Citrix ADC machine. First, the 10 largest directories on the respective affected area are checked.

All commands must be executed in configuration mode (shell).

du -a /flash | sort -n -r | head -n 10

Now you can check why the directories are consuming so much disk space. In the listed images, I would delete the old firmware states under /var/nsinstall (build-12.1-62.25) and /flash/ (ns-12.1-62.25 & ns-12.1-62.23), as well as clean up the oldest logs under /var/nslog. However, it is important here not to delete the data of the currently used firmware!

Classically, even without the previous command, the following directories can be cleaned up.

Verzeichnis / DateiBefehl
/var/nstracerm -r /var/nstrace/*
/var/ns_system_backuprm -r /var/ns_system_backup/*
/var/tmp/supportrm -r /var/tmp/support/*
/var/nsinstallrm -r /var/nsinstall/<Old Firmware Version>
/var/corerm -r /var/core/*
/var/crashrm -r /var/crash/*
/flash/<Old Firmware Version>.gzrm -r /flash/<Old Firmware Version> (Nicht die aktuelle!)

HA Sync error

Citrix has enabled the security option for all RPC nodes by default starting with version 13.0 build 64.35 & 12.1 build 61.18.

This means that the communication between ADC nodes in the HA network, cluster or GSLB is only secure via port 3008 and 3009. So if necessary, the network firewalls must also be adapted so that the traffic gets through.

Secure HA is automatically activated for communication between the HA pairs. This can lead to the following message appearing after the update, when logging in for the first time.

Unable to establish connection with the secondary. Command propagation failed

The status of the HA pair (System > High Availability > Nodes) also shows the Synchronization State FAILED with the message “Unable to connect to primary, please check the network connectivity from secondary to primary”.

Unable to connect to primary, please check the network connectivity from secondary to primary System > High Availability > Nodes Synchronisation State FAILED

First, check the appropriate RPC nodes (nsrpcs-127.0.0.1-3008) under Traffic Management > Load Balancing > Services > Internal Services.

nsrpcs-127.0.0.1-3008 Traffic Management > Load Balancing > Services > Internal Services

Here you can see that a certificate is connected, but TLSv12 is not activated under Protocol.

Protocol TLSv12

If we enable this on both Citrix ADC nodes for the RPC point, the sync will work again.

Synchronisation State SUCCESS

This should be repeated for the remaining Internal Services so that all features can also use TLSv12.

Another solution to the issue is to enable one of the Default SSL Profiles under System > Profiles > SSL Profile.

Default SSL Profile System > Profiles > SSL Profile

One thought on “Citrix ADC 101 – Fundamentals”

Leave a Reply

Your email address will not be published. Required fields are marked *

* I consent to having this website store my submitted information so they can respond to my inquiry.