Table of Contents
The following is a collection of basic information about Citrix ADC. From licensing, to the most important commands, to the update procedures that can be performed.
General Information
Here is some basic information about Citrix ADC.
Operating System and Architecture
Citrix ADC is based on the open source operating system FreeBSD. Unlike the very similar Linux, FreeBSD has a modular kernel and Citrix has taken advantage of this to modify FreeBSD’s Bash shell by removing the networking subsystem and replacing it with its own. The modifications were placed in a custom kernel module called NetScaler Core Packet Processing Engine (PPE).
So the Citrix ADC consists of two shells: the BSD kernel and the NetScaler kernel. Both work as a cohesive unit thanks to the strict delineation of roles. The BSD kernel manages the boot process, file system access and long-term logging. The NetScaler kernel controls time slicing for BSD, network access, SSL offloading, SNMP and syslog processing.
The PPE (alternatively referred to as the Packet Engine (PE)) is designed to take advantage of the performance gains that can be achieved through parallelization. Each PPE process is assigned to a core and operates as follows:
- Monitor incoming packets
- pull them off the package queue
- handle them accordingly for content switching, frontend optimization, caching, etc.
- put the packets back into the packet queue
- wait for more packages
So the process is either working on a packet or waiting for packets at any time. With multi-core CPUs this can be done in parallel. Certain cores are entrusted with certain functions. For example, core 1 might be responsible for managing network traffic, core 2 for processing TCP/IP, core 3 for processing Layer 7 (e.g. HTTP), and so on. This is possible because each process is a mini ADC that can perform all application optimization tasks supported by ADC.
The upper limit of how much parallel processing can take place at any given time is determined by the number of cores in the CPU. For example, for a CPU with 4 cores, 3 cores are assigned to 3 separate PSAs, with 1 core reserved for management functions, such as SNMP. Note that one core is always reserved for management.
When the ADC is powered on, FreeBSD boots and loads the NetScaler kernel. It lets the NetScaler kernel take over all CPUs except the management core, and then passes the reins to the ADC to complete the boot.
Platforms
Citrix ADC is available in 4 platform versions. The two virtual versions VPX and CPX. VPX for the well-known hypervisors and CPX for Docker hosts. As well as the three physical versions MPX, SDX and BLX. MPX and SDX comes directly as hardware from Citrix, where SDX is a Citrix hypervisor that can include up to 115 independent VPX (Depends on the hardware). BLX is a bare metal software version that can run on its own hardware. Not all hardware is supported here!
Each of the above mentioned platforms has a bandwidth limit stored in the license. This can be adjusted by importing a new, higher license (pay-as-you-grow). The naming scheme of the licenses directly reveals the maximum bandwidth, e.g. a VPX50 has a maximum incoming bandwidth of 50 Mbps. The outgoing traffic is not included in the Citrix bandwidth limitation.
A machine without a license installed is called Citrix ADC Express and has the following limitations:
- 20 Mbps bandwidth
- All ADC standard license features, except Citrix Gateway and L4 and L7 defenses
- Maximum 250 SSL Sessions
- 20 Mbps SSL throughput
Licensing
Citrix ADC is available in four different license models. Three different Citrix ADC models and the Citrix Gateway license. The different supported features are shown in the following list.
| Feature | Premium | Advanced | Standard | Gateway |
|---|---|---|---|---|
| Load Balancing | YES | YES | YES | |
| Content Switching | YES | YES | YES | |
| AppExpert Rate Controls | YES | YES | YES | |
| IPv6 Support | YES | YES | YES | |
| Traffic Domains | YES | YES | YES | |
| Subscriber-Aware Traffic Steering | YES | YES | YES | |
| Global Server Load Balancing (GSLB) | YES | YES | Optional | |
| Carrier-Grade Network Address Translation (CGNAT) | YES | YES | ||
| Dynamic Routing Protocols | YES | YES | ||
| Surge Protection | YES | YES | ||
| Priority Queuing | YES | YES | ||
| TriScale Clustering | YES | YES | ||
| TCP Optimizations | YES | YES | YES | |
| AppCompress | YES | YES | Optional | |
| AppCache | YES | Optional | ||
| DoS Defenses | YES | YES | YES | |
| Rewrite and Responder | YES | YES | YES | |
| AAA for Traffic Management | YES | YES | ||
| Citrix Web AppFirewall (WAF) | YES | Optional | ||
| IP Reputation | YES | Optional | ||
| nFactor Authentication | YES | YES | ||
| Cloud Connector | YES | |||
| Insight Center-Web Insight | YES | YES | YES | |
| AppExpert | YES | YES | YES | |
| ActionAnalytics | YES | YES | YES | |
| Configuration Wizards | YES | YES | YES | |
| Native Citrix Web Interface | YES | YES | ||
| Citrix Command Center | YES | YES | YES | |
| Federated Identity | YES | YES | ||
| One URL/SSO using SAML 2.0 | YES | YES | ||
| Cluster for ICA Proxy (Striped) | YES | YES | ||
| Monitoring of Citrix Apps and Desktops Traffic (Real Time) | YES | YES | ||
| Monitoring of Citrix Apps and Desktops Traffic (Historical) | YES | |||
| Monitoring of Gateway Traffic (Real Time) | YES | YES | ||
| Monitoring of Gateway Traffic (Historical) | YES | |||
| Customizable Web Portal | YES | YES | YES | YES |
| SSL VPN Remote Access | YES | YES | YES | YES |
| ICA Proxy to Citrix Virtual Apps and Desktops | YES | YES | YES | YES |
| Contextual Policies for Citrix Apps and Desktops | YES | YES | YES | YES |
| End Point Analysis | YES | YES | YES | YES |
| Secure Browser-Only Access (CVPN) | YES | YES | YES | YES |
| Always-On | YES | YES | YES | |
| Integration with StoreFront | YES | YES | YES |
Troubleshooting
Useful information and commands for troubleshooting.
Directories & Files
A list of the most important directories and files on the Citrix ADC machine.
| Explanation | Directory / File |
|---|---|
| System Syslog File | /var/log/ns.log |
| Alle logged entries | /var/log/messages |
| Authentication /Authorization Logs | /var/log/auth.log |
| Hardware Error & Boot Sequence Error Log | /var/log/dmesg.* |
| Main Log File in NS Data Format. Older files are archived in the same folder but in GZ format. | /var/nslog/newnslog |
| Core Crash Dump Files | /var/crash/vmcore.*.gz /var/core/NSPPE-**-*.gz |
| Kernel Crash Dump Files | /var/crash/kernel.* |
| Core Dump Log File | /tmp/savecore.log |
| Symbolic link to /flash/nsconfig | /nsconfig |
| Location of Citrix License Files | /flash/nsconfig/license/*.lic |
| Current configuration file. Older configurations are stored in the same folder as ns.conf.*. | /flash/nsconfig/ns.conf |
| SSL certificates location | /flash/nsconfig/ssl |
| Location of the custom monitors | /flash/nsconfig/monitors |
| Location of the firmware update files | /var/nsinstall /flash |
Processes
List of the most important processes that can be found on the Citrix ADC machine.
| Explanation | Process |
|---|---|
| NetScaler Packet Engine | nsppe |
| RBA and SSL VPN External Auth | nsaaad |
| Write the ns.conf file | nsconf |
| Controls the logging for newnslog | nslog.sh |
| HA Sync | nssync |
| Reads SSL Cert files | nsreadfile |
| SSL CRL List Update | nscrlrefresh |
| Synchronizes bookmarks and SSL certificates | nsfsyncd |
| Configuration changes through the GUI | nsnetsvc |
| Runs the monitors with script | nsumond |
| Controls the writing of the newnslog | nsconmsg |
| Collects statistics data for the Historical Reporting | nscollect |
| Routing processes | imi / ripd / ospfd / bgpd |
Command Line Interface (CLI) commands
The CLI is part of the NetScaler kernel and is the first thing you see when you connect to the machine.
General Commands
| Explanation | Command |
|---|---|
| Enables CLI Color Mode | set cli mode -color ON |
| Adding current user, hostname, time and node status to the CLI | set cli prompt %u@%h-%T-%s |
| Increase timeout for CLI session (here to 30 minutes (1800 seconds)) | set cli mode -timeout 1800 |
| History of executed commands | history | more |
| Help display for specific command | help <Command> |
| Display MAN page for specific command | man <Command> |
| Configuration menu | config ns |
| Creates backup of configuration files (/nsconfig/, /var/, /netscaler/, ns.conf) in folder /var/ns_sys_backup | create system backup <Backup Name> -level basic |
| Creates extended backup (/nsconfig/, /var/, /netscaler/, ns.conf, Certificates, License Files)in the folder /var/ns_sys_backup | create system backup <Backup Name> -level full |
| Displays existing backups | show system backup |
| Restore from existing backup | restore system backup <Backup Name> |
| Configuration mode | shell |
| Features (Available & Configured) | show feature |
| Enables certain feature (if it is supported by the installed license) | enable feature <Acronym> |
| Disables specific feature | disable feature <Acronym> |
| Mode (Available & Configured) | show ns mode |
| Enables specific mode | enable ns mode <Acronym> |
| Disables specific mode | disable ns mode <Acronym> |
| Saved configuration | show savedConfig | more |
| Running configuration | show run | more |
| Differences between the running configuration with the saved configuration | diff ns config -outtype CLI |
| Save running configuration | save config |
| Creates file under /var/tmp/support/ for manual upload to cis.citrix.com (health check of Citrix ADC) | show techsupport |
| Creates file and uploads it automatically to cis.citrix.com. The login is done via the supplied credentials. | show techsupport -upload -username <Citrix Username> -password <Citrix Password> |
| HA Node status | show ha node |
| Set the current HA node to Stayprimary. (For Staysecondary just adapt the command) | set ha node -hastatus stayprimary |
| Perform HA synchronization (parameters for single synchronization instead of all are: bookmarks, ssl, htmlinjection, imports, misc, all_plus_misc). | sync ha files all |
| Disable HA Sync | set ha node -hasync disabled |
| HA Failover | force ha failover |
| Routing table | show route |
| Add static route | add route <Network> <Netmask> <Gateway> |
| Remove static route | rm route <Network> <Netmask> <Gateway> |
| Network Interfaces Detailed | show interface |
| Network Interfaces Compact | show interface -summary |
| Detailed information network interface | show interface <Interface Number> |
| Enables network interface | enable interface <Interface Number> |
| Disables network interface | disable interface <Interface Number> |
System Information
| Explanation | Command |
|---|---|
| Collection of information (e.g. firmware, host names, etc.) | show ns info |
| Firmware version | show version |
| Hostname | show hostname |
| License details | show license |
| Hardware Details & Serial Number | show hardware |
| HA Node configuration | show node |
| IP addresses (NSIP, SNIP,VIP, MIP) | show ip |
| ARP table | show arp |
| VLANs | show vlan |
| DNS Server | show dns addrec -type proxy |
| RPC Node Information | show ns rpcnode |
| All current connections | show connectiontable |
| All current connections, filtered on defined IP address | show connectiontable | grep <IP Address> |
| Current AAA Sessions | show aaa session |
| Current Persistence Sessions | show persistentsessions |
| Cached http objects | show cache object |
| Cached http objects limited to specific ContentGroup | show cache object | grep -i “<ContentGroup>” |
| Detailed display of cached http objects (locator can be retrieved via previous command) | show cache object -locator <locator> |
Load Balancing
| Explanation | Command |
|---|---|
| Load Balancing vServer List & Configuration | show lb vserver | more |
| Detailed Load Balancing vServer configuration | show lb vserver <LB vServer Name> |
| Enables Load Balancing vServer | enable lb vserver <LB vServer Name> |
| Disables Load Balancing vServer | disable lb vserver <LB vServer Name> |
| Load Balancing Service List & Configuration | show service | more |
| Detaillierte Load Balancing Service Konfiguration | show service <LB Service Name> |
| Enables Load Balancing Service | enable service <LB Service Name> |
| Disables Load Balancing Service | disable service <LB Service Name> |
| Load Balancing Service Group List & Configuration | show servicegroup | more |
| Detailed Load Balancing Service Group Configuration | show servicegroup <LB Servicegroup Name> |
| Enables Load Balancing Service Group | enable servicegroup <LB Service Group Name> |
| Disables Load Balancing Service Group (Delay in seconds) | disable servicegroup <LB Service Group Name> -delay <Seconds> |
| Load Balancing Server List & Configuration | show server | more |
| Detailed Load Balancing Server Configuration | show server <LB Server Name> |
| Enables Load Balancing Server | enable server <LB Server Name> |
| Disables Load Balancing Server (Delay in seconds) | disable server <LB Server Name> -delay <Seconds> |
| Load Balancing Monitor List & Configuration | show monitor | more |
| Detailed Load Balancing Monitor Configuration | show monitor <LB Monitor Name> |
| Enables Load Balancing Monitor | enable monitor <LB Monitor Name> |
| Disables Load Balancing Monitor | disable monitor <LB Monitor Name> |
| CLI configuration for a specific Citrix ADC object (here Load Balancer vServer) | sh run | grep -i “<LB vServer Name>” |
Content Switching
| Explanation | Command |
|---|---|
| Content Switch vServer List & Configuration | show cs vserver | more |
| Detailed Content Switch vServer Configuration | show cs vserver <CS vServer Name> |
| Enables Content Switch vServer | enable cs vserver <CS vServer Name> |
| Disables Content Switch vServer | disable cs vserver <CS vServer Name> |
| Content Switch Action List & Configuration | show cs action | more |
| Content Switch Policy List & Configuration | show cs policy | more |
| Detailed Content Switch Policy Configuration | show cs policy <CS Policy Name> |
| CLI configuration for a specific Citrix ADC object (here Content Switch Action) | sh run | grep -i “<CS Action Name>” |
VPN / Gateway
| Explanation | Command |
|---|---|
| VPN / Gateway vServer List & Configuration | show vpn vserver | more |
| Detailed VPN / Gateway vServer Configuration | show vpn vserver <VPN / Gateway vServer Name> |
| Enables VPN vServer | enable vpn vserver <VPN / Gateway vServer Name> |
| Disables VPN vServer | disable vpn vserver <VPN / Gateway vServer Name> |
| CLI configuration for a specific Citrix ADC object (here VPN / Gateway vServer) | sh run | grep -i “<VPN / Gateway vServer Name>” |
AAA
| Explanation | Command |
|---|---|
| AAA vServer List & Configuration | show authentication vserver | more |
| Detailed AAA vServer Configuration | show authentication vserver <AAA vServer Name> |
| Enables AAA vServer | enable authentication vserver <AAA vServer Name> |
| Disables AAA vServer | disable authentication vserver <AAA vServer Name> |
| AAA Policy List & Configuration | show authentication policy | more |
| Detailed AAA Policy Configuration | show authentication policy <AAA Policy Name> |
| AAA LDAP Action List & Configuration | show authentication ldapaction | more |
| Detailed AAA LDAP Action Configuration | show authentication ldapaction <AAA LDAP Action Name> |
| AAA LDAP Policy List & Configuration | show authentication ldappolicy | more |
| Detailed AAA LDAP Policy Configuration | show authentication ldappolicy <AAA LDAP Policy Name> |
| AAA SAML Policy List & Configuration | show authentication samlpolicy | more |
| Detailed AAA SAML Policy Configuration | show authentication samlpolicy <AAA SAML Policy Name> |
| AAA SAML Action List & Configuration | show authentication samlaction | more |
| Detailed AAA SAML Action Configuration | show authentication samlaction <AAA SAML Action Name> |
| AAA SAML IdP Policy List & Configuration | show authentication samlIdPpolicy | more |
| Detailed AAA SAML IdP Policy Configuration | show authentication samlIdPpolicy <AAA samlIdPpolicy Name> |
| AAA SAML IdP Profile List & Configuration | show authentication samlIdPprofile | more |
| Detailed AAA SAML IdP Profile Configuration | show authentication samlIdPprofile <AAA SAML IdP Profile Name> |
| AAA Radius Action List & Configuration | show authentication radiusaction | more |
| Detailed AAA Radius Action Configuration | show authentication radiusaction <AAA Radius Action Name> |
| AAA Radius Policy List & Configuration | show authentication radiuspolicy | more |
| Detailed AAA Radius Policy Configuration | show authentication radiuspolicy <AAA Radius Policy Name> |
| CLI configuration for a specific Citrix ADC object (here AAA vServer) | sh run | grep -i “<AAA vServer Name>” |
SSL
| Explanation | Command |
|---|---|
| Advanced SSL parameters | show ssl parameter |
| SSL vServer List & Configuration | show ssl vserver | more |
| Detailed SSL vServer Configuration | show ssl vserver <SSL vServer Name> |
| SSL Policy List & Configuration | show ssl policy | more |
| Detailed SSL Policy Configuration | show ssl policy <SSL Policy Name> |
| SSL Action List & Configuration | show ssl action | more |
| Detailed SSL Action Configuration | show ssl action <SSL Action Name> |
| SSL Profile List & Configuration | show ssl profile | more |
| Detailed SSL Profile Configuration | show ssl profile <SSL Profile Name> |
| SSL Service List & Configuration | show ssl service | more |
| Detailed SSL Policy Configuration | show ssl service <SSL Service Name> |
| SSL Service Group List & Configuration | show ssl servicegroup | more |
| Detailed SSL Service Group Configuration | show ssl servicegroup <SSL Service Group Name> |
| SSL Certificates / CA List & Configuration | show ssl certkey | more |
| Detailed SSL Certificate / CA Configuration | show ssl certkey <SSL Certificate / CA Name> |
| Certificates linking | show ssl certlink |
| CLI configuration for a specific Citrix ADC object (here SSL vServer) | sh run | grep -i “<SSL vServer Name>” |
Statistics
| Explanation | Command |
|---|---|
| Citrix ADC statistics | stat ns |
| SSL statistics | stat ssl |
| Interface statistics | stat interface |
| Detailed interface statistics | stat interface <Interface Name> |
| CPU statistics | stat cpu |
| RAM consumption | stat cache -detail | grep -i “Utilized memory” |
| AAA statistics | show aaa stats |
| Statistics of all LB vServers | stat lb vserver -full |
| Load Balancing vServer statistics | stat lb vserver <LB vServer Name> |
| Statistics of all LB Services | stat service -full |
| Load Balancing Service statistics | stat service <LB Service Name> |
| Statistics of all LB Service Groups | stat servicegroup -full |
| Load Balancing Service Group statistics | stat servicegroup <LB Service Group Name> |
| Statistics of all LB Servers | stat server -full |
| Load Balancing Server statistics | stat server <LB Server Name> |
| Statistics of all CS vServer | stat cs vserver -full |
| Content Switching vServer statistics | stat cs vserver <CS vServer Name> |
| Statistics of all VPN / Gateway vServers | stat vpn vserver -full |
| VPN / Gateway vServer statistics | stat vpn vserver <VPN / Gateway vServer Name> |
| Statistics of all AAA vServers | stat authentication vserver -full |
| AAA vServer statistics | stat authentication vserver <AAA vServer Name> |
| Statistics of all AAA Policy | stat authentication policy -full |
| AAA Policy statistics | stat authentication policy <AAA Policy Name> |
| Statistics of all AAA SAML IdP Policy | stat authentication samlIdPpolicy -full |
| AAA SAML IdP Policy statistics | stat authentication samlIdPpolicy <AAA SAML IdP Policy Name> |
| Statistics of all SSL vServers | stat ssl vserver -full |
| SSL vServer statistics | stat ssl vserver <SSL vServer Name> |
Configuration mode (shell) commands
The configuration mode belongs to the BSD kernel and is accessible via the CLI. In the CLI you have to execute the command shell to get into the configuration mode.
General Commands
| Explanation | Command |
|---|---|
| Exit configuration mode | exit (Ctrl + D) |
| Traceroute | traceroute <IP or DNS Name> |
| Ping | ping <IP or DNS Name> |
| Telnet | telnet <IP or DNS Name> |
| Dig (DNS Utility) [until BSD kernel 10.x] | dig <IP or DNS Name> |
| Drill (DNS Utility) [from BSD kernel 10.x] | drill <IP or DNS Name> |
| List of running processes | ps -ax |
| ADC “Task Manager” | top |
| Unpacking of .tar.gz files (Here e.g. Historical newnslog file for later analysis) | tar xvfz /var/nslog/newnslog.99.tar.gz |
System Information
| Explanation | Command |
|---|---|
| Current operating time ADC | uptime |
| Detailed ADC info (description, model, platform, CPU, etc.) | sysctl -a netscaler | more |
| Disk space | df -h |
| View the integrated cache | nscachemgr -a |
Logging
| Explanation | Command |
|---|---|
| LDAP Authentication Log Output | cat /tmp/aaad.debug |
| Delete Kerberos tickets (Important for troubleshooting of the Kerberos auth) | nsapimgr_wr.sh -ys call=ns_aaa_flush_kerberos_tickets |
| Kerberos Authentication Log Output | cat /tmp/nskrb.debug |
| Current real-time info from ns.log | tail -f /var/log/ns.log |
| Current real-time info regarding SNMP from ns.log | tail -f /var/log/ns.log | grep -i “snmp” |
| Current Hardware Error & Boot Sequence Error Log | dmesg |
| Displays real-time packets from / to <IP Address> | nstcpdump.sh host <IP Address> |
| Displays real-time packets between <IP Address> and <IP Address> | nstcpdump.sh host <IP Address> and host <IP Address> |
| Displays real-time packets on port <Port Number> | nstcpdump.sh port <Port Number> |
| Displays real-time packets from / to <IP Address> on port <Port Number> | nstcpdump.sh host <IP Address> and port <Port Number> |
| Displays real-time packets from / to <Network Address> with <Subnet Mask> | nstcpdump.sh net <Network Address> mask <Subnet Mask> |
| Displays real-time packets from / to <IP Address> between port <Start Port> and port <End Port> | nstcpdump.sh host <IP Address> and portrange <Start Port-End Port> |
| Displays <Packet Count> of real-time packets from / to <IP Address> | nstcpdump.sh -c <Packet Count> host <IP Address> |
| Displays real-time packets from / to <IP Address> or <IP Address> | nstcpdump.sh host <IP Address> or host <IP Address> |
| Displays real-time tcp packets from / to <IP Address> | nstcpdump.sh host <IP Address> and tcp |
| Displays real-time udp packets from / to <IP Address> | nstcpdump.sh host <IP Address> and udp |
| Displays real-time arp packets from / to <IP Address> | nstcpdump.sh host <IP Address> and arp |
| Displays real-time icmp packets from / to <IP Address> | nstcpdump.sh host <IP Address> and icmp |
| Capture of real-time packets in Wireshark capture format | nstcpdump.sh port <Port Number> -w /var/tmp/test.pcap |
| Capture the real-time packets of the specified interfaces in Wireshark capture format (Important -i keyword works only in Wireshark capture format) | nstcpdump.sh -w /var/tmp/test.pcap -i <Interface Number> -i <Interface Number> |
nsconmsg
The most important tool for troubleshooting in configuration mode is nsconmsg. A small briefing follows, how this tool is to be served. Later still special commands follow, which are more understandable thereby.
General nsconmsg parameters are:
-d <Operation>
| -d | Current | Current performance data |
| -d | Stats | Current statistics counter |
| -d | Memstats | Current memory statistics |
-K <File Name>
| -K | newnslog | Performance information from this log file |
-s <name=value>
| -s | ConBL=2 | Load Balancing performance data |
| -s | ConCSW=2 | Content Switch performance data |
| -s | ConSSL=3 | SSL performance data (1 = Front End Connections / 2 = Back End Connections / 3 = Front & Back End Connections) |
-g <Match String>
| -g | nic_err | Filters to only the information that matches the string |
| Explanation | Command |
|---|---|
| Analyze the unpacked newnslog.99 file. Here on historical memory usage. | nsconmsg -K /var/nslog/newnslog.99 -s ConMEM=2 -d oldconmsg | more |
| Check if network packets were dropped by ADC due to a bandwidth limitation | nsconmsg -K /var/nslog/newnslog -g nic_err_rl -d current -s disptime=1 | more |
| Policy Hits for Session Policies | nsconmsg -d current -g _hits |
| Policy Hits for Rewrites | nsconmsg –d current | egrep –i rewrite |
| Policy Hits for Responder | nsconmsg –d current | egrep –i responder |
| Current memory statistics | nsconmsg -K /var/nslog/newnslog -d memstats |
| Current memory errors | nsconmsg -K /var/nslog/newnslog -g mem_err -d statswt0 |
| Data file start and end time | nsconmsg -K /var/nslog/newnslog -d setime |
| Archive file start and end time | zcat /var/nslog/newnslog.99.gz | nsconmsg -K pipe -d setime |
| Restricting the log file to a specific time range | nsconmsg -K /var/log/newnslog -s time=12Aug2021:00:00 -k short_log.nsl -T 1200 -d copy |
| Current statistics counter | nsconmsg -K /var/nslog/newnslog -d stats | more |
| Statistics of the specific counter, here ssl_err & nic_err | nsconmsg -K /var/nslog/newnslog -g nic_err -g ssl_err –s disptime=1 -d current |
| Current statistics SAML Auth. | nsconmsg -d current -g saml |
| Historical statistics SAML Auth. | nsconmsg -d stats -g saml |
| Network statistics of the specified Load Balancer vServer. Via ConLb the level of detail of the output can be defined (1 or 2) | nsconmsg -K /var/nslog/newnslog -j <LB vServer Name> -T 7 -s ConLb=2 -d oldconmsg |
| Current CPU utilization (Pay attention to totalcount-val, 463 would be e.g. 46,3 %) | nsconmsg -K /var/nslog/newnslog -g cpu_use -s disptime=1 -d current | more |
| Current Packet Engine (PE) CPU utilization (pay attention to totalcount-val, 463 would be e.g. 46,3 %) | nsconmsg -K /var/nslog/newnslog -g cc_cpu_use -s disptime=1 -d current | more |
| Current management CPU utilization (pay attention to totalcount-val, 463 would be e.g. 46,3 %) | nsconmsg -K /var/nslog/newnslog -g mgmt_cpu_use -s disptime=1 -d current | more |
| Time span covered by a given newnslog file. | nsconmsg -K /var/nslog/newnslog -d setime |
| Current events | nsconmsg -d current -d event |
| All ADC monitors currently marked as DOWN and the reason why | nsconmsg -K /var/nslog/newnslog -d event | grep -i “DOWN;” |
| Checks the HA packets (pay attention to the delta column. If the number here changes upwards, there are network problems between the ADC nodes). | nsconmsg -K /var/nslog/newnslog -s disptime=1 -d current -g ha_tot_pkt_rx | more |
| Consoles messages | nsconmsg -K /var/nslog/newnslog -d consmsg |
| Checks if IP conflicts have been detected in a subnet used by the Citrix ADC | nsconmsg -K /var/nslog/newnslog -d consmsg | grep -i conflict |
Citrix ADC Update
The Citrix ADC update comes at a regular interval. It is important to note here that an update always affects users and should therefore not be carried out carelessly.
Procedure
- Create snapshot of the machine (if VPX)
- Save current configuration
- Create system backup (CLI: create system backup -level full or GUI: System > Backup and Restore) and download from system (CLI: /var/ns_sys_backup/ or GUI)
- Check used features, as far as possible, before update (gateway access, load balancer, LDAP access)
- Only update in hours of low operation, because even in the HA cluster short-term connection problems occur (user receives a message that he must reconnect for approx. 3 seconds)
- Update in HA Cluster the Secondary Node
- Check Secondary Node configuration for completeness
- Switch the Secondary Node to Primary Node
- Functional test of used features (gateway access, load balancer, LDAP access)
- Update of the former Primary Node
- Check HA status and synchronization
- Switch the HA Nodes
- Functional test of used features (gateway access, load balancer, LDAP access)
Using the CLI
Since the update via the GUI sometimes hangs, it actually always makes sense to perform this via the CLI. To do this, connect to the Citrix ADC via putty (NetScaler IP).
Then enters its credentials in the following window.
To be on the safe side, we first save the running configuration using the command.
|
1 |
save config |
Now you have to switch to configuration mode and create a folder for the new image.
|
1 2 |
shell mkdir /var/nsinstall/<Version> |
The update can be copied into this folder via WinSCP or similar. After this is done, the file must be unpacked.
|
1 2 |
cd /var/nsinstall/<Version>/ tar xzvf <Update File>.tgz |
Starts the update after unpacking with the command.
|
1 |
./installns |
Then restart the system and check if everything is working.
|
1 2 |
exit reboot -warm |
Using the GUI
Of course, you can also start the update via GUI. This helps you to avoid uploading and unpacking the new firmware in case of errors. First, log on to the Citrix ADC machine (NetScaler IP).
If it is an HA cluster, the following message should appear. With this we know that we can safely perform the update without restricting the users.
First we save the running configuration for safety’s sake, for this we click on the disk in the upper right corner. Under HA status we also see that we are on the secondary node.
After that click on System Upgrade.
In the following window we check if there is still enough space (used >55%) available on the /var directory for the update.
If there is enough free space, click Choose File and click Local.
Selects the downloaded firmware file there.
Check the settings under Upgrade Options and Citrix ADM Service Connect. If no Citric ADM is available, the option under there can be disabled.
Important is under Upgrade Options. If Reboot after successful installation is selected there, it does not get a clean message that the system is rebooting.
It just seems to hang in the installation step. After refreshing the browser, you see the new firmware and that the Citrix ADC is already booted.
Start the update by clicking on Upgrade.
A window opens and you can see that the firmware data is being uploaded.
After that the firmware update will be installed and you will see the following message at the end.
As indicated, simply restart the machine.
Free Disk Space
If one of the following messages appears during the update:
|
1 2 3 4 5 6 7 8 9 10 11 |
Error: No space left on /var/ filesystem, aborting Error: No space left on /var/ filesystem, aborting installation… installns: [1455]: Error: No space left on /var/ filesystem, aborting installation… Error: No space left on /flash/ filesystem, aborting Error: No space left on /flash/ filesystem, aborting installation… installns: [1455]: Error: No space left on /flash/ filesystem, aborting installation… |
Then space must be freed on the respective drive of the Citrix ADC machine. First, the 10 largest directories on the respective affected area are checked.
All commands must be executed in configuration mode (shell).
|
1 |
du -a /var | sort -n -r | head -n 10 |
|
1 |
du -a /flash | sort -n -r | head -n 10 |
Now you can check why the directories are consuming so much disk space. In the listed images, I would delete the old firmware states under /var/nsinstall (build-12.1-62.25) and /flash/ (ns-12.1-62.25 & ns-12.1-62.23), as well as clean up the oldest logs under /var/nslog. However, it is important here not to delete the data of the currently used firmware!
Classically, even without the previous command, the following directories can be cleaned up.
| Verzeichnis / Datei | Befehl |
|---|---|
| /var/nstrace | rm -r /var/nstrace/* |
| /var/ns_system_backup | rm -r /var/ns_system_backup/* |
| /var/tmp/support | rm -r /var/tmp/support/* |
| /var/nsinstall | rm -r /var/nsinstall/<Old Firmware Version> |
| /var/core | rm -r /var/core/* |
| /var/crash | rm -r /var/crash/* |
| /flash/<Old Firmware Version>.gz | rm -r /flash/<Old Firmware Version> (Nicht die aktuelle!) |
HA Sync error
Citrix has enabled the security option for all RPC nodes by default starting with version 13.0 build 64.35 & 12.1 build 61.18.
This means that the communication between ADC nodes in the HA network, cluster or GSLB is only secure via port 3008 and 3009. So if necessary, the network firewalls must also be adapted so that the traffic gets through.
Secure HA is automatically activated for communication between the HA pairs. This can lead to the following message appearing after the update, when logging in for the first time.
The status of the HA pair (System > High Availability > Nodes) also shows the Synchronization State FAILED with the message “Unable to connect to primary, please check the network connectivity from secondary to primary”.
First, check the appropriate RPC nodes (nsrpcs-127.0.0.1-3008) under Traffic Management > Load Balancing > Services > Internal Services.
Here you can see that a certificate is connected, but TLSv12 is not activated under Protocol.
If we enable this on both Citrix ADC nodes for the RPC point, the sync will work again.
This should be repeated for the remaining Internal Services so that all features can also use TLSv12.
Another solution to the issue is to enable one of the Default SSL Profiles under System > Profiles > SSL Profile.
