Finding in the Wild | XenServer 9 – NUMA Node Drift

An interesting performance topic in the current XenServer 9 known issues that is easy to overlook, particularly in larger Citrix environments. When a XenServer host’s CPUs are fully utilized, so-called NUMA node drift may occur. What does that

An interesting performance topic in the current XenServer 9 known issues that is easy to overlook, particularly in larger Citrix environments.

When a XenServer host’s CPUs are fully utilized, so-called NUMA node drift may occur.

What does that mean?

Normally, XenServer tries to run a VM’s vCPUs on the NUMA node that also holds the associated memory.

Under heavy CPU saturation, this association may be lost:

vCPU → NUMA node 0
RAM → NUMA node 1

The CPU then has to access memory on the other NUMA node.

➡️ Higher latency
➡️ Worse performance
➡️ Citrix VDAs may become noticeably slower

The tricky part:

Inside the VDA, this may initially look fairly unremarkable.

You can quickly end up investigating Windows, Citrix policies, FSLogix, storage or the application even though the actual issue is at the hypervisor layer.

XenServer 9 provides a particularly useful performance counter for this:

NUMA non-affine vCPUs

If this value rises alongside high host CPU utilization, it is worth examining NUMA distribution.

The workaround documented by XenServer is also noteworthy:

👉 Leave at least one CPU free on the host.

This allows XenServer to restore NUMA affinity.

My learning for Citrix performance analysis:

Do not look only at the VDA’s CPU utilization.

Always check the NUMA layer on heavily utilized XenServer hosts.

Sometimes a VDA is slow not because it lacks CPU, but because its CPU is accessing the wrong memory. 😉

Optimization of VMs by NUMA

Introduction CPU & NUMA architecture

The main task of every CPU is to process data. But here is the misconception, the faster the CPU (or the more CPUs I allocate), the faster the data will be processed. This is unfortunately not quite that easy, because before the CPU can process the data, it must be read out by the slower system RAM and that latency can slow the CPU processing. In order to minimize the time the CPU is waiting on reading data, CPU architectures include on-chip memory caches (local RAM) that are much faster than RAM (the access is up to 95% faster).

Optimization of VMs by NUMA
Continue reading “Optimization of VMs by NUMA”