Re: [PATCH v11 6/6] docs: Document CPU hotplug

[Volodymyr Babchuk <Volodymyr_Babchuk@epam.com>]

Hi,

Mykyta Poturai <Mykyta_Poturai@epam.com> writes:

> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>

Reviewed-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>

> ---
> v10->v11:
> * sync with latest changes
>
> v9->v10:
> * explain the actual ITS incompatibility
> * say FFA and TEE are incompatible, matching the Kconfig dependency
>
> v8->v9:
> * no changes
>
> v7->v8:
> * remove support status update
> * update config option name
>
> v6->v7:
> * add testing and limitations
>
> v5->v6:
> * no changes
>
> v4->v5:
> * s/supported/implemented/
> * update SUPPORT.md
>
> v3->v4:
> * update configuration section
>
> v2->v3:
> * patch introduced
> ---
>  docs/misc/cpu-hotplug.txt | 102 ++++++++++++++++++++++++++++++++++++++
>  1 file changed, 102 insertions(+)
>  create mode 100644 docs/misc/cpu-hotplug.txt
>
> diff --git a/docs/misc/cpu-hotplug.txt b/docs/misc/cpu-hotplug.txt
> new file mode 100644
> index 0000000000..e1b474ca69
> --- /dev/null
> +++ b/docs/misc/cpu-hotplug.txt
> @@ -0,0 +1,102 @@
> +CPU Hotplug
> +===========
> +
> +CPU hotplug is a feature that allows pCPU cores to be added to or removed from a
> +running system without requiring a reboot. It is implemented on x86 and Arm64
> +architectures.
> +
> +Implementation Details
> +----------------------
> +
> +CPU hotplug is implemented through the `XEN_SYSCTL_CPU_HOTPLUG_*` sysctl calls.
> +The specific calls are:
> +
> +- `XEN_SYSCTL_CPU_HOTPLUG_ONLINE`: Brings a pCPU online
> +- `XEN_SYSCTL_CPU_HOTPLUG_OFFLINE`: Takes a pCPU offline
> +- `XEN_SYSCTL_CPU_HOTPLUG_SMT_ENABLE`: Enables SMT threads (x86 only)
> +- `XEN_SYSCTL_CPU_HOTPLUG_SMT_DISABLE`: Disables SMT threads (x86 only)
> +
> +All cores can be disabled, assuming hardware support, except for the boot core.
> +Sysctl calls are routed to the boot core before doing any actual up/down
> +operations on other cores.
> +
> +If there are Xen-bound interrupts pinned to the pCPU being offlined, they will
> +be automatically migrated to other online pCPUs. Interrupts used by guest
> +domains are handled by the scheduler when it reschedules the vCPUs to a new,
> +online, pCPU. When a pCPU is being onlined, some Xen-bound interrupts will get
> +redistributed to the newly onlined pCPU to prevent imbalance.
> +
> +If pCPU being offlined has some vCPUs pinned to it, they will be automatically
> +unpinned and migrated to other online pCPUs.
> +
> +Limitations
> +-----------
> +
> +On Arm64 cpu hotplug is currently not compatible with the ITS: the per-CPU LPI
> +pending table and GICR_CTLR.EnableLPIs are set up once at boot and are not
> +re-usable, so onlining a pCPU a second time fails.
> +
> +On Arm64 it is also not compatible with the FFA and TEE mediators, as both use
> +non-static IRQ actions.
> +
> +Configuration
> +-------------
> +
> +CPU hotplug support in Xen is controlled by the CONFIG_CPU_ONLINE_OFFLINE
> +Kconfig option, which requires EXPERT and depends on HAS_CPU_ONLINE_OFFLINE.
> +HAS_CPU_ONLINE_OFFLINE is selected by x86, and by Arm64 when neither ITS nor
> +TEE support is enabled. The option is enabled by default on x86 and disabled
> +by default on Arm64.
> +
> +The xen-hptool userspace tool is built unconditionally.
> +
> +Usage
> +-----
> +
> +Disable core:
> +
> +$ xen-hptool cpu-offline 2
> +Prepare to offline CPU 2
> +(XEN) Removing cpu 2 from runqueue 0
> +CPU 2 offlined successfully
> +
> +Enable core:
> +
> +$ xen-hptool cpu-online 2
> +Prepare to online CPU 2
> +(XEN) Bringing up CPU2
> +(XEN) GICv3: CPU2: Found redistributor in region 0 @00000a004005c000
> +(XEN) CPU2: Guest atomics will try 1 times before pausing the domain
> +(XEN) CPU 2 booted.
> +(XEN) Adding cpu 2 to runqueue 0
> +CPU 2 onlined successfully
> +
> +Disabling a core with pinned vCPUs:
> +
> +$ xl vcpu-pin 0 3 3 3
> +$ xl vcpu-pin 0 2 3 3
> +$ xl vcpu-pin 0 1 3 3
> +$ xl vcpu-pin 0 0 3 3
> +$ xen-hptool cpu-offline 3
> +Prepare to offline CPU 3
> +(XEN) Breaking affinity for d0v0
> +(XEN) Breaking affinity for d0v1
> +(XEN) Breaking affinity for d0v2
> +(XEN) Breaking affinity for d0v3
> +(XEN) Removing cpu 3 from runqueue 0
> +CPU 3 offlined successfully
> +
> +Testing
> +-------
> +
> +The CPU hotplug feature has been tested on both x86 and Arm64 QEMU setups and on
> +R-Car Gen5 (Arm64) hardware.
> +
> +The tests included:
> +- Offlining and onlining cores with no pinned vCPUs
> +- Offlining cores with pinned vCPUs
> +- Offlining cores with Xen-bound interrupts
> +- Offlining all cores except the boot core
> +- Offlining the boot core (expected to fail)
> +- Enabling and disabling SMT threads (x86 only)
> +- Offlining cores to which guests with passthrough devices are pinned

-- 
WBR, Volodymyr