Re: [PATCH v9 6/6] docs: Document CPU hotplug

[Volodymyr Babchuk <Volodymyr_Babchuk@epam.com>]

Hi,

Mykyta Poturai <Mykyta_Poturai@epam.com> writes:

I have a couple of questions, please see below

> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> ---
> v8->v9:
> * fix typos
>
> v7->v8:
> * remove support status update
> * update config option name
>
> v6->v7:
> * add testing and limitations
>
> v5->v6:
> * no changes
>
> v4->v5:
> * s/supported/implemented/
> * update SUPPORT.md
>
> v3->v4:
> * update configuration section
>
> v2->v3:
> * patch introduced
> ---
>  docs/misc/cpu-hotplug.txt | 98 +++++++++++++++++++++++++++++++++++++++
>  1 file changed, 98 insertions(+)
>  create mode 100644 docs/misc/cpu-hotplug.txt
>
> diff --git a/docs/misc/cpu-hotplug.txt b/docs/misc/cpu-hotplug.txt
> new file mode 100644
> index 0000000000..4f6068db30
> --- /dev/null
> +++ b/docs/misc/cpu-hotplug.txt
> @@ -0,0 +1,98 @@
> +CPU Hotplug
> +===========
> +
> +CPU hotplug is a feature that allows pCPU cores to be added to or removed from a
> +running system without requiring a reboot. It is implemented on x86 and Arm64
> +architectures.
> +
> +Implementation Details
> +----------------------
> +
> +CPU hotplug is implemented through the `XEN_SYSCTL_CPU_HOTPLUG_*` sysctl calls.
> +The specific calls are:
> +
> +- `XEN_SYSCTL_CPU_HOTPLUG_ONLINE`: Brings a pCPU online
> +- `XEN_SYSCTL_CPU_HOTPLUG_OFFLINE`: Takes a pCPU offline
> +- `XEN_SYSCTL_CPU_HOTPLUG_SMT_ENABLE`: Enables SMT threads (x86 only)
> +- `XEN_SYSCTL_CPU_HOTPLUG_SMT_DISABLE`: Disables SMT threads (x86 only)
> +
> +All cores can be disabled, assuming hardware support, except for the boot core.
> +Sysctl calls are routed to the boot core before doing any actual up/down
> +operations on other cores.
> +
> +If there are Xen-bound interrupts pinned to the pCPU being offlined, they will
> +be automatically migrated to other online pCPUs. Interrupts used by guest
> +domains are handled by the scheduler when it reschedules the vCPUs to a new,
> +online, pCPU. When a pCPU is being onlined, some Xen-bound interrupts will get
> +redistributed to the newly onlined pCPU to prevent imbalance.
> +
> +If pCPU being offlined has some vCPUs pinned to it, they will be automatically
> +unpinned and migrated to other online pCPUs.
> +
> +Limitations
> +-----------
> +
> +On Arm64 cpu hotplug is currently not compatible with ITS, due to an issue with
> +the redistributor assignment.

In this case you probably want to make sure that
CONFIG_CPU_ONLINE_OFFLINE is incompatible with CONFIG_HAS_ITS

> +
> +On Arm64 there can be problems with FFA if secure FW supports the notification
> +ABI, and with TEE, as both use non-static IRQ actions.

Could you please elaborate on this? AFAIK, OP-TEE has own
cpu_up/cpu_down callbacks, so it should know which CPUs are online.

> +
> +Configuration
> +-------------
> +
> +The presence of the feature is controlled by CONFIG_CPU_ONLINE_OFFLINE option.
> +It is enabled by default on x86 architecture. On Arm64, the option is disabled
> +by default and marked as EXPERT. xen-hptool userspace tool is built
> +unconditionally.
> +
> +Usage
> +-----
> +
> +Disable core:
> +
> +$ xen-hptool cpu-offline 2
> +Prepare to offline CPU 2
> +(XEN) Removing cpu 2 from runqueue 0
> +CPU 2 offlined successfully
> +
> +Enable core:
> +
> +$ xen-hptool cpu-online 2
> +Prepare to online CPU 2
> +(XEN) Bringing up CPU2
> +(XEN) GICv3: CPU2: Found redistributor in region 0 @00000a004005c000
> +(XEN) CPU2: Guest atomics will try 1 times before pausing the domain
> +(XEN) CPU 2 booted.
> +(XEN) Adding cpu 2 to runqueue 0
> +CPU 2 onlined successfully
> +
> +Disabling a core with pinned vCPUs:
> +
> +$ xl vcpu-pin 0 3 3 3
> +$ xl vcpu-pin 0 2 3 3
> +$ xl vcpu-pin 0 1 3 3
> +$ xl vcpu-pin 0 0 3 3
> +$ xen-hptool cpu-offline 3
> +Prepare to offline CPU 3
> +(XEN) Breaking affinity for d0v0
> +(XEN) Breaking affinity for d0v1
> +(XEN) Breaking affinity for d0v2
> +(XEN) Breaking affinity for d0v3
> +(XEN) Removing cpu 3 from runqueue 0
> +CPU 3 offlined successfully
> +
> +Testing
> +-------
> +
> +The CPU hotplug feature has been tested on both x86 and Arm64 QEMU setups and on
> +R-Car Gen5 (Arm64) hardware.
> +
> +The tests included:
> +- Offlining and onlining cores with no pinned vCPUs
> +- Offlining cores with pinned vCPUs
> +- Offlining cores with Xen-bound interrupts
> +- Offlining all cores except the boot core
> +- Offlining the boot core (expected to fail)
> +- Enabling and disabling SMT threads (x86 only)
> +- Offlining cores to which guests with passthrough devices are pinned

-- 
WBR, Volodymyr