Re: [RFC 00/52] Introduce hybrid CPU topology

[Zhao Liu <zhao1.liu@linux.intel.com>]

On Mon, Feb 13, 2023 at 10:14:22AM +0000, Alex Bennée wrote:
> Date: Mon, 13 Feb 2023 10:14:22 +0000
> From: Alex Bennée <alex.bennee@linaro.org>
> Subject: Re: [RFC 00/52] Introduce hybrid CPU topology
> 
> 
> Zhao Liu <zhao1.liu@linux.intel.com> writes:
> 
> > From: Zhao Liu <zhao1.liu@intel.com>
> >
> > Hi list,
> >
> > This RFC series is to introduce hybrid topology in QEMU, and is based
> > on our previous patches about cluster support for x86 [1] ([PATCH RESEND
> > 00/18] Support smp.clusters for x86).
> >
> > Though our cluster support patches [1] has not yet accepted, we are
> > still posting the hybrid topology support patches in advance, hoping to
> > get discussion and feedbacks.
> >
> > In this RFC series, we mainly have the following work:
> > * Introduce the new cpu topology under heterogeneous architecture, which
> >   we called "hybrid" cpu topology.
> >       - since the name "hmp" is already used in QEMU and "hybrid" is
> >         also a synonym for heterogeneous.
> > * Reorganized the relevant topology information in qemu to adapt to the
> >   hybrid topology.
> > * Introduce a new command "hybrid", which allows user to set a custom
> >   hybrid topology.
> >       - currently limited to custom cluster and custom core
> > * Support hybrid topology in i386 PC machine.
> > * Expose basic topology information (topology type and core type) in
> >   query_cpus_fast.
> >
> > We hope that the hybrid topology can be general enough to be compatible
> > with the needs of hybrids with various architectures in the future.
> >
> > Welcome your feedbacks!
> >
> >
> > # 1. Background
> >
> > Heterogeneous computing refers to systems that use more than one kind of
> > processor or cores [2]. Now there are more and more heterogeneous
> > computing architectures, typically including arm's big.LITTLE [3] and
> > intel hybrid architecture [4].
> >
> > The development of heterogeneous computing brings new challenges to
> > virtualization. The first problem we face is how to support the vCPU
> > topology under the heterogeneous architecture.
> >
> > Here, to distinguish it from "smp" cpu topology, we call it "hybrid" cpu
> > topology.
> >
> >
> > # 2. Motivation
> >
> > In a typical hybrid system, different types of core may have different
> > capabilities. This difference may be reflected in different capacities,
> > performance or power efficiencies, and even different instruction-per-
> > cycle (IPC) capabilities. The direct consequence of these differences is
> > to affect the performance of the workload.
> 
> This elides a use case for heterogeneous computing under emulation where
> different cores can have totally different ISA's. We already support
> emulation models where we have varying classes of the same ISA, for
> example xlnx-zyncmp supports 4 A profile and 2 R profile Arm chips.

This is emulation mode, so you also need to define heterogeneous CPU
topology in this case, right?

> 
> I've glanced over the patch set and I don't think there is any direct
> clash with that as this is mainly about representing the topology for
> x86 non-emulation accelerators but I just want ensure we don't end up
> stepping on each others toes.

Yes, this is for non-emulation case.

> 
> > For a virtual machines, vCPUs are just threads, although we can bind
> > vCPUs to different cores to benefit from the different advantages of
> > these physical cores, it's obviously not enough since CPU topology (and
> > cache topology) will still have a significant impact on scheduling
> > performance.
> >
> <snip>
> > ## 3.3. "-hybrid" command
> >
> > For hybrid cpu topology configuration, the original "-smp" lack of
> > flexibility to expand, and unables to customize different cores.
> >
> > So we introduce "-hybrid" command and implement it as the multi-
> > line command. The multi-line command format is more complex than the
> > single-line one, but it can bring stronger scalability and
> > intuitiveness. In the future, it can also be easily extended to more
> > heterogeneous topology levels.
> >
> > "-hybrid" command is as follows:
> >
> > -hybrid socket,sockets=n
> > -hybrid die,dies=n
> > -hybrid cluster,clusters=n
> > -hybrid core,cores=n,type=core_type[,threads=threads]
> >         [,clusterid=cluster]
> >
> > Here, we first define the corresponding qapi options for these 4
> > topology levels: core, cluster, die and socket.
> >
> > We doesn't need a thread level since thread doesn't have different
> > type.
> >
> > For example:
> >
> > -hybrid socket,sockets=1
> > -hybrid die,dies=1
> > -hybrid cluster,clusters=4
> > -hybrid core,cores=1,coretype="core",threads=2,clusterid=0-2
> > -hybrid core,cores=4,coretype="atom",threads=1
> >
> > Here we can build a hybrid cpu topology, which has 1 socket, 1 die per
> > socket, 4 clusters per die. And in each die, every clusters has 4 "atom"
> > core with 1 threads, and cluster0, cluster1 and cluster2 have 1 "core"
> > cores with 2 threads.
> >
> > Please note this example is not an actual machine topology, but it shows
> > the powerful flexibility of "hybrid" command.
> 
> Is there any intention to apply this command to emulation or will it
> only be for hardware virtualisation? For example the x86 translator
> should be capable of emulating two different classes of x86 with
> different subsets of instructions.

We have not considered the hybrid case in emulation mode, and
the current hybrid CPUs of intel are same-ISA.

> It requires ensuring each CPU is
> assigned to the correct TYPE_CPU_CLUSTER so we don't re-use TBs from
> the "wrong" CPU.

From my understanding, this also seems like the topology of heterogeneous
core and heterogeneous cluster.

Zhao

> 
> <snip>
> 
> -- 
> Alex Bennée
> Virtualisation Tech Lead @ Linaro