Re: [PATCH 2/5] KVM: PPC: Book3S HV: Implement dynamic micro-threading on POWER8

[Alexander Graf <agraf@suse.de>]

On 06/24/15 13:18, Paul Mackerras wrote:
> This builds on the ability to run more than one vcore on a physical
> core by using the micro-threading (split-core) modes of the POWER8
> chip.  Previously, only vcores from the same VM could be run together,
> and (on POWER8) only if they had just one thread per core.  With the
> ability to split the core on guest entry and unsplit it on guest exit,
> we can run up to 8 vcpu threads from up to 4 different VMs, and we can
> run multiple vcores with 2 or 4 vcpus per vcore.
>
> Dynamic micro-threading is only available if the static configuration
> of the cores is whole-core mode (unsplit), and only on POWER8.
>
> To manage this, we introduce a new kvm_split_mode struct which is
> shared across all of the subcores in the core, with a pointer in the
> paca on each thread.  In addition we extend the core_info struct to
> have information on each subcore.  When deciding whether to add a
> vcore to the set already on the core, we now have two possibilities:
> (a) piggyback the vcore onto an existing subcore, or (b) start a new
> subcore.
>
> Currently, when any vcpu needs to exit the guest and switch to host
> virtual mode, we interrupt all the threads in all subcores and switch
> the core back to whole-core mode.  It may be possible in future to
> allow some of the subcores to keep executing in the guest while
> subcore 0 switches to the host, but that is not implemented in this
> patch.
>
> This adds a module parameter called dynamic_mt_modes which controls
> which micro-threading (split-core) modes the code will consider, as a
> bitmap.  In other words, if it is 0, no micro-threading mode is
> considered; if it is 2, only 2-way micro-threading is considered; if
> it is 4, only 4-way, and if it is 6, both 2-way and 4-way
> micro-threading mode will be considered.  The default is 6.
>
> With this, we now have secondary threads which are the primary thread
> for their subcore and therefore need to do the MMU switch.  These
> threads will need to be started even if they have no vcpu to run, so
> we use the vcore pointer in the PACA rather than the vcpu pointer to
> trigger them.
>
> It is now possible for thread 0 to find that an exit has been
> requested before it gets to switch the subcore state to the guest.  In
> that case we haven't added the guest's timebase offset to the
> timebase, so we need to be careful not to subtract the offset in the
> guest exit path.  In fact we just skip the whole path that switches
> back to host context, since we haven't switched to the guest context.
>
> Signed-off-by: Paul Mackerras <paulus@samba.org>
> ---
>   arch/powerpc/include/asm/kvm_book3s_asm.h |  20 ++
>   arch/powerpc/include/asm/kvm_host.h       |   3 +
>   arch/powerpc/kernel/asm-offsets.c         |   7 +
>   arch/powerpc/kvm/book3s_hv.c              | 369 ++++++++++++++++++++++++++----
>   arch/powerpc/kvm/book3s_hv_builtin.c      |  25 +-
>   arch/powerpc/kvm/book3s_hv_rmhandlers.S   | 113 +++++++--
>   6 files changed, 475 insertions(+), 62 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/kvm_book3s_asm.h b/arch/powerpc/include/asm/kvm_book3s_asm.h
> index 5bdfb5d..4024d24 100644
> --- a/arch/powerpc/include/asm/kvm_book3s_asm.h
> +++ b/arch/powerpc/include/asm/kvm_book3s_asm.h
> @@ -25,6 +25,12 @@
>   #define XICS_MFRR		0xc
>   #define XICS_IPI		2	/* interrupt source # for IPIs */
>   
> +/* Maximum number of threads per physical core */
> +#define MAX_THREADS		8
> +
> +/* Maximum number of subcores per physical core */
> +#define MAX_SUBCORES		4
> +
>   #ifdef __ASSEMBLY__
>   
>   #ifdef CONFIG_KVM_BOOK3S_HANDLER
> @@ -65,6 +71,19 @@ kvmppc_resume_\intno:
>   
>   #else  /*__ASSEMBLY__ */
>   
> +struct kvmppc_vcore;
> +
> +/* Struct used for coordinating micro-threading (split-core) mode changes */
> +struct kvm_split_mode {
> +	unsigned long	rpr;
> +	unsigned long	pmmar;
> +	unsigned long	ldbar;
> +	u8		subcore_size;
> +	u8		do_nap;
> +	u8		napped[MAX_THREADS];
> +	struct kvmppc_vcore *master_vcs[MAX_SUBCORES];
> +};
> +
>   /*
>    * This struct goes in the PACA on 64-bit processors.  It is used
>    * to store host state that needs to be saved when we enter a guest
> @@ -100,6 +119,7 @@ struct kvmppc_host_state {
>   	u64 host_spurr;
>   	u64 host_dscr;
>   	u64 dec_expires;
> +	struct kvm_split_mode *kvm_split_mode;
>   #endif
>   #ifdef CONFIG_PPC_BOOK3S_64
>   	u64 cfar;
> diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
> index 2b74490..80eb29a 100644
> --- a/arch/powerpc/include/asm/kvm_host.h
> +++ b/arch/powerpc/include/asm/kvm_host.h
> @@ -302,6 +302,9 @@ struct kvmppc_vcore {
>   #define VCORE_EXIT_MAP(vc)	((vc)->entry_exit_map >> 8)
>   #define VCORE_IS_EXITING(vc)	(VCORE_EXIT_MAP(vc) != 0)
>   
> +/* This bit is used when a vcore exit is triggered from outside the vcore */
> +#define VCORE_EXIT_REQ		0x10000
> +
>   /*
>    * Values for vcore_state.
>    * Note that these are arranged such that lower values
> diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c
> index d333664..c3e11e0 100644
> --- a/arch/powerpc/kernel/asm-offsets.c
> +++ b/arch/powerpc/kernel/asm-offsets.c
> @@ -676,7 +676,14 @@ int main(void)
>   	HSTATE_FIELD(HSTATE_DSCR, host_dscr);
>   	HSTATE_FIELD(HSTATE_DABR, dabr);
>   	HSTATE_FIELD(HSTATE_DECEXP, dec_expires);
> +	HSTATE_FIELD(HSTATE_SPLIT_MODE, kvm_split_mode);
>   	DEFINE(IPI_PRIORITY, IPI_PRIORITY);
> +	DEFINE(KVM_SPLIT_RPR, offsetof(struct kvm_split_mode, rpr));
> +	DEFINE(KVM_SPLIT_PMMAR, offsetof(struct kvm_split_mode, pmmar));
> +	DEFINE(KVM_SPLIT_LDBAR, offsetof(struct kvm_split_mode, ldbar));
> +	DEFINE(KVM_SPLIT_SIZE, offsetof(struct kvm_split_mode, subcore_size));
> +	DEFINE(KVM_SPLIT_DO_NAP, offsetof(struct kvm_split_mode, do_nap));
> +	DEFINE(KVM_SPLIT_NAPPED, offsetof(struct kvm_split_mode, napped));
>   #endif /* CONFIG_KVM_BOOK3S_HV_POSSIBLE */
>   
>   #ifdef CONFIG_PPC_BOOK3S_64
> diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
> index 2048309..32377de 100644
> --- a/arch/powerpc/kvm/book3s_hv.c
> +++ b/arch/powerpc/kvm/book3s_hv.c
> @@ -81,6 +81,9 @@ static DECLARE_BITMAP(default_enabled_hcalls, MAX_HCALL_OPCODE/4 + 1);
>   #define MPP_BUFFER_ORDER	3
>   #endif
>   
> +static int dynamic_mt_modes = 6;
> +module_param(dynamic_mt_modes, int, S_IRUGO | S_IWUSR);
> +MODULE_PARM_DESC(dynamic_mt_modes, "Set of allowed dynamic micro-threading modes");

Please be more verbose here. Just list the possible choices like you do 
in the patch description.


Alex