From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alexander Graf Date: Tue, 30 Jun 2015 09:35:33 +0000 Subject: Re: [PATCH 2/5] KVM: PPC: Book3S HV: Implement dynamic micro-threading on POWER8 Message-Id: <559262E5.50108@suse.de> List-Id: References: <1435144687-8071-1-git-send-email-paulus@samba.org> <1435144687-8071-3-git-send-email-paulus@samba.org> In-Reply-To: <1435144687-8071-3-git-send-email-paulus@samba.org> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Paul Mackerras , kvm@vger.kernel.org, kvm-ppc@vger.kernel.org On 06/24/15 13:18, Paul Mackerras wrote: > This builds on the ability to run more than one vcore on a physical > core by using the micro-threading (split-core) modes of the POWER8 > chip. Previously, only vcores from the same VM could be run together, > and (on POWER8) only if they had just one thread per core. With the > ability to split the core on guest entry and unsplit it on guest exit, > we can run up to 8 vcpu threads from up to 4 different VMs, and we can > run multiple vcores with 2 or 4 vcpus per vcore. > > Dynamic micro-threading is only available if the static configuration > of the cores is whole-core mode (unsplit), and only on POWER8. > > To manage this, we introduce a new kvm_split_mode struct which is > shared across all of the subcores in the core, with a pointer in the > paca on each thread. In addition we extend the core_info struct to > have information on each subcore. When deciding whether to add a > vcore to the set already on the core, we now have two possibilities: > (a) piggyback the vcore onto an existing subcore, or (b) start a new > subcore. > > Currently, when any vcpu needs to exit the guest and switch to host > virtual mode, we interrupt all the threads in all subcores and switch > the core back to whole-core mode. It may be possible in future to > allow some of the subcores to keep executing in the guest while > subcore 0 switches to the host, but that is not implemented in this > patch. > > This adds a module parameter called dynamic_mt_modes which controls > which micro-threading (split-core) modes the code will consider, as a > bitmap. In other words, if it is 0, no micro-threading mode is > considered; if it is 2, only 2-way micro-threading is considered; if > it is 4, only 4-way, and if it is 6, both 2-way and 4-way > micro-threading mode will be considered. The default is 6. > > With this, we now have secondary threads which are the primary thread > for their subcore and therefore need to do the MMU switch. These > threads will need to be started even if they have no vcpu to run, so > we use the vcore pointer in the PACA rather than the vcpu pointer to > trigger them. > > It is now possible for thread 0 to find that an exit has been > requested before it gets to switch the subcore state to the guest. In > that case we haven't added the guest's timebase offset to the > timebase, so we need to be careful not to subtract the offset in the > guest exit path. In fact we just skip the whole path that switches > back to host context, since we haven't switched to the guest context. > > Signed-off-by: Paul Mackerras > --- > arch/powerpc/include/asm/kvm_book3s_asm.h | 20 ++ > arch/powerpc/include/asm/kvm_host.h | 3 + > arch/powerpc/kernel/asm-offsets.c | 7 + > arch/powerpc/kvm/book3s_hv.c | 369 ++++++++++++++++++++++++++---- > arch/powerpc/kvm/book3s_hv_builtin.c | 25 +- > arch/powerpc/kvm/book3s_hv_rmhandlers.S | 113 +++++++-- > 6 files changed, 475 insertions(+), 62 deletions(-) > > diff --git a/arch/powerpc/include/asm/kvm_book3s_asm.h b/arch/powerpc/include/asm/kvm_book3s_asm.h > index 5bdfb5d..4024d24 100644 > --- a/arch/powerpc/include/asm/kvm_book3s_asm.h > +++ b/arch/powerpc/include/asm/kvm_book3s_asm.h > @@ -25,6 +25,12 @@ > #define XICS_MFRR 0xc > #define XICS_IPI 2 /* interrupt source # for IPIs */ > > +/* Maximum number of threads per physical core */ > +#define MAX_THREADS 8 > + > +/* Maximum number of subcores per physical core */ > +#define MAX_SUBCORES 4 > + > #ifdef __ASSEMBLY__ > > #ifdef CONFIG_KVM_BOOK3S_HANDLER > @@ -65,6 +71,19 @@ kvmppc_resume_\intno: > > #else /*__ASSEMBLY__ */ > > +struct kvmppc_vcore; > + > +/* Struct used for coordinating micro-threading (split-core) mode changes */ > +struct kvm_split_mode { > + unsigned long rpr; > + unsigned long pmmar; > + unsigned long ldbar; > + u8 subcore_size; > + u8 do_nap; > + u8 napped[MAX_THREADS]; > + struct kvmppc_vcore *master_vcs[MAX_SUBCORES]; > +}; > + > /* > * This struct goes in the PACA on 64-bit processors. It is used > * to store host state that needs to be saved when we enter a guest > @@ -100,6 +119,7 @@ struct kvmppc_host_state { > u64 host_spurr; > u64 host_dscr; > u64 dec_expires; > + struct kvm_split_mode *kvm_split_mode; > #endif > #ifdef CONFIG_PPC_BOOK3S_64 > u64 cfar; > diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h > index 2b74490..80eb29a 100644 > --- a/arch/powerpc/include/asm/kvm_host.h > +++ b/arch/powerpc/include/asm/kvm_host.h > @@ -302,6 +302,9 @@ struct kvmppc_vcore { > #define VCORE_EXIT_MAP(vc) ((vc)->entry_exit_map >> 8) > #define VCORE_IS_EXITING(vc) (VCORE_EXIT_MAP(vc) != 0) > > +/* This bit is used when a vcore exit is triggered from outside the vcore */ > +#define VCORE_EXIT_REQ 0x10000 > + > /* > * Values for vcore_state. > * Note that these are arranged such that lower values > diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c > index d333664..c3e11e0 100644 > --- a/arch/powerpc/kernel/asm-offsets.c > +++ b/arch/powerpc/kernel/asm-offsets.c > @@ -676,7 +676,14 @@ int main(void) > HSTATE_FIELD(HSTATE_DSCR, host_dscr); > HSTATE_FIELD(HSTATE_DABR, dabr); > HSTATE_FIELD(HSTATE_DECEXP, dec_expires); > + HSTATE_FIELD(HSTATE_SPLIT_MODE, kvm_split_mode); > DEFINE(IPI_PRIORITY, IPI_PRIORITY); > + DEFINE(KVM_SPLIT_RPR, offsetof(struct kvm_split_mode, rpr)); > + DEFINE(KVM_SPLIT_PMMAR, offsetof(struct kvm_split_mode, pmmar)); > + DEFINE(KVM_SPLIT_LDBAR, offsetof(struct kvm_split_mode, ldbar)); > + DEFINE(KVM_SPLIT_SIZE, offsetof(struct kvm_split_mode, subcore_size)); > + DEFINE(KVM_SPLIT_DO_NAP, offsetof(struct kvm_split_mode, do_nap)); > + DEFINE(KVM_SPLIT_NAPPED, offsetof(struct kvm_split_mode, napped)); > #endif /* CONFIG_KVM_BOOK3S_HV_POSSIBLE */ > > #ifdef CONFIG_PPC_BOOK3S_64 > diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c > index 2048309..32377de 100644 > --- a/arch/powerpc/kvm/book3s_hv.c > +++ b/arch/powerpc/kvm/book3s_hv.c > @@ -81,6 +81,9 @@ static DECLARE_BITMAP(default_enabled_hcalls, MAX_HCALL_OPCODE/4 + 1); > #define MPP_BUFFER_ORDER 3 > #endif > > +static int dynamic_mt_modes = 6; > +module_param(dynamic_mt_modes, int, S_IRUGO | S_IWUSR); > +MODULE_PARM_DESC(dynamic_mt_modes, "Set of allowed dynamic micro-threading modes"); Please be more verbose here. Just list the possible choices like you do in the patch description. Alex From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alexander Graf Subject: Re: [PATCH 2/5] KVM: PPC: Book3S HV: Implement dynamic micro-threading on POWER8 Date: Tue, 30 Jun 2015 11:35:33 +0200 Message-ID: <559262E5.50108@suse.de> References: <1435144687-8071-1-git-send-email-paulus@samba.org> <1435144687-8071-3-git-send-email-paulus@samba.org> Mime-Version: 1.0 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit To: Paul Mackerras , kvm@vger.kernel.org, kvm-ppc@vger.kernel.org Return-path: Received: from cantor2.suse.de ([195.135.220.15]:38802 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752883AbbF3Jff (ORCPT ); Tue, 30 Jun 2015 05:35:35 -0400 In-Reply-To: <1435144687-8071-3-git-send-email-paulus@samba.org> Sender: kvm-owner@vger.kernel.org List-ID: On 06/24/15 13:18, Paul Mackerras wrote: > This builds on the ability to run more than one vcore on a physical > core by using the micro-threading (split-core) modes of the POWER8 > chip. Previously, only vcores from the same VM could be run together, > and (on POWER8) only if they had just one thread per core. With the > ability to split the core on guest entry and unsplit it on guest exit, > we can run up to 8 vcpu threads from up to 4 different VMs, and we can > run multiple vcores with 2 or 4 vcpus per vcore. > > Dynamic micro-threading is only available if the static configuration > of the cores is whole-core mode (unsplit), and only on POWER8. > > To manage this, we introduce a new kvm_split_mode struct which is > shared across all of the subcores in the core, with a pointer in the > paca on each thread. In addition we extend the core_info struct to > have information on each subcore. When deciding whether to add a > vcore to the set already on the core, we now have two possibilities: > (a) piggyback the vcore onto an existing subcore, or (b) start a new > subcore. > > Currently, when any vcpu needs to exit the guest and switch to host > virtual mode, we interrupt all the threads in all subcores and switch > the core back to whole-core mode. It may be possible in future to > allow some of the subcores to keep executing in the guest while > subcore 0 switches to the host, but that is not implemented in this > patch. > > This adds a module parameter called dynamic_mt_modes which controls > which micro-threading (split-core) modes the code will consider, as a > bitmap. In other words, if it is 0, no micro-threading mode is > considered; if it is 2, only 2-way micro-threading is considered; if > it is 4, only 4-way, and if it is 6, both 2-way and 4-way > micro-threading mode will be considered. The default is 6. > > With this, we now have secondary threads which are the primary thread > for their subcore and therefore need to do the MMU switch. These > threads will need to be started even if they have no vcpu to run, so > we use the vcore pointer in the PACA rather than the vcpu pointer to > trigger them. > > It is now possible for thread 0 to find that an exit has been > requested before it gets to switch the subcore state to the guest. In > that case we haven't added the guest's timebase offset to the > timebase, so we need to be careful not to subtract the offset in the > guest exit path. In fact we just skip the whole path that switches > back to host context, since we haven't switched to the guest context. > > Signed-off-by: Paul Mackerras > --- > arch/powerpc/include/asm/kvm_book3s_asm.h | 20 ++ > arch/powerpc/include/asm/kvm_host.h | 3 + > arch/powerpc/kernel/asm-offsets.c | 7 + > arch/powerpc/kvm/book3s_hv.c | 369 ++++++++++++++++++++++++++---- > arch/powerpc/kvm/book3s_hv_builtin.c | 25 +- > arch/powerpc/kvm/book3s_hv_rmhandlers.S | 113 +++++++-- > 6 files changed, 475 insertions(+), 62 deletions(-) > > diff --git a/arch/powerpc/include/asm/kvm_book3s_asm.h b/arch/powerpc/include/asm/kvm_book3s_asm.h > index 5bdfb5d..4024d24 100644 > --- a/arch/powerpc/include/asm/kvm_book3s_asm.h > +++ b/arch/powerpc/include/asm/kvm_book3s_asm.h > @@ -25,6 +25,12 @@ > #define XICS_MFRR 0xc > #define XICS_IPI 2 /* interrupt source # for IPIs */ > > +/* Maximum number of threads per physical core */ > +#define MAX_THREADS 8 > + > +/* Maximum number of subcores per physical core */ > +#define MAX_SUBCORES 4 > + > #ifdef __ASSEMBLY__ > > #ifdef CONFIG_KVM_BOOK3S_HANDLER > @@ -65,6 +71,19 @@ kvmppc_resume_\intno: > > #else /*__ASSEMBLY__ */ > > +struct kvmppc_vcore; > + > +/* Struct used for coordinating micro-threading (split-core) mode changes */ > +struct kvm_split_mode { > + unsigned long rpr; > + unsigned long pmmar; > + unsigned long ldbar; > + u8 subcore_size; > + u8 do_nap; > + u8 napped[MAX_THREADS]; > + struct kvmppc_vcore *master_vcs[MAX_SUBCORES]; > +}; > + > /* > * This struct goes in the PACA on 64-bit processors. It is used > * to store host state that needs to be saved when we enter a guest > @@ -100,6 +119,7 @@ struct kvmppc_host_state { > u64 host_spurr; > u64 host_dscr; > u64 dec_expires; > + struct kvm_split_mode *kvm_split_mode; > #endif > #ifdef CONFIG_PPC_BOOK3S_64 > u64 cfar; > diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h > index 2b74490..80eb29a 100644 > --- a/arch/powerpc/include/asm/kvm_host.h > +++ b/arch/powerpc/include/asm/kvm_host.h > @@ -302,6 +302,9 @@ struct kvmppc_vcore { > #define VCORE_EXIT_MAP(vc) ((vc)->entry_exit_map >> 8) > #define VCORE_IS_EXITING(vc) (VCORE_EXIT_MAP(vc) != 0) > > +/* This bit is used when a vcore exit is triggered from outside the vcore */ > +#define VCORE_EXIT_REQ 0x10000 > + > /* > * Values for vcore_state. > * Note that these are arranged such that lower values > diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c > index d333664..c3e11e0 100644 > --- a/arch/powerpc/kernel/asm-offsets.c > +++ b/arch/powerpc/kernel/asm-offsets.c > @@ -676,7 +676,14 @@ int main(void) > HSTATE_FIELD(HSTATE_DSCR, host_dscr); > HSTATE_FIELD(HSTATE_DABR, dabr); > HSTATE_FIELD(HSTATE_DECEXP, dec_expires); > + HSTATE_FIELD(HSTATE_SPLIT_MODE, kvm_split_mode); > DEFINE(IPI_PRIORITY, IPI_PRIORITY); > + DEFINE(KVM_SPLIT_RPR, offsetof(struct kvm_split_mode, rpr)); > + DEFINE(KVM_SPLIT_PMMAR, offsetof(struct kvm_split_mode, pmmar)); > + DEFINE(KVM_SPLIT_LDBAR, offsetof(struct kvm_split_mode, ldbar)); > + DEFINE(KVM_SPLIT_SIZE, offsetof(struct kvm_split_mode, subcore_size)); > + DEFINE(KVM_SPLIT_DO_NAP, offsetof(struct kvm_split_mode, do_nap)); > + DEFINE(KVM_SPLIT_NAPPED, offsetof(struct kvm_split_mode, napped)); > #endif /* CONFIG_KVM_BOOK3S_HV_POSSIBLE */ > > #ifdef CONFIG_PPC_BOOK3S_64 > diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c > index 2048309..32377de 100644 > --- a/arch/powerpc/kvm/book3s_hv.c > +++ b/arch/powerpc/kvm/book3s_hv.c > @@ -81,6 +81,9 @@ static DECLARE_BITMAP(default_enabled_hcalls, MAX_HCALL_OPCODE/4 + 1); > #define MPP_BUFFER_ORDER 3 > #endif > > +static int dynamic_mt_modes = 6; > +module_param(dynamic_mt_modes, int, S_IRUGO | S_IWUSR); > +MODULE_PARM_DESC(dynamic_mt_modes, "Set of allowed dynamic micro-threading modes"); Please be more verbose here. Just list the possible choices like you do in the patch description. Alex