[PATCH v3] xen/arm: introduce vwfi parameter

[PATCH v3] xen/arm: introduce vwfi parameter

[Stefano Stabellini]

Introduce new Xen command line parameter called "vwfi", which stands for
virtual wfi. The default is "trap": Xen traps guest wfi and wfe
instructions. In the case of wfi, Xen calls vcpu_block on the guest
vcpu; in the case of guest wfe, Xen calls vcpu_yield on the guest vcpu.
The behavior can be changed by setting vwfi to "native", in that case
Xen doesn't trap neither wfi nor wfe, running them in guest context.

The result is strong reduction in irq latency (from 5000ns to 2000ns,
measured using https://github.com/edgarigl/tbm, the physical timer, and
1 pcpu dedicated to 1 vcpu). The downside is that the scheduler thinks
that the guest is busy when actually is sleeping, leading to suboptimal
scheduling decisions.

Signed-off-by: Stefano Stabellini <sstabellini@kernel.org>
CC: dario.faggioli@citrix.com

---
Changes in v3:
- do the same for wfe
---
 docs/misc/xen-command-line.markdown | 14 ++++++++++++++
 xen/arch/arm/traps.c                | 17 +++++++++++++++--
 2 files changed, 29 insertions(+), 2 deletions(-)

diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown
index c0106fb..de4642b 100644
--- a/docs/misc/xen-command-line.markdown
+++ b/docs/misc/xen-command-line.markdown
@@ -1638,6 +1638,20 @@ Note that if **watchdog** option is also specified vpmu will be turned off.
 As the virtualisation is not 100% safe, don't use the vpmu flag on
 production systems (see http://xenbits.xen.org/xsa/advisory-163.html)!
 
+### vwfi
+> `= trap | native
+
+> Default: `trap`
+
+WFI is the ARM instruction to "wait for interrupt". WFE is similar and
+means "wait for event". This option, which is ARM specific, changes the
+way guest WFI and WFE are implemented in Xen. By default, Xen traps both
+instructions. In the case of WFI, Xen blocks the guest vcpu; in the case
+of WFE, Xen yield the guest vcpu. When setting vwfi to `native`, Xen
+doesn't trap either instruction, running them in guest context. Setting
+vwfi to `native` reduces irq latency, but leads to suboptimal scheduling
+decisions.
+
 ### watchdog
 > `= force | <boolean>`
 
diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 4f96a2b..8117bd5 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -102,6 +102,19 @@ static int debug_stack_lines = 40;
 
 integer_param("debug_stack_lines", debug_stack_lines);
 
+static enum {
+	TRAP,
+	NATIVE,
+} vwfi;
+
+static void __init parse_vwfi(const char *s)
+{
+	if ( !strcmp(s, "native") )
+		vwfi = NATIVE;
+	else
+		vwfi = TRAP;
+}
+custom_param("vwfi", parse_vwfi);
 
 void init_traps(void)
 {
@@ -128,8 +141,8 @@ void init_traps(void)
 
     /* Setup hypervisor traps */
     WRITE_SYSREG(HCR_PTW|HCR_BSU_INNER|HCR_AMO|HCR_IMO|HCR_FMO|HCR_VM|
-                 HCR_TWE|HCR_TWI|HCR_TSC|HCR_TAC|HCR_SWIO|HCR_TIDCP|HCR_FB,
-                 HCR_EL2);
+                 (vwfi != NATIVE ? (HCR_TWI|HCR_TWE) : 0) |
+                 HCR_TSC|HCR_TAC|HCR_SWIO|HCR_TIDCP|HCR_FB,HCR_EL2);
     isb();
 }
 
-- 
1.9.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v3] xen/arm: introduce vwfi parameter

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1834 bytes --]

On Tue, 2017-02-28 at 13:12 -0800, Stefano Stabellini wrote:
> --- a/docs/misc/xen-command-line.markdown
> +++ b/docs/misc/xen-command-line.markdown
> @@ -1638,6 +1638,20 @@ Note that if **watchdog** option is also
> specified vpmu will be turned off.
>  As the virtualisation is not 100% safe, don't use the vpmu flag on
>  production systems (see http://xenbits.xen.org/xsa/advisory-163.html
> )!
>  
> +### vwfi
> +> `= trap | native
> +
> +> Default: `trap`
> +
> +WFI is the ARM instruction to "wait for interrupt". WFE is similar
> and
> +means "wait for event". This option, which is ARM specific, changes
> the
> +way guest WFI and WFE are implemented in Xen. By default, Xen traps
> both
> +instructions. In the case of WFI, Xen blocks the guest vcpu; in the
> case
> +of WFE, Xen yield the guest vcpu. When setting vwfi to `native`, Xen
> +doesn't trap either instruction, running them in guest context.
> Setting
> +vwfi to `native` reduces irq latency, but leads to suboptimal
> scheduling
> +decisions.
> +
These last few words sounds a bit generic to me. They may alarm people
and discourage using 'native', even when it is actually ok.

Being a bit more specific, although without going into too much
technical detail, seems worth to me.

So, how about something like:

"Using `native` reduces irq latency significantly. It can also lead to
suboptimal scheduling decisions and affect performance, but only when
the system is oversubscribed (i.e., there are, in total, more vCPUs
than pCPUs)."

?

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v3] xen/arm: introduce vwfi parameter

[Stefano Stabellini]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1672 bytes --]

On Wed, 1 Mar 2017, Dario Faggioli wrote:
> On Tue, 2017-02-28 at 13:12 -0800, Stefano Stabellini wrote:
> > --- a/docs/misc/xen-command-line.markdown
> > +++ b/docs/misc/xen-command-line.markdown
> > @@ -1638,6 +1638,20 @@ Note that if **watchdog** option is also
> > specified vpmu will be turned off.
> >  As the virtualisation is not 100% safe, don't use the vpmu flag on
> >  production systems (see http://xenbits.xen.org/xsa/advisory-163.html
> > )!
> >  
> > +### vwfi
> > +> `= trap | native
> > +
> > +> Default: `trap`
> > +
> > +WFI is the ARM instruction to "wait for interrupt". WFE is similar
> > and
> > +means "wait for event". This option, which is ARM specific, changes
> > the
> > +way guest WFI and WFE are implemented in Xen. By default, Xen traps
> > both
> > +instructions. In the case of WFI, Xen blocks the guest vcpu; in the
> > case
> > +of WFE, Xen yield the guest vcpu. When setting vwfi to `native`, Xen
> > +doesn't trap either instruction, running them in guest context.
> > Setting
> > +vwfi to `native` reduces irq latency, but leads to suboptimal
> > scheduling
> > +decisions.
> > +
> These last few words sounds a bit generic to me. They may alarm people
> and discourage using 'native', even when it is actually ok.
> 
> Being a bit more specific, although without going into too much
> technical detail, seems worth to me.
> 
> So, how about something like:
> 
> "Using `native` reduces irq latency significantly. It can also lead to
> suboptimal scheduling decisions and affect performance, but only when
> the system is oversubscribed (i.e., there are, in total, more vCPUs
> than pCPUs)."

That's a good suggestion, thank you.

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel