From mboxrd@z Thu Jan 1 00:00:00 1970 From: Oleksandr Tyshchenko Subject: Re: [PATCH] xen: arm: increase priority of SGIs used as IPIs Date: Tue, 28 Jan 2014 20:20:00 +0200 Message-ID: References: <1390927878-7048-1-git-send-email-ian.campbell@citrix.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============3196472893408834412==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Stefano Stabellini Cc: george.dunlap@citrix.com, julien.grall@linaro.org, tim@xen.org, Ian Campbell , xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org --===============3196472893408834412== Content-Type: multipart/alternative; boundary=047d7bd6bd765833a904f10bdf64 --047d7bd6bd765833a904f10bdf64 Content-Type: text/plain; charset=ISO-8859-1 Hi, all. Sorry for late response. A lot of thanks to all for your advices and full solutions. I will check the current patch and earlier solution (with tasklet_schedule) too. On Tue, Jan 28, 2014 at 7:50 PM, Stefano Stabellini < stefano.stabellini@eu.citrix.com> wrote: > On Tue, 28 Jan 2014, Ian Campbell wrote: > > Code such as on_selected_cpus expects/requires that an IPI can preempt a > > processor which is just handling a normal interrupt. Lacking this > property can > > result in a deadlock between two CPUs trying to IPI each other from > interrupt > > context. > > > > For the time being there is only two priorities, IRQ and IPI, although > it is > > also conceivable that in the future some IPIs might be higher priority > than > > others. This could be used to implement a better BUG() than we have now, > but I > > haven't tackled that yet. > > > > Tested with a debug patch which sends a local IPI from a keyhandler, > which is > > run in serial interrupt context. > > > > This should also fix the issue reported by Oleksandr in "xen/arm: > > maintenance_interrupt SMP fix" without resorting to trylock. > > > > Signed-off-by: Ian Campbell > > Cc: Oleksandr Tyshchenko > > It looks simple enough. > Oleksandr, I would appreciate if could test the patch and tell us if it > is working well for you. > > > > I think this is probably 4.5 material at this point. > > > > Tested with "HACK: dump pcpu state keyhandler" which I'll post for > > completeness. It gives: > > (XEN) Xen call trace: > > (XEN) [<0000000000212048>] dump_pcpus+0x28/0x2c (PC) > > (XEN) [<000000000021256c>] handle_keypress+0x70/0xb0 (LR) > > (XEN) [<000000000023ed00>] __serial_rx+0x20/0x6c > > (XEN) [<000000000023f8ac>] serial_rx+0xb4/0xc4 > > (XEN) [<00000000002409ec>] serial_rx_interrupt+0xb0/0xd4 > > (XEN) [<00000000002404b4>] ns16550_interrupt+0x6c/0x90 > > (XEN) [<0000000000245fc0>] do_IRQ+0x144/0x1b4 > > (XEN) [<0000000000245a28>] gic_interrupt+0x60/0xf8 > > (XEN) [<000000000024be64>] do_trap_irq+0x10/0x18 > > (XEN) [<000000000024e240>] hyp_irq+0x5c/0x60 > > (XEN) [<0000000000249324>] init_done+0x10/0x18 > > (XEN) [<0000000000000080>] 0000000000000080 > > --- > > xen/arch/arm/gic.c | 19 +++++++++++++------ > > xen/arch/arm/time.c | 6 +++--- > > xen/include/asm-arm/gic.h | 22 ++++++++++++++++++++++ > > 3 files changed, 38 insertions(+), 9 deletions(-) > > > > diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c > > index dcf9cd4..ee37019 100644 > > --- a/xen/arch/arm/gic.c > > +++ b/xen/arch/arm/gic.c > > @@ -319,7 +319,8 @@ static void __init gic_dist_init(void) > > > > /* Default priority for global interrupts */ > > for ( i = 32; i < gic.lines; i += 4 ) > > - GICD[GICD_IPRIORITYR + i / 4] = 0xa0a0a0a0; > > + GICD[GICD_IPRIORITYR + i / 4] = > > + GIC_PRI_IRQ<<24 | GIC_PRI_IRQ<<16 | GIC_PRI_IRQ<<8 | > GIC_PRI_IRQ; > > > > /* Disable all global interrupts */ > > for ( i = 32; i < gic.lines; i += 32 ) > > @@ -341,8 +342,12 @@ static void __cpuinit gic_cpu_init(void) > > GICD[GICD_ICENABLER] = 0xffff0000; /* Disable all PPI */ > > GICD[GICD_ISENABLER] = 0x0000ffff; /* Enable all SGI */ > > /* Set PPI and SGI priorities */ > > - for (i = 0; i < 32; i += 4) > > - GICD[GICD_IPRIORITYR + i / 4] = 0xa0a0a0a0; > > + for (i = 0; i < 16; i += 4) > > + GICD[GICD_IPRIORITYR + i / 4] = > > + GIC_PRI_IPI<<24 | GIC_PRI_IPI<<16 | GIC_PRI_IPI<<8 | > GIC_PRI_IPI; > > + for (i = 16; i < 32; i += 4) > > + GICD[GICD_IPRIORITYR + i / 4] = > > + GIC_PRI_IRQ<<24 | GIC_PRI_IRQ<<16 | GIC_PRI_IRQ<<8 | > GIC_PRI_IRQ; > > > > /* Local settings: interface controller */ > > GICC[GICC_PMR] = 0xff; /* Don't mask by priority */ > > @@ -538,7 +543,8 @@ void gic_disable_cpu(void) > > void gic_route_ppis(void) > > { > > /* GIC maintenance */ > > - gic_route_dt_irq(&gic.maintenance, cpumask_of(smp_processor_id()), > 0xa0); > > + gic_route_dt_irq(&gic.maintenance, cpumask_of(smp_processor_id()), > > + GIC_PRI_IRQ); > > /* Route timer interrupt */ > > route_timer_interrupt(); > > } > > @@ -553,7 +559,8 @@ void gic_route_spis(void) > > if ( (irq = serial_dt_irq(seridx)) == NULL ) > > continue; > > > > - gic_route_dt_irq(irq, cpumask_of(smp_processor_id()), 0xa0); > > + gic_route_dt_irq(irq, cpumask_of(smp_processor_id()), > > + GIC_PRI_IRQ); > > } > > } > > > > @@ -777,7 +784,7 @@ int gic_route_irq_to_guest(struct domain *d, const > struct dt_irq *irq, > > level = dt_irq_is_level_triggered(irq); > > > > gic_set_irq_properties(irq->irq, level, > cpumask_of(smp_processor_id()), > > - 0xa0); > > + GIC_PRI_IRQ); > > > > retval = __setup_irq(desc, irq->irq, action); > > if (retval) { > > diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c > > index 81e3e28..68b939d 100644 > > --- a/xen/arch/arm/time.c > > +++ b/xen/arch/arm/time.c > > @@ -222,11 +222,11 @@ static void vtimer_interrupt(int irq, void > *dev_id, struct cpu_user_regs *regs) > > void __cpuinit route_timer_interrupt(void) > > { > > gic_route_dt_irq(&timer_irq[TIMER_PHYS_NONSECURE_PPI], > > - cpumask_of(smp_processor_id()), 0xa0); > > + cpumask_of(smp_processor_id()), GIC_PRI_IRQ); > > gic_route_dt_irq(&timer_irq[TIMER_HYP_PPI], > > - cpumask_of(smp_processor_id()), 0xa0); > > + cpumask_of(smp_processor_id()), GIC_PRI_IRQ); > > gic_route_dt_irq(&timer_irq[TIMER_VIRT_PPI], > > - cpumask_of(smp_processor_id()), 0xa0); > > + cpumask_of(smp_processor_id()), GIC_PRI_IRQ); > > } > > > > /* Set up the timer interrupt on this CPU */ > > diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h > > index 9c6f9bb..25b2b24 100644 > > --- a/xen/include/asm-arm/gic.h > > +++ b/xen/include/asm-arm/gic.h > > @@ -129,6 +129,28 @@ > > #define GICH_LR_CPUID_SHIFT 9 > > #define GICH_VTR_NRLRGS 0x3f > > > > +/* > > + * The minimum GICC_BPR is required to be in the range 0-3. We set > > + * GICC_BPR to 0 but we must expect that it might be 3. This means we > > + * can rely on premption between the following ranges: > > + * 0xf0..0xff > > + * 0xe0..0xdf > > + * 0xc0..0xcf > > + * 0xb0..0xbf > > + * 0xa0..0xaf > > + * 0x90..0x9f > > + * 0x80..0x8f > > + * > > + * Priorities within a range will not preempt each other. > > + * > > + * A GIC must support a mimimum of 16 priority levels. > > + */ > > +#define GIC_PRI_LOWEST 0xf0 > > +#define GIC_PRI_IRQ 0xa0 > > +#define GIC_PRI_IPI 0x90 /* IPIs must preempt normal interrupts > */ > > +#define GIC_PRI_HIGHEST 0x80 /* Higher priorities belong to > Secure-World */ > > + > > + > > #ifndef __ASSEMBLY__ > > #include > > > > -- > > 1.7.10.4 > > > -- Name | Title GlobalLogic P +x.xxx.xxx.xxxx M +x.xxx.xxx.xxxx S skype www.globallogic.com http://www.globallogic.com/email_disclaimer.txt --047d7bd6bd765833a904f10bdf64 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi, all.
Sorry for late response.
A lot of thanks to= all for your advices and full solutions.
I will check the current patch= and earlier solution (with tasklet_schedule) too.


On Tue, Jan 28, 2014 at 7:50 PM, Stefano= Stabellini <stefano.stabellini@eu.citrix.com> wrote:
On Tue, 28 Jan 2014, Ian C= ampbell wrote:
> Code such as on_selected_cpus expects/requires that an IPI can preempt= a
> processor which is just handling a normal interrupt. Lacking this prop= erty can
> result in a deadlock between two CPUs trying to IPI each other from in= terrupt
> context.
>
> For the time being there is only two priorities, IRQ and IPI, although= it is
> also conceivable that in the future some IPIs might be higher priority= than
> others. This could be used to implement a better BUG() than we have no= w, but I
> haven't tackled that yet.
>
> Tested with a debug patch which sends a local IPI from a keyhandler, w= hich is
> run in serial interrupt context.
>
> This should also fix the issue reported by Oleksandr in "xen/arm:=
> maintenance_interrupt SMP fix" without resorting to trylock.
>
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> Cc: Oleksandr Tyshchenko <oleksandr.tyshchenko@globallogic.com>

It looks simple enough.
Oleksandr, I would appreciate if could test the patch and tell us if it
is working well for you.


> I think this is probably 4.5 material at this point.
>
> Tested with "HACK: dump pcpu state keyhandler" which I'l= l post for
> completeness. It gives:
> (XEN) Xen call trace:
> (XEN) =A0 =A0[<0000000000212048>] dump_pcpus+0x28/0x2c (PC)
> (XEN) =A0 =A0[<000000000021256c>] handle_keypress+0x70/0xb0 (LR)=
> (XEN) =A0 =A0[<000000000023ed00>] __serial_rx+0x20/0x6c
> (XEN) =A0 =A0[<000000000023f8ac>] serial_rx+0xb4/0xc4
> (XEN) =A0 =A0[<00000000002409ec>] serial_rx_interrupt+0xb0/0xd4<= br> > (XEN) =A0 =A0[<00000000002404b4>] ns16550_interrupt+0x6c/0x90 > (XEN) =A0 =A0[<0000000000245fc0>] do_IRQ+0x144/0x1b4
> (XEN) =A0 =A0[<0000000000245a28>] gic_interrupt+0x60/0xf8
> (XEN) =A0 =A0[<000000000024be64>] do_trap_irq+0x10/0x18
> (XEN) =A0 =A0[<000000000024e240>] hyp_irq+0x5c/0x60
> (XEN) =A0 =A0[<0000000000249324>] init_done+0x10/0x18
> (XEN) =A0 =A0[<0000000000000080>] 0000000000000080
> ---
> =A0xen/arch/arm/gic.c =A0 =A0 =A0 =A0| =A0 19 +++++++++++++------
> =A0xen/arch/arm/time.c =A0 =A0 =A0 | =A0 =A06 +++---
> =A0xen/include/asm-arm/gic.h | =A0 22 ++++++++++++++++++++++
> =A03 files changed, 38 insertions(+), 9 deletions(-)
>
> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> index dcf9cd4..ee37019 100644
> --- a/xen/arch/arm/gic.c
> +++ b/xen/arch/arm/gic.c
> @@ -319,7 +319,8 @@ static void __init gic_dist_init(void)
>
> =A0 =A0 =A0/* Default priority for global interrupts */
> =A0 =A0 =A0for ( i =3D 32; i < gic.lines; i +=3D 4 )
> - =A0 =A0 =A0 =A0GICD[GICD_IPRIORITYR + i / 4] =3D 0xa0a0a0a0;
> + =A0 =A0 =A0 =A0GICD[GICD_IPRIORITYR + i / 4] =3D
> + =A0 =A0 =A0 =A0 =A0 =A0GIC_PRI_IRQ<<24 | GIC_PRI_IRQ<<16= | GIC_PRI_IRQ<<8 | GIC_PRI_IRQ;
>
> =A0 =A0 =A0/* Disable all global interrupts */
> =A0 =A0 =A0for ( i =3D 32; i < gic.lines; i +=3D 32 )
> @@ -341,8 +342,12 @@ static void __cpuinit gic_cpu_init(void)
> =A0 =A0 =A0GICD[GICD_ICENABLER] =3D 0xffff0000; /* Disable all PPI */<= br> > =A0 =A0 =A0GICD[GICD_ISENABLER] =3D 0x0000ffff; /* Enable all SGI */ > =A0 =A0 =A0/* Set PPI and SGI priorities */
> - =A0 =A0for (i =3D 0; i < 32; i +=3D 4)
> - =A0 =A0 =A0 =A0GICD[GICD_IPRIORITYR + i / 4] =3D 0xa0a0a0a0;
> + =A0 =A0for (i =3D 0; i < 16; i +=3D 4)
> + =A0 =A0 =A0 =A0GICD[GICD_IPRIORITYR + i / 4] =3D
> + =A0 =A0 =A0 =A0 =A0 =A0GIC_PRI_IPI<<24 | GIC_PRI_IPI<<16= | GIC_PRI_IPI<<8 | GIC_PRI_IPI;
> + =A0 =A0for (i =3D 16; i < 32; i +=3D 4)
> + =A0 =A0 =A0 =A0GICD[GICD_IPRIORITYR + i / 4] =3D
> + =A0 =A0 =A0 =A0 =A0 =A0GIC_PRI_IRQ<<24 | GIC_PRI_IRQ<<16= | GIC_PRI_IRQ<<8 | GIC_PRI_IRQ;
>
> =A0 =A0 =A0/* Local settings: interface controller */
> =A0 =A0 =A0GICC[GICC_PMR] =3D 0xff; =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0/* = Don't mask by priority */
> @@ -538,7 +543,8 @@ void gic_disable_cpu(void)
> =A0void gic_route_ppis(void)
> =A0{
> =A0 =A0 =A0/* GIC maintenance */
> - =A0 =A0gic_route_dt_irq(&gic.maintenance, cpumask_of(smp_process= or_id()), 0xa0);
> + =A0 =A0gic_route_dt_irq(&gic.maintenance, cpumask_of(smp_process= or_id()),
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 GIC_PRI_IRQ);
> =A0 =A0 =A0/* Route timer interrupt */
> =A0 =A0 =A0route_timer_interrupt();
> =A0}
> @@ -553,7 +559,8 @@ void gic_route_spis(void)
> =A0 =A0 =A0 =A0 =A0if ( (irq =3D serial_dt_irq(seridx)) =3D=3D NULL )<= br> > =A0 =A0 =A0 =A0 =A0 =A0 =A0continue;
>
> - =A0 =A0 =A0 =A0gic_route_dt_irq(irq, cpumask_of(smp_processor_id()),= 0xa0);
> + =A0 =A0 =A0 =A0gic_route_dt_irq(irq, cpumask_of(smp_processor_id()),=
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 GIC_PRI_IRQ);
> =A0 =A0 =A0}
> =A0}
>
> @@ -777,7 +784,7 @@ int gic_route_irq_to_guest(struct domain *d, const= struct dt_irq *irq,
> =A0 =A0 =A0level =3D dt_irq_is_level_triggered(irq);
>
> =A0 =A0 =A0gic_set_irq_properties(irq->irq, level, cpumask_of(smp_p= rocessor_id()),
> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0xa0);
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 GIC_PRI_IRQ); >
> =A0 =A0 =A0retval =3D __setup_irq(desc, irq->irq, action);
> =A0 =A0 =A0if (retval) {
> diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c
> index 81e3e28..68b939d 100644
> --- a/xen/arch/arm/time.c
> +++ b/xen/arch/arm/time.c
> @@ -222,11 +222,11 @@ static void vtimer_interrupt(int irq, void *dev_= id, struct cpu_user_regs *regs)
> =A0void __cpuinit route_timer_interrupt(void)
> =A0{
> =A0 =A0 =A0gic_route_dt_irq(&timer_irq[TIMER_PHYS_NONSECURE_PPI],<= br> > - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 cpumask_of(smp_processor_id(= )), 0xa0);
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 cpumask_of(smp_processor_id(= )), GIC_PRI_IRQ);
> =A0 =A0 =A0gic_route_dt_irq(&timer_irq[TIMER_HYP_PPI],
> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 cpumask_of(smp_processor_id(= )), 0xa0);
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 cpumask_of(smp_processor_id(= )), GIC_PRI_IRQ);
> =A0 =A0 =A0gic_route_dt_irq(&timer_irq[TIMER_VIRT_PPI],
> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 cpumask_of(smp_processor_id(= )), 0xa0);
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 cpumask_of(smp_processor_id(= )), GIC_PRI_IRQ);
> =A0}
>
> =A0/* Set up the timer interrupt on this CPU */
> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
> index 9c6f9bb..25b2b24 100644
> --- a/xen/include/asm-arm/gic.h
> +++ b/xen/include/asm-arm/gic.h
> @@ -129,6 +129,28 @@
> =A0#define GICH_LR_CPUID_SHIFT =A0 =A0 9
> =A0#define GICH_VTR_NRLRGS =A0 =A0 =A0 =A0 0x3f
>
> +/*
> + * The minimum GICC_BPR is required to be in the range 0-3. We set > + * GICC_BPR to 0 but we must expect that it might be 3. This means we=
> + * can rely on premption between the following ranges:
> + * 0xf0..0xff
> + * 0xe0..0xdf
> + * 0xc0..0xcf
> + * 0xb0..0xbf
> + * 0xa0..0xaf
> + * 0x90..0x9f
> + * 0x80..0x8f
> + *
> + * Priorities within a range will not preempt each other.
> + *
> + * A GIC must support a mimimum of 16 priority levels.
> + */
> +#define GIC_PRI_LOWEST =A0 =A0 0xf0
> +#define GIC_PRI_IRQ =A0 =A0 =A0 =A00xa0
> +#define GIC_PRI_IPI =A0 =A0 =A0 =A00x90 /* IPIs must preempt normal i= nterrupts */
> +#define GIC_PRI_HIGHEST =A0 =A00x80 /* Higher priorities belong to Se= cure-World */
> +
> +
> =A0#ifndef __ASSEMBLY__
> =A0#include <xen/device_tree.h>
>
> --
> 1.7.10.4
>



--

Name | Title
GlobalLogic
P +x.xxx.xxx.xxxx=A0=A0M +x.xxx.xxx.xxxx=A0=A0S=A0skype
www.globallogic.com

http://www.globallogic.= com/email_disclaimer.txt
--047d7bd6bd765833a904f10bdf64-- --===============3196472893408834412== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============3196472893408834412==--