[ipipe 4.19][RFC PATCH] ipipe: Do not report spurious interrupts by using printk

[ipipe 4.19][RFC PATCH] ipipe: Do not report spurious interrupts by using printk

[Florian Bezdeka]

Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
---

Hi all,

I need this patch to get the Hikey board (part of the Xenomai CI lab) up
and running. The problem appeared when updating the test images from
Debian 11 to Debian 12 and with that from gcc 10 to gcc 12.

My understanding is:
 - During boot a UART hangup occurs, so we release the IRQ
 - While that happens there is still on IRQ in flight on a different CPU
 - The in-flight IRQ is detected as spurious interrupt
 - We try to report that using printk()

printk() internally calls ipipe_unstall_root() - by using one of the 
_irqrestore() macros/functions - which unconditionally calls 
hard_local_irq_enable() so HW IRQs are enabled now.

I end up in an IRQ stack overflow.

I assume this happens because the UART IRQ is a level-triggered IRQ. So
it simply fires again.

Does that make sense?

How to fix that properly? Ideas welcome...

The problem can be reproduced on the hikey board quite well. Nearly
every boot ends up in the IRQ stack overflow.

Best regards,
Florian
 

 kernel/ipipe/core.c | 2 +-
 kernel/irq/handle.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/kernel/ipipe/core.c b/kernel/ipipe/core.c
index d12e71103424..17ad6f2841f9 100644
--- a/kernel/ipipe/core.c
+++ b/kernel/ipipe/core.c
@@ -1491,7 +1491,7 @@ void __ipipe_dispatch_irq(unsigned int irq, int flags) /* hw interrupts off */
 
 #ifdef CONFIG_IPIPE_DEBUG
 	if (irq >= IPIPE_NR_IRQS) {
-		pr_err("I-pipe: spurious interrupt %u\n", irq);
+		//pr_err("I-pipe: spurious interrupt %u\n", irq);
 		return;
 	}
 #endif
diff --git a/kernel/irq/handle.c b/kernel/irq/handle.c
index e2f7afcb1ae6..1c3221c72977 100644
--- a/kernel/irq/handle.c
+++ b/kernel/irq/handle.c
@@ -32,7 +32,7 @@ void handle_bad_irq(struct irq_desc *desc)
 {
 	unsigned int irq = irq_desc_get_irq(desc);
 
-	print_irq_desc(irq, desc);
+	//print_irq_desc(irq, desc);
 	kstat_incr_irqs_this_cpu(desc);
 	ack_bad_irq(irq);
 }
-- 
2.35.1

Re: [ipipe 4.19][RFC PATCH] ipipe: Do not report spurious interrupts by using printk

[Jan Kiszka]

On 25.11.22 10:08, Florian Bezdeka wrote:
> Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
> ---
> 
> Hi all,
> 
> I need this patch to get the Hikey board (part of the Xenomai CI lab) up
> and running. The problem appeared when updating the test images from
> Debian 11 to Debian 12 and with that from gcc 10 to gcc 12.
> 
> My understanding is:
>  - During boot a UART hangup occurs, so we release the IRQ
>  - While that happens there is still on IRQ in flight on a different CPU
>  - The in-flight IRQ is detected as spurious interrupt
>  - We try to report that using printk()
> 
> printk() internally calls ipipe_unstall_root() - by using one of the 
> _irqrestore() macros/functions - which unconditionally calls 
> hard_local_irq_enable() so HW IRQs are enabled now.
> 
> I end up in an IRQ stack overflow.
> 
> I assume this happens because the UART IRQ is a level-triggered IRQ. So
> it simply fires again.
> 
> Does that make sense?
> 
> How to fix that properly? Ideas welcome...
> 
> The problem can be reproduced on the hikey board quite well. Nearly
> every boot ends up in the IRQ stack overflow.
> 
> Best regards,
> Florian
>  
> 
>  kernel/ipipe/core.c | 2 +-
>  kernel/irq/handle.c | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/ipipe/core.c b/kernel/ipipe/core.c
> index d12e71103424..17ad6f2841f9 100644
> --- a/kernel/ipipe/core.c
> +++ b/kernel/ipipe/core.c
> @@ -1491,7 +1491,7 @@ void __ipipe_dispatch_irq(unsigned int irq, int flags) /* hw interrupts off */
>  
>  #ifdef CONFIG_IPIPE_DEBUG
>  	if (irq >= IPIPE_NR_IRQS) {
> -		pr_err("I-pipe: spurious interrupt %u\n", irq);
> +		//pr_err("I-pipe: spurious interrupt %u\n", irq);
>  		return;
>  	}
>  #endif
> diff --git a/kernel/irq/handle.c b/kernel/irq/handle.c
> index e2f7afcb1ae6..1c3221c72977 100644
> --- a/kernel/irq/handle.c
> +++ b/kernel/irq/handle.c
> @@ -32,7 +32,7 @@ void handle_bad_irq(struct irq_desc *desc)
>  {
>  	unsigned int irq = irq_desc_get_irq(desc);
>  
> -	print_irq_desc(irq, desc);
> +	//print_irq_desc(irq, desc);
>  	kstat_incr_irqs_this_cpu(desc);
>  	ack_bad_irq(irq);
>  }

Is this issue 4.19-specific, resolved in 5.4? Or in dovetail?

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: [ipipe 4.19][RFC PATCH] ipipe: Do not report spurious interrupts by using printk

[Florian Bezdeka]

On Fri, 2022-11-25 at 11:53 +0100, Jan Kiszka wrote:
> On 25.11.22 10:08, Florian Bezdeka wrote:
> > Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
> > ---
> > 
> > Hi all,
> > 
> > I need this patch to get the Hikey board (part of the Xenomai CI lab) up
> > and running. The problem appeared when updating the test images from
> > Debian 11 to Debian 12 and with that from gcc 10 to gcc 12.
> > 
> > My understanding is:
> >  - During boot a UART hangup occurs, so we release the IRQ
> >  - While that happens there is still on IRQ in flight on a different CPU
> >  - The in-flight IRQ is detected as spurious interrupt
> >  - We try to report that using printk()
> > 
> > printk() internally calls ipipe_unstall_root() - by using one of the 
> > _irqrestore() macros/functions - which unconditionally calls 
> > hard_local_irq_enable() so HW IRQs are enabled now.
> > 
> > I end up in an IRQ stack overflow.
> > 
> > I assume this happens because the UART IRQ is a level-triggered IRQ. So
> > it simply fires again.
> > 
> > Does that make sense?
> > 
> > How to fix that properly? Ideas welcome...
> > 
> > The problem can be reproduced on the hikey board quite well. Nearly
> > every boot ends up in the IRQ stack overflow.
> > 
> > Best regards,
> > Florian
> >  
> > 
> >  kernel/ipipe/core.c | 2 +-
> >  kernel/irq/handle.c | 2 +-
> >  2 files changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/kernel/ipipe/core.c b/kernel/ipipe/core.c
> > index d12e71103424..17ad6f2841f9 100644
> > --- a/kernel/ipipe/core.c
> > +++ b/kernel/ipipe/core.c
> > @@ -1491,7 +1491,7 @@ void __ipipe_dispatch_irq(unsigned int irq, int flags) /* hw interrupts off */
> >  
> >  #ifdef CONFIG_IPIPE_DEBUG
> >  	if (irq >= IPIPE_NR_IRQS) {
> > -		pr_err("I-pipe: spurious interrupt %u\n", irq);
> > +		//pr_err("I-pipe: spurious interrupt %u\n", irq);
> >  		return;
> >  	}
> >  #endif
> > diff --git a/kernel/irq/handle.c b/kernel/irq/handle.c
> > index e2f7afcb1ae6..1c3221c72977 100644
> > --- a/kernel/irq/handle.c
> > +++ b/kernel/irq/handle.c
> > @@ -32,7 +32,7 @@ void handle_bad_irq(struct irq_desc *desc)
> >  {
> >  	unsigned int irq = irq_desc_get_irq(desc);
> >  
> > -	print_irq_desc(irq, desc);
> > +	//print_irq_desc(irq, desc);
> >  	kstat_incr_irqs_this_cpu(desc);
> >  	ack_bad_irq(irq);
> >  }
> 
> Is this issue 4.19-specific, resolved in 5.4? Or in dovetail?

The affected code looks the same in 4.19 and 5.4 but it never triggered
on 5.4 (AFAICT). 

I compared the UART drivers between this versions and there was one
noticeable difference. A patch will follow as this difference
fixed/introduced a bug (UART hang), but it has no effect on this one
(irq stack overflow). 

> 
> Jan
> 

Re: [ipipe 4.19][RFC PATCH] ipipe: Do not report spurious interrupts by using printk

[Philippe Gerum]

Jan Kiszka <jan.kiszka@siemens.com> writes:

> On 25.11.22 10:08, Florian Bezdeka wrote:
>> Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
>> ---
>> 
>> Hi all,
>> 
>> I need this patch to get the Hikey board (part of the Xenomai CI lab) up
>> and running. The problem appeared when updating the test images from
>> Debian 11 to Debian 12 and with that from gcc 10 to gcc 12.
>> 
>> My understanding is:
>>  - During boot a UART hangup occurs, so we release the IRQ
>>  - While that happens there is still on IRQ in flight on a different CPU
>>  - The in-flight IRQ is detected as spurious interrupt
>>  - We try to report that using printk()
>> 
>> printk() internally calls ipipe_unstall_root() - by using one of the 
>> _irqrestore() macros/functions - which unconditionally calls 
>> hard_local_irq_enable() so HW IRQs are enabled now.
>> 
>> I end up in an IRQ stack overflow.
>> 
>> I assume this happens because the UART IRQ is a level-triggered IRQ. So
>> it simply fires again.
>> 
>> Does that make sense?
>> 
>> How to fix that properly? Ideas welcome...
>> 
>> The problem can be reproduced on the hikey board quite well. Nearly
>> every boot ends up in the IRQ stack overflow.
>> 
>> Best regards,
>> Florian
>>  
>> 
>>  kernel/ipipe/core.c | 2 +-
>>  kernel/irq/handle.c | 2 +-
>>  2 files changed, 2 insertions(+), 2 deletions(-)
>> 
>> diff --git a/kernel/ipipe/core.c b/kernel/ipipe/core.c
>> index d12e71103424..17ad6f2841f9 100644
>> --- a/kernel/ipipe/core.c
>> +++ b/kernel/ipipe/core.c
>> @@ -1491,7 +1491,7 @@ void __ipipe_dispatch_irq(unsigned int irq, int flags) /* hw interrupts off */
>>  
>>  #ifdef CONFIG_IPIPE_DEBUG
>>  	if (irq >= IPIPE_NR_IRQS) {
>> -		pr_err("I-pipe: spurious interrupt %u\n", irq);
>> +		//pr_err("I-pipe: spurious interrupt %u\n", irq);
>>  		return;
>>  	}
>>  #endif
>> diff --git a/kernel/irq/handle.c b/kernel/irq/handle.c
>> index e2f7afcb1ae6..1c3221c72977 100644
>> --- a/kernel/irq/handle.c
>> +++ b/kernel/irq/handle.c
>> @@ -32,7 +32,7 @@ void handle_bad_irq(struct irq_desc *desc)
>>  {
>>  	unsigned int irq = irq_desc_get_irq(desc);
>>  
>> -	print_irq_desc(irq, desc);
>> +	//print_irq_desc(irq, desc);
>>  	kstat_incr_irqs_this_cpu(desc);
>>  	ack_bad_irq(irq);
>>  }
>
> Is this issue 4.19-specific, resolved in 5.4? Or in dovetail?

Dovetail has no such issue because unstalling the in-band stage does not
re-enable hard irqs.

-- 
Philippe.

Re: [ipipe 4.19][RFC PATCH] ipipe: Do not report spurious interrupts by using printk

[Florian Bezdeka]

On Fri, 2022-11-25 at 13:59 +0100, Philippe Gerum wrote:
> Jan Kiszka <jan.kiszka@siemens.com> writes:
> 
> > On 25.11.22 10:08, Florian Bezdeka wrote:
> > > Signed-off-by: Florian Bezdeka <florian.bezdeka@siemens.com>
> > > ---
> > > 
> > > Hi all,
> > > 
> > > I need this patch to get the Hikey board (part of the Xenomai CI lab) up
> > > and running. The problem appeared when updating the test images from
> > > Debian 11 to Debian 12 and with that from gcc 10 to gcc 12.
> > > 
> > > My understanding is:
> > >  - During boot a UART hangup occurs, so we release the IRQ
> > >  - While that happens there is still on IRQ in flight on a different CPU
> > >  - The in-flight IRQ is detected as spurious interrupt
> > >  - We try to report that using printk()
> > > 
> > > printk() internally calls ipipe_unstall_root() - by using one of the 
> > > _irqrestore() macros/functions - which unconditionally calls 
> > > hard_local_irq_enable() so HW IRQs are enabled now.
> > > 
> > > I end up in an IRQ stack overflow.
> > > 
> > > I assume this happens because the UART IRQ is a level-triggered IRQ. So
> > > it simply fires again.
> > > 
> > > Does that make sense?
> > > 
> > > How to fix that properly? Ideas welcome...
> > > 
> > > The problem can be reproduced on the hikey board quite well. Nearly
> > > every boot ends up in the IRQ stack overflow.
> > > 
> > > Best regards,
> > > Florian
> > >  
> > > 
> > >  kernel/ipipe/core.c | 2 +-
> > >  kernel/irq/handle.c | 2 +-
> > >  2 files changed, 2 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/kernel/ipipe/core.c b/kernel/ipipe/core.c
> > > index d12e71103424..17ad6f2841f9 100644
> > > --- a/kernel/ipipe/core.c
> > > +++ b/kernel/ipipe/core.c
> > > @@ -1491,7 +1491,7 @@ void __ipipe_dispatch_irq(unsigned int irq, int flags) /* hw interrupts off */
> > >  
> > >  #ifdef CONFIG_IPIPE_DEBUG
> > >  	if (irq >= IPIPE_NR_IRQS) {
> > > -		pr_err("I-pipe: spurious interrupt %u\n", irq);
> > > +		//pr_err("I-pipe: spurious interrupt %u\n", irq);
> > >  		return;
> > >  	}
> > >  #endif
> > > diff --git a/kernel/irq/handle.c b/kernel/irq/handle.c
> > > index e2f7afcb1ae6..1c3221c72977 100644
> > > --- a/kernel/irq/handle.c
> > > +++ b/kernel/irq/handle.c
> > > @@ -32,7 +32,7 @@ void handle_bad_irq(struct irq_desc *desc)
> > >  {
> > >  	unsigned int irq = irq_desc_get_irq(desc);
> > >  
> > > -	print_irq_desc(irq, desc);
> > > +	//print_irq_desc(irq, desc);
> > >  	kstat_incr_irqs_this_cpu(desc);
> > >  	ack_bad_irq(irq);
> > >  }
> > 
> > Is this issue 4.19-specific, resolved in 5.4? Or in dovetail?
> 
> Dovetail has no such issue because unstalling the in-band stage does not
> re-enable hard irqs.

Thanks for the clarification, Philippe.

I'm still lost here. How to properly solve this issue? Any ideas?

In addition I still have no idea why this problem can be reproduced on
4.19 easily but not with 5.4. The trees that I played around with were
based on different stable patch levels, so there might be a difference
in one of the drivers, but I was not able find it yet.

Florian