intel-lpss 0000:00:15.1: idma64_irq: status=0x0, millions of lines spamming journal

intel-lpss 0000:00:15.1: idma64_irq: status=0x0, millions of lines spamming journal

[Chris Murphy]

Downstream bug has dmesg, lspci, acpidump attached
https://bugzilla.redhat.com/show_bug.cgi?id=2188969

The gist is repeating message:

kernel: intel-lpss 0000:00:15.1: idma64_irq: status=0x0

~6800 times in a couple seconds, and in a few hours racked up over 3.3 million.

Bug first appears in 6.3.0-0.rc2.20230317git38e04b3e4240.27.fc39.x86_64+debug
Last good version is 6.3.0-0.rc2.20230315git6015b1aca1a2.25.fc39.x86_64+debug

Bug appears in 6.3.0 release. It does not appear in 6.2 series kernels or older.


--
Chris Murphy

Re: intel-lpss 0000:00:15.1: idma64_irq: status=0x0, millions of lines spamming journal

[Wolfram Sang]

[-- Attachment #1: Type: text/plain, Size: 879 bytes --]


CCing the designware maintainers. Not sure if it really is the I2C part
of lpss which regressed, though. There wasn't a change to the driver
since 6.3-rc1. The changes in rc1 seem unrelated to me, but I leave that
to the pros.

On Mon, Apr 24, 2023 at 05:51:15PM -0400, Chris Murphy wrote:
> Downstream bug has dmesg, lspci, acpidump attached
> https://bugzilla.redhat.com/show_bug.cgi?id=2188969
> 
> The gist is repeating message:
> 
> kernel: intel-lpss 0000:00:15.1: idma64_irq: status=0x0
> 
> ~6800 times in a couple seconds, and in a few hours racked up over 3.3 million.
> 
> Bug first appears in 6.3.0-0.rc2.20230317git38e04b3e4240.27.fc39.x86_64+debug
> Last good version is 6.3.0-0.rc2.20230315git6015b1aca1a2.25.fc39.x86_64+debug
> 
> Bug appears in 6.3.0 release. It does not appear in 6.2 series kernels or older.
> 
> 
> --
> Chris Murphy

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: intel-lpss 0000:00:15.1: idma64_irq: status=0x0, millions of lines spamming journal

[Mika Westerberg]

Hi Chris,

Would you be able to bisect this to a mainline commit?

At least looking at the changes between v6.3-rc1 and v6.3-rc7 there is
virtually nothing to any of these drivers involved. The log itself looks
like:

       dev_vdbg(idma64->dma.dev, "%s: status=%#x\n", __func__, status);

so this should not be enabled at all unless CONFIG_DMADEVICES_VDEBUG is
set to y which seems odd in distro kernel.

Also what does /proc/interrupts show for this?

On Tue, Apr 25, 2023 at 07:59:11AM +0200, Wolfram Sang wrote:
> 
> CCing the designware maintainers. Not sure if it really is the I2C part
> of lpss which regressed, though. There wasn't a change to the driver
> since 6.3-rc1. The changes in rc1 seem unrelated to me, but I leave that
> to the pros.
> 
> On Mon, Apr 24, 2023 at 05:51:15PM -0400, Chris Murphy wrote:
> > Downstream bug has dmesg, lspci, acpidump attached
> > https://bugzilla.redhat.com/show_bug.cgi?id=2188969
> > 
> > The gist is repeating message:
> > 
> > kernel: intel-lpss 0000:00:15.1: idma64_irq: status=0x0
> > 
> > ~6800 times in a couple seconds, and in a few hours racked up over 3.3 million.
> > 
> > Bug first appears in 6.3.0-0.rc2.20230317git38e04b3e4240.27.fc39.x86_64+debug
> > Last good version is 6.3.0-0.rc2.20230315git6015b1aca1a2.25.fc39.x86_64+debug
> > 
> > Bug appears in 6.3.0 release. It does not appear in 6.2 series kernels or older.
> > 
> > 
> > --
> > Chris Murphy

Re: intel-lpss 0000:00:15.1: idma64_irq: status=0x0, millions of lines spamming journal

[Chris Murphy]

On Tue, Apr 25, 2023, at 3:01 AM, Mika Westerberg wrote:
> Hi Chris,
>
> Would you be able to bisect this to a mainline commit?

Difficult in the near term.


> At least looking at the changes between v6.3-rc1 and v6.3-rc7 there is
> virtually nothing to any of these drivers involved. The log itself looks
> like:
>
>        dev_vdbg(idma64->dma.dev, "%s: status=%#x\n", __func__, status);
>
> so this should not be enabled at all unless CONFIG_DMADEVICES_VDEBUG is
> set to y which seems odd in distro kernel.

$ grep DMADEVICES /boot/config-6.3.0-0.rc2.20230315git6015b1aca1a2.25.fc39.x86_64+debug 
CONFIG_DMADEVICES=y
CONFIG_DMADEVICES_DEBUG=y
# CONFIG_DMADEVICES_VDEBUG is not set
$ grep DMADEVICES /boot/config-6.3.0-0.rc2.20230317git38e04b3e4240.27.fc39.x86_64+debug 
CONFIG_DMADEVICES=y
CONFIG_DMADEVICES_DEBUG=y
CONFIG_DMADEVICES_VDEBUG=y

It follows the bug, though I'm not sure if this is the true source of the problem?


> Also what does /proc/interrupts show for this?

Attached to bug report.
https://bugzilla-attachments.redhat.com/attachment.cgi?id=1959838



-- 
Chris Murphy

Re: intel-lpss 0000:00:15.1: idma64_irq: status=0x0, millions of lines spamming journal

[Mika Westerberg]

On Tue, Apr 25, 2023 at 01:14:35PM -0400, Chris Murphy wrote:
> 
> 
> On Tue, Apr 25, 2023, at 3:01 AM, Mika Westerberg wrote:
> > Hi Chris,
> >
> > Would you be able to bisect this to a mainline commit?
> 
> Difficult in the near term.
> 
> 
> > At least looking at the changes between v6.3-rc1 and v6.3-rc7 there is
> > virtually nothing to any of these drivers involved. The log itself looks
> > like:
> >
> >        dev_vdbg(idma64->dma.dev, "%s: status=%#x\n", __func__, status);
> >
> > so this should not be enabled at all unless CONFIG_DMADEVICES_VDEBUG is
> > set to y which seems odd in distro kernel.
> 
> $ grep DMADEVICES /boot/config-6.3.0-0.rc2.20230315git6015b1aca1a2.25.fc39.x86_64+debug 
> CONFIG_DMADEVICES=y
> CONFIG_DMADEVICES_DEBUG=y
> # CONFIG_DMADEVICES_VDEBUG is not set
> $ grep DMADEVICES /boot/config-6.3.0-0.rc2.20230317git38e04b3e4240.27.fc39.x86_64+debug 
> CONFIG_DMADEVICES=y
> CONFIG_DMADEVICES_DEBUG=y
> CONFIG_DMADEVICES_VDEBUG=y
> 
> It follows the bug, though I'm not sure if this is the true source of the problem?

Okay, so the issue here I think is just that the VDEBUG is enabled.
Since idma64 and I2C share the interrupt, each time a I2C transaction is
done the idma64 interrupt handler is called as well:

17:          0          0      18841          0          0          0          0          0  IR-IO-APIC   17-fasteoi   i2c_designware.1, idma64.1

so these end up in the dmesg and journal. I suggest to just disable the
VDEBUG. Probably was enabled in the Fedora .config by accident as this
is something that should not be enabled by distro kernels.