Re: Kernel panic with niu module

[Bjorn Helgaas <helgaas@kernel.org>]

[+cc Thomas, author of 7d5ec3d36123 ("PCI/MSI: Mask all unused MSI-X
entries")]

On Mon, Nov 04, 2024 at 05:34:42AM -0600, Dullfire wrote:
> Hello,
> 
> I am working on a set of patches that address a panic on bind in the niu
> module. However, none of the approaches I see integrate well with the kernels
> frameworks, so any feed back you could provide would be appreciated.
> 
> On sparcv9 systems (and possibly others), when the niu drivers sets up the
> MSIX IRQ vectors, a fatal trap[0] is encountered. I have done a number of
> tests[1]. From these tests I have believe that any read from a specific MSIX
> table entry must come after a write to it's PCI_MSIX_ENTRY_DATA field,
> otherwise it will cause a fatal trap.
> 
> I see types of approaches:
> 1) Add writes to the ENTRY_DATA field to niu before it call into the 
>     msi(x) code.
> 2) Adjust the MSIX code to either skip the read, or write to ENTRY_DATA first
> 3) Add a PCI quirk for this device to "initialize" the MSIX vector table.
> 
> Approach 1 encounters issues in needing to write to the MSIX table. The
> functions needed to do this are internal to msi.c (or drivers/pci/msi/msi.h),
> so they would have to either be reproduces in niu, or exposed in a public
> header. Neither of those seem like a good approach to me.
> 
> Approach 2 can be done in a small amount of code, but it would either require
> the addition of a struct pci_dev flag of some sort, or it would be invasive
> to lots of other devices.
> 
> While approach 3 seems to be the most correct location, it suffers many of
> the same issues as approach 1.
> 
> I have also bisected the kernel, and determined that  upstream commit
> 7d5ec3d3612396dc6d4b76366d20ab9fc06f399f revealed this issue. This commit
> adds read to the mask status before any write to PCI_MSIX_ENTRY_DATA, thus
> provoking the issue.

7d5ec3d36123 ("PCI/MSI: Mask all unused MSI-X entries") appeared in
v5.14 in 2021.  Surely other drivers use MSI-X and would have been
tested on sparcv9 since then?  Just based on the age of 7d5ec3d36123,
I would guess some kind of niu issue.  But Thomas will know much more.

> If you have any suggestions, please let me know.
> 
> Regards,
> Jonathan Currier
> 
> 
> [0] The trap looks like this:
> -----------------------------------------------------------------------------
> [   25.166817] niu: niu.c:v1.1 (Apr 22, 2010)
> [   25.166952] niu 0001:04:00.0: enabling device (0144 -> 0146)
> [   25.174100] niu: niu0: Found PHY 002063b0 type MII at phy_port 26
> [   25.174559] niu: niu0: Found PHY 002063b0 type MII at phy_port 27
> [   25.175004] niu: niu0: Found PHY 002063b0 type MII at phy_port 28
> [   25.175449] niu: niu0: Found PHY 002063b0 type MII at phy_port 29
> [   25.176298] niu: niu0: Port 0 [4 RX chans] [6 TX chans]
> [   25.176405] niu: niu0: Port 1 [4 RX chans] [6 TX chans]
> [   25.176507] niu: niu0: Port 2 [4 RX chans] [6 TX chans]
> [   25.176548] niu: niu0: Port 3 [4 RX chans] [6 TX chans]
> [   25.176590] niu: niu0: Port 0 RDC tbl(0) [ 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 ]
> [   25.176757] niu: niu0: Port 0 RDC tbl(1) [ 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 ]
> [   25.176890] niu: niu0: Port 1 RDC tbl(2) [ 4 5 6 7 4 5 6 7 4 5 6 7 4 5 6 7 ]
> [   25.177053] niu: niu0: Port 1 RDC tbl(3) [ 4 5 6 7 4 5 6 7 4 5 6 7 4 5 6 7 ]
> [   25.177185] niu: niu0: Port 2 RDC tbl(4) [ 8 9 10 11 8 9 10 11 8 9 10 11 8 9 10 11 ]
> [   25.177349] niu: niu0: Port 2 RDC tbl(5) [ 8 9 10 11 8 9 10 11 8 9 10 11 8 9 10 11 ]
> [   25.177483] niu: niu0: Port 3 RDC tbl(6) [ 12 13 14 15 12 13 14 15 12 13 14 15 12 13 14 15 ]
> [   25.177649] niu: niu0: Port 3 RDC tbl(7) [ 12 13 14 15 12 13 14 15 12 13 14 15 12 13 14 15 ]
> [   25.245863] NON-RESUMABLE ERROR: Reporting on cpu 64
> [   25.245973] NON-RESUMABLE ERROR: TPC [0x00000000005f6900] <msix_prepare_msi_desc+0x90/0xa0>
> [   25.246106] NON-RESUMABLE ERROR: RAW [4010000000000016:00000e37f93e32ff:0000000202000080:ffffffffffffffff
> [   25.246215] NON-RESUMABLE ERROR:      0000000800000000:0000000000000000:0000000000000000:0000000000000000]
> [   25.246291] NON-RESUMABLE ERROR: handle [0x4010000000000016] stick [0x00000e37f93e32ff]
> [   25.246335] NON-RESUMABLE ERROR: type [precise nonresumable]
> [   25.246373] NON-RESUMABLE ERROR: attrs [0x02000080] < ASI sp-faulted priv >
> [   25.246435] NON-RESUMABLE ERROR: raddr [0xffffffffffffffff]
> [   25.246476] NON-RESUMABLE ERROR: insn effective address [0x000000c50020000c]
> [   25.246517] NON-RESUMABLE ERROR: size [0x8]
> [   25.246544] NON-RESUMABLE ERROR: asi [0x00]
> [   25.246573] CPU: 64 UID: 0 PID: 745 Comm: kworker/64:1 Not tainted 6.11.5 #63
> [   25.246625] Workqueue: events work_for_cpu_fn
> [   25.246671] TSTATE: 0000000011001602 TPC: 00000000005f6900 TNPC: 00000000005f6904 Y: 00000000    Not tainted
> [   25.246729] TPC: <msix_prepare_msi_desc+0x90/0xa0>
> [   25.246771] g0: 00000000000002e9 g1: 000000000000000c g2: 000000c50020000c g3: 0000000000000100
> [   25.246815] g4: ffff8000470307c0 g5: ffff800fec5be000 g6: ffff800047a08000 g7: 0000000000000000
> [   25.246861] o0: ffff800014feb000 o1: ffff800047a0b620 o2: 0000000000000011 o3: ffff800047a0b620
> [   25.246906] o4: 0000000000000080 o5: 0000000000000011 sp: ffff800047a0ad51 ret_pc: 00000000005f7128
> [   25.246951] RPC: <__pci_enable_msix_range+0x3cc/0x460>
> [   25.247004] l0: 000000000000000d l1: 000000000000c01f l2: ffff800014feb0a8 l3: 0000000000000020
> [   25.247049] l4: 000000000000c000 l5: 0000000000000001 l6: 0000000020000000 l7: ffff800047a0b734
> [   25.247094] i0: ffff800014feb000 i1: ffff800047a0b730 i2: 0000000000000001 i3: 000000000000000d
> [   25.247138] i4: 0000000000000000 i5: 0000000000000000 i6: ffff800047a0ae81 i7: 00000000101888b0
> [   25.247182] I7: <niu_try_msix.constprop.0+0xc0/0x130 [niu]>
> [   25.247321] Call Trace:
> [   25.247346] [<00000000101888b0>] niu_try_msix.constprop.0+0xc0/0x130 [niu]
> [   25.247442] [<000000001018f840>] niu_get_invariants+0x183c/0x207c [niu]
> [   25.247536] [<00000000101902fc>] niu_pci_init_one+0x27c/0x2fc [niu]
> [   25.247630] [<00000000005ef3e4>] local_pci_probe+0x28/0x74
> [   25.247677] [<0000000000469240>] work_for_cpu_fn+0x8/0x1c
> [   25.247726] [<000000000046b008>] process_scheduled_works+0x144/0x210
> [   25.247782] [<000000000046b518>] worker_thread+0x13c/0x1c0
> [   25.247833] [<00000000004710e0>] kthread+0xb8/0xc8
> [   25.247874] [<00000000004060c8>] ret_from_fork+0x1c/0x2c
> [   25.247931] [<0000000000000000>] 0x0
> [   25.247961] Kernel panic - not syncing: Non-resumable error.
> -----------------------------------------------------------------------------
> 
> [1] Tests I have done (and their results)
> All tests done on a T5240 (UltaSPARC T2).
> In my test cases: niu driver tries to use up to 13 vectors (table size of 32 entries).
> "SUCCCESS" - Test case booted with functional networking
> "FAILED" - Test case experienced a fatal trap after loading the niu module 
>  - writing 0 to all of the MSIX table entrys' ENTRY_VECTOR_CTRL: FAILED 
>  - writing 0 to all of the MSIX table entrys' ENTRY_LOWER_ADDR: FAILED 
>  - writing 0 to all of the MSIX table entrys' ENTRY_UPPER_ADDR: FAILED 
>  - writing 0 to all of the MSIX table entrys' ENTRY_DATA: SUCCESS
>  - writing 0 to only one MSIX table entry's ENTRY_DATA: FAILED
>  - writing 0 to only the first 1/2 of the MSIX table entrys' ENTRY_DATA: SUCCESS
>  - writing ~0 to only the first 1/2 of the MSIX table entrys' ENTRY_DATA: SUCCESS
>  - writing 0 to only the first 12 of the MSIX table entrys' ENTRY_DATA: FAILED
>  - writing 0 to only the first 13 of the MSIX table entrys' ENTRY_DATA: SUCCESS
>  - reading ENTRY_DATA before writing it: FAILED 
>  - reading ENTRY_LOWER_ADDR before writing it: FAILED 
>  - reading ENTRY_UPPER_ADDR before writing it: FAILED