[Intel-wired-lan] ice crashes when not enough IRQs

[Intel-wired-lan] ice crashes when not enough IRQs

[Jakub Kicinski]

Hi!

Trying to build a minimal kernel I dropped CONFIG_IRQ_REMAP=y from 
my config, and (on AMD) that caused IRQ shortage. 

This seems to crash ice after commit ad61cd9c67ad ("ice: get rid of
num_lan_msix field"). Sorry for the lack of line numbers, I also
dropped DEBUG_INFO. But I think the problem itself is pretty obvious.
The fix less so, short of reverting ad61cd9c67ad. We can't just clamp
the queues in ice_vsi_alloc_q_vectors() because AFAICT that would make
ethtool -L succeed but driver would have a lower queue count than
requested.


[   80.931193] kselftest: Running tests in drivers/net
[   81.071248] ice 0000:e1:00.0 ens1f0np0: Dedicated RX or TX channels cannot be used simultaneously
[   81.090784] ice 0000:e1:00.0 ens1f0np0: Dedicated RX or TX channels cannot be used simultaneously
[   81.103099] ice 0000:e1:00.0: Failed to allocate 32 q_vectors for VSI 6, new value 8

2026-03-15 14:51:18.171508 [   81.123795] ------------[ cut here ]------------
[   81.128979] WARNING: net/core/dev.c:2861 at __netif_set_xps_queue+0x90a/0xb10, CPU#30: python3/3163
[   81.139148] Modules linked in:
[   81.142587] CPU: 30 UID: 0 PID: 3163 Comm: python3 Not tainted 7.0.0-rc3-banc-g76f4f4965ab1 #1 PREEMPT(lazy) 
[   81.153728] Hardware name: Giga Computing E163-Z34-AAH1-000/MZ33-DC1-000, BIOS R30_F44 12/24/2025
[   81.163696] RIP: 0010:__netif_set_xps_queue+0x90a/0xb10
[   81.169562] Code: 10 48 c7 04 c2 00 00 00 00 e8 62 88 64 ff e9 95 fe ff ff 45 31 c9 bf 40 00 00 00 45 31 d2 41 bb 14 00 00 00 e9 e7 f8 ff ff 90 <0f> 0b 90 e9 1d f7 ff ff 48 8b 5c 24 28 41 0f b7 c7 48 8d 04 80 48
[   81.190649] RSP: 0018:ff70e7fbedc0b7e8 EFLAGS: 00010246
[   81.196522] RAX: ff25cc4a83409000 RBX: 0000000000000008 RCX: 0000000000000000
[   81.204541] RDX: 0000000000000008 RSI: ff25cc4a809a6468 RDI: 0000000000000008
[   81.212559] RBP: ff70e7fbedc0b928 R08: ff25cc4a809a6468 R09: ff25cc4a832fa840
[   81.220575] R10: ff70e7fbe204e460 R11: ff25cc4a81beb0c8 R12: ff70e7fbedc0b8b8
[   81.228590] R13: ff25cc4a98c13828 R14: 0000000000000000 R15: 0000000000000008
[   81.236610] FS:  00007efe2b50af00(0000) GS:ff25cc6225af0000(0000) knlGS:0000000000000000
[   81.245701] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   81.252150] CR2: 000055c982e676a8 CR3: 000000014927c006 CR4: 0000000000771ef0
[   81.260169] PKRU: 55555554
[   81.263202] Call Trace:
[   81.265954]  <TASK>
[   81.268315]  ? ice_ena_vsi_txq+0x1eb/0x300
[   81.272924]  netif_set_xps_queue+0x2c/0x40
[   81.277522]  ice_vsi_cfg_txq+0x221/0x2a0
[   81.281937]  ice_vsi_cfg_lan_txqs+0x60/0x90
[   81.286640]  ice_vsi_cfg_lan+0x24/0xf0
[   81.290856]  ice_vsi_open+0x8a/0x180
[   81.294876]  ice_vsi_recfg_qs+0x10f/0x160
[   81.299384]  ice_set_channels+0x161/0x2b0
[   81.303893]  ethnl_set_channels+0x187/0x2b0
[   81.308601]  ethnl_default_set_doit+0xf6/0x200
[   81.313596]  genl_family_rcv_msg_doit+0xf3/0x150
[   81.318783]  genl_rcv_msg+0x1a1/0x2a0
[   81.322902]  ? ethnl_notify+0xb0/0xb0
[   81.327014]  ? genl_family_rcv_msg_dumpit+0xf0/0xf0
[   81.332496]  netlink_rcv_skb+0x54/0x100
[   81.336811]  genl_rcv+0x23/0x30
[   81.340333]  netlink_unicast+0x250/0x380
[   81.344745]  netlink_sendmsg+0x1ed/0x410
[   81.349156]  __sock_sendmsg+0x33/0x60
[   81.353277]  __sys_sendto+0x120/0x170
[   81.357397]  ? file_has_perm+0xcf/0xe0
[   81.361614]  __x64_sys_sendto+0x1f/0x30
[   81.365926]  do_syscall_64+0xe2/0x570
[   81.370045]  ? exc_page_fault+0x65/0x160
[   81.374456]  entry_SYSCALL_64_after_hwframe+0x4b/0x53
[   81.380128] RIP: 0033:0x7efe2b66fc5e
[   81.384150] Code: 4d 89 d8 e8 34 bd 00 00 4c 8b 5d f8 41 8b 93 08 03 00 00 59 5e 48 83 f8 fc 74 11 c9 c3 0f 1f 80 00 00 00 00 48 8b 45 10 0f 05 <c9> c3 83 e2 39 83 fa 08 75 e7 e8 13 ff ff ff 0f 1f 00 f3 0f 1e fa
[   81.405234] RSP: 002b:00007ffc680ce3f0 EFLAGS: 00000202 ORIG_RAX: 000000000000002c
[   81.413729] RAX: ffffffffffffffda RBX: 00007ffc680ce500 RCX: 00007efe2b66fc5e
[   81.421747] RDX: 0000000000000038 RSI: 00007efe2ac6c410 RDI: 0000000000000006
[   81.429762] RBP: 00007ffc680ce400 R08: 0000000000000000 R09: 0000000000000000
[   81.437778] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000000
[   81.445798] R13: 00007efe2abc1160 R14: 0000000000000000 R15: 00007efe2aee6390
[   81.453806]  </TASK>
[   81.456266] ---[ end trace 0000000000000000 ]---
[   81.463586] BUG: unable to handle page fault for address: 0000000e00001039
[   81.471309] #PF: supervisor read access in kernel mode
[   81.477080] #PF: error_code(0x0000) - not-present page
[   81.482852] PGD 13ce4f067 P4D 0 
[   81.486481] Oops: Oops: 0000 [#1] SMP
[   81.490596] CPU: 30 UID: 0 PID: 3163 Comm: python3 Tainted: G        W           7.0.0-rc3-banc-g76f4f4965ab1 #1 PREEMPT(lazy) 
[   81.503488] Tainted: [W]=WARN
[   81.506822] Hardware name: Giga Computing E163-Z34-AAH1-000/MZ33-DC1-000, BIOS R30_F44 12/24/2025
[   81.516789] RIP: 0010:__netif_set_xps_queue+0x6e/0xb10
[   81.522561] Code: 89 f3 85 f6 0f 88 77 03 00 00 41 0f b7 c7 48 8d 04 80 48 c1 e0 06 48 03 42 18 48 8b 40 70 48 85 c0 0f 84 ff 02 00 00 45 31 d2 <66> 83 78 36 00 48 89 44 24 28 0f 85 ec 02 00 00 48 c7 c7 60 ba 48
[   81.543643] RSP: 0018:ff70e7fbedc0b7e8 EFLAGS: 00010246
[   81.549513] RAX: 0000000e00001003 RBX: 0000000000000001 RCX: 0000000000000000
[   81.557527] RDX: ff25cc4a83409000 RSI: 0000000000000001 RDI: 000000000000000d
[   81.565540] RBP: ff70e7fbedc0b928 R08: ff25cc4a809a6490 R09: ff25cc4a832fa480
[   81.573553] R10: 0000000000000000 R11: ff25cc4a81beb0c8 R12: ff70e7fbedc0b8b8
[   81.581565] R13: ff25cc4a98c13828 R14: 0000000000000000 R15: 000000000000000d
[   81.589579] FS:  00007efe2b50af00(0000) GS:ff25cc6225af0000(0000) knlGS:0000000000000000
[   81.598666] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   81.605120] CR2: 0000000e00001039 CR3: 000000014927c006 CR4: 0000000000771ef0
[   81.613132] PKRU: 55555554
[   81.616175] Call Trace:
[   81.618925]  <TASK>
[   81.621286]  ? ice_ena_vsi_txq+0x1eb/0x300
[   81.625890]  netif_set_xps_queue+0x2c/0x40
[   81.630493]  ice_vsi_cfg_txq+0x221/0x2a0
[   81.634901]  ice_vsi_cfg_lan_txqs+0x60/0x90
[   81.639601]  ice_vsi_cfg_lan+0x24/0xf0
[   81.643815]  ice_vsi_open+0x8a/0x180
[   81.647832]  ice_vsi_recfg_qs+0x10f/0x160
[   81.652338]  ice_set_channels+0x161/0x2b0
[   81.656842]  ethnl_set_channels+0x187/0x2b0
[   81.661543]  ethnl_default_set_doit+0xf6/0x200
[   81.666536]  genl_family_rcv_msg_doit+0xf3/0x150
[   81.671724]  genl_rcv_msg+0x1a1/0x2a0
[   81.675840]  ? ethnl_notify+0xb0/0xb0
[   81.679955]  ? genl_family_rcv_msg_dumpit+0xf0/0xf0
[   81.685434]  netlink_rcv_skb+0x54/0x100
[   81.689745]  genl_rcv+0x23/0x30
[   81.693276]  netlink_unicast+0x250/0x380
[   81.697683]  netlink_sendmsg+0x1ed/0x410
[   81.702092]  __sock_sendmsg+0x33/0x60
[   81.706206]  __sys_sendto+0x120/0x170
[   81.710320]  ? file_has_perm+0xcf/0xe0
[   81.714533]  __x64_sys_sendto+0x1f/0x30
[   81.718843]  do_syscall_64+0xe2/0x570
[   81.722957]  ? exc_page_fault+0x65/0x160
[   81.727365]  entry_SYSCALL_64_after_hwframe+0x4b/0x53
[   81.733039] RIP: 0033:0x7efe2b66fc5e
[   81.737057] Code: 4d 89 d8 e8 34 bd 00 00 4c 8b 5d f8 41 8b 93 08 03 00 00 59 5e 48 83 f8 fc 74 11 c9 c3 0f 1f 80 00 00 00 00 48 8b 45 10 0f 05 <c9> c3 83 e2 39 83 fa 08 75 e7 e8 13 ff ff ff 0f 1f 00 f3 0f 1e fa
[   81.758138] RSP: 002b:00007ffc680ce3f0 EFLAGS: 00000202 ORIG_RAX: 000000000000002c
[   81.766643] RAX: ffffffffffffffda RBX: 00007ffc680ce500 RCX: 00007efe2b66fc5e
[   81.774656] RDX: 0000000000000038 RSI: 00007efe2ac6c410 RDI: 0000000000000006
[   81.782668] RBP: 00007ffc680ce400 R08: 0000000000000000 R09: 0000000000000000
[   81.790681] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000000
[   81.798695] R13: 00007efe2abc1160 R14: 0000000000000000 R15: 00007efe2aee6390
[   81.806709]  </TASK>
[   81.809166] Modules linked in:
[   81.812599] CR2: 0000000e00001039
[   81.816325] ---[ end trace 0000000000000000 ]---
[   81.821513] RIP: 0010:__netif_set_xps_queue+0x6e/0xb10
[   81.827287] Code: 89 f3 85 f6 0f 88 77 03 00 00 41 0f b7 c7 48 8d 04 80 48 c1 e0 06 48 03 42 18 48 8b 40 70 48 85 c0 0f 84 ff 02 00 00 45 31 d2 <66> 83 78 36 00 48 89 44 24 28 0f 85 ec 02 00 00 48 c7 c7 60 ba 48
[   81.848368] RSP: 0018:ff70e7fbedc0b7e8 EFLAGS: 00010246
[   81.854237] RAX: 0000000e00001003 RBX: 0000000000000001 RCX: 0000000000000000
[   81.862251] RDX: ff25cc4a83409000 RSI: 0000000000000001 RDI: 000000000000000d
[   81.870264] RBP: ff70e7fbedc0b928 R08: ff25cc4a809a6490 R09: ff25cc4a832fa480
[   81.878277] R10: 0000000000000000 R11: ff25cc4a81beb0c8 R12: ff70e7fbedc0b8b8
[   81.886290] R13: ff25cc4a98c13828 R14: 0000000000000000 R15: 000000000000000d
[   81.894302] FS:  00007efe2b50af00(0000) GS:ff25cc6225af0000(0000) knlGS:0000000000000000
[   81.903388] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   81.909841] CR2: 0000000e00001039 CR3: 000000014927c006 CR4: 0000000000771

Re: [Intel-wired-lan] ice crashes when not enough IRQs

[Jakub Kicinski]

On Sun, 15 Mar 2026 12:54:51 -0700 Jakub Kicinski wrote:
> Trying to build a minimal kernel I dropped CONFIG_IRQ_REMAP=y from 
> my config, and (on AMD) that caused IRQ shortage. 
> 
> This seems to crash ice after commit ad61cd9c67ad ("ice: get rid of
> num_lan_msix field"). Sorry for the lack of line numbers, I also
> dropped DEBUG_INFO. But I think the problem itself is pretty obvious.
> The fix less so, short of reverting ad61cd9c67ad. We can't just clamp
> the queues in ice_vsi_alloc_q_vectors() because AFAICT that would make
> ethtool -L succeed but driver would have a lower queue count than
> requested.

Hm, maybe it's not just CONFIG_IRQ_REMAP=y
Enabling it makes no difference. Let me try to see what state the IRQ
allocation machinery is in on this kernel. On distro kernel ice gets
all the IRQs it wants at boot. But it also barfs something RDMA so
I can't really compare..

Re: [Intel-wired-lan] ice crashes when not enough IRQs

[Jakub Kicinski]

On Sun, 15 Mar 2026 13:01:50 -0700 Jakub Kicinski wrote:
> On Sun, 15 Mar 2026 12:54:51 -0700 Jakub Kicinski wrote:
> > Trying to build a minimal kernel I dropped CONFIG_IRQ_REMAP=y from 
> > my config, and (on AMD) that caused IRQ shortage. 
> > 
> > This seems to crash ice after commit ad61cd9c67ad ("ice: get rid of
> > num_lan_msix field"). Sorry for the lack of line numbers, I also
> > dropped DEBUG_INFO. But I think the problem itself is pretty obvious.
> > The fix less so, short of reverting ad61cd9c67ad. We can't just clamp
> > the queues in ice_vsi_alloc_q_vectors() because AFAICT that would make
> > ethtool -L succeed but driver would have a lower queue count than
> > requested.  
> 
> Hm, maybe it's not just CONFIG_IRQ_REMAP=y
> Enabling it makes no difference. Let me try to see what state the IRQ
> allocation machinery is in on this kernel. On distro kernel ice gets
> all the IRQs it wants at boot. But it also barfs something RDMA so
> I can't really compare..

I think it's ee13aa1a2c5a ("ice: use netif_get_num_default_rss_queues()")
It clamped the number of allocated queues but I think it meant to only
clamp the default enabled queue count. No idea how y'all gonna get the
extra IRQs later or whether you intended to pack multiple queues per IRQ
so I'll let you figure this out..

Thanks for letting me test crash detection and recovery in NIPA, I guess :D

Re: [Intel-wired-lan] ice crashes when not enough IRQs

[Tony Nguyen]

On 3/15/2026 1:22 PM, Jakub Kicinski wrote:
> On Sun, 15 Mar 2026 13:01:50 -0700 Jakub Kicinski wrote:
>> On Sun, 15 Mar 2026 12:54:51 -0700 Jakub Kicinski wrote:
>>> Trying to build a minimal kernel I dropped CONFIG_IRQ_REMAP=y from
>>> my config, and (on AMD) that caused IRQ shortage.
>>>
>>> This seems to crash ice after commit ad61cd9c67ad ("ice: get rid of
>>> num_lan_msix field"). Sorry for the lack of line numbers, I also
>>> dropped DEBUG_INFO. But I think the problem itself is pretty obvious.
>>> The fix less so, short of reverting ad61cd9c67ad. We can't just clamp
>>> the queues in ice_vsi_alloc_q_vectors() because AFAICT that would make
>>> ethtool -L succeed but driver would have a lower queue count than
>>> requested.
>>
>> Hm, maybe it's not just CONFIG_IRQ_REMAP=y
>> Enabling it makes no difference. Let me try to see what state the IRQ
>> allocation machinery is in on this kernel. On distro kernel ice gets
>> all the IRQs it wants at boot. But it also barfs something RDMA so
>> I can't really compare..
> 
> I think it's ee13aa1a2c5a ("ice: use netif_get_num_default_rss_queues()")
> It clamped the number of allocated queues but I think it meant to only
> clamp the default enabled queue count. No idea how y'all gonna get the
> extra IRQs later or whether you intended to pack multiple queues per IRQ
> so I'll let you figure this out..

Hi Jakub,

Thanks for letting us know. I think we have the fix for this in the 
pipeline [1]. I'll try to get it tested and out to you ASAP.

Thanks,
Tony

[1] 
https://lore.kernel.org/intel-wired-lan/20260223125157.819135-1-michal.swiatkowski@linux.intel.com/

> Thanks for letting me test crash detection and recovery in NIPA, I guess :D

Re: [Intel-wired-lan] ice crashes when not enough IRQs

[Jakub Kicinski]

On Mon, 16 Mar 2026 10:29:56 -0700 Tony Nguyen wrote:
> On 3/15/2026 1:22 PM, Jakub Kicinski wrote:
> > On Sun, 15 Mar 2026 13:01:50 -0700 Jakub Kicinski wrote:  
> >> On Sun, 15 Mar 2026 12:54:51 -0700 Jakub Kicinski wrote:  
>  [...]  
> >>
> >> Hm, maybe it's not just CONFIG_IRQ_REMAP=y
> >> Enabling it makes no difference. Let me try to see what state the IRQ
> >> allocation machinery is in on this kernel. On distro kernel ice gets
> >> all the IRQs it wants at boot. But it also barfs something RDMA so
> >> I can't really compare..  
> > 
> > I think it's ee13aa1a2c5a ("ice: use netif_get_num_default_rss_queues()")
> > It clamped the number of allocated queues but I think it meant to only
> > clamp the default enabled queue count. No idea how y'all gonna get the
> > extra IRQs later or whether you intended to pack multiple queues per IRQ
> > so I'll let you figure this out..  
> 
> Hi Jakub,
> 
> Thanks for letting us know. I think we have the fix for this in the 
> pipeline [1]. I'll try to get it tested and out to you ASAP.

Thanks, do you also have any locking fixes for i40e ?
It's doing even worse, after running these tests:

[   66.614062][ T1267] nipa-hw-worker: [1/36] Running drivers/net:gro.py
[  138.058617][ T1267] nipa-hw-worker: [2/36] Running drivers/net:hds.py
[  138.856172][ T1267] nipa-hw-worker: [3/36] Running drivers/net:napi_id.py
[  140.195848][ T1267] nipa-hw-worker: [4/36] Running drivers/net:napi_threaded.py

The machine locks up waiting on the netdev instance lock.

Looks like on your own NIPA instance this test doesn't get to run
because ethtool is ancient and doesn't support --json for -l

Re: [Intel-wired-lan] ice crashes when not enough IRQs

[Tony Nguyen]

On 3/16/2026 11:59 AM, Jakub Kicinski wrote:
> On Mon, 16 Mar 2026 10:29:56 -0700 Tony Nguyen wrote:
>> On 3/15/2026 1:22 PM, Jakub Kicinski wrote:
>>> On Sun, 15 Mar 2026 13:01:50 -0700 Jakub Kicinski wrote:
>>>> On Sun, 15 Mar 2026 12:54:51 -0700 Jakub Kicinski wrote:
>>   [...]
>>>>
>>>> Hm, maybe it's not just CONFIG_IRQ_REMAP=y
>>>> Enabling it makes no difference. Let me try to see what state the IRQ
>>>> allocation machinery is in on this kernel. On distro kernel ice gets
>>>> all the IRQs it wants at boot. But it also barfs something RDMA so
>>>> I can't really compare..
>>>
>>> I think it's ee13aa1a2c5a ("ice: use netif_get_num_default_rss_queues()")
>>> It clamped the number of allocated queues but I think it meant to only
>>> clamp the default enabled queue count. No idea how y'all gonna get the
>>> extra IRQs later or whether you intended to pack multiple queues per IRQ
>>> so I'll let you figure this out..
>>
>> Hi Jakub,
>>
>> Thanks for letting us know. I think we have the fix for this in the
>> pipeline [1]. I'll try to get it tested and out to you ASAP.
> 
> Thanks, do you also have any locking fixes for i40e ?

No, no pending locking fixes for i40e :(

> It's doing even worse, after running these tests:
> 
> [   66.614062][ T1267] nipa-hw-worker: [1/36] Running drivers/net:gro.py
> [  138.058617][ T1267] nipa-hw-worker: [2/36] Running drivers/net:hds.py
> [  138.856172][ T1267] nipa-hw-worker: [3/36] Running drivers/net:napi_id.py
> [  140.195848][ T1267] nipa-hw-worker: [4/36] Running drivers/net:napi_threaded.py
> 
> The machine locks up waiting on the netdev instance lock.
> 
> Looks like on your own NIPA instance this test doesn't get to run
> because ethtool is ancient and doesn't support --json for -l

I'll see if we can get these updated.

Thanks,
Tony

Re: [Intel-wired-lan] ice crashes when not enough IRQs

[Jakub Kicinski]

On Mon, 16 Mar 2026 12:57:37 -0700 Tony Nguyen wrote:
> >> Thanks for letting us know. I think we have the fix for this in the
> >> pipeline [1]. I'll try to get it tested and out to you ASAP.  
> > 
> > Thanks, do you also have any locking fixes for i40e ?  
> 
> No, no pending locking fixes for i40e :(

Correction, looking closer at the output I think the lock is a second
order effect. The problem is that napi_set_threaded() is waiting forever
on some napi stage change. SOL logs here:

https://netdev-ctrl.bots.linux.dev/logs/hwksft/X710-dbg/results/560801/sol-machine-2