[TEST] CX7 timeouts on reconfig w/ page pool failure injection

[TEST] CX7 timeouts on reconfig w/ page pool failure injection

[Jakub Kicinski]

Hi!

The CX7 testing in NIPA is looking pretty good, one major source of
noise, tho, is that the pp-alloc test sometimes locks rtnl for
a minute.

Here's a failure in that test itself:

# # Exception| subprocess.TimeoutExpired: Command '['ethtool', '-G', 'ens25f1np1', 'rx', '2048']' timed out after 20 seconds
# # Exception| 
# not ok 1 pp_alloc_fail.test_pp_alloc

https://netdev.bots.linux.dev/logview.html?f=/logs/hwksft/CX7-dbg/results/763842/test-outputs/36-pp-alloc-fail-py/stdout

Then the next test that runs also times out during setup:

# subprocess.TimeoutExpired: Command '['ip', '--json', '-d', 'link', 'show', 'dev', 'ens25f1np1']' timed out after 20 seconds
not ok 1 selftests: drivers/net/hw: rss_api.py # exit=1

https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/763842/test-outputs/37-rss-api-py/stdout

pp_alloc test can fail, but spilling into the next test is a problem.

At the retry things clear up enough to return to happiness, tho:

https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/763842/test-outputs/37-rss-api-py-retry/stdout

I haven't try to dig into where it is blocked but does this ring any
bells?

Re: [TEST] CX7 timeouts on reconfig w/ page pool failure injection

[Dragos Tatulea]

On 04.08.26 18:45, Jakub Kicinski wrote:
> Hi!
> 
> The CX7 testing in NIPA is looking pretty good, one major source of
> noise, tho, is that the pp-alloc test sometimes locks rtnl for
> a minute.
> 
> Here's a failure in that test itself:
> 
> # # Exception| subprocess.TimeoutExpired: Command '['ethtool', '-G', 'ens25f1np1', 'rx', '2048']' timed out after 20 seconds
> # # Exception| 
> # not ok 1 pp_alloc_fail.test_pp_alloc
> 
> https://netdev.bots.linux.dev/logview.html?f=/logs/hwksft/CX7-dbg/results/763842/test-outputs/36-pp-alloc-fail-py/stdout
> 
> Then the next test that runs also times out during setup:
> 
> # subprocess.TimeoutExpired: Command '['ip', '--json', '-d', 'link', 'show', 'dev', 'ens25f1np1']' timed out after 20 seconds
> not ok 1 selftests: drivers/net/hw: rss_api.py # exit=1
> 
> https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/763842/test-outputs/37-rss-api-py/stdout
> 
> pp_alloc test can fail, but spilling into the next test is a problem.
> 
> At the retry things clear up enough to return to happiness, tho:
> 
> https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/763842/test-outputs/37-rss-api-py-retry/stdout
> 
> I haven't try to dig into where it is blocked but does this ring any
> bells?
Not to me. We can try an internal reproduction.

Randomly clicking around the flakes page, I found another instance of
pp_alloc_fail.py running and getting crashes in mlx5 during page
deallocation:
https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/762402/sol-machine-2

Is this also something new?

Thanks,
Dragos

Re: [TEST] CX7 timeouts on reconfig w/ page pool failure injection

[Jakub Kicinski]

On Tue, 4 Aug 2026 19:16:19 +0200 Dragos Tatulea wrote:
> Randomly clicking around the flakes page, I found another instance of
> pp_alloc_fail.py running and getting crashes in mlx5 during page
> deallocation:
> https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/762402/sol-machine-2
> 
> Is this also something new?

Ah, that was going to be my next ask. (there were some fake "crashes"
reported in NIPA because the system was retrying and running out of
time, that's hopefully fixed as of this morning so the signal will
be cleaner)

The crash has been hitting since the beginning of July (if it gets
line wrapped into oblivion the tl;dr is that first hit was
net-next-hw-2026-07-03--00-00):

            branch             |       test       |                                                                                                                    crashes                                                                                                             
       
-------------------------------+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------
 net-next-hw-2026-08-04--16-00 | pp-alloc-fail-py | ["", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:mlx5e_free_rx_mpwqe:__napi_poll"]
 net-next-hw-2026-08-03--08-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-08-02--16-00 | pp-alloc-fail-py | ["", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-08-02--08-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-07-31--00-00 | pp-alloc-fail-py | [""]
 net-next-hw-2026-07-28--08-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-07-28--00-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-07-26--16-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-07-25--16-00 | pp-alloc-fail-py | ["", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-07-25--08-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-07-25--00-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_dealloc_rx_mpwqe:mlx5e_free_rx_descs:mlx5e_close_rq:mlx5e_close_queues"]
 net-next-hw-2026-07-23--00-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-07-19--16-03 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-07-19--08-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_dealloc_rx_mpwqe:mlx5e_free_rx_descs:mlx5e_close_rq:mlx5e_close_queues", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-07-18--00-00 | pp-alloc-fail-py | [""]
 net-next-hw-2026-07-17--16-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-07-17--00-00 | irq-py           | ["mlx5e_free_rx_mpwqe:mlx5e_dealloc_rx_mpwqe:mlx5e_free_rx_descs:mlx5e_close_rq:mlx5e_close_queues"]
 net-next-hw-2026-07-15--08-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes"]
 net-next-hw-2026-07-13--16-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-07-13--08-00 | pp-alloc-fail-py | ["", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-07-13--00-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-07-12--16-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-07-11--00-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-07-10--08-00 | pp-alloc-fail-py | ["", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-07-10--00-00 | pp-alloc-fail-py | ["", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-07-09--08-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:mlx5e_free_rx_mpwqe"]
 net-next-hw-2026-07-09--00-00 | pp-alloc-fail-py | ["handle_softirqs:run_ksoftirqd:mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-07-08--08-00 | pp-alloc-fail-py | [""]
 net-next-hw-2026-07-07--16-00 | ethtool-rmon-sh  | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-07-07--08-00 | pp-alloc-fail-py | ["", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-07-07--00-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-07-06--16-00 | pp-alloc-fail-py | ["mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action", "ret_from_fork_asm"]
 net-next-hw-2026-07-06--00-00 | irq-py           | ["mlx5e_free_rx_mpwqe:mlx5e_dealloc_rx_mpwqe:mlx5e_free_rx_descs:mlx5e_close_rq:mlx5e_close_queues"]
 net-next-hw-2026-07-05--16-00 | pp-alloc-fail-py | ["", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]
 net-next-hw-2026-07-03--00-00 | pp-alloc-fail-py | ["", "mlx5e_free_rx_mpwqe:mlx5e_post_rx_mpwqes:mlx5e_napi_poll:__napi_poll:net_rx_action"]


(for my own record)
 SELECT
      r.branch,       
      x.test,
      x.crashes
  FROM results AS r
  CROSS JOIN LATERAL jsonb_to_recordset(
      r.json_full::jsonb -> 'results'
  ) AS x(test text, crashes jsonb)
  WHERE r.remote = 'netdev-CX7-dbg'
    AND x.crashes IS NOT NULL 
    AND x.crashes <> '["kernel crash during test"]'::jsonb
  ORDER BY r.branch_date DESC;

Re: [TEST] CX7 timeouts on reconfig w/ page pool failure injection

[Nimrod Oren]

On 04/08/2026 19:45, Jakub Kicinski wrote:
> Hi!
> 
> The CX7 testing in NIPA is looking pretty good, one major source of
> noise, tho, is that the pp-alloc test sometimes locks rtnl for
> a minute.
> 
> Here's a failure in that test itself:
> 
> # # Exception| subprocess.TimeoutExpired: Command '['ethtool', '-G', 'ens25f1np1', 'rx', '2048']' timed out after 20 seconds
> # # Exception| 
> # not ok 1 pp_alloc_fail.test_pp_alloc
> 
> https://netdev.bots.linux.dev/logview.html?f=/logs/hwksft/CX7-dbg/results/763842/test-outputs/36-pp-alloc-fail-py/stdout
> 
> Then the next test that runs also times out during setup:
> 
> # subprocess.TimeoutExpired: Command '['ip', '--json', '-d', 'link', 'show', 'dev', 'ens25f1np1']' timed out after 20 seconds
> not ok 1 selftests: drivers/net/hw: rss_api.py # exit=1
> 
> https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/763842/test-outputs/37-rss-api-py/stdout
> 
> pp_alloc test can fail, but spilling into the next test is a problem.
>

Seems like the bigger culprit is the earlier xdp test, which leaves MTU
at 9000:

# # Defer Exception| net.lib.py.utils.CmdExitFailure: Command failed
# # Defer Exception| CMD: ip link set dev ens25f1np1 mtu 1500 xdpdrv off
# # Defer Exception|   EXIT: -15
# # Defer Exception|
# not ok 17 xdp.test_xdp_native_adjst_head_grow_data.ipv4
# # Totals: pass:14 fail:3 xfail:0 xpass:0 skip:0 error:0
not ok 1 selftests: drivers/net: xdp.py # TIMEOUT 360 seconds

https://netdev.bots.linux.dev/logview.html?f=/logs/hwksft/CX7-dbg/results/763842/test-outputs/13-xdp-py/stdout

I'll try to reproduce this internally and investigate what happened.

Also, I noticed that the pp-alloc test isn't properly restoring the rx
ring size due to the timeout. In the subsequent retry, it's attempting
to double it again, this time from 2K to 4K:

# # Exception| subprocess.TimeoutExpired: Command '['ethtool', '-G',
'ens25f1np1', 'rx', '4096']' timed out after 20 seconds
# # Exception|
# not ok 1 pp_alloc_fail.test_pp_alloc

https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/763842/test-outputs/36-pp-alloc-fail-py-retry/stdout

The high MTU and ring size along with the heavy debug kernel likely make
the pp-alloc test significantly slower. I'll also prepare a patch to fix
this pp-alloc cleanup issue.

Thanks,
Nimrod

Re: [TEST] CX7 timeouts on reconfig w/ page pool failure injection

[Jakub Kicinski]

On Tue, 11 Aug 2026 10:20:52 +0300 Nimrod Oren wrote:
> On 04/08/2026 19:45, Jakub Kicinski wrote:
> > Hi!
> > 
> > The CX7 testing in NIPA is looking pretty good, one major source of
> > noise, tho, is that the pp-alloc test sometimes locks rtnl for
> > a minute.
> > 
> > Here's a failure in that test itself:
> > 
> > # # Exception| subprocess.TimeoutExpired: Command '['ethtool', '-G', 'ens25f1np1', 'rx', '2048']' timed out after 20 seconds
> > # # Exception| 
> > # not ok 1 pp_alloc_fail.test_pp_alloc
> > 
> > https://netdev.bots.linux.dev/logview.html?f=/logs/hwksft/CX7-dbg/results/763842/test-outputs/36-pp-alloc-fail-py/stdout
> > 
> > Then the next test that runs also times out during setup:
> > 
> > # subprocess.TimeoutExpired: Command '['ip', '--json', '-d', 'link', 'show', 'dev', 'ens25f1np1']' timed out after 20 seconds
> > not ok 1 selftests: drivers/net/hw: rss_api.py # exit=1
> > 
> > https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/763842/test-outputs/37-rss-api-py/stdout
> > 
> > pp_alloc test can fail, but spilling into the next test is a problem.
> >  
> 
> Seems like the bigger culprit is the earlier xdp test, which leaves MTU
> at 9000:
> 
> # # Defer Exception| net.lib.py.utils.CmdExitFailure: Command failed
> # # Defer Exception| CMD: ip link set dev ens25f1np1 mtu 1500 xdpdrv off
> # # Defer Exception|   EXIT: -15
> # # Defer Exception|
> # not ok 17 xdp.test_xdp_native_adjst_head_grow_data.ipv4
> # # Totals: pass:14 fail:3 xfail:0 xpass:0 skip:0 error:0
> not ok 1 selftests: drivers/net: xdp.py # TIMEOUT 360 seconds
> 
> https://netdev.bots.linux.dev/logview.html?f=/logs/hwksft/CX7-dbg/results/763842/test-outputs/13-xdp-py/stdout

Ah, well spotted. That's the test timeout, I'll bump the timeout from
the current 6min to 10min. Once it passes we'll know the true time
needed, and we shall set the timeout to 150% of the expected case..

> I'll try to reproduce this internally and investigate what happened.
> 
> Also, I noticed that the pp-alloc test isn't properly restoring the rx
> ring size due to the timeout. In the subsequent retry, it's attempting
> to double it again, this time from 2K to 4K:
> 
> # # Exception| subprocess.TimeoutExpired: Command '['ethtool', '-G',
> 'ens25f1np1', 'rx', '4096']' timed out after 20 seconds
> # # Exception|
> # not ok 1 pp_alloc_fail.test_pp_alloc
> 
> https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/763842/test-outputs/36-pp-alloc-fail-py-retry/stdout
> 
> The high MTU and ring size along with the heavy debug kernel likely make
> the pp-alloc test significantly slower. I'll also prepare a patch to fix
> this pp-alloc cleanup issue.

To be clear - if the commands time out - how can we fix the cleanup?
Or do you mean some other bug?

Re: [TEST] CX7 timeouts on reconfig w/ page pool failure injection

[Nimrod Oren]

On 11/08/2026 18:04, Jakub Kicinski wrote:
> On Tue, 11 Aug 2026 10:20:52 +0300 Nimrod Oren wrote:
>> On 04/08/2026 19:45, Jakub Kicinski wrote:
>>> Hi!
>>>
>>> The CX7 testing in NIPA is looking pretty good, one major source of
>>> noise, tho, is that the pp-alloc test sometimes locks rtnl for
>>> a minute.
>>>
>>> Here's a failure in that test itself:
>>>
>>> # # Exception| subprocess.TimeoutExpired: Command '['ethtool', '-G', 'ens25f1np1', 'rx', '2048']' timed out after 20 seconds
>>> # # Exception| 
>>> # not ok 1 pp_alloc_fail.test_pp_alloc
>>>
>>> https://netdev.bots.linux.dev/logview.html?f=/logs/hwksft/CX7-dbg/results/763842/test-outputs/36-pp-alloc-fail-py/stdout
>>>
>>> Then the next test that runs also times out during setup:
>>>
>>> # subprocess.TimeoutExpired: Command '['ip', '--json', '-d', 'link', 'show', 'dev', 'ens25f1np1']' timed out after 20 seconds
>>> not ok 1 selftests: drivers/net/hw: rss_api.py # exit=1
>>>
>>> https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/763842/test-outputs/37-rss-api-py/stdout
>>>
>>> pp_alloc test can fail, but spilling into the next test is a problem.
>>>  
>>
>> Seems like the bigger culprit is the earlier xdp test, which leaves MTU
>> at 9000:
>>
>> # # Defer Exception| net.lib.py.utils.CmdExitFailure: Command failed
>> # # Defer Exception| CMD: ip link set dev ens25f1np1 mtu 1500 xdpdrv off
>> # # Defer Exception|   EXIT: -15
>> # # Defer Exception|
>> # not ok 17 xdp.test_xdp_native_adjst_head_grow_data.ipv4
>> # # Totals: pass:14 fail:3 xfail:0 xpass:0 skip:0 error:0
>> not ok 1 selftests: drivers/net: xdp.py # TIMEOUT 360 seconds
>>
>> https://netdev.bots.linux.dev/logview.html?f=/logs/hwksft/CX7-dbg/results/763842/test-outputs/13-xdp-py/stdout
> 
> Ah, well spotted. That's the test timeout, I'll bump the timeout from
> the current 6min to 10min. Once it passes we'll know the true time
> needed, and we shall set the timeout to 150% of the expected case..
> 

Sounds good, thanks.

>> I'll try to reproduce this internally and investigate what happened.
>>
>> Also, I noticed that the pp-alloc test isn't properly restoring the rx
>> ring size due to the timeout. In the subsequent retry, it's attempting
>> to double it again, this time from 2K to 4K:
>>
>> # # Exception| subprocess.TimeoutExpired: Command '['ethtool', '-G',
>> 'ens25f1np1', 'rx', '4096']' timed out after 20 seconds
>> # # Exception|
>> # not ok 1 pp_alloc_fail.test_pp_alloc
>>
>> https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/763842/test-outputs/36-pp-alloc-fail-py-retry/stdout
>>
>> The high MTU and ring size along with the heavy debug kernel likely make
>> the pp-alloc test significantly slower. I'll also prepare a patch to fix
>> this pp-alloc cleanup issue.
> 
> To be clear - if the commands time out - how can we fix the cleanup?
> Or do you mean some other bug?

In the pp-alloc test, the ring size command timed out, but the change
was still applied in the end.

Currently that test only attempts to restore the original ring size if
the set command returns successfully. This case makes me think that it
should attempt the restoration regardless...

Re: [TEST] CX7 timeouts on reconfig w/ page pool failure injection

[Jakub Kicinski]

On Tue, 11 Aug 2026 19:57:31 +0300 Nimrod Oren wrote:
> On 11/08/2026 18:04, Jakub Kicinski wrote:
> > On Tue, 11 Aug 2026 10:20:52 +0300 Nimrod Oren wrote:  
> >> On 04/08/2026 19:45, Jakub Kicinski wrote:  
>  [...]  
> >>
> >> Seems like the bigger culprit is the earlier xdp test, which leaves MTU
> >> at 9000:
> >>
> >> # # Defer Exception| net.lib.py.utils.CmdExitFailure: Command failed
> >> # # Defer Exception| CMD: ip link set dev ens25f1np1 mtu 1500 xdpdrv off
> >> # # Defer Exception|   EXIT: -15
> >> # # Defer Exception|
> >> # not ok 17 xdp.test_xdp_native_adjst_head_grow_data.ipv4
> >> # # Totals: pass:14 fail:3 xfail:0 xpass:0 skip:0 error:0
> >> not ok 1 selftests: drivers/net: xdp.py # TIMEOUT 360 seconds
> >>
> >> https://netdev.bots.linux.dev/logview.html?f=/logs/hwksft/CX7-dbg/results/763842/test-outputs/13-xdp-py/stdout  
> > 
> > Ah, well spotted. That's the test timeout, I'll bump the timeout from
> > the current 6min to 10min. Once it passes we'll know the true time
> > needed, and we shall set the timeout to 150% of the expected case..
> 
> Sounds good, thanks.

10min wasn't enough, bumping to 15min :(

> >> I'll try to reproduce this internally and investigate what happened.
> >>
> >> Also, I noticed that the pp-alloc test isn't properly restoring the rx
> >> ring size due to the timeout. In the subsequent retry, it's attempting
> >> to double it again, this time from 2K to 4K:
> >>
> >> # # Exception| subprocess.TimeoutExpired: Command '['ethtool', '-G',
> >> 'ens25f1np1', 'rx', '4096']' timed out after 20 seconds
> >> # # Exception|
> >> # not ok 1 pp_alloc_fail.test_pp_alloc
> >>
> >> https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/763842/test-outputs/36-pp-alloc-fail-py-retry/stdout
> >>
> >> The high MTU and ring size along with the heavy debug kernel likely make
> >> the pp-alloc test significantly slower. I'll also prepare a patch to fix
> >> this pp-alloc cleanup issue.  
> > 
> > To be clear - if the commands time out - how can we fix the cleanup?
> > Or do you mean some other bug?  
> 
> In the pp-alloc test, the ring size command timed out, but the change
> was still applied in the end.
> 
> Currently that test only attempts to restore the original ring size if
> the set command returns successfully. This case makes me think that it
> should attempt the restoration regardless...

Would the defer() not time out as well if the command to apply 
the config timed out? Any idea what the main time consumer is 
for the reconfig? It can't be anything device related since 
the device isn't really impacted by kernel config options...