Re: [TEST] CX7 timeouts on reconfig w/ page pool failure injection

[Nimrod Oren <noren@nvidia.com>]

On 11/08/2026 20:38, Jakub Kicinski wrote:
> On Tue, 11 Aug 2026 19:57:31 +0300 Nimrod Oren wrote:
>> On 11/08/2026 18:04, Jakub Kicinski wrote:
>>> On Tue, 11 Aug 2026 10:20:52 +0300 Nimrod Oren wrote:  
>>>> On 04/08/2026 19:45, Jakub Kicinski wrote:  
>>  [...]  
>>>>
>>>> Seems like the bigger culprit is the earlier xdp test, which leaves MTU
>>>> at 9000:
>>>>
>>>> # # Defer Exception| net.lib.py.utils.CmdExitFailure: Command failed
>>>> # # Defer Exception| CMD: ip link set dev ens25f1np1 mtu 1500 xdpdrv off
>>>> # # Defer Exception|   EXIT: -15
>>>> # # Defer Exception|
>>>> # not ok 17 xdp.test_xdp_native_adjst_head_grow_data.ipv4
>>>> # # Totals: pass:14 fail:3 xfail:0 xpass:0 skip:0 error:0
>>>> not ok 1 selftests: drivers/net: xdp.py # TIMEOUT 360 seconds
>>>>
>>>> https://netdev.bots.linux.dev/logview.html?f=/logs/hwksft/CX7-dbg/results/763842/test-outputs/13-xdp-py/stdout  
>>>
>>> Ah, well spotted. That's the test timeout, I'll bump the timeout from
>>> the current 6min to 10min. Once it passes we'll know the true time
>>> needed, and we shall set the timeout to 150% of the expected case..
>>
>> Sounds good, thanks.
> 
> 10min wasn't enough, bumping to 15min :(
> 

In these runs xdp-py took ~9.5 minutes. Both pp-alloc and rss-api tests
passed:

https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/774403/test-outputs/

https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/774889/test-outputs/

https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/775361/test-outputs/

>>>> I'll try to reproduce this internally and investigate what happened.
>>>>
>>>> Also, I noticed that the pp-alloc test isn't properly restoring the rx
>>>> ring size due to the timeout. In the subsequent retry, it's attempting
>>>> to double it again, this time from 2K to 4K:
>>>>
>>>> # # Exception| subprocess.TimeoutExpired: Command '['ethtool', '-G',
>>>> 'ens25f1np1', 'rx', '4096']' timed out after 20 seconds
>>>> # # Exception|
>>>> # not ok 1 pp_alloc_fail.test_pp_alloc
>>>>
>>>> https://netdev.bots.linux.dev/logs/hwksft/CX7-dbg/results/763842/test-outputs/36-pp-alloc-fail-py-retry/stdout
>>>>
>>>> The high MTU and ring size along with the heavy debug kernel likely make
>>>> the pp-alloc test significantly slower. I'll also prepare a patch to fix
>>>> this pp-alloc cleanup issue.  
>>>
>>> To be clear - if the commands time out - how can we fix the cleanup?
>>> Or do you mean some other bug?  
>>
>> In the pp-alloc test, the ring size command timed out, but the change
>> was still applied in the end.
>>
>> Currently that test only attempts to restore the original ring size if
>> the set command returns successfully. This case makes me think that it
>> should attempt the restoration regardless...
> 
> Would the defer() not time out as well if the command to apply 
> the config timed out? Any idea what the main time consumer is 
> for the reconfig? It can't be anything device related since 
> the device isn't really impacted by kernel config options...

It looks like most of the time is being spent in SLUB debugging. In the
pp-alloc test, reconfiguration runs while background traffic is active.
The SLUB overhead disappears a short while after the test traffic stops,
so a deferred restore command is more likely to succeed.