Re: [TEST] Google's Netdev-CI: a few flaky tests

[Matthieu Baerts <matttbe@kernel.org>]

Hi Sheena,

On 24/10/2026 19:49, Sheena Mohan wrote:
> On Fri, Oct 23, 2026 at 9:02 AM Matthieu Baerts <matttbe@kernel.org> wrote:
>>
>> Hi Sheena,
>>
>> I hope you are doing well.
>>
>> While looking at the results on the Netdev CI for the Google runners, I
>> notice some tests are quite flaky or constantly failing. I marked them
>> as "ignored" on the Netdev CI while this is being investigated.
>>
>> - IDPF: psp-py started to a month ago. I guess you are monitoring the
>>   results, but then if it is normal, it might be easier if you block the
>>   known bad tests from running on your side. Just in case, a few links:
>>
>>   https://netdev.bots.linux.dev/flakes.html?tn-needle=psp-py&br-pfx=net-next-hw&ld-remote=google-idpf
>>
>>   Because the previous branch didn't get tested, the regression might be
>>   due to the modifications introduced in the previous branch as well:
>>
>>   https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-09-23--16-00.html
>>   https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-09-23--08-00.html
>>
>>
>> - GVE: "queues-py" test started to fail recently: I *think* it is a
>>   regression due to the following patch that has just been dropped, but
>>   we will only know that in a couple of hours:
>>
>>   https://lore.kernel.org/20261021155611.68473-1-phx0fer@gmail.com
>>
>>   https://netdev.bots.linux.dev/contest.html?remote=google-gve&executor=gve-nipa&test=queues-py
>>
>>   https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-10-22--00-00.html
>>
>>
>> - GVE: A few tests are unstable for a while and have been ignored: do
>>   you mind looking at them, please?
>>
>>   https://netdev.bots.linux.dev/flakes.html?min-flip=0&pw-y=0&br-pfx=net-next-hw&ld-remote=google-gve
>>
>>
>> - IDPF: sometimes, it takes ~4h to run the results (+1h to start).
>>   That's fine because we have a 6h window, but just wondering: is it
>>   normal? (The GVE runner seems to start quicker, but I guess that's
>>   because one is on bare metal and the other in a VM)
>>
>> Thank you for your help!
>>
>> Cheers,
>> Matt
> 
> Hi Matt,
> 
> Thanks for reaching out and flagging these issues. Here is an update
> on each of the items:
> 
> IDPF
> 
>   - psp-py: We agree that blocking known bad tests on our side is
> best. While we resolve these
>     failures, we will disable the failing test cases to reduce noise
> on the CI dashboard.
>   - Run/Setup Duration: The longer setup and execution times are due
> to running in a bare metal
>     environment. We are looking into optimizing this to reduce the duration.

Thank you!

> GVE
> 
>   - queues-py: We also noticed this failure recently. Since it is
> suspected to be a kernel
>     regression, we will continue monitoring the results.

It looks like the issue has been fixed when the bad patch has been removed.

>   - Unstable/Ignored Tests:
>       - devmem-py: Should be fixed by patch 4 of the series recently
> sent upstream:
>         https://lore.kernel.org/netdev/20261023204706.3112215-5-hramamurthy@google.com/T/#m0f075c545d9de1c0ad3bd95da0f009a82e80b6c5

Great! Indeed, it looks good so far.

>       - napi-id-py & bond-options-sh: We will investigate both tests.

When fixes are applied or unstable tests are blocked on your side, do
you mind pinging me so I can remove them from this ignore list, please?

  https://netdev.bots.linux.dev/contest/filters-hw.json

Cheers,
Matt