[TEST] Google's Netdev-CI: a few flaky tests

[TEST] Google's Netdev-CI: a few flaky tests

[Matthieu Baerts]

Hi Sheena,

I hope you are doing well.

While looking at the results on the Netdev CI for the Google runners, I
notice some tests are quite flaky or constantly failing. I marked them
as "ignored" on the Netdev CI while this is being investigated.

- IDPF: psp-py started to a month ago. I guess you are monitoring the
  results, but then if it is normal, it might be easier if you block the
  known bad tests from running on your side. Just in case, a few links:

  https://netdev.bots.linux.dev/flakes.html?tn-needle=psp-py&br-pfx=net-next-hw&ld-remote=google-idpf

  Because the previous branch didn't get tested, the regression might be
  due to the modifications introduced in the previous branch as well:

  https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-09-23--16-00.html
  https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-09-23--08-00.html


- GVE: "queues-py" test started to fail recently: I *think* it is a
  regression due to the following patch that has just been dropped, but
  we will only know that in a couple of hours:

  https://lore.kernel.org/20261021155611.68473-1-phx0fer@gmail.com

  https://netdev.bots.linux.dev/contest.html?remote=google-gve&executor=gve-nipa&test=queues-py

  https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-10-22--00-00.html


- GVE: A few tests are unstable for a while and have been ignored: do
  you mind looking at them, please?

  https://netdev.bots.linux.dev/flakes.html?min-flip=0&pw-y=0&br-pfx=net-next-hw&ld-remote=google-gve


- IDPF: sometimes, it takes ~4h to run the results (+1h to start).
  That's fine because we have a 6h window, but just wondering: is it
  normal? (The GVE runner seems to start quicker, but I guess that's
  because one is on bare metal and the other in a VM)

Thank you for your help!

Cheers,
Matt

Re: [TEST] Google's Netdev-CI: a few flaky tests

[Sheena Mohan]

On Fri, Oct 23, 2026 at 9:02 AM Matthieu Baerts <matttbe@kernel.org> wrote:
>
> Hi Sheena,
>
> I hope you are doing well.
>
> While looking at the results on the Netdev CI for the Google runners, I
> notice some tests are quite flaky or constantly failing. I marked them
> as "ignored" on the Netdev CI while this is being investigated.
>
> - IDPF: psp-py started to a month ago. I guess you are monitoring the
>   results, but then if it is normal, it might be easier if you block the
>   known bad tests from running on your side. Just in case, a few links:
>
>   https://netdev.bots.linux.dev/flakes.html?tn-needle=psp-py&br-pfx=net-next-hw&ld-remote=google-idpf
>
>   Because the previous branch didn't get tested, the regression might be
>   due to the modifications introduced in the previous branch as well:
>
>   https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-09-23--16-00.html
>   https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-09-23--08-00.html
>
>
> - GVE: "queues-py" test started to fail recently: I *think* it is a
>   regression due to the following patch that has just been dropped, but
>   we will only know that in a couple of hours:
>
>   https://lore.kernel.org/20261021155611.68473-1-phx0fer@gmail.com
>
>   https://netdev.bots.linux.dev/contest.html?remote=google-gve&executor=gve-nipa&test=queues-py
>
>   https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-10-22--00-00.html
>
>
> - GVE: A few tests are unstable for a while and have been ignored: do
>   you mind looking at them, please?
>
>   https://netdev.bots.linux.dev/flakes.html?min-flip=0&pw-y=0&br-pfx=net-next-hw&ld-remote=google-gve
>
>
> - IDPF: sometimes, it takes ~4h to run the results (+1h to start).
>   That's fine because we have a 6h window, but just wondering: is it
>   normal? (The GVE runner seems to start quicker, but I guess that's
>   because one is on bare metal and the other in a VM)
>
> Thank you for your help!
>
> Cheers,
> Matt

Hi Matt,

Thanks for reaching out and flagging these issues. Here is an update
on each of the items:

IDPF

  - psp-py: We agree that blocking known bad tests on our side is
best. While we resolve these
    failures, we will disable the failing test cases to reduce noise
on the CI dashboard.
  - Run/Setup Duration: The longer setup and execution times are due
to running in a bare metal
    environment. We are looking into optimizing this to reduce the duration.

GVE

  - queues-py: We also noticed this failure recently. Since it is
suspected to be a kernel
    regression, we will continue monitoring the results.
  - Unstable/Ignored Tests:
      - devmem-py: Should be fixed by patch 4 of the series recently
sent upstream:
        https://lore.kernel.org/netdev/20261023204706.3112215-5-hramamurthy@google.com/T/#m0f075c545d9de1c0ad3bd95da0f009a82e80b6c5
      - napi-id-py & bond-options-sh: We will investigate both tests.

Thanks,
Sheena

Re: [TEST] Google's Netdev-CI: a few flaky tests

[Matthieu Baerts]

Hi Sheena,

On 24/10/2026 19:49, Sheena Mohan wrote:
> On Fri, Oct 23, 2026 at 9:02 AM Matthieu Baerts <matttbe@kernel.org> wrote:
>>
>> Hi Sheena,
>>
>> I hope you are doing well.
>>
>> While looking at the results on the Netdev CI for the Google runners, I
>> notice some tests are quite flaky or constantly failing. I marked them
>> as "ignored" on the Netdev CI while this is being investigated.
>>
>> - IDPF: psp-py started to a month ago. I guess you are monitoring the
>>   results, but then if it is normal, it might be easier if you block the
>>   known bad tests from running on your side. Just in case, a few links:
>>
>>   https://netdev.bots.linux.dev/flakes.html?tn-needle=psp-py&br-pfx=net-next-hw&ld-remote=google-idpf
>>
>>   Because the previous branch didn't get tested, the regression might be
>>   due to the modifications introduced in the previous branch as well:
>>
>>   https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-09-23--16-00.html
>>   https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-09-23--08-00.html
>>
>>
>> - GVE: "queues-py" test started to fail recently: I *think* it is a
>>   regression due to the following patch that has just been dropped, but
>>   we will only know that in a couple of hours:
>>
>>   https://lore.kernel.org/20261021155611.68473-1-phx0fer@gmail.com
>>
>>   https://netdev.bots.linux.dev/contest.html?remote=google-gve&executor=gve-nipa&test=queues-py
>>
>>   https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-10-22--00-00.html
>>
>>
>> - GVE: A few tests are unstable for a while and have been ignored: do
>>   you mind looking at them, please?
>>
>>   https://netdev.bots.linux.dev/flakes.html?min-flip=0&pw-y=0&br-pfx=net-next-hw&ld-remote=google-gve
>>
>>
>> - IDPF: sometimes, it takes ~4h to run the results (+1h to start).
>>   That's fine because we have a 6h window, but just wondering: is it
>>   normal? (The GVE runner seems to start quicker, but I guess that's
>>   because one is on bare metal and the other in a VM)
>>
>> Thank you for your help!
>>
>> Cheers,
>> Matt
> 
> Hi Matt,
> 
> Thanks for reaching out and flagging these issues. Here is an update
> on each of the items:
> 
> IDPF
> 
>   - psp-py: We agree that blocking known bad tests on our side is
> best. While we resolve these
>     failures, we will disable the failing test cases to reduce noise
> on the CI dashboard.
>   - Run/Setup Duration: The longer setup and execution times are due
> to running in a bare metal
>     environment. We are looking into optimizing this to reduce the duration.

Thank you!

> GVE
> 
>   - queues-py: We also noticed this failure recently. Since it is
> suspected to be a kernel
>     regression, we will continue monitoring the results.

It looks like the issue has been fixed when the bad patch has been removed.

>   - Unstable/Ignored Tests:
>       - devmem-py: Should be fixed by patch 4 of the series recently
> sent upstream:
>         https://lore.kernel.org/netdev/20261023204706.3112215-5-hramamurthy@google.com/T/#m0f075c545d9de1c0ad3bd95da0f009a82e80b6c5

Great! Indeed, it looks good so far.

>       - napi-id-py & bond-options-sh: We will investigate both tests.

When fixes are applied or unstable tests are blocked on your side, do
you mind pinging me so I can remove them from this ignore list, please?

  https://netdev.bots.linux.dev/contest/filters-hw.json

Cheers,
Matt