* [TEST] Google's Netdev-CI: a few flaky tests @ 2026-10-23 16:02 Matthieu Baerts 2026-10-24 17:49 ` Sheena Mohan 0 siblings, 1 reply; 5+ messages in thread From: Matthieu Baerts @ 2026-10-23 16:02 UTC (permalink / raw) To: Sheena Mohan; +Cc: Willem de Bruijn, Netdev, Jakub Kicinski Hi Sheena, I hope you are doing well. While looking at the results on the Netdev CI for the Google runners, I notice some tests are quite flaky or constantly failing. I marked them as "ignored" on the Netdev CI while this is being investigated. - IDPF: psp-py started to a month ago. I guess you are monitoring the results, but then if it is normal, it might be easier if you block the known bad tests from running on your side. Just in case, a few links: https://netdev.bots.linux.dev/flakes.html?tn-needle=psp-py&br-pfx=net-next-hw&ld-remote=google-idpf Because the previous branch didn't get tested, the regression might be due to the modifications introduced in the previous branch as well: https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-09-23--16-00.html https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-09-23--08-00.html - GVE: "queues-py" test started to fail recently: I *think* it is a regression due to the following patch that has just been dropped, but we will only know that in a couple of hours: https://lore.kernel.org/20261021155611.68473-1-phx0fer@gmail.com https://netdev.bots.linux.dev/contest.html?remote=google-gve&executor=gve-nipa&test=queues-py https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-10-22--00-00.html - GVE: A few tests are unstable for a while and have been ignored: do you mind looking at them, please? https://netdev.bots.linux.dev/flakes.html?min-flip=0&pw-y=0&br-pfx=net-next-hw&ld-remote=google-gve - IDPF: sometimes, it takes ~4h to run the results (+1h to start). That's fine because we have a 6h window, but just wondering: is it normal? (The GVE runner seems to start quicker, but I guess that's because one is on bare metal and the other in a VM) Thank you for your help! Cheers, Matt ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [TEST] Google's Netdev-CI: a few flaky tests 2026-10-23 16:02 [TEST] Google's Netdev-CI: a few flaky tests Matthieu Baerts @ 2026-10-24 17:49 ` Sheena Mohan 2026-10-26 16:41 ` Matthieu Baerts 0 siblings, 1 reply; 5+ messages in thread From: Sheena Mohan @ 2026-10-24 17:49 UTC (permalink / raw) To: Matthieu Baerts; +Cc: Willem de Bruijn, Netdev, Jakub Kicinski On Fri, Oct 23, 2026 at 9:02 AM Matthieu Baerts <matttbe@kernel.org> wrote: > > Hi Sheena, > > I hope you are doing well. > > While looking at the results on the Netdev CI for the Google runners, I > notice some tests are quite flaky or constantly failing. I marked them > as "ignored" on the Netdev CI while this is being investigated. > > - IDPF: psp-py started to a month ago. I guess you are monitoring the > results, but then if it is normal, it might be easier if you block the > known bad tests from running on your side. Just in case, a few links: > > https://netdev.bots.linux.dev/flakes.html?tn-needle=psp-py&br-pfx=net-next-hw&ld-remote=google-idpf > > Because the previous branch didn't get tested, the regression might be > due to the modifications introduced in the previous branch as well: > > https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-09-23--16-00.html > https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-09-23--08-00.html > > > - GVE: "queues-py" test started to fail recently: I *think* it is a > regression due to the following patch that has just been dropped, but > we will only know that in a couple of hours: > > https://lore.kernel.org/20261021155611.68473-1-phx0fer@gmail.com > > https://netdev.bots.linux.dev/contest.html?remote=google-gve&executor=gve-nipa&test=queues-py > > https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-10-22--00-00.html > > > - GVE: A few tests are unstable for a while and have been ignored: do > you mind looking at them, please? > > https://netdev.bots.linux.dev/flakes.html?min-flip=0&pw-y=0&br-pfx=net-next-hw&ld-remote=google-gve > > > - IDPF: sometimes, it takes ~4h to run the results (+1h to start). > That's fine because we have a 6h window, but just wondering: is it > normal? (The GVE runner seems to start quicker, but I guess that's > because one is on bare metal and the other in a VM) > > Thank you for your help! > > Cheers, > Matt Hi Matt, Thanks for reaching out and flagging these issues. Here is an update on each of the items: IDPF - psp-py: We agree that blocking known bad tests on our side is best. While we resolve these failures, we will disable the failing test cases to reduce noise on the CI dashboard. - Run/Setup Duration: The longer setup and execution times are due to running in a bare metal environment. We are looking into optimizing this to reduce the duration. GVE - queues-py: We also noticed this failure recently. Since it is suspected to be a kernel regression, we will continue monitoring the results. - Unstable/Ignored Tests: - devmem-py: Should be fixed by patch 4 of the series recently sent upstream: https://lore.kernel.org/netdev/20261023204706.3112215-5-hramamurthy@google.com/T/#m0f075c545d9de1c0ad3bd95da0f009a82e80b6c5 - napi-id-py & bond-options-sh: We will investigate both tests. Thanks, Sheena ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [TEST] Google's Netdev-CI: a few flaky tests 2026-10-24 17:49 ` Sheena Mohan @ 2026-10-26 16:41 ` Matthieu Baerts 2026-11-20 3:12 ` Pin-yen Lin 0 siblings, 1 reply; 5+ messages in thread From: Matthieu Baerts @ 2026-10-26 16:41 UTC (permalink / raw) To: Sheena Mohan; +Cc: Willem de Bruijn, Netdev, Jakub Kicinski Hi Sheena, On 24/10/2026 19:49, Sheena Mohan wrote: > On Fri, Oct 23, 2026 at 9:02 AM Matthieu Baerts <matttbe@kernel.org> wrote: >> >> Hi Sheena, >> >> I hope you are doing well. >> >> While looking at the results on the Netdev CI for the Google runners, I >> notice some tests are quite flaky or constantly failing. I marked them >> as "ignored" on the Netdev CI while this is being investigated. >> >> - IDPF: psp-py started to a month ago. I guess you are monitoring the >> results, but then if it is normal, it might be easier if you block the >> known bad tests from running on your side. Just in case, a few links: >> >> https://netdev.bots.linux.dev/flakes.html?tn-needle=psp-py&br-pfx=net-next-hw&ld-remote=google-idpf >> >> Because the previous branch didn't get tested, the regression might be >> due to the modifications introduced in the previous branch as well: >> >> https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-09-23--16-00.html >> https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-09-23--08-00.html >> >> >> - GVE: "queues-py" test started to fail recently: I *think* it is a >> regression due to the following patch that has just been dropped, but >> we will only know that in a couple of hours: >> >> https://lore.kernel.org/20261021155611.68473-1-phx0fer@gmail.com >> >> https://netdev.bots.linux.dev/contest.html?remote=google-gve&executor=gve-nipa&test=queues-py >> >> https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-10-22--00-00.html >> >> >> - GVE: A few tests are unstable for a while and have been ignored: do >> you mind looking at them, please? >> >> https://netdev.bots.linux.dev/flakes.html?min-flip=0&pw-y=0&br-pfx=net-next-hw&ld-remote=google-gve >> >> >> - IDPF: sometimes, it takes ~4h to run the results (+1h to start). >> That's fine because we have a 6h window, but just wondering: is it >> normal? (The GVE runner seems to start quicker, but I guess that's >> because one is on bare metal and the other in a VM) >> >> Thank you for your help! >> >> Cheers, >> Matt > > Hi Matt, > > Thanks for reaching out and flagging these issues. Here is an update > on each of the items: > > IDPF > > - psp-py: We agree that blocking known bad tests on our side is > best. While we resolve these > failures, we will disable the failing test cases to reduce noise > on the CI dashboard. > - Run/Setup Duration: The longer setup and execution times are due > to running in a bare metal > environment. We are looking into optimizing this to reduce the duration. Thank you! > GVE > > - queues-py: We also noticed this failure recently. Since it is > suspected to be a kernel > regression, we will continue monitoring the results. It looks like the issue has been fixed when the bad patch has been removed. > - Unstable/Ignored Tests: > - devmem-py: Should be fixed by patch 4 of the series recently > sent upstream: > https://lore.kernel.org/netdev/20261023204706.3112215-5-hramamurthy@google.com/T/#m0f075c545d9de1c0ad3bd95da0f009a82e80b6c5 Great! Indeed, it looks good so far. > - napi-id-py & bond-options-sh: We will investigate both tests. When fixes are applied or unstable tests are blocked on your side, do you mind pinging me so I can remove them from this ignore list, please? https://netdev.bots.linux.dev/contest/filters-hw.json Cheers, Matt ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [TEST] Google's Netdev-CI: a few flaky tests 2026-10-26 16:41 ` Matthieu Baerts @ 2026-11-20 3:12 ` Pin-yen Lin 2026-11-20 9:09 ` Matthieu Baerts 0 siblings, 1 reply; 5+ messages in thread From: Pin-yen Lin @ 2026-11-20 3:12 UTC (permalink / raw) To: Matthieu Baerts; +Cc: Sheena Mohan, Willem de Bruijn, Netdev, Jakub Kicinski Hi Matthieu, On Mon, Oct 26, 2026 at 9:41 AM Matthieu Baerts <matttbe@kernel.org> wrote: > > Hi Sheena, > > On 24/10/2026 19:49, Sheena Mohan wrote: > > On Fri, Oct 23, 2026 at 9:02 AM Matthieu Baerts <matttbe@kernel.org> wrote: > >> > >> Hi Sheena, > >> > >> I hope you are doing well. > >> > >> While looking at the results on the Netdev CI for the Google runners, I > >> notice some tests are quite flaky or constantly failing. I marked them > >> as "ignored" on the Netdev CI while this is being investigated. > >> > >> - IDPF: psp-py started to a month ago. I guess you are monitoring the > >> results, but then if it is normal, it might be easier if you block the > >> known bad tests from running on your side. Just in case, a few links: > >> > >> https://netdev.bots.linux.dev/flakes.html?tn-needle=psp-py&br-pfx=net-next-hw&ld-remote=google-idpf > >> > >> Because the previous branch didn't get tested, the regression might be > >> due to the modifications introduced in the previous branch as well: > >> > >> https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-09-23--16-00.html > >> https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-09-23--08-00.html > >> > >> > >> - GVE: "queues-py" test started to fail recently: I *think* it is a > >> regression due to the following patch that has just been dropped, but > >> we will only know that in a couple of hours: > >> > >> https://lore.kernel.org/20261021155611.68473-1-phx0fer@gmail.com > >> > >> https://netdev.bots.linux.dev/contest.html?remote=google-gve&executor=gve-nipa&test=queues-py > >> > >> https://netdev.bots.linux.dev/branch_deltas/net-next-hw-2026-10-22--00-00.html > >> > >> > >> - GVE: A few tests are unstable for a while and have been ignored: do > >> you mind looking at them, please? > >> > >> https://netdev.bots.linux.dev/flakes.html?min-flip=0&pw-y=0&br-pfx=net-next-hw&ld-remote=google-gve > >> > >> > >> - IDPF: sometimes, it takes ~4h to run the results (+1h to start). > >> That's fine because we have a 6h window, but just wondering: is it > >> normal? (The GVE runner seems to start quicker, but I guess that's > >> because one is on bare metal and the other in a VM) > >> > >> Thank you for your help! > >> > >> Cheers, > >> Matt > > > > Hi Matt, > > > > Thanks for reaching out and flagging these issues. Here is an update > > on each of the items: > > > > IDPF > > > > - psp-py: We agree that blocking known bad tests on our side is > > best. While we resolve these > > failures, we will disable the failing test cases to reduce noise > > on the CI dashboard. > > - Run/Setup Duration: The longer setup and execution times are due > > to running in a bare metal > > environment. We are looking into optimizing this to reduce the duration. > > Thank you! > > > GVE > > > > - queues-py: We also noticed this failure recently. Since it is > > suspected to be a kernel > > regression, we will continue monitoring the results. > > It looks like the issue has been fixed when the bad patch has been removed. > > > - Unstable/Ignored Tests: > > - devmem-py: Should be fixed by patch 4 of the series recently > > sent upstream: > > https://lore.kernel.org/netdev/20261023204706.3112215-5-hramamurthy@google.com/T/#m0f075c545d9de1c0ad3bd95da0f009a82e80b6c5 > > Great! Indeed, it looks good so far. > > > - napi-id-py & bond-options-sh: We will investigate both tests. > > When fixes are applied or unstable tests are blocked on your side, do > you mind pinging me so I can remove them from this ignore list, please? > > https://netdev.bots.linux.dev/contest/filters-hw.json Both napi-id-py and devmem-py should be fixed now given that [1] has been accepted and merged to net-next. Could you remove these two from the ignore list? The fix for bond-options-sh has been sent out as [2] and the discussions are still ongoing on the mailing list. [1]: https://lore.kernel.org/netdev/20261023204706.3112215-5-hramamurthy@google.com/T/#m0f075c545d9de1c0ad3bd95da0f009a82e80b6c5 [2]: https://lore.kernel.org/all/20261112014942.2946303-1-treapking@google.com/ > > Cheers, > Matt Regards, Pin-yen ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [TEST] Google's Netdev-CI: a few flaky tests 2026-11-20 3:12 ` Pin-yen Lin @ 2026-11-20 9:09 ` Matthieu Baerts 0 siblings, 0 replies; 5+ messages in thread From: Matthieu Baerts @ 2026-11-20 9:09 UTC (permalink / raw) To: Pin-yen Lin; +Cc: Sheena Mohan, Willem de Bruijn, Netdev, Jakub Kicinski Hi Pin-yen, Thank you for this email. On 20/11/2026 04:12, Pin-yen Lin wrote: > On Mon, Oct 26, 2026 at 9:41 AM Matthieu Baerts <matttbe@kernel.org> wrote: >> On 24/10/2026 19:49, Sheena Mohan wrote: (...) >>> - napi-id-py & bond-options-sh: We will investigate both tests. >> >> When fixes are applied or unstable tests are blocked on your side, do >> you mind pinging me so I can remove them from this ignore list, please? >> >> https://netdev.bots.linux.dev/contest/filters-hw.json > > Both napi-id-py and devmem-py should be fixed now given that [1] has > been accepted and merged to net-next. Could you remove these two from > the ignore list? Thank you for the fix and the ping, I just removed these two tests, plus psp-py for IDPF which seems to have been disabled on your side. > The fix for bond-options-sh has been sent out as [2] and the > discussions are still ongoing on the mailing list. That's good, and the v1 already seems to help. Please ping me when the new version is applied. (Note: as mentioned on [1], the Google runners are currently disabled.) [1] https://lore.kernel.org/0469d224-3baa-48c2-9d28-ac0d9fa2ec73@kernel.org Cheers, Matt ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-11-20 9:09 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-10-23 16:02 [TEST] Google's Netdev-CI: a few flaky tests Matthieu Baerts 2026-10-24 17:49 ` Sheena Mohan 2026-10-26 16:41 ` Matthieu Baerts 2026-11-20 3:12 ` Pin-yen Lin 2026-11-20 9:09 ` Matthieu Baerts
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.