[PATCH net] selftests: drv-net: rss_ctx: Add short delay between per-context traffic checks

[PATCH net] selftests: drv-net: rss_ctx: Add short delay between per-context traffic checks

[Nimrod Oren]

A few packets may still be sent and received during the termination of
the iperf processes. These late packets cause failures when they arrive
on queues expected to be empty.

Add a one second delay between repeated _send_traffic_check() calls in
rss_ctx tests to ensure such packets are processed before the next
traffic checks are performed.

Example failure observed:

  Check failed 2 != 0 traffic on inactive queues (context 1):
  [0, 0, 1, 1, 386385, 397196, 0, 0, 0, 0, ...]

  Check failed 4 != 0 traffic on inactive queues (context 2):
  [0, 0, 0, 0, 2, 2, 247152, 253013, 0, 0, ...]

  Check failed 2 != 0 traffic on inactive queues (context 3):
  [0, 0, 0, 0, 0, 0, 1, 1, 282434, 283070, ...]

Note: While the `noise` parameter could be used to tolerate these late
packets, it would be inappropriate here. `noise` tolerates far more
traffic than acceptable in this case, risking false positives.
Inactive queues are supposed to see zero traffic.

Fixes: 847aa551fa78 ("selftests: drv-net: rss_ctx: factor out send traffic and check")
Reviewed-by: Gal Pressman <gal@nvidia.com>
Reviewed-by: Carolina Jubran <cjubran@nvidia.com>
Signed-off-by: Nimrod Oren <noren@nvidia.com>
---
 tools/testing/selftests/drivers/net/hw/rss_ctx.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tools/testing/selftests/drivers/net/hw/rss_ctx.py b/tools/testing/selftests/drivers/net/hw/rss_ctx.py
index 7bb552f8b182..19be69227693 100755
--- a/tools/testing/selftests/drivers/net/hw/rss_ctx.py
+++ b/tools/testing/selftests/drivers/net/hw/rss_ctx.py
@@ -4,6 +4,7 @@
 import datetime
 import random
 import re
+import time
 from lib.py import ksft_run, ksft_pr, ksft_exit
 from lib.py import ksft_eq, ksft_ne, ksft_ge, ksft_in, ksft_lt, ksft_true, ksft_raises
 from lib.py import NetDrvEpEnv
@@ -492,6 +493,7 @@ def test_rss_context(cfg, ctx_cnt=1, create_with_cfg=None):
                             { 'target': (2+i*2, 3+i*2),
                               'noise': (0, 1),
                               'empty': list(range(2, 2+i*2)) + list(range(4+i*2, 2+2*ctx_cnt)) })
+        time.sleep(1)
 
     if requested_ctx_cnt != ctx_cnt:
         raise KsftSkipEx(f"Tested only {ctx_cnt} contexts, wanted {requested_ctx_cnt}")
@@ -559,6 +561,7 @@ def test_rss_context_out_of_order(cfg, ctx_cnt=4):
                 }
 
             _send_traffic_check(cfg, ports[i], f"context {i}", expected)
+            time.sleep(1)
 
     # Use queues 0 and 1 for normal traffic
     ethtool(f"-X {cfg.ifname} equal 2")
-- 
2.37.1

Re: [PATCH net] selftests: drv-net: rss_ctx: Add short delay between per-context traffic checks

[Jakub Kicinski]

On Sun, 29 Jun 2025 14:18:12 +0300 Nimrod Oren wrote:
> A few packets may still be sent and received during the termination of
> the iperf processes. These late packets cause failures when they arrive
> on queues expected to be empty.
> 
> Add a one second delay between repeated _send_traffic_check() calls in
> rss_ctx tests to ensure such packets are processed before the next
> traffic checks are performed.

Sprinklings sleeps should be last resort. Is there a way to wait for
iperf to shut down cleanly, or wait for the socket to be closed fully?
Like wait_port_listen() ?

Re: [PATCH net] selftests: drv-net: rss_ctx: Add short delay between per-context traffic checks

[Nimrod Oren]

On 02/07/2025 3:23, Jakub Kicinski wrote:
> On Sun, 29 Jun 2025 14:18:12 +0300 Nimrod Oren wrote:
>> A few packets may still be sent and received during the termination of
>> the iperf processes. These late packets cause failures when they arrive
>> on queues expected to be empty.
>>
>> Add a one second delay between repeated _send_traffic_check() calls in
>> rss_ctx tests to ensure such packets are processed before the next
>> traffic checks are performed.
> 
> Sprinklings sleeps should be last resort. Is there a way to wait for
> iperf to shut down cleanly, or wait for the socket to be closed fully?
> Like wait_port_listen() ?

The socket may end up in TIME_WAIT state, so waiting for it to be fully
closed can take ~2 mins. We could opt for a single sleep during
GenerateTraffic.stop(), but that would also affect tests that aren't
impacted by late packets. WDYT?

Re: [PATCH net] selftests: drv-net: rss_ctx: Add short delay between per-context traffic checks

[Jakub Kicinski]

On Wed, 9 Jul 2025 20:10:14 +0300 Nimrod Oren wrote:
> On 02/07/2025 3:23, Jakub Kicinski wrote:
> > On Sun, 29 Jun 2025 14:18:12 +0300 Nimrod Oren wrote:  
> >> A few packets may still be sent and received during the termination of
> >> the iperf processes. These late packets cause failures when they arrive
> >> on queues expected to be empty.
> >>
> >> Add a one second delay between repeated _send_traffic_check() calls in
> >> rss_ctx tests to ensure such packets are processed before the next
> >> traffic checks are performed.  
> > 
> > Sprinklings sleeps should be last resort. Is there a way to wait for
> > iperf to shut down cleanly, or wait for the socket to be closed fully?
> > Like wait_port_listen() ?  
> 
> The socket may end up in TIME_WAIT state, so waiting for it to be fully
> closed can take ~2 mins.

TIME_WAIT is as good as CLOSED for our purposes. Once we got a FIN
the chances of more traffic should be minuscule.