[PATCH v1] dts: resolve variance in performance reporting

[PATCH v1] dts: resolve variance in performance reporting

[Andrew Bailey]

Currently, the single core forward performance test suite reports
variable results with a standard deviation of ~10Mpps. Currently,
traffic is measured over 30 seconds a single time and the stats are
reported from this metric. This commit reduces the time traffic is sent
to only 5 seconds but runs numerous times to take an average and limit
variance to a standard deviation of ~1Mpps.

Bugzilla ID: 1969
Fixes: d77d7f04f24c ("dts: add single-core performance test suite")

Signed-off-by: Andrew Bailey <abailey@iol.unh.edu>
---
 .../TestSuite_single_core_forward_perf.py     | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/dts/tests/TestSuite_single_core_forward_perf.py b/dts/tests/TestSuite_single_core_forward_perf.py
index acdf8ae2f6..47cb592692 100644
--- a/dts/tests/TestSuite_single_core_forward_perf.py
+++ b/dts/tests/TestSuite_single_core_forward_perf.py
@@ -53,12 +53,13 @@ def set_up_suite(self):
         self.test_parameters = self.config.test_parameters
         self.delta_tolerance = self.config.delta_tolerance
 
-    def _transmit(self, testpmd: TestPmd, frame_size: int) -> float:
+    def _transmit(self, testpmd: TestPmd, frame_size: int, repetitions: int = 1) -> float:
         """Create a testpmd session with every rule in the given list, verify jump behavior.
 
         Args:
             testpmd: The testpmd shell to use for forwarding packets.
             frame_size: The size of the frame to transmit.
+            repetitions: The number of times to rerun the transmission.
 
         Returns:
             The MPPS (millions of packets per second) forwarded by the SUT.
@@ -71,13 +72,13 @@ def _transmit(self, testpmd: TestPmd, frame_size: int) -> float:
         )
 
         testpmd.start()
+        rx_avg: float = 0
 
-        # Transmit for 30 seconds.
-        stats = assess_performance_by_packet(packet=packet, duration=30)
-
-        rx_mpps = stats.rx_pps / 1_000_000
-
-        return rx_mpps
+        for _ in range(repetitions):
+            # Transmit for 5 seconds.
+            stats = assess_performance_by_packet(packet=packet, duration=5)
+            rx_avg += stats.rx_pps
+        return rx_avg / (repetitions * 1_000_000)
 
     def _produce_stats_table(self, test_parameters: list[dict[str, int | float]]) -> None:
         """Display performance results in table format and write to structured JSON file.
@@ -131,7 +132,9 @@ def single_core_forward_perf(self) -> None:
             with TestPmd(
                 **driver_specific_testpmd_args,
             ) as testpmd:
-                params["measured_mpps"] = round(self._transmit(testpmd, frame_size), 3)
+                params["measured_mpps"] = round(
+                    self._transmit(testpmd, frame_size, repetitions=5), 3
+                )
                 params["performance_delta"] = round(
                     (float(params["measured_mpps"]) - float(params["expected_mpps"]))
                     / float(params["expected_mpps"]),
-- 
2.54.0

Re: [PATCH v1] dts: resolve variance in performance reporting

[Luca Vizzarro]

Applied to next-dts with Claude's suggestion and a check to ensure that 
repetitions is always greater than 0.

Thanks!