DPDK-dev Archive on lore.kernel.org
 help / color / mirror / Atom feed
From: Luca Vizzarro <Luca.Vizzarro@arm.com>
To: Koushik Bhargav Nimoji <knimoji@iol.unh.edu>, patrickrobb1997@gmail.com
Cc: dev@dpdk.org, dladd@iol.unh.edu, abailey@iol.unh.edu,
	lylavoie@iol.unh.edu
Subject: Re: [PATCH v4 1/2] dts: add pvp multi paths vhost performance testsuite
Date: Tue, 10 Nov 2026 12:50:10 +0000	[thread overview]
Message-ID: <9e507b8c-b106-4177-8ce2-e09fc630e4d4@arm.com> (raw)
In-Reply-To: <20261103220218.902765-1-knimoji@iol.unh.edu>

On 03/11/2026 22:02, Koushik Bhargav Nimoji wrote:
> diff --git a/dts/api/testbed_model/virtual_device.py b/dts/api/testbed_model/virtual_device.py
> index 1a4794e695..8ac9a625f5 100644
> --- a/dts/api/testbed_model/virtual_device.py
> +++ b/dts/api/testbed_model/virtual_device.py
> @@ -6,6 +6,8 @@
>   Alongside support for physical hardware, DPDK can create various virtual devices.
>   """
>   
> +from api.context import get_ctx
> +
>   
>   class VirtualDevice:
>       """Base class for virtual devices used by DPDK.
> @@ -27,3 +29,9 @@ def __init__(self, name: str) -> None:
>       def __str__(self) -> str:
>           """This corresponds to the name used for DPDK devices."""
>           return self.name
> +
> +
> +def clean_up_vhost() -> None:
> +    """Deletes stale vhost-user Unix domain socket files."""
> +    ctx = get_ctx()
> +    ctx.sut_node.main_session.send_command("rm -rf /tmp/vhost-net*", privileged=True)
I welcome this change but send_command is still something that should be 
called by the relevant OSSession class. In theory main_session can be a 
session for a host of any OS, be Linux, FreeBSD, Windows etc. Therefore 
the changes should be OS-independent.

I've reviewed the usage better and the test suite shouldn't be able to 
arbitrarily create files out of arbitrary paths. We have the class 
Artifact to help abstract the remote filesystem away. On remote session 
this also takes care of cleanup.

Instead of introducing this change which has no real ownership in the 
framework and api packages, I'd suggest letting the test suite create 
the vhost-net using the Artifact class instead.
> diff --git a/dts/configurations/tests_config.example.yaml b/dts/configurations/tests_config.example.yaml
> index ce67385b0b..36a0e7c2f1 100644
> --- a/dts/configurations/tests_config.example.yaml
> +++ b/dts/configurations/tests_config.example.yaml
> @@ -18,6 +18,30 @@ hello_world:
>   #       num_descriptors: 1024
>   #       expected_mpps: 1.0
>   #   delta_tolerance: 0.05
> +# pvp_multi_paths_vhost_single_core_perf:
> +#   test_parameters: # Add frame size / descriptor count combinations as needed
> +#     - frame_size: 64
> +#       num_descriptors: 512
> +#       expected_mpps: 1.0 # Set millions of packets per second according to your devices expected throughput for this given frame size / descriptor count
> +#     - frame_size: 64
> +#       num_descriptors: 1024
> +#       expected_mpps: 1.0
> +#     - frame_size: 512
> +#       num_descriptors: 1024
> +#       expected_mpps: 1.0
> +#   delta_tolerance: 0.05
> +# pvp_multi_paths_virtio_single_core_perf:
> +#   test_parameters: # Add frame size / descriptor count combinations as needed
> +#     - frame_size: 64
> +#       num_descriptors: 512
> +#       expected_mpps: 1.0 # Set millions of packets per second according to your devices expected throughput for this given frame size / descriptor count
> +#     - frame_size: 64
> +#       num_descriptors: 1024
> +#       expected_mpps: 1.0
> +#     - frame_size: 512
> +#       num_descriptors: 1024
> +#       expected_mpps: 1.0
> +#   delta_tolerance: 0.05
>   # pmd_rss:
>   #   NUM_QUEUES: 4
>   #   ACTUAL_KEY_SIZE: 52
> diff --git a/dts/tests/TestSuite_pvp_multi_paths_vhost_single_core_perf.py b/dts/tests/TestSuite_pvp_multi_paths_vhost_single_core_perf.py
> new file mode 100644
> index 0000000000..269fad1782
> --- /dev/null
> +++ b/dts/tests/TestSuite_pvp_multi_paths_vhost_single_core_perf.py
> @@ -0,0 +1,523 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2026 University of New Hampshire
> +
> +"""Vhost-user single-core forwarding performance test suite.
> +
> +This suite measures the packet forwarding performance of DPDK's vhost-user PMD running on a
> +single CPU core in a Physical-Virtual-Physical (PVP) topology. Performance is evaluated across
> +multiple virtio-user backend options, including packed versus split ring formats, in-order packet
> +processing, mergeable buffers, and SIMD vectorization. Test cases are parameterized using
> +combinations of frame sizes, descriptor ring sizes, and target MPPS baselines specified in the
> +test configuration. High-speed traffic is injected from an external generator through SUT physical
> +interfaces, routed through Vhost and Virtio loops, and returned to the generator to measure
> +aggregate MPPS.
> +"""
> +
> +from typing import TypedDict, cast
> +
> +from scapy.layers.inet import IP
> +from scapy.layers.l2 import Ether
> +from scapy.packet import Raw
> +from typing_extensions import NotRequired
> +
> +from api.capabilities import (
> +    LinkTopology,
> +    requires_link_topology,
> +)
> +from api.packet import assess_performance_by_packet
> +from api.params.eal import VirtualDevice
> +from api.params.types import LogicalCoreList, RSSSetting, TestPmdParamsDict
> +from api.test import get_current_test_case_name, verify, write_performance_json
> +from api.test_suite import BaseConfig, TestSuite, perf_suite, perf_test
> +from api.testbed_model.virtual_device import clean_up_vhost
> +from api.testpmd import TestPmd
> +from api.testpmd.config import (
> +    Params,
> +    PortTopology,
> +    RXRingParams,
> +    SimpleForwardingModes,
> +    TXRingParams,
> +)
> +
> +
> +class TestParameter(TypedDict):
> +    """Performance test metrics."""
> +
> +    frame_size: int
> +    num_descriptors: int
> +    expected_mpps: float
> +    performance_delta: NotRequired[float | None]
> +    measured_mpps: NotRequired[float | None]
> +    passing: NotRequired[float | None]
I've noticed that this kind of configuration is ubiquitous across 
performance test suites. I'd propose to abstract the Config and this 
away as a generic PerfSuiteConfig and re-use it accordingly. In that 
case you can move away from using a TypedDict and use a Pydantic 
BaseConfig instead which will allow you to use more powerful constructs.

You won't need to use NotRequired – which I am assuming makes doing `| 
None` redundant, in that case. Moreover the fields should be documented 
appropriately similarly to what was done in the config package.> +
> +
> +class Config(BaseConfig):
> +    """Test configuration for pvp_multi_paths_vhost_single_core_perf test suite."""
> +
> +    test_parameters: list[TestParameter] = [
> +        {
> +            "frame_size": 64,
> +            "num_descriptors": 1024,
> +            "expected_mpps": 1.00,
> +            "performance_delta": None,
> +            "measured_mpps": None,
> +            "passing": None,
> +        },
> +        {
> +            "frame_size": 128,
> +            "num_descriptors": 1024,
> +            "expected_mpps": 1.00,
> +            "performance_delta": None,
> +            "measured_mpps": None,
> +            "passing": None,
> +        },
> +        {
> +            "frame_size": 256,
> +            "num_descriptors": 1024,
> +            "expected_mpps": 1.00,
> +            "performance_delta": None,
> +            "measured_mpps": None,
> +            "passing": None,
> +        },
> +        {
> +            "frame_size": 512,
> +            "num_descriptors": 1024,
> +            "expected_mpps": 1.00,
> +            "performance_delta": None,
> +            "measured_mpps": None,
> +            "passing": None,
> +        },
> +        {
> +            "frame_size": 1024,
> +            "num_descriptors": 1024,
> +            "expected_mpps": 1.00,
> +            "performance_delta": None,
> +            "measured_mpps": None,
> +            "passing": None,
> +        },
> +        {
> +            "frame_size": 1518,
> +            "num_descriptors": 1024,
> +            "expected_mpps": 1.00,
> +            "performance_delta": None,
> +            "measured_mpps": None,
> +            "passing": None,
> +        },
> +    ]
> +    delta_tolerance: float = 0.05
So all of the above could be a common Config class that can be re-used 
across perf test suites.> +
> +
> +@perf_suite
> +@requires_link_topology(LinkTopology.TWO_LINKS)
> +class TestPvpMultiPathsVhostSingleCorePerf(TestSuite):
> +    """pvp multi paths vhost single core performance test suite."""
> +
> +    config: Config
> +
> +    def set_up_test_case(self) -> None:
> +        """Delete stale vhost-user Unix domain socket files."""
> +        clean_up_vhost()
> +
> +    def _transmit(
> +        self, vhost: TestPmd, virtio: TestPmd, frame_size: int, repetitions: int = 1
> +    ) -> float:
> +        """Transmit packets using testpmd instances and compute average MPPS.
> +
> +        Args:
> +            vhost: the running vhost testpmd shell.
> +            virtio: the running virtio testpmd shell.
> +            frame_size: The size of the frame to transmit.
> +            repetitions: The number of times to rerun the transmission.
> +
> +        Returns:
> +            The MPPS (millions of packets per second) forwarded by the SUT.
> +        """
> +        assert repetitions > 0, "Invalid number of repetitions given, must be greater than 0."
> +        # Build packet with dummy values, and account for the 14B and 20B Ether and IP headers
> +        packet = (
> +            Ether(src="52:00:00:00:00:00")
> +            / IP(src="1.2.3.4", dst="192.18.1.0")
> +            / Raw(load="x" * (frame_size - 14 - 20))
> +        )
> +
> +        vhost.start()
> +        virtio.start()
> +        rx_avg = 0.0
> +
> +        for _ in range(repetitions):
> +            # Transmit for 5 seconds.
> +            stats = assess_performance_by_packet(packet=packet, duration=5)
> +            rx_avg += stats.rx_pps
> +        return rx_avg / (repetitions * 1_000_000)
> +
> +    def _create_and_transmit(
> +        self,
> +        ring_format: int,
> +        in_order: int,
> +        buffers: int,
> +        vectorized: int,
> +        extra_args: TestPmdParamsDict,
> +    ) -> None:
> +        """Create testpmd instances with specified params and send traffic.
> +
> +        Args:
> +            ring_format: selects the virtio queue descriptor layout.
> +            in_order: toggles in-order packet processing.
> +            buffers: enables mergeable rx buffers.
> +            vectorized: enables SIMD vectorized rx/tx data paths.
> +            extra_args: extra/generic TestPmd arguments.
> +        """
> +        sut_dpdk_driver = self._ctx.sut_node.config.ports[0].os_driver_for_dpdk
Here you should be able to do:

   vhost_iface = Artifact("sut", "vhost-net", save_local_copy=False)> + 
       vhost_user_vdev = 
VirtualDevice("net_vhost0,iface=/tmp/vhost-net,queues=1")
   VirtualDevice(f"net_vhost0,iface={vhost_iface.path},queues=1")

etc.
> +        virtio_user_vdev = VirtualDevice(
> +            "net_virtio_user0,"
> +            "mac=00:11:22:33:44:10,"
> +            "path=/tmp/vhost-net,"
> +            "queues=1,"
> +            f"packed_vq={ring_format},"
> +            f"mrg_rxbuf={buffers},"
> +            f"in_order={in_order},"
> +            f"vectorized={vectorized}"
> +        )
> +
> +        for params in self.config.test_parameters:
> +            frame_size = params.get("frame_size")
> +            num_descriptors = params.get("num_descriptors") or 1024
> +            expected_mpps = params.get("expected_mpps") or 1.00
Wouldn't using the default field more appropriate here? What if the 
param is 0? Thinking of:

   num_descriptors = params.get("num_descriptors", 1024)
   expected_mpps = params.get("expected_mpps", 1.00)> +
> +            default_args: TestPmdParamsDict = {
> +                "tx_ring": TXRingParams(descriptors=int(num_descriptors)),
> +                "rx_ring": RXRingParams(descriptors=int(num_descriptors)),
> +            }
> +
> +            if sut_dpdk_driver == "mlx5_core":
> +                default_args["burst"] = 64
> +                default_args["mbcache"] = 512
> +            elif sut_dpdk_driver == "i40e":
> +                default_args["rx_queues"] = 1
> +                default_args["tx_queues"] = 1
> +
> +            vhost_params: TestPmdParamsDict = {
> +                "prefix": "vhost",
> +                "allowed_ports": [],
> +                "memory_channels": 4,
> +                "port_topology": PortTopology.chained,
> +                "vdevs": [vhost_user_vdev],
> +                "lcore_list": LogicalCoreList([1, 2]),
> +                "nb_cores": 1,
> +                **default_args,
> +            }
> +
> +            virtio_params: TestPmdParamsDict = {
> +                "prefix": "virtio",
> +                "no_pci": True,
> +                "memory_channels": 4,
> +                "allowed_ports": [],
> +                "vdevs": [virtio_user_vdev],
> +                "lcore_list": LogicalCoreList([3, 4, 5]),
> +                "nb_cores": 2,
> +                **extra_args,
> +                **default_args,
> +            }
> +
> +            with (
> +                TestPmd(**vhost_params) as vhost,
> +                TestPmd(**virtio_params) as virtio,
> +            ):
> +                vhost.set_forward_mode(SimpleForwardingModes.mac)
> +                virtio.set_forward_mode(SimpleForwardingModes.io)
> +                vhost.set_portlist([0, 2, 1])
> +
> +                assert expected_mpps > 0, "Invalid expected mpps, must be greater than 0."

Is this assertion a check for a test suite bug or an actual testing 
detail? If the latter then verify should be used here.

> +
> +                measured_mpps = round(
> +                    self._transmit(vhost, virtio, int(frame_size), repetitions=5), 3
> +                )
> +
> +                performance_delta = round(
> +                    (float(measured_mpps) - float(expected_mpps)) / float(expected_mpps),
> +                    3,
> +                )
> +
> +                params["measured_mpps"] = measured_mpps
> +                params["performance_delta"] = performance_delta
> +                params["passing"] = float(performance_delta) >= -self.config.delta_tolerance
> +
> +        self._produce_stats_table(self.config.test_parameters)
> +        for params in self.config.test_parameters:
> +            verify(
> +                params["passing"] is True,
> +                f"""Packets forwarded is less than {(1 - self.config.delta_tolerance) * 100}%
> +                of the expected baseline.
> +                Measured MPPS = {params["measured_mpps"]}
> +                Expected MPPS = {params["expected_mpps"]}""",
> +            )

  parent reply	other threads:[~2026-11-10 12:50 UTC|newest]

Thread overview: 28+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-09-01 18:37 [RFC PATCH] dts: add pvp multi paths virtio performance testsuite Koushik Bhargav Nimoji
2026-09-03 20:11 ` [PATCH v1] " Koushik Bhargav Nimoji
2026-10-08 19:11 ` [PATCH v2 1/2] dts: add pvp multi paths vhost " Koushik Bhargav Nimoji
2026-10-08 19:11   ` [PATCH v2 2/2] dts: add pvp multi paths virtio " Koushik Bhargav Nimoji
2026-10-19 18:58     ` Andrew Bailey
2026-10-19 18:36   ` [PATCH v2 1/2] dts: add pvp multi paths vhost " Andrew Bailey
2026-10-20 17:30   ` [PATCH v3 " Koushik Bhargav Nimoji
2026-10-20 17:30     ` [PATCH v3 2/2] dts: add pvp multi paths virtio " Koushik Bhargav Nimoji
2026-10-29 17:01       ` Luca Vizzarro
2026-11-03 15:37         ` Koushik Bhargav Nimoji
2026-11-03 15:47           ` Luca Vizzarro
2026-10-29 16:53     ` [PATCH v3 1/2] dts: add pvp multi paths vhost " Luca Vizzarro
2026-11-03 20:28       ` Koushik Bhargav Nimoji
2026-11-03 22:02   ` [PATCH v4 " Koushik Bhargav Nimoji
2026-11-03 22:02     ` [PATCH v4 2/2] dts: add pvp multi paths virtio " Koushik Bhargav Nimoji
2026-11-10 12:51       ` Luca Vizzarro
2026-11-10 12:50     ` Luca Vizzarro [this message]
2026-11-10 15:40       ` [PATCH v4 1/2] dts: add pvp multi paths vhost " Koushik Bhargav Nimoji
2026-11-10 20:10   ` [PATCH v5 " Koushik Bhargav Nimoji
2026-11-10 20:10     ` [PATCH v5 2/2] dts: add pvp multi paths virtio " Koushik Bhargav Nimoji
2026-11-11 16:30     ` [PATCH v5 1/2] dts: add pvp multi paths vhost " Luca Vizzarro
2026-11-12 15:48       ` Koushik Bhargav Nimoji
2026-11-12 21:00   ` [PATCH v6 1/3] dts: create default performance test configuration Koushik Bhargav Nimoji
2026-11-12 21:00     ` [PATCH v6 2/3] dts: add pvp multi paths vhost performance testsuite Koushik Bhargav Nimoji
2026-11-12 21:00     ` [PATCH v6 3/3] dts: add pvp multi paths virtio " Koushik Bhargav Nimoji
2026-11-19 18:41     ` [PATCH v6 1/3] dts: create default performance test configuration Luca Vizzarro
2026-11-20 12:02       ` Thomas Monjalon
2026-10-20 18:20 ` [RFC PATCH] dts: add pvp multi paths virtio performance testsuite Koushik Bhargav Nimoji

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=9e507b8c-b106-4177-8ce2-e09fc630e4d4@arm.com \
    --to=luca.vizzarro@arm.com \
    --cc=abailey@iol.unh.edu \
    --cc=dev@dpdk.org \
    --cc=dladd@iol.unh.edu \
    --cc=knimoji@iol.unh.edu \
    --cc=lylavoie@iol.unh.edu \
    --cc=patrickrobb1997@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox