[RFC 0/1] virtio-net: add support for SR-IOV emulation

[RFC 0/1] virtio-net: add support for SR-IOV emulation

[Yui Washizu]

This patch series is the first step towards enabling
hardware offloading of the L2 packet switching feature on virtio-net device to host machine.
We are considering that this hardware offloading enables
the use of high-performance networks in virtual infrastructures,
such as container infrastructures on VMs.

To enable L2 packet switching by SR-IOV VFs, we are considering the following:
- making the guest recognize virtio-net devices as SR-IOV PF devices
  (archived with this patch series)
- allowing virtio-net devices to connect SR-IOV VFs to the backend networks,
  leaving the L2 packet switching feature to the management layer like libvirt
  - This makes hardware offloading of L2 packet switching possible.
    For example, when using vDPA devices, it allows the guest
    to utilize SR-IOV NIC embedded switch of hosts.

This patch series aims to enable SR-IOV emulation on virtio-net devices.
With this series, the guest can identify the virtio-net device as an SR-IOV PF device.
The newly added property 'sriov_max_vfs' allows us to enable the SR-IOV feature
on the virtio-net device.
Currently, we are unable to specify the properties of a VF created from the guest.
The properties are set to their default values.
In the future, we plan to allow users to set the properties.

qemu-system-x86_64 --device virtio-net,sriov_max_vfs=<num>
# when 'sriov_max_vfs' is present, the SR-IOV feature will be automatically enabled
# <num> means the max number of VF on guest

Example commands to create VFs in virtio-net device from the guest:

guest% readlink -f /sys/class/net/eth1/device
 /sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/virtio1
guest% echo "2" > /sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/sriov_numvfs
guest% ip link show
 eth0: ....
 eth1: ....
 eth2: .... #virtual VF created
 eth3: .... #virtual VF created
 
Please note that communication between VF and PF/VF is not possible by this patch series itself.

Yui Washizu (1):
  virtio-pci: add SR-IOV capability

 hw/pci/msix.c                  |  8 +++--
 hw/pci/pci.c                   |  4 +++
 hw/virtio/virtio-pci.c         | 62 ++++++++++++++++++++++++++++++----
 include/hw/virtio/virtio-pci.h |  1 +
 4 files changed, 66 insertions(+), 9 deletions(-)

-- 
2.39.3

[RFC 1/1] virtio-pci: add SR-IOV capability

[Yui Washizu]

This enables SR-IOV emulation on virtio-pci devices by adding SR-IOV capability
It also introduces a newly added property 'sriov_max_vfs'
to enable or disable the SR-IOV feature on the virtio-pci device in guest,
as well as to specify the maximum number of VFs that can be created in the guest.
Currently only virtio-net is supported.
Also, the vendor ID and device ID remain the same for both the PF and VF,
enabling existing guest PF drivers to be used in the VF without any modifications.

Signed-off-by: Yui Washizu <yui.washidu@gmail.com>
---
 hw/pci/msix.c                  |  8 +++--
 hw/pci/pci.c                   |  4 +++
 hw/virtio/virtio-pci.c         | 62 ++++++++++++++++++++++++++++++----
 include/hw/virtio/virtio-pci.h |  1 +
 4 files changed, 66 insertions(+), 9 deletions(-)

diff --git a/hw/pci/msix.c b/hw/pci/msix.c
index ab8869d9d0..3b94ce389f 100644
--- a/hw/pci/msix.c
+++ b/hw/pci/msix.c
@@ -421,8 +421,12 @@ int msix_init_exclusive_bar(PCIDevice *dev, unsigned short nentries,
         return ret;
     }
 
-    pci_register_bar(dev, bar_nr, PCI_BASE_ADDRESS_SPACE_MEMORY,
-                     &dev->msix_exclusive_bar);
+    if (pci_is_vf(dev)) {
+        pcie_sriov_vf_register_bar(dev, bar_nr, &dev->msix_exclusive_bar);
+    } else {
+        pci_register_bar(dev, bar_nr, PCI_BASE_ADDRESS_SPACE_MEMORY,
+                         &dev->msix_exclusive_bar);
+    }
 
     return 0;
 }
diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index e2eb4c3b4a..cbd50b38ea 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -2325,6 +2325,10 @@ static void pci_add_option_rom(PCIDevice *pdev, bool is_default_rom,
         return;
     }
 
+    if (pci_is_vf(pdev)) {
+        return;
+    }
+
     if (!pdev->rom_bar) {
         /*
          * Load rom via fw_cfg instead of creating a rom bar,
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index edbc0daa18..2315c2647a 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -49,6 +49,8 @@
  * configuration space */
 #define VIRTIO_PCI_CONFIG_SIZE(dev)     VIRTIO_PCI_CONFIG_OFF(msix_enabled(dev))
 
+#define VIRTIO_MAX_VFS 127
+
 static void virtio_pci_bus_new(VirtioBusState *bus, size_t bus_size,
                                VirtIOPCIProxy *dev);
 static void virtio_pci_reset(DeviceState *qdev);
@@ -1907,6 +1909,11 @@ static void virtio_pci_pre_plugged(DeviceState *d, Error **errp)
 
     if (virtio_pci_modern(proxy)) {
         virtio_add_feature(&vdev->host_features, VIRTIO_F_VERSION_1);
+        if (proxy->sriov_max_vfs) {
+            virtio_add_feature(&vdev->host_features, VIRTIO_F_SR_IOV);
+        }
+    } else if (proxy->sriov_max_vfs) {
+        error_setg(errp, "VirtIO PCI modern is required for the use of SR-IOV");
     }
 
     virtio_add_feature(&vdev->host_features, VIRTIO_F_BAD_FEATURE);
@@ -2015,22 +2022,62 @@ static void virtio_pci_device_plugged(DeviceState *d, Error **errp)
         virtio_pci_modern_mem_region_map(proxy, &proxy->device, &cap);
         virtio_pci_modern_mem_region_map(proxy, &proxy->notify, &notify.cap);
 
+        if (!pci_is_vf(&proxy->pci_dev) && proxy->sriov_max_vfs) {
+            if (virtio_bus_get_vdev_id(bus) != VIRTIO_ID_NET) {
+                error_setg(errp, "sriov_max_vfs prop is not supported by %s",
+                           proxy->pci_dev.name);
+                return;
+            }
+            if (proxy->sriov_max_vfs > VIRTIO_MAX_VFS) {
+                error_setg(errp, "sriov_max_vfs must be between 0 and %d",
+                           VIRTIO_MAX_VFS);
+                return;
+            }
+
+            pcie_sriov_pf_init(&proxy->pci_dev, PCI_CONFIG_SPACE_SIZE,
+                               proxy->pci_dev.name,
+                               PCI_DEVICE_ID_VIRTIO_10_BASE
+                               + virtio_bus_get_vdev_id(bus),
+                               proxy->sriov_max_vfs, proxy->sriov_max_vfs, 1, 1);
+            if (proxy->flags & VIRTIO_PCI_FLAG_MODERN_PIO_NOTIFY) {
+                pcie_sriov_pf_init_vf_bar(&proxy->pci_dev, proxy->modern_io_bar_idx,
+                                          PCI_BASE_ADDRESS_SPACE_IO, 4);
+            }
+            if (proxy->nvectors) {
+                pcie_sriov_pf_init_vf_bar(&proxy->pci_dev, proxy->msix_bar_idx,
+                                          PCI_BASE_ADDRESS_SPACE_MEMORY, 4 * 1024);
+            }
+            pcie_sriov_pf_init_vf_bar(&proxy->pci_dev, proxy->modern_mem_bar_idx,
+                                      PCI_BASE_ADDRESS_SPACE_MEMORY |
+                                      PCI_BASE_ADDRESS_MEM_PREFETCH |
+                                      PCI_BASE_ADDRESS_MEM_TYPE_64,
+                                      16 * 1024);
+        }
+
         if (modern_pio) {
             memory_region_init(&proxy->io_bar, OBJECT(proxy),
                                "virtio-pci-io", 0x4);
 
-            pci_register_bar(&proxy->pci_dev, proxy->modern_io_bar_idx,
-                             PCI_BASE_ADDRESS_SPACE_IO, &proxy->io_bar);
+            if (pci_is_vf(&proxy->pci_dev))
+                pcie_sriov_vf_register_bar(&proxy->pci_dev, proxy->modern_io_bar_idx,
+                                           &proxy->io_bar);
+            else
+                pci_register_bar(&proxy->pci_dev, proxy->modern_io_bar_idx,
+                                 PCI_BASE_ADDRESS_SPACE_IO, &proxy->io_bar);
 
             virtio_pci_modern_io_region_map(proxy, &proxy->notify_pio,
                                             &notify_pio.cap);
         }
 
-        pci_register_bar(&proxy->pci_dev, proxy->modern_mem_bar_idx,
-                         PCI_BASE_ADDRESS_SPACE_MEMORY |
-                         PCI_BASE_ADDRESS_MEM_PREFETCH |
-                         PCI_BASE_ADDRESS_MEM_TYPE_64,
-                         &proxy->modern_bar);
+        if (pci_is_vf(&proxy->pci_dev))
+            pcie_sriov_vf_register_bar(&proxy->pci_dev, proxy->modern_mem_bar_idx,
+                                       &proxy->modern_bar);
+        else
+            pci_register_bar(&proxy->pci_dev, proxy->modern_mem_bar_idx,
+                             PCI_BASE_ADDRESS_SPACE_MEMORY |
+                             PCI_BASE_ADDRESS_MEM_PREFETCH |
+                             PCI_BASE_ADDRESS_MEM_TYPE_64,
+                             &proxy->modern_bar);
 
         proxy->config_cap = virtio_pci_add_mem_cap(proxy, &cfg.cap);
         cfg_mask = (void *)(proxy->pci_dev.wmask + proxy->config_cap);
@@ -2298,6 +2345,7 @@ static Property virtio_pci_properties[] = {
                     VIRTIO_PCI_FLAG_INIT_FLR_BIT, true),
     DEFINE_PROP_BIT("aer", VirtIOPCIProxy, flags,
                     VIRTIO_PCI_FLAG_AER_BIT, false),
+    DEFINE_PROP_UINT16("sriov_max_vfs", VirtIOPCIProxy, sriov_max_vfs, 0),
     DEFINE_PROP_END_OF_LIST(),
 };
 
diff --git a/include/hw/virtio/virtio-pci.h b/include/hw/virtio/virtio-pci.h
index ab2051b64b..05ab6ddc3f 100644
--- a/include/hw/virtio/virtio-pci.h
+++ b/include/hw/virtio/virtio-pci.h
@@ -150,6 +150,7 @@ struct VirtIOPCIProxy {
     uint32_t flags;
     bool disable_modern;
     bool ignore_backend_features;
+    uint16_t sriov_max_vfs;
     OnOffAuto disable_legacy;
     /* Transitional device id */
     uint16_t trans_devid;
-- 
2.39.3

Re: [RFC 1/1] virtio-pci: add SR-IOV capability

[Akihiko Odaki]

On 2023/07/19 10:56, Yui Washizu wrote:
> This enables SR-IOV emulation on virtio-pci devices by adding SR-IOV capability
> It also introduces a newly added property 'sriov_max_vfs'
> to enable or disable the SR-IOV feature on the virtio-pci device in guest,
> as well as to specify the maximum number of VFs that can be created in the guest.
> Currently only virtio-net is supported.
> Also, the vendor ID and device ID remain the same for both the PF and VF,
> enabling existing guest PF drivers to be used in the VF without any modifications.
> 
> Signed-off-by: Yui Washizu <yui.washidu@gmail.com>
> ---
>   hw/pci/msix.c                  |  8 +++--
>   hw/pci/pci.c                   |  4 +++
>   hw/virtio/virtio-pci.c         | 62 ++++++++++++++++++++++++++++++----
>   include/hw/virtio/virtio-pci.h |  1 +
>   4 files changed, 66 insertions(+), 9 deletions(-)
> 
> diff --git a/hw/pci/msix.c b/hw/pci/msix.c
> index ab8869d9d0..3b94ce389f 100644
> --- a/hw/pci/msix.c
> +++ b/hw/pci/msix.c
> @@ -421,8 +421,12 @@ int msix_init_exclusive_bar(PCIDevice *dev, unsigned short nentries,
>           return ret;
>       }
>   
> -    pci_register_bar(dev, bar_nr, PCI_BASE_ADDRESS_SPACE_MEMORY,
> -                     &dev->msix_exclusive_bar);
> +    if (pci_is_vf(dev)) {
> +        pcie_sriov_vf_register_bar(dev, bar_nr, &dev->msix_exclusive_bar);
> +    } else {
> +        pci_register_bar(dev, bar_nr, PCI_BASE_ADDRESS_SPACE_MEMORY,
> +                         &dev->msix_exclusive_bar);
> +    }
>   
>       return 0;
>   }
> diff --git a/hw/pci/pci.c b/hw/pci/pci.c
> index e2eb4c3b4a..cbd50b38ea 100644
> --- a/hw/pci/pci.c
> +++ b/hw/pci/pci.c
> @@ -2325,6 +2325,10 @@ static void pci_add_option_rom(PCIDevice *pdev, bool is_default_rom,
>           return;
>       }
>   
> +    if (pci_is_vf(pdev)) {
> +        return;
> +    }
> +
>       if (!pdev->rom_bar) {
>           /*
>            * Load rom via fw_cfg instead of creating a rom bar,
> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
> index edbc0daa18..2315c2647a 100644
> --- a/hw/virtio/virtio-pci.c
> +++ b/hw/virtio/virtio-pci.c
> @@ -49,6 +49,8 @@
>    * configuration space */
>   #define VIRTIO_PCI_CONFIG_SIZE(dev)     VIRTIO_PCI_CONFIG_OFF(msix_enabled(dev))
>   
> +#define VIRTIO_MAX_VFS 127
> +
>   static void virtio_pci_bus_new(VirtioBusState *bus, size_t bus_size,
>                                  VirtIOPCIProxy *dev);
>   static void virtio_pci_reset(DeviceState *qdev);
> @@ -1907,6 +1909,11 @@ static void virtio_pci_pre_plugged(DeviceState *d, Error **errp)
>   
>       if (virtio_pci_modern(proxy)) {
>           virtio_add_feature(&vdev->host_features, VIRTIO_F_VERSION_1);
> +        if (proxy->sriov_max_vfs) {
> +            virtio_add_feature(&vdev->host_features, VIRTIO_F_SR_IOV);
> +        }
> +    } else if (proxy->sriov_max_vfs) {
> +        error_setg(errp, "VirtIO PCI modern is required for the use of SR-IOV");

Missing: return;

>       }
>   
>       virtio_add_feature(&vdev->host_features, VIRTIO_F_BAD_FEATURE);
> @@ -2015,22 +2022,62 @@ static void virtio_pci_device_plugged(DeviceState *d, Error **errp)
>           virtio_pci_modern_mem_region_map(proxy, &proxy->device, &cap);
>           virtio_pci_modern_mem_region_map(proxy, &proxy->notify, &notify.cap);
>   
> +        if (!pci_is_vf(&proxy->pci_dev) && proxy->sriov_max_vfs) {
> +            if (virtio_bus_get_vdev_id(bus) != VIRTIO_ID_NET) {
> +                error_setg(errp, "sriov_max_vfs prop is not supported by %s",
> +                           proxy->pci_dev.name);
> +                return;
> +            }
> +            if (proxy->sriov_max_vfs > VIRTIO_MAX_VFS) {
> +                error_setg(errp, "sriov_max_vfs must be between 0 and %d",
> +                           VIRTIO_MAX_VFS);
> +                return;
> +            }
> +
> +            pcie_sriov_pf_init(&proxy->pci_dev, PCI_CONFIG_SPACE_SIZE,
> +                               proxy->pci_dev.name,
> +                               PCI_DEVICE_ID_VIRTIO_10_BASE
> +                               + virtio_bus_get_vdev_id(bus),
> +                               proxy->sriov_max_vfs, proxy->sriov_max_vfs, 1, 1);
> +            if (proxy->flags & VIRTIO_PCI_FLAG_MODERN_PIO_NOTIFY) {
> +                pcie_sriov_pf_init_vf_bar(&proxy->pci_dev, proxy->modern_io_bar_idx,
> +                                          PCI_BASE_ADDRESS_SPACE_IO, 4);
> +            }
> +            if (proxy->nvectors) {
> +                pcie_sriov_pf_init_vf_bar(&proxy->pci_dev, proxy->msix_bar_idx,
> +                                          PCI_BASE_ADDRESS_SPACE_MEMORY, 4 * 1024);
> +            }
> +            pcie_sriov_pf_init_vf_bar(&proxy->pci_dev, proxy->modern_mem_bar_idx,
> +                                      PCI_BASE_ADDRESS_SPACE_MEMORY |
> +                                      PCI_BASE_ADDRESS_MEM_PREFETCH |
> +                                      PCI_BASE_ADDRESS_MEM_TYPE_64,
> +                                      16 * 1024);
> +        }
> +
>           if (modern_pio) {
>               memory_region_init(&proxy->io_bar, OBJECT(proxy),
>                                  "virtio-pci-io", 0x4);
>   
> -            pci_register_bar(&proxy->pci_dev, proxy->modern_io_bar_idx,
> -                             PCI_BASE_ADDRESS_SPACE_IO, &proxy->io_bar);
> +            if (pci_is_vf(&proxy->pci_dev))
> +                pcie_sriov_vf_register_bar(&proxy->pci_dev, proxy->modern_io_bar_idx,
> +                                           &proxy->io_bar);
> +            else
> +                pci_register_bar(&proxy->pci_dev, proxy->modern_io_bar_idx,
> +                                 PCI_BASE_ADDRESS_SPACE_IO, &proxy->io_bar);
>   
>               virtio_pci_modern_io_region_map(proxy, &proxy->notify_pio,
>                                               &notify_pio.cap);
>           }
>   
> -        pci_register_bar(&proxy->pci_dev, proxy->modern_mem_bar_idx,
> -                         PCI_BASE_ADDRESS_SPACE_MEMORY |
> -                         PCI_BASE_ADDRESS_MEM_PREFETCH |
> -                         PCI_BASE_ADDRESS_MEM_TYPE_64,
> -                         &proxy->modern_bar);
> +        if (pci_is_vf(&proxy->pci_dev))
> +            pcie_sriov_vf_register_bar(&proxy->pci_dev, proxy->modern_mem_bar_idx,
> +                                       &proxy->modern_bar);
> +        else
> +            pci_register_bar(&proxy->pci_dev, proxy->modern_mem_bar_idx,
> +                             PCI_BASE_ADDRESS_SPACE_MEMORY |
> +                             PCI_BASE_ADDRESS_MEM_PREFETCH |
> +                             PCI_BASE_ADDRESS_MEM_TYPE_64,
> +                             &proxy->modern_bar);
>   
>           proxy->config_cap = virtio_pci_add_mem_cap(proxy, &cfg.cap);
>           cfg_mask = (void *)(proxy->pci_dev.wmask + proxy->config_cap);
> @@ -2298,6 +2345,7 @@ static Property virtio_pci_properties[] = {
>                       VIRTIO_PCI_FLAG_INIT_FLR_BIT, true),
>       DEFINE_PROP_BIT("aer", VirtIOPCIProxy, flags,
>                       VIRTIO_PCI_FLAG_AER_BIT, false),
> +    DEFINE_PROP_UINT16("sriov_max_vfs", VirtIOPCIProxy, sriov_max_vfs, 0),
>       DEFINE_PROP_END_OF_LIST(),
>   };
>   
> diff --git a/include/hw/virtio/virtio-pci.h b/include/hw/virtio/virtio-pci.h
> index ab2051b64b..05ab6ddc3f 100644
> --- a/include/hw/virtio/virtio-pci.h
> +++ b/include/hw/virtio/virtio-pci.h
> @@ -150,6 +150,7 @@ struct VirtIOPCIProxy {
>       uint32_t flags;
>       bool disable_modern;
>       bool ignore_backend_features;
> +    uint16_t sriov_max_vfs;
>       OnOffAuto disable_legacy;
>       /* Transitional device id */
>       uint16_t trans_devid;

Re: [RFC 1/1] virtio-pci: add SR-IOV capability

[Jason Wang]

On Wed, Jul 19, 2023 at 9:59 AM Yui Washizu <yui.washidu@gmail.com> wrote:
>
> This enables SR-IOV emulation on virtio-pci devices by adding SR-IOV capability
> It also introduces a newly added property 'sriov_max_vfs'
> to enable or disable the SR-IOV feature on the virtio-pci device in guest,
> as well as to specify the maximum number of VFs that can be created in the guest.
> Currently only virtio-net is supported.
> Also, the vendor ID and device ID remain the same for both the PF and VF,
> enabling existing guest PF drivers to be used in the VF without any modifications.
>
> Signed-off-by: Yui Washizu <yui.washidu@gmail.com>
> ---
>  hw/pci/msix.c                  |  8 +++--
>  hw/pci/pci.c                   |  4 +++
>  hw/virtio/virtio-pci.c         | 62 ++++++++++++++++++++++++++++++----
>  include/hw/virtio/virtio-pci.h |  1 +
>  4 files changed, 66 insertions(+), 9 deletions(-)
>
> diff --git a/hw/pci/msix.c b/hw/pci/msix.c
> index ab8869d9d0..3b94ce389f 100644
> --- a/hw/pci/msix.c
> +++ b/hw/pci/msix.c
> @@ -421,8 +421,12 @@ int msix_init_exclusive_bar(PCIDevice *dev, unsigned short nentries,
>          return ret;
>      }
>
> -    pci_register_bar(dev, bar_nr, PCI_BASE_ADDRESS_SPACE_MEMORY,
> -                     &dev->msix_exclusive_bar);
> +    if (pci_is_vf(dev)) {
> +        pcie_sriov_vf_register_bar(dev, bar_nr, &dev->msix_exclusive_bar);
> +    } else {
> +        pci_register_bar(dev, bar_nr, PCI_BASE_ADDRESS_SPACE_MEMORY,
> +                         &dev->msix_exclusive_bar);
> +    }
>
>      return 0;
>  }
> diff --git a/hw/pci/pci.c b/hw/pci/pci.c
> index e2eb4c3b4a..cbd50b38ea 100644
> --- a/hw/pci/pci.c
> +++ b/hw/pci/pci.c
> @@ -2325,6 +2325,10 @@ static void pci_add_option_rom(PCIDevice *pdev, bool is_default_rom,
>          return;
>      }
>
> +    if (pci_is_vf(pdev)) {
> +        return;
> +    }
> +
>      if (!pdev->rom_bar) {
>          /*
>           * Load rom via fw_cfg instead of creating a rom bar,
> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
> index edbc0daa18..2315c2647a 100644
> --- a/hw/virtio/virtio-pci.c
> +++ b/hw/virtio/virtio-pci.c
> @@ -49,6 +49,8 @@
>   * configuration space */
>  #define VIRTIO_PCI_CONFIG_SIZE(dev)     VIRTIO_PCI_CONFIG_OFF(msix_enabled(dev))
>
> +#define VIRTIO_MAX_VFS 127
> +
>  static void virtio_pci_bus_new(VirtioBusState *bus, size_t bus_size,
>                                 VirtIOPCIProxy *dev);
>  static void virtio_pci_reset(DeviceState *qdev);
> @@ -1907,6 +1909,11 @@ static void virtio_pci_pre_plugged(DeviceState *d, Error **errp)
>
>      if (virtio_pci_modern(proxy)) {
>          virtio_add_feature(&vdev->host_features, VIRTIO_F_VERSION_1);
> +        if (proxy->sriov_max_vfs) {
> +            virtio_add_feature(&vdev->host_features, VIRTIO_F_SR_IOV);
> +        }
> +    } else if (proxy->sriov_max_vfs) {
> +        error_setg(errp, "VirtIO PCI modern is required for the use of SR-IOV");
>      }
>
>      virtio_add_feature(&vdev->host_features, VIRTIO_F_BAD_FEATURE);
> @@ -2015,22 +2022,62 @@ static void virtio_pci_device_plugged(DeviceState *d, Error **errp)
>          virtio_pci_modern_mem_region_map(proxy, &proxy->device, &cap);
>          virtio_pci_modern_mem_region_map(proxy, &proxy->notify, &notify.cap);
>
> +        if (!pci_is_vf(&proxy->pci_dev) && proxy->sriov_max_vfs) {
> +            if (virtio_bus_get_vdev_id(bus) != VIRTIO_ID_NET) {
> +                error_setg(errp, "sriov_max_vfs prop is not supported by %s",
> +                           proxy->pci_dev.name);
> +                return;
> +            }
> +            if (proxy->sriov_max_vfs > VIRTIO_MAX_VFS) {
> +                error_setg(errp, "sriov_max_vfs must be between 0 and %d",
> +                           VIRTIO_MAX_VFS);
> +                return;
> +            }
> +
> +            pcie_sriov_pf_init(&proxy->pci_dev, PCI_CONFIG_SPACE_SIZE,
> +                               proxy->pci_dev.name,
> +                               PCI_DEVICE_ID_VIRTIO_10_BASE
> +                               + virtio_bus_get_vdev_id(bus),
> +                               proxy->sriov_max_vfs, proxy->sriov_max_vfs, 1, 1);
> +            if (proxy->flags & VIRTIO_PCI_FLAG_MODERN_PIO_NOTIFY) {
> +                pcie_sriov_pf_init_vf_bar(&proxy->pci_dev, proxy->modern_io_bar_idx,
> +                                          PCI_BASE_ADDRESS_SPACE_IO, 4);
> +            }
> +            if (proxy->nvectors) {
> +                pcie_sriov_pf_init_vf_bar(&proxy->pci_dev, proxy->msix_bar_idx,
> +                                          PCI_BASE_ADDRESS_SPACE_MEMORY, 4 * 1024);
> +            }
> +            pcie_sriov_pf_init_vf_bar(&proxy->pci_dev, proxy->modern_mem_bar_idx,
> +                                      PCI_BASE_ADDRESS_SPACE_MEMORY |
> +                                      PCI_BASE_ADDRESS_MEM_PREFETCH |
> +                                      PCI_BASE_ADDRESS_MEM_TYPE_64,
> +                                      16 * 1024);
> +        }
> +
>          if (modern_pio) {
>              memory_region_init(&proxy->io_bar, OBJECT(proxy),
>                                 "virtio-pci-io", 0x4);
>
> -            pci_register_bar(&proxy->pci_dev, proxy->modern_io_bar_idx,
> -                             PCI_BASE_ADDRESS_SPACE_IO, &proxy->io_bar);
> +            if (pci_is_vf(&proxy->pci_dev))
> +                pcie_sriov_vf_register_bar(&proxy->pci_dev, proxy->modern_io_bar_idx,
> +                                           &proxy->io_bar);

We probably don't need another memory bar for notification.

Thanks

> +            else
> +                pci_register_bar(&proxy->pci_dev, proxy->modern_io_bar_idx,
> +                                 PCI_BASE_ADDRESS_SPACE_IO, &proxy->io_bar);
>
>              virtio_pci_modern_io_region_map(proxy, &proxy->notify_pio,
>                                              &notify_pio.cap);
>          }
>
> -        pci_register_bar(&proxy->pci_dev, proxy->modern_mem_bar_idx,
> -                         PCI_BASE_ADDRESS_SPACE_MEMORY |
> -                         PCI_BASE_ADDRESS_MEM_PREFETCH |
> -                         PCI_BASE_ADDRESS_MEM_TYPE_64,
> -                         &proxy->modern_bar);
> +        if (pci_is_vf(&proxy->pci_dev))
> +            pcie_sriov_vf_register_bar(&proxy->pci_dev, proxy->modern_mem_bar_idx,
> +                                       &proxy->modern_bar);
> +        else
> +            pci_register_bar(&proxy->pci_dev, proxy->modern_mem_bar_idx,
> +                             PCI_BASE_ADDRESS_SPACE_MEMORY |
> +                             PCI_BASE_ADDRESS_MEM_PREFETCH |
> +                             PCI_BASE_ADDRESS_MEM_TYPE_64,
> +                             &proxy->modern_bar);
>
>          proxy->config_cap = virtio_pci_add_mem_cap(proxy, &cfg.cap);
>          cfg_mask = (void *)(proxy->pci_dev.wmask + proxy->config_cap);
> @@ -2298,6 +2345,7 @@ static Property virtio_pci_properties[] = {
>                      VIRTIO_PCI_FLAG_INIT_FLR_BIT, true),
>      DEFINE_PROP_BIT("aer", VirtIOPCIProxy, flags,
>                      VIRTIO_PCI_FLAG_AER_BIT, false),
> +    DEFINE_PROP_UINT16("sriov_max_vfs", VirtIOPCIProxy, sriov_max_vfs, 0),
>      DEFINE_PROP_END_OF_LIST(),
>  };
>
> diff --git a/include/hw/virtio/virtio-pci.h b/include/hw/virtio/virtio-pci.h
> index ab2051b64b..05ab6ddc3f 100644
> --- a/include/hw/virtio/virtio-pci.h
> +++ b/include/hw/virtio/virtio-pci.h
> @@ -150,6 +150,7 @@ struct VirtIOPCIProxy {
>      uint32_t flags;
>      bool disable_modern;
>      bool ignore_backend_features;
> +    uint16_t sriov_max_vfs;
>      OnOffAuto disable_legacy;
>      /* Transitional device id */
>      uint16_t trans_devid;
> --
> 2.39.3
>

Re: [RFC 0/1] virtio-net: add support for SR-IOV emulation

[Jason Wang]

On Wed, Jul 19, 2023 at 9:59 AM Yui Washizu <yui.washidu@gmail.com> wrote:
>
> This patch series is the first step towards enabling
> hardware offloading of the L2 packet switching feature on virtio-net device to host machine.
> We are considering that this hardware offloading enables
> the use of high-performance networks in virtual infrastructures,
> such as container infrastructures on VMs.
>
> To enable L2 packet switching by SR-IOV VFs, we are considering the following:
> - making the guest recognize virtio-net devices as SR-IOV PF devices
>   (archived with this patch series)
> - allowing virtio-net devices to connect SR-IOV VFs to the backend networks,
>   leaving the L2 packet switching feature to the management layer like libvirt

Could you please show the qemu command line you want to propose here?

>   - This makes hardware offloading of L2 packet switching possible.
>     For example, when using vDPA devices, it allows the guest
>     to utilize SR-IOV NIC embedded switch of hosts.

This would be interesting.

Thanks

>
> This patch series aims to enable SR-IOV emulation on virtio-net devices.
> With this series, the guest can identify the virtio-net device as an SR-IOV PF device.
> The newly added property 'sriov_max_vfs' allows us to enable the SR-IOV feature
> on the virtio-net device.
> Currently, we are unable to specify the properties of a VF created from the guest.
> The properties are set to their default values.
> In the future, we plan to allow users to set the properties.
>
> qemu-system-x86_64 --device virtio-net,sriov_max_vfs=<num>
> # when 'sriov_max_vfs' is present, the SR-IOV feature will be automatically enabled
> # <num> means the max number of VF on guest
>
> Example commands to create VFs in virtio-net device from the guest:
>
> guest% readlink -f /sys/class/net/eth1/device
>  /sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/virtio1
> guest% echo "2" > /sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/sriov_numvfs
> guest% ip link show
>  eth0: ....
>  eth1: ....
>  eth2: .... #virtual VF created
>  eth3: .... #virtual VF created
>
> Please note that communication between VF and PF/VF is not possible by this patch series itself.
>
> Yui Washizu (1):
>   virtio-pci: add SR-IOV capability
>
>  hw/pci/msix.c                  |  8 +++--
>  hw/pci/pci.c                   |  4 +++
>  hw/virtio/virtio-pci.c         | 62 ++++++++++++++++++++++++++++++----
>  include/hw/virtio/virtio-pci.h |  1 +
>  4 files changed, 66 insertions(+), 9 deletions(-)
>
> --
> 2.39.3
>

Re: [RFC 0/1] virtio-net: add support for SR-IOV emulation

[Yui Washizu]

On 2023/07/20 11:20, Jason Wang wrote:
> On Wed, Jul 19, 2023 at 9:59 AM Yui Washizu <yui.washidu@gmail.com> wrote:
>> This patch series is the first step towards enabling
>> hardware offloading of the L2 packet switching feature on virtio-net device to host machine.
>> We are considering that this hardware offloading enables
>> the use of high-performance networks in virtual infrastructures,
>> such as container infrastructures on VMs.
>>
>> To enable L2 packet switching by SR-IOV VFs, we are considering the following:
>> - making the guest recognize virtio-net devices as SR-IOV PF devices
>>    (archived with this patch series)
>> - allowing virtio-net devices to connect SR-IOV VFs to the backend networks,
>>    leaving the L2 packet switching feature to the management layer like libvirt
> Could you please show the qemu command line you want to propose here?


I am considering how to specify the properties of VFs to connect SR-IOV 
VFs to the backend networks.


For example:


qemu-system-x86_64 -device 
pcie-root-port,port=8,chassis=8,id=pci.8,bus=pcie.0,multifunction=on
                    -netdev tap,id=hostnet0,vhost=on
                    -netdev tap,id=vfnet1,vhost=on # backend network for 
SR-IOV VF 1
                    -netdev tap,id=vfnet2,vhost=on # backend network for 
SR-IOV VF 2
                    -device 
virtio-net-pci,netdev=hostnet0,sriov_max_vfs=2,sriov_netdev=vfnet1:vfnet2,...


In this example, we can specify multiple backend networks to the VFs
by adding "sriov_netdev" and separating them with ":".
Additionally, when passing properties like "rx_queue_size" to VFs, we 
can utilize new properties,
such as "sriov_rx_queue_size_per_vfs," to ensure that the same value is 
passed to all VFs.

I'm still considering about how to specify it, so please give me any 
comments if you have any.


>>    - This makes hardware offloading of L2 packet switching possible.
>>      For example, when using vDPA devices, it allows the guest
>>      to utilize SR-IOV NIC embedded switch of hosts.
> This would be interesting.
>
> Thanks
>
>> This patch series aims to enable SR-IOV emulation on virtio-net devices.
>> With this series, the guest can identify the virtio-net device as an SR-IOV PF device.
>> The newly added property 'sriov_max_vfs' allows us to enable the SR-IOV feature
>> on the virtio-net device.
>> Currently, we are unable to specify the properties of a VF created from the guest.
>> The properties are set to their default values.
>> In the future, we plan to allow users to set the properties.
>>
>> qemu-system-x86_64 --device virtio-net,sriov_max_vfs=<num>
>> # when 'sriov_max_vfs' is present, the SR-IOV feature will be automatically enabled
>> # <num> means the max number of VF on guest
>>
>> Example commands to create VFs in virtio-net device from the guest:
>>
>> guest% readlink -f /sys/class/net/eth1/device
>>   /sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/virtio1
>> guest% echo "2" > /sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/sriov_numvfs
>> guest% ip link show
>>   eth0: ....
>>   eth1: ....
>>   eth2: .... #virtual VF created
>>   eth3: .... #virtual VF created
>>
>> Please note that communication between VF and PF/VF is not possible by this patch series itself.
>>
>> Yui Washizu (1):
>>    virtio-pci: add SR-IOV capability
>>
>>   hw/pci/msix.c                  |  8 +++--
>>   hw/pci/pci.c                   |  4 +++
>>   hw/virtio/virtio-pci.c         | 62 ++++++++++++++++++++++++++++++----
>>   include/hw/virtio/virtio-pci.h |  1 +
>>   4 files changed, 66 insertions(+), 9 deletions(-)
>>
>> --
>> 2.39.3
>>

Re: [RFC 0/1] virtio-net: add support for SR-IOV emulation

[Jason Wang]

On Mon, Jul 24, 2023 at 10:32 AM Yui Washizu <yui.washidu@gmail.com> wrote:
>
>
> On 2023/07/20 11:20, Jason Wang wrote:
> > On Wed, Jul 19, 2023 at 9:59 AM Yui Washizu <yui.washidu@gmail.com> wrote:
> >> This patch series is the first step towards enabling
> >> hardware offloading of the L2 packet switching feature on virtio-net device to host machine.
> >> We are considering that this hardware offloading enables
> >> the use of high-performance networks in virtual infrastructures,
> >> such as container infrastructures on VMs.
> >>
> >> To enable L2 packet switching by SR-IOV VFs, we are considering the following:
> >> - making the guest recognize virtio-net devices as SR-IOV PF devices
> >>    (archived with this patch series)
> >> - allowing virtio-net devices to connect SR-IOV VFs to the backend networks,
> >>    leaving the L2 packet switching feature to the management layer like libvirt
> > Could you please show the qemu command line you want to propose here?
>
>
> I am considering how to specify the properties of VFs to connect SR-IOV
> VFs to the backend networks.
>
>
> For example:
>
>
> qemu-system-x86_64 -device
> pcie-root-port,port=8,chassis=8,id=pci.8,bus=pcie.0,multifunction=on
>                     -netdev tap,id=hostnet0,vhost=on
>                     -netdev tap,id=vfnet1,vhost=on # backend network for
> SR-IOV VF 1
>                     -netdev tap,id=vfnet2,vhost=on # backend network for
> SR-IOV VF 2
>                     -device
> virtio-net-pci,netdev=hostnet0,sriov_max_vfs=2,sriov_netdev=vfnet1:vfnet2,...
>
>
> In this example, we can specify multiple backend networks to the VFs
> by adding "sriov_netdev" and separating them with ":".

This seems what is in my mind as well, more below

> Additionally, when passing properties like "rx_queue_size" to VFs, we
> can utilize new properties,
> such as "sriov_rx_queue_size_per_vfs," to ensure that the same value is
> passed to all VFs.

Or we can introduce new device like:

-netdev tap,id=hn0 \
-device virtio-net-pci,netdev=hn0,id=vnet_pf \
-netdev tap,netdev=hn1 \
-device virtio-net-pci-vf,netdev=hn1,id=vf0,pf=vnet_pf,rx_queue_size=XYZ ... \

This allows us to reuse the codes for configuring vf parameters. But
note that rx_queue_size doesn't make too much sense to vhost-vDPA, as
qemu can perform nothing more than a simple sanity test.

Thanks

>
> I'm still considering about how to specify it, so please give me any
> comments if you have any.
>
>
> >>    - This makes hardware offloading of L2 packet switching possible.
> >>      For example, when using vDPA devices, it allows the guest
> >>      to utilize SR-IOV NIC embedded switch of hosts.
> > This would be interesting.
> >
> > Thanks
> >
> >> This patch series aims to enable SR-IOV emulation on virtio-net devices.
> >> With this series, the guest can identify the virtio-net device as an SR-IOV PF device.
> >> The newly added property 'sriov_max_vfs' allows us to enable the SR-IOV feature
> >> on the virtio-net device.
> >> Currently, we are unable to specify the properties of a VF created from the guest.
> >> The properties are set to their default values.
> >> In the future, we plan to allow users to set the properties.
> >>
> >> qemu-system-x86_64 --device virtio-net,sriov_max_vfs=<num>
> >> # when 'sriov_max_vfs' is present, the SR-IOV feature will be automatically enabled
> >> # <num> means the max number of VF on guest
> >>
> >> Example commands to create VFs in virtio-net device from the guest:
> >>
> >> guest% readlink -f /sys/class/net/eth1/device
> >>   /sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/virtio1
> >> guest% echo "2" > /sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/sriov_numvfs
> >> guest% ip link show
> >>   eth0: ....
> >>   eth1: ....
> >>   eth2: .... #virtual VF created
> >>   eth3: .... #virtual VF created
> >>
> >> Please note that communication between VF and PF/VF is not possible by this patch series itself.
> >>
> >> Yui Washizu (1):
> >>    virtio-pci: add SR-IOV capability
> >>
> >>   hw/pci/msix.c                  |  8 +++--
> >>   hw/pci/pci.c                   |  4 +++
> >>   hw/virtio/virtio-pci.c         | 62 ++++++++++++++++++++++++++++++----
> >>   include/hw/virtio/virtio-pci.h |  1 +
> >>   4 files changed, 66 insertions(+), 9 deletions(-)
> >>
> >> --
> >> 2.39.3
> >>
>

Re: [RFC 0/1] virtio-net: add support for SR-IOV emulation

[Yui Washizu]

On 2023/07/24 15:58, Jason Wang wrote:
> On Mon, Jul 24, 2023 at 10:32 AM Yui Washizu <yui.washidu@gmail.com> wrote:
>>
>> On 2023/07/20 11:20, Jason Wang wrote:
>>> On Wed, Jul 19, 2023 at 9:59 AM Yui Washizu <yui.washidu@gmail.com> wrote:
>>>> This patch series is the first step towards enabling
>>>> hardware offloading of the L2 packet switching feature on virtio-net device to host machine.
>>>> We are considering that this hardware offloading enables
>>>> the use of high-performance networks in virtual infrastructures,
>>>> such as container infrastructures on VMs.
>>>>
>>>> To enable L2 packet switching by SR-IOV VFs, we are considering the following:
>>>> - making the guest recognize virtio-net devices as SR-IOV PF devices
>>>>     (archived with this patch series)
>>>> - allowing virtio-net devices to connect SR-IOV VFs to the backend networks,
>>>>     leaving the L2 packet switching feature to the management layer like libvirt
>>> Could you please show the qemu command line you want to propose here?
>>
>> I am considering how to specify the properties of VFs to connect SR-IOV
>> VFs to the backend networks.
>>
>>
>> For example:
>>
>>
>> qemu-system-x86_64 -device
>> pcie-root-port,port=8,chassis=8,id=pci.8,bus=pcie.0,multifunction=on
>>                      -netdev tap,id=hostnet0,vhost=on
>>                      -netdev tap,id=vfnet1,vhost=on # backend network for
>> SR-IOV VF 1
>>                      -netdev tap,id=vfnet2,vhost=on # backend network for
>> SR-IOV VF 2
>>                      -device
>> virtio-net-pci,netdev=hostnet0,sriov_max_vfs=2,sriov_netdev=vfnet1:vfnet2,...
>>
>>
>> In this example, we can specify multiple backend networks to the VFs
>> by adding "sriov_netdev" and separating them with ":".
> This seems what is in my mind as well, more below
>
>> Additionally, when passing properties like "rx_queue_size" to VFs, we
>> can utilize new properties,
>> such as "sriov_rx_queue_size_per_vfs," to ensure that the same value is
>> passed to all VFs.
> Or we can introduce new device like:
>
> -netdev tap,id=hn0 \
> -device virtio-net-pci,netdev=hn0,id=vnet_pf \
> -netdev tap,netdev=hn1 \
> -device virtio-net-pci-vf,netdev=hn1,id=vf0,pf=vnet_pf,rx_queue_size=XYZ ... \
>
> This allows us to reuse the codes for configuring vf parameters.


Yes, I was also considering this method was one of the option.


I was concerned that the implementation
for it might be different from other devices specified by the "-device" 
option
because VF devices are created when the guest creates VFs from PFs,
not when the guest starts,
and the implementation may be complicated.
So I dropped the idea then.


However, I think it is a cleaner way for specifying VFs than my 
suggested one.


I will further consider whether the implementation will be really 
difficult or not.


Thank you.


> But
> note that rx_queue_size doesn't make too much sense to vhost-vDPA, as
> qemu can perform nothing more than a simple sanity test.
>
> Thanks
>
>> I'm still considering about how to specify it, so please give me any
>> comments if you have any.
>>
>>
>>>>     - This makes hardware offloading of L2 packet switching possible.
>>>>       For example, when using vDPA devices, it allows the guest
>>>>       to utilize SR-IOV NIC embedded switch of hosts.
>>> This would be interesting.
>>>
>>> Thanks
>>>
>>>> This patch series aims to enable SR-IOV emulation on virtio-net devices.
>>>> With this series, the guest can identify the virtio-net device as an SR-IOV PF device.
>>>> The newly added property 'sriov_max_vfs' allows us to enable the SR-IOV feature
>>>> on the virtio-net device.
>>>> Currently, we are unable to specify the properties of a VF created from the guest.
>>>> The properties are set to their default values.
>>>> In the future, we plan to allow users to set the properties.
>>>>
>>>> qemu-system-x86_64 --device virtio-net,sriov_max_vfs=<num>
>>>> # when 'sriov_max_vfs' is present, the SR-IOV feature will be automatically enabled
>>>> # <num> means the max number of VF on guest
>>>>
>>>> Example commands to create VFs in virtio-net device from the guest:
>>>>
>>>> guest% readlink -f /sys/class/net/eth1/device
>>>>    /sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/virtio1
>>>> guest% echo "2" > /sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/sriov_numvfs
>>>> guest% ip link show
>>>>    eth0: ....
>>>>    eth1: ....
>>>>    eth2: .... #virtual VF created
>>>>    eth3: .... #virtual VF created
>>>>
>>>> Please note that communication between VF and PF/VF is not possible by this patch series itself.
>>>>
>>>> Yui Washizu (1):
>>>>     virtio-pci: add SR-IOV capability
>>>>
>>>>    hw/pci/msix.c                  |  8 +++--
>>>>    hw/pci/pci.c                   |  4 +++
>>>>    hw/virtio/virtio-pci.c         | 62 ++++++++++++++++++++++++++++++----
>>>>    include/hw/virtio/virtio-pci.h |  1 +
>>>>    4 files changed, 66 insertions(+), 9 deletions(-)
>>>>
>>>> --
>>>> 2.39.3
>>>>

Re: [RFC 0/1] virtio-net: add support for SR-IOV emulation

[Yui Washizu]

On 2023/07/24 15:58, Jason Wang wrote:
> On Mon, Jul 24, 2023 at 10:32 AM Yui Washizu <yui.washidu@gmail.com> wrote:
>>
>> On 2023/07/20 11:20, Jason Wang wrote:
>>> On Wed, Jul 19, 2023 at 9:59 AM Yui Washizu <yui.washidu@gmail.com> wrote:
>>>> This patch series is the first step towards enabling
>>>> hardware offloading of the L2 packet switching feature on virtio-net device to host machine.
>>>> We are considering that this hardware offloading enables
>>>> the use of high-performance networks in virtual infrastructures,
>>>> such as container infrastructures on VMs.
>>>>
>>>> To enable L2 packet switching by SR-IOV VFs, we are considering the following:
>>>> - making the guest recognize virtio-net devices as SR-IOV PF devices
>>>>     (archived with this patch series)
>>>> - allowing virtio-net devices to connect SR-IOV VFs to the backend networks,
>>>>     leaving the L2 packet switching feature to the management layer like libvirt
>>> Could you please show the qemu command line you want to propose here?
>>
>> I am considering how to specify the properties of VFs to connect SR-IOV
>> VFs to the backend networks.
>>
>>
>> For example:
>>
>>
>> qemu-system-x86_64 -device
>> pcie-root-port,port=8,chassis=8,id=pci.8,bus=pcie.0,multifunction=on
>>                      -netdev tap,id=hostnet0,vhost=on
>>                      -netdev tap,id=vfnet1,vhost=on # backend network for
>> SR-IOV VF 1
>>                      -netdev tap,id=vfnet2,vhost=on # backend network for
>> SR-IOV VF 2
>>                      -device
>> virtio-net-pci,netdev=hostnet0,sriov_max_vfs=2,sriov_netdev=vfnet1:vfnet2,...
>>
>>
>> In this example, we can specify multiple backend networks to the VFs
>> by adding "sriov_netdev" and separating them with ":".
> This seems what is in my mind as well, more below
>
>> Additionally, when passing properties like "rx_queue_size" to VFs, we
>> can utilize new properties,
>> such as "sriov_rx_queue_size_per_vfs," to ensure that the same value is
>> passed to all VFs.
> Or we can introduce new device like:
>
> -netdev tap,id=hn0 \
> -device virtio-net-pci,netdev=hn0,id=vnet_pf \
> -netdev tap,netdev=hn1 \
> -device virtio-net-pci-vf,netdev=hn1,id=vf0,pf=vnet_pf,rx_queue_size=XYZ ... \
>
> This allows us to reuse the codes for configuring vf parameters. But
> note that rx_queue_size doesn't make too much sense to vhost-vDPA, as
> qemu can perform nothing more than a simple sanity test.
>
> Thanks


Thanks for proposing this new way.

I have considered how to implement this.


As virtio-net-pci-vf device should show up

on the guest only when the guest OS creates a VF,

the guest must not be able to see the VF device on PCI bus when qemu starts.

However, it's hard to realize this without overcomplicating

relevant code due to current qemu implementation.

It's because qdev_device_add_from_qdict,

a function which is called when devices are specified

with "-device" option of qemu startup command,

always create devices by qdev_new and qdev_realize.

It might be possible that we fix it

so that qdev_new/qdev_realize aren't triggered for virtio-net-pci-vf 
devices,

but It seems that we need to special case the device in very generic code

like qdev_device_add_from_qdict(), qdev_device_add(),

device_init_func() or their caller function.


Given my current ideas,

it seems like this PATCH could become complex.

Woule you have any suggestions

for achieving this in more simple way possible ?



>> I'm still considering about how to specify it, so please give me any
>> comments if you have any.
>>
>>
>>>>     - This makes hardware offloading of L2 packet switching possible.
>>>>       For example, when using vDPA devices, it allows the guest
>>>>       to utilize SR-IOV NIC embedded switch of hosts.
>>> This would be interesting.
>>>
>>> Thanks
>>>
>>>> This patch series aims to enable SR-IOV emulation on virtio-net devices.
>>>> With this series, the guest can identify the virtio-net device as an SR-IOV PF device.
>>>> The newly added property 'sriov_max_vfs' allows us to enable the SR-IOV feature
>>>> on the virtio-net device.
>>>> Currently, we are unable to specify the properties of a VF created from the guest.
>>>> The properties are set to their default values.
>>>> In the future, we plan to allow users to set the properties.
>>>>
>>>> qemu-system-x86_64 --device virtio-net,sriov_max_vfs=<num>
>>>> # when 'sriov_max_vfs' is present, the SR-IOV feature will be automatically enabled
>>>> # <num> means the max number of VF on guest
>>>>
>>>> Example commands to create VFs in virtio-net device from the guest:
>>>>
>>>> guest% readlink -f /sys/class/net/eth1/device
>>>>    /sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/virtio1
>>>> guest% echo "2" > /sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/sriov_numvfs
>>>> guest% ip link show
>>>>    eth0: ....
>>>>    eth1: ....
>>>>    eth2: .... #virtual VF created
>>>>    eth3: .... #virtual VF created
>>>>
>>>> Please note that communication between VF and PF/VF is not possible by this patch series itself.
>>>>
>>>> Yui Washizu (1):
>>>>     virtio-pci: add SR-IOV capability
>>>>
>>>>    hw/pci/msix.c                  |  8 +++--
>>>>    hw/pci/pci.c                   |  4 +++
>>>>    hw/virtio/virtio-pci.c         | 62 ++++++++++++++++++++++++++++++----
>>>>    include/hw/virtio/virtio-pci.h |  1 +
>>>>    4 files changed, 66 insertions(+), 9 deletions(-)
>>>>
>>>> --
>>>> 2.39.3
>>>>

Re: [RFC 0/1] virtio-net: add support for SR-IOV emulation

[Yui Washizu]

Hi Jason,


On 2023/08/30 14:28, Yui Washizu wrote:
>
> On 2023/07/24 15:58, Jason Wang wrote:
>> On Mon, Jul 24, 2023 at 10:32 AM Yui Washizu <yui.washidu@gmail.com> 
>> wrote:
>>>
>>> On 2023/07/20 11:20, Jason Wang wrote:
>>>> On Wed, Jul 19, 2023 at 9:59 AM Yui Washizu <yui.washidu@gmail.com> 
>>>> wrote:
>>>>> This patch series is the first step towards enabling
>>>>> hardware offloading of the L2 packet switching feature on 
>>>>> virtio-net device to host machine.
>>>>> We are considering that this hardware offloading enables
>>>>> the use of high-performance networks in virtual infrastructures,
>>>>> such as container infrastructures on VMs.
>>>>>
>>>>> To enable L2 packet switching by SR-IOV VFs, we are considering 
>>>>> the following:
>>>>> - making the guest recognize virtio-net devices as SR-IOV PF devices
>>>>>     (archived with this patch series)
>>>>> - allowing virtio-net devices to connect SR-IOV VFs to the backend 
>>>>> networks,
>>>>>     leaving the L2 packet switching feature to the management 
>>>>> layer like libvirt
>>>> Could you please show the qemu command line you want to propose here?
>>>
>>> I am considering how to specify the properties of VFs to connect SR-IOV
>>> VFs to the backend networks.
>>>
>>>
>>> For example:
>>>
>>>
>>> qemu-system-x86_64 -device
>>> pcie-root-port,port=8,chassis=8,id=pci.8,bus=pcie.0,multifunction=on
>>>                      -netdev tap,id=hostnet0,vhost=on
>>>                      -netdev tap,id=vfnet1,vhost=on # backend 
>>> network for
>>> SR-IOV VF 1
>>>                      -netdev tap,id=vfnet2,vhost=on # backend 
>>> network for
>>> SR-IOV VF 2
>>>                      -device
>>> virtio-net-pci,netdev=hostnet0,sriov_max_vfs=2,sriov_netdev=vfnet1:vfnet2,... 
>>>
>>>
>>>
>>> In this example, we can specify multiple backend networks to the VFs
>>> by adding "sriov_netdev" and separating them with ":".
>> This seems what is in my mind as well, more below
>>
>>> Additionally, when passing properties like "rx_queue_size" to VFs, we
>>> can utilize new properties,
>>> such as "sriov_rx_queue_size_per_vfs," to ensure that the same value is
>>> passed to all VFs.
>> Or we can introduce new device like:
>>
>> -netdev tap,id=hn0 \
>> -device virtio-net-pci,netdev=hn0,id=vnet_pf \
>> -netdev tap,netdev=hn1 \
>> -device 
>> virtio-net-pci-vf,netdev=hn1,id=vf0,pf=vnet_pf,rx_queue_size=XYZ ... \
>>
>> This allows us to reuse the codes for configuring vf parameters. But
>> note that rx_queue_size doesn't make too much sense to vhost-vDPA, as
>> qemu can perform nothing more than a simple sanity test.
>>
>> Thanks
>
>
> Thanks for proposing this new way.
>
> I have considered how to implement this.
>
>
> As virtio-net-pci-vf device should show up
>
> on the guest only when the guest OS creates a VF,
>
> the guest must not be able to see the VF device on PCI bus when qemu 
> starts.
>
> However, it's hard to realize this without overcomplicating
>
> relevant code due to current qemu implementation.
>
> It's because qdev_device_add_from_qdict,
>
> a function which is called when devices are specified
>
> with "-device" option of qemu startup command,
>
> always create devices by qdev_new and qdev_realize.
>
> It might be possible that we fix it
>
> so that qdev_new/qdev_realize aren't triggered for virtio-net-pci-vf 
> devices,
>
> but It seems that we need to special case the device in very generic code
>
> like qdev_device_add_from_qdict(), qdev_device_add(),
>
> device_init_func() or their caller function.
>
>
> Given my current ideas,
>
> it seems like this PATCH could become complex.
>
> Woule you have any suggestions
>
> for achieving this in more simple way possible ?
>
>

I was wondering if you could give me some feedback.
Best regard.


>
>>> I'm still considering about how to specify it, so please give me any
>>> comments if you have any.
>>>
>>>
>>>>>     - This makes hardware offloading of L2 packet switching possible.
>>>>>       For example, when using vDPA devices, it allows the guest
>>>>>       to utilize SR-IOV NIC embedded switch of hosts.
>>>> This would be interesting.
>>>>
>>>> Thanks
>>>>
>>>>> This patch series aims to enable SR-IOV emulation on virtio-net 
>>>>> devices.
>>>>> With this series, the guest can identify the virtio-net device as 
>>>>> an SR-IOV PF device.
>>>>> The newly added property 'sriov_max_vfs' allows us to enable the 
>>>>> SR-IOV feature
>>>>> on the virtio-net device.
>>>>> Currently, we are unable to specify the properties of a VF created 
>>>>> from the guest.
>>>>> The properties are set to their default values.
>>>>> In the future, we plan to allow users to set the properties.
>>>>>
>>>>> qemu-system-x86_64 --device virtio-net,sriov_max_vfs=<num>
>>>>> # when 'sriov_max_vfs' is present, the SR-IOV feature will be 
>>>>> automatically enabled
>>>>> # <num> means the max number of VF on guest
>>>>>
>>>>> Example commands to create VFs in virtio-net device from the guest:
>>>>>
>>>>> guest% readlink -f /sys/class/net/eth1/device
>>>>> /sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/virtio1
>>>>> guest% echo "2" > 
>>>>> /sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/sriov_numvfs
>>>>> guest% ip link show
>>>>>    eth0: ....
>>>>>    eth1: ....
>>>>>    eth2: .... #virtual VF created
>>>>>    eth3: .... #virtual VF created
>>>>>
>>>>> Please note that communication between VF and PF/VF is not 
>>>>> possible by this patch series itself.
>>>>>
>>>>> Yui Washizu (1):
>>>>>     virtio-pci: add SR-IOV capability
>>>>>
>>>>>    hw/pci/msix.c                  |  8 +++--
>>>>>    hw/pci/pci.c                   |  4 +++
>>>>>    hw/virtio/virtio-pci.c         | 62 
>>>>> ++++++++++++++++++++++++++++++----
>>>>>    include/hw/virtio/virtio-pci.h |  1 +
>>>>>    4 files changed, 66 insertions(+), 9 deletions(-)
>>>>>
>>>>> -- 
>>>>> 2.39.3
>>>>>

Re: [RFC 0/1] virtio-net: add support for SR-IOV emulation

[Jason Wang]

On Wed, Sep 6, 2023 at 1:11 PM Yui Washizu <yui.washidu@gmail.com> wrote:
>
>
> Hi Jason,
>
>
> On 2023/08/30 14:28, Yui Washizu wrote:
> >
> > On 2023/07/24 15:58, Jason Wang wrote:
> >> On Mon, Jul 24, 2023 at 10:32 AM Yui Washizu <yui.washidu@gmail.com>
> >> wrote:
> >>>
> >>> On 2023/07/20 11:20, Jason Wang wrote:
> >>>> On Wed, Jul 19, 2023 at 9:59 AM Yui Washizu <yui.washidu@gmail.com>
> >>>> wrote:
> >>>>> This patch series is the first step towards enabling
> >>>>> hardware offloading of the L2 packet switching feature on
> >>>>> virtio-net device to host machine.
> >>>>> We are considering that this hardware offloading enables
> >>>>> the use of high-performance networks in virtual infrastructures,
> >>>>> such as container infrastructures on VMs.
> >>>>>
> >>>>> To enable L2 packet switching by SR-IOV VFs, we are considering
> >>>>> the following:
> >>>>> - making the guest recognize virtio-net devices as SR-IOV PF devices
> >>>>>     (archived with this patch series)
> >>>>> - allowing virtio-net devices to connect SR-IOV VFs to the backend
> >>>>> networks,
> >>>>>     leaving the L2 packet switching feature to the management
> >>>>> layer like libvirt
> >>>> Could you please show the qemu command line you want to propose here?
> >>>
> >>> I am considering how to specify the properties of VFs to connect SR-IOV
> >>> VFs to the backend networks.
> >>>
> >>>
> >>> For example:
> >>>
> >>>
> >>> qemu-system-x86_64 -device
> >>> pcie-root-port,port=8,chassis=8,id=pci.8,bus=pcie.0,multifunction=on
> >>>                      -netdev tap,id=hostnet0,vhost=on
> >>>                      -netdev tap,id=vfnet1,vhost=on # backend
> >>> network for
> >>> SR-IOV VF 1
> >>>                      -netdev tap,id=vfnet2,vhost=on # backend
> >>> network for
> >>> SR-IOV VF 2
> >>>                      -device
> >>> virtio-net-pci,netdev=hostnet0,sriov_max_vfs=2,sriov_netdev=vfnet1:vfnet2,...
> >>>
> >>>
> >>>
> >>> In this example, we can specify multiple backend networks to the VFs
> >>> by adding "sriov_netdev" and separating them with ":".
> >> This seems what is in my mind as well, more below
> >>
> >>> Additionally, when passing properties like "rx_queue_size" to VFs, we
> >>> can utilize new properties,
> >>> such as "sriov_rx_queue_size_per_vfs," to ensure that the same value is
> >>> passed to all VFs.
> >> Or we can introduce new device like:
> >>
> >> -netdev tap,id=hn0 \
> >> -device virtio-net-pci,netdev=hn0,id=vnet_pf \
> >> -netdev tap,netdev=hn1 \
> >> -device
> >> virtio-net-pci-vf,netdev=hn1,id=vf0,pf=vnet_pf,rx_queue_size=XYZ ... \
> >>
> >> This allows us to reuse the codes for configuring vf parameters. But
> >> note that rx_queue_size doesn't make too much sense to vhost-vDPA, as
> >> qemu can perform nothing more than a simple sanity test.
> >>
> >> Thanks
> >
> >
> > Thanks for proposing this new way.
> >
> > I have considered how to implement this.
> >
> >
> > As virtio-net-pci-vf device should show up
> >
> > on the guest only when the guest OS creates a VF,
> >
> > the guest must not be able to see the VF device on PCI bus when qemu
> > starts.
> >
> > However, it's hard to realize this without overcomplicating
> >
> > relevant code due to current qemu implementation.
> >
> > It's because qdev_device_add_from_qdict,
> >
> > a function which is called when devices are specified
> >
> > with "-device" option of qemu startup command,
> >
> > always create devices by qdev_new and qdev_realize.
> >
> > It might be possible that we fix it
> >
> > so that qdev_new/qdev_realize aren't triggered for virtio-net-pci-vf
> > devices,
> >
> > but It seems that we need to special case the device in very generic code
> >
> > like qdev_device_add_from_qdict(), qdev_device_add(),
> >
> > device_init_func() or their caller function.
> >
> >
> > Given my current ideas,
> >
> > it seems like this PATCH could become complex.
> >
> > Woule you have any suggestions
> >
> > for achieving this in more simple way possible ?
> >
> >
>
> I was wondering if you could give me some feedback.
> Best regard.

Sorry for the late reply, I think we can start from the easy way from
your point and see.

Thanks

>
>
> >
> >>> I'm still considering about how to specify it, so please give me any
> >>> comments if you have any.
> >>>
> >>>
> >>>>>     - This makes hardware offloading of L2 packet switching possible.
> >>>>>       For example, when using vDPA devices, it allows the guest
> >>>>>       to utilize SR-IOV NIC embedded switch of hosts.
> >>>> This would be interesting.
> >>>>
> >>>> Thanks
> >>>>
> >>>>> This patch series aims to enable SR-IOV emulation on virtio-net
> >>>>> devices.
> >>>>> With this series, the guest can identify the virtio-net device as
> >>>>> an SR-IOV PF device.
> >>>>> The newly added property 'sriov_max_vfs' allows us to enable the
> >>>>> SR-IOV feature
> >>>>> on the virtio-net device.
> >>>>> Currently, we are unable to specify the properties of a VF created
> >>>>> from the guest.
> >>>>> The properties are set to their default values.
> >>>>> In the future, we plan to allow users to set the properties.
> >>>>>
> >>>>> qemu-system-x86_64 --device virtio-net,sriov_max_vfs=<num>
> >>>>> # when 'sriov_max_vfs' is present, the SR-IOV feature will be
> >>>>> automatically enabled
> >>>>> # <num> means the max number of VF on guest
> >>>>>
> >>>>> Example commands to create VFs in virtio-net device from the guest:
> >>>>>
> >>>>> guest% readlink -f /sys/class/net/eth1/device
> >>>>> /sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/virtio1
> >>>>> guest% echo "2" >
> >>>>> /sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0/sriov_numvfs
> >>>>> guest% ip link show
> >>>>>    eth0: ....
> >>>>>    eth1: ....
> >>>>>    eth2: .... #virtual VF created
> >>>>>    eth3: .... #virtual VF created
> >>>>>
> >>>>> Please note that communication between VF and PF/VF is not
> >>>>> possible by this patch series itself.
> >>>>>
> >>>>> Yui Washizu (1):
> >>>>>     virtio-pci: add SR-IOV capability
> >>>>>
> >>>>>    hw/pci/msix.c                  |  8 +++--
> >>>>>    hw/pci/pci.c                   |  4 +++
> >>>>>    hw/virtio/virtio-pci.c         | 62
> >>>>> ++++++++++++++++++++++++++++++----
> >>>>>    include/hw/virtio/virtio-pci.h |  1 +
> >>>>>    4 files changed, 66 insertions(+), 9 deletions(-)
> >>>>>
> >>>>> --
> >>>>> 2.39.3
> >>>>>
>