[PATCH v2 0/2] Vhost-vdpa Shadow Virtqueue Hash calculation Support

[PATCH v2 0/2] Vhost-vdpa Shadow Virtqueue Hash calculation Support

[Hawkins Jiawei]

This series enables shadowed CVQ to intercept
VIRTIO_NET_CTRL_MQ_HASH_CONFIG command through shadowed CVQ,
update the virtio NIC device model so qemu send it in a
migration, and the restore of that Hash calculation state
in the destination.

Note that this patch should be based on
patch "vdpa: Send all CVQ state load commands in parallel" at [1].

[1]. https://lore.kernel.org/all/cover.1693287885.git.yin31149@gmail.com/

TestStep
========
1. test the migration using vp-vdpa device
  - For L0 guest, boot QEMU with two virtio-net-pci net device with
`ctrl_vq`, `mq`, `hash` features on, command line like:
	-netdev tap,...
	-device virtio-net-pci,disable-legacy=on,disable-modern=off,
iommu_platform=on,mq=on,ctrl_vq=on,hash=on,guest_announce=off,
indirect_desc=off,queue_reset=off,...

  - For L1 guest, apply the relative patch series and compile the
source code, start QEMU with two vdpa device with svq mode on,
enable the `ctrl_vq`, `mq`, `hash` features on, command line like:
      -netdev type=vhost-vdpa,x-svq=true,...
      -device virtio-net-pci,mq=on,guest_announce=off,ctrl_vq=on,
hash=on,...

  - For L2 source guest, run the following bash command:
```bash
#!/bin/sh

ethtool -K eth0 rxhash on
```
  - Gdb attach the destination VM and break at the
vhost_vdpa_net_load_rss()

  - Execute the live migration in L2 source monitor

  - Result
    * with this series, gdb can hit the breakpoint and continue
the executing without triggering any error or warning.


ChangeLog
=========
v2:
  - resolve conflict with updated patch
"vdpa: Send all CVQ state load commands in parallel", move the
`table` declaration at the beginning of the vhost_vdpa_net_load_rss()
in patch
"vdpa: Restore hash calculation state"

RFC: https://lore.kernel.org/all/cover.1691762906.git.yin31149@gmail.com/#t

Hawkins Jiawei (2):
  vdpa: Restore hash calculation state
  vdpa: Allow VIRTIO_NET_F_HASH_REPORT in SVQ

 net/vhost-vdpa.c | 92 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 92 insertions(+)

-- 
2.25.1

[PATCH v2 1/2] vdpa: Restore hash calculation state

[Hawkins Jiawei]

This patch introduces vhost_vdpa_net_load_rss() to restore
the hash calculation state at device's startup.

Note that vhost_vdpa_net_load_rss() has `do_rss` argument,
which allows future code to reuse this function to restore
the receive-side scaling state when the VIRTIO_NET_F_RSS
feature is enabled in SVQ. Currently, vhost_vdpa_net_load_rss()
could only be invoked when `do_rss` is set to false.

Signed-off-by: Hawkins Jiawei <yin31149@gmail.com>
---
Question:

It seems that virtio_net_handle_rss() currently does not restore the
hash key length parsed from the CVQ command sent from the guest into
n->rss_data and uses the maximum key length in other code.

So for `hash_key_length` field in VIRTIO_NET_CTRL_MQ_HASH_CONFIG command
sent to device, is it okay to also use the maximum key length as its value?
Or should we introduce the `hash_key_length` field in n->rss_data
structure to record the key length from guest and use this value?

ChangeLog:

v2:
  - resolve conflict with updated patch
"vdpa: Send all CVQ state load commands in parallel"
  - move the `table` declaration at the beginning of the
vhost_vdpa_net_load_rss()

RFC: https://lore.kernel.org/all/a54ca70b12ebe2f3c391864e41241697ab1aba30.1691762906.git.yin31149@gmail.com/

 net/vhost-vdpa.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 91 insertions(+)

diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c
index 818464b702..139bb79468 100644
--- a/net/vhost-vdpa.c
+++ b/net/vhost-vdpa.c
@@ -805,6 +805,88 @@ static int vhost_vdpa_net_load_mac(VhostVDPAState *s, const VirtIONet *n,
     return 0;
 }
 
+static int vhost_vdpa_net_load_rss(VhostVDPAState *s, const VirtIONet *n,
+                                   struct iovec *out_cursor,
+                                   struct iovec *in_cursor, bool do_rss)
+{
+    struct virtio_net_rss_config cfg;
+    ssize_t r;
+    g_autofree uint16_t *table = NULL;
+
+    /*
+     * According to VirtIO standard, "Initially the device has all hash
+     * types disabled and reports only VIRTIO_NET_HASH_REPORT_NONE.".
+     *
+     * Therefore, there is no need to send this CVQ command if the
+     * driver disable the all hash types, which aligns with
+     * the device's defaults.
+     *
+     * Note that the device's defaults can mismatch the driver's
+     * configuration only at live migration.
+     */
+    if (!n->rss_data.enabled ||
+        n->rss_data.hash_types == VIRTIO_NET_HASH_REPORT_NONE) {
+        return 0;
+    }
+
+    cfg.hash_types = cpu_to_le32(n->rss_data.hash_types);
+
+    /*
+     * According to VirtIO standard, "Field reserved MUST contain zeroes.
+     * It is defined to make the structure to match the layout of
+     * virtio_net_rss_config structure, defined in 5.1.6.5.7.".
+     *
+     * Therefore, we need to zero the fields in struct virtio_net_rss_config,
+     * which corresponds the `reserved` field in
+     * struct virtio_net_hash_config.
+     */
+    memset(&cfg.indirection_table_mask, 0,
+           sizeof_field(struct virtio_net_hash_config, reserved));
+
+    table = g_malloc_n(n->rss_data.indirections_len,
+                       sizeof(n->rss_data.indirections_table[0]));
+    for (int i = 0; i < n->rss_data.indirections_len; ++i) {
+        table[i] = cpu_to_le16(n->rss_data.indirections_table[i]);
+    }
+
+    /*
+     * Consider that virtio_net_handle_rss() currently does not restore the
+     * hash key length parsed from the CVQ command sent from the guest into
+     * n->rss_data and uses the maximum key length in other code, so we also
+     * employthe the maxium key length here.
+     */
+    cfg.hash_key_length = sizeof(n->rss_data.key);
+
+    const struct iovec data[] = {
+        {
+            .iov_base = &cfg,
+            .iov_len = offsetof(struct virtio_net_rss_config,
+                                indirection_table),
+        }, {
+            .iov_base = table,
+            .iov_len = n->rss_data.indirections_len *
+                       sizeof(n->rss_data.indirections_table[0]),
+        }, {
+            .iov_base = &cfg.max_tx_vq,
+            .iov_len = offsetof(struct virtio_net_rss_config, hash_key_data) -
+                       offsetof(struct virtio_net_rss_config, max_tx_vq),
+        }, {
+            .iov_base = (void *)n->rss_data.key,
+            .iov_len = sizeof(n->rss_data.key),
+        }
+    };
+
+    r = vhost_vdpa_net_load_cmd(s, out_cursor, in_cursor,
+                                VIRTIO_NET_CTRL_MQ,
+                                VIRTIO_NET_CTRL_MQ_HASH_CONFIG,
+                                data, ARRAY_SIZE(data));
+    if (unlikely(r < 0)) {
+        return r;
+    }
+
+    return 0;
+}
+
 static int vhost_vdpa_net_load_mq(VhostVDPAState *s,
                                   const VirtIONet *n,
                                   struct iovec *out_cursor,
@@ -830,6 +912,15 @@ static int vhost_vdpa_net_load_mq(VhostVDPAState *s,
         return r;
     }
 
+    if (!virtio_vdev_has_feature(&n->parent_obj, VIRTIO_NET_F_HASH_REPORT)) {
+        return 0;
+    }
+
+    r = vhost_vdpa_net_load_rss(s, n, out_cursor, in_cursor, false);
+    if (unlikely(r < 0)) {
+        return r;
+    }
+
     return 0;
 }
 
-- 
2.25.1

[PATCH v2 2/2] vdpa: Allow VIRTIO_NET_F_HASH_REPORT in SVQ

[Hawkins Jiawei]

Enable SVQ with VIRTIO_NET_F_HASH_REPORT feature.

Signed-off-by: Hawkins Jiawei <yin31149@gmail.com>
---
 net/vhost-vdpa.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c
index 139bb79468..b29f84f54c 100644
--- a/net/vhost-vdpa.c
+++ b/net/vhost-vdpa.c
@@ -118,6 +118,7 @@ static const uint64_t vdpa_svq_device_features =
     BIT_ULL(VIRTIO_NET_F_CTRL_MAC_ADDR) |
     /* VHOST_F_LOG_ALL is exposed by SVQ */
     BIT_ULL(VHOST_F_LOG_ALL) |
+    BIT_ULL(VIRTIO_NET_F_HASH_REPORT) |
     BIT_ULL(VIRTIO_NET_F_RSC_EXT) |
     BIT_ULL(VIRTIO_NET_F_STANDBY) |
     BIT_ULL(VIRTIO_NET_F_SPEED_DUPLEX);
-- 
2.25.1

Re: [PATCH v2 0/2] Vhost-vdpa Shadow Virtqueue Hash calculation Support

[Michael S. Tsirkin]

On Tue, Aug 29, 2023 at 04:47:33PM +0800, Hawkins Jiawei wrote:
> This series enables shadowed CVQ to intercept
> VIRTIO_NET_CTRL_MQ_HASH_CONFIG command through shadowed CVQ,
> update the virtio NIC device model so qemu send it in a
> migration, and the restore of that Hash calculation state
> in the destination.
> 
> Note that this patch should be based on
> patch "vdpa: Send all CVQ state load commands in parallel" at [1].
> 
> [1]. https://lore.kernel.org/all/cover.1693287885.git.yin31149@gmail.com/

Dropped due to CI failures:
https://gitlab.com/qemu-project/qemu/-/jobs/5329820077


> TestStep
> ========
> 1. test the migration using vp-vdpa device
>   - For L0 guest, boot QEMU with two virtio-net-pci net device with
> `ctrl_vq`, `mq`, `hash` features on, command line like:
> 	-netdev tap,...
> 	-device virtio-net-pci,disable-legacy=on,disable-modern=off,
> iommu_platform=on,mq=on,ctrl_vq=on,hash=on,guest_announce=off,
> indirect_desc=off,queue_reset=off,...
> 
>   - For L1 guest, apply the relative patch series and compile the
> source code, start QEMU with two vdpa device with svq mode on,
> enable the `ctrl_vq`, `mq`, `hash` features on, command line like:
>       -netdev type=vhost-vdpa,x-svq=true,...
>       -device virtio-net-pci,mq=on,guest_announce=off,ctrl_vq=on,
> hash=on,...
> 
>   - For L2 source guest, run the following bash command:
> ```bash
> #!/bin/sh
> 
> ethtool -K eth0 rxhash on
> ```
>   - Gdb attach the destination VM and break at the
> vhost_vdpa_net_load_rss()
> 
>   - Execute the live migration in L2 source monitor
> 
>   - Result
>     * with this series, gdb can hit the breakpoint and continue
> the executing without triggering any error or warning.
> 
> 
> ChangeLog
> =========
> v2:
>   - resolve conflict with updated patch
> "vdpa: Send all CVQ state load commands in parallel", move the
> `table` declaration at the beginning of the vhost_vdpa_net_load_rss()
> in patch
> "vdpa: Restore hash calculation state"
> 
> RFC: https://lore.kernel.org/all/cover.1691762906.git.yin31149@gmail.com/#t
> 
> Hawkins Jiawei (2):
>   vdpa: Restore hash calculation state
>   vdpa: Allow VIRTIO_NET_F_HASH_REPORT in SVQ
> 
>  net/vhost-vdpa.c | 92 ++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 92 insertions(+)
> 
> -- 
> 2.25.1