* [PATCH 9.0 01/13] vdpa: add VhostVDPAShared
2023-11-24 17:14 [PATCH 9.0 00/13] Consolidate common vdpa members in VhostVDPAShared Eugenio Pérez
@ 2023-11-24 17:14 ` Eugenio Pérez
2023-12-01 5:35 ` Jason Wang
2023-11-24 17:14 ` [PATCH 9.0 02/13] vdpa: move iova tree to the shared struct Eugenio Pérez
` (13 subsequent siblings)
14 siblings, 1 reply; 21+ messages in thread
From: Eugenio Pérez @ 2023-11-24 17:14 UTC (permalink / raw)
To: qemu-devel
Cc: Parav Pandit, si-wei.liu, Stefano Garzarella, Zhu Lingshan,
Lei Yang, Michael S. Tsirkin, Jason Wang, Dragos Tatulea,
Laurent Vivier
It will hold properties shared among all vhost_vdpa instances associated
with of the same device. For example, we just need one iova_tree or one
memory listener for the entire device.
Next patches will register the vhost_vdpa memory listener at the
beginning of the VM migration at the destination. This enables QEMU to
map the memory to the device before stopping the VM at the source,
instead of doing while both source and destination are stopped, thus
minimizing the downtime.
However, the destination QEMU is unaware of which vhost_vdpa struct will
register its memory_listener. If the source guest has CVQ enabled, it
will be the one associated with the CVQ. Otherwise, it will be the
first one.
Save the memory operations related members in a common place rather than
always in the first / last vhost_vdpa.
Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
include/hw/virtio/vhost-vdpa.h | 5 +++++
net/vhost-vdpa.c | 24 ++++++++++++++++++++++--
2 files changed, 27 insertions(+), 2 deletions(-)
diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h
index 5407d54fd7..eb1a56d75a 100644
--- a/include/hw/virtio/vhost-vdpa.h
+++ b/include/hw/virtio/vhost-vdpa.h
@@ -30,6 +30,10 @@ typedef struct VhostVDPAHostNotifier {
void *addr;
} VhostVDPAHostNotifier;
+/* Info shared by all vhost_vdpa device models */
+typedef struct vhost_vdpa_shared {
+} VhostVDPAShared;
+
typedef struct vhost_vdpa {
int device_fd;
int index;
@@ -46,6 +50,7 @@ typedef struct vhost_vdpa {
bool suspended;
/* IOVA mapping used by the Shadow Virtqueue */
VhostIOVATree *iova_tree;
+ VhostVDPAShared *shared;
GPtrArray *shadow_vqs;
const VhostShadowVirtqueueOps *shadow_vq_ops;
void *shadow_vq_ops_opaque;
diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c
index d0614d7954..8b661b9e6d 100644
--- a/net/vhost-vdpa.c
+++ b/net/vhost-vdpa.c
@@ -240,6 +240,10 @@ static void vhost_vdpa_cleanup(NetClientState *nc)
qemu_close(s->vhost_vdpa.device_fd);
s->vhost_vdpa.device_fd = -1;
}
+ if (s->vhost_vdpa.index != 0) {
+ return;
+ }
+ g_free(s->vhost_vdpa.shared);
}
/** Dummy SetSteeringEBPF to support RSS for vhost-vdpa backend */
@@ -1661,6 +1665,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
bool svq,
struct vhost_vdpa_iova_range iova_range,
uint64_t features,
+ VhostVDPAShared *shared,
Error **errp)
{
NetClientState *nc = NULL;
@@ -1696,6 +1701,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
if (queue_pair_index == 0) {
vhost_vdpa_net_valid_svq_features(features,
&s->vhost_vdpa.migration_blocker);
+ s->vhost_vdpa.shared = g_new0(VhostVDPAShared, 1);
} else if (!is_datapath) {
s->cvq_cmd_out_buffer = mmap(NULL, vhost_vdpa_net_cvq_cmd_page_len(),
PROT_READ | PROT_WRITE,
@@ -1708,11 +1714,16 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
s->vhost_vdpa.shadow_vq_ops_opaque = s;
s->cvq_isolated = cvq_isolated;
}
+ if (queue_pair_index != 0) {
+ s->vhost_vdpa.shared = shared;
+ }
+
ret = vhost_vdpa_add(nc, (void *)&s->vhost_vdpa, queue_pair_index, nvqs);
if (ret) {
qemu_del_net_client(nc);
return NULL;
}
+
return nc;
}
@@ -1824,17 +1835,26 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,
ncs = g_malloc0(sizeof(*ncs) * queue_pairs);
for (i = 0; i < queue_pairs; i++) {
+ VhostVDPAShared *shared = NULL;
+
+ if (i) {
+ shared = DO_UPCAST(VhostVDPAState, nc, ncs[0])->vhost_vdpa.shared;
+ }
ncs[i] = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name,
vdpa_device_fd, i, 2, true, opts->x_svq,
- iova_range, features, errp);
+ iova_range, features, shared, errp);
if (!ncs[i])
goto err;
}
if (has_cvq) {
+ VhostVDPAState *s0 = DO_UPCAST(VhostVDPAState, nc, ncs[0]);
+ VhostVDPAShared *shared = s0->vhost_vdpa.shared;
+
nc = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name,
vdpa_device_fd, i, 1, false,
- opts->x_svq, iova_range, features, errp);
+ opts->x_svq, iova_range, features, shared,
+ errp);
if (!nc)
goto err;
}
--
2.39.3
^ permalink raw reply related [flat|nested] 21+ messages in thread* Re: [PATCH 9.0 01/13] vdpa: add VhostVDPAShared
2023-11-24 17:14 ` [PATCH 9.0 01/13] vdpa: add VhostVDPAShared Eugenio Pérez
@ 2023-12-01 5:35 ` Jason Wang
2023-12-01 6:41 ` Eugenio Perez Martin
0 siblings, 1 reply; 21+ messages in thread
From: Jason Wang @ 2023-12-01 5:35 UTC (permalink / raw)
To: Eugenio Pérez
Cc: qemu-devel, Parav Pandit, si-wei.liu, Stefano Garzarella,
Zhu Lingshan, Lei Yang, Michael S. Tsirkin, Dragos Tatulea,
Laurent Vivier
On Sat, Nov 25, 2023 at 1:14 AM Eugenio Pérez <eperezma@redhat.com> wrote:
>
> It will hold properties shared among all vhost_vdpa instances associated
> with of the same device. For example, we just need one iova_tree or one
> memory listener for the entire device.
>
> Next patches will register the vhost_vdpa memory listener at the
> beginning of the VM migration at the destination. This enables QEMU to
> map the memory to the device before stopping the VM at the source,
> instead of doing while both source and destination are stopped, thus
> minimizing the downtime.
>
> However, the destination QEMU is unaware of which vhost_vdpa struct will
> register its memory_listener. If the source guest has CVQ enabled, it
> will be the one associated with the CVQ. Otherwise, it will be the
> first one.
>
> Save the memory operations related members in a common place rather than
> always in the first / last vhost_vdpa.
Great.
Patch looks good but I think we probably need a better name like
VhostVDPAParent?
And it would be better in the future if we can convert it to QOM.
Thanks
>
> Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> ---
> include/hw/virtio/vhost-vdpa.h | 5 +++++
> net/vhost-vdpa.c | 24 ++++++++++++++++++++++--
> 2 files changed, 27 insertions(+), 2 deletions(-)
>
> diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h
> index 5407d54fd7..eb1a56d75a 100644
> --- a/include/hw/virtio/vhost-vdpa.h
> +++ b/include/hw/virtio/vhost-vdpa.h
> @@ -30,6 +30,10 @@ typedef struct VhostVDPAHostNotifier {
> void *addr;
> } VhostVDPAHostNotifier;
>
> +/* Info shared by all vhost_vdpa device models */
> +typedef struct vhost_vdpa_shared {
> +} VhostVDPAShared;
> +
> typedef struct vhost_vdpa {
> int device_fd;
> int index;
> @@ -46,6 +50,7 @@ typedef struct vhost_vdpa {
> bool suspended;
> /* IOVA mapping used by the Shadow Virtqueue */
> VhostIOVATree *iova_tree;
> + VhostVDPAShared *shared;
> GPtrArray *shadow_vqs;
> const VhostShadowVirtqueueOps *shadow_vq_ops;
> void *shadow_vq_ops_opaque;
> diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c
> index d0614d7954..8b661b9e6d 100644
> --- a/net/vhost-vdpa.c
> +++ b/net/vhost-vdpa.c
> @@ -240,6 +240,10 @@ static void vhost_vdpa_cleanup(NetClientState *nc)
> qemu_close(s->vhost_vdpa.device_fd);
> s->vhost_vdpa.device_fd = -1;
> }
> + if (s->vhost_vdpa.index != 0) {
> + return;
> + }
> + g_free(s->vhost_vdpa.shared);
> }
>
> /** Dummy SetSteeringEBPF to support RSS for vhost-vdpa backend */
> @@ -1661,6 +1665,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
> bool svq,
> struct vhost_vdpa_iova_range iova_range,
> uint64_t features,
> + VhostVDPAShared *shared,
> Error **errp)
> {
> NetClientState *nc = NULL;
> @@ -1696,6 +1701,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
> if (queue_pair_index == 0) {
> vhost_vdpa_net_valid_svq_features(features,
> &s->vhost_vdpa.migration_blocker);
> + s->vhost_vdpa.shared = g_new0(VhostVDPAShared, 1);
> } else if (!is_datapath) {
> s->cvq_cmd_out_buffer = mmap(NULL, vhost_vdpa_net_cvq_cmd_page_len(),
> PROT_READ | PROT_WRITE,
> @@ -1708,11 +1714,16 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
> s->vhost_vdpa.shadow_vq_ops_opaque = s;
> s->cvq_isolated = cvq_isolated;
> }
> + if (queue_pair_index != 0) {
> + s->vhost_vdpa.shared = shared;
> + }
> +
> ret = vhost_vdpa_add(nc, (void *)&s->vhost_vdpa, queue_pair_index, nvqs);
> if (ret) {
> qemu_del_net_client(nc);
> return NULL;
> }
> +
> return nc;
> }
>
> @@ -1824,17 +1835,26 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,
> ncs = g_malloc0(sizeof(*ncs) * queue_pairs);
>
> for (i = 0; i < queue_pairs; i++) {
> + VhostVDPAShared *shared = NULL;
> +
> + if (i) {
> + shared = DO_UPCAST(VhostVDPAState, nc, ncs[0])->vhost_vdpa.shared;
> + }
> ncs[i] = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name,
> vdpa_device_fd, i, 2, true, opts->x_svq,
> - iova_range, features, errp);
> + iova_range, features, shared, errp);
> if (!ncs[i])
> goto err;
> }
>
> if (has_cvq) {
> + VhostVDPAState *s0 = DO_UPCAST(VhostVDPAState, nc, ncs[0]);
> + VhostVDPAShared *shared = s0->vhost_vdpa.shared;
> +
> nc = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name,
> vdpa_device_fd, i, 1, false,
> - opts->x_svq, iova_range, features, errp);
> + opts->x_svq, iova_range, features, shared,
> + errp);
> if (!nc)
> goto err;
> }
> --
> 2.39.3
>
^ permalink raw reply [flat|nested] 21+ messages in thread* Re: [PATCH 9.0 01/13] vdpa: add VhostVDPAShared
2023-12-01 5:35 ` Jason Wang
@ 2023-12-01 6:41 ` Eugenio Perez Martin
0 siblings, 0 replies; 21+ messages in thread
From: Eugenio Perez Martin @ 2023-12-01 6:41 UTC (permalink / raw)
To: Jason Wang
Cc: qemu-devel, Parav Pandit, si-wei.liu, Stefano Garzarella,
Zhu Lingshan, Lei Yang, Michael S. Tsirkin, Dragos Tatulea,
Laurent Vivier
On Fri, Dec 1, 2023 at 6:35 AM Jason Wang <jasowang@redhat.com> wrote:
>
> On Sat, Nov 25, 2023 at 1:14 AM Eugenio Pérez <eperezma@redhat.com> wrote:
> >
> > It will hold properties shared among all vhost_vdpa instances associated
> > with of the same device. For example, we just need one iova_tree or one
> > memory listener for the entire device.
> >
> > Next patches will register the vhost_vdpa memory listener at the
> > beginning of the VM migration at the destination. This enables QEMU to
> > map the memory to the device before stopping the VM at the source,
> > instead of doing while both source and destination are stopped, thus
> > minimizing the downtime.
> >
> > However, the destination QEMU is unaware of which vhost_vdpa struct will
> > register its memory_listener. If the source guest has CVQ enabled, it
> > will be the one associated with the CVQ. Otherwise, it will be the
> > first one.
> >
> > Save the memory operations related members in a common place rather than
> > always in the first / last vhost_vdpa.
>
> Great.
>
> Patch looks good but I think we probably need a better name like
> VhostVDPAParent?
>
Sure, I'm ok with the renaming. I'll change it for v2.
Thanks!
> And it would be better in the future if we can convert it to QOM.
>
> Thanks
>
> >
> > Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> > ---
> > include/hw/virtio/vhost-vdpa.h | 5 +++++
> > net/vhost-vdpa.c | 24 ++++++++++++++++++++++--
> > 2 files changed, 27 insertions(+), 2 deletions(-)
> >
> > diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h
> > index 5407d54fd7..eb1a56d75a 100644
> > --- a/include/hw/virtio/vhost-vdpa.h
> > +++ b/include/hw/virtio/vhost-vdpa.h
> > @@ -30,6 +30,10 @@ typedef struct VhostVDPAHostNotifier {
> > void *addr;
> > } VhostVDPAHostNotifier;
> >
> > +/* Info shared by all vhost_vdpa device models */
> > +typedef struct vhost_vdpa_shared {
> > +} VhostVDPAShared;
> > +
> > typedef struct vhost_vdpa {
> > int device_fd;
> > int index;
> > @@ -46,6 +50,7 @@ typedef struct vhost_vdpa {
> > bool suspended;
> > /* IOVA mapping used by the Shadow Virtqueue */
> > VhostIOVATree *iova_tree;
> > + VhostVDPAShared *shared;
> > GPtrArray *shadow_vqs;
> > const VhostShadowVirtqueueOps *shadow_vq_ops;
> > void *shadow_vq_ops_opaque;
> > diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c
> > index d0614d7954..8b661b9e6d 100644
> > --- a/net/vhost-vdpa.c
> > +++ b/net/vhost-vdpa.c
> > @@ -240,6 +240,10 @@ static void vhost_vdpa_cleanup(NetClientState *nc)
> > qemu_close(s->vhost_vdpa.device_fd);
> > s->vhost_vdpa.device_fd = -1;
> > }
> > + if (s->vhost_vdpa.index != 0) {
> > + return;
> > + }
> > + g_free(s->vhost_vdpa.shared);
> > }
> >
> > /** Dummy SetSteeringEBPF to support RSS for vhost-vdpa backend */
> > @@ -1661,6 +1665,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
> > bool svq,
> > struct vhost_vdpa_iova_range iova_range,
> > uint64_t features,
> > + VhostVDPAShared *shared,
> > Error **errp)
> > {
> > NetClientState *nc = NULL;
> > @@ -1696,6 +1701,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
> > if (queue_pair_index == 0) {
> > vhost_vdpa_net_valid_svq_features(features,
> > &s->vhost_vdpa.migration_blocker);
> > + s->vhost_vdpa.shared = g_new0(VhostVDPAShared, 1);
> > } else if (!is_datapath) {
> > s->cvq_cmd_out_buffer = mmap(NULL, vhost_vdpa_net_cvq_cmd_page_len(),
> > PROT_READ | PROT_WRITE,
> > @@ -1708,11 +1714,16 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
> > s->vhost_vdpa.shadow_vq_ops_opaque = s;
> > s->cvq_isolated = cvq_isolated;
> > }
> > + if (queue_pair_index != 0) {
> > + s->vhost_vdpa.shared = shared;
> > + }
> > +
> > ret = vhost_vdpa_add(nc, (void *)&s->vhost_vdpa, queue_pair_index, nvqs);
> > if (ret) {
> > qemu_del_net_client(nc);
> > return NULL;
> > }
> > +
> > return nc;
> > }
> >
> > @@ -1824,17 +1835,26 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name,
> > ncs = g_malloc0(sizeof(*ncs) * queue_pairs);
> >
> > for (i = 0; i < queue_pairs; i++) {
> > + VhostVDPAShared *shared = NULL;
> > +
> > + if (i) {
> > + shared = DO_UPCAST(VhostVDPAState, nc, ncs[0])->vhost_vdpa.shared;
> > + }
> > ncs[i] = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name,
> > vdpa_device_fd, i, 2, true, opts->x_svq,
> > - iova_range, features, errp);
> > + iova_range, features, shared, errp);
> > if (!ncs[i])
> > goto err;
> > }
> >
> > if (has_cvq) {
> > + VhostVDPAState *s0 = DO_UPCAST(VhostVDPAState, nc, ncs[0]);
> > + VhostVDPAShared *shared = s0->vhost_vdpa.shared;
> > +
> > nc = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name,
> > vdpa_device_fd, i, 1, false,
> > - opts->x_svq, iova_range, features, errp);
> > + opts->x_svq, iova_range, features, shared,
> > + errp);
> > if (!nc)
> > goto err;
> > }
> > --
> > 2.39.3
> >
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* [PATCH 9.0 02/13] vdpa: move iova tree to the shared struct
2023-11-24 17:14 [PATCH 9.0 00/13] Consolidate common vdpa members in VhostVDPAShared Eugenio Pérez
2023-11-24 17:14 ` [PATCH 9.0 01/13] vdpa: add VhostVDPAShared Eugenio Pérez
@ 2023-11-24 17:14 ` Eugenio Pérez
2023-11-24 17:14 ` [PATCH 9.0 03/13] vdpa: move iova_range to vhost_vdpa_shared Eugenio Pérez
` (12 subsequent siblings)
14 siblings, 0 replies; 21+ messages in thread
From: Eugenio Pérez @ 2023-11-24 17:14 UTC (permalink / raw)
To: qemu-devel
Cc: Parav Pandit, si-wei.liu, Stefano Garzarella, Zhu Lingshan,
Lei Yang, Michael S. Tsirkin, Jason Wang, Dragos Tatulea,
Laurent Vivier
Next patches will register the vhost_vdpa memory listener while the VM
is migrating at the destination, so we can map the memory to the device
before stopping the VM at the source. The main goal is to reduce the
downtime.
However, the destination QEMU is unaware of which vhost_vdpa device will
register its memory_listener. If the source guest has CVQ enabled, it
will be the CVQ device. Otherwise, it will be the first one.
Move the iova tree to VhostVDPAShared so all vhost_vdpa can use it,
rather than always in the first or last vhost_vdpa.
Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
include/hw/virtio/vhost-vdpa.h | 4 +--
hw/virtio/vhost-vdpa.c | 19 ++++++------
net/vhost-vdpa.c | 54 +++++++++++++++-------------------
3 files changed, 35 insertions(+), 42 deletions(-)
diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h
index eb1a56d75a..ac036055d3 100644
--- a/include/hw/virtio/vhost-vdpa.h
+++ b/include/hw/virtio/vhost-vdpa.h
@@ -32,6 +32,8 @@ typedef struct VhostVDPAHostNotifier {
/* Info shared by all vhost_vdpa device models */
typedef struct vhost_vdpa_shared {
+ /* IOVA mapping used by the Shadow Virtqueue */
+ VhostIOVATree *iova_tree;
} VhostVDPAShared;
typedef struct vhost_vdpa {
@@ -48,8 +50,6 @@ typedef struct vhost_vdpa {
bool shadow_data;
/* Device suspended successfully */
bool suspended;
- /* IOVA mapping used by the Shadow Virtqueue */
- VhostIOVATree *iova_tree;
VhostVDPAShared *shared;
GPtrArray *shadow_vqs;
const VhostShadowVirtqueueOps *shadow_vq_ops;
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 819b2d811a..9cee38cb6d 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -358,7 +358,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
mem_region.size = int128_get64(llsize) - 1,
mem_region.perm = IOMMU_ACCESS_FLAG(true, section->readonly),
- r = vhost_iova_tree_map_alloc(v->iova_tree, &mem_region);
+ r = vhost_iova_tree_map_alloc(v->shared->iova_tree, &mem_region);
if (unlikely(r != IOVA_OK)) {
error_report("Can't allocate a mapping (%d)", r);
goto fail;
@@ -379,7 +379,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
fail_map:
if (v->shadow_data) {
- vhost_iova_tree_remove(v->iova_tree, mem_region);
+ vhost_iova_tree_remove(v->shared->iova_tree, mem_region);
}
fail:
@@ -441,13 +441,13 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
.size = int128_get64(llsize) - 1,
};
- result = vhost_iova_tree_find_iova(v->iova_tree, &mem_region);
+ result = vhost_iova_tree_find_iova(v->shared->iova_tree, &mem_region);
if (!result) {
/* The memory listener map wasn't mapped */
return;
}
iova = result->iova;
- vhost_iova_tree_remove(v->iova_tree, *result);
+ vhost_iova_tree_remove(v->shared->iova_tree, *result);
}
vhost_vdpa_iotlb_batch_begin_once(v);
/*
@@ -1059,7 +1059,8 @@ static void vhost_vdpa_svq_unmap_ring(struct vhost_vdpa *v, hwaddr addr)
const DMAMap needle = {
.translated_addr = addr,
};
- const DMAMap *result = vhost_iova_tree_find_iova(v->iova_tree, &needle);
+ const DMAMap *result = vhost_iova_tree_find_iova(v->shared->iova_tree,
+ &needle);
hwaddr size;
int r;
@@ -1075,7 +1076,7 @@ static void vhost_vdpa_svq_unmap_ring(struct vhost_vdpa *v, hwaddr addr)
return;
}
- vhost_iova_tree_remove(v->iova_tree, *result);
+ vhost_iova_tree_remove(v->shared->iova_tree, *result);
}
static void vhost_vdpa_svq_unmap_rings(struct vhost_dev *dev,
@@ -1103,7 +1104,7 @@ static bool vhost_vdpa_svq_map_ring(struct vhost_vdpa *v, DMAMap *needle,
{
int r;
- r = vhost_iova_tree_map_alloc(v->iova_tree, needle);
+ r = vhost_iova_tree_map_alloc(v->shared->iova_tree, needle);
if (unlikely(r != IOVA_OK)) {
error_setg(errp, "Cannot allocate iova (%d)", r);
return false;
@@ -1115,7 +1116,7 @@ static bool vhost_vdpa_svq_map_ring(struct vhost_vdpa *v, DMAMap *needle,
needle->perm == IOMMU_RO);
if (unlikely(r != 0)) {
error_setg_errno(errp, -r, "Cannot map region to device");
- vhost_iova_tree_remove(v->iova_tree, *needle);
+ vhost_iova_tree_remove(v->shared->iova_tree, *needle);
}
return r == 0;
@@ -1216,7 +1217,7 @@ static bool vhost_vdpa_svqs_start(struct vhost_dev *dev)
goto err;
}
- vhost_svq_start(svq, dev->vdev, vq, v->iova_tree);
+ vhost_svq_start(svq, dev->vdev, vq, v->shared->iova_tree);
ok = vhost_vdpa_svq_map_rings(dev, svq, &addr, &err);
if (unlikely(!ok)) {
goto err_map;
diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c
index 8b661b9e6d..10703e5833 100644
--- a/net/vhost-vdpa.c
+++ b/net/vhost-vdpa.c
@@ -354,8 +354,8 @@ static void vhost_vdpa_net_data_start_first(VhostVDPAState *s)
migration_add_notifier(&s->migration_state,
vdpa_net_migration_state_notifier);
if (v->shadow_vqs_enabled) {
- v->iova_tree = vhost_iova_tree_new(v->iova_range.first,
- v->iova_range.last);
+ v->shared->iova_tree = vhost_iova_tree_new(v->iova_range.first,
+ v->iova_range.last);
}
}
@@ -380,11 +380,6 @@ static int vhost_vdpa_net_data_start(NetClientState *nc)
return 0;
}
- if (v->shadow_vqs_enabled) {
- VhostVDPAState *s0 = vhost_vdpa_net_first_nc_vdpa(s);
- v->iova_tree = s0->vhost_vdpa.iova_tree;
- }
-
return 0;
}
@@ -417,9 +412,8 @@ static void vhost_vdpa_net_client_stop(NetClientState *nc)
dev = s->vhost_vdpa.dev;
if (dev->vq_index + dev->nvqs == dev->vq_index_end) {
- g_clear_pointer(&s->vhost_vdpa.iova_tree, vhost_iova_tree_delete);
- } else {
- s->vhost_vdpa.iova_tree = NULL;
+ g_clear_pointer(&s->vhost_vdpa.shared->iova_tree,
+ vhost_iova_tree_delete);
}
}
@@ -474,7 +468,7 @@ static int vhost_vdpa_set_address_space_id(struct vhost_vdpa *v,
static void vhost_vdpa_cvq_unmap_buf(struct vhost_vdpa *v, void *addr)
{
- VhostIOVATree *tree = v->iova_tree;
+ VhostIOVATree *tree = v->shared->iova_tree;
DMAMap needle = {
/*
* No need to specify size or to look for more translations since
@@ -508,7 +502,7 @@ static int vhost_vdpa_cvq_map_buf(struct vhost_vdpa *v, void *buf, size_t size,
map.translated_addr = (hwaddr)(uintptr_t)buf;
map.size = size - 1;
map.perm = write ? IOMMU_RW : IOMMU_RO,
- r = vhost_iova_tree_map_alloc(v->iova_tree, &map);
+ r = vhost_iova_tree_map_alloc(v->shared->iova_tree, &map);
if (unlikely(r != IOVA_OK)) {
error_report("Cannot map injected element");
return r;
@@ -523,7 +517,7 @@ static int vhost_vdpa_cvq_map_buf(struct vhost_vdpa *v, void *buf, size_t size,
return 0;
dma_map_err:
- vhost_iova_tree_remove(v->iova_tree, map);
+ vhost_iova_tree_remove(v->shared->iova_tree, map);
return r;
}
@@ -583,24 +577,22 @@ out:
return 0;
}
- if (s0->vhost_vdpa.iova_tree) {
- /*
- * SVQ is already configured for all virtqueues. Reuse IOVA tree for
- * simplicity, whether CVQ shares ASID with guest or not, because:
- * - Memory listener need access to guest's memory addresses allocated
- * in the IOVA tree.
- * - There should be plenty of IOVA address space for both ASID not to
- * worry about collisions between them. Guest's translations are
- * still validated with virtio virtqueue_pop so there is no risk for
- * the guest to access memory that it shouldn't.
- *
- * To allocate a iova tree per ASID is doable but it complicates the
- * code and it is not worth it for the moment.
- */
- v->iova_tree = s0->vhost_vdpa.iova_tree;
- } else {
- v->iova_tree = vhost_iova_tree_new(v->iova_range.first,
- v->iova_range.last);
+ /*
+ * If other vhost_vdpa already have an iova_tree, reuse it for simplicity,
+ * whether CVQ shares ASID with guest or not, because:
+ * - Memory listener need access to guest's memory addresses allocated in
+ * the IOVA tree.
+ * - There should be plenty of IOVA address space for both ASID not to
+ * worry about collisions between them. Guest's translations are still
+ * validated with virtio virtqueue_pop so there is no risk for the guest
+ * to access memory that it shouldn't.
+ *
+ * To allocate a iova tree per ASID is doable but it complicates the code
+ * and it is not worth it for the moment.
+ */
+ if (!v->shared->iova_tree) {
+ v->shared->iova_tree = vhost_iova_tree_new(v->iova_range.first,
+ v->iova_range.last);
}
r = vhost_vdpa_cvq_map_buf(&s->vhost_vdpa, s->cvq_cmd_out_buffer,
--
2.39.3
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 9.0 03/13] vdpa: move iova_range to vhost_vdpa_shared
2023-11-24 17:14 [PATCH 9.0 00/13] Consolidate common vdpa members in VhostVDPAShared Eugenio Pérez
2023-11-24 17:14 ` [PATCH 9.0 01/13] vdpa: add VhostVDPAShared Eugenio Pérez
2023-11-24 17:14 ` [PATCH 9.0 02/13] vdpa: move iova tree to the shared struct Eugenio Pérez
@ 2023-11-24 17:14 ` Eugenio Pérez
2023-11-24 17:14 ` [PATCH 9.0 04/13] vdpa: move shadow_data " Eugenio Pérez
` (11 subsequent siblings)
14 siblings, 0 replies; 21+ messages in thread
From: Eugenio Pérez @ 2023-11-24 17:14 UTC (permalink / raw)
To: qemu-devel
Cc: Parav Pandit, si-wei.liu, Stefano Garzarella, Zhu Lingshan,
Lei Yang, Michael S. Tsirkin, Jason Wang, Dragos Tatulea,
Laurent Vivier
Next patches will register the vhost_vdpa memory listener while the VM
is migrating at the destination, so we can map the memory to the device
before stopping the VM at the source. The main goal is to reduce the
downtime.
However, the destination QEMU is unaware of which vhost_vdpa device will
register its memory_listener. If the source guest has CVQ enabled, it
will be the CVQ device. Otherwise, it will be the first one.
Move the iova range to VhostVDPAShared so all vhost_vdpa can use it,
rather than always in the first or last vhost_vdpa.
Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
include/hw/virtio/vhost-vdpa.h | 3 ++-
hw/virtio/vdpa-dev.c | 5 ++++-
hw/virtio/vhost-vdpa.c | 16 ++++++++++------
net/vhost-vdpa.c | 10 +++++-----
4 files changed, 21 insertions(+), 13 deletions(-)
diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h
index ac036055d3..8d52a7e498 100644
--- a/include/hw/virtio/vhost-vdpa.h
+++ b/include/hw/virtio/vhost-vdpa.h
@@ -32,6 +32,8 @@ typedef struct VhostVDPAHostNotifier {
/* Info shared by all vhost_vdpa device models */
typedef struct vhost_vdpa_shared {
+ struct vhost_vdpa_iova_range iova_range;
+
/* IOVA mapping used by the Shadow Virtqueue */
VhostIOVATree *iova_tree;
} VhostVDPAShared;
@@ -43,7 +45,6 @@ typedef struct vhost_vdpa {
bool iotlb_batch_begin_sent;
uint32_t address_space_id;
MemoryListener listener;
- struct vhost_vdpa_iova_range iova_range;
uint64_t acked_features;
bool shadow_vqs_enabled;
/* Vdpa must send shadow addresses as IOTLB key for data queues, not GPA */
diff --git a/hw/virtio/vdpa-dev.c b/hw/virtio/vdpa-dev.c
index f22d5d5bc0..457960d28a 100644
--- a/hw/virtio/vdpa-dev.c
+++ b/hw/virtio/vdpa-dev.c
@@ -114,7 +114,8 @@ static void vhost_vdpa_device_realize(DeviceState *dev, Error **errp)
strerror(-ret));
goto free_vqs;
}
- v->vdpa.iova_range = iova_range;
+ v->vdpa.shared = g_new0(VhostVDPAShared, 1);
+ v->vdpa.shared->iova_range = iova_range;
ret = vhost_dev_init(&v->dev, &v->vdpa, VHOST_BACKEND_TYPE_VDPA, 0, NULL);
if (ret < 0) {
@@ -162,6 +163,7 @@ vhost_cleanup:
vhost_dev_cleanup(&v->dev);
free_vqs:
g_free(vqs);
+ g_free(v->vdpa.shared);
out:
qemu_close(v->vhostfd);
v->vhostfd = -1;
@@ -184,6 +186,7 @@ static void vhost_vdpa_device_unrealize(DeviceState *dev)
g_free(s->config);
g_free(s->dev.vqs);
vhost_dev_cleanup(&s->dev);
+ g_free(s->vdpa.shared);
qemu_close(s->vhostfd);
s->vhostfd = -1;
}
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 9cee38cb6d..2bceadd118 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -213,10 +213,10 @@ static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb)
RCU_READ_LOCK_GUARD();
/* check if RAM section out of device range */
llend = int128_add(int128_makes64(iotlb->addr_mask), int128_makes64(iova));
- if (int128_gt(llend, int128_make64(v->iova_range.last))) {
+ if (int128_gt(llend, int128_make64(v->shared->iova_range.last))) {
error_report("RAM section out of device range (max=0x%" PRIx64
", end addr=0x%" PRIx64 ")",
- v->iova_range.last, int128_get64(llend));
+ v->shared->iova_range.last, int128_get64(llend));
return;
}
@@ -316,8 +316,10 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
int page_size = qemu_target_page_size();
int page_mask = -page_size;
- if (vhost_vdpa_listener_skipped_section(section, v->iova_range.first,
- v->iova_range.last, page_mask)) {
+ if (vhost_vdpa_listener_skipped_section(section,
+ v->shared->iova_range.first,
+ v->shared->iova_range.last,
+ page_mask)) {
return;
}
if (memory_region_is_iommu(section->mr)) {
@@ -403,8 +405,10 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
int page_size = qemu_target_page_size();
int page_mask = -page_size;
- if (vhost_vdpa_listener_skipped_section(section, v->iova_range.first,
- v->iova_range.last, page_mask)) {
+ if (vhost_vdpa_listener_skipped_section(section,
+ v->shared->iova_range.first,
+ v->shared->iova_range.last,
+ page_mask)) {
return;
}
if (memory_region_is_iommu(section->mr)) {
diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c
index 10703e5833..7be2c30ad3 100644
--- a/net/vhost-vdpa.c
+++ b/net/vhost-vdpa.c
@@ -354,8 +354,8 @@ static void vhost_vdpa_net_data_start_first(VhostVDPAState *s)
migration_add_notifier(&s->migration_state,
vdpa_net_migration_state_notifier);
if (v->shadow_vqs_enabled) {
- v->shared->iova_tree = vhost_iova_tree_new(v->iova_range.first,
- v->iova_range.last);
+ v->shared->iova_tree = vhost_iova_tree_new(v->shared->iova_range.first,
+ v->shared->iova_range.last);
}
}
@@ -591,8 +591,8 @@ out:
* and it is not worth it for the moment.
*/
if (!v->shared->iova_tree) {
- v->shared->iova_tree = vhost_iova_tree_new(v->iova_range.first,
- v->iova_range.last);
+ v->shared->iova_tree = vhost_iova_tree_new(v->shared->iova_range.first,
+ v->shared->iova_range.last);
}
r = vhost_vdpa_cvq_map_buf(&s->vhost_vdpa, s->cvq_cmd_out_buffer,
@@ -1688,12 +1688,12 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
s->always_svq = svq;
s->migration_state.notify = NULL;
s->vhost_vdpa.shadow_vqs_enabled = svq;
- s->vhost_vdpa.iova_range = iova_range;
s->vhost_vdpa.shadow_data = svq;
if (queue_pair_index == 0) {
vhost_vdpa_net_valid_svq_features(features,
&s->vhost_vdpa.migration_blocker);
s->vhost_vdpa.shared = g_new0(VhostVDPAShared, 1);
+ s->vhost_vdpa.shared->iova_range = iova_range;
} else if (!is_datapath) {
s->cvq_cmd_out_buffer = mmap(NULL, vhost_vdpa_net_cvq_cmd_page_len(),
PROT_READ | PROT_WRITE,
--
2.39.3
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 9.0 04/13] vdpa: move shadow_data to vhost_vdpa_shared
2023-11-24 17:14 [PATCH 9.0 00/13] Consolidate common vdpa members in VhostVDPAShared Eugenio Pérez
` (2 preceding siblings ...)
2023-11-24 17:14 ` [PATCH 9.0 03/13] vdpa: move iova_range to vhost_vdpa_shared Eugenio Pérez
@ 2023-11-24 17:14 ` Eugenio Pérez
2023-12-06 6:04 ` Si-Wei Liu
2023-11-24 17:14 ` [PATCH 9.0 05/13] vdpa: use vdpa shared for tracing Eugenio Pérez
` (10 subsequent siblings)
14 siblings, 1 reply; 21+ messages in thread
From: Eugenio Pérez @ 2023-11-24 17:14 UTC (permalink / raw)
To: qemu-devel
Cc: Parav Pandit, si-wei.liu, Stefano Garzarella, Zhu Lingshan,
Lei Yang, Michael S. Tsirkin, Jason Wang, Dragos Tatulea,
Laurent Vivier
Next patches will register the vhost_vdpa memory listener while the VM
is migrating at the destination, so we can map the memory to the device
before stopping the VM at the source. The main goal is to reduce the
downtime.
However, the destination QEMU is unaware of which vhost_vdpa device will
register its memory_listener. If the source guest has CVQ enabled, it
will be the CVQ device. Otherwise, it will be the first one.
Move the shadow_data member to VhostVDPAShared so all vhost_vdpa can use
it, rather than always in the first or last vhost_vdpa.
Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
v1 from RFC:
* Fix vhost_vdpa_net_cvq_start checking for always_svq instead of
shadow_data. This could cause CVQ not being shadowed if
vhost_vdpa_net_cvq_start was called in the middle of a migration.
---
include/hw/virtio/vhost-vdpa.h | 5 +++--
hw/virtio/vhost-vdpa.c | 6 +++---
net/vhost-vdpa.c | 23 ++++++-----------------
3 files changed, 12 insertions(+), 22 deletions(-)
diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h
index 8d52a7e498..01e0f25e27 100644
--- a/include/hw/virtio/vhost-vdpa.h
+++ b/include/hw/virtio/vhost-vdpa.h
@@ -36,6 +36,9 @@ typedef struct vhost_vdpa_shared {
/* IOVA mapping used by the Shadow Virtqueue */
VhostIOVATree *iova_tree;
+
+ /* Vdpa must send shadow addresses as IOTLB key for data queues, not GPA */
+ bool shadow_data;
} VhostVDPAShared;
typedef struct vhost_vdpa {
@@ -47,8 +50,6 @@ typedef struct vhost_vdpa {
MemoryListener listener;
uint64_t acked_features;
bool shadow_vqs_enabled;
- /* Vdpa must send shadow addresses as IOTLB key for data queues, not GPA */
- bool shadow_data;
/* Device suspended successfully */
bool suspended;
VhostVDPAShared *shared;
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 2bceadd118..ec028e4c56 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -353,7 +353,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
vaddr, section->readonly);
llsize = int128_sub(llend, int128_make64(iova));
- if (v->shadow_data) {
+ if (v->shared->shadow_data) {
int r;
mem_region.translated_addr = (hwaddr)(uintptr_t)vaddr,
@@ -380,7 +380,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
return;
fail_map:
- if (v->shadow_data) {
+ if (v->shared->shadow_data) {
vhost_iova_tree_remove(v->shared->iova_tree, mem_region);
}
@@ -435,7 +435,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
llsize = int128_sub(llend, int128_make64(iova));
- if (v->shadow_data) {
+ if (v->shared->shadow_data) {
const DMAMap *result;
const void *vaddr = memory_region_get_ram_ptr(section->mr) +
section->offset_within_region +
diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c
index 7be2c30ad3..2376d9989a 100644
--- a/net/vhost-vdpa.c
+++ b/net/vhost-vdpa.c
@@ -290,15 +290,6 @@ static ssize_t vhost_vdpa_receive(NetClientState *nc, const uint8_t *buf,
return size;
}
-/** From any vdpa net client, get the netclient of the first queue pair */
-static VhostVDPAState *vhost_vdpa_net_first_nc_vdpa(VhostVDPAState *s)
-{
- NICState *nic = qemu_get_nic(s->nc.peer);
- NetClientState *nc0 = qemu_get_peer(nic->ncs, 0);
-
- return DO_UPCAST(VhostVDPAState, nc, nc0);
-}
-
static void vhost_vdpa_net_log_global_enable(VhostVDPAState *s, bool enable)
{
struct vhost_vdpa *v = &s->vhost_vdpa;
@@ -369,10 +360,10 @@ static int vhost_vdpa_net_data_start(NetClientState *nc)
if (s->always_svq ||
migration_is_setup_or_active(migrate_get_current()->state)) {
v->shadow_vqs_enabled = true;
- v->shadow_data = true;
+ v->shared->shadow_data = true;
} else {
v->shadow_vqs_enabled = false;
- v->shadow_data = false;
+ v->shared->shadow_data = false;
}
if (v->index == 0) {
@@ -523,7 +514,7 @@ dma_map_err:
static int vhost_vdpa_net_cvq_start(NetClientState *nc)
{
- VhostVDPAState *s, *s0;
+ VhostVDPAState *s;
struct vhost_vdpa *v;
int64_t cvq_group;
int r;
@@ -534,12 +525,10 @@ static int vhost_vdpa_net_cvq_start(NetClientState *nc)
s = DO_UPCAST(VhostVDPAState, nc, nc);
v = &s->vhost_vdpa;
- s0 = vhost_vdpa_net_first_nc_vdpa(s);
- v->shadow_data = s0->vhost_vdpa.shadow_vqs_enabled;
- v->shadow_vqs_enabled = s0->vhost_vdpa.shadow_vqs_enabled;
+ v->shadow_vqs_enabled = v->shared->shadow_data;
s->vhost_vdpa.address_space_id = VHOST_VDPA_GUEST_PA_ASID;
- if (s->vhost_vdpa.shadow_data) {
+ if (v->shared->shadow_data) {
/* SVQ is already configured for all virtqueues */
goto out;
}
@@ -1688,12 +1677,12 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
s->always_svq = svq;
s->migration_state.notify = NULL;
s->vhost_vdpa.shadow_vqs_enabled = svq;
- s->vhost_vdpa.shadow_data = svq;
if (queue_pair_index == 0) {
vhost_vdpa_net_valid_svq_features(features,
&s->vhost_vdpa.migration_blocker);
s->vhost_vdpa.shared = g_new0(VhostVDPAShared, 1);
s->vhost_vdpa.shared->iova_range = iova_range;
+ s->vhost_vdpa.shared->shadow_data = svq;
} else if (!is_datapath) {
s->cvq_cmd_out_buffer = mmap(NULL, vhost_vdpa_net_cvq_cmd_page_len(),
PROT_READ | PROT_WRITE,
--
2.39.3
^ permalink raw reply related [flat|nested] 21+ messages in thread* Re: [PATCH 9.0 04/13] vdpa: move shadow_data to vhost_vdpa_shared
2023-11-24 17:14 ` [PATCH 9.0 04/13] vdpa: move shadow_data " Eugenio Pérez
@ 2023-12-06 6:04 ` Si-Wei Liu
0 siblings, 0 replies; 21+ messages in thread
From: Si-Wei Liu @ 2023-12-06 6:04 UTC (permalink / raw)
To: Eugenio Pérez, qemu-devel
Cc: Parav Pandit, Stefano Garzarella, Zhu Lingshan, Lei Yang,
Michael S. Tsirkin, Jason Wang, Dragos Tatulea, Laurent Vivier
On 11/24/2023 9:14 AM, Eugenio Pérez wrote:
> Next patches will register the vhost_vdpa memory listener while the VM
> is migrating at the destination, so we can map the memory to the device
> before stopping the VM at the source. The main goal is to reduce the
> downtime.
>
> However, the destination QEMU is unaware of which vhost_vdpa device will
> register its memory_listener. If the source guest has CVQ enabled, it
> will be the CVQ device. Otherwise, it will be the first one.
>
> Move the shadow_data member to VhostVDPAShared so all vhost_vdpa can use
> it, rather than always in the first or last vhost_vdpa.
>
> Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> ---
> v1 from RFC:
> * Fix vhost_vdpa_net_cvq_start checking for always_svq instead of
> shadow_data. This could cause CVQ not being shadowed if
> vhost_vdpa_net_cvq_start was called in the middle of a migration.
> ---
> include/hw/virtio/vhost-vdpa.h | 5 +++--
> hw/virtio/vhost-vdpa.c | 6 +++---
> net/vhost-vdpa.c | 23 ++++++-----------------
> 3 files changed, 12 insertions(+), 22 deletions(-)
>
> diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h
> index 8d52a7e498..01e0f25e27 100644
> --- a/include/hw/virtio/vhost-vdpa.h
> +++ b/include/hw/virtio/vhost-vdpa.h
> @@ -36,6 +36,9 @@ typedef struct vhost_vdpa_shared {
>
> /* IOVA mapping used by the Shadow Virtqueue */
> VhostIOVATree *iova_tree;
> +
> + /* Vdpa must send shadow addresses as IOTLB key for data queues, not GPA */
> + bool shadow_data;
> } VhostVDPAShared;
>
> typedef struct vhost_vdpa {
> @@ -47,8 +50,6 @@ typedef struct vhost_vdpa {
> MemoryListener listener;
> uint64_t acked_features;
> bool shadow_vqs_enabled;
> - /* Vdpa must send shadow addresses as IOTLB key for data queues, not GPA */
> - bool shadow_data;
> /* Device suspended successfully */
> bool suspended;
> VhostVDPAShared *shared;
> diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
> index 2bceadd118..ec028e4c56 100644
> --- a/hw/virtio/vhost-vdpa.c
> +++ b/hw/virtio/vhost-vdpa.c
> @@ -353,7 +353,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
> vaddr, section->readonly);
>
> llsize = int128_sub(llend, int128_make64(iova));
> - if (v->shadow_data) {
> + if (v->shared->shadow_data) {
> int r;
>
> mem_region.translated_addr = (hwaddr)(uintptr_t)vaddr,
> @@ -380,7 +380,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
> return;
>
> fail_map:
> - if (v->shadow_data) {
> + if (v->shared->shadow_data) {
> vhost_iova_tree_remove(v->shared->iova_tree, mem_region);
> }
>
> @@ -435,7 +435,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
>
> llsize = int128_sub(llend, int128_make64(iova));
>
> - if (v->shadow_data) {
> + if (v->shared->shadow_data) {
> const DMAMap *result;
> const void *vaddr = memory_region_get_ram_ptr(section->mr) +
> section->offset_within_region +
> diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c
> index 7be2c30ad3..2376d9989a 100644
> --- a/net/vhost-vdpa.c
> +++ b/net/vhost-vdpa.c
> @@ -290,15 +290,6 @@ static ssize_t vhost_vdpa_receive(NetClientState *nc, const uint8_t *buf,
> return size;
> }
>
> -/** From any vdpa net client, get the netclient of the first queue pair */
> -static VhostVDPAState *vhost_vdpa_net_first_nc_vdpa(VhostVDPAState *s)
> -{
> - NICState *nic = qemu_get_nic(s->nc.peer);
> - NetClientState *nc0 = qemu_get_peer(nic->ncs, 0);
> -
> - return DO_UPCAST(VhostVDPAState, nc, nc0);
> -}
> -
Fine with the removal as the only caller is gone. However, in my series
I will resurrect this function for fields that are not shared and vq
specific. Not sure there's sort of way not failing the build while
keeping unused function around.
> static void vhost_vdpa_net_log_global_enable(VhostVDPAState *s, bool enable)
> {
> struct vhost_vdpa *v = &s->vhost_vdpa;
> @@ -369,10 +360,10 @@ static int vhost_vdpa_net_data_start(NetClientState *nc)
> if (s->always_svq ||
> migration_is_setup_or_active(migrate_get_current()->state)) {
> v->shadow_vqs_enabled = true;
> - v->shadow_data = true;
> + v->shared->shadow_data = true;
Noted this shared shadow_data needs to be set only once. I will also
change this in my series. No actual code change is requested.
> } else {
> v->shadow_vqs_enabled = false;
> - v->shadow_data = false;
> + v->shared->shadow_data = false;
> }
>
> if (v->index == 0) {
> @@ -523,7 +514,7 @@ dma_map_err:
>
> static int vhost_vdpa_net_cvq_start(NetClientState *nc)
> {
> - VhostVDPAState *s, *s0;
> + VhostVDPAState *s;
> struct vhost_vdpa *v;
> int64_t cvq_group;
> int r;
> @@ -534,12 +525,10 @@ static int vhost_vdpa_net_cvq_start(NetClientState *nc)
> s = DO_UPCAST(VhostVDPAState, nc, nc);
> v = &s->vhost_vdpa;
>
> - s0 = vhost_vdpa_net_first_nc_vdpa(s);
> - v->shadow_data = s0->vhost_vdpa.shadow_vqs_enabled;
> - v->shadow_vqs_enabled = s0->vhost_vdpa.shadow_vqs_enabled;
> + v->shadow_vqs_enabled = v->shared->shadow_data;
This new code looks fine.
Reviewed-by: Si-Wei Liu <si-wei.liu@oracle.com>
> s->vhost_vdpa.address_space_id = VHOST_VDPA_GUEST_PA_ASID;
>
> - if (s->vhost_vdpa.shadow_data) {
> + if (v->shared->shadow_data) {
> /* SVQ is already configured for all virtqueues */
> goto out;
> }
> @@ -1688,12 +1677,12 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
> s->always_svq = svq;
> s->migration_state.notify = NULL;
> s->vhost_vdpa.shadow_vqs_enabled = svq;
> - s->vhost_vdpa.shadow_data = svq;
> if (queue_pair_index == 0) {
> vhost_vdpa_net_valid_svq_features(features,
> &s->vhost_vdpa.migration_blocker);
> s->vhost_vdpa.shared = g_new0(VhostVDPAShared, 1);
> s->vhost_vdpa.shared->iova_range = iova_range;
> + s->vhost_vdpa.shared->shadow_data = svq;
> } else if (!is_datapath) {
> s->cvq_cmd_out_buffer = mmap(NULL, vhost_vdpa_net_cvq_cmd_page_len(),
> PROT_READ | PROT_WRITE,
^ permalink raw reply [flat|nested] 21+ messages in thread
* [PATCH 9.0 05/13] vdpa: use vdpa shared for tracing
2023-11-24 17:14 [PATCH 9.0 00/13] Consolidate common vdpa members in VhostVDPAShared Eugenio Pérez
` (3 preceding siblings ...)
2023-11-24 17:14 ` [PATCH 9.0 04/13] vdpa: move shadow_data " Eugenio Pérez
@ 2023-11-24 17:14 ` Eugenio Pérez
2023-11-24 17:14 ` [PATCH 9.0 06/13] vdpa: move file descriptor to vhost_vdpa_shared Eugenio Pérez
` (9 subsequent siblings)
14 siblings, 0 replies; 21+ messages in thread
From: Eugenio Pérez @ 2023-11-24 17:14 UTC (permalink / raw)
To: qemu-devel
Cc: Parav Pandit, si-wei.liu, Stefano Garzarella, Zhu Lingshan,
Lei Yang, Michael S. Tsirkin, Jason Wang, Dragos Tatulea,
Laurent Vivier
By the end of this series dma_map and dma_unmap functions don't have the
vdpa device for tracing. Movinge trace function to shared member one.
Print it also in the vdpa initialization so log reader can relate them.
Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
hw/virtio/vhost-vdpa.c | 26 ++++++++++++++------------
hw/virtio/trace-events | 14 +++++++-------
2 files changed, 21 insertions(+), 19 deletions(-)
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index ec028e4c56..85de60b184 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -101,7 +101,7 @@ int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
msg.iotlb.perm = readonly ? VHOST_ACCESS_RO : VHOST_ACCESS_RW;
msg.iotlb.type = VHOST_IOTLB_UPDATE;
- trace_vhost_vdpa_dma_map(v, fd, msg.type, msg.asid, msg.iotlb.iova,
+ trace_vhost_vdpa_dma_map(v->shared, fd, msg.type, msg.asid, msg.iotlb.iova,
msg.iotlb.size, msg.iotlb.uaddr, msg.iotlb.perm,
msg.iotlb.type);
@@ -131,8 +131,8 @@ int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
msg.iotlb.size = size;
msg.iotlb.type = VHOST_IOTLB_INVALIDATE;
- trace_vhost_vdpa_dma_unmap(v, fd, msg.type, msg.asid, msg.iotlb.iova,
- msg.iotlb.size, msg.iotlb.type);
+ trace_vhost_vdpa_dma_unmap(v->shared, fd, msg.type, msg.asid,
+ msg.iotlb.iova, msg.iotlb.size, msg.iotlb.type);
if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {
error_report("failed to write, fd=%d, errno=%d (%s)",
@@ -151,7 +151,8 @@ static void vhost_vdpa_listener_begin_batch(struct vhost_vdpa *v)
.iotlb.type = VHOST_IOTLB_BATCH_BEGIN,
};
- trace_vhost_vdpa_listener_begin_batch(v, fd, msg.type, msg.iotlb.type);
+ trace_vhost_vdpa_listener_begin_batch(v->shared, fd, msg.type,
+ msg.iotlb.type);
if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {
error_report("failed to write, fd=%d, errno=%d (%s)",
fd, errno, strerror(errno));
@@ -186,7 +187,7 @@ static void vhost_vdpa_listener_commit(MemoryListener *listener)
msg.type = v->msg_type;
msg.iotlb.type = VHOST_IOTLB_BATCH_END;
- trace_vhost_vdpa_listener_commit(v, fd, msg.type, msg.iotlb.type);
+ trace_vhost_vdpa_listener_commit(v->shared, fd, msg.type, msg.iotlb.type);
if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {
error_report("failed to write, fd=%d, errno=%d (%s)",
fd, errno, strerror(errno));
@@ -329,7 +330,8 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
if (unlikely((section->offset_within_address_space & ~page_mask) !=
(section->offset_within_region & ~page_mask))) {
- trace_vhost_vdpa_listener_region_add_unaligned(v, section->mr->name,
+ trace_vhost_vdpa_listener_region_add_unaligned(v->shared,
+ section->mr->name,
section->offset_within_address_space & ~page_mask,
section->offset_within_region & ~page_mask);
return;
@@ -349,7 +351,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
section->offset_within_region +
(iova - section->offset_within_address_space);
- trace_vhost_vdpa_listener_region_add(v, iova, int128_get64(llend),
+ trace_vhost_vdpa_listener_region_add(v->shared, iova, int128_get64(llend),
vaddr, section->readonly);
llsize = int128_sub(llend, int128_make64(iova));
@@ -417,7 +419,8 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
if (unlikely((section->offset_within_address_space & ~page_mask) !=
(section->offset_within_region & ~page_mask))) {
- trace_vhost_vdpa_listener_region_del_unaligned(v, section->mr->name,
+ trace_vhost_vdpa_listener_region_del_unaligned(v->shared,
+ section->mr->name,
section->offset_within_address_space & ~page_mask,
section->offset_within_region & ~page_mask);
return;
@@ -426,7 +429,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
iova = ROUND_UP(section->offset_within_address_space, page_size);
llend = vhost_vdpa_section_end(section, page_mask);
- trace_vhost_vdpa_listener_region_del(v, iova,
+ trace_vhost_vdpa_listener_region_del(v->shared, iova,
int128_get64(int128_sub(llend, int128_one())));
if (int128_ge(int128_make64(iova), llend)) {
@@ -583,12 +586,11 @@ static void vhost_vdpa_init_svq(struct vhost_dev *hdev, struct vhost_vdpa *v)
static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
{
- struct vhost_vdpa *v;
+ struct vhost_vdpa *v = opaque;
assert(dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_VDPA);
- trace_vhost_vdpa_init(dev, opaque);
+ trace_vhost_vdpa_init(dev, v->shared, opaque);
int ret;
- v = opaque;
v->dev = dev;
dev->opaque = opaque ;
v->listener = vhost_vdpa_memory_listener;
diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events
index 637cac4edf..77905d1994 100644
--- a/hw/virtio/trace-events
+++ b/hw/virtio/trace-events
@@ -30,16 +30,16 @@ vhost_user_write(uint32_t req, uint32_t flags) "req:%d flags:0x%"PRIx32""
vhost_user_create_notifier(int idx, void *n) "idx:%d n:%p"
# vhost-vdpa.c
-vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8
-vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8
-vhost_vdpa_listener_begin_batch(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8
-vhost_vdpa_listener_commit(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8
-vhost_vdpa_listener_region_add_unaligned(void *v, const char *name, uint64_t offset_as, uint64_t offset_page) "vdpa: %p region %s offset_within_address_space %"PRIu64" offset_within_region %"PRIu64
+vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8
+vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint8_t type) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8
+vhost_vdpa_listener_begin_batch(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8
+vhost_vdpa_listener_commit(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8
+vhost_vdpa_listener_region_add_unaligned(void *v, const char *name, uint64_t offset_as, uint64_t offset_page) "vdpa_shared: %p region %s offset_within_address_space %"PRIu64" offset_within_region %"PRIu64
vhost_vdpa_listener_region_add(void *vdpa, uint64_t iova, uint64_t llend, void *vaddr, bool readonly) "vdpa: %p iova 0x%"PRIx64" llend 0x%"PRIx64" vaddr: %p read-only: %d"
-vhost_vdpa_listener_region_del_unaligned(void *v, const char *name, uint64_t offset_as, uint64_t offset_page) "vdpa: %p region %s offset_within_address_space %"PRIu64" offset_within_region %"PRIu64
+vhost_vdpa_listener_region_del_unaligned(void *v, const char *name, uint64_t offset_as, uint64_t offset_page) "vdpa_shared: %p region %s offset_within_address_space %"PRIu64" offset_within_region %"PRIu64
vhost_vdpa_listener_region_del(void *vdpa, uint64_t iova, uint64_t llend) "vdpa: %p iova 0x%"PRIx64" llend 0x%"PRIx64
vhost_vdpa_add_status(void *dev, uint8_t status) "dev: %p status: 0x%"PRIx8
-vhost_vdpa_init(void *dev, void *vdpa) "dev: %p vdpa: %p"
+vhost_vdpa_init(void *dev, void *s, void *vdpa) "dev: %p, common dev: %p vdpa: %p"
vhost_vdpa_cleanup(void *dev, void *vdpa) "dev: %p vdpa: %p"
vhost_vdpa_memslots_limit(void *dev, int ret) "dev: %p = 0x%x"
vhost_vdpa_set_mem_table(void *dev, uint32_t nregions, uint32_t padding) "dev: %p nregions: %"PRIu32" padding: 0x%"PRIx32
--
2.39.3
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 9.0 06/13] vdpa: move file descriptor to vhost_vdpa_shared
2023-11-24 17:14 [PATCH 9.0 00/13] Consolidate common vdpa members in VhostVDPAShared Eugenio Pérez
` (4 preceding siblings ...)
2023-11-24 17:14 ` [PATCH 9.0 05/13] vdpa: use vdpa shared for tracing Eugenio Pérez
@ 2023-11-24 17:14 ` Eugenio Pérez
2023-11-24 17:14 ` [PATCH 9.0 07/13] vdpa: move iotlb_batch_begin_sent " Eugenio Pérez
` (8 subsequent siblings)
14 siblings, 0 replies; 21+ messages in thread
From: Eugenio Pérez @ 2023-11-24 17:14 UTC (permalink / raw)
To: qemu-devel
Cc: Parav Pandit, si-wei.liu, Stefano Garzarella, Zhu Lingshan,
Lei Yang, Michael S. Tsirkin, Jason Wang, Dragos Tatulea,
Laurent Vivier
Next patches will register the vhost_vdpa memory listener while the VM
is migrating at the destination, so we can map the memory to the device
before stopping the VM at the source. The main goal is to reduce the
downtime.
However, the destination QEMU is unaware of which vhost_vdpa device will
register its memory_listener. If the source guest has CVQ enabled, it
will be the CVQ device. Otherwise, it will be the first one.
Move the file descriptor to VhostVDPAShared so all vhost_vdpa can use
it, rather than always in the first / last vhost_vdpa.
Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
include/hw/virtio/vhost-vdpa.h | 2 +-
hw/virtio/vdpa-dev.c | 2 +-
hw/virtio/vhost-vdpa.c | 14 +++++++-------
net/vhost-vdpa.c | 11 ++++-------
4 files changed, 13 insertions(+), 16 deletions(-)
diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h
index 01e0f25e27..796a180afa 100644
--- a/include/hw/virtio/vhost-vdpa.h
+++ b/include/hw/virtio/vhost-vdpa.h
@@ -32,6 +32,7 @@ typedef struct VhostVDPAHostNotifier {
/* Info shared by all vhost_vdpa device models */
typedef struct vhost_vdpa_shared {
+ int device_fd;
struct vhost_vdpa_iova_range iova_range;
/* IOVA mapping used by the Shadow Virtqueue */
@@ -42,7 +43,6 @@ typedef struct vhost_vdpa_shared {
} VhostVDPAShared;
typedef struct vhost_vdpa {
- int device_fd;
int index;
uint32_t msg_type;
bool iotlb_batch_begin_sent;
diff --git a/hw/virtio/vdpa-dev.c b/hw/virtio/vdpa-dev.c
index 457960d28a..8774986571 100644
--- a/hw/virtio/vdpa-dev.c
+++ b/hw/virtio/vdpa-dev.c
@@ -66,7 +66,6 @@ static void vhost_vdpa_device_realize(DeviceState *dev, Error **errp)
if (*errp) {
return;
}
- v->vdpa.device_fd = v->vhostfd;
v->vdev_id = vhost_vdpa_device_get_u32(v->vhostfd,
VHOST_VDPA_GET_DEVICE_ID, errp);
@@ -115,6 +114,7 @@ static void vhost_vdpa_device_realize(DeviceState *dev, Error **errp)
goto free_vqs;
}
v->vdpa.shared = g_new0(VhostVDPAShared, 1);
+ v->vdpa.shared->device_fd = v->vhostfd;
v->vdpa.shared->iova_range = iova_range;
ret = vhost_dev_init(&v->dev, &v->vdpa, VHOST_BACKEND_TYPE_VDPA, 0, NULL);
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 85de60b184..095543395b 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -90,7 +90,7 @@ int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
hwaddr size, void *vaddr, bool readonly)
{
struct vhost_msg_v2 msg = {};
- int fd = v->device_fd;
+ int fd = v->shared->device_fd;
int ret = 0;
msg.type = v->msg_type;
@@ -122,7 +122,7 @@ int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
hwaddr size)
{
struct vhost_msg_v2 msg = {};
- int fd = v->device_fd;
+ int fd = v->shared->device_fd;
int ret = 0;
msg.type = v->msg_type;
@@ -145,7 +145,7 @@ int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
static void vhost_vdpa_listener_begin_batch(struct vhost_vdpa *v)
{
- int fd = v->device_fd;
+ int fd = v->shared->device_fd;
struct vhost_msg_v2 msg = {
.type = v->msg_type,
.iotlb.type = VHOST_IOTLB_BATCH_BEGIN,
@@ -174,7 +174,7 @@ static void vhost_vdpa_listener_commit(MemoryListener *listener)
struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener);
struct vhost_dev *dev = v->dev;
struct vhost_msg_v2 msg = {};
- int fd = v->device_fd;
+ int fd = v->shared->device_fd;
if (!(dev->backend_cap & (0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH))) {
return;
@@ -499,7 +499,7 @@ static int vhost_vdpa_call(struct vhost_dev *dev, unsigned long int request,
void *arg)
{
struct vhost_vdpa *v = dev->opaque;
- int fd = v->device_fd;
+ int fd = v->shared->device_fd;
int ret;
assert(dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_VDPA);
@@ -657,7 +657,7 @@ static int vhost_vdpa_host_notifier_init(struct vhost_dev *dev, int queue_index)
struct vhost_vdpa *v = dev->opaque;
VirtIODevice *vdev = dev->vdev;
VhostVDPAHostNotifier *n;
- int fd = v->device_fd;
+ int fd = v->shared->device_fd;
void *addr;
char *name;
@@ -1286,7 +1286,7 @@ static void vhost_vdpa_suspend(struct vhost_dev *dev)
if (dev->backend_cap & BIT_ULL(VHOST_BACKEND_F_SUSPEND)) {
trace_vhost_vdpa_suspend(dev);
- r = ioctl(v->device_fd, VHOST_VDPA_SUSPEND);
+ r = ioctl(v->shared->device_fd, VHOST_VDPA_SUSPEND);
if (unlikely(r)) {
error_report("Cannot suspend: %s(%d)", g_strerror(errno), errno);
} else {
diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c
index 2376d9989a..3aefd77968 100644
--- a/net/vhost-vdpa.c
+++ b/net/vhost-vdpa.c
@@ -235,14 +235,11 @@ static void vhost_vdpa_cleanup(NetClientState *nc)
vhost_net_cleanup(s->vhost_net);
g_free(s->vhost_net);
s->vhost_net = NULL;
- }
- if (s->vhost_vdpa.device_fd >= 0) {
- qemu_close(s->vhost_vdpa.device_fd);
- s->vhost_vdpa.device_fd = -1;
}
if (s->vhost_vdpa.index != 0) {
return;
}
+ qemu_close(s->vhost_vdpa.shared->device_fd);
g_free(s->vhost_vdpa.shared);
}
@@ -449,7 +446,7 @@ static int vhost_vdpa_set_address_space_id(struct vhost_vdpa *v,
};
int r;
- r = ioctl(v->device_fd, VHOST_VDPA_SET_GROUP_ASID, &asid);
+ r = ioctl(v->shared->device_fd, VHOST_VDPA_SET_GROUP_ASID, &asid);
if (unlikely(r < 0)) {
error_report("Can't set vq group %u asid %u, errno=%d (%s)",
asid.index, asid.num, errno, g_strerror(errno));
@@ -545,7 +542,7 @@ static int vhost_vdpa_net_cvq_start(NetClientState *nc)
return 0;
}
- cvq_group = vhost_vdpa_get_vring_group(v->device_fd,
+ cvq_group = vhost_vdpa_get_vring_group(v->shared->device_fd,
v->dev->vq_index_end - 1,
&err);
if (unlikely(cvq_group < 0)) {
@@ -1672,7 +1669,6 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
qemu_set_info_str(nc, TYPE_VHOST_VDPA);
s = DO_UPCAST(VhostVDPAState, nc, nc);
- s->vhost_vdpa.device_fd = vdpa_device_fd;
s->vhost_vdpa.index = queue_pair_index;
s->always_svq = svq;
s->migration_state.notify = NULL;
@@ -1681,6 +1677,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
vhost_vdpa_net_valid_svq_features(features,
&s->vhost_vdpa.migration_blocker);
s->vhost_vdpa.shared = g_new0(VhostVDPAShared, 1);
+ s->vhost_vdpa.shared->device_fd = vdpa_device_fd;
s->vhost_vdpa.shared->iova_range = iova_range;
s->vhost_vdpa.shared->shadow_data = svq;
} else if (!is_datapath) {
--
2.39.3
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 9.0 07/13] vdpa: move iotlb_batch_begin_sent to vhost_vdpa_shared
2023-11-24 17:14 [PATCH 9.0 00/13] Consolidate common vdpa members in VhostVDPAShared Eugenio Pérez
` (5 preceding siblings ...)
2023-11-24 17:14 ` [PATCH 9.0 06/13] vdpa: move file descriptor to vhost_vdpa_shared Eugenio Pérez
@ 2023-11-24 17:14 ` Eugenio Pérez
2023-11-24 17:14 ` [PATCH 9.0 08/13] vdpa: move backend_cap " Eugenio Pérez
` (7 subsequent siblings)
14 siblings, 0 replies; 21+ messages in thread
From: Eugenio Pérez @ 2023-11-24 17:14 UTC (permalink / raw)
To: qemu-devel
Cc: Parav Pandit, si-wei.liu, Stefano Garzarella, Zhu Lingshan,
Lei Yang, Michael S. Tsirkin, Jason Wang, Dragos Tatulea,
Laurent Vivier
Next patches will register the vhost_vdpa memory listener while the VM
is migrating at the destination, so we can map the memory to the device
before stopping the VM at the source. The main goal is to reduce the
downtime.
However, the destination QEMU is unaware of which vhost_vdpa device will
register its memory_listener. If the source guest has CVQ enabled, it
will be the CVQ device. Otherwise, it will be the first one.
Move the iotlb_batch_begin_sent member to VhostVDPAShared so all
vhost_vdpa can use it, rather than always in the first / last
vhost_vdpa.
Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
include/hw/virtio/vhost-vdpa.h | 3 ++-
hw/virtio/vhost-vdpa.c | 8 ++++----
2 files changed, 6 insertions(+), 5 deletions(-)
diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h
index 796a180afa..05219bbcf7 100644
--- a/include/hw/virtio/vhost-vdpa.h
+++ b/include/hw/virtio/vhost-vdpa.h
@@ -38,6 +38,8 @@ typedef struct vhost_vdpa_shared {
/* IOVA mapping used by the Shadow Virtqueue */
VhostIOVATree *iova_tree;
+ bool iotlb_batch_begin_sent;
+
/* Vdpa must send shadow addresses as IOTLB key for data queues, not GPA */
bool shadow_data;
} VhostVDPAShared;
@@ -45,7 +47,6 @@ typedef struct vhost_vdpa_shared {
typedef struct vhost_vdpa {
int index;
uint32_t msg_type;
- bool iotlb_batch_begin_sent;
uint32_t address_space_id;
MemoryListener listener;
uint64_t acked_features;
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 095543395b..85b13e09f4 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -162,11 +162,11 @@ static void vhost_vdpa_listener_begin_batch(struct vhost_vdpa *v)
static void vhost_vdpa_iotlb_batch_begin_once(struct vhost_vdpa *v)
{
if (v->dev->backend_cap & (0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH) &&
- !v->iotlb_batch_begin_sent) {
+ !v->shared->iotlb_batch_begin_sent) {
vhost_vdpa_listener_begin_batch(v);
}
- v->iotlb_batch_begin_sent = true;
+ v->shared->iotlb_batch_begin_sent = true;
}
static void vhost_vdpa_listener_commit(MemoryListener *listener)
@@ -180,7 +180,7 @@ static void vhost_vdpa_listener_commit(MemoryListener *listener)
return;
}
- if (!v->iotlb_batch_begin_sent) {
+ if (!v->shared->iotlb_batch_begin_sent) {
return;
}
@@ -193,7 +193,7 @@ static void vhost_vdpa_listener_commit(MemoryListener *listener)
fd, errno, strerror(errno));
}
- v->iotlb_batch_begin_sent = false;
+ v->shared->iotlb_batch_begin_sent = false;
}
static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb)
--
2.39.3
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 9.0 08/13] vdpa: move backend_cap to vhost_vdpa_shared
2023-11-24 17:14 [PATCH 9.0 00/13] Consolidate common vdpa members in VhostVDPAShared Eugenio Pérez
` (6 preceding siblings ...)
2023-11-24 17:14 ` [PATCH 9.0 07/13] vdpa: move iotlb_batch_begin_sent " Eugenio Pérez
@ 2023-11-24 17:14 ` Eugenio Pérez
2023-11-24 17:14 ` [PATCH 9.0 09/13] vdpa: remove msg type of vhost_vdpa Eugenio Pérez
` (6 subsequent siblings)
14 siblings, 0 replies; 21+ messages in thread
From: Eugenio Pérez @ 2023-11-24 17:14 UTC (permalink / raw)
To: qemu-devel
Cc: Parav Pandit, si-wei.liu, Stefano Garzarella, Zhu Lingshan,
Lei Yang, Michael S. Tsirkin, Jason Wang, Dragos Tatulea,
Laurent Vivier
Next patches will register the vhost_vdpa memory listener while the VM
is migrating at the destination, so we can map the memory to the device
before stopping the VM at the source. The main goal is to reduce the
downtime.
However, the destination QEMU is unaware of which vhost_vdpa device will
register its memory_listener. If the source guest has CVQ enabled, it
will be the CVQ device. Otherwise, it will be the first one.
Move the backend_cap member to VhostVDPAShared so all vhost_vdpa can use
it, rather than always in the first / last vhost_vdpa.
Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
include/hw/virtio/vhost-vdpa.h | 3 +++
hw/virtio/vhost-vdpa.c | 8 +++++---
2 files changed, 8 insertions(+), 3 deletions(-)
diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h
index 05219bbcf7..11ac14085a 100644
--- a/include/hw/virtio/vhost-vdpa.h
+++ b/include/hw/virtio/vhost-vdpa.h
@@ -38,6 +38,9 @@ typedef struct vhost_vdpa_shared {
/* IOVA mapping used by the Shadow Virtqueue */
VhostIOVATree *iova_tree;
+ /* Copy of backend features */
+ uint64_t backend_cap;
+
bool iotlb_batch_begin_sent;
/* Vdpa must send shadow addresses as IOTLB key for data queues, not GPA */
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 85b13e09f4..458e46befd 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -161,7 +161,7 @@ static void vhost_vdpa_listener_begin_batch(struct vhost_vdpa *v)
static void vhost_vdpa_iotlb_batch_begin_once(struct vhost_vdpa *v)
{
- if (v->dev->backend_cap & (0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH) &&
+ if (v->shared->backend_cap & (0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH) &&
!v->shared->iotlb_batch_begin_sent) {
vhost_vdpa_listener_begin_batch(v);
}
@@ -172,11 +172,10 @@ static void vhost_vdpa_iotlb_batch_begin_once(struct vhost_vdpa *v)
static void vhost_vdpa_listener_commit(MemoryListener *listener)
{
struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener);
- struct vhost_dev *dev = v->dev;
struct vhost_msg_v2 msg = {};
int fd = v->shared->device_fd;
- if (!(dev->backend_cap & (0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH))) {
+ if (!(v->shared->backend_cap & (0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH))) {
return;
}
@@ -834,6 +833,8 @@ static int vhost_vdpa_set_features(struct vhost_dev *dev,
static int vhost_vdpa_set_backend_cap(struct vhost_dev *dev)
{
+ struct vhost_vdpa *v = dev->opaque;
+
uint64_t features;
uint64_t f = 0x1ULL << VHOST_BACKEND_F_IOTLB_MSG_V2 |
0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH |
@@ -855,6 +856,7 @@ static int vhost_vdpa_set_backend_cap(struct vhost_dev *dev)
}
dev->backend_cap = features;
+ v->shared->backend_cap = features;
return 0;
}
--
2.39.3
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 9.0 09/13] vdpa: remove msg type of vhost_vdpa
2023-11-24 17:14 [PATCH 9.0 00/13] Consolidate common vdpa members in VhostVDPAShared Eugenio Pérez
` (7 preceding siblings ...)
2023-11-24 17:14 ` [PATCH 9.0 08/13] vdpa: move backend_cap " Eugenio Pérez
@ 2023-11-24 17:14 ` Eugenio Pérez
2023-11-24 17:14 ` [PATCH 9.0 10/13] vdpa: move iommu_list to vhost_vdpa_shared Eugenio Pérez
` (5 subsequent siblings)
14 siblings, 0 replies; 21+ messages in thread
From: Eugenio Pérez @ 2023-11-24 17:14 UTC (permalink / raw)
To: qemu-devel
Cc: Parav Pandit, si-wei.liu, Stefano Garzarella, Zhu Lingshan,
Lei Yang, Michael S. Tsirkin, Jason Wang, Dragos Tatulea,
Laurent Vivier
It is always VHOST_IOTLB_MSG_V2. We can always make it back per
vhost_dev if needed.
This change makes easier for vhost_vdpa_map and unmap not to depend on
vhost_vdpa but only in VhostVDPAShared.
Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
include/hw/virtio/vhost-vdpa.h | 1 -
hw/virtio/vhost-vdpa.c | 9 ++++-----
2 files changed, 4 insertions(+), 6 deletions(-)
diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h
index 11ac14085a..5bd964dac5 100644
--- a/include/hw/virtio/vhost-vdpa.h
+++ b/include/hw/virtio/vhost-vdpa.h
@@ -49,7 +49,6 @@ typedef struct vhost_vdpa_shared {
typedef struct vhost_vdpa {
int index;
- uint32_t msg_type;
uint32_t address_space_id;
MemoryListener listener;
uint64_t acked_features;
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 458e46befd..38afcbf1c9 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -93,7 +93,7 @@ int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
int fd = v->shared->device_fd;
int ret = 0;
- msg.type = v->msg_type;
+ msg.type = VHOST_IOTLB_MSG_V2;
msg.asid = asid;
msg.iotlb.iova = iova;
msg.iotlb.size = size;
@@ -125,7 +125,7 @@ int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
int fd = v->shared->device_fd;
int ret = 0;
- msg.type = v->msg_type;
+ msg.type = VHOST_IOTLB_MSG_V2;
msg.asid = asid;
msg.iotlb.iova = iova;
msg.iotlb.size = size;
@@ -147,7 +147,7 @@ static void vhost_vdpa_listener_begin_batch(struct vhost_vdpa *v)
{
int fd = v->shared->device_fd;
struct vhost_msg_v2 msg = {
- .type = v->msg_type,
+ .type = VHOST_IOTLB_MSG_V2,
.iotlb.type = VHOST_IOTLB_BATCH_BEGIN,
};
@@ -183,7 +183,7 @@ static void vhost_vdpa_listener_commit(MemoryListener *listener)
return;
}
- msg.type = v->msg_type;
+ msg.type = VHOST_IOTLB_MSG_V2;
msg.iotlb.type = VHOST_IOTLB_BATCH_END;
trace_vhost_vdpa_listener_commit(v->shared, fd, msg.type, msg.iotlb.type);
@@ -593,7 +593,6 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
v->dev = dev;
dev->opaque = opaque ;
v->listener = vhost_vdpa_memory_listener;
- v->msg_type = VHOST_IOTLB_MSG_V2;
vhost_vdpa_init_svq(dev, v);
error_propagate(&dev->migration_blocker, v->migration_blocker);
--
2.39.3
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 9.0 10/13] vdpa: move iommu_list to vhost_vdpa_shared
2023-11-24 17:14 [PATCH 9.0 00/13] Consolidate common vdpa members in VhostVDPAShared Eugenio Pérez
` (8 preceding siblings ...)
2023-11-24 17:14 ` [PATCH 9.0 09/13] vdpa: remove msg type of vhost_vdpa Eugenio Pérez
@ 2023-11-24 17:14 ` Eugenio Pérez
2023-11-24 17:14 ` [PATCH 9.0 11/13] vdpa: use VhostVDPAShared in vdpa_dma_map and unmap Eugenio Pérez
` (4 subsequent siblings)
14 siblings, 0 replies; 21+ messages in thread
From: Eugenio Pérez @ 2023-11-24 17:14 UTC (permalink / raw)
To: qemu-devel
Cc: Parav Pandit, si-wei.liu, Stefano Garzarella, Zhu Lingshan,
Lei Yang, Michael S. Tsirkin, Jason Wang, Dragos Tatulea,
Laurent Vivier
Next patches will register the vhost_vdpa memory listener while the VM
is migrating at the destination, so we can map the memory to the device
before stopping the VM at the source. The main goal is to reduce the
downtime.
However, the destination QEMU is unaware of which vhost_vdpa device will
register its memory_listener. If the source guest has CVQ enabled, it
will be the CVQ device. Otherwise, it will be the first one.
Move the iommu_list member to VhostVDPAShared so all vhost_vdpa can use
it, rather than always in the first / last vhost_vdpa.
Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
include/hw/virtio/vhost-vdpa.h | 2 +-
hw/virtio/vhost-vdpa.c | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h
index 5bd964dac5..3880b9e7f2 100644
--- a/include/hw/virtio/vhost-vdpa.h
+++ b/include/hw/virtio/vhost-vdpa.h
@@ -34,6 +34,7 @@ typedef struct VhostVDPAHostNotifier {
typedef struct vhost_vdpa_shared {
int device_fd;
struct vhost_vdpa_iova_range iova_range;
+ QLIST_HEAD(, vdpa_iommu) iommu_list;
/* IOVA mapping used by the Shadow Virtqueue */
VhostIOVATree *iova_tree;
@@ -62,7 +63,6 @@ typedef struct vhost_vdpa {
struct vhost_dev *dev;
Error *migration_blocker;
VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX];
- QLIST_HEAD(, vdpa_iommu) iommu_list;
IOMMUNotifier n;
} VhostVDPA;
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 38afcbf1c9..a07cd85081 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -279,7 +279,7 @@ static void vhost_vdpa_iommu_region_add(MemoryListener *listener,
return;
}
- QLIST_INSERT_HEAD(&v->iommu_list, iommu, iommu_next);
+ QLIST_INSERT_HEAD(&v->shared->iommu_list, iommu, iommu_next);
memory_region_iommu_replay(iommu->iommu_mr, &iommu->n);
return;
@@ -292,7 +292,7 @@ static void vhost_vdpa_iommu_region_del(MemoryListener *listener,
struct vdpa_iommu *iommu;
- QLIST_FOREACH(iommu, &v->iommu_list, iommu_next)
+ QLIST_FOREACH(iommu, &v->shared->iommu_list, iommu_next)
{
if (MEMORY_REGION(iommu->iommu_mr) == section->mr &&
iommu->n.start == section->offset_within_region) {
--
2.39.3
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 9.0 11/13] vdpa: use VhostVDPAShared in vdpa_dma_map and unmap
2023-11-24 17:14 [PATCH 9.0 00/13] Consolidate common vdpa members in VhostVDPAShared Eugenio Pérez
` (9 preceding siblings ...)
2023-11-24 17:14 ` [PATCH 9.0 10/13] vdpa: move iommu_list to vhost_vdpa_shared Eugenio Pérez
@ 2023-11-24 17:14 ` Eugenio Pérez
2023-11-24 17:14 ` [PATCH 9.0 12/13] vdpa: use dev_shared in vdpa_iommu Eugenio Pérez
` (3 subsequent siblings)
14 siblings, 0 replies; 21+ messages in thread
From: Eugenio Pérez @ 2023-11-24 17:14 UTC (permalink / raw)
To: qemu-devel
Cc: Parav Pandit, si-wei.liu, Stefano Garzarella, Zhu Lingshan,
Lei Yang, Michael S. Tsirkin, Jason Wang, Dragos Tatulea,
Laurent Vivier
The callers only have the shared information by the end of this series.
Start converting this functions.
Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
include/hw/virtio/vhost-vdpa.h | 4 +--
hw/virtio/vhost-vdpa.c | 50 +++++++++++++++++-----------------
net/vhost-vdpa.c | 5 ++--
3 files changed, 30 insertions(+), 29 deletions(-)
diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h
index 3880b9e7f2..705c754776 100644
--- a/include/hw/virtio/vhost-vdpa.h
+++ b/include/hw/virtio/vhost-vdpa.h
@@ -69,9 +69,9 @@ typedef struct vhost_vdpa {
int vhost_vdpa_get_iova_range(int fd, struct vhost_vdpa_iova_range *iova_range);
int vhost_vdpa_set_vring_ready(struct vhost_vdpa *v, unsigned idx);
-int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+int vhost_vdpa_dma_map(VhostVDPAShared *s, uint32_t asid, hwaddr iova,
hwaddr size, void *vaddr, bool readonly);
-int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+int vhost_vdpa_dma_unmap(VhostVDPAShared *s, uint32_t asid, hwaddr iova,
hwaddr size);
typedef struct vdpa_iommu {
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index a07cd85081..0ed6550aad 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -86,11 +86,11 @@ static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,
* The caller must set asid = 0 if the device does not support asid.
* This is not an ABI break since it is set to 0 by the initializer anyway.
*/
-int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+int vhost_vdpa_dma_map(VhostVDPAShared *s, uint32_t asid, hwaddr iova,
hwaddr size, void *vaddr, bool readonly)
{
struct vhost_msg_v2 msg = {};
- int fd = v->shared->device_fd;
+ int fd = s->device_fd;
int ret = 0;
msg.type = VHOST_IOTLB_MSG_V2;
@@ -101,7 +101,7 @@ int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
msg.iotlb.perm = readonly ? VHOST_ACCESS_RO : VHOST_ACCESS_RW;
msg.iotlb.type = VHOST_IOTLB_UPDATE;
- trace_vhost_vdpa_dma_map(v->shared, fd, msg.type, msg.asid, msg.iotlb.iova,
+ trace_vhost_vdpa_dma_map(s, fd, msg.type, msg.asid, msg.iotlb.iova,
msg.iotlb.size, msg.iotlb.uaddr, msg.iotlb.perm,
msg.iotlb.type);
@@ -118,11 +118,11 @@ int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
* The caller must set asid = 0 if the device does not support asid.
* This is not an ABI break since it is set to 0 by the initializer anyway.
*/
-int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+int vhost_vdpa_dma_unmap(VhostVDPAShared *s, uint32_t asid, hwaddr iova,
hwaddr size)
{
struct vhost_msg_v2 msg = {};
- int fd = v->shared->device_fd;
+ int fd = s->device_fd;
int ret = 0;
msg.type = VHOST_IOTLB_MSG_V2;
@@ -131,8 +131,8 @@ int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
msg.iotlb.size = size;
msg.iotlb.type = VHOST_IOTLB_INVALIDATE;
- trace_vhost_vdpa_dma_unmap(v->shared, fd, msg.type, msg.asid,
- msg.iotlb.iova, msg.iotlb.size, msg.iotlb.type);
+ trace_vhost_vdpa_dma_unmap(s, fd, msg.type, msg.asid, msg.iotlb.iova,
+ msg.iotlb.size, msg.iotlb.type);
if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {
error_report("failed to write, fd=%d, errno=%d (%s)",
@@ -143,30 +143,29 @@ int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
return ret;
}
-static void vhost_vdpa_listener_begin_batch(struct vhost_vdpa *v)
+static void vhost_vdpa_listener_begin_batch(VhostVDPAShared *s)
{
- int fd = v->shared->device_fd;
+ int fd = s->device_fd;
struct vhost_msg_v2 msg = {
.type = VHOST_IOTLB_MSG_V2,
.iotlb.type = VHOST_IOTLB_BATCH_BEGIN,
};
- trace_vhost_vdpa_listener_begin_batch(v->shared, fd, msg.type,
- msg.iotlb.type);
+ trace_vhost_vdpa_listener_begin_batch(s, fd, msg.type, msg.iotlb.type);
if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {
error_report("failed to write, fd=%d, errno=%d (%s)",
fd, errno, strerror(errno));
}
}
-static void vhost_vdpa_iotlb_batch_begin_once(struct vhost_vdpa *v)
+static void vhost_vdpa_iotlb_batch_begin_once(VhostVDPAShared *s)
{
- if (v->shared->backend_cap & (0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH) &&
- !v->shared->iotlb_batch_begin_sent) {
- vhost_vdpa_listener_begin_batch(v);
+ if (s->backend_cap & (0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH) &&
+ !s->iotlb_batch_begin_sent) {
+ vhost_vdpa_listener_begin_batch(s);
}
- v->shared->iotlb_batch_begin_sent = true;
+ s->iotlb_batch_begin_sent = true;
}
static void vhost_vdpa_listener_commit(MemoryListener *listener)
@@ -226,7 +225,7 @@ static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb)
if (!memory_get_xlat_addr(iotlb, &vaddr, NULL, &read_only, NULL)) {
return;
}
- ret = vhost_vdpa_dma_map(v, VHOST_VDPA_GUEST_PA_ASID, iova,
+ ret = vhost_vdpa_dma_map(v->shared, VHOST_VDPA_GUEST_PA_ASID, iova,
iotlb->addr_mask + 1, vaddr, read_only);
if (ret) {
error_report("vhost_vdpa_dma_map(%p, 0x%" HWADDR_PRIx ", "
@@ -234,7 +233,7 @@ static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb)
v, iova, iotlb->addr_mask + 1, vaddr, ret);
}
} else {
- ret = vhost_vdpa_dma_unmap(v, VHOST_VDPA_GUEST_PA_ASID, iova,
+ ret = vhost_vdpa_dma_unmap(v->shared, VHOST_VDPA_GUEST_PA_ASID, iova,
iotlb->addr_mask + 1);
if (ret) {
error_report("vhost_vdpa_dma_unmap(%p, 0x%" HWADDR_PRIx ", "
@@ -370,8 +369,8 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
iova = mem_region.iova;
}
- vhost_vdpa_iotlb_batch_begin_once(v);
- ret = vhost_vdpa_dma_map(v, VHOST_VDPA_GUEST_PA_ASID, iova,
+ vhost_vdpa_iotlb_batch_begin_once(v->shared);
+ ret = vhost_vdpa_dma_map(v->shared, VHOST_VDPA_GUEST_PA_ASID, iova,
int128_get64(llsize), vaddr, section->readonly);
if (ret) {
error_report("vhost vdpa map fail!");
@@ -455,13 +454,13 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
iova = result->iova;
vhost_iova_tree_remove(v->shared->iova_tree, *result);
}
- vhost_vdpa_iotlb_batch_begin_once(v);
+ vhost_vdpa_iotlb_batch_begin_once(v->shared);
/*
* The unmap ioctl doesn't accept a full 64-bit. need to check it
*/
if (int128_eq(llsize, int128_2_64())) {
llsize = int128_rshift(llsize, 1);
- ret = vhost_vdpa_dma_unmap(v, VHOST_VDPA_GUEST_PA_ASID, iova,
+ ret = vhost_vdpa_dma_unmap(v->shared, VHOST_VDPA_GUEST_PA_ASID, iova,
int128_get64(llsize));
if (ret) {
@@ -471,7 +470,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
}
iova += int128_get64(llsize);
}
- ret = vhost_vdpa_dma_unmap(v, VHOST_VDPA_GUEST_PA_ASID, iova,
+ ret = vhost_vdpa_dma_unmap(v->shared, VHOST_VDPA_GUEST_PA_ASID, iova,
int128_get64(llsize));
if (ret) {
@@ -1077,7 +1076,8 @@ static void vhost_vdpa_svq_unmap_ring(struct vhost_vdpa *v, hwaddr addr)
}
size = ROUND_UP(result->size, qemu_real_host_page_size());
- r = vhost_vdpa_dma_unmap(v, v->address_space_id, result->iova, size);
+ r = vhost_vdpa_dma_unmap(v->shared, v->address_space_id, result->iova,
+ size);
if (unlikely(r < 0)) {
error_report("Unable to unmap SVQ vring: %s (%d)", g_strerror(-r), -r);
return;
@@ -1117,7 +1117,7 @@ static bool vhost_vdpa_svq_map_ring(struct vhost_vdpa *v, DMAMap *needle,
return false;
}
- r = vhost_vdpa_dma_map(v, v->address_space_id, needle->iova,
+ r = vhost_vdpa_dma_map(v->shared, v->address_space_id, needle->iova,
needle->size + 1,
(void *)(uintptr_t)needle->translated_addr,
needle->perm == IOMMU_RO);
diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c
index 3aefd77968..3fb209cd35 100644
--- a/net/vhost-vdpa.c
+++ b/net/vhost-vdpa.c
@@ -472,7 +472,8 @@ static void vhost_vdpa_cvq_unmap_buf(struct vhost_vdpa *v, void *addr)
return;
}
- r = vhost_vdpa_dma_unmap(v, v->address_space_id, map->iova, map->size + 1);
+ r = vhost_vdpa_dma_unmap(v->shared, v->address_space_id, map->iova,
+ map->size + 1);
if (unlikely(r != 0)) {
error_report("Device cannot unmap: %s(%d)", g_strerror(r), r);
}
@@ -496,7 +497,7 @@ static int vhost_vdpa_cvq_map_buf(struct vhost_vdpa *v, void *buf, size_t size,
return r;
}
- r = vhost_vdpa_dma_map(v, v->address_space_id, map.iova,
+ r = vhost_vdpa_dma_map(v->shared, v->address_space_id, map.iova,
vhost_vdpa_net_cvq_cmd_page_len(), buf, !write);
if (unlikely(r < 0)) {
goto dma_map_err;
--
2.39.3
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 9.0 12/13] vdpa: use dev_shared in vdpa_iommu
2023-11-24 17:14 [PATCH 9.0 00/13] Consolidate common vdpa members in VhostVDPAShared Eugenio Pérez
` (10 preceding siblings ...)
2023-11-24 17:14 ` [PATCH 9.0 11/13] vdpa: use VhostVDPAShared in vdpa_dma_map and unmap Eugenio Pérez
@ 2023-11-24 17:14 ` Eugenio Pérez
2023-11-24 17:14 ` [PATCH 9.0 13/13] vdpa: move memory listener to vhost_vdpa_shared Eugenio Pérez
` (2 subsequent siblings)
14 siblings, 0 replies; 21+ messages in thread
From: Eugenio Pérez @ 2023-11-24 17:14 UTC (permalink / raw)
To: qemu-devel
Cc: Parav Pandit, si-wei.liu, Stefano Garzarella, Zhu Lingshan,
Lei Yang, Michael S. Tsirkin, Jason Wang, Dragos Tatulea,
Laurent Vivier
The memory listener functions can call these too. Make vdpa_iommu work
with VhostVDPAShared.
Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
include/hw/virtio/vhost-vdpa.h | 2 +-
hw/virtio/vhost-vdpa.c | 16 ++++++++--------
2 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h
index 705c754776..2abee2164a 100644
--- a/include/hw/virtio/vhost-vdpa.h
+++ b/include/hw/virtio/vhost-vdpa.h
@@ -75,7 +75,7 @@ int vhost_vdpa_dma_unmap(VhostVDPAShared *s, uint32_t asid, hwaddr iova,
hwaddr size);
typedef struct vdpa_iommu {
- struct vhost_vdpa *dev;
+ VhostVDPAShared *dev_shared;
IOMMUMemoryRegion *iommu_mr;
hwaddr iommu_offset;
IOMMUNotifier n;
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 0ed6550aad..61553ad196 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -199,7 +199,7 @@ static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb)
struct vdpa_iommu *iommu = container_of(n, struct vdpa_iommu, n);
hwaddr iova = iotlb->iova + iommu->iommu_offset;
- struct vhost_vdpa *v = iommu->dev;
+ VhostVDPAShared *s = iommu->dev_shared;
void *vaddr;
int ret;
Int128 llend;
@@ -212,10 +212,10 @@ static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb)
RCU_READ_LOCK_GUARD();
/* check if RAM section out of device range */
llend = int128_add(int128_makes64(iotlb->addr_mask), int128_makes64(iova));
- if (int128_gt(llend, int128_make64(v->shared->iova_range.last))) {
+ if (int128_gt(llend, int128_make64(s->iova_range.last))) {
error_report("RAM section out of device range (max=0x%" PRIx64
", end addr=0x%" PRIx64 ")",
- v->shared->iova_range.last, int128_get64(llend));
+ s->iova_range.last, int128_get64(llend));
return;
}
@@ -225,20 +225,20 @@ static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb)
if (!memory_get_xlat_addr(iotlb, &vaddr, NULL, &read_only, NULL)) {
return;
}
- ret = vhost_vdpa_dma_map(v->shared, VHOST_VDPA_GUEST_PA_ASID, iova,
+ ret = vhost_vdpa_dma_map(s, VHOST_VDPA_GUEST_PA_ASID, iova,
iotlb->addr_mask + 1, vaddr, read_only);
if (ret) {
error_report("vhost_vdpa_dma_map(%p, 0x%" HWADDR_PRIx ", "
"0x%" HWADDR_PRIx ", %p) = %d (%m)",
- v, iova, iotlb->addr_mask + 1, vaddr, ret);
+ s, iova, iotlb->addr_mask + 1, vaddr, ret);
}
} else {
- ret = vhost_vdpa_dma_unmap(v->shared, VHOST_VDPA_GUEST_PA_ASID, iova,
+ ret = vhost_vdpa_dma_unmap(s, VHOST_VDPA_GUEST_PA_ASID, iova,
iotlb->addr_mask + 1);
if (ret) {
error_report("vhost_vdpa_dma_unmap(%p, 0x%" HWADDR_PRIx ", "
"0x%" HWADDR_PRIx ") = %d (%m)",
- v, iova, iotlb->addr_mask + 1, ret);
+ s, iova, iotlb->addr_mask + 1, ret);
}
}
}
@@ -270,7 +270,7 @@ static void vhost_vdpa_iommu_region_add(MemoryListener *listener,
iommu_idx);
iommu->iommu_offset = section->offset_within_address_space -
section->offset_within_region;
- iommu->dev = v;
+ iommu->dev_shared = v->shared;
ret = memory_region_register_iommu_notifier(section->mr, &iommu->n, NULL);
if (ret) {
--
2.39.3
^ permalink raw reply related [flat|nested] 21+ messages in thread* [PATCH 9.0 13/13] vdpa: move memory listener to vhost_vdpa_shared
2023-11-24 17:14 [PATCH 9.0 00/13] Consolidate common vdpa members in VhostVDPAShared Eugenio Pérez
` (11 preceding siblings ...)
2023-11-24 17:14 ` [PATCH 9.0 12/13] vdpa: use dev_shared in vdpa_iommu Eugenio Pérez
@ 2023-11-24 17:14 ` Eugenio Pérez
2023-11-30 3:21 ` [PATCH 9.0 00/13] Consolidate common vdpa members in VhostVDPAShared Lei Yang
2023-12-01 7:04 ` Jason Wang
14 siblings, 0 replies; 21+ messages in thread
From: Eugenio Pérez @ 2023-11-24 17:14 UTC (permalink / raw)
To: qemu-devel
Cc: Parav Pandit, si-wei.liu, Stefano Garzarella, Zhu Lingshan,
Lei Yang, Michael S. Tsirkin, Jason Wang, Dragos Tatulea,
Laurent Vivier
Next patches will register the vhost_vdpa memory listener while the VM
is migrating at the destination, so we can map the memory to the device
before stopping the VM at the source. The main goal is to reduce the
downtime.
However, the destination QEMU is unaware of which vhost_vdpa device will
register its memory_listener. If the source guest has CVQ enabled, it
will be the CVQ device. Otherwise, it will be the first one.
Move the memory listener to a common place rather than always in the
first / last vhost_vdpa.
Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
include/hw/virtio/vhost-vdpa.h | 2 +-
hw/virtio/vhost-vdpa.c | 84 ++++++++++++++++------------------
2 files changed, 40 insertions(+), 46 deletions(-)
diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h
index 2abee2164a..8f54e5edd4 100644
--- a/include/hw/virtio/vhost-vdpa.h
+++ b/include/hw/virtio/vhost-vdpa.h
@@ -33,6 +33,7 @@ typedef struct VhostVDPAHostNotifier {
/* Info shared by all vhost_vdpa device models */
typedef struct vhost_vdpa_shared {
int device_fd;
+ MemoryListener listener;
struct vhost_vdpa_iova_range iova_range;
QLIST_HEAD(, vdpa_iommu) iommu_list;
@@ -51,7 +52,6 @@ typedef struct vhost_vdpa_shared {
typedef struct vhost_vdpa {
int index;
uint32_t address_space_id;
- MemoryListener listener;
uint64_t acked_features;
bool shadow_vqs_enabled;
/* Device suspended successfully */
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 61553ad196..7500c2fc82 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -170,28 +170,28 @@ static void vhost_vdpa_iotlb_batch_begin_once(VhostVDPAShared *s)
static void vhost_vdpa_listener_commit(MemoryListener *listener)
{
- struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener);
+ VhostVDPAShared *s = container_of(listener, VhostVDPAShared, listener);
struct vhost_msg_v2 msg = {};
- int fd = v->shared->device_fd;
+ int fd = s->device_fd;
- if (!(v->shared->backend_cap & (0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH))) {
+ if (!(s->backend_cap & (0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH))) {
return;
}
- if (!v->shared->iotlb_batch_begin_sent) {
+ if (!s->iotlb_batch_begin_sent) {
return;
}
msg.type = VHOST_IOTLB_MSG_V2;
msg.iotlb.type = VHOST_IOTLB_BATCH_END;
- trace_vhost_vdpa_listener_commit(v->shared, fd, msg.type, msg.iotlb.type);
+ trace_vhost_vdpa_listener_commit(s, fd, msg.type, msg.iotlb.type);
if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) {
error_report("failed to write, fd=%d, errno=%d (%s)",
fd, errno, strerror(errno));
}
- v->shared->iotlb_batch_begin_sent = false;
+ s->iotlb_batch_begin_sent = false;
}
static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb)
@@ -246,7 +246,7 @@ static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb)
static void vhost_vdpa_iommu_region_add(MemoryListener *listener,
MemoryRegionSection *section)
{
- struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener);
+ VhostVDPAShared *s = container_of(listener, VhostVDPAShared, listener);
struct vdpa_iommu *iommu;
Int128 end;
@@ -270,7 +270,7 @@ static void vhost_vdpa_iommu_region_add(MemoryListener *listener,
iommu_idx);
iommu->iommu_offset = section->offset_within_address_space -
section->offset_within_region;
- iommu->dev_shared = v->shared;
+ iommu->dev_shared = s;
ret = memory_region_register_iommu_notifier(section->mr, &iommu->n, NULL);
if (ret) {
@@ -278,7 +278,7 @@ static void vhost_vdpa_iommu_region_add(MemoryListener *listener,
return;
}
- QLIST_INSERT_HEAD(&v->shared->iommu_list, iommu, iommu_next);
+ QLIST_INSERT_HEAD(&s->iommu_list, iommu, iommu_next);
memory_region_iommu_replay(iommu->iommu_mr, &iommu->n);
return;
@@ -287,11 +287,11 @@ static void vhost_vdpa_iommu_region_add(MemoryListener *listener,
static void vhost_vdpa_iommu_region_del(MemoryListener *listener,
MemoryRegionSection *section)
{
- struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener);
+ VhostVDPAShared *s = container_of(listener, VhostVDPAShared, listener);
struct vdpa_iommu *iommu;
- QLIST_FOREACH(iommu, &v->shared->iommu_list, iommu_next)
+ QLIST_FOREACH(iommu, &s->iommu_list, iommu_next)
{
if (MEMORY_REGION(iommu->iommu_mr) == section->mr &&
iommu->n.start == section->offset_within_region) {
@@ -307,7 +307,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
MemoryRegionSection *section)
{
DMAMap mem_region = {};
- struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener);
+ VhostVDPAShared *s = container_of(listener, VhostVDPAShared, listener);
hwaddr iova;
Int128 llend, llsize;
void *vaddr;
@@ -315,10 +315,8 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
int page_size = qemu_target_page_size();
int page_mask = -page_size;
- if (vhost_vdpa_listener_skipped_section(section,
- v->shared->iova_range.first,
- v->shared->iova_range.last,
- page_mask)) {
+ if (vhost_vdpa_listener_skipped_section(section, s->iova_range.first,
+ s->iova_range.last, page_mask)) {
return;
}
if (memory_region_is_iommu(section->mr)) {
@@ -328,8 +326,7 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
if (unlikely((section->offset_within_address_space & ~page_mask) !=
(section->offset_within_region & ~page_mask))) {
- trace_vhost_vdpa_listener_region_add_unaligned(v->shared,
- section->mr->name,
+ trace_vhost_vdpa_listener_region_add_unaligned(s, section->mr->name,
section->offset_within_address_space & ~page_mask,
section->offset_within_region & ~page_mask);
return;
@@ -349,18 +346,18 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
section->offset_within_region +
(iova - section->offset_within_address_space);
- trace_vhost_vdpa_listener_region_add(v->shared, iova, int128_get64(llend),
+ trace_vhost_vdpa_listener_region_add(s, iova, int128_get64(llend),
vaddr, section->readonly);
llsize = int128_sub(llend, int128_make64(iova));
- if (v->shared->shadow_data) {
+ if (s->shadow_data) {
int r;
mem_region.translated_addr = (hwaddr)(uintptr_t)vaddr,
mem_region.size = int128_get64(llsize) - 1,
mem_region.perm = IOMMU_ACCESS_FLAG(true, section->readonly),
- r = vhost_iova_tree_map_alloc(v->shared->iova_tree, &mem_region);
+ r = vhost_iova_tree_map_alloc(s->iova_tree, &mem_region);
if (unlikely(r != IOVA_OK)) {
error_report("Can't allocate a mapping (%d)", r);
goto fail;
@@ -369,8 +366,8 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
iova = mem_region.iova;
}
- vhost_vdpa_iotlb_batch_begin_once(v->shared);
- ret = vhost_vdpa_dma_map(v->shared, VHOST_VDPA_GUEST_PA_ASID, iova,
+ vhost_vdpa_iotlb_batch_begin_once(s);
+ ret = vhost_vdpa_dma_map(s, VHOST_VDPA_GUEST_PA_ASID, iova,
int128_get64(llsize), vaddr, section->readonly);
if (ret) {
error_report("vhost vdpa map fail!");
@@ -380,8 +377,8 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener,
return;
fail_map:
- if (v->shared->shadow_data) {
- vhost_iova_tree_remove(v->shared->iova_tree, mem_region);
+ if (s->shadow_data) {
+ vhost_iova_tree_remove(s->iova_tree, mem_region);
}
fail:
@@ -398,17 +395,15 @@ fail:
static void vhost_vdpa_listener_region_del(MemoryListener *listener,
MemoryRegionSection *section)
{
- struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener);
+ VhostVDPAShared *s = container_of(listener, VhostVDPAShared, listener);
hwaddr iova;
Int128 llend, llsize;
int ret;
int page_size = qemu_target_page_size();
int page_mask = -page_size;
- if (vhost_vdpa_listener_skipped_section(section,
- v->shared->iova_range.first,
- v->shared->iova_range.last,
- page_mask)) {
+ if (vhost_vdpa_listener_skipped_section(section, s->iova_range.first,
+ s->iova_range.last, page_mask)) {
return;
}
if (memory_region_is_iommu(section->mr)) {
@@ -417,8 +412,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
if (unlikely((section->offset_within_address_space & ~page_mask) !=
(section->offset_within_region & ~page_mask))) {
- trace_vhost_vdpa_listener_region_del_unaligned(v->shared,
- section->mr->name,
+ trace_vhost_vdpa_listener_region_del_unaligned(s, section->mr->name,
section->offset_within_address_space & ~page_mask,
section->offset_within_region & ~page_mask);
return;
@@ -427,7 +421,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
iova = ROUND_UP(section->offset_within_address_space, page_size);
llend = vhost_vdpa_section_end(section, page_mask);
- trace_vhost_vdpa_listener_region_del(v->shared, iova,
+ trace_vhost_vdpa_listener_region_del(s, iova,
int128_get64(int128_sub(llend, int128_one())));
if (int128_ge(int128_make64(iova), llend)) {
@@ -436,7 +430,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
llsize = int128_sub(llend, int128_make64(iova));
- if (v->shared->shadow_data) {
+ if (s->shadow_data) {
const DMAMap *result;
const void *vaddr = memory_region_get_ram_ptr(section->mr) +
section->offset_within_region +
@@ -446,37 +440,37 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener,
.size = int128_get64(llsize) - 1,
};
- result = vhost_iova_tree_find_iova(v->shared->iova_tree, &mem_region);
+ result = vhost_iova_tree_find_iova(s->iova_tree, &mem_region);
if (!result) {
/* The memory listener map wasn't mapped */
return;
}
iova = result->iova;
- vhost_iova_tree_remove(v->shared->iova_tree, *result);
+ vhost_iova_tree_remove(s->iova_tree, *result);
}
- vhost_vdpa_iotlb_batch_begin_once(v->shared);
+ vhost_vdpa_iotlb_batch_begin_once(s);
/*
* The unmap ioctl doesn't accept a full 64-bit. need to check it
*/
if (int128_eq(llsize, int128_2_64())) {
llsize = int128_rshift(llsize, 1);
- ret = vhost_vdpa_dma_unmap(v->shared, VHOST_VDPA_GUEST_PA_ASID, iova,
+ ret = vhost_vdpa_dma_unmap(s, VHOST_VDPA_GUEST_PA_ASID, iova,
int128_get64(llsize));
if (ret) {
error_report("vhost_vdpa_dma_unmap(%p, 0x%" HWADDR_PRIx ", "
"0x%" HWADDR_PRIx ") = %d (%m)",
- v, iova, int128_get64(llsize), ret);
+ s, iova, int128_get64(llsize), ret);
}
iova += int128_get64(llsize);
}
- ret = vhost_vdpa_dma_unmap(v->shared, VHOST_VDPA_GUEST_PA_ASID, iova,
+ ret = vhost_vdpa_dma_unmap(s, VHOST_VDPA_GUEST_PA_ASID, iova,
int128_get64(llsize));
if (ret) {
error_report("vhost_vdpa_dma_unmap(%p, 0x%" HWADDR_PRIx ", "
"0x%" HWADDR_PRIx ") = %d (%m)",
- v, iova, int128_get64(llsize), ret);
+ s, iova, int128_get64(llsize), ret);
}
memory_region_unref(section->mr);
@@ -591,7 +585,7 @@ static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
v->dev = dev;
dev->opaque = opaque ;
- v->listener = vhost_vdpa_memory_listener;
+ v->shared->listener = vhost_vdpa_memory_listener;
vhost_vdpa_init_svq(dev, v);
error_propagate(&dev->migration_blocker, v->migration_blocker);
@@ -754,7 +748,7 @@ static int vhost_vdpa_cleanup(struct vhost_dev *dev)
}
vhost_vdpa_host_notifiers_uninit(dev, dev->nvqs);
- memory_listener_unregister(&v->listener);
+ memory_listener_unregister(&v->shared->listener);
vhost_vdpa_svq_cleanup(dev);
dev->opaque = NULL;
@@ -1327,7 +1321,7 @@ static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started)
"IOMMU and try again");
return -1;
}
- memory_listener_register(&v->listener, dev->vdev->dma_as);
+ memory_listener_register(&v->shared->listener, dev->vdev->dma_as);
return vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_DRIVER_OK);
}
@@ -1346,7 +1340,7 @@ static void vhost_vdpa_reset_status(struct vhost_dev *dev)
vhost_vdpa_reset_device(dev);
vhost_vdpa_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE |
VIRTIO_CONFIG_S_DRIVER);
- memory_listener_unregister(&v->listener);
+ memory_listener_unregister(&v->shared->listener);
}
static int vhost_vdpa_set_log_base(struct vhost_dev *dev, uint64_t base,
--
2.39.3
^ permalink raw reply related [flat|nested] 21+ messages in thread* Re: [PATCH 9.0 00/13] Consolidate common vdpa members in VhostVDPAShared
2023-11-24 17:14 [PATCH 9.0 00/13] Consolidate common vdpa members in VhostVDPAShared Eugenio Pérez
` (12 preceding siblings ...)
2023-11-24 17:14 ` [PATCH 9.0 13/13] vdpa: move memory listener to vhost_vdpa_shared Eugenio Pérez
@ 2023-11-30 3:21 ` Lei Yang
2023-11-30 7:38 ` Eugenio Perez Martin
2023-12-01 7:04 ` Jason Wang
14 siblings, 1 reply; 21+ messages in thread
From: Lei Yang @ 2023-11-30 3:21 UTC (permalink / raw)
To: Eugenio Pérez
Cc: qemu-devel, Parav Pandit, si-wei.liu, Stefano Garzarella,
Zhu Lingshan, Michael S. Tsirkin, Jason Wang, Dragos Tatulea,
Laurent Vivier
[-- Attachment #1: Type: text/plain, Size: 2609 bytes --]
Hi Eugenio
QE performed regression testing after applying this patch. This series
patch introduced a qemu core dump bug, for the core dump information
please review the attached file.
Tested-by: Lei Yang <leiyang@redhat.com>
On Sat, Nov 25, 2023 at 1:14 AM Eugenio Pérez <eperezma@redhat.com> wrote:
>
> Current memory operations like pinning may take a lot of time at the
> destination. Currently they are done after the source of the migration is
> stopped, and before the workload is resumed at the destination. This is a
> period where neigher traffic can flow, nor the VM workload can continue
> (downtime).
>
> We can do better as we know the memory layout of the guest RAM at the
> destination from the moment the migration starts. Moving that operation allows
> QEMU to communicate the kernel the maps while the workload is still running in
> the source, so Linux can start mapping them. Ideally, all IOMMU is configured,
> but if the vDPA parent driver uses on-chip IOMMU and .set_map we're still
> saving all the pinning time.
>
> This is a first required step to consolidate all the members in a common
> struct. This is needed because the destination does not know what vhost_vdpa
> struct will have the registered listener member, so it is easier to place them
> in a shared struct rather to keep them in vhost_vdpa struct.
>
> v1 from RFC:
> * Fix vhost_vdpa_net_cvq_start checking for always_svq instead of
> shadow_data. This could cause CVQ not being shadowed if
> vhost_vdpa_net_cvq_start was called in the middle of a migration.
>
> Eugenio Pérez (13):
> vdpa: add VhostVDPAShared
> vdpa: move iova tree to the shared struct
> vdpa: move iova_range to vhost_vdpa_shared
> vdpa: move shadow_data to vhost_vdpa_shared
> vdpa: use vdpa shared for tracing
> vdpa: move file descriptor to vhost_vdpa_shared
> vdpa: move iotlb_batch_begin_sent to vhost_vdpa_shared
> vdpa: move backend_cap to vhost_vdpa_shared
> vdpa: remove msg type of vhost_vdpa
> vdpa: move iommu_list to vhost_vdpa_shared
> vdpa: use VhostVDPAShared in vdpa_dma_map and unmap
> vdpa: use dev_shared in vdpa_iommu
> vdpa: move memory listener to vhost_vdpa_shared
>
> include/hw/virtio/vhost-vdpa.h | 36 +++++---
> hw/virtio/vdpa-dev.c | 7 +-
> hw/virtio/vhost-vdpa.c | 160 +++++++++++++++++----------------
> net/vhost-vdpa.c | 117 ++++++++++++------------
> hw/virtio/trace-events | 14 +--
> 5 files changed, 174 insertions(+), 160 deletions(-)
>
> --
> 2.39.3
>
>
[-- Attachment #2: core_dump_info --]
[-- Type: application/octet-stream, Size: 5901 bytes --]
(gdb) bt full
#0 memory_listener_unregister (listener=0x55a1ed7e2d98) at ../system/memory.c:3123
#1 0x000055a1e8ae5eba in vhost_vdpa_cleanup (dev=0x55a1ebe60470) at ../hw/virtio/vhost-vdpa.c:751
v = 0x55a1ecfd7f90
__PRETTY_FUNCTION__ = "vhost_vdpa_cleanup"
#2 0x000055a1e8addf7b in vhost_dev_cleanup (hdev=0x55a1ebe60470) at ../hw/virtio/vhost.c:1603
i = 1
__PRETTY_FUNCTION__ = "vhost_dev_cleanup"
#3 0x000055a1e89b7d26 in vhost_net_cleanup (net=0x55a1ebe60470) at ../hw/net/vhost_net.c:469
#4 0x000055a1e8b97976 in vhost_vdpa_cleanup (nc=0x55a1ecfd7e10) at ../net/vhost-vdpa.c:235
s = 0x55a1ecfd7e10
#5 0x000055a1e8b82aab in qemu_cleanup_net_client (nc=0x55a1ecfd7e10) at ../net/net.c:383
#6 0x000055a1e8b82d7b in qemu_del_net_client (nc=0x55a1ed52e260) at ../net/net.c:446
ncs =
{0x55a1ed52e260, 0x55a1ecfd7e10, 0x200000002, 0x55a1ed5e8b18, 0x55a1ed5e8b18, 0x55a1ed5e8c90, 0x0 <repeats 45 times>, 0x7f32e874f45f <__libc_connect+95>, 0x7ffdf7d70130, 0x6effffffff, 0x0, 0x7f32e879274e <open_socket+606>, 0x0, 0x0, 0x0, 0x7f32e8792530 <open_socket+64>, 0x0, 0xd, 0x0, 0x0, 0x0, 0x0, 0x722f7261762f0001, 0x2f6463736e2f6e75, 0x74656b636f73, 0x0 <repeats 12 times>, 0x571484d67a787b00, 0x0, 0xfffffffffffffb08, 0xd, 0x5, 0x7f32e87bbda8, 0xffffffffffffffff, 0x7ffdf7d703a0, 0x7f32e8792b1d <__nscd_get_mapping+189>, 0x0, 0x7f32e8792a9c <__nscd_get_mapping+60>, 0x0, 0x0, 0x7ffdf7d701f0, 0x7f32e880dd68 <__hst_map_handle+8>, 0x200000000, 0x6, 0x0 <repeats 11 times>, 0x7f32e86ae3b0 <malloc_consolidate+368>, 0x0, 0x7f32e87ffce0 <main_arena+96>, 0x0, 0x0, 0x0, 0x0, 0x0, 0x620, 0x618, 0x62, 0x618, 0x6e, 0x7c, 0x7f32e86b0a81 <_int_malloc+3281>, 0x7f32e87ffc80 <main_arena>, 0x60, 0x0, 0x7, 0x618, 0x7107d30e5b873e01, 0x48, 0x70, 0x640, 0x18, 0x4800000062, 0x0, 0x0, 0x571484d67a787b00, 0x770000007c, 0x0, 0x0, 0x7ffdf7d714a0, 0xf, 0x55a1ec3750a0, 0x7ffdf7d71530, 0x7f32e87909c6 <__nscd_get_nl_timestamp+214>, 0x16ca, 0x571484d67a787b00, 0x7ffdf7d71530, 0x571484d67a787b00, 0x14, 0x7ffdf7d71530, 0x0, 0x7f32e876a3a3 <__check_pf+1523>, 0x2000300000014, 0x16ca6567f272, 0x100000000, 0x10014, 0x0, 0x6001401000000, 0xffffffffffffffff, 0x29100000291, 0x8000080008, 0x2001400000048, 0x16ca6567f272, 0x60000400a, 0x5200202600010014, 0x10bfe58788480000, 0x6001435492812, 0x278cdb00093a5b, 0x3b6d700000a89, 0x20000080008, 0x2001400000048, 0x16ca6567f272, 0x6fd80400a, 0x80fe00010014, 0x1eafba9a00000000, 0x600142fddac91, 0xffffffffffffffff, 0x9ff000009ff, 0x28000080008, 0x2001400000048, 0x16ca6567f272, 0x16fd80400a, 0x80fe00010014, 0xfff6ceba00000000, 0x60014f0110afe, 0xffffffffffffffff, 0xda900000da90, 0x8000080008, 0x0 <repeats 83 times>, 0x7f32e86b0a81 <_int_malloc+3281>, 0x7f32e87ffc80 <main_arena>, 0x13f, 0x0, 0x7, 0x1401, 0x7107d30e5b873e01, 0x65, 0x7f32e86af3be <_int_free+2030>, 0x1430, 0x50, 0x8000000141, 0x100000000, 0x1, 0xa, 0x770000007c, 0x571484d67a787b00, 0x0, 0x17b0, 0xa0, 0x0, 0x81a00000, 0x0, 0xa0, 0x0, 0x81a00000, 0x0, 0x81a00100, 0x7ffdf7d70ab0, 0x81a00000, 0x0, 0x81a000a0, 0x0, 0x7ffdf7d70920, 0x55a1e8d3cd42 <addrrange_intersection+226>, 0x7ffdf7d708e0, 0x7ffdf7d70ab0, 0x81a00000...}
queues = 2
i = 1
nf = 0x0
next = 0x7ffdf7d71f88
__PRETTY_FUNCTION__ = "qemu_del_net_client"
#7 0x000055a1e8b84a9f in qmp_netdev_del (id=0x55a1ed18c770 "idqT0hia", errp=0x7ffdf7d71f98) at ../net/net.c:1325
nc = 0x55a1ed52e260
opts = 0x55a1ec2dc020
__func__ = "qmp_netdev_del"
#8 0x000055a1e8f5d7b8 in qmp_marshal_netdev_del (args=0x7f32dc0058f0, ret=0x7f32e7c34d98, errp=0x7f32e7c34d90) at qapi/qapi-commands-net.c:135
err = 0x0
ok = true
v = 0x55a1ec2dc020
arg = {id = 0x55a1ed18c770 "idqT0hia"}
#9 0x000055a1e8f9b241 in do_qmp_dispatch_bh (opaque=0x7f32e7c34e30) at ../qapi/qmp-dispatch.c:128
data = 0x7f32e7c34e30
__PRETTY_FUNCTION__ = "do_qmp_dispatch_bh"
#10 0x000055a1e8fc58e9 in aio_bh_call (bh=0x7f32dc004150) at ../util/async.c:169
last_engaged_in_io = false
reentrancy_guard = 0x0
#11 0x000055a1e8fc5a04 in aio_bh_poll (ctx=0x55a1ebed6fb0) at ../util/async.c:216
bh = 0x7f32dc004150
flags = 11
slice = {bh_list = {slh_first = 0x0}, next = {sqe_next = 0x0}}
s = 0x7ffdf7d72090
ret = 1
#12 0x000055a1e8fa85af in aio_dispatch (ctx=0x55a1ebed6fb0) at ../util/aio-posix.c:423
#13 0x000055a1e8fc5e43 in aio_ctx_dispatch (source=0x55a1ebed6fb0, callback=0x0, user_data=0x0) at ../util/async.c:358
ctx = 0x55a1ebed6fb0
__PRETTY_FUNCTION__ = "aio_ctx_dispatch"
#14 0x00007f32e8a7ef4f in g_main_dispatch (context=0x55a1ebed9520) at ../glib/gmain.c:3364
dispatch = 0x55a1e8fc5dec <aio_ctx_dispatch>
prev_source = 0x0
begin_time_nsec = 2687834462600
was_in_call = <optimized out>
user_data = 0x0
callback = 0x0
cb_funcs = 0x0
cb_data = 0x0
need_destroy = <optimized out>
source = 0x55a1ebed6fb0
current = 0x55a1ec33ffa0
i = 0
#15 g_main_context_dispatch (context=0x55a1ebed9520) at ../glib/gmain.c:4079
#16 0x000055a1e8fc73ae in glib_pollfds_poll () at ../util/main-loop.c:290
context = 0x55a1ebed9520
pfds = 0x55a1ec3ece00
#17 0x000055a1e8fc742b in os_host_main_loop_wait (timeout=0) at ../util/main-loop.c:313
context = 0x55a1ebed9520
ret = 1
#18 0x000055a1e8fc7539 in main_loop_wait (nonblocking=0) at ../util/main-loop.c:592
mlpoll = {state = 0, timeout = 4294967295, pollfds = 0x55a1ebebe860}
ret = 32765
timeout_ns = 798542174
#19 0x000055a1e8b22d27 in qemu_main_loop () at ../system/runstate.c:782
status = 0
#20 0x000055a1e881b706 in qemu_default_main () at ../system/main.c:37
status = 0
#21 0x000055a1e881b741 in main (argc=84, argv=0x7ffdf7d723c8) at ../system/main.c:48
^ permalink raw reply [flat|nested] 21+ messages in thread* Re: [PATCH 9.0 00/13] Consolidate common vdpa members in VhostVDPAShared
2023-11-30 3:21 ` [PATCH 9.0 00/13] Consolidate common vdpa members in VhostVDPAShared Lei Yang
@ 2023-11-30 7:38 ` Eugenio Perez Martin
2023-11-30 8:19 ` Lei Yang
0 siblings, 1 reply; 21+ messages in thread
From: Eugenio Perez Martin @ 2023-11-30 7:38 UTC (permalink / raw)
To: Lei Yang
Cc: qemu-devel, Parav Pandit, si-wei.liu, Stefano Garzarella,
Zhu Lingshan, Michael S. Tsirkin, Jason Wang, Dragos Tatulea,
Laurent Vivier
On Thu, Nov 30, 2023 at 4:22 AM Lei Yang <leiyang@redhat.com> wrote:
>
> Hi Eugenio
>
> QE performed regression testing after applying this patch. This series
> patch introduced a qemu core dump bug, for the core dump information
> please review the attached file.
>
Hi Lei, thank you very much for the testing!
Can you describe the test steps that lead to the crash? I think you
removed the vdpa device via QMP, but I'd like to be sure.
Thanks!
> Tested-by: Lei Yang <leiyang@redhat.com>
>
>
>
>
> On Sat, Nov 25, 2023 at 1:14 AM Eugenio Pérez <eperezma@redhat.com> wrote:
> >
> > Current memory operations like pinning may take a lot of time at the
> > destination. Currently they are done after the source of the migration is
> > stopped, and before the workload is resumed at the destination. This is a
> > period where neigher traffic can flow, nor the VM workload can continue
> > (downtime).
> >
> > We can do better as we know the memory layout of the guest RAM at the
> > destination from the moment the migration starts. Moving that operation allows
> > QEMU to communicate the kernel the maps while the workload is still running in
> > the source, so Linux can start mapping them. Ideally, all IOMMU is configured,
> > but if the vDPA parent driver uses on-chip IOMMU and .set_map we're still
> > saving all the pinning time.
> >
> > This is a first required step to consolidate all the members in a common
> > struct. This is needed because the destination does not know what vhost_vdpa
> > struct will have the registered listener member, so it is easier to place them
> > in a shared struct rather to keep them in vhost_vdpa struct.
> >
> > v1 from RFC:
> > * Fix vhost_vdpa_net_cvq_start checking for always_svq instead of
> > shadow_data. This could cause CVQ not being shadowed if
> > vhost_vdpa_net_cvq_start was called in the middle of a migration.
> >
> > Eugenio Pérez (13):
> > vdpa: add VhostVDPAShared
> > vdpa: move iova tree to the shared struct
> > vdpa: move iova_range to vhost_vdpa_shared
> > vdpa: move shadow_data to vhost_vdpa_shared
> > vdpa: use vdpa shared for tracing
> > vdpa: move file descriptor to vhost_vdpa_shared
> > vdpa: move iotlb_batch_begin_sent to vhost_vdpa_shared
> > vdpa: move backend_cap to vhost_vdpa_shared
> > vdpa: remove msg type of vhost_vdpa
> > vdpa: move iommu_list to vhost_vdpa_shared
> > vdpa: use VhostVDPAShared in vdpa_dma_map and unmap
> > vdpa: use dev_shared in vdpa_iommu
> > vdpa: move memory listener to vhost_vdpa_shared
> >
> > include/hw/virtio/vhost-vdpa.h | 36 +++++---
> > hw/virtio/vdpa-dev.c | 7 +-
> > hw/virtio/vhost-vdpa.c | 160 +++++++++++++++++----------------
> > net/vhost-vdpa.c | 117 ++++++++++++------------
> > hw/virtio/trace-events | 14 +--
> > 5 files changed, 174 insertions(+), 160 deletions(-)
> >
> > --
> > 2.39.3
> >
> >
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH 9.0 00/13] Consolidate common vdpa members in VhostVDPAShared
2023-11-30 7:38 ` Eugenio Perez Martin
@ 2023-11-30 8:19 ` Lei Yang
0 siblings, 0 replies; 21+ messages in thread
From: Lei Yang @ 2023-11-30 8:19 UTC (permalink / raw)
To: Eugenio Perez Martin
Cc: qemu-devel, Parav Pandit, si-wei.liu, Stefano Garzarella,
Zhu Lingshan, Michael S. Tsirkin, Jason Wang, Dragos Tatulea,
Laurent Vivier
On Thu, Nov 30, 2023 at 3:38 PM Eugenio Perez Martin
<eperezma@redhat.com> wrote:
>
> On Thu, Nov 30, 2023 at 4:22 AM Lei Yang <leiyang@redhat.com> wrote:
> >
> > Hi Eugenio
> >
> > QE performed regression testing after applying this patch. This series
> > patch introduced a qemu core dump bug, for the core dump information
> > please review the attached file.
> >
>
> Hi Lei, thank you very much for the testing!
>
Hi Eugenio
> Can you describe the test steps that lead to the crash? I think you
> removed the vdpa device via QMP, but I'd like to be sure.
Yes, you're right, the core dump occurs when hot unplug nic, please
review the following simple test steps:
Test Steps:
1. create two vdpa device(vdpa0 and vdpa1) with multi queues
2. Boot a guest with vdpa0
3. set_link false to vdpa0
4. hotplug vdpa1
5. stop and resume guest via QMP
6. hotunplug vdpa1, hit core dump in this time
Thanks
Lei
>
> Thanks!
>
> > Tested-by: Lei Yang <leiyang@redhat.com>
> >
> >
> >
> >
> > On Sat, Nov 25, 2023 at 1:14 AM Eugenio Pérez <eperezma@redhat.com> wrote:
> > >
> > > Current memory operations like pinning may take a lot of time at the
> > > destination. Currently they are done after the source of the migration is
> > > stopped, and before the workload is resumed at the destination. This is a
> > > period where neigher traffic can flow, nor the VM workload can continue
> > > (downtime).
> > >
> > > We can do better as we know the memory layout of the guest RAM at the
> > > destination from the moment the migration starts. Moving that operation allows
> > > QEMU to communicate the kernel the maps while the workload is still running in
> > > the source, so Linux can start mapping them. Ideally, all IOMMU is configured,
> > > but if the vDPA parent driver uses on-chip IOMMU and .set_map we're still
> > > saving all the pinning time.
> > >
> > > This is a first required step to consolidate all the members in a common
> > > struct. This is needed because the destination does not know what vhost_vdpa
> > > struct will have the registered listener member, so it is easier to place them
> > > in a shared struct rather to keep them in vhost_vdpa struct.
> > >
> > > v1 from RFC:
> > > * Fix vhost_vdpa_net_cvq_start checking for always_svq instead of
> > > shadow_data. This could cause CVQ not being shadowed if
> > > vhost_vdpa_net_cvq_start was called in the middle of a migration.
> > >
> > > Eugenio Pérez (13):
> > > vdpa: add VhostVDPAShared
> > > vdpa: move iova tree to the shared struct
> > > vdpa: move iova_range to vhost_vdpa_shared
> > > vdpa: move shadow_data to vhost_vdpa_shared
> > > vdpa: use vdpa shared for tracing
> > > vdpa: move file descriptor to vhost_vdpa_shared
> > > vdpa: move iotlb_batch_begin_sent to vhost_vdpa_shared
> > > vdpa: move backend_cap to vhost_vdpa_shared
> > > vdpa: remove msg type of vhost_vdpa
> > > vdpa: move iommu_list to vhost_vdpa_shared
> > > vdpa: use VhostVDPAShared in vdpa_dma_map and unmap
> > > vdpa: use dev_shared in vdpa_iommu
> > > vdpa: move memory listener to vhost_vdpa_shared
> > >
> > > include/hw/virtio/vhost-vdpa.h | 36 +++++---
> > > hw/virtio/vdpa-dev.c | 7 +-
> > > hw/virtio/vhost-vdpa.c | 160 +++++++++++++++++----------------
> > > net/vhost-vdpa.c | 117 ++++++++++++------------
> > > hw/virtio/trace-events | 14 +--
> > > 5 files changed, 174 insertions(+), 160 deletions(-)
> > >
> > > --
> > > 2.39.3
> > >
> > >
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH 9.0 00/13] Consolidate common vdpa members in VhostVDPAShared
2023-11-24 17:14 [PATCH 9.0 00/13] Consolidate common vdpa members in VhostVDPAShared Eugenio Pérez
` (13 preceding siblings ...)
2023-11-30 3:21 ` [PATCH 9.0 00/13] Consolidate common vdpa members in VhostVDPAShared Lei Yang
@ 2023-12-01 7:04 ` Jason Wang
14 siblings, 0 replies; 21+ messages in thread
From: Jason Wang @ 2023-12-01 7:04 UTC (permalink / raw)
To: Eugenio Pérez
Cc: qemu-devel, Parav Pandit, si-wei.liu, Stefano Garzarella,
Zhu Lingshan, Lei Yang, Michael S. Tsirkin, Dragos Tatulea,
Laurent Vivier
On Sat, Nov 25, 2023 at 1:14 AM Eugenio Pérez <eperezma@redhat.com> wrote:
>
> Current memory operations like pinning may take a lot of time at the
> destination. Currently they are done after the source of the migration is
> stopped, and before the workload is resumed at the destination. This is a
> period where neigher traffic can flow, nor the VM workload can continue
> (downtime).
>
> We can do better as we know the memory layout of the guest RAM at the
> destination from the moment the migration starts. Moving that operation allows
> QEMU to communicate the kernel the maps while the workload is still running in
> the source, so Linux can start mapping them. Ideally, all IOMMU is configured,
> but if the vDPA parent driver uses on-chip IOMMU and .set_map we're still
> saving all the pinning time.
>
> This is a first required step to consolidate all the members in a common
> struct. This is needed because the destination does not know what vhost_vdpa
> struct will have the registered listener member, so it is easier to place them
> in a shared struct rather to keep them in vhost_vdpa struct.
>
> v1 from RFC:
> * Fix vhost_vdpa_net_cvq_start checking for always_svq instead of
> shadow_data. This could cause CVQ not being shadowed if
> vhost_vdpa_net_cvq_start was called in the middle of a migration.
With the renaming of the VhostVDPAShared to VhostVDPAParent.
Acked-by: Jason Wang <jasowang@redhat.com>
Thanks
>
> Eugenio Pérez (13):
> vdpa: add VhostVDPAShared
> vdpa: move iova tree to the shared struct
> vdpa: move iova_range to vhost_vdpa_shared
> vdpa: move shadow_data to vhost_vdpa_shared
> vdpa: use vdpa shared for tracing
> vdpa: move file descriptor to vhost_vdpa_shared
> vdpa: move iotlb_batch_begin_sent to vhost_vdpa_shared
> vdpa: move backend_cap to vhost_vdpa_shared
> vdpa: remove msg type of vhost_vdpa
> vdpa: move iommu_list to vhost_vdpa_shared
> vdpa: use VhostVDPAShared in vdpa_dma_map and unmap
> vdpa: use dev_shared in vdpa_iommu
> vdpa: move memory listener to vhost_vdpa_shared
>
> include/hw/virtio/vhost-vdpa.h | 36 +++++---
> hw/virtio/vdpa-dev.c | 7 +-
> hw/virtio/vhost-vdpa.c | 160 +++++++++++++++++----------------
> net/vhost-vdpa.c | 117 ++++++++++++------------
> hw/virtio/trace-events | 14 +--
> 5 files changed, 174 insertions(+), 160 deletions(-)
>
> --
> 2.39.3
>
>
^ permalink raw reply [flat|nested] 21+ messages in thread