From: "Cédric Le Goater" <clg@redhat.com>
To: "Maciej S. Szmigiero" <mail@maciej.szmigiero.name>
Cc: "Alex Williamson" <alex.williamson@redhat.com>,
"Eric Blake" <eblake@redhat.com>, "Peter Xu" <peterx@redhat.com>,
"Fabiano Rosas" <farosas@suse.de>,
"Markus Armbruster" <armbru@redhat.com>,
"Daniel P . Berrangé" <berrange@redhat.com>,
"Avihai Horon" <avihaih@nvidia.com>,
"Joao Martins" <joao.m.martins@oracle.com>,
qemu-devel@nongnu.org
Subject: Re: [PATCH v4 27/33] vfio/migration: Multifd device state transfer support - received buffers queuing
Date: Mon, 17 Feb 2025 14:48:10 +0100 [thread overview]
Message-ID: <c1d138eb-687c-4158-931a-dfc6400622d7@redhat.com> (raw)
In-Reply-To: <50715039-1eb8-454b-9ab7-fb1490e27841@maciej.szmigiero.name>
On 2/14/25 21:58, Maciej S. Szmigiero wrote:
> On 12.02.2025 14:47, Cédric Le Goater wrote:
>> On 1/30/25 11:08, Maciej S. Szmigiero wrote:
>>> From: "Maciej S. Szmigiero" <maciej.szmigiero@oracle.com>
>>>
>>> The multifd received data needs to be reassembled since device state
>>> packets sent via different multifd channels can arrive out-of-order.
>>>
>>> Therefore, each VFIO device state packet carries a header indicating its
>>> position in the stream.
>>> The raw device state data is saved into a VFIOStateBuffer for later
>>> in-order loading into the device.
>>>
>>> The last such VFIO device state packet should have
>>> VFIO_DEVICE_STATE_CONFIG_STATE flag set and carry the device config state.
>>>
>>> Signed-off-by: Maciej S. Szmigiero <maciej.szmigiero@oracle.com>
>>> ---
>>> hw/vfio/migration.c | 116 ++++++++++++++++++++++++++++++++++
>>> hw/vfio/pci.c | 2 +
>>> hw/vfio/trace-events | 1 +
>>> include/hw/vfio/vfio-common.h | 1 +
>>> 4 files changed, 120 insertions(+)
>>>
>>> diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c
>>> index bcdf204d5cf4..0c0caec1bd64 100644
>>> --- a/hw/vfio/migration.c
>>> +++ b/hw/vfio/migration.c
>>> @@ -301,6 +301,12 @@ typedef struct VFIOStateBuffer {
>>> } VFIOStateBuffer;
>>> typedef struct VFIOMultifd {
>>> + VFIOStateBuffers load_bufs;
>>> + QemuCond load_bufs_buffer_ready_cond;
>>> + QemuMutex load_bufs_mutex; /* Lock order: this lock -> BQL */
>>> + uint32_t load_buf_idx;
>>> + uint32_t load_buf_idx_last;
>>> + uint32_t load_buf_queued_pending_buffers;
>>> } VFIOMultifd;
>>> static void vfio_state_buffer_clear(gpointer data)
>>> @@ -346,6 +352,103 @@ static VFIOStateBuffer *vfio_state_buffers_at(VFIOStateBuffers *bufs, guint idx)
>>> return &g_array_index(bufs->array, VFIOStateBuffer, idx);
>>> }
>> Each routine executed from a migration thread should have a preliminary
>> comment saying from which context it is called: migration or VFIO
>
> Do you mean like whether it is called from the code in qemu/migration/
> directory or the code in hw/vfio/ directory?
Threads are spawned from different subsystems: migration callbacks
(save), and from VFIO (load, well, not load phase, switchover phase).
It would be good to provide hints to the reader.
I am struggling to understand how this works. Imagine a new comer
looking at the code and at the git history in 2y time ... Check
vfio in QEMU 1.3 (one small file) and see what it has become today.
> What about internal linkage ("static") functions?
There shouldn't be any static left when all multifd code is moved
to its own hw/vfio/migration-multifd.c file.
> Do they need such comment too? That would actually decrease the readability> of these one-or-two line helpers due to high comment-to-code ratio.
I meant the higher level routines.
Tbh, this lacks tons of documentation, under docs, under each file,
for the properties, etc. This should be addressed before resend.
> As far as I can see, pretty much no existing VFIO migration function
> has such comment.> >>> +static bool vfio_load_state_buffer_insert(VFIODevice *vbasedev,
>>> + VFIODeviceStatePacket *packet,
>>> + size_t packet_total_size,
>>> + Error **errp)
>>> +{
>>> + VFIOMigration *migration = vbasedev->migration;
>>> + VFIOMultifd *multifd = migration->multifd;
>>> + VFIOStateBuffer *lb;
>>> +
>>> + vfio_state_buffers_assert_init(&multifd->load_bufs);
>>> + if (packet->idx >= vfio_state_buffers_size_get(&multifd->load_bufs)) {
>>> + vfio_state_buffers_size_set(&multifd->load_bufs, packet->idx + 1);
>>> + }
>>> +
>>> + lb = vfio_state_buffers_at(&multifd->load_bufs, packet->idx);
>>> + if (lb->is_present) {
>>> + error_setg(errp, "state buffer %" PRIu32 " already filled",
>>> + packet->idx);
>>> + return false;
>>> + }
>>> +
>>> + assert(packet->idx >= multifd->load_buf_idx);
>>> +
>>> + multifd->load_buf_queued_pending_buffers++;
>>> + if (multifd->load_buf_queued_pending_buffers >
>>> + vbasedev->migration_max_queued_buffers) {
>>> + error_setg(errp,
>>> + "queuing state buffer %" PRIu32 " would exceed the max of %" PRIu64,
>>> + packet->idx, vbasedev->migration_max_queued_buffers);
>>> + return false;
>>> + }
>>
>> AFAICT, attributes multifd->load_buf_queued_pending_buffers and
>> vbasedev->migration_max_queued_buffers are not strictly necessary.
>> They allow to count buffers and check an arbitrary limit, which
>> is UINT64_MAX today. It makes me wonder how useful they are.
>
> You are right they aren't strictly necessary and in fact they weren't
> there in early versions of this patch set.
>
> It was introduced upon Peter's request since otherwise the source> could theoretically cause the target QEMU to allocate unlimited
> amounts of memory for buffers-in-flight:
> https://lore.kernel.org/qemu-devel/9e85016e-ac72-4207-8e69-8cba054cefb7@maciej.szmigiero.name/
> (scroll to the "Risk of OOM on unlimited VFIO buffering" section).
>
> If that's an actual risk in someone's use case then that person
> could lower that limit from UINT64_MAX to, for example, 10 buffers.
> >> Please introduce them in a separate patch at the end of the series,
>> adding documentation on the "x-migration-max-queued-buffers" property
>> and also general documentation on why and how to use it.
>
> I can certainly move it to the end of the series - done now.
Great. Please add the comment above in the commit log. We will decide
it this is experimental or not.
Also, I wonder if this should be a global migration property.
>>> +
>>> + lb->data = g_memdup2(&packet->data, packet_total_size - sizeof(*packet));
>>> + lb->len = packet_total_size - sizeof(*packet);
>>> + lb->is_present = true;
>>> +
>>> + return true;
>>> +}
>>> +
>>> +static bool vfio_load_state_buffer(void *opaque, char *data, size_t data_size,
>>> + Error **errp)
>>> +{
>>> + VFIODevice *vbasedev = opaque;
>>> + VFIOMigration *migration = vbasedev->migration;
>>> + VFIOMultifd *multifd = migration->multifd;
>>> + VFIODeviceStatePacket *packet = (VFIODeviceStatePacket *)data;
>>> +
>>> + /*
>>> + * Holding BQL here would violate the lock order and can cause
>>> + * a deadlock once we attempt to lock load_bufs_mutex below.
>>> + */
>>> + assert(!bql_locked());
>>> +
>>> + if (!migration->multifd_transfer) {
>>> + error_setg(errp,
>>> + "got device state packet but not doing multifd transfer");
>>> + return false;
>>> + }
>>> +
>>> + assert(multifd);
>>> +
>>> + if (data_size < sizeof(*packet)) {
>>> + error_setg(errp, "packet too short at %zu (min is %zu)",
>>> + data_size, sizeof(*packet));
>>> + return false;
>>> + }
>>> +
>>> + if (packet->version != 0) {
>>
>> Please add a define for version, even if 0.
>
> I've introduced a new define VFIO_DEVICE_STATE_PACKET_VER_CURRENT.
>
>>> + error_setg(errp, "packet has unknown version %" PRIu32,
>>> + packet->version);
>>> + return false;
>>> + }
>>> +
>>> + if (packet->idx == UINT32_MAX) {
>>> + error_setg(errp, "packet has too high idx %" PRIu32,
>>> + packet->idx);
>>
>> I don't think printing out packet->idx is useful here.
>
> Yeah, it's unlikely that the value of UINT32_MAX will ever change :)
>
> Removed now.
>
>>> + return false;
>>> + }
>>> +
>>> + trace_vfio_load_state_device_buffer_incoming(vbasedev->name, packet->idx);
>>
>> I wonder if we can add thread ids to trace events. It would be useful.
>
> load_state_buffer is called from multifd channel receive threads
> so passing multifd channel id there would require adding this multifd-specific
> parameter to qemu_loadvm_load_state_buffer() and load_state_buffer
> SaveVMHandler.
'-msg timestamp=on' should be enough. Having logical thread names would
be nice. It's another topic.
>>> +
>>> + QEMU_LOCK_GUARD(&multifd->load_bufs_mutex);
>>> +
>>> + /* config state packet should be the last one in the stream */
>>> + if (packet->flags & VFIO_DEVICE_STATE_CONFIG_STATE) {
>>> + multifd->load_buf_idx_last = packet->idx;
>>> + }
>>> +
>>> + if (!vfio_load_state_buffer_insert(vbasedev, packet, data_size, errp)) {
>>
>> So the migration thread calling multifd_device_state_recv() will
>> exit
>
> The thread is calling multifd_device_state_recv() is a multifd
> channel receive thread.
>
>> and the vfio thread loading the state into the device will
>> hang until its aborted ?
>
> In the normal (successful) migration flow the vfio_load_bufs_thread()
> will exit after loading (write()'ing) all buffers into the device
> and then loading its config state.
>
> In the aborted/error/unsuccessful migration flow it will get
> terminated from vfio_load_cleanup() -> vfio_multifd_free() ->
> vfio_load_cleanup_load_bufs_thread().
>
> vfio_load_cleanup_load_bufs_thread() will signal
> load_bufs_buffer_ready_cond and load_bufs_iter_done_cond since
> the load thread indeed could be waiting on them.
>
>
>>
>> This sequence is expected to be called to release the vfio thread
>>
>> while (multifd->load_bufs_thread_running) {
>> multifd->load_bufs_thread_want_exit = true;
>>
>> qemu_cond_signal(&multifd->load_bufs_buffer_ready_cond);
>> ...
>> }
>>
>> right ?
>
> Right, that's a part of the code in vfio_load_cleanup_load_bufs_thread().
ok. So I think this lacks comments on thread termination points.
Please try to comment a bit more these areas in the code. I will
check next version more closely.
>> The way the series is presented makes it a bit complex to follow the
>> proposition, especially regarding the creation and termination of
>> threads, something the reader should be aware of.
>>
>> As an initial step in clarifying the design, I would have preferred
>> a series of patches introducing the various threads, migration threads
>> and VFIO threads, without any workload. Once the creation and termination
>> points are established I would then introduce the work load for each
>> thread.
>
> When I am doing review of anything more complex (though it's not usually
> in QEMU) I mainly follow the final code flow as an operation is handled
> since looking just from top to down at individual commits rarely gives
> enough context to see how every part interacts together.
>
> But for this the reviewer needs to see the whole code for the logical
> operation, rather than just a part of it.
and this is the very problematic :/ Very very hard to maintain on the
long run. I also don't have *time* to dig in all the context. So please
try to keep it as simple as possible.
> I think that adding the load operation in parts doesn't really
> help since the reason why things are done such way in earlier patches
> are only apparent in later patches and the earlier parts doesn't
> really have much sense on their own.
> Not to mention extra code churn when rebasing/reworking that increases
> chance of a typo or a copy-paste mistake happening at some point.
> > I also see that in comments to a later patch you dislike that
> a dummy vfio_load_bufs_thread_load_config() gets added in one patch
> then immediately replaced by the real implementation in the next patch.
> Previously, you also said that vfio_load_config_after_iter() seems
> to be unused in the patch that adds it - that's exactly the kind of
> issues that bringing the complete operation in one patch avoids.
May be I did. Sorry I switched context may times already and this
was lost in oblivion. Again, please help the reviewer. Changes
should be made obvious.
> I agree that, for example, x-migration-load-config-after-iter feature
> could be a separate patch as it is a relatively simple change.
>
> Same goes for x-migration-max-queued-buffers checking/enforcement,
> compat changes, exporting existing settings (variables) as properties
> or adding a g_autoptr() cleanup function for an existing type.
>
> That's why originally the VFIO part of the series was divided into two
> parts - receive and send, since these are two separate, yet internally
> complete operations.
I am now asking to have a better understanding of how threads are
created/terminated. It's another sub split of the load part AFAICT.
If you prefer we can forget about the load thread first, like I
asked initially iirc. I would very much prefer that for QEMU 10.0.
> I also export the whole series (including the current WiP state, with
> code moved to migration-multifd.{c,h} files, etc.) as a git tree at
> https://gitlab.com/maciejsszmigiero/qemu/-/commits/multifd-device-state-transfer-vfio
> since this way it can be easily seen how the QEMU code currently
> looks after the whole patch set or set of patches there.
Overall, I think this is making great progress. For such a complex
work, I would imagine a couple of RFCs first and half dozen normal
series. So ~10 iterations. We are only at v4. At least two more are
expected.
Thanks,
C.
next prev parent reply other threads:[~2025-02-17 13:48 UTC|newest]
Thread overview: 137+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-01-30 10:08 [PATCH v4 00/33] Multifd 🔀 device state transfer support with VFIO consumer Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 01/33] migration: Clarify that {load, save}_cleanup handlers can run without setup Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 02/33] thread-pool: Remove thread_pool_submit() function Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 03/33] thread-pool: Rename AIO pool functions to *_aio() and data types to *Aio Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 04/33] thread-pool: Implement generic (non-AIO) pool support Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 05/33] migration: Add MIG_CMD_SWITCHOVER_START and its load handler Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 06/33] migration: Add qemu_loadvm_load_state_buffer() and its handler Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 07/33] io: tls: Allow terminating the TLS session gracefully with EOF Maciej S. Szmigiero
2025-02-04 15:15 ` Daniel P. Berrangé
2025-02-04 16:02 ` Maciej S. Szmigiero
2025-02-04 16:14 ` Daniel P. Berrangé
2025-02-04 18:25 ` Maciej S. Szmigiero
2025-02-06 21:53 ` Peter Xu
2025-01-30 10:08 ` [PATCH v4 08/33] migration/multifd: Allow premature EOF on TLS incoming channels Maciej S. Szmigiero
2025-02-03 18:20 ` Peter Xu
2025-02-03 18:53 ` Maciej S. Szmigiero
2025-02-03 20:20 ` Peter Xu
2025-02-03 21:41 ` Maciej S. Szmigiero
2025-02-03 22:56 ` Peter Xu
2025-02-04 13:51 ` Fabiano Rosas
2025-02-04 14:39 ` Maciej S. Szmigiero
2025-02-04 15:00 ` Fabiano Rosas
2025-02-04 15:10 ` Maciej S. Szmigiero
2025-02-04 15:31 ` Peter Xu
2025-02-04 15:39 ` Daniel P. Berrangé
2025-02-05 19:09 ` Fabiano Rosas
2025-02-05 20:42 ` Fabiano Rosas
2025-02-05 20:55 ` Maciej S. Szmigiero
2025-02-06 14:13 ` Fabiano Rosas
2025-02-06 14:53 ` Maciej S. Szmigiero
2025-02-06 15:20 ` Fabiano Rosas
2025-02-06 16:01 ` Maciej S. Szmigiero
2025-02-06 17:32 ` Fabiano Rosas
2025-02-06 17:55 ` Maciej S. Szmigiero
2025-02-06 21:51 ` Peter Xu
2025-02-07 13:17 ` Fabiano Rosas
2025-02-07 14:04 ` Peter Xu
2025-02-07 14:16 ` Fabiano Rosas
2025-02-05 21:13 ` Peter Xu
2025-02-06 14:19 ` Fabiano Rosas
2025-02-04 15:10 ` Daniel P. Berrangé
2025-02-04 15:08 ` Daniel P. Berrangé
2025-02-04 16:02 ` Peter Xu
2025-02-04 16:12 ` Daniel P. Berrangé
2025-02-04 16:29 ` Peter Xu
2025-02-04 18:25 ` Fabiano Rosas
2025-02-04 19:34 ` Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 09/33] migration: postcopy_ram_listen_thread() needs to take BQL for some calls Maciej S. Szmigiero
2025-02-02 2:06 ` Dr. David Alan Gilbert
2025-02-02 11:55 ` Maciej S. Szmigiero
2025-02-02 12:45 ` Dr. David Alan Gilbert
2025-02-03 13:57 ` Maciej S. Szmigiero
2025-02-03 19:58 ` Peter Xu
2025-02-03 20:15 ` Maciej S. Szmigiero
2025-02-03 20:36 ` Peter Xu
2025-02-03 21:41 ` Maciej S. Szmigiero
2025-02-03 23:02 ` Peter Xu
2025-02-04 14:57 ` Maciej S. Szmigiero
2025-02-04 15:39 ` Peter Xu
2025-02-04 19:32 ` Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 10/33] error: define g_autoptr() cleanup function for the Error type Maciej S. Szmigiero
2025-02-03 20:53 ` Peter Xu
2025-02-03 21:13 ` Daniel P. Berrangé
2025-02-03 21:51 ` Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 11/33] migration: Add thread pool of optional load threads Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 12/33] migration/multifd: Split packet into header and RAM data Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 13/33] migration/multifd: Device state transfer support - receive side Maciej S. Szmigiero
2025-02-03 21:27 ` Peter Xu
2025-02-03 22:18 ` Maciej S. Szmigiero
2025-02-03 22:59 ` Peter Xu
2025-02-04 14:40 ` Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 14/33] migration/multifd: Make multifd_send() thread safe Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 15/33] migration/multifd: Add an explicit MultiFDSendData destructor Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 16/33] migration/multifd: Device state transfer support - send side Maciej S. Szmigiero
2025-02-03 21:47 ` Peter Xu
2025-01-30 10:08 ` [PATCH v4 17/33] migration/multifd: Make MultiFDSendData a struct Maciej S. Szmigiero
2025-02-07 14:36 ` Fabiano Rosas
2025-02-07 19:43 ` Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 18/33] migration/multifd: Add multifd_device_state_supported() Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 19/33] migration: Add save_live_complete_precopy_thread handler Maciej S. Szmigiero
2025-02-04 17:54 ` Peter Xu
2025-02-04 19:32 ` Maciej S. Szmigiero
2025-02-04 20:34 ` Peter Xu
2025-02-05 11:53 ` Maciej S. Szmigiero
2025-02-05 15:55 ` Peter Xu
2025-02-06 11:41 ` Maciej S. Szmigiero
2025-02-06 22:16 ` Peter Xu
2025-01-30 10:08 ` [PATCH v4 20/33] vfio/migration: Add x-migration-load-config-after-iter VFIO property Maciej S. Szmigiero
2025-02-10 17:24 ` Cédric Le Goater
2025-02-11 14:37 ` Maciej S. Szmigiero
2025-02-11 15:00 ` Cédric Le Goater
2025-02-11 15:57 ` Maciej S. Szmigiero
2025-02-11 16:28 ` Cédric Le Goater
2025-01-30 10:08 ` [PATCH v4 21/33] vfio/migration: Add load_device_config_state_start trace event Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 22/33] vfio/migration: Convert bytes_transferred counter to atomic Maciej S. Szmigiero
2025-01-30 21:35 ` Cédric Le Goater
2025-01-31 9:47 ` Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 23/33] vfio/migration: Multifd device state transfer support - basic types Maciej S. Szmigiero
2025-02-10 17:17 ` Cédric Le Goater
2025-01-30 10:08 ` [PATCH v4 24/33] vfio/migration: Multifd device state transfer support - VFIOStateBuffer(s) Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 25/33] vfio/migration: Multifd device state transfer - add support checking function Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 26/33] vfio/migration: Multifd device state transfer support - receive init/cleanup Maciej S. Szmigiero
2025-02-12 10:55 ` Cédric Le Goater
2025-02-14 20:55 ` Maciej S. Szmigiero
2025-02-17 9:38 ` Cédric Le Goater
2025-02-17 22:13 ` Maciej S. Szmigiero
2025-02-18 7:54 ` Cédric Le Goater
2025-01-30 10:08 ` [PATCH v4 27/33] vfio/migration: Multifd device state transfer support - received buffers queuing Maciej S. Szmigiero
2025-02-12 13:47 ` Cédric Le Goater
2025-02-14 20:58 ` Maciej S. Szmigiero
2025-02-17 13:48 ` Cédric Le Goater [this message]
2025-02-17 22:15 ` Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 28/33] vfio/migration: Multifd device state transfer support - load thread Maciej S. Szmigiero
2025-02-12 15:48 ` Cédric Le Goater
2025-02-12 16:19 ` Cédric Le Goater
2025-02-17 22:09 ` Maciej S. Szmigiero
2025-02-17 22:09 ` Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 29/33] vfio/migration: Multifd device state transfer support - config loading support Maciej S. Szmigiero
2025-02-12 16:21 ` Cédric Le Goater
2025-02-17 22:09 ` Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 30/33] migration/qemu-file: Define g_autoptr() cleanup function for QEMUFile Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 31/33] vfio/migration: Multifd device state transfer support - send side Maciej S. Szmigiero
2025-02-12 17:03 ` Cédric Le Goater
2025-02-17 22:12 ` Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 32/33] vfio/migration: Add x-migration-multifd-transfer VFIO property Maciej S. Szmigiero
2025-02-12 17:10 ` Cédric Le Goater
2025-02-14 20:56 ` Maciej S. Szmigiero
2025-02-17 13:57 ` Cédric Le Goater
2025-02-17 14:16 ` Maciej S. Szmigiero
2025-01-30 10:08 ` [PATCH v4 33/33] hw/core/machine: Add compat for " Maciej S. Szmigiero
2025-01-30 20:19 ` [PATCH v4 00/33] Multifd 🔀 device state transfer support with VFIO consumer Fabiano Rosas
2025-01-30 20:27 ` Maciej S. Szmigiero
2025-01-30 20:46 ` Fabiano Rosas
2025-01-31 18:16 ` Maciej S. Szmigiero
2025-02-03 14:19 ` Cédric Le Goater
2025-02-21 6:57 ` Yanghang Liu
2025-02-22 9:51 ` Maciej S. Szmigiero
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=c1d138eb-687c-4158-931a-dfc6400622d7@redhat.com \
--to=clg@redhat.com \
--cc=alex.williamson@redhat.com \
--cc=armbru@redhat.com \
--cc=avihaih@nvidia.com \
--cc=berrange@redhat.com \
--cc=eblake@redhat.com \
--cc=farosas@suse.de \
--cc=joao.m.martins@oracle.com \
--cc=mail@maciej.szmigiero.name \
--cc=peterx@redhat.com \
--cc=qemu-devel@nongnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).