[RFC PATCH v2 0/3] xl: Allow PCI devices to be passthrough'd via Qemu command line

[RFC PATCH v2 0/3] xl: Allow PCI devices to be passthrough'd via Qemu command line

[Thierry Escande]

This series adds support for a new 'hotplug' option for PCI devices to be
passthrough'd to guests.

The current mechanism for device passthrough uses QMP device_add command
to hot-plug PCI devices to the guest Qemu instance. This is an issue for
guests running on Q35 chipset (preliminary support posted at [1]) since the
Q35 PCI root bus does not support hotplug. Devices could be hotplugged to
a secondary PCI bus but Xen only support 1 PCI bus for now.

The 'hotplug' option allows to control how devices are attached to the
guest, either by using the legacy QMP mechanism (this is the default) or by
passing it directly to the Qemu command line using xen-pci-passthrough
device.

Example usage in cfg file:
 pci = [ "00:03.0,seize=1,hotplug=0" ]

Since Qemu -device option accepts parameters in json format, this
patchset adds a new internal function libxl__device_pci_get_qmp_json() that
generates the device json object used for both QMP hotplug and command line
passing.
Also, the function libxl_pci_assignable() is made available from libxl.h
for access in libxl_dm.c. I don't know if it's ok to make it public or if
it should stay private. Maybe it should be renamed as
libxl_device_pci_assignable() to match the other PCI device APIs.

This has been successfully tested on Xen 4.22-dev and Qemu 10.0 running
Debian VMs in both Bios and UEFI mode with a passthrough'd nvme disk.
Disabling the hotplug mechanism might be made mandatory for Q35 machines
later, once Q35 support is merged upstream.

[1] https://lore.kernel.org/xen-devel/20260313163455.790692-1-thierry.escande@vates.tech/

Changes in v2:
- Add support for YAJL json parser

Thierry Escande (3):
  xl: Add pci device hotplug option
  libxl: Allow PCI device passthrough using -device Qemu command line
  docs: provide description for pci hotplug option

 docs/man/xl-pci-configuration.5.pod | 17 ++++++
 docs/man/xl.cfg.5.pod.in            |  6 +++
 tools/include/libxl.h               |  1 +
 tools/libs/light/libxl_dm.c         | 80 +++++++++++++++++++++++++++++
 tools/libs/light/libxl_internal.h   |  2 +
 tools/libs/light/libxl_pci.c        | 41 +++++++++++----
 tools/libs/light/libxl_types.idl    |  1 +
 tools/libs/util/libxlu_pci.c        |  2 +
 tools/xl/xl_parse.c                 |  5 ++
 9 files changed, 145 insertions(+), 10 deletions(-)

-- 
2.53.0



--
Thierry Escande | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

[RFC PATCH v2 1/3] xl: Add pci device hotplug option

[Thierry Escande]

This option is used to passthrough PCI devices using the Qemu command
line (-device driver,...) instead of the QMP hot-plug mechanism.

This is needed for Q35 support since its PCI root bus doesn't support
hotplugging.

If not specified, the default behavior is to use hotplug via QMP.

Signed-off-by: Thierry Escande <thierry.escande@vates.tech>
---
v2: no change
---
 tools/libs/light/libxl_types.idl | 1 +
 tools/libs/util/libxlu_pci.c     | 2 ++
 tools/xl/xl_parse.c              | 5 +++++
 3 files changed, 8 insertions(+)

diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl
index a7893460f0..aca0e93793 100644
--- a/tools/libs/light/libxl_types.idl
+++ b/tools/libs/light/libxl_types.idl
@@ -922,6 +922,7 @@ libxl_device_pci = Struct("device_pci", [
     ("seize", bool),
     ("rdm_policy", libxl_rdm_reserve_policy),
     ("name", string),
+    ("hotplug", bool),
     ])
 
 libxl_device_rdm = Struct("device_rdm", [
diff --git a/tools/libs/util/libxlu_pci.c b/tools/libs/util/libxlu_pci.c
index 294482c6d7..f6440c878b 100644
--- a/tools/libs/util/libxlu_pci.c
+++ b/tools/libs/util/libxlu_pci.c
@@ -192,6 +192,8 @@ int xlu_pci_parse_spec_string(XLU_Config *cfg, libxl_device_pci *pci,
             name_present = true;
             pci->name = strdup(val);
             if (!pci->name) ret = ERROR_NOMEM;
+        } else if (!strcmp(key, "hotplug")) {
+            pci->hotplug = atoi(val);
         } else {
             XLU__PCI_ERR(cfg, "Unknown PCI_SPEC_STRING option: %s", key);
             ret = ERROR_INVAL;
diff --git a/tools/xl/xl_parse.c b/tools/xl/xl_parse.c
index 48c72dce9c..7ea2a76662 100644
--- a/tools/xl/xl_parse.c
+++ b/tools/xl/xl_parse.c
@@ -1361,6 +1361,7 @@ void parse_config_data(const char *config_source,
     int pci_msitranslate = 0;
     int pci_permissive = 0;
     int pci_seize = 0;
+    int pci_hotplug = 1;
     int i, e;
     int num_llc_colors;
     int num_xs_quota;
@@ -1699,6 +1700,9 @@ void parse_config_data(const char *config_source,
     if (!xlu_cfg_get_long (config, "pci_seize", &l, 0))
         pci_seize = l;
 
+    if (!xlu_cfg_get_long (config, "pci_hotplug", &l, 0))
+        pci_hotplug = l;
+
     if (!xlu_cfg_get_string(config, "rdm", &buf, 0)) {
         libxl_rdm_reserve rdm;
         if (!xlu_rdm_parse(config, &rdm, buf)) {
@@ -1720,6 +1724,7 @@ void parse_config_data(const char *config_source,
             pci->power_mgmt = pci_power_mgmt;
             pci->permissive = pci_permissive;
             pci->seize = pci_seize;
+            pci->hotplug = pci_hotplug;
             /*
              * Like other pci option, the per-device policy always follows
              * the global policy by default.
-- 
2.53.0



--
Thierry Escande | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

[RFC PATCH v2 2/3] libxl: Allow PCI device passthrough using -device Qemu command line

[Thierry Escande]

This change makes use of the new option 'hotplug' for host PCI devices
passthrough'd to the guest. If hotplug=0 is used in the pci device
configuration table, the device will be attached to the guest using the
Qemu command line as '-device xen-pci-passthrough,hostaddr=...'

The host device configuration is passed to the -device option as a json
array, just like it's done for hotplug using QMP. The json array is
created by a new internal function libxl__device_pci_get_qmp_json() that
is also used by pci_add_qmp_device_add().

Then, instead of sending the 'device_add' command, the device_add
callback is called to perform the 'query-pci' check to make sure the
passthrough'd device is present.

In the same way at shutdown, the device is not removed using QMP and
only the pci_remove_done() function is called.

As with QMP, the use of the 'hotplug=0' option honors the 'seize' option
by adding the PCI device to the assignable list if needed.

Example use:
 pci = [ "00:03.0,seize=1,hotplug=0" ]

Signed-off-by: Thierry Escande <thierry.escande@vates.tech>
---
v2:
- Add support for YAJL json parser
---
 tools/include/libxl.h             |  1 +
 tools/libs/light/libxl_dm.c       | 80 +++++++++++++++++++++++++++++++
 tools/libs/light/libxl_internal.h |  2 +
 tools/libs/light/libxl_pci.c      | 41 ++++++++++++----
 4 files changed, 114 insertions(+), 10 deletions(-)

diff --git a/tools/include/libxl.h b/tools/include/libxl.h
index 7c098edab6..66fb07ad67 100644
--- a/tools/include/libxl.h
+++ b/tools/include/libxl.h
@@ -2666,6 +2666,7 @@ int libxl_device_pci_assignable_add(libxl_ctx *ctx, libxl_device_pci *pci, int r
 int libxl_device_pci_assignable_remove(libxl_ctx *ctx, libxl_device_pci *pci, int rebind);
 libxl_device_pci *libxl_device_pci_assignable_list(libxl_ctx *ctx, int *num);
 void libxl_device_pci_assignable_list_free(libxl_device_pci *list, int num);
+bool libxl_pci_assignable(libxl_ctx *ctx, libxl_device_pci *pci);
 
 /* CPUID handling */
 int libxl_cpuid_parse_config(libxl_cpuid_policy_list *cpuid, const char* str);
diff --git a/tools/libs/light/libxl_dm.c b/tools/libs/light/libxl_dm.c
index 511ec76a65..4f4781b36d 100644
--- a/tools/libs/light/libxl_dm.c
+++ b/tools/libs/light/libxl_dm.c
@@ -1169,6 +1169,86 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,
                     }
                 }
             }
+
+            if (guest_config->num_pcidevs) {
+                libxl_ctx *ctx = libxl__gc_owner(gc);
+                libxl_device_pci *pci;
+                libxl__json_object *qmp_json;
+                char *json_str;
+#ifdef HAVE_LIBJSONC
+                json_object *jso;
+                const char *buf;
+#elif defined(HAVE_LIBYAJL)
+                yajl_gen hand;
+                /* memory for 'buf' is owned by 'hand' */
+                const unsigned char *buf;
+                libxl_yajl_length len;
+#else
+#               error Missing JSON library
+#endif
+
+                for (i = 0; i < guest_config->num_pcidevs; i++) {
+                    pci = &guest_config->pcidevs[i];
+
+                    if (pci->hotplug)
+                        continue;
+
+                    if (!libxl_pci_assignable(ctx, pci) && pci->seize) {
+                        rc = libxl_device_pci_assignable_add(ctx, pci, 1);
+                        if (rc)
+                            return rc;
+                    }
+
+                    qmp_json = libxl__device_pci_get_qmp_json(gc, pci);
+
+#ifdef HAVE_LIBJSONC
+                    rc = libxl__json_object_to_json_object(gc, &jso, qmp_json);
+                    if (rc)
+                        return rc;
+
+                    buf = json_object_to_json_string_ext(jso,
+                                                         JSON_C_TO_STRING_PLAIN);
+                    if (!buf) {
+                        json_object_put(jso);
+                        return ERROR_NOMEM;
+                    }
+#elif defined(HAVE_LIBYAJL)
+                    hand = libxl_yajl_gen_alloc(NULL);
+                    if (!hand) {
+                        return ERROR_NOMEM;
+                    }
+#if HAVE_YAJL_V2
+                    /* Disable beautify for data sent to QEMU */
+                    yajl_gen_config(hand, yajl_gen_beautify, 0);
+#endif
+
+                    rc = libxl__json_object_to_yajl_gen(gc, hand, qmp_json);
+                    if (rc) {
+                        yajl_gen_free(hand);
+                        return rc;
+                    }
+
+                    rc = yajl_gen_get_buf(hand, &buf, &len);
+                    if (rc != yajl_gen_status_ok) {
+                        yajl_gen_free(hand);
+                        return rc;
+                    }
+#endif
+
+                    json_str = libxl__strdup(gc, (const char *)buf);
+                    if (json_str)
+                        flexarray_vappend(dm_args, "-device", json_str, NULL);
+
+#ifdef HAVE_LIBJSONC
+                    json_object_put(jso);
+#elif defined(HAVE_LIBYAJL)
+                    yajl_gen_free(hand);
+#endif
+
+                    if (!json_str)
+                        return ERROR_NOMEM;
+                }
+            }
         }
 
         if (libxl_defbool_val(b_info->u.hvm.nographic) && (!sdl && !vnc)) {
diff --git a/tools/libs/light/libxl_internal.h b/tools/libs/light/libxl_internal.h
index b65e0064b9..06b5a14409 100644
--- a/tools/libs/light/libxl_internal.h
+++ b/tools/libs/light/libxl_internal.h
@@ -1729,6 +1729,8 @@ _hidden int libxl__device_pci_setdefault(libxl__gc *gc, uint32_t domid,
                                          libxl_device_pci *pci, bool hotplug);
 _hidden bool libxl__is_igd_vga_passthru(libxl__gc *gc,
                                         const libxl_domain_config *d_config);
+_hidden libxl__json_object *libxl__device_pci_get_qmp_json(libxl__gc *gc,
+                                                         libxl_device_pci *pci);
 
 /* from libxl_dtdev */
 
diff --git a/tools/libs/light/libxl_pci.c b/tools/libs/light/libxl_pci.c
index 49d272d0de..3ef2c43412 100644
--- a/tools/libs/light/libxl_pci.c
+++ b/tools/libs/light/libxl_pci.c
@@ -1098,16 +1098,10 @@ out:
     pci_add_dm_done(egc, pas, rc); /* must be last */
 }
 
-static void pci_add_qmp_device_add(libxl__egc *egc, pci_add_state *pas)
+libxl__json_object *libxl__device_pci_get_qmp_json(libxl__gc *gc,
+                                                   libxl_device_pci *pci)
 {
-    STATE_AO_GC(pas->aodev->ao);
     libxl__json_object *args = NULL;
-    int rc;
-
-    /* Convenience aliases */
-    libxl_domid domid = pas->domid;
-    libxl_device_pci *pci = &pas->pci;
-    libxl__ev_qmp *const qmp = &pas->qmp;
 
     libxl__qmp_param_add_string(gc, &args, "driver",
                                 "xen-pci-passthrough");
@@ -1134,11 +1128,30 @@ static void pci_add_qmp_device_add(libxl__egc *egc, pci_add_state *pas)
     if (pci->permissive)
         libxl__qmp_param_add_bool(gc, &args, "permissive", true);
 
+    return args;
+}
+
+static void pci_add_qmp_device_add(libxl__egc *egc, pci_add_state *pas)
+{
+    STATE_AO_GC(pas->aodev->ao);
+    libxl__json_object *args = NULL;
+    int rc = 0;
+
+    /* Convenience aliases */
+    libxl_domid domid = pas->domid;
+    libxl_device_pci *pci = &pas->pci;
+    libxl__ev_qmp *const qmp = &pas->qmp;
+
+    args = libxl__device_pci_get_qmp_json(gc, pci);
+
     qmp->ao = pas->aodev->ao;
     qmp->domid = domid;
     qmp->payload_fd = -1;
     qmp->callback = pci_add_qmp_device_add_cb;
-    rc = libxl__ev_qmp_send(egc, qmp, "device_add", args);
+    if (pci->hotplug)
+        rc = libxl__ev_qmp_send(egc, qmp, "device_add", args);
+    else
+        pci_add_qmp_device_add_cb(egc, qmp, NULL, 0);
     if (rc) goto out;
     return;
 
@@ -1509,7 +1522,7 @@ int libxl_device_pci_add(libxl_ctx *ctx, uint32_t domid,
     return AO_INPROGRESS;
 }
 
-static bool libxl_pci_assignable(libxl_ctx *ctx, libxl_device_pci *pci)
+bool libxl_pci_assignable(libxl_ctx *ctx, libxl_device_pci *pci)
 {
     libxl_device_pci *pcis;
     int num;
@@ -1820,6 +1833,14 @@ static void do_pci_remove(libxl__egc *egc, pci_remove_state *prs)
     libxl_domain_type type = libxl__domain_type(gc, domid);
     libxl_device_pci *pci = &prs->pci;
     int rc, num;
+
+    /* Passthrough'd device has been passed to Qemu command line so there is
+     * no need to remove it via QMP */
+    if (!pci->hotplug) {
+        pci_remove_done(egc, prs, 0);
+        return;
+    }
+
     pcis = libxl_device_pci_list(ctx, domid, &num);
     if (!pcis) {
         rc = ERROR_FAIL;
-- 
2.53.0



--
Thierry Escande | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

[RFC PATCH v2 3/3] docs: provide description for pci hotplug option

[Thierry Escande]

This patch adds description for the 'hotplug' PCI option, allowing
devices to be attached to the guest using the legacy QMP hotplug
mechanism or by passing them to the Qemu command line using '-device'.

Signed-off-by: Thierry Escande <thierry.escande@vates.tech>
---
v2: no change
---
 docs/man/xl-pci-configuration.5.pod | 17 +++++++++++++++++
 docs/man/xl.cfg.5.pod.in            |  6 ++++++
 2 files changed, 23 insertions(+)

diff --git a/docs/man/xl-pci-configuration.5.pod b/docs/man/xl-pci-configuration.5.pod
index 0691f06ad3..8b664173cb 100644
--- a/docs/man/xl-pci-configuration.5.pod
+++ b/docs/man/xl-pci-configuration.5.pod
@@ -166,6 +166,23 @@ dom0 without confirmation.  Please use with care.
 
 =back
 
+=item B<hotplug>=I<BOOLEAN>
+
+=over 4
+
+=item Description
+
+Tells L<xl(1)> to use QMP hotplug mechanism to attach assignable device to
+the guest, or to pass it via Qemu command line using
+'B<-device xen-pci-passthrough,...>'. The default is to use the legacy QMP
+mechanism.
+
+=item Default Value
+
+1
+
+=back
+
 =item B<power_mgmt>=I<BOOLEAN>
 
 =over 4
diff --git a/docs/man/xl.cfg.5.pod.in b/docs/man/xl.cfg.5.pod.in
index 3aac0bc4fb..181aaffa0f 100644
--- a/docs/man/xl.cfg.5.pod.in
+++ b/docs/man/xl.cfg.5.pod.in
@@ -1230,6 +1230,12 @@ B<(HVM only)> Changes the default value of B<power_mgmt> for all PCI
 devices passed through to this VM. See B<power_mgmt>
 above.
 
+=item B<pci_hotplug=BOOLEAN>
+
+B<(HVM only)> Changes the default value of B<hotplug> for all PCI
+devices passed through to this VM. See L<xl-pci-configuration(5)> for
+more details.
+
 =item B<gfx_passthru=BOOLEAN|"STRING">
 
 Enable graphics device PCI passthrough. This option makes an assigned
-- 
2.53.0



--
Thierry Escande | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [RFC PATCH v2 0/3] xl: Allow PCI devices to be passthrough'd via Qemu command line

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 1967 bytes --]

On Thu, Apr 09, 2026 at 04:20:57PM +0000, Thierry Escande wrote:
> This series adds support for a new 'hotplug' option for PCI devices to be
> passthrough'd to guests.
> 
> The current mechanism for device passthrough uses QMP device_add command
> to hot-plug PCI devices to the guest Qemu instance. This is an issue for
> guests running on Q35 chipset (preliminary support posted at [1]) since the
> Q35 PCI root bus does not support hotplug. Devices could be hotplugged to
> a secondary PCI bus but Xen only support 1 PCI bus for now.
> 
> The 'hotplug' option allows to control how devices are attached to the
> guest, either by using the legacy QMP mechanism (this is the default) or by
> passing it directly to the Qemu command line using xen-pci-passthrough
> device.
> 
> Example usage in cfg file:
>  pci = [ "00:03.0,seize=1,hotplug=0" ]
> 
> Since Qemu -device option accepts parameters in json format, this
> patchset adds a new internal function libxl__device_pci_get_qmp_json() that
> generates the device json object used for both QMP hotplug and command line
> passing.
> Also, the function libxl_pci_assignable() is made available from libxl.h
> for access in libxl_dm.c. I don't know if it's ok to make it public or if
> it should stay private. Maybe it should be renamed as
> libxl_device_pci_assignable() to match the other PCI device APIs.
> 
> This has been successfully tested on Xen 4.22-dev and Qemu 10.0 running
> Debian VMs in both Bios and UEFI mode with a passthrough'd nvme disk.
> Disabling the hotplug mechanism might be made mandatory for Q35 machines
> later, once Q35 support is merged upstream.
> 
> [1] https://lore.kernel.org/xen-devel/20260313163455.790692-1-thierry.escande@vates.tech/

I don't see relevant device option added to the QEMU cmdline:
https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/13860278916

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [RFC PATCH v2 0/3] xl: Allow PCI devices to be passthrough'd via Qemu command line

[Thierry Escande]

On 4/10/26 02:55, Marek Marczykowski-Górecki wrote:
> On Thu, Apr 09, 2026 at 04:20:57PM +0000, Thierry Escande wrote:
>> This series adds support for a new 'hotplug' option for PCI devices to be
>> passthrough'd to guests.
>>
>> The current mechanism for device passthrough uses QMP device_add command
>> to hot-plug PCI devices to the guest Qemu instance. This is an issue for
>> guests running on Q35 chipset (preliminary support posted at [1]) since the
>> Q35 PCI root bus does not support hotplug. Devices could be hotplugged to
>> a secondary PCI bus but Xen only support 1 PCI bus for now.
>>
>> The 'hotplug' option allows to control how devices are attached to the
>> guest, either by using the legacy QMP mechanism (this is the default) or by
>> passing it directly to the Qemu command line using xen-pci-passthrough
>> device.
>>
>> Example usage in cfg file:
>>  pci = [ "00:03.0,seize=1,hotplug=0" ]
>>
>> Since Qemu -device option accepts parameters in json format, this
>> patchset adds a new internal function libxl__device_pci_get_qmp_json() that
>> generates the device json object used for both QMP hotplug and command line
>> passing.
>> Also, the function libxl_pci_assignable() is made available from libxl.h
>> for access in libxl_dm.c. I don't know if it's ok to make it public or if
>> it should stay private. Maybe it should be renamed as
>> libxl_device_pci_assignable() to match the other PCI device APIs.
>>
>> This has been successfully tested on Xen 4.22-dev and Qemu 10.0 running
>> Debian VMs in both Bios and UEFI mode with a passthrough'd nvme disk.
>> Disabling the hotplug mechanism might be made mandatory for Q35 machines
>> later, once Q35 support is merged upstream.
>>
>> [1] https://lore.kernel.org/xen-devel/20260313163455.790692-1-thierry.escande@vates.tech/
>
> I don't see relevant device option added to the QEMU cmdline:
> https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/13860278916
>
My bad. The code that adds the -device option is not in the correct code
block. (bad copy/paste at some point I guess...). v3 on its way.

Regards,
Thierry


--
Thierry Escande | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech