From mboxrd@z Thu Jan 1 00:00:00 1970 From: Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= Subject: Re: PCI passthrough with stubdomain Date: Fri, 23 Sep 2016 16:25:41 +0200 Message-ID: <20160923142541.GB31510@mail-itl> References: <20160923084814.GS7339@mail-itl> <20160923132707.GH20069@var.bordeaux.inria.fr> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============8828737941281037821==" Return-path: In-Reply-To: <20160923132707.GH20069@var.bordeaux.inria.fr> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: Samuel Thibault , xen-devel , HW42 List-Id: xen-devel@lists.xenproject.org --===============8828737941281037821== Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="el3ucdwC6Myv/ATe" Content-Disposition: inline --el3ucdwC6Myv/ATe Content-Type: multipart/mixed; boundary="7nIo0v9TABfW/D7j" Content-Disposition: inline --7nIo0v9TABfW/D7j Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Sep 23, 2016 at 03:27:07PM +0200, Samuel Thibault wrote: > Marek Marczykowski-G=C3=B3recki, on Fri 23 Sep 2016 10:48:14 +0200, wrote: > > I'm still trying to get PCI passthrough working with stubdomain and > > without qemu in dom0 (even for just vfb/vkbd backends). How is this > > supposed to work? >=20 > Just as I recall from my memory: >=20 > > 1. Should xen-pcifront in the target domain be used (and consequently, > > should xen-pciback be set for it)? >=20 > I guess that could work. Could, or should? ;) In the meantime, I've found this in xen-pcifront driver: static int __init pcifront_init(void) { if (!xen_pv_domain() || xen_initial_domain()) return -ENODEV; So, it looks like pcifront is not used in PV domain. > > Currently xen-pciback is set for both > > stubdomain and target domain, which presents a race condition and > > xen-pciback refuses to setup one of them. >=20 > Yes, that's expected, for the reason you say. In the meantime I've tried to modify libxl to not setup pciback for target domain. This, along with other modifications (see below) improved the situation. > * to summarize > ************** > > If running PV drivers in the guest, you set the pciback on the guest, be > it run with stubdom or not.=20 > If running plain drivers in the guest, > * when not using a stubdom you don't need to set a pciback. > * when using a stubdom you need to set a pciback on the stubdom. Thanks for the explanation! > So the unfortunate thing is that when using stubdom, you'd have to set > the pciback either on the guest (to run a PV driver in it), or on the > stubdom (to run a plain driver in the guest, and let mini-os use PV to > let qemu pass the board through) I've tried only on Linux HVM guest and as noted above xen-pcifront doesn't look to be involved. Is it any different on other OSes? As for getting PCI passthrough working with stubdomain, for now I hit those problems: 1. getdomaininfo not allowed from stubdomain (already fixed in master), 2. double setup of pciback (explained above) - for now I've disabled one of them, 3. race condition between pcifront in mini-os and qemu. Regarding the last one: Toolstack during (stub)domain startup setup two things, mostly asynchronously: 1. pciback/pcifront (through standard xenstore entries) 2. instruct qemu to attach device (by writing "pci-ins" to device-model/xxx/command) The later fails if pcifront is not initialized yet. For now I've moved the second step to be really after the first one, including synchronization through libxl__wait_for_backend call.=20 Is it ok? Or maybe it should be done on stubdomain side (handling "pci-ins" should wait on pcifront)? I guess the same will apply to qemu-xen case, or any other stubdomain, so probably better have it in toolstack, right? Work-in-progress patches attached. The result is that lspci inside the guest finally list the device. No idea if it's really working yet. --=20 Best Regards, Marek Marczykowski-G=C3=B3recki Invisible Things Lab A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? --7nIo0v9TABfW/D7j Content-Type: text/plain; charset=utf-8 Content-Disposition: attachment; filename="0001-libxl-attach-xen-pciback-only-to-stubdomain.patch" Content-Transfer-Encoding: quoted-printable =46rom b8a34904411cc6e7a7abdc6296657ff5f3eb92e9 Mon Sep 17 00:00:00 2001 =46rom: =3D?UTF-8?q?Marek=3D20Marczykowski-G=3DC3=3DB3recki?=3D Date: Thu, 22 Sep 2016 16:07:14 +0200 Subject: [PATCH 1/2] libxl: attach xen-pciback only to stubdomain MIME-Version: 1.0 Content-Type: text/plain; charset=3DUTF-8 Content-Transfer-Encoding: 8bit Organization: Invisible Things Lab Cc: Marek Marczykowski-G=C3=B3recki Signed-off-by: Marek Marczykowski-G=C3=B3recki --- tools/libxl/libxl_create.c | 2 +- tools/libxl/libxl_pci.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c index c6862b8..5625ef2 100644 --- a/tools/libxl/libxl_create.c +++ b/tools/libxl/libxl_create.c @@ -1429,7 +1429,7 @@ static void domcreate_attach_pci(libxl__egc *egc, lib= xl__multidev *multidev, } } =20 - if (d_config->num_pcidevs > 0) { + if (d_config->num_pcidevs > 0 && d_config->c_info.type =3D=3D LIBXL_DO= MAIN_TYPE_PV) { ret =3D libxl__create_pci_backend(gc, domid, d_config->pcidevs, d_config->num_pcidevs); if (ret < 0) { diff --git a/tools/libxl/libxl_pci.c b/tools/libxl/libxl_pci.c index 19c597e..2942772 100644 --- a/tools/libxl/libxl_pci.c +++ b/tools/libxl/libxl_pci.c @@ -1001,7 +1001,7 @@ out: } } =20 - if (!starting) + if (!starting && !hvm) rc =3D libxl__device_pci_add_xenstore(gc, domid, pcidev, starting); else rc =3D 0; --=20 2.5.5 --7nIo0v9TABfW/D7j Content-Type: text/plain; charset=utf-8 Content-Disposition: attachment; filename="0002-libxl-attach-PCI-device-to-qemu-only-after-setting-p.patch" Content-Transfer-Encoding: quoted-printable =46rom 930398583d5a590c511e58057d000e1ed7defb82 Mon Sep 17 00:00:00 2001 =46rom: =3D?UTF-8?q?Marek=3D20Marczykowski-G=3DC3=3DB3recki?=3D Date: Fri, 23 Sep 2016 16:13:37 +0200 Subject: [PATCH 2/2] libxl: attach PCI device to qemu only after setting pciback/pcifront MIME-Version: 1.0 Content-Type: text/plain; charset=3DUTF-8 Content-Transfer-Encoding: 8bit Organization: Invisible Things Lab Cc: Marek Marczykowski-G=C3=B3recki When qemu is running in stubdomain, handling "pci-ins" command will fail if pcifront is not initialized already. Fix this by sending such command only after confirming that pciback/front is running. Signed-off-by: Marek Marczykowski-G=C3=B3recki --- tools/libxl/libxl_pci.c | 50 ++++++++++++++++++++++++++-------------------= ---- 1 file changed, 27 insertions(+), 23 deletions(-) diff --git a/tools/libxl/libxl_pci.c b/tools/libxl/libxl_pci.c index 2942772..88f8f5b 100644 --- a/tools/libxl/libxl_pci.c +++ b/tools/libxl/libxl_pci.c @@ -891,34 +891,15 @@ static int do_pci_add(libxl__gc *gc, uint32_t domid, = libxl_device_pci *pcidev, i libxl_ctx *ctx =3D libxl__gc_owner(gc); libxl_domain_type type =3D libxl__domain_type(gc, domid); char *sysfs_path; + char *be_path; FILE *f; unsigned long long start, end, flags, size; - int irq, i, rc, hvm =3D 0; + int irq, i, rc, hvm =3D (type =3D=3D LIBXL_DOMAIN_TYPE_HVM); uint32_t flag =3D XEN_DOMCTL_DEV_RDM_RELAXED; =20 if (type =3D=3D LIBXL_DOMAIN_TYPE_INVALID) return ERROR_FAIL; =20 - if (type =3D=3D LIBXL_DOMAIN_TYPE_HVM) { - hvm =3D 1; - if (libxl__wait_for_device_model_deprecated(gc, domid, "running", - NULL, NULL, NULL) < 0) { - return ERROR_FAIL; - } - switch (libxl__device_model_version_running(gc, domid)) { - case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN_TRADITIONAL: - rc =3D qemu_pci_add_xenstore(gc, domid, pcidev); - break; - case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN: - rc =3D libxl__qmp_pci_add(gc, domid, pcidev); - break; - default: - return ERROR_INVAL; - } - if ( rc ) - return ERROR_FAIL; - } - sysfs_path =3D libxl__sprintf(gc, SYSFS_PCI_DEV"/"PCI_BDF"/resource", = pcidev->domain, pcidev->bus, pcidev->dev, pcidev->func); f =3D fopen(sysfs_path, "r"); @@ -1001,10 +982,33 @@ out: } } =20 - if (!starting && !hvm) + if (!starting && !hvm) { rc =3D libxl__device_pci_add_xenstore(gc, domid, pcidev, starting); - else + be_path =3D libxl__sprintf(gc, "%s/backend/pci/%d/0", libxl__xs_ge= t_dompath(gc, 0), domid); + if (libxl__wait_for_backend(gc, be_path, GCSPRINTF("%d", XenbusSta= teConnected)) < 0) + return ERROR_FAIL; + } else rc =3D 0; + + if (type =3D=3D LIBXL_DOMAIN_TYPE_HVM) { + if (libxl__wait_for_device_model_deprecated(gc, domid, "running", + NULL, NULL, NULL) < 0) { + return ERROR_FAIL; + } + switch (libxl__device_model_version_running(gc, domid)) { + case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN_TRADITIONAL: + rc =3D qemu_pci_add_xenstore(gc, domid, pcidev); + break; + case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN: + rc =3D libxl__qmp_pci_add(gc, domid, pcidev); + break; + default: + return ERROR_INVAL; + } + if ( rc ) + return ERROR_FAIL; + } + return rc; } =20 --=20 2.5.5 --7nIo0v9TABfW/D7j-- --el3ucdwC6Myv/ATe Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBCAAGBQJX5TtlAAoJENuP0xzK19csNeEH/34bITIG+6mX8RR8l0ihEsDt VQiveeKdze6tY3ggrtQLTYk+A6X0tVjSU7V86nnSN5A4bx+DsNleM/FiUbkENof5 7TUuLeBOgnfrnMvPMZ5NQh1YoKssSOLQCJv/XL9kKVqJoYdAInFduMdCBm1gQTZP sNCD8k1F2jKaJIgKzvP7/1iRESx9SpZDMEtMfVlshAqYZ/ypBFIZGYNewPc8kXdE paVzYiB0SzxxKu8JspOYpfFftJRzXLq4/rwmGa42xjPFJam9TTi9C4hb0F4BEqG+ EUO1EuogJ+0T0ZbDMHX5J/TAXIqzWF9nSXEDgnHoWCgVUFexyTd38UR7PpJcVM4= =82WV -----END PGP SIGNATURE----- --el3ucdwC6Myv/ATe-- --===============8828737941281037821== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwczovL2xpc3RzLnhlbi5v cmcveGVuLWRldmVsCg== --===============8828737941281037821==--