* [PATCH 1 of 4 v3] libxl: Make a helper function write a BDF to a sysfs path
2012-05-15 14:35 [PATCH 0 of 4 v3] Add commands to automatically prep devices for pass-through George Dunlap
@ 2012-05-15 14:35 ` George Dunlap
2012-05-15 14:35 ` [PATCH 2 of 4 v3] libxl: Rename pci_list_assignable to pci_assignable_list George Dunlap
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: George Dunlap @ 2012-05-15 14:35 UTC (permalink / raw)
To: xen-devel; +Cc: george.dunlap
This functionality will be used several times in subsequent patches.
Signed-off-by: George Dunlap <george.dunlap@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
diff -r 804eb2962984 -r a2b528299c12 tools/libxl/libxl_pci.c
--- a/tools/libxl/libxl_pci.c Tue May 15 15:13:45 2012 +0100
+++ b/tools/libxl/libxl_pci.c Tue May 15 15:24:02 2012 +0100
@@ -327,6 +327,36 @@ static int is_pcidev_in_array(libxl_devi
return 0;
}
+/* Write the standard BDF into the sysfs path given by sysfs_path. */
+static int sysfs_write_bdf(libxl__gc *gc, const char * sysfs_path,
+ libxl_device_pci *pcidev)
+{
+ libxl_ctx *ctx = libxl__gc_owner(gc);
+ int rc, fd;
+ char *buf;
+
+ fd = open(sysfs_path, O_WRONLY);
+ if (fd < 0) {
+ LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "Couldn't open %s",
+ sysfs_path);
+ return ERROR_FAIL;
+ }
+
+ buf = libxl__sprintf(gc, PCI_BDF, pcidev->domain, pcidev->bus,
+ pcidev->dev, pcidev->func);
+ rc = write(fd, buf, strlen(buf));
+ /* Annoying to have two if's, but we need the errno */
+ if (rc < 0)
+ LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR,
+ "write to %s returned %d", sysfs_path, rc);
+ close(fd);
+
+ if (rc < 0)
+ return ERROR_FAIL;
+
+ return 0;
+}
+
libxl_device_pci *libxl_device_pci_list_assignable(libxl_ctx *ctx, int *num)
{
GC_INIT(ctx);
@@ -571,27 +601,12 @@ static int do_pci_add(libxl__gc *gc, uin
/* Don't restrict writes to the PCI config space from this VM */
if (pcidev->permissive) {
- int fd;
- char *buf;
-
- sysfs_path = libxl__sprintf(gc, SYSFS_PCIBACK_DRIVER"/permissive");
- fd = open(sysfs_path, O_WRONLY);
- if (fd < 0) {
- LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "Couldn't open %s",
- sysfs_path);
+ if ( sysfs_write_bdf(gc, SYSFS_PCIBACK_DRIVER"/permissive",
+ pcidev) < 0 ) {
+ LIBXL__LOG(ctx, LIBXL__LOG_ERROR,
+ "Setting permissive for device");
return ERROR_FAIL;
}
-
- buf = libxl__sprintf(gc, PCI_BDF, pcidev->domain, pcidev->bus,
- pcidev->dev, pcidev->func);
- rc = write(fd, buf, strlen(buf));
- /* Annoying to have two if's, but we need the errno */
- if (rc < 0)
- LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR,
- "write to %s returned %d", sysfs_path, rc);
- close(fd);
- if (rc < 0)
- return ERROR_FAIL;
}
break;
}
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH 2 of 4 v3] libxl: Rename pci_list_assignable to pci_assignable_list
2012-05-15 14:35 [PATCH 0 of 4 v3] Add commands to automatically prep devices for pass-through George Dunlap
2012-05-15 14:35 ` [PATCH 1 of 4 v3] libxl: Make a helper function write a BDF to a sysfs path George Dunlap
@ 2012-05-15 14:35 ` George Dunlap
2012-05-15 14:35 ` [PATCH 3 of 4 v3] libxl: Introduce pci_assignable_add and pci_assignable_remove George Dunlap
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: George Dunlap @ 2012-05-15 14:35 UTC (permalink / raw)
To: xen-devel; +Cc: george.dunlap
...to prepare for a consistent "pci_assignable_*" naming scheme.
Also move the man page entry into the PCI PASS-THROUGH section, rather
than the XEN HOST section.
No functional changes.
v3:
- Port to xen-unstable tip
Signed-off-by: George Dunlap <george.dunlap@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
diff -r a2b528299c12 -r 62e682bdb103 docs/man/xl.pod.1
--- a/docs/man/xl.pod.1 Tue May 15 15:24:02 2012 +0100
+++ b/docs/man/xl.pod.1 Tue May 15 15:26:24 2012 +0100
@@ -693,13 +693,6 @@ explanatory.
Prints the current uptime of the domains running.
-=item B<pci-list-assignable-devices>
-
-List all the assignable PCI devices.
-These are devices in the system which are configured to be
-available for passthrough and are bound to a suitable PCI
-backend driver in domain 0 rather than a real driver.
-
=back
=head1 SCHEDULER SUBCOMMANDS
@@ -1032,6 +1025,13 @@ List virtual network interfaces for a do
=over 4
+=item B<pci-assignable-list>
+
+List all the assignable PCI devices.
+These are devices in the system which are configured to be
+available for passthrough and are bound to a suitable PCI
+backend driver in domain 0 rather than a real driver.
+
=item B<pci-attach> I<domain-id> I<BDF>
Hot-plug a new pass-through pci device to the specified domain.
diff -r a2b528299c12 -r 62e682bdb103 tools/libxl/libxl.h
--- a/tools/libxl/libxl.h Tue May 15 15:24:02 2012 +0100
+++ b/tools/libxl/libxl.h Tue May 15 15:26:24 2012 +0100
@@ -722,7 +722,7 @@ libxl_device_pci *libxl_device_pci_list(
* could be assigned to a domain (i.e. are bound to the backend
* driver) but are not currently.
*/
-libxl_device_pci *libxl_device_pci_list_assignable(libxl_ctx *ctx, int *num);
+libxl_device_pci *libxl_device_pci_assignable_list(libxl_ctx *ctx, int *num);
/* CPUID handling */
int libxl_cpuid_parse_config(libxl_cpuid_policy_list *cpuid, const char* str);
diff -r a2b528299c12 -r 62e682bdb103 tools/libxl/libxl_pci.c
--- a/tools/libxl/libxl_pci.c Tue May 15 15:24:02 2012 +0100
+++ b/tools/libxl/libxl_pci.c Tue May 15 15:26:24 2012 +0100
@@ -357,7 +357,7 @@ static int sysfs_write_bdf(libxl__gc *gc
return 0;
}
-libxl_device_pci *libxl_device_pci_list_assignable(libxl_ctx *ctx, int *num)
+libxl_device_pci *libxl_device_pci_assignable_list(libxl_ctx *ctx, int *num)
{
GC_INIT(ctx);
libxl_device_pci *pcidevs = NULL, *new, *assigned;
@@ -684,7 +684,7 @@ static int libxl_pcidev_assignable(libxl
libxl_device_pci *pcidevs;
int num, i;
- pcidevs = libxl_device_pci_list_assignable(ctx, &num);
+ pcidevs = libxl_device_pci_assignable_list(ctx, &num);
for (i = 0; i < num; i++) {
if (pcidevs[i].domain == pcidev->domain &&
pcidevs[i].bus == pcidev->bus &&
diff -r a2b528299c12 -r 62e682bdb103 tools/libxl/xl.h
--- a/tools/libxl/xl.h Tue May 15 15:24:02 2012 +0100
+++ b/tools/libxl/xl.h Tue May 15 15:26:24 2012 +0100
@@ -35,9 +35,9 @@ int main_cd_insert(int argc, char **argv
int main_console(int argc, char **argv);
int main_vncviewer(int argc, char **argv);
int main_pcilist(int argc, char **argv);
-int main_pcilist_assignable(int argc, char **argv);
int main_pcidetach(int argc, char **argv);
int main_pciattach(int argc, char **argv);
+int main_pciassignable_list(int argc, char **argv);
int main_restore(int argc, char **argv);
int main_migrate_receive(int argc, char **argv);
int main_save(int argc, char **argv);
diff -r a2b528299c12 -r 62e682bdb103 tools/libxl/xl_cmdimpl.c
--- a/tools/libxl/xl_cmdimpl.c Tue May 15 15:24:02 2012 +0100
+++ b/tools/libxl/xl_cmdimpl.c Tue May 15 15:26:24 2012 +0100
@@ -2222,34 +2222,6 @@ int main_vncviewer(int argc, char **argv
return 0;
}
-static void pcilist_assignable(void)
-{
- libxl_device_pci *pcidevs;
- int num, i;
-
- pcidevs = libxl_device_pci_list_assignable(ctx, &num);
-
- if ( pcidevs == NULL )
- return;
- for (i = 0; i < num; i++) {
- printf("%04x:%02x:%02x.%01x\n",
- pcidevs[i].domain, pcidevs[i].bus, pcidevs[i].dev, pcidevs[i].func);
- libxl_device_pci_dispose(&pcidevs[i]);
- }
- free(pcidevs);
-}
-
-int main_pcilist_assignable(int argc, char **argv)
-{
- int opt;
-
- if ((opt = def_getopt(argc, argv, "", "pci-list-assignable-devices", 0)) != -1)
- return opt;
-
- pcilist_assignable();
- return 0;
-}
-
static void pcilist(const char *dom)
{
libxl_device_pci *pcidevs;
@@ -2371,6 +2343,34 @@ int main_pciattach(int argc, char **argv
return 0;
}
+static void pciassignable_list(void)
+{
+ libxl_device_pci *pcidevs;
+ int num, i;
+
+ pcidevs = libxl_device_pci_assignable_list(ctx, &num);
+
+ if ( pcidevs == NULL )
+ return;
+ for (i = 0; i < num; i++) {
+ printf("%04x:%02x:%02x.%01x\n",
+ pcidevs[i].domain, pcidevs[i].bus, pcidevs[i].dev, pcidevs[i].func);
+ libxl_device_pci_dispose(&pcidevs[i]);
+ }
+ free(pcidevs);
+}
+
+int main_pciassignable_list(int argc, char **argv)
+{
+ int opt;
+
+ if ((opt = def_getopt(argc, argv, "", "pci-assignable-list", 0)) != -1)
+ return opt;
+
+ pciassignable_list();
+ return 0;
+}
+
static void pause_domain(const char *p)
{
find_domain(p);
diff -r a2b528299c12 -r 62e682bdb103 tools/libxl/xl_cmdtable.c
--- a/tools/libxl/xl_cmdtable.c Tue May 15 15:24:02 2012 +0100
+++ b/tools/libxl/xl_cmdtable.c Tue May 15 15:26:24 2012 +0100
@@ -86,8 +86,8 @@ struct cmd_spec cmd_table[] = {
"List pass-through pci devices for a domain",
"<Domain>",
},
- { "pci-list-assignable-devices",
- &main_pcilist_assignable, 0, 0,
+ { "pci-assignable-list",
+ &main_pciassignable_list, 0, 0,
"List all the assignable pci devices",
"",
},
diff -r a2b528299c12 -r 62e682bdb103 tools/python/xen/lowlevel/xl/xl.c
--- a/tools/python/xen/lowlevel/xl/xl.c Tue May 15 15:24:02 2012 +0100
+++ b/tools/python/xen/lowlevel/xl/xl.c Tue May 15 15:26:24 2012 +0100
@@ -566,13 +566,13 @@ static PyObject *pyxl_pci_parse(XlObject
return (PyObject *)pci;
}
-static PyObject *pyxl_pci_list_assignable(XlObject *self, PyObject *args)
+static PyObject *pyxl_pci_assignable_list(XlObject *self, PyObject *args)
{
libxl_device_pci *dev;
PyObject *list;
int nr_dev, i;
- dev = libxl_device_pci_list_assignable(self->ctx, &nr_dev);
+ dev = libxl_device_pci_assignable_list(self->ctx, &nr_dev);
if ( dev == NULL ) {
PyErr_SetString(xl_error_obj, "Cannot list assignable devices");
return NULL;
@@ -662,8 +662,8 @@ static PyMethodDef pyxl_methods[] = {
"Parse pass-through PCI device spec (BDF)"},
{"device_pci_list", (PyCFunction)pyxl_pci_list, METH_VARARGS,
"List PCI devices assigned to a domain"},
- {"device_pci_list_assignable",
- (PyCFunction)pyxl_pci_list_assignable, METH_NOARGS,
+ {"device_pci_assignable_list",
+ (PyCFunction)pyxl_pci_assignable_list, METH_NOARGS,
"List assignable PCI devices"},
{ NULL, NULL, 0, NULL }
};
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH 3 of 4 v3] libxl: Introduce pci_assignable_add and pci_assignable_remove
2012-05-15 14:35 [PATCH 0 of 4 v3] Add commands to automatically prep devices for pass-through George Dunlap
2012-05-15 14:35 ` [PATCH 1 of 4 v3] libxl: Make a helper function write a BDF to a sysfs path George Dunlap
2012-05-15 14:35 ` [PATCH 2 of 4 v3] libxl: Rename pci_list_assignable to pci_assignable_list George Dunlap
@ 2012-05-15 14:35 ` George Dunlap
2012-05-15 14:35 ` [PATCH 4 of 4 v3] xl: Add pci_assignable_add and remove commands George Dunlap
2012-05-15 15:29 ` [PATCH 0 of 4 v3] Add commands to automatically prep devices for pass-through Ian Campbell
4 siblings, 0 replies; 6+ messages in thread
From: George Dunlap @ 2012-05-15 14:35 UTC (permalink / raw)
To: xen-devel; +Cc: george.dunlap
Introduce libxl helper functions to prepare devices to be passed
through to guests. This is meant to replace of all the manual sysfs
commands which are currently required.
pci_assignable_add accepts a BDF for a device and will:
* Unbind a device from its current driver, if any
* If "rebind" is set, it will store the path of the driver from which we
unplugged it in /libxl/pciback/$BDF/driver_path
* If create a slot for it in pciback if one doesn't yet exist
* Bind the device to pciback
At this point it will show up in pci_assignable_list, and is ready to
be passed through to a guest.
pci_assignable_remove accepts a BDF for a device and will:
* Unbind the device from pciback
* Remove the slot from pciback
* If "rebind" is set, and /libx/pciback/$BDF/driver_path exists, it
will attempt to rebind the device to its original driver.
Both functions are idempotent: if the desired end state has already
been reached, they return SUCCESS.
NB that "$BDF" in this case uses '-' instead of ':' and '.', because
':' and '.' are illegal characters in xenstore paths.
v2:
- sysfs_dev_unbind uses a local var for the path pointer, and sets
only at the end
- Actually read pci domain when looking at slots, instead of assuming
0000
- Call LOG_ERRNO after failed sysfs_write_bdf calls, rather than just
LOG
- Removed stray FIXME
- Made xenstore reads and writes single operations, and removed
transaction infrastructure
- Wrapped a couple of lines that were above 80 characters
- Added a comment explaining the patch's treatment of slots
- Clarified patch description of when it creates slots
v3:
- Fix bug in pciback_dev_has_slot() introduced in v2.
- Raise loglevel to WARNING when pci device is already assigned
- Update comment in libxl.h to make add and remove explicitly
idempotent
Signed-off-by: George Dunlap <george.dunlap@eu.citrix.com>
diff -r 62e682bdb103 -r d6739b5dfd9a tools/libxl/libxl.h
--- a/tools/libxl/libxl.h Tue May 15 15:26:24 2012 +0100
+++ b/tools/libxl/libxl.h Tue May 15 15:26:25 2012 +0100
@@ -718,10 +718,29 @@ int libxl_device_pci_destroy(libxl_ctx *
libxl_device_pci *libxl_device_pci_list(libxl_ctx *ctx, uint32_t domid, int *num);
/*
- * Similar to libxl_device_pci_list but returns all devices which
- * could be assigned to a domain (i.e. are bound to the backend
- * driver) but are not currently.
+ * Functions related to making devices assignable -- that is, bound to
+ * the pciback driver, ready to be given to a guest via
+ * libxl_pci_device_add.
+ *
+ * - ..._add() will unbind the device from its current driver (if
+ * already bound) and re-bind it to pciback; at that point it will be
+ * ready to be assigned to a VM. If rebind is set, it will store the
+ * path to the old driver in xenstore so that it can be handed back to
+ * dom0 on restore.
+ *
+ * - ..._remove() will unbind the device from pciback, and if
+ * rebind is non-zero, attempt to assign it back to the driver
+ * from whence it came.
+ *
+ * - ..._list() will return a list of the PCI devices available to be
+ * assigned.
+ *
+ * add and remove are idempotent: if the device in question is already
+ * added or is not bound, the functions will emit a warning but return
+ * SUCCESS.
*/
+int libxl_device_pci_assignable_add(libxl_ctx *ctx, libxl_device_pci *pcidev, int rebind);
+int libxl_device_pci_assignable_remove(libxl_ctx *ctx, libxl_device_pci *pcidev, int rebind);
libxl_device_pci *libxl_device_pci_assignable_list(libxl_ctx *ctx, int *num);
/* CPUID handling */
diff -r 62e682bdb103 -r d6739b5dfd9a tools/libxl/libxl_pci.c
--- a/tools/libxl/libxl_pci.c Tue May 15 15:26:24 2012 +0100
+++ b/tools/libxl/libxl_pci.c Tue May 15 15:26:25 2012 +0100
@@ -21,6 +21,7 @@
#define PCI_BDF "%04x:%02x:%02x.%01x"
#define PCI_BDF_SHORT "%02x:%02x.%01x"
#define PCI_BDF_VDEVFN "%04x:%02x:%02x.%01x@%02x"
+#define PCI_BDF_XSPATH "%04x-%02x-%02x-%01x"
static unsigned int pcidev_encode_bdf(libxl_device_pci *pcidev)
{
@@ -408,6 +409,334 @@ out:
return pcidevs;
}
+/* Unbind device from its current driver, if any. If driver_path is non-NULL,
+ * store the path to the original driver in it. */
+static int sysfs_dev_unbind(libxl__gc *gc, libxl_device_pci *pcidev,
+ char **driver_path)
+{
+ libxl_ctx *ctx = libxl__gc_owner(gc);
+ char * spath, *dp = NULL;
+ struct stat st;
+
+ spath = libxl__sprintf(gc, SYSFS_PCI_DEV"/"PCI_BDF"/driver",
+ pcidev->domain,
+ pcidev->bus,
+ pcidev->dev,
+ pcidev->func);
+ if ( !lstat(spath, &st) ) {
+ /* Find the canonical path to the driver. */
+ dp = libxl__zalloc(gc, PATH_MAX);
+ dp = realpath(spath, dp);
+ if ( !dp ) {
+ LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "realpath() failed");
+ return -1;
+ }
+
+ LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "Driver re-plug path: %s",
+ dp);
+
+ /* Unbind from the old driver */
+ spath = libxl__sprintf(gc, "%s/unbind", dp);
+ if ( sysfs_write_bdf(gc, spath, pcidev) < 0 ) {
+ LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "Couldn't unbind device");
+ return -1;
+ }
+ }
+
+ if ( driver_path )
+ *driver_path = dp;
+
+ return 0;
+}
+
+/*
+ * A brief comment about slots. I don't know what slots are for; however,
+ * I have by experimentation determined:
+ * - Before a device can be bound to pciback, its BDF must first be listed
+ * in pciback/slots
+ * - The way to get the BDF listed there is to write BDF to
+ * pciback/new_slot
+ * - Writing the same BDF to pciback/new_slot is not idempotent; it results
+ * in two entries of the BDF in pciback/slots
+ * It's not clear whether having two entries in pciback/slots is a problem
+ * or not. Just to be safe, this code does the conservative thing, and
+ * first checks to see if there is a slot, adding one only if one does not
+ * already exist.
+ */
+
+/* Scan through /sys/.../pciback/slots looking for pcidev's BDF */
+static int pciback_dev_has_slot(libxl__gc *gc, libxl_device_pci *pcidev)
+{
+ libxl_ctx *ctx = libxl__gc_owner(gc);
+ FILE *f;
+ int rc = 0;
+ unsigned dom, bus, dev, func;
+
+ f = fopen(SYSFS_PCIBACK_DRIVER"/slots", "r");
+
+ if (f == NULL) {
+ LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "Couldn't open %s",
+ SYSFS_PCIBACK_DRIVER"/slots");
+ return ERROR_FAIL;
+ }
+
+ while(fscanf(f, "%x:%x:%x.%x\n", &dom, &bus, &dev, &func)==4) {
+ if(dom == pcidev->domain
+ && bus == pcidev->bus
+ && dev == pcidev->dev
+ && func == pcidev->func) {
+ rc = 1;
+ goto out;
+ }
+ }
+out:
+ fclose(f);
+ return rc;
+}
+
+static int pciback_dev_is_assigned(libxl__gc *gc, libxl_device_pci *pcidev)
+{
+ libxl_ctx *ctx = libxl__gc_owner(gc);
+ char * spath;
+ int rc;
+ struct stat st;
+
+ spath = libxl__sprintf(gc, SYSFS_PCIBACK_DRIVER"/"PCI_BDF,
+ pcidev->domain, pcidev->bus,
+ pcidev->dev, pcidev->func);
+ rc = lstat(spath, &st);
+
+ if( rc == 0 )
+ return 1;
+ if ( rc < 0 && errno == ENOENT )
+ return 0;
+ LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "Accessing %s", spath);
+ return -1;
+}
+
+static int pciback_dev_assign(libxl__gc *gc, libxl_device_pci *pcidev)
+{
+ libxl_ctx *ctx = libxl__gc_owner(gc);
+ int rc;
+
+ if ( (rc=pciback_dev_has_slot(gc, pcidev)) < 0 ) {
+ LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR,
+ "Error checking for pciback slot");
+ return ERROR_FAIL;
+ } else if (rc == 0) {
+ if ( sysfs_write_bdf(gc, SYSFS_PCIBACK_DRIVER"/new_slot",
+ pcidev) < 0 ) {
+ LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR,
+ "Couldn't bind device to pciback!");
+ return ERROR_FAIL;
+ }
+ }
+
+ if ( sysfs_write_bdf(gc, SYSFS_PCIBACK_DRIVER"/bind", pcidev) < 0 ) {
+ LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR,
+ "Couldn't bind device to pciback!");
+ return ERROR_FAIL;
+ }
+ return 0;
+}
+
+static int pciback_dev_unassign(libxl__gc *gc, libxl_device_pci *pcidev)
+{
+ libxl_ctx *ctx = libxl__gc_owner(gc);
+
+ /* Remove from pciback */
+ if ( sysfs_dev_unbind(gc, pcidev, NULL) < 0 ) {
+ LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "Couldn't unbind device!");
+ return ERROR_FAIL;
+ }
+
+ /* Remove slot if necessary */
+ if ( pciback_dev_has_slot(gc, pcidev) > 0 ) {
+ if ( sysfs_write_bdf(gc, SYSFS_PCIBACK_DRIVER"/remove_slot",
+ pcidev) < 0 ) {
+ LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR,
+ "Couldn't remove pciback slot");
+ return ERROR_FAIL;
+ }
+ }
+ return 0;
+}
+
+#define PCIBACK_INFO_PATH "/libxl/pciback"
+
+static void pci_assignable_driver_path_write(libxl__gc *gc,
+ libxl_device_pci *pcidev,
+ char *driver_path)
+{
+ libxl_ctx *ctx = libxl__gc_owner(gc);
+ char *path;
+
+ path = libxl__sprintf(gc, PCIBACK_INFO_PATH"/"PCI_BDF_XSPATH"/driver_path",
+ pcidev->domain,
+ pcidev->bus,
+ pcidev->dev,
+ pcidev->func);
+ if ( libxl__xs_write(gc, XBT_NULL, path, "%s", driver_path) < 0 ) {
+ LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_WARNING,
+ "Write of %s to node %s failed.",
+ driver_path, path);
+ }
+}
+
+static char * pci_assignable_driver_path_read(libxl__gc *gc,
+ libxl_device_pci *pcidev)
+{
+ return libxl__xs_read(gc, XBT_NULL,
+ libxl__sprintf(gc,
+ PCIBACK_INFO_PATH "/" PCI_BDF_XSPATH "/driver_path",
+ pcidev->domain,
+ pcidev->bus,
+ pcidev->dev,
+ pcidev->func));
+}
+
+static void pci_assignable_driver_path_remove(libxl__gc *gc,
+ libxl_device_pci *pcidev)
+{
+ libxl_ctx *ctx = libxl__gc_owner(gc);
+
+ /* Remove the xenstore entry */
+ xs_rm(ctx->xsh, XBT_NULL,
+ libxl__sprintf(gc, PCIBACK_INFO_PATH "/" PCI_BDF_XSPATH,
+ pcidev->domain,
+ pcidev->bus,
+ pcidev->dev,
+ pcidev->func) );
+}
+
+static int libxl__device_pci_assignable_add(libxl__gc *gc,
+ libxl_device_pci *pcidev,
+ int rebind)
+{
+ libxl_ctx *ctx = libxl__gc_owner(gc);
+ unsigned dom, bus, dev, func;
+ char *spath, *driver_path = NULL;
+ struct stat st;
+
+ /* Local copy for convenience */
+ dom = pcidev->domain;
+ bus = pcidev->bus;
+ dev = pcidev->dev;
+ func = pcidev->func;
+
+ /* See if the device exists */
+ spath = libxl__sprintf(gc, SYSFS_PCI_DEV"/"PCI_BDF, dom, bus, dev, func);
+ if ( lstat(spath, &st) ) {
+ LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "Couldn't lstat %s", spath);
+ return ERROR_FAIL;
+ }
+
+ /* Check to see if it's already assigned to pciback */
+ if ( pciback_dev_is_assigned(gc, pcidev) ) {
+ LIBXL__LOG(ctx, LIBXL__LOG_WARNING, PCI_BDF" already assigned to pciback",
+ dom, bus, dev, func);
+ return 0;
+ }
+
+ /* Check to see if there's already a driver that we need to unbind from */
+ if ( sysfs_dev_unbind(gc, pcidev, &driver_path ) ) {
+ LIBXL__LOG(ctx, LIBXL__LOG_ERROR,
+ "Couldn't unbind "PCI_BDF" from driver",
+ dom, bus, dev, func);
+ return ERROR_FAIL;
+ }
+
+ /* Store driver_path for rebinding to dom0 */
+ if ( rebind ) {
+ if ( driver_path ) {
+ pci_assignable_driver_path_write(gc, pcidev, driver_path);
+ } else {
+ LIBXL__LOG(ctx, LIBXL__LOG_WARNING,
+ PCI_BDF" not bound to a driver, will not be rebound.",
+ dom, bus, dev, func);
+ }
+ }
+
+ if ( pciback_dev_assign(gc, pcidev) ) {
+ LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "Couldn't bind device to pciback!");
+ return ERROR_FAIL;
+ }
+
+ return 0;
+}
+
+static int libxl__device_pci_assignable_remove(libxl__gc *gc,
+ libxl_device_pci *pcidev,
+ int rebind)
+{
+ libxl_ctx *ctx = libxl__gc_owner(gc);
+ int rc;
+ char *driver_path;
+
+ /* Unbind from pciback */
+ if ( (rc=pciback_dev_is_assigned(gc, pcidev)) < 0 ) {
+ LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "Checking if pciback was assigned");
+ return ERROR_FAIL;
+ } else if ( rc ) {
+ pciback_dev_unassign(gc, pcidev);
+ } else {
+ LIBXL__LOG(ctx, LIBXL__LOG_WARNING,
+ "Not bound to pciback");
+ }
+
+ /* Rebind if necessary */
+ driver_path = pci_assignable_driver_path_read(gc, pcidev);
+
+ if ( driver_path ) {
+ if ( rebind ) {
+ LIBXL__LOG(ctx, LIBXL__LOG_INFO, "Rebinding to driver at %s",
+ driver_path);
+
+ if ( sysfs_write_bdf(gc,
+ libxl__sprintf(gc, "%s/bind", driver_path),
+ pcidev) < 0 ) {
+ LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR,
+ "Couldn't bind device to %s", driver_path);
+ return -1;
+ }
+ }
+
+ pci_assignable_driver_path_remove(gc, pcidev);
+ } else {
+ if ( rebind ) {
+ LIBXL__LOG(ctx, LIBXL__LOG_WARNING,
+ "Couldn't find path for original driver; not rebinding");
+ }
+ }
+
+ return 0;
+}
+
+int libxl_device_pci_assignable_add(libxl_ctx *ctx, libxl_device_pci *pcidev,
+ int rebind)
+{
+ GC_INIT(ctx);
+ int rc;
+
+ rc = libxl__device_pci_assignable_add(gc, pcidev, rebind);
+
+ GC_FREE;
+ return rc;
+}
+
+
+int libxl_device_pci_assignable_remove(libxl_ctx *ctx, libxl_device_pci *pcidev,
+ int rebind)
+{
+ GC_INIT(ctx);
+ int rc;
+
+ rc = libxl__device_pci_assignable_remove(gc, pcidev, rebind);
+
+ GC_FREE;
+ return rc;
+}
+
/*
* This function checks that all functions of a device are bound to pciback
* driver. It also initialises a bit-mask of which function numbers are present
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH 4 of 4 v3] xl: Add pci_assignable_add and remove commands
2012-05-15 14:35 [PATCH 0 of 4 v3] Add commands to automatically prep devices for pass-through George Dunlap
` (2 preceding siblings ...)
2012-05-15 14:35 ` [PATCH 3 of 4 v3] libxl: Introduce pci_assignable_add and pci_assignable_remove George Dunlap
@ 2012-05-15 14:35 ` George Dunlap
2012-05-15 15:29 ` [PATCH 0 of 4 v3] Add commands to automatically prep devices for pass-through Ian Campbell
4 siblings, 0 replies; 6+ messages in thread
From: George Dunlap @ 2012-05-15 14:35 UTC (permalink / raw)
To: xen-devel; +Cc: george.dunlap
pci-assignable-add will always store the driver rebind path, but
pci-assignable-remove will only actually rebind if asked to do so.
v2:
- Use libxl_device_pci_init() instead of memset()
- Call xlu_cfg_destroy() properly
v3:
- Fix typo in failure message
- Specify idempotence in manpage
- Mark add/remove as commands which modify state
Signed-off-by: George Dunlap <george.dunlap@eu.citrix.com>
diff -r d6739b5dfd9a -r 34a8cded4b39 docs/man/xl.pod.1
--- a/docs/man/xl.pod.1 Tue May 15 15:26:25 2012 +0100
+++ b/docs/man/xl.pod.1 Tue May 15 15:30:32 2012 +0100
@@ -1032,6 +1032,28 @@ These are devices in the system which ar
available for passthrough and are bound to a suitable PCI
backend driver in domain 0 rather than a real driver.
+=item B<pci-assignable-add> I<BDF>
+
+Make the device at PCI Bus/Device/Function BDF assignable to guests.
+This will bind the device to the pciback driver. If it is already
+bound to a driver, it will first be unbound, and the original driver
+stored so that it can be re-bound to the same driver later if desired.
+If the device is already bound, it will return success.
+
+CAUTION: This will make the device unusable by Domain 0 until it is
+returned with pci-assignable-remove. Care should therefore be taken
+not to do this on a device critical to domain 0's operation, such as
+storage controllers, network interfaces, or GPUs that are currently
+being used.
+
+=item B<pci-assignable-remove> [I<-r>] I<BDF>
+
+Make the device at PCI Bus/Device/Function BDF assignable to guests. This
+will at least unbind the device from pciback. If the -r option is specified,
+it will also attempt to re-bind the device to its original driver, making it
+usable by Domain 0 again. If the device is not bound to pciback, it will
+return success.
+
=item B<pci-attach> I<domain-id> I<BDF>
Hot-plug a new pass-through pci device to the specified domain.
diff -r d6739b5dfd9a -r 34a8cded4b39 tools/libxl/xl.h
--- a/tools/libxl/xl.h Tue May 15 15:26:25 2012 +0100
+++ b/tools/libxl/xl.h Tue May 15 15:30:32 2012 +0100
@@ -37,6 +37,8 @@ int main_vncviewer(int argc, char **argv
int main_pcilist(int argc, char **argv);
int main_pcidetach(int argc, char **argv);
int main_pciattach(int argc, char **argv);
+int main_pciassignable_add(int argc, char **argv);
+int main_pciassignable_remove(int argc, char **argv);
int main_pciassignable_list(int argc, char **argv);
int main_restore(int argc, char **argv);
int main_migrate_receive(int argc, char **argv);
diff -r d6739b5dfd9a -r 34a8cded4b39 tools/libxl/xl_cmdimpl.c
--- a/tools/libxl/xl_cmdimpl.c Tue May 15 15:26:25 2012 +0100
+++ b/tools/libxl/xl_cmdimpl.c Tue May 15 15:30:32 2012 +0100
@@ -2371,6 +2371,86 @@ int main_pciassignable_list(int argc, ch
return 0;
}
+static void pciassignable_add(const char *bdf, int rebind)
+{
+ libxl_device_pci pcidev;
+ XLU_Config *config;
+
+ libxl_device_pci_init(&pcidev);
+
+ config = xlu_cfg_init(stderr, "command line");
+ if (!config) { perror("xlu_cfg_init"); exit(-1); }
+
+ if (xlu_pci_parse_bdf(config, &pcidev, bdf)) {
+ fprintf(stderr, "pci-assignable-add: malformed BDF specification \"%s\"\n", bdf);
+ exit(2);
+ }
+ libxl_device_pci_assignable_add(ctx, &pcidev, rebind);
+
+ libxl_device_pci_dispose(&pcidev);
+ xlu_cfg_destroy(config);
+}
+
+int main_pciassignable_add(int argc, char **argv)
+{
+ int opt;
+ const char *bdf = NULL;
+
+ while ((opt = def_getopt(argc, argv, "", "pci-assignable-add", 1)) != -1) {
+ switch (opt) {
+ case 0: case 2:
+ return opt;
+ }
+ }
+
+ bdf = argv[optind];
+
+ pciassignable_add(bdf, 1);
+ return 0;
+}
+
+static void pciassignable_remove(const char *bdf, int rebind)
+{
+ libxl_device_pci pcidev;
+ XLU_Config *config;
+
+ libxl_device_pci_init(&pcidev);
+
+ config = xlu_cfg_init(stderr, "command line");
+ if (!config) { perror("xlu_cfg_init"); exit(-1); }
+
+ if (xlu_pci_parse_bdf(config, &pcidev, bdf)) {
+ fprintf(stderr, "pci-assignable-remove: malformed BDF specification \"%s\"\n", bdf);
+ exit(2);
+ }
+ libxl_device_pci_assignable_remove(ctx, &pcidev, rebind);
+
+ libxl_device_pci_dispose(&pcidev);
+ xlu_cfg_destroy(config);
+}
+
+int main_pciassignable_remove(int argc, char **argv)
+{
+ int opt;
+ const char *bdf = NULL;
+ int rebind = 0;
+
+ while ((opt = def_getopt(argc, argv, "r", "pci-assignable-remove", 1)) != -1) {
+ switch (opt) {
+ case 0: case 2:
+ return opt;
+ case 'r':
+ rebind=1;
+ break;
+ }
+ }
+
+ bdf = argv[optind];
+
+ pciassignable_remove(bdf, rebind);
+ return 0;
+}
+
static void pause_domain(const char *p)
{
find_domain(p);
diff -r d6739b5dfd9a -r 34a8cded4b39 tools/libxl/xl_cmdtable.c
--- a/tools/libxl/xl_cmdtable.c Tue May 15 15:26:25 2012 +0100
+++ b/tools/libxl/xl_cmdtable.c Tue May 15 15:30:32 2012 +0100
@@ -86,6 +86,20 @@ struct cmd_spec cmd_table[] = {
"List pass-through pci devices for a domain",
"<Domain>",
},
+ { "pci-assignable-add",
+ &main_pciassignable_add, 0, 1,
+ "Make a device assignable for pci-passthru",
+ "<BDF>",
+ "-h Print this help.\n"
+ },
+ { "pci-assignable-remove",
+ &main_pciassignable_remove, 0, 1,
+ "Remove a device from being assignable",
+ "[options] <BDF>",
+ "-h Print this help.\n"
+ "-r Attempt to re-assign the device to the\n"
+ " original driver"
+ },
{ "pci-assignable-list",
&main_pciassignable_list, 0, 0,
"List all the assignable pci devices",
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH 0 of 4 v3] Add commands to automatically prep devices for pass-through
2012-05-15 14:35 [PATCH 0 of 4 v3] Add commands to automatically prep devices for pass-through George Dunlap
` (3 preceding siblings ...)
2012-05-15 14:35 ` [PATCH 4 of 4 v3] xl: Add pci_assignable_add and remove commands George Dunlap
@ 2012-05-15 15:29 ` Ian Campbell
4 siblings, 0 replies; 6+ messages in thread
From: Ian Campbell @ 2012-05-15 15:29 UTC (permalink / raw)
To: George Dunlap; +Cc: xen-devel@lists.xensource.com
On Tue, 2012-05-15 at 15:35 +0100, George Dunlap wrote:
> Add commands to automatically prep devices for pass-through
>
> The current method for passing through devices requires users to
> either modify cryptic Linux boot parameters and reboot, or do a lot of
> manual reads and writes into sysfs nodes.
>
> This set of patches introduces commands to make this easier. It expands
> on the concept of "assignable" (from the list_assignable_devices command).
>
> The new xl commands are:
>
> pci_assignable_add: Make a device assignable to guests. This involves
> unbinding the device from its old driver, creating a slot for it in
> pciback (if necessary), and binding it to pciback.
>
> pci_assignable_list: List devices assignable to guests. Just renamed
> from pci_list_assignable.
>
> pci_assignable_remove: Make the device no longer assignable to guests.
> This involves unbinding the device from pciback and removing the slot. It
> optionally involves rebinding the device to the driver from which we stole
> it.
>
> Signed-off-by: George Dunlap <george.dunlap@eu.citrix.com>
All 4 patches:
Committed-by: Ian Campbell <ian.campbell@citrix.com>
Please do watch out for adding trailing whitespace, I cleaned this up in
the series and a couple of your other patches which I applied today.
Ian.
^ permalink raw reply [flat|nested] 6+ messages in thread