* Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough [not found] ` <53273B3C.40707@web2web.at> @ 2014-03-18 10:15 ` Ian Campbell 2014-03-18 13:01 ` Atom2 0 siblings, 1 reply; 23+ messages in thread From: Ian Campbell @ 2014-03-18 10:15 UTC (permalink / raw) To: Atom2; +Cc: xen-devel, Ian Jackson, Roger Pau Monne, xen-users Adding xen-devel. Full thread starts at http://lists.xen.org/archives/html/xen-users/2014-03/msg00102.html On Mon, 2014-03-17 at 19:13 +0100, Atom2 wrote: > > Any chance you could try 4.3.2, or even 4.4.0? > Unfortunately neither of these versions are currently available as > stable ebuilds for my distribution, but I assume it shouldn't be long > before there's some movement. Looking at the diff to tools/libxl/libxl_pci.c I don't see any pertinent looking fixes so it seems probably this issue still exists. > > > >> The system is capable of vt-d and uses a Xeon E3-1260L processor. > >> > >> Do these observations ring a bell with anybody or is this even expected > >> behaviour. If this is not normal - which I would expect as I have not > >> been able to find any information relating to substantial delays during > >> shutdown - how would I go about getting to the grounds of this? > > > > My guess would be that xl process which is managing the domain destroy > > is waiting for something (perhaps pciback) to confirm shutdown for each > > device and this is timing out in series, leading to the delays. You > > might find something in the logs /var/log/xen pointing to something like > > this. > > > > If not then if you start the guest with "xl -vvv create -F <cfg>" then > > the xl process which is monitoring the domain will stay in the > > foreground and be logging to stdout (I think). If you then issue the > > shutdown from another shell perhaps there will be some obvious gaps in > > the logs as things shutdown which might help. > That worked and there also was some output - please find the log from > start to finnish attached to this mail. I have marked various points in > the log: First the point where the startup was done and the domU was > live and secondly those 4 points in time (or rather output) where the > 10s delay occured. Quoting the relevant bit for -devel, full log is at http://lists.xen.org/archives/html/xen-users/2014-03/txtl6VscE4NMf.txt: Domain 3 has shut down, reason code 0 0x0 Action for shutdown reason code 0 is destroy Domain 3 needs to be cleaned up: destroying the domain libxl: debug: libxl.c:1252:libxl_domain_destroy: ao 0x7f1dddf2b850: create: how=(nil) callback=(nil) poller=0x7f1dddf2cd70 libxl: error: libxl_pci.c:1248:do_pci_remove: xc_domain_irq_permission irq=17 <NOTE: at this point a 10s pause happens> libxl: error: libxl_device.c:1127:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/3/0 not ready libxl: debug: libxl_pci.c:173:libxl__device_pci_remove_xenstore: pci backend at /local/domain/0/backend/pci/3/0 is not ready libxl: error: libxl_pci.c:1248:do_pci_remove: xc_domain_irq_permission irq=16 <NOTE: at this point a 10s pause happens> libxl: error: libxl_device.c:1127:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/3/0 not ready libxl: debug: libxl_pci.c:173:libxl__device_pci_remove_xenstore: pci backend at /local/domain/0/backend/pci/3/0 is not ready [repeat for more devices] Do you get anything in "xl dmesg" or dom0's "dmesg" corresponding to these events? Looking at do_pci_remove after the call to xc_domain_irq_permission (which fails, but I don't think that relates to the delay) we then call (conditionally) libxl__device_pci_reset, xc_deassign_device, libxl__device_pci_remove_common and libxl__device_pci_remove_xenstore, with no logging to indicate which we are calling (not helpful!). The "is not ready" message comes from libxl__device_pci_remove_xenstore which calls libxl__wait_for_backend. The latter has been rewritten a bit since 4.3.1 but not in a way which I think would affect this case. libxl__wait_for_backend does have a usleep(10000) in it -- which is certainly the source of the delay, but I'd like to explain how we got to waiting like that anyway (IanJ: do you have PCI on your hitlist for asyncing up?) This thing about pciback not being ready rings a bell. I've cc'd a few folks who I think might remember more. While the domain is happily running can you provide the output of "xenstore-ls -fp" -- I'm curious what state pciback is in. It should be 4, if not then that would be the problem. > BTW: I don't know whether it makes any difference, but I am only using > xen-pciback.hide=(bb:dd.f)(...) on the grub command line for a number of > devices including those that I pass through to this domU - there's > nothing else happening in the dom0 with those devices priot to starting > the domU and there are also no driver modules available for any of the > hidden hardware (except for one of the hidden USB Controllers of the > motherboard which is also passed through) in dom0. I don't think that should matter here. Ian. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough 2014-03-18 10:15 ` [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough Ian Campbell @ 2014-03-18 13:01 ` Atom2 2014-03-18 15:07 ` Ian Campbell 0 siblings, 1 reply; 23+ messages in thread From: Atom2 @ 2014-03-18 13:01 UTC (permalink / raw) To: Ian Campbell; +Cc: xen-devel, Ian Jackson, Roger Pau Monne, xen-users [-- Attachment #1: Type: text/plain, Size: 9075 bytes --] Am 18.03.14 11:15, schrieb Ian Campbell: > Adding xen-devel. Full thread starts at > http://lists.xen.org/archives/html/xen-users/2014-03/msg00102.html > > On Mon, 2014-03-17 at 19:13 +0100, Atom2 wrote: > >>> Any chance you could try 4.3.2, or even 4.4.0? >> Unfortunately neither of these versions are currently available as >> stable ebuilds for my distribution, but I assume it shouldn't be long >> before there's some movement. > > Looking at the diff to tools/libxl/libxl_pci.c I don't see any pertinent > looking fixes so it seems probably this issue still exists. > >>> >>>> The system is capable of vt-d and uses a Xeon E3-1260L processor. >>>> >>>> Do these observations ring a bell with anybody or is this even expected >>>> behaviour. If this is not normal - which I would expect as I have not >>>> been able to find any information relating to substantial delays during >>>> shutdown - how would I go about getting to the grounds of this? >>> >>> My guess would be that xl process which is managing the domain destroy >>> is waiting for something (perhaps pciback) to confirm shutdown for each >>> device and this is timing out in series, leading to the delays. You >>> might find something in the logs /var/log/xen pointing to something like >>> this. >>> >>> If not then if you start the guest with "xl -vvv create -F <cfg>" then >>> the xl process which is monitoring the domain will stay in the >>> foreground and be logging to stdout (I think). If you then issue the >>> shutdown from another shell perhaps there will be some obvious gaps in >>> the logs as things shutdown which might help. >> That worked and there also was some output - please find the log from >> start to finnish attached to this mail. I have marked various points in >> the log: First the point where the startup was done and the domU was >> live and secondly those 4 points in time (or rather output) where the >> 10s delay occured. > > Quoting the relevant bit for -devel, full log is at > http://lists.xen.org/archives/html/xen-users/2014-03/txtl6VscE4NMf.txt: > > Domain 3 has shut down, reason code 0 0x0 > Action for shutdown reason code 0 is destroy > Domain 3 needs to be cleaned up: destroying the domain > libxl: debug: libxl.c:1252:libxl_domain_destroy: ao 0x7f1dddf2b850: create: how=(nil) callback=(nil) poller=0x7f1dddf2cd70 > libxl: error: libxl_pci.c:1248:do_pci_remove: xc_domain_irq_permission irq=17 > > <NOTE: at this point a 10s pause happens> > > libxl: error: libxl_device.c:1127:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/3/0 not ready > libxl: debug: libxl_pci.c:173:libxl__device_pci_remove_xenstore: pci backend at /local/domain/0/backend/pci/3/0 is not ready > libxl: error: libxl_pci.c:1248:do_pci_remove: xc_domain_irq_permission irq=16 > > <NOTE: at this point a 10s pause happens> > > libxl: error: libxl_device.c:1127:libxl__wait_for_backend: Backend /local/domain/0/backend/pci/3/0 not ready > libxl: debug: libxl_pci.c:173:libxl__device_pci_remove_xenstore: pci backend at /local/domain/0/backend/pci/3/0 is not ready > [repeat for more devices] > > Do you get anything in "xl dmesg" or dom0's "dmesg" corresponding to > these events? xl dmesg: There is no additional output in dom0's 'xl dmesg' after creation of the domain with xl create -c There are however two additional lines in dom0's 'xl dmesg' output after I have executed 'shutdown -h now' from the domU's command line following a login as follows: ============== (XEN) irq.c:2028: dom3: forcing unbind of pirq 23 (XEN) tmem: flushing tmem pools for domid=3 ============== dmesg: Additional output of dmesg in dom0 after creating the the domain with xl create -c is as follows: ============== [ 322.225345] device vif3.0 entered promiscuous mode [ 322.828304] xen_pciback: vpci: 0000:03:00.0: assign to virtual slot 0 [ 322.829486] xen_pciback: vpci: 0000:06:00.0: assign to virtual slot 1 [ 322.840174] xen_pciback: vpci: 0000:09:02.0: assign to virtual slot 2 [ 322.841134] xen_pciback: vpci: 0000:00:1d.0: assign to virtual slot 3 [ 322.937946] xen-blkback:ring-ref 2047, event-channel 4, protocol 1 (x86_64-abi) [ 322.947106] xen-blkback:ring-ref 2046, event-channel 5, protocol 1 (x86_64-abi) [ 322.955840] xen-blkback:ring-ref 2045, event-channel 6, protocol 1 (x86_64-abi) [ 327.171287] xen-blkback:backend/vbd/3/51713: prepare for reconnect [ 327.194441] xen-blkback:backend/vbd/3/51714: prepare for reconnect [ 327.198687] xen-blkback:backend/vbd/3/51715: prepare for reconnect [ 327.925926] pciback 0000:00:1d.0: enabling device (0000 -> 0002) [ 327.926053] xen: registering gsi 23 triggering 0 polarity 1 [ 327.926059] Already setup the GSI :23 [ 327.926219] pciback 0000:00:1d.0: Driver tried to write to a read-only configuration space field at offset 0x6c, size 4. This may be harmless, but if you have problems with your device: 1) see permissive attribute in sysfs 2) report problems to the xen-devel mailing list along with details of your device obtained from lspci. [ 327.932217] xen-blkback:ring-ref 9, event-channel 18, protocol 1 (x86_64-abi) persistent grants [ 327.938142] xen-blkback:ring-ref 10, event-channel 19, protocol 1 (x86_64-abi) persistent grants [ 327.943674] xen-blkback:ring-ref 11, event-channel 20, protocol 1 (x86_64-abi) persistent grants [ 328.965061] xenbr0: port 4(vif3.0) entered forwarding state [ 328.965067] xenbr0: port 4(vif3.0) entered forwarding state [ 330.716771] pciback 0000:00:1d.0: setting latency timer to 64 [ 335.728218] pciback 0000:06:00.0: enabling device (0000 -> 0002) [ 335.728286] xen: registering gsi 16 triggering 0 polarity 1 [ 335.728290] Already setup the GSI :16 [ 343.992293] xenbr0: port 4(vif3.0) entered forwarding state ============== FYI: The 0000:00:1d.0 device is the USB host controller being passed through (Intel Sandybridge, C206 chipset); output of lspci: 00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05) Howvere currently there seems to be no issue with the USB host controller in domU: There are two USB bluetooth dongles attached and they do work (they connect to mobile phones). I'm more than happy to provide more info should this be required. Further output of dmesg in dom0 after I have executed 'shutdown -h now' from the domU's command line following a login is follows: ============== [ 556.864398] xenbr0: port 4(vif3.0) entered disabled state [ 597.584588] xenbr0: port 4(vif3.0) entered disabled state [ 597.584663] device vif3.0 left promiscuous mode [ 597.584668] xenbr0: port 4(vif3.0) entered disabled state ============== So I guess probably not too much of value ... > > Looking at do_pci_remove after the call to xc_domain_irq_permission > (which fails, but I don't think that relates to the delay) we then call > (conditionally) libxl__device_pci_reset, xc_deassign_device, > libxl__device_pci_remove_common and libxl__device_pci_remove_xenstore, > with no logging to indicate which we are calling (not helpful!). > > The "is not ready" message comes from libxl__device_pci_remove_xenstore > which calls libxl__wait_for_backend. The latter has been rewritten a bit > since 4.3.1 but not in a way which I think would affect this case. > libxl__wait_for_backend does have a usleep(10000) in it -- which is > certainly the source of the delay, but I'd like to explain how we got to > waiting like that anyway (IanJ: do you have PCI on your hitlist for > asyncing up?) > > This thing about pciback not being ready rings a bell. I've cc'd a few > folks who I think might remember more. > > While the domain is happily running can you provide the output of > "xenstore-ls -fp" -- I'm curious what state pciback is in. It should be > 4, if not then that would be the problem. Full output of xenstore-ls -fp (from dom0) is attached as it is 627 lines long and I am not quiet sure what you are actually after): There's nothing in it that reads pciback; there are however a few entries named /local/domain/0/backend/pci and for one of the subentries named 3/0/state (3 is the domain-id) the value seems to be 4: /local/domain/0/backend/pci/3/0/state = "4" (n0,r3) But then there is also another entry further down named /local/domain/3/device/pci/0/state = "4" (n3,r0) But I am speculating here and I guess it's better to leave the interpretation to your expertise ... > >> BTW: I don't know whether it makes any difference, but I am only using >> xen-pciback.hide=(bb:dd.f)(...) on the grub command line for a number of >> devices including those that I pass through to this domU - there's >> nothing else happening in the dom0 with those devices priot to starting >> the domU and there are also no driver modules available for any of the >> hidden hardware (except for one of the hidden USB Controllers of the >> motherboard which is also passed through) in dom0. > > I don't think that should matter here. > > Ian. > [-- Attachment #2: xenstore --] [-- Type: text/plain, Size: 38891 bytes --] /tool = "" (n0) /tool/xenstored = "" (n0) /vm = "" (n0) /vm/f8ed69cc-3f5e-4245-995e-0839553892e9 = "" (n0,r1) /vm/f8ed69cc-3f5e-4245-995e-0839553892e9/uuid = "f8ed69cc-3f5e-4245-995e-0839553892e9" (n0,r1) /vm/f8ed69cc-3f5e-4245-995e-0839553892e9/name = "ldap" (n0,r1) /vm/f8ed69cc-3f5e-4245-995e-0839553892e9/image = "" (n0,r1) /vm/f8ed69cc-3f5e-4245-995e-0839553892e9/image/ostype = "linux" (n0,r1) /vm/f8ed69cc-3f5e-4245-995e-0839553892e9/image/kernel = "/usr/lib/xen/boot/pv-grub-x86_64.gz" (n0,r1) /vm/f8ed69cc-3f5e-4245-995e-0839553892e9/image/ramdisk = "/etc/xen/guests/grub.d/ldap.grub" (n0,r1) /vm/f8ed69cc-3f5e-4245-995e-0839553892e9/image/cmdline = "" (n0,r1) /vm/f8ed69cc-3f5e-4245-995e-0839553892e9/start_time = "1393262439.59" (n0,r1) /vm/c72b254e-5887-4636-aa17-8c08dc2a756b = "" (n0,r6) /vm/c72b254e-5887-4636-aa17-8c08dc2a756b/uuid = "c72b254e-5887-4636-aa17-8c08dc2a756b" (n0,r6) /vm/c72b254e-5887-4636-aa17-8c08dc2a756b/name = "pki" (n0,r6) /vm/c72b254e-5887-4636-aa17-8c08dc2a756b/image = "" (n0,r6) /vm/c72b254e-5887-4636-aa17-8c08dc2a756b/image/ostype = "linux" (n0,r6) /vm/c72b254e-5887-4636-aa17-8c08dc2a756b/image/kernel = "/usr/lib/xen/boot/pv-grub-x86_64.gz" (n0,r6) /vm/c72b254e-5887-4636-aa17-8c08dc2a756b/image/ramdisk = "/etc/xen/guests/grub.d/pki.grub" (n0,r6) /vm/c72b254e-5887-4636-aa17-8c08dc2a756b/image/cmdline = "" (n0,r6) /vm/c72b254e-5887-4636-aa17-8c08dc2a756b/start_time = "1394963622.94" (n0,r6) /vm/ffe4ca76-e888-4a07-af5d-015d72497fd6 = "" (n0,r1) /vm/ffe4ca76-e888-4a07-af5d-015d72497fd6/uuid = "ffe4ca76-e888-4a07-af5d-015d72497fd6" (n0,r1) /vm/ffe4ca76-e888-4a07-af5d-015d72497fd6/name = "ldap" (n0,r1) /vm/ffe4ca76-e888-4a07-af5d-015d72497fd6/image = "" (n0,r1) /vm/ffe4ca76-e888-4a07-af5d-015d72497fd6/image/ostype = "linux" (n0,r1) /vm/ffe4ca76-e888-4a07-af5d-015d72497fd6/image/kernel = "/usr/lib/xen/boot/pv-grub-x86_64.gz" (n0,r1) /vm/ffe4ca76-e888-4a07-af5d-015d72497fd6/image/ramdisk = "/etc/xen/guests/grub.d/ldap.grub" (n0,r1) /vm/ffe4ca76-e888-4a07-af5d-015d72497fd6/image/cmdline = "" (n0,r1) /vm/ffe4ca76-e888-4a07-af5d-015d72497fd6/start_time = "1395146456.96" (n0,r1) /vm/3d564878-91c5-4384-a4ae-343508f41d60 = "" (n0,r2) /vm/3d564878-91c5-4384-a4ae-343508f41d60/uuid = "3d564878-91c5-4384-a4ae-343508f41d60" (n0,r2) /vm/3d564878-91c5-4384-a4ae-343508f41d60/name = "www" (n0,r2) /vm/3d564878-91c5-4384-a4ae-343508f41d60/image = "" (n0,r2) /vm/3d564878-91c5-4384-a4ae-343508f41d60/image/ostype = "linux" (n0,r2) /vm/3d564878-91c5-4384-a4ae-343508f41d60/image/kernel = "/usr/lib/xen/boot/pv-grub-x86_64.gz" (n0,r2) /vm/3d564878-91c5-4384-a4ae-343508f41d60/image/ramdisk = "/etc/xen/guests/grub.d/www.grub" (n0,r2) /vm/3d564878-91c5-4384-a4ae-343508f41d60/image/cmdline = "" (n0,r2) /vm/3d564878-91c5-4384-a4ae-343508f41d60/start_time = "1395146458.12" (n0,r2) /vm/8d28f3d9-1d51-41d9-853b-f67f1a750567 = "" (n0,r3) /vm/8d28f3d9-1d51-41d9-853b-f67f1a750567/uuid = "8d28f3d9-1d51-41d9-853b-f67f1a750567" (n0,r3) /vm/8d28f3d9-1d51-41d9-853b-f67f1a750567/name = "voip" (n0,r3) /vm/8d28f3d9-1d51-41d9-853b-f67f1a750567/image = "" (n0,r3) /vm/8d28f3d9-1d51-41d9-853b-f67f1a750567/image/ostype = "linux" (n0,r3) /vm/8d28f3d9-1d51-41d9-853b-f67f1a750567/image/kernel = "/usr/lib/xen/boot/pv-grub-x86_64.gz" (n0,r3) /vm/8d28f3d9-1d51-41d9-853b-f67f1a750567/image/ramdisk = "/etc/xen/guests/grub.d/voip.grub" (n0,r3) /vm/8d28f3d9-1d51-41d9-853b-f67f1a750567/image/cmdline = "" (n0,r3) /vm/8d28f3d9-1d51-41d9-853b-f67f1a750567/start_time = "1395146707.98" (n0,r3) /libxl = "" (n0) /libxl/1 = "" (n0) /libxl/1/dm-version = "qemu_xen" (n0) /libxl/2 = "" (n0) /libxl/2/dm-version = "qemu_xen" (n0) /libxl/3 = "" (n0) /libxl/3/dm-version = "qemu_xen" (n0) /local = "" (n0) /local/domain = "" (n0) /local/domain/0 = "" (n0) /local/domain/0/name = "Domain-0" (n0) /local/domain/0/libxl = "" (n0) /local/domain/0/libxl/disable_udev = "1" (n0) /local/domain/0/backend = "" (n0) /local/domain/0/backend/vbd = "" (n0) /local/domain/0/backend/vbd/1 = "" (n0) /local/domain/0/backend/vbd/1/51713 = "" (n0,r1) /local/domain/0/backend/vbd/1/51713/frontend = "/local/domain/1/device/vbd/51713" (n0,r1) /local/domain/0/backend/vbd/1/51713/params = "/etc/xen/guests/root.d/live.root" (n0,r1) /local/domain/0/backend/vbd/1/51713/script = "/etc/xen/scripts/block" (n0,r1) /local/domain/0/backend/vbd/1/51713/physical-device = "fd:2" (n0,r1) /local/domain/0/backend/vbd/1/51713/frontend-id = "1" (n0,r1) /local/domain/0/backend/vbd/1/51713/online = "1" (n0,r1) /local/domain/0/backend/vbd/1/51713/removable = "0" (n0,r1) /local/domain/0/backend/vbd/1/51713/bootable = "1" (n0,r1) /local/domain/0/backend/vbd/1/51713/state = "4" (n0,r1) /local/domain/0/backend/vbd/1/51713/dev = "xvda1" (n0,r1) /local/domain/0/backend/vbd/1/51713/type = "phy" (n0,r1) /local/domain/0/backend/vbd/1/51713/mode = "r" (n0,r1) /local/domain/0/backend/vbd/1/51713/device-type = "disk" (n0,r1) /local/domain/0/backend/vbd/1/51713/feature-flush-cache = "1" (n0,r1) /local/domain/0/backend/vbd/1/51713/feature-discard = "0" (n0,r1) /local/domain/0/backend/vbd/1/51713/feature-barrier = "1" (n0,r1) /local/domain/0/backend/vbd/1/51713/feature-persistent = "1" (n0,r1) /local/domain/0/backend/vbd/1/51713/feature-max-indirect-segments = "256" (n0,r1) /local/domain/0/backend/vbd/1/51713/sectors = "16777216" (n0,r1) /local/domain/0/backend/vbd/1/51713/info = "4" (n0,r1) /local/domain/0/backend/vbd/1/51713/sector-size = "512" (n0,r1) /local/domain/0/backend/vbd/1/51713/physical-sector-size = "512" (n0,r1) /local/domain/0/backend/vbd/1/51714 = "" (n0,r1) /local/domain/0/backend/vbd/1/51714/frontend = "/local/domain/1/device/vbd/51714" (n0,r1) /local/domain/0/backend/vbd/1/51714/params = "/etc/xen/guests/swap.d/ldap.swap" (n0,r1) /local/domain/0/backend/vbd/1/51714/script = "/etc/xen/scripts/block" (n0,r1) /local/domain/0/backend/vbd/1/51714/physical-device = "fd:4" (n0,r1) /local/domain/0/backend/vbd/1/51714/frontend-id = "1" (n0,r1) /local/domain/0/backend/vbd/1/51714/online = "1" (n0,r1) /local/domain/0/backend/vbd/1/51714/removable = "0" (n0,r1) /local/domain/0/backend/vbd/1/51714/bootable = "1" (n0,r1) /local/domain/0/backend/vbd/1/51714/state = "4" (n0,r1) /local/domain/0/backend/vbd/1/51714/dev = "xvda2" (n0,r1) /local/domain/0/backend/vbd/1/51714/type = "phy" (n0,r1) /local/domain/0/backend/vbd/1/51714/mode = "w" (n0,r1) /local/domain/0/backend/vbd/1/51714/device-type = "disk" (n0,r1) /local/domain/0/backend/vbd/1/51714/feature-flush-cache = "1" (n0,r1) /local/domain/0/backend/vbd/1/51714/feature-discard = "0" (n0,r1) /local/domain/0/backend/vbd/1/51714/feature-barrier = "1" (n0,r1) /local/domain/0/backend/vbd/1/51714/feature-persistent = "1" (n0,r1) /local/domain/0/backend/vbd/1/51714/feature-max-indirect-segments = "256" (n0,r1) /local/domain/0/backend/vbd/1/51714/sectors = "2097152" (n0,r1) /local/domain/0/backend/vbd/1/51714/info = "0" (n0,r1) /local/domain/0/backend/vbd/1/51714/sector-size = "512" (n0,r1) /local/domain/0/backend/vbd/1/51714/physical-sector-size = "512" (n0,r1) /local/domain/0/backend/vbd/1/51715 = "" (n0,r1) /local/domain/0/backend/vbd/1/51715/frontend = "/local/domain/1/device/vbd/51715" (n0,r1) /local/domain/0/backend/vbd/1/51715/params = "/etc/xen/guests/overlay.d/ldap.ovly" (n0,r1) /local/domain/0/backend/vbd/1/51715/script = "/etc/xen/scripts/block" (n0,r1) /local/domain/0/backend/vbd/1/51715/physical-device = "fd:a" (n0,r1) /local/domain/0/backend/vbd/1/51715/frontend-id = "1" (n0,r1) /local/domain/0/backend/vbd/1/51715/online = "1" (n0,r1) /local/domain/0/backend/vbd/1/51715/removable = "0" (n0,r1) /local/domain/0/backend/vbd/1/51715/bootable = "1" (n0,r1) /local/domain/0/backend/vbd/1/51715/state = "4" (n0,r1) /local/domain/0/backend/vbd/1/51715/dev = "xvda3" (n0,r1) /local/domain/0/backend/vbd/1/51715/type = "phy" (n0,r1) /local/domain/0/backend/vbd/1/51715/mode = "w" (n0,r1) /local/domain/0/backend/vbd/1/51715/device-type = "disk" (n0,r1) /local/domain/0/backend/vbd/1/51715/feature-flush-cache = "1" (n0,r1) /local/domain/0/backend/vbd/1/51715/feature-discard = "0" (n0,r1) /local/domain/0/backend/vbd/1/51715/feature-barrier = "1" (n0,r1) /local/domain/0/backend/vbd/1/51715/feature-persistent = "1" (n0,r1) /local/domain/0/backend/vbd/1/51715/feature-max-indirect-segments = "256" (n0,r1) /local/domain/0/backend/vbd/1/51715/sectors = "2097152" (n0,r1) /local/domain/0/backend/vbd/1/51715/info = "0" (n0,r1) /local/domain/0/backend/vbd/1/51715/sector-size = "512" (n0,r1) /local/domain/0/backend/vbd/1/51715/physical-sector-size = "512" (n0,r1) /local/domain/0/backend/vbd/2 = "" (n0) /local/domain/0/backend/vbd/2/51713 = "" (n0,r2) /local/domain/0/backend/vbd/2/51713/frontend = "/local/domain/2/device/vbd/51713" (n0,r2) /local/domain/0/backend/vbd/2/51713/params = "/etc/xen/guests/root.d/live.root" (n0,r2) /local/domain/0/backend/vbd/2/51713/script = "/etc/xen/scripts/block" (n0,r2) /local/domain/0/backend/vbd/2/51713/physical-device = "fd:2" (n0,r2) /local/domain/0/backend/vbd/2/51713/frontend-id = "2" (n0,r2) /local/domain/0/backend/vbd/2/51713/online = "1" (n0,r2) /local/domain/0/backend/vbd/2/51713/removable = "0" (n0,r2) /local/domain/0/backend/vbd/2/51713/bootable = "1" (n0,r2) /local/domain/0/backend/vbd/2/51713/state = "4" (n0,r2) /local/domain/0/backend/vbd/2/51713/dev = "xvda1" (n0,r2) /local/domain/0/backend/vbd/2/51713/type = "phy" (n0,r2) /local/domain/0/backend/vbd/2/51713/mode = "r" (n0,r2) /local/domain/0/backend/vbd/2/51713/device-type = "disk" (n0,r2) /local/domain/0/backend/vbd/2/51713/feature-flush-cache = "1" (n0,r2) /local/domain/0/backend/vbd/2/51713/feature-discard = "0" (n0,r2) /local/domain/0/backend/vbd/2/51713/feature-barrier = "1" (n0,r2) /local/domain/0/backend/vbd/2/51713/feature-persistent = "1" (n0,r2) /local/domain/0/backend/vbd/2/51713/feature-max-indirect-segments = "256" (n0,r2) /local/domain/0/backend/vbd/2/51713/sectors = "16777216" (n0,r2) /local/domain/0/backend/vbd/2/51713/info = "4" (n0,r2) /local/domain/0/backend/vbd/2/51713/sector-size = "512" (n0,r2) /local/domain/0/backend/vbd/2/51713/physical-sector-size = "512" (n0,r2) /local/domain/0/backend/vbd/2/51714 = "" (n0,r2) /local/domain/0/backend/vbd/2/51714/frontend = "/local/domain/2/device/vbd/51714" (n0,r2) /local/domain/0/backend/vbd/2/51714/params = "/etc/xen/guests/swap.d/www.swap" (n0,r2) /local/domain/0/backend/vbd/2/51714/script = "/etc/xen/scripts/block" (n0,r2) /local/domain/0/backend/vbd/2/51714/physical-device = "fd:e" (n0,r2) /local/domain/0/backend/vbd/2/51714/frontend-id = "2" (n0,r2) /local/domain/0/backend/vbd/2/51714/online = "1" (n0,r2) /local/domain/0/backend/vbd/2/51714/removable = "0" (n0,r2) /local/domain/0/backend/vbd/2/51714/bootable = "1" (n0,r2) /local/domain/0/backend/vbd/2/51714/state = "4" (n0,r2) /local/domain/0/backend/vbd/2/51714/dev = "xvda2" (n0,r2) /local/domain/0/backend/vbd/2/51714/type = "phy" (n0,r2) /local/domain/0/backend/vbd/2/51714/mode = "w" (n0,r2) /local/domain/0/backend/vbd/2/51714/device-type = "disk" (n0,r2) /local/domain/0/backend/vbd/2/51714/feature-flush-cache = "1" (n0,r2) /local/domain/0/backend/vbd/2/51714/feature-discard = "0" (n0,r2) /local/domain/0/backend/vbd/2/51714/feature-barrier = "1" (n0,r2) /local/domain/0/backend/vbd/2/51714/feature-persistent = "1" (n0,r2) /local/domain/0/backend/vbd/2/51714/feature-max-indirect-segments = "256" (n0,r2) /local/domain/0/backend/vbd/2/51714/sectors = "2097152" (n0,r2) /local/domain/0/backend/vbd/2/51714/info = "0" (n0,r2) /local/domain/0/backend/vbd/2/51714/sector-size = "512" (n0,r2) /local/domain/0/backend/vbd/2/51714/physical-sector-size = "512" (n0,r2) /local/domain/0/backend/vbd/2/51715 = "" (n0,r2) /local/domain/0/backend/vbd/2/51715/frontend = "/local/domain/2/device/vbd/51715" (n0,r2) /local/domain/0/backend/vbd/2/51715/params = "/etc/xen/guests/overlay.d/www.ovly" (n0,r2) /local/domain/0/backend/vbd/2/51715/script = "/etc/xen/scripts/block" (n0,r2) /local/domain/0/backend/vbd/2/51715/physical-device = "fd:d" (n0,r2) /local/domain/0/backend/vbd/2/51715/frontend-id = "2" (n0,r2) /local/domain/0/backend/vbd/2/51715/online = "1" (n0,r2) /local/domain/0/backend/vbd/2/51715/removable = "0" (n0,r2) /local/domain/0/backend/vbd/2/51715/bootable = "1" (n0,r2) /local/domain/0/backend/vbd/2/51715/state = "4" (n0,r2) /local/domain/0/backend/vbd/2/51715/dev = "xvda3" (n0,r2) /local/domain/0/backend/vbd/2/51715/type = "phy" (n0,r2) /local/domain/0/backend/vbd/2/51715/mode = "w" (n0,r2) /local/domain/0/backend/vbd/2/51715/device-type = "disk" (n0,r2) /local/domain/0/backend/vbd/2/51715/feature-flush-cache = "1" (n0,r2) /local/domain/0/backend/vbd/2/51715/feature-discard = "0" (n0,r2) /local/domain/0/backend/vbd/2/51715/feature-barrier = "1" (n0,r2) /local/domain/0/backend/vbd/2/51715/feature-persistent = "1" (n0,r2) /local/domain/0/backend/vbd/2/51715/feature-max-indirect-segments = "256" (n0,r2) /local/domain/0/backend/vbd/2/51715/sectors = "2097152" (n0,r2) /local/domain/0/backend/vbd/2/51715/info = "0" (n0,r2) /local/domain/0/backend/vbd/2/51715/sector-size = "512" (n0,r2) /local/domain/0/backend/vbd/2/51715/physical-sector-size = "512" (n0,r2) /local/domain/0/backend/vbd/3 = "" (n0) /local/domain/0/backend/vbd/3/51713 = "" (n0,r3) /local/domain/0/backend/vbd/3/51713/frontend = "/local/domain/3/device/vbd/51713" (n0,r3) /local/domain/0/backend/vbd/3/51713/params = "/etc/xen/guests/root.d/live.root" (n0,r3) /local/domain/0/backend/vbd/3/51713/script = "/etc/xen/scripts/block" (n0,r3) /local/domain/0/backend/vbd/3/51713/physical-device = "fd:2" (n0,r3) /local/domain/0/backend/vbd/3/51713/frontend-id = "3" (n0,r3) /local/domain/0/backend/vbd/3/51713/online = "1" (n0,r3) /local/domain/0/backend/vbd/3/51713/removable = "0" (n0,r3) /local/domain/0/backend/vbd/3/51713/bootable = "1" (n0,r3) /local/domain/0/backend/vbd/3/51713/state = "4" (n0,r3) /local/domain/0/backend/vbd/3/51713/dev = "xvda1" (n0,r3) /local/domain/0/backend/vbd/3/51713/type = "phy" (n0,r3) /local/domain/0/backend/vbd/3/51713/mode = "r" (n0,r3) /local/domain/0/backend/vbd/3/51713/device-type = "disk" (n0,r3) /local/domain/0/backend/vbd/3/51713/feature-flush-cache = "1" (n0,r3) /local/domain/0/backend/vbd/3/51713/feature-discard = "0" (n0,r3) /local/domain/0/backend/vbd/3/51713/feature-barrier = "1" (n0,r3) /local/domain/0/backend/vbd/3/51713/feature-persistent = "1" (n0,r3) /local/domain/0/backend/vbd/3/51713/feature-max-indirect-segments = "256" (n0,r3) /local/domain/0/backend/vbd/3/51713/sectors = "16777216" (n0,r3) /local/domain/0/backend/vbd/3/51713/info = "4" (n0,r3) /local/domain/0/backend/vbd/3/51713/sector-size = "512" (n0,r3) /local/domain/0/backend/vbd/3/51713/physical-sector-size = "512" (n0,r3) /local/domain/0/backend/vbd/3/51714 = "" (n0,r3) /local/domain/0/backend/vbd/3/51714/frontend = "/local/domain/3/device/vbd/51714" (n0,r3) /local/domain/0/backend/vbd/3/51714/params = "/etc/xen/guests/swap.d/voip.swap" (n0,r3) /local/domain/0/backend/vbd/3/51714/script = "/etc/xen/scripts/block" (n0,r3) /local/domain/0/backend/vbd/3/51714/physical-device = "fd:11" (n0,r3) /local/domain/0/backend/vbd/3/51714/frontend-id = "3" (n0,r3) /local/domain/0/backend/vbd/3/51714/online = "1" (n0,r3) /local/domain/0/backend/vbd/3/51714/removable = "0" (n0,r3) /local/domain/0/backend/vbd/3/51714/bootable = "1" (n0,r3) /local/domain/0/backend/vbd/3/51714/state = "4" (n0,r3) /local/domain/0/backend/vbd/3/51714/dev = "xvda2" (n0,r3) /local/domain/0/backend/vbd/3/51714/type = "phy" (n0,r3) /local/domain/0/backend/vbd/3/51714/mode = "w" (n0,r3) /local/domain/0/backend/vbd/3/51714/device-type = "disk" (n0,r3) /local/domain/0/backend/vbd/3/51714/feature-flush-cache = "1" (n0,r3) /local/domain/0/backend/vbd/3/51714/feature-discard = "0" (n0,r3) /local/domain/0/backend/vbd/3/51714/feature-barrier = "1" (n0,r3) /local/domain/0/backend/vbd/3/51714/feature-persistent = "1" (n0,r3) /local/domain/0/backend/vbd/3/51714/feature-max-indirect-segments = "256" (n0,r3) /local/domain/0/backend/vbd/3/51714/sectors = "2097152" (n0,r3) /local/domain/0/backend/vbd/3/51714/info = "0" (n0,r3) /local/domain/0/backend/vbd/3/51714/sector-size = "512" (n0,r3) /local/domain/0/backend/vbd/3/51714/physical-sector-size = "512" (n0,r3) /local/domain/0/backend/vbd/3/51715 = "" (n0,r3) /local/domain/0/backend/vbd/3/51715/frontend = "/local/domain/3/device/vbd/51715" (n0,r3) /local/domain/0/backend/vbd/3/51715/params = "/etc/xen/guests/overlay.d/voip.ovly" (n0,r3) /local/domain/0/backend/vbd/3/51715/script = "/etc/xen/scripts/block" (n0,r3) /local/domain/0/backend/vbd/3/51715/physical-device = "fd:12" (n0,r3) /local/domain/0/backend/vbd/3/51715/frontend-id = "3" (n0,r3) /local/domain/0/backend/vbd/3/51715/online = "1" (n0,r3) /local/domain/0/backend/vbd/3/51715/removable = "0" (n0,r3) /local/domain/0/backend/vbd/3/51715/bootable = "1" (n0,r3) /local/domain/0/backend/vbd/3/51715/state = "4" (n0,r3) /local/domain/0/backend/vbd/3/51715/dev = "xvda3" (n0,r3) /local/domain/0/backend/vbd/3/51715/type = "phy" (n0,r3) /local/domain/0/backend/vbd/3/51715/mode = "w" (n0,r3) /local/domain/0/backend/vbd/3/51715/device-type = "disk" (n0,r3) /local/domain/0/backend/vbd/3/51715/feature-flush-cache = "1" (n0,r3) /local/domain/0/backend/vbd/3/51715/feature-discard = "0" (n0,r3) /local/domain/0/backend/vbd/3/51715/feature-barrier = "1" (n0,r3) /local/domain/0/backend/vbd/3/51715/feature-persistent = "1" (n0,r3) /local/domain/0/backend/vbd/3/51715/feature-max-indirect-segments = "256" (n0,r3) /local/domain/0/backend/vbd/3/51715/sectors = "2097152" (n0,r3) /local/domain/0/backend/vbd/3/51715/info = "0" (n0,r3) /local/domain/0/backend/vbd/3/51715/sector-size = "512" (n0,r3) /local/domain/0/backend/vbd/3/51715/physical-sector-size = "512" (n0,r3) /local/domain/0/backend/console = "" (n0) /local/domain/0/backend/console/1 = "" (n0) /local/domain/0/backend/console/1/0 = "" (n0,r1) /local/domain/0/backend/console/1/0/frontend = "/local/domain/1/console" (n0,r1) /local/domain/0/backend/console/1/0/frontend-id = "1" (n0,r1) /local/domain/0/backend/console/1/0/online = "1" (n0,r1) /local/domain/0/backend/console/1/0/state = "1" (n0,r1) /local/domain/0/backend/console/1/0/domain = "ldap" (n0,r1) /local/domain/0/backend/console/1/0/protocol = "vt100" (n0,r1) /local/domain/0/backend/console/2 = "" (n0) /local/domain/0/backend/console/2/0 = "" (n0,r2) /local/domain/0/backend/console/2/0/frontend = "/local/domain/2/console" (n0,r2) /local/domain/0/backend/console/2/0/frontend-id = "2" (n0,r2) /local/domain/0/backend/console/2/0/online = "1" (n0,r2) /local/domain/0/backend/console/2/0/state = "1" (n0,r2) /local/domain/0/backend/console/2/0/domain = "www" (n0,r2) /local/domain/0/backend/console/2/0/protocol = "vt100" (n0,r2) /local/domain/0/backend/console/3 = "" (n0) /local/domain/0/backend/console/3/0 = "" (n0,r3) /local/domain/0/backend/console/3/0/frontend = "/local/domain/3/console" (n0,r3) /local/domain/0/backend/console/3/0/frontend-id = "3" (n0,r3) /local/domain/0/backend/console/3/0/online = "1" (n0,r3) /local/domain/0/backend/console/3/0/state = "1" (n0,r3) /local/domain/0/backend/console/3/0/domain = "voip" (n0,r3) /local/domain/0/backend/console/3/0/protocol = "vt100" (n0,r3) /local/domain/0/backend/vif = "" (n0) /local/domain/0/backend/vif/1 = "" (n0) /local/domain/0/backend/vif/1/0 = "" (n0,r1) /local/domain/0/backend/vif/1/0/frontend = "/local/domain/1/device/vif/0" (n0,r1) /local/domain/0/backend/vif/1/0/frontend-id = "1" (n0,r1) /local/domain/0/backend/vif/1/0/online = "1" (n0,r1) /local/domain/0/backend/vif/1/0/state = "4" (n0,r1) /local/domain/0/backend/vif/1/0/script = "/etc/xen/scripts/vif-bridge" (n0,r1) /local/domain/0/backend/vif/1/0/mac = "00:16:3e:a0:64:04" (n0,r1) /local/domain/0/backend/vif/1/0/bridge = "xenbr0" (n0,r1) /local/domain/0/backend/vif/1/0/handle = "0" (n0,r1) /local/domain/0/backend/vif/1/0/type = "vif" (n0,r1) /local/domain/0/backend/vif/1/0/feature-sg = "1" (n0,r1) /local/domain/0/backend/vif/1/0/feature-gso-tcpv4 = "1" (n0,r1) /local/domain/0/backend/vif/1/0/feature-rx-copy = "1" (n0,r1) /local/domain/0/backend/vif/1/0/feature-rx-flip = "0" (n0,r1) /local/domain/0/backend/vif/1/0/feature-split-event-channels = "1" (n0,r1) /local/domain/0/backend/vif/1/0/hotplug-status = "connected" (n0,r1) /local/domain/0/backend/vif/2 = "" (n0) /local/domain/0/backend/vif/2/0 = "" (n0,r2) /local/domain/0/backend/vif/2/0/frontend = "/local/domain/2/device/vif/0" (n0,r2) /local/domain/0/backend/vif/2/0/frontend-id = "2" (n0,r2) /local/domain/0/backend/vif/2/0/online = "1" (n0,r2) /local/domain/0/backend/vif/2/0/state = "4" (n0,r2) /local/domain/0/backend/vif/2/0/script = "/etc/xen/scripts/vif-bridge" (n0,r2) /local/domain/0/backend/vif/2/0/mac = "00:16:3e:a0:64:07" (n0,r2) /local/domain/0/backend/vif/2/0/bridge = "xenbr0" (n0,r2) /local/domain/0/backend/vif/2/0/handle = "0" (n0,r2) /local/domain/0/backend/vif/2/0/type = "vif" (n0,r2) /local/domain/0/backend/vif/2/0/feature-sg = "1" (n0,r2) /local/domain/0/backend/vif/2/0/feature-gso-tcpv4 = "1" (n0,r2) /local/domain/0/backend/vif/2/0/feature-rx-copy = "1" (n0,r2) /local/domain/0/backend/vif/2/0/feature-rx-flip = "0" (n0,r2) /local/domain/0/backend/vif/2/0/feature-split-event-channels = "1" (n0,r2) /local/domain/0/backend/vif/2/0/hotplug-status = "connected" (n0,r2) /local/domain/0/backend/vif/3 = "" (n0) /local/domain/0/backend/vif/3/0 = "" (n0,r3) /local/domain/0/backend/vif/3/0/frontend = "/local/domain/3/device/vif/0" (n0,r3) /local/domain/0/backend/vif/3/0/frontend-id = "3" (n0,r3) /local/domain/0/backend/vif/3/0/online = "1" (n0,r3) /local/domain/0/backend/vif/3/0/state = "4" (n0,r3) /local/domain/0/backend/vif/3/0/script = "/etc/xen/scripts/vif-bridge" (n0,r3) /local/domain/0/backend/vif/3/0/mac = "00:16:3e:a0:64:09" (n0,r3) /local/domain/0/backend/vif/3/0/bridge = "xenbr0" (n0,r3) /local/domain/0/backend/vif/3/0/handle = "0" (n0,r3) /local/domain/0/backend/vif/3/0/type = "vif" (n0,r3) /local/domain/0/backend/vif/3/0/feature-sg = "1" (n0,r3) /local/domain/0/backend/vif/3/0/feature-gso-tcpv4 = "1" (n0,r3) /local/domain/0/backend/vif/3/0/feature-rx-copy = "1" (n0,r3) /local/domain/0/backend/vif/3/0/feature-rx-flip = "0" (n0,r3) /local/domain/0/backend/vif/3/0/feature-split-event-channels = "1" (n0,r3) /local/domain/0/backend/vif/3/0/hotplug-status = "connected" (n0,r3) /local/domain/0/backend/pci = "" (n0) /local/domain/0/backend/pci/3 = "" (n0) /local/domain/0/backend/pci/3/0 = "" (n0,r3) /local/domain/0/backend/pci/3/0/frontend = "/local/domain/3/device/pci/0" (n0,r3) /local/domain/0/backend/pci/3/0/frontend-id = "3" (n0,r3) /local/domain/0/backend/pci/3/0/online = "1" (n0,r3) /local/domain/0/backend/pci/3/0/state = "4" (n0,r3) /local/domain/0/backend/pci/3/0/domain = "voip" (n0,r3) /local/domain/0/backend/pci/3/0/key-0 = "0000:03:00.0" (n0,r3) /local/domain/0/backend/pci/3/0/dev-0 = "0000:03:00.0" (n0,r3) /local/domain/0/backend/pci/3/0/opts-0 = "msitranslate=0,power_mgmt=0,permissive=0" (n0,r3) /local/domain/0/backend/pci/3/0/state-0 = "3" (n0,r3) /local/domain/0/backend/pci/3/0/key-1 = "0000:06:00.0" (n0,r3) /local/domain/0/backend/pci/3/0/dev-1 = "0000:06:00.0" (n0,r3) /local/domain/0/backend/pci/3/0/opts-1 = "msitranslate=0,power_mgmt=0,permissive=0" (n0,r3) /local/domain/0/backend/pci/3/0/state-1 = "3" (n0,r3) /local/domain/0/backend/pci/3/0/key-2 = "0000:09:02.0" (n0,r3) /local/domain/0/backend/pci/3/0/dev-2 = "0000:09:02.0" (n0,r3) /local/domain/0/backend/pci/3/0/opts-2 = "msitranslate=0,power_mgmt=0,permissive=0" (n0,r3) /local/domain/0/backend/pci/3/0/state-2 = "3" (n0,r3) /local/domain/0/backend/pci/3/0/key-3 = "0000:00:1d.0" (n0,r3) /local/domain/0/backend/pci/3/0/dev-3 = "0000:00:1d.0" (n0,r3) /local/domain/0/backend/pci/3/0/opts-3 = "msitranslate=0,power_mgmt=0,permissive=0" (n0,r3) /local/domain/0/backend/pci/3/0/state-3 = "3" (n0,r3) /local/domain/0/backend/pci/3/0/num_devs = "4" (n0,r3) /local/domain/0/backend/pci/3/0/vdev-0 = "0000:00:00.00" (n0,r3) /local/domain/0/backend/pci/3/0/vdev-1 = "0000:00:01.00" (n0,r3) /local/domain/0/backend/pci/3/0/vdev-2 = "0000:00:02.00" (n0,r3) /local/domain/0/backend/pci/3/0/vdev-3 = "0000:00:03.00" (n0,r3) /local/domain/0/backend/pci/3/0/root-0 = "0000:00" (n0,r3) /local/domain/0/backend/pci/3/0/root_num = "1" (n0,r3) /local/domain/0/device-model = "" (n0) /local/domain/0/device-model/0 = "" (n0) /local/domain/0/device-model/0/state = "running" (n0) /local/domain/1 = "" (n0,r1) /local/domain/1/vm = "/vm/ffe4ca76-e888-4a07-af5d-015d72497fd6" (n0,r1) /local/domain/1/name = "ldap" (n0,r1) /local/domain/1/cpu = "" (n0,r1) /local/domain/1/cpu/0 = "" (n0,r1) /local/domain/1/cpu/0/availability = "online" (n0,r1) /local/domain/1/cpu/1 = "" (n0,r1) /local/domain/1/cpu/1/availability = "online" (n0,r1) /local/domain/1/memory = "" (n0,r1) /local/domain/1/memory/static-max = "2097152" (n0,r1) /local/domain/1/memory/target = "1048577" (n0,r1) /local/domain/1/memory/videoram = "-1" (n0,r1) /local/domain/1/device = "" (n0,r1) /local/domain/1/device/suspend = "" (n0,r1) /local/domain/1/device/suspend/event-channel = "" (n1) /local/domain/1/device/vbd = "" (n0,r1) /local/domain/1/device/vbd/51713 = "" (n1,r0) /local/domain/1/device/vbd/51713/backend = "/local/domain/0/backend/vbd/1/51713" (n1,r0) /local/domain/1/device/vbd/51713/backend-id = "0" (n1,r0) /local/domain/1/device/vbd/51713/state = "4" (n1,r0) /local/domain/1/device/vbd/51713/virtual-device = "51713" (n1,r0) /local/domain/1/device/vbd/51713/device-type = "disk" (n1,r0) /local/domain/1/device/vbd/51713/protocol = "x86_64-abi" (n1,r0) /local/domain/1/device/vbd/51713/ring-ref = "8" (n1,r0) /local/domain/1/device/vbd/51713/event-channel = "17" (n1,r0) /local/domain/1/device/vbd/51713/feature-persistent = "1" (n1,r0) /local/domain/1/device/vbd/51714 = "" (n1,r0) /local/domain/1/device/vbd/51714/backend = "/local/domain/0/backend/vbd/1/51714" (n1,r0) /local/domain/1/device/vbd/51714/backend-id = "0" (n1,r0) /local/domain/1/device/vbd/51714/state = "4" (n1,r0) /local/domain/1/device/vbd/51714/virtual-device = "51714" (n1,r0) /local/domain/1/device/vbd/51714/device-type = "disk" (n1,r0) /local/domain/1/device/vbd/51714/protocol = "x86_64-abi" (n1,r0) /local/domain/1/device/vbd/51714/ring-ref = "9" (n1,r0) /local/domain/1/device/vbd/51714/event-channel = "18" (n1,r0) /local/domain/1/device/vbd/51714/feature-persistent = "1" (n1,r0) /local/domain/1/device/vbd/51715 = "" (n1,r0) /local/domain/1/device/vbd/51715/backend = "/local/domain/0/backend/vbd/1/51715" (n1,r0) /local/domain/1/device/vbd/51715/backend-id = "0" (n1,r0) /local/domain/1/device/vbd/51715/state = "4" (n1,r0) /local/domain/1/device/vbd/51715/virtual-device = "51715" (n1,r0) /local/domain/1/device/vbd/51715/device-type = "disk" (n1,r0) /local/domain/1/device/vbd/51715/protocol = "x86_64-abi" (n1,r0) /local/domain/1/device/vbd/51715/ring-ref = "10" (n1,r0) /local/domain/1/device/vbd/51715/event-channel = "19" (n1,r0) /local/domain/1/device/vbd/51715/feature-persistent = "1" (n1,r0) /local/domain/1/device/vif = "" (n0,r1) /local/domain/1/device/vif/0 = "" (n1,r0) /local/domain/1/device/vif/0/backend = "/local/domain/0/backend/vif/1/0" (n1,r0) /local/domain/1/device/vif/0/backend-id = "0" (n1,r0) /local/domain/1/device/vif/0/state = "4" (n1,r0) /local/domain/1/device/vif/0/handle = "0" (n1,r0) /local/domain/1/device/vif/0/mac = "00:16:3e:a0:64:04" (n1,r0) /local/domain/1/device/vif/0/tx-ring-ref = "11" (n1,r0) /local/domain/1/device/vif/0/rx-ring-ref = "12" (n1,r0) /local/domain/1/device/vif/0/event-channel-tx = "20" (n1,r0) /local/domain/1/device/vif/0/event-channel-rx = "21" (n1,r0) /local/domain/1/device/vif/0/request-rx-copy = "1" (n1,r0) /local/domain/1/device/vif/0/feature-rx-notify = "1" (n1,r0) /local/domain/1/device/vif/0/feature-sg = "1" (n1,r0) /local/domain/1/device/vif/0/feature-gso-tcpv4 = "1" (n1,r0) /local/domain/1/control = "" (n0,r1) /local/domain/1/control/shutdown = "" (n1) /local/domain/1/control/platform-feature-multiprocessor-suspend = "1" (n0,r1) /local/domain/1/control/platform-feature-xs_reset_watches = "1" (n0,r1) /local/domain/1/data = "" (n1) /local/domain/1/domid = "1" (n0,r1) /local/domain/1/store = "" (n0,r1) /local/domain/1/store/port = "1" (n0,r1) /local/domain/1/store/ring-ref = "7443678" (n0,r1) /local/domain/1/console = "" (n0,r1) /local/domain/1/console/backend = "/local/domain/0/backend/console/1/0" (n0,r1) /local/domain/1/console/backend-id = "0" (n1,r0) /local/domain/1/console/limit = "1048576" (n0,r1) /local/domain/1/console/type = "xenconsoled" (n0,r1) /local/domain/1/console/output = "pty" (n0,r1) /local/domain/1/console/tty = "/dev/pts/4" (n0,r1) /local/domain/1/console/port = "2" (n0,r1) /local/domain/1/console/ring-ref = "7443677" (n0,r1) /local/domain/2 = "" (n0,r2) /local/domain/2/vm = "/vm/3d564878-91c5-4384-a4ae-343508f41d60" (n0,r2) /local/domain/2/name = "www" (n0,r2) /local/domain/2/cpu = "" (n0,r2) /local/domain/2/cpu/0 = "" (n0,r2) /local/domain/2/cpu/0/availability = "online" (n0,r2) /local/domain/2/memory = "" (n0,r2) /local/domain/2/memory/static-max = "524288" (n0,r2) /local/domain/2/memory/target = "524289" (n0,r2) /local/domain/2/memory/videoram = "-1" (n0,r2) /local/domain/2/device = "" (n0,r2) /local/domain/2/device/suspend = "" (n0,r2) /local/domain/2/device/suspend/event-channel = "" (n2) /local/domain/2/device/vbd = "" (n0,r2) /local/domain/2/device/vbd/51713 = "" (n2,r0) /local/domain/2/device/vbd/51713/backend = "/local/domain/0/backend/vbd/2/51713" (n2,r0) /local/domain/2/device/vbd/51713/backend-id = "0" (n2,r0) /local/domain/2/device/vbd/51713/state = "4" (n2,r0) /local/domain/2/device/vbd/51713/virtual-device = "51713" (n2,r0) /local/domain/2/device/vbd/51713/device-type = "disk" (n2,r0) /local/domain/2/device/vbd/51713/protocol = "x86_64-abi" (n2,r0) /local/domain/2/device/vbd/51713/ring-ref = "8" (n2,r0) /local/domain/2/device/vbd/51713/event-channel = "10" (n2,r0) /local/domain/2/device/vbd/51713/feature-persistent = "1" (n2,r0) /local/domain/2/device/vbd/51714 = "" (n2,r0) /local/domain/2/device/vbd/51714/backend = "/local/domain/0/backend/vbd/2/51714" (n2,r0) /local/domain/2/device/vbd/51714/backend-id = "0" (n2,r0) /local/domain/2/device/vbd/51714/state = "4" (n2,r0) /local/domain/2/device/vbd/51714/virtual-device = "51714" (n2,r0) /local/domain/2/device/vbd/51714/device-type = "disk" (n2,r0) /local/domain/2/device/vbd/51714/protocol = "x86_64-abi" (n2,r0) /local/domain/2/device/vbd/51714/ring-ref = "9" (n2,r0) /local/domain/2/device/vbd/51714/event-channel = "11" (n2,r0) /local/domain/2/device/vbd/51714/feature-persistent = "1" (n2,r0) /local/domain/2/device/vbd/51715 = "" (n2,r0) /local/domain/2/device/vbd/51715/backend = "/local/domain/0/backend/vbd/2/51715" (n2,r0) /local/domain/2/device/vbd/51715/backend-id = "0" (n2,r0) /local/domain/2/device/vbd/51715/state = "4" (n2,r0) /local/domain/2/device/vbd/51715/virtual-device = "51715" (n2,r0) /local/domain/2/device/vbd/51715/device-type = "disk" (n2,r0) /local/domain/2/device/vbd/51715/protocol = "x86_64-abi" (n2,r0) /local/domain/2/device/vbd/51715/ring-ref = "10" (n2,r0) /local/domain/2/device/vbd/51715/event-channel = "12" (n2,r0) /local/domain/2/device/vbd/51715/feature-persistent = "1" (n2,r0) /local/domain/2/device/vif = "" (n0,r2) /local/domain/2/device/vif/0 = "" (n2,r0) /local/domain/2/device/vif/0/backend = "/local/domain/0/backend/vif/2/0" (n2,r0) /local/domain/2/device/vif/0/backend-id = "0" (n2,r0) /local/domain/2/device/vif/0/state = "4" (n2,r0) /local/domain/2/device/vif/0/handle = "0" (n2,r0) /local/domain/2/device/vif/0/mac = "00:16:3e:a0:64:07" (n2,r0) /local/domain/2/device/vif/0/tx-ring-ref = "11" (n2,r0) /local/domain/2/device/vif/0/rx-ring-ref = "12" (n2,r0) /local/domain/2/device/vif/0/event-channel-tx = "13" (n2,r0) /local/domain/2/device/vif/0/event-channel-rx = "14" (n2,r0) /local/domain/2/device/vif/0/request-rx-copy = "1" (n2,r0) /local/domain/2/device/vif/0/feature-rx-notify = "1" (n2,r0) /local/domain/2/device/vif/0/feature-sg = "1" (n2,r0) /local/domain/2/device/vif/0/feature-gso-tcpv4 = "1" (n2,r0) /local/domain/2/control = "" (n0,r2) /local/domain/2/control/shutdown = "" (n2) /local/domain/2/control/platform-feature-multiprocessor-suspend = "1" (n0,r2) /local/domain/2/control/platform-feature-xs_reset_watches = "1" (n0,r2) /local/domain/2/data = "" (n2) /local/domain/2/domid = "2" (n0,r2) /local/domain/2/store = "" (n0,r2) /local/domain/2/store/port = "1" (n0,r2) /local/domain/2/store/ring-ref = "7631284" (n0,r2) /local/domain/2/console = "" (n0,r2) /local/domain/2/console/backend = "/local/domain/0/backend/console/2/0" (n0,r2) /local/domain/2/console/backend-id = "0" (n2,r0) /local/domain/2/console/limit = "1048576" (n0,r2) /local/domain/2/console/type = "xenconsoled" (n0,r2) /local/domain/2/console/output = "pty" (n0,r2) /local/domain/2/console/tty = "/dev/pts/5" (n0,r2) /local/domain/2/console/port = "2" (n0,r2) /local/domain/2/console/ring-ref = "7631283" (n0,r2) /local/domain/3 = "" (n0,r3) /local/domain/3/vm = "/vm/8d28f3d9-1d51-41d9-853b-f67f1a750567" (n0,r3) /local/domain/3/name = "voip" (n0,r3) /local/domain/3/cpu = "" (n0,r3) /local/domain/3/cpu/0 = "" (n0,r3) /local/domain/3/cpu/0/availability = "online" (n0,r3) /local/domain/3/cpu/1 = "" (n0,r3) /local/domain/3/cpu/1/availability = "online" (n0,r3) /local/domain/3/memory = "" (n0,r3) /local/domain/3/memory/static-max = "1048576" (n0,r3) /local/domain/3/memory/target = "1048577" (n0,r3) /local/domain/3/memory/videoram = "-1" (n0,r3) /local/domain/3/device = "" (n0,r3) /local/domain/3/device/suspend = "" (n0,r3) /local/domain/3/device/suspend/event-channel = "" (n3) /local/domain/3/device/vbd = "" (n0,r3) /local/domain/3/device/vbd/51713 = "" (n3,r0) /local/domain/3/device/vbd/51713/backend = "/local/domain/0/backend/vbd/3/51713" (n3,r0) /local/domain/3/device/vbd/51713/backend-id = "0" (n3,r0) /local/domain/3/device/vbd/51713/state = "4" (n3,r0) /local/domain/3/device/vbd/51713/virtual-device = "51713" (n3,r0) /local/domain/3/device/vbd/51713/device-type = "disk" (n3,r0) /local/domain/3/device/vbd/51713/protocol = "x86_64-abi" (n3,r0) /local/domain/3/device/vbd/51713/ring-ref = "9" (n3,r0) /local/domain/3/device/vbd/51713/event-channel = "18" (n3,r0) /local/domain/3/device/vbd/51713/feature-persistent = "1" (n3,r0) /local/domain/3/device/vbd/51714 = "" (n3,r0) /local/domain/3/device/vbd/51714/backend = "/local/domain/0/backend/vbd/3/51714" (n3,r0) /local/domain/3/device/vbd/51714/backend-id = "0" (n3,r0) /local/domain/3/device/vbd/51714/state = "4" (n3,r0) /local/domain/3/device/vbd/51714/virtual-device = "51714" (n3,r0) /local/domain/3/device/vbd/51714/device-type = "disk" (n3,r0) /local/domain/3/device/vbd/51714/protocol = "x86_64-abi" (n3,r0) /local/domain/3/device/vbd/51714/ring-ref = "10" (n3,r0) /local/domain/3/device/vbd/51714/event-channel = "19" (n3,r0) /local/domain/3/device/vbd/51714/feature-persistent = "1" (n3,r0) /local/domain/3/device/vbd/51715 = "" (n3,r0) /local/domain/3/device/vbd/51715/backend = "/local/domain/0/backend/vbd/3/51715" (n3,r0) /local/domain/3/device/vbd/51715/backend-id = "0" (n3,r0) /local/domain/3/device/vbd/51715/state = "4" (n3,r0) /local/domain/3/device/vbd/51715/virtual-device = "51715" (n3,r0) /local/domain/3/device/vbd/51715/device-type = "disk" (n3,r0) /local/domain/3/device/vbd/51715/protocol = "x86_64-abi" (n3,r0) /local/domain/3/device/vbd/51715/ring-ref = "11" (n3,r0) /local/domain/3/device/vbd/51715/event-channel = "20" (n3,r0) /local/domain/3/device/vbd/51715/feature-persistent = "1" (n3,r0) /local/domain/3/device/vif = "" (n0,r3) /local/domain/3/device/vif/0 = "" (n3,r0) /local/domain/3/device/vif/0/backend = "/local/domain/0/backend/vif/3/0" (n3,r0) /local/domain/3/device/vif/0/backend-id = "0" (n3,r0) /local/domain/3/device/vif/0/state = "4" (n3,r0) /local/domain/3/device/vif/0/handle = "0" (n3,r0) /local/domain/3/device/vif/0/mac = "00:16:3e:a0:64:09" (n3,r0) /local/domain/3/device/vif/0/tx-ring-ref = "12" (n3,r0) /local/domain/3/device/vif/0/rx-ring-ref = "13" (n3,r0) /local/domain/3/device/vif/0/event-channel-tx = "21" (n3,r0) /local/domain/3/device/vif/0/event-channel-rx = "22" (n3,r0) /local/domain/3/device/vif/0/request-rx-copy = "1" (n3,r0) /local/domain/3/device/vif/0/feature-rx-notify = "1" (n3,r0) /local/domain/3/device/vif/0/feature-sg = "1" (n3,r0) /local/domain/3/device/vif/0/feature-gso-tcpv4 = "1" (n3,r0) /local/domain/3/device/pci = "" (n0,r3) /local/domain/3/device/pci/0 = "" (n3,r0) /local/domain/3/device/pci/0/backend = "/local/domain/0/backend/pci/3/0" (n3,r0) /local/domain/3/device/pci/0/backend-id = "0" (n3,r0) /local/domain/3/device/pci/0/state = "4" (n3,r0) /local/domain/3/device/pci/0/pci-op-ref = "8" (n3,r0) /local/domain/3/device/pci/0/event-channel = "17" (n3,r0) /local/domain/3/device/pci/0/magic = "7" (n3,r0) /local/domain/3/control = "" (n0,r3) /local/domain/3/control/shutdown = "" (n3) /local/domain/3/control/platform-feature-multiprocessor-suspend = "1" (n0,r3) /local/domain/3/control/platform-feature-xs_reset_watches = "1" (n0,r3) /local/domain/3/data = "" (n3) /local/domain/3/domid = "3" (n0,r3) /local/domain/3/store = "" (n0,r3) /local/domain/3/store/port = "1" (n0,r3) /local/domain/3/store/ring-ref = "7627436" (n0,r3) /local/domain/3/console = "" (n0,r3) /local/domain/3/console/backend = "/local/domain/0/backend/console/3/0" (n0,r3) /local/domain/3/console/backend-id = "0" (n3,r0) /local/domain/3/console/limit = "1048576" (n0,r3) /local/domain/3/console/type = "xenconsoled" (n0,r3) /local/domain/3/console/output = "pty" (n0,r3) /local/domain/3/console/tty = "/dev/pts/6" (n0,r3) /local/domain/3/console/port = "2" (n0,r3) /local/domain/3/console/ring-ref = "7627435" (n0,r3) [-- Attachment #3: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough 2014-03-18 13:01 ` Atom2 @ 2014-03-18 15:07 ` Ian Campbell 2014-03-19 0:25 ` Atom2 0 siblings, 1 reply; 23+ messages in thread From: Ian Campbell @ 2014-03-18 15:07 UTC (permalink / raw) To: Atom2; +Cc: xen-devel, Ian Jackson, Roger Pau Monne, xen-users On Tue, 2014-03-18 at 14:01 +0100, Atom2 wrote: [...] > So I guess probably not too much of value ... Unfortunately not, but thanks anyway, it's good to check. > > While the domain is happily running can you provide the output of > > "xenstore-ls -fp" -- I'm curious what state pciback is in. It should be > > 4, if not then that would be the problem. > Full output of xenstore-ls -fp (from dom0) is attached as it is 627 > lines long and I am not quiet sure what you are actually after): There's > nothing in it that reads pciback; there are however a few entries named > /local/domain/0/backend/pci > and for one of the subentries named 3/0/state (3 is the domain-id) > the value seems to be 4: > /local/domain/0/backend/pci/3/0/state = "4" (n0,r3) Yes, this is the one which libxl appears to be looking for and it is set to "4" which is what libxl is looking for. Please can you try this again but take the dump while the destruction is going on, i.e. in among the 10s delays somewhere (I don't think where exactly will matter. I don't see it in the logs but I'm wondering if the pciback directory is getting torn down too soon. Ian. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough 2014-03-18 15:07 ` Ian Campbell @ 2014-03-19 0:25 ` Atom2 2014-03-19 11:26 ` Ian Campbell 0 siblings, 1 reply; 23+ messages in thread From: Atom2 @ 2014-03-19 0:25 UTC (permalink / raw) To: Ian Campbell; +Cc: xen-devel, Ian Jackson, Roger Pau Monne, xen-users Am 18.03.14 16:07, schrieb Ian Campbell: > On Tue, 2014-03-18 at 14:01 +0100, Atom2 wrote: > [...] >> So I guess probably not too much of value ... > > Unfortunately not, but thanks anyway, it's good to check. > >>> While the domain is happily running can you provide the output of >>> "xenstore-ls -fp" -- I'm curious what state pciback is in. It should be >>> 4, if not then that would be the problem. >> Full output of xenstore-ls -fp (from dom0) is attached as it is 627 >> lines long and I am not quiet sure what you are actually after): There's >> nothing in it that reads pciback; there are however a few entries named >> /local/domain/0/backend/pci >> and for one of the subentries named 3/0/state (3 is the domain-id) >> the value seems to be 4: >> /local/domain/0/backend/pci/3/0/state = "4" (n0,r3) > > Yes, this is the one which libxl appears to be looking for and it is set > to "4" which is what libxl is looking for. > > Please can you try this again but take the dump while the destruction is > going on, i.e. in among the 10s delays somewhere (I don't think where > exactly will matter. I don't see it in the logs but I'm wondering if the > pciback directory is getting torn down too soon. > O.K. - that's what I have done: I created a loop at the shell prompt in dom0 that executes 'xenstore-ls -fp' for a total of 16 times with a delay of one second between invocations and stores each output in a separate file with an extension that counts up from 0. I have started that "script" just before the "reboot: System halted" message appeared in the domU, so at least the first file must still contain the info prior to the 10s delay when the domU is still sort of up and running. The output below is the result of grep -E '/local/domain/(0|8)/(backend|device)/pci/.*/state' xenstore.n where the domain id this time was 8 and n is the counter of my loop. Please find the output below: .0 and .1 file: /local/domain/0/backend/pci/8/0/state = "4" (n0,r8) /local/domain/0/backend/pci/8/0/state-0 = "3" (n0,r8) /local/domain/0/backend/pci/8/0/state-1 = "3" (n0,r8) /local/domain/0/backend/pci/8/0/state-2 = "3" (n0,r8) /local/domain/0/backend/pci/8/0/state-3 = "3" (n0,r8) /local/domain/8/device/pci/0/state = "4" (n8,r0) .2 upto .15 file: /local/domain/0/backend/pci/8/0/state = "6" (n0,r8) /local/domain/0/backend/pci/8/0/state-0 = "3" (n0,r8) /local/domain/0/backend/pci/8/0/state-1 = "3" (n0,r8) /local/domain/0/backend/pci/8/0/state-2 = "3" (n0,r8) /local/domain/0/backend/pci/8/0/state-3 = "3" (n0,r8) /local/domain/8/device/pci/0/state = "6" (n8,r0) So it seems that pretty much at the start of the 10s delay the state changed from 4 to 6 and stays at that value even after the first 10s delay is over - whatever that means. And for the sake of being complete: Below is the output from starting the domain prior to the grub menu showing on screen. There is one device with which xl seems to have an issue (for which I still have to find and compile a driver anyway), but I think that's unrelated because if I drop that one device from being passed through, the delay of 10s per remaining device nevertheless remains. =========================== Parsing config from 5:voip.9 libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:09:02.0 Daemon running with PID 4759 Xen Minimal OS! start_info: 0xbab000(VA) nr_pages: 0x40000 shared_inf: 0xdb4f9000(MA) pt_base: 0xbae000(VA) nr_pt_frames: 0xb mfn_list: 0x9ab000(VA) mod_start: 0x9aa000(VA) mod_len: 4096 flags: 0x0 cmd_line: stack: 0x9690e0-0x9890e0 MM: Init _text: 0x0(VA) _etext: 0x7b4c5(VA) _erodata: 0x96000(VA) _edata: 0x9bd20(VA) stack start: 0x9690e0(VA) _end: 0x9a96e0(VA) start_pfn: bbc max_pfn: 40000 Mapping memory range 0x1000000 - 0x40000000 setting 0x0-0x96000 readonly skipped 0x1000 MM: Initialise page allocator for db4000(db4000)-40000000(40000000) MM: done Demand map pfns at 40001000-2040001000. Heap resides at 2040002000-4040002000. Initialising timer interface Initialising console ... done. gnttab_table mapped at 0x40001000. Initialising scheduler Thread "Idle": pointer: 0x2040002050, stack: 0xfd0000 Thread "xenstore": pointer: 0x2040002800, stack: 0xfe0000 xenbus initialised on irq 1 mfn 0x7a512a Thread "shutdown": pointer: 0x2040002fb0, stack: 0xff0000 Dummy main: start_info=0x9891e0 Thread "main": pointer: 0x2040003760, stack: 0x1000000 "main" vbd 51713 is hd0 ******************* BLKFRONT for device/vbd/51713 ********** Shutting down () Shutdown requested: 3 Thread "shutdown" exited. backend at /local/domain/0/backend/vbd/8/51713 16777216 sectors of 512 bytes ************************** vbd 51714 is hd1 ******************* BLKFRONT for device/vbd/51714 ********** backend at /local/domain/0/backend/vbd/8/51714 2097152 sectors of 512 bytes ************************** vbd 51715 is hd2 ******************* BLKFRONT for device/vbd/51715 ********** backend at /local/domain/0/backend/vbd/8/51715 2097152 sectors of 512 bytes ************************** =========================== Thanks Atom2 ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough 2014-03-19 0:25 ` Atom2 @ 2014-03-19 11:26 ` Ian Campbell 2014-03-19 13:00 ` Konrad Rzeszutek Wilk 0 siblings, 1 reply; 23+ messages in thread From: Ian Campbell @ 2014-03-19 11:26 UTC (permalink / raw) To: Atom2; +Cc: xen-devel, Ian Jackson, Roger Pau Monne, xen-users On Wed, 2014-03-19 at 01:25 +0100, Atom2 wrote: > So it seems that pretty much at the start of the 10s delay the state > changed from 4 to 6 and stays at that value even after the first 10s > delay is over - whatever that means. 4 == Connected 6 == Closed I think what is happening is that the domain is shutting down, which causes pciback to transition to the closed state (because the f.e. went away, so this is a reasonable thing for it to do). The bug appears to be that libxl is trying to "hot unplug" the devices on shutdown when they have already been effectively "cold unplugged" by the domain going down. Perhaps libxl__device_pci_remove_xenstore should observe that the state is > 4 (hence closing/closed) and not bother doing anything, i.e. only waiting iff the state is <4 (init, connecting etc)? Or unconditionally removing the nodes if state > 4. (perhaps state 7, reconfiguring needs handling here too) Or perhaps the force parameter passed to remove_common (which indicates destroy rather than unplug) ought to be propagated down to this code and $something done with it. Roger, Ian, any thoughts on that? Ian. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough 2014-03-19 11:26 ` Ian Campbell @ 2014-03-19 13:00 ` Konrad Rzeszutek Wilk 2014-03-19 14:03 ` Atom2 0 siblings, 1 reply; 23+ messages in thread From: Konrad Rzeszutek Wilk @ 2014-03-19 13:00 UTC (permalink / raw) To: Ian Campbell; +Cc: Roger Pau Monne, xen-devel, Ian Jackson, Atom2, xen-users On Wed, Mar 19, 2014 at 11:26:24AM +0000, Ian Campbell wrote: > On Wed, 2014-03-19 at 01:25 +0100, Atom2 wrote: > > So it seems that pretty much at the start of the 10s delay the state > > changed from 4 to 6 and stays at that value even after the first 10s > > delay is over - whatever that means. > > 4 == Connected > 6 == Closed > > I think what is happening is that the domain is shutting down, which > causes pciback to transition to the closed state (because the f.e. went > away, so this is a reasonable thing for it to do). > > The bug appears to be that libxl is trying to "hot unplug" the devices > on shutdown when they have already been effectively "cold unplugged" by > the domain going down. > > Perhaps libxl__device_pci_remove_xenstore should observe that the state > is > 4 (hence closing/closed) and not bother doing anything, i.e. only > waiting iff the state is <4 (init, connecting etc)? Or unconditionally > removing the nodes if state > 4. (perhaps state 7, reconfiguring needs > handling here too) > > Or perhaps the force parameter passed to remove_common (which indicates > destroy rather than unplug) ought to be propagated down to this code and > $something done with it. > > Roger, Ian, any thoughts on that? This reminds me of this bug: commit 098b1aeaf4d6149953b8f1f8d55c21d85536fbff Author: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> Date: Mon Jun 10 16:48:09 2013 -0400 xen/pcifront: Deal with toolstack missing 'XenbusStateClosing' state. ... snip.. In other words, this 4(Connected)->5(Closing)->4(Connected) state was expected, while 4(Connected)->.... anything but 5(Closing)->4(Connected) was not. This patch removes that aggressive check and allows Xen pcifront to work with the 'xl' toolstack (for one or more PCI devices) and with 'xm' toolstack (for more than two PCI devices). But this seems to be a different state issue? Ariel/Atom2, do you see this behavior with 'xend'? And what is the version of Linux kernel you are running as guest? > > Ian. > ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough 2014-03-19 13:00 ` Konrad Rzeszutek Wilk @ 2014-03-19 14:03 ` Atom2 2014-03-19 15:45 ` Ian Jackson 0 siblings, 1 reply; 23+ messages in thread From: Atom2 @ 2014-03-19 14:03 UTC (permalink / raw) To: Konrad Rzeszutek Wilk, Ian Campbell Cc: xen-devel, Ian Jackson, Roger Pau Monne, xen-users Am 19.03.14 14:00, schrieb Konrad Rzeszutek Wilk: > On Wed, Mar 19, 2014 at 11:26:24AM +0000, Ian Campbell wrote: >> On Wed, 2014-03-19 at 01:25 +0100, Atom2 wrote: >>> So it seems that pretty much at the start of the 10s delay the state >>> changed from 4 to 6 and stays at that value even after the first 10s >>> delay is over - whatever that means. >> >> 4 == Connected >> 6 == Closed >> >> I think what is happening is that the domain is shutting down, which >> causes pciback to transition to the closed state (because the f.e. went >> away, so this is a reasonable thing for it to do). >> >> The bug appears to be that libxl is trying to "hot unplug" the devices >> on shutdown when they have already been effectively "cold unplugged" by >> the domain going down. I might be wrong, but this behaviour is somehow reminescent of (although not identical to) the bug in the vif-bridge script that I reported some time ago (see http://xen.markmail.org/thread/auroivzr4vje3bzn ; btw discussions there seem to have stalled): The vif-bridge script also tried to do something (i.e. deleting an i/f from the bridge and bringing down the i/f) which obviously has already been done through shutting down the guest domain. >> >> Perhaps libxl__device_pci_remove_xenstore should observe that the state >> is > 4 (hence closing/closed) and not bother doing anything, i.e. only >> waiting iff the state is <4 (init, connecting etc)? Or unconditionally >> removing the nodes if state > 4. (perhaps state 7, reconfiguring needs >> handling here too) >> >> Or perhaps the force parameter passed to remove_common (which indicates >> destroy rather than unplug) ought to be propagated down to this code and >> $something done with it. >> >> Roger, Ian, any thoughts on that? > > This reminds me of this bug: > > commit 098b1aeaf4d6149953b8f1f8d55c21d85536fbff > Author: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> > Date: Mon Jun 10 16:48:09 2013 -0400 > > xen/pcifront: Deal with toolstack missing 'XenbusStateClosing' state. > > ... snip.. > > In other words, this 4(Connected)->5(Closing)->4(Connected) state > was expected, while 4(Connected)->.... anything but 5(Closing)->4(Connected) > was not. This patch removes that aggressive check and allows > Xen pcifront to work with the 'xl' toolstack (for one or more > PCI devices) and with 'xm' toolstack (for more than two PCI > devices). > > But this seems to be a different state issue? > > Ariel/Atom2, do you see this behavior with 'xend'? And what is the version of Linux > kernel you are running as guest? Hi Konrad - nope, I am using xl; there is no xend or xm installed on the machine or involved anyhow (I assumed with xend you referred back to xm instead of xl). The xen (and xen-tools) version is 4.3.1-r5 and the linux kernel is 3.11.7-r1 from gentoo hardened-sources (that's both for guest and for dom0 - although clearly with different kernel configs). Both the kernel and xen/xen-tools are the latest stable versions available as ebuilds from gentoo. >> >> Ian. >> ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough 2014-03-19 14:03 ` Atom2 @ 2014-03-19 15:45 ` Ian Jackson 2014-03-20 2:31 ` Atom2 0 siblings, 1 reply; 23+ messages in thread From: Ian Jackson @ 2014-03-19 15:45 UTC (permalink / raw) To: Atom2; +Cc: Roger Pau Monne, xen-users, xen-devel, Ian Campbell Atom2 writes ("Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough"): > nope, I am using xl; there is no xend or xm installed on the machine or > involved anyhow (I assumed with xend you referred back to xm instead of xl). Can you try this patch ? Thanks, Ian. diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c index 30b0b06..1583498 100644 --- a/tools/libxl/libxl.c +++ b/tools/libxl/libxl.c @@ -2728,7 +2728,7 @@ static void local_device_attach_cb(libxl__egc *egc, libxl__ao_device *aodev) if (rc < 0) goto out; be_path = libxl__device_backend_path(gc, &device); - rc = libxl__wait_for_backend(gc, be_path, "4"); + rc = libxl__wait_for_backend_deprecated(gc, be_path, "4", (char*)0); if (rc < 0) goto out; diff --git a/tools/libxl/libxl_device.c b/tools/libxl/libxl_device.c index fa99f77..11a9885 100644 --- a/tools/libxl/libxl_device.c +++ b/tools/libxl/libxl_device.c @@ -1208,12 +1208,14 @@ int libxl__wait_for_device_model_deprecated(libxl__gc *gc, check_callback, check_callback_userdata); } -int libxl__wait_for_backend(libxl__gc *gc, const char *be_path, - const char *state) +int libxl__wait_for_backend_deprecated(libxl__gc *gc, const char *be_path, + ...) { int watchdog = 100; const char *p, *path = GCSPRINTF("%s/state", be_path); + const char *want; int rc; + va_list al; while (watchdog-- > 0) { rc = libxl__xs_read_checked(gc, XBT_NULL, path, &p); @@ -1224,8 +1226,14 @@ int libxl__wait_for_backend(libxl__gc *gc, const char *be_path, return ERROR_FAIL; } - if (!strcmp(p, state)) - return 0; + va_start(al,be_path); + while ((want = va_arg(al, char*))) { + if (!strcmp(p, want)) { + va_end(al); + return 0; + } + } + va_end(al); usleep(100000); } diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h index b3a200d..bdcce35 100644 --- a/tools/libxl/libxl_internal.h +++ b/tools/libxl/libxl_internal.h @@ -1025,8 +1025,8 @@ _hidden char *libxl__device_frontend_path(libxl__gc *gc, libxl__device *device); _hidden int libxl__parse_backend_path(libxl__gc *gc, const char *path, libxl__device *dev); _hidden int libxl__device_destroy(libxl__gc *gc, libxl__device *dev); -_hidden int libxl__wait_for_backend(libxl__gc *gc, const char *be_path, - const char *state); +_hidden int libxl__wait_for_backend_deprecated(libxl__gc *gc, + const char *be_path, ...) __attribute__((sentinel)); _hidden int libxl__nic_type(libxl__gc *gc, libxl__device *dev, libxl_nic_type *nictype); diff --git a/tools/libxl/libxl_pci.c b/tools/libxl/libxl_pci.c index 44d0453..43ffd57 100644 --- a/tools/libxl/libxl_pci.c +++ b/tools/libxl/libxl_pci.c @@ -126,7 +126,7 @@ static int libxl__device_pci_add_xenstore(libxl__gc *gc, uint32_t domid, libxl_d return ERROR_FAIL; if (!starting && domtype == LIBXL_DOMAIN_TYPE_PV) { - if (libxl__wait_for_backend(gc, be_path, "4") < 0) + if (libxl__wait_for_backend_deprecated(gc, be_path, "4", (char*)0) < 0) return ERROR_FAIL; } @@ -169,7 +169,8 @@ static int libxl__device_pci_remove_xenstore(libxl__gc *gc, uint32_t domid, libx return ERROR_FAIL; if (domtype == LIBXL_DOMAIN_TYPE_PV) { - if (libxl__wait_for_backend(gc, be_path, "4") < 0) { + if (libxl__wait_for_backend_deprecated(gc, be_path, "4", "6", (char*)0) + < 0) { LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "pci backend at %s is not ready", be_path); return ERROR_FAIL; } @@ -198,7 +199,8 @@ retry_transaction: goto retry_transaction; if (domtype == LIBXL_DOMAIN_TYPE_PV) { - if (libxl__wait_for_backend(gc, be_path, "4") < 0) { + if (libxl__wait_for_backend_deprecated(gc, be_path, "4", "6", (char*)0) + < 0) { LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "pci backend at %s is not ready", be_path); return ERROR_FAIL; } ^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough 2014-03-19 15:45 ` Ian Jackson @ 2014-03-20 2:31 ` Atom2 2014-03-20 11:52 ` Ian Jackson 0 siblings, 1 reply; 23+ messages in thread From: Atom2 @ 2014-03-20 2:31 UTC (permalink / raw) To: Ian Jackson; +Cc: xen-users, xen-devel, Ian Campbell, Roger Pau Monne [-- Attachment #1: Type: text/plain, Size: 5510 bytes --] Sorry for my delay in answering - this is a resend as the first e-Mail with uncompressed attachments did not go through. Am 19.03.14 16:45, schrieb Ian Jackson: > Atom2 writes ("Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough"): >> nope, I am using xl; there is no xend or xm installed on the machine or >> involved anyhow (I assumed with xend you referred back to xm instead of xl). > > Can you try this patch ? > > Thanks, > Ian. Hi Ian, the patch unfortunately doesn't apply to my sources - some comments to the reasons why further below. Just FYI: the version I am using is 4.3.1-r5; I have attached the relevant source files referred to by your patches. Thanks and regards, Atom2 > > diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c > index 30b0b06..1583498 100644 > --- a/tools/libxl/libxl.c > +++ b/tools/libxl/libxl.c > @@ -2728,7 +2728,7 @@ static void local_device_attach_cb(libxl__egc *egc, libxl__ao_device *aodev) > if (rc < 0) > goto out; > be_path = libxl__device_backend_path(gc, &device); > - rc = libxl__wait_for_backend(gc, be_path, "4"); > + rc = libxl__wait_for_backend_deprecated(gc, be_path, "4", (char*)0); > if (rc < 0) > goto out; > This one would apply with an offset of 43 lines - it should therefore be o.k I guess. > diff --git a/tools/libxl/libxl_device.c b/tools/libxl/libxl_device.c > index fa99f77..11a9885 100644 > --- a/tools/libxl/libxl_device.c > +++ b/tools/libxl/libxl_device.c > @@ -1208,12 +1208,14 @@ int libxl__wait_for_device_model_deprecated(libxl__gc *gc, > check_callback, check_callback_userdata); > } > > -int libxl__wait_for_backend(libxl__gc *gc, const char *be_path, > - const char *state) > +int libxl__wait_for_backend_deprecated(libxl__gc *gc, const char *be_path, > + ...) > { > int watchdog = 100; > const char *p, *path = GCSPRINTF("%s/state", be_path); > + const char *want; > int rc; > + va_list al; > > while (watchdog-- > 0) { > rc = libxl__xs_read_checked(gc, XBT_NULL, path, &p); > @@ -1224,8 +1226,14 @@ int libxl__wait_for_backend(libxl__gc *gc, const char *be_path, > return ERROR_FAIL; > } > > - if (!strcmp(p, state)) > - return 0; > + va_start(al,be_path); > + while ((want = va_arg(al, char*))) { > + if (!strcmp(p, want)) { > + va_end(al); > + return 0; > + } > + } > + va_end(al); > > usleep(100000); > } This one does not apply, not the least because there is no function libxl__wait_for_device_model_deprecated in the source. Furthermore the definition and the body of libxl__wait_for_backend looks differently to what the patch seems to expect - please see attached source file. > diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h > index b3a200d..bdcce35 100644 > --- a/tools/libxl/libxl_internal.h > +++ b/tools/libxl/libxl_internal.h > @@ -1025,8 +1025,8 @@ _hidden char *libxl__device_frontend_path(libxl__gc *gc, libxl__device *device); > _hidden int libxl__parse_backend_path(libxl__gc *gc, const char *path, > libxl__device *dev); > _hidden int libxl__device_destroy(libxl__gc *gc, libxl__device *dev); > -_hidden int libxl__wait_for_backend(libxl__gc *gc, const char *be_path, > - const char *state); > +_hidden int libxl__wait_for_backend_deprecated(libxl__gc *gc, > + const char *be_path, ...) __attribute__((sentinel)); > _hidden int libxl__nic_type(libxl__gc *gc, libxl__device *dev, > libxl_nic_type *nictype); > This one fails because the definition of libxl__wait_for_backend(...) is different (see above) - please see attached source file. > diff --git a/tools/libxl/libxl_pci.c b/tools/libxl/libxl_pci.c > index 44d0453..43ffd57 100644 > --- a/tools/libxl/libxl_pci.c > +++ b/tools/libxl/libxl_pci.c > @@ -126,7 +126,7 @@ static int libxl__device_pci_add_xenstore(libxl__gc *gc, uint32_t domid, libxl_d > return ERROR_FAIL; > > if (!starting && domtype == LIBXL_DOMAIN_TYPE_PV) { > - if (libxl__wait_for_backend(gc, be_path, "4") < 0) > + if (libxl__wait_for_backend_deprecated(gc, be_path, "4", (char*)0) < 0) > return ERROR_FAIL; > } > > @@ -169,7 +169,8 @@ static int libxl__device_pci_remove_xenstore(libxl__gc *gc, uint32_t domid, libx > return ERROR_FAIL; > > if (domtype == LIBXL_DOMAIN_TYPE_PV) { > - if (libxl__wait_for_backend(gc, be_path, "4") < 0) { > + if (libxl__wait_for_backend_deprecated(gc, be_path, "4", "6", (char*)0) > + < 0) { > LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "pci backend at %s is not ready", be_path); > return ERROR_FAIL; > } > @@ -198,7 +199,8 @@ retry_transaction: > goto retry_transaction; > > if (domtype == LIBXL_DOMAIN_TYPE_PV) { > - if (libxl__wait_for_backend(gc, be_path, "4") < 0) { > + if (libxl__wait_for_backend_deprecated(gc, be_path, "4", "6", (char*)0) > + < 0) { > LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "pci backend at %s is not ready", be_path); > return ERROR_FAIL; > } > The last one does apply cleanly - even without any offset. [-- Attachment #2: libxl_device.c.7z --] [-- Type: application/octet-stream, Size: 8161 bytes --] [-- Attachment #3: libxl_internal.h.7z --] [-- Type: application/octet-stream, Size: 28765 bytes --] [-- Attachment #4: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough 2014-03-20 2:31 ` Atom2 @ 2014-03-20 11:52 ` Ian Jackson 2014-03-20 13:53 ` Pasi Kärkkäinen 2014-03-20 19:32 ` Atom2 0 siblings, 2 replies; 23+ messages in thread From: Ian Jackson @ 2014-03-20 11:52 UTC (permalink / raw) To: Atom2; +Cc: xen-users, xen-devel, Ian Campbell, Roger Pau Monne Atom2 writes ("Re: [Xen-devel] [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough"): > the patch unfortunately doesn't apply to my sources - some comments to > the reasons why further below. Here's a backport. I have compiled but not executed it. > Sorry for my delay in answering - this is a resend as the first e-Mail > with uncompressed attachments did not go through. > Just FYI: the version I am using is 4.3.1-r5; I have attached the > relevant source files referred to by your patches. Thanks, but our revision control system enables us to retrieve old versions very easily :-). So there is not any need to provide us with these files. Having said that, I have no record of 4.3.1-rc5. But I'm pretty sure the patch below, which is against staging-4.3, will apply to your tree. It applies cleanly to 4.3.0-rc5 and 4.3.1-rc2, which are my two guesses as to which version you mean. Thanks, Ian. >From f9df128cd4d4ad6c7ed6ffd9bd8ba0633af78389 Mon Sep 17 00:00:00 2001 From: Ian Jackson <ian.jackson@eu.citrix.com> Date: Wed, 19 Mar 2014 15:47:02 +0000 Subject: [PATCH] libxl: Tolerate backend state "6" on pciback remove When shutting down a domain with pci passthrough, it can happen that the backend has actually shut down (xenbus state 6) before we try to remove it. When this happens, libxl would time out waiting for the backend to reach state 4. Instead, deal with this by having libxl__wait_for_backend take a list of suitable states. The arrangements are still fundamentally incorrect: - libxl__wait_for_backend is a slow synchronous function, which is forbidden; - There is no way to deal properly with the various xenbus states that might arise (including erroneous ones). We will hopefully fix this later, although it's not trivial. For the moment, rename the function to libxl__wait_for_backend_deprecated. Reported-by: Atom2 <ariel.atom2@web2web.at> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> CC: Atom2 <ariel.atom2@web2web.at> CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> CC: Roger Pau Monne <roger.pau@citrix.com> CC: Ian Campbell <Ian.Campbell@citrix.com> Backported to 4.3. Conflicts: tools/libxl/libxl_device.c tools/libxl/libxl_internal.h Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> --- tools/libxl/libxl.c | 2 +- tools/libxl/libxl_device.c | 21 ++++++++++++++------- tools/libxl/libxl_internal.h | 3 ++- tools/libxl/libxl_pci.c | 8 +++++--- 4 files changed, 22 insertions(+), 12 deletions(-) diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c index 3d9543b..c0cc0b7 100644 --- a/tools/libxl/libxl.c +++ b/tools/libxl/libxl.c @@ -2689,7 +2689,7 @@ static void local_device_attach_cb(libxl__egc *egc, libxl__ao_device *aodev) if (rc < 0) goto out; be_path = libxl__device_backend_path(gc, &device); - rc = libxl__wait_for_backend(gc, be_path, "4"); + rc = libxl__wait_for_backend_deprecated(gc, be_path, "4", (char*)0); if (rc < 0) goto out; diff --git a/tools/libxl/libxl_device.c b/tools/libxl/libxl_device.c index ea845b7..779b38b 100644 --- a/tools/libxl/libxl_device.c +++ b/tools/libxl/libxl_device.c @@ -1094,7 +1094,8 @@ int libxl__wait_for_device_model(libxl__gc *gc, check_callback, check_callback_userdata); } -int libxl__wait_for_backend(libxl__gc *gc, char *be_path, char *state) +int libxl__wait_for_backend_deprecated(libxl__gc *gc, const char *be_path, + ...) { libxl_ctx *ctx = libxl__gc_owner(gc); int watchdog = 100; @@ -1115,13 +1116,19 @@ int libxl__wait_for_backend(libxl__gc *gc, char *be_path, char *state) } goto out; } else { - if (!strcmp(p, state)) { - rc = 0; - goto out; - } else { - usleep(100000); - watchdog--; + const char *want; + va_list al; + va_start(al,be_path); + while ((want = va_arg(al, char*))) { + if (!strcmp(p, want)) { + va_end(al); + rc = 0; + goto out; + } } + va_end(al); + usleep(100000); + watchdog--; } } LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "Backend %s not ready", be_path); diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h index f051d91..4485c56 100644 --- a/tools/libxl/libxl_internal.h +++ b/tools/libxl/libxl_internal.h @@ -944,7 +944,8 @@ _hidden char *libxl__device_frontend_path(libxl__gc *gc, libxl__device *device); _hidden int libxl__parse_backend_path(libxl__gc *gc, const char *path, libxl__device *dev); _hidden int libxl__device_destroy(libxl__gc *gc, libxl__device *dev); -_hidden int libxl__wait_for_backend(libxl__gc *gc, char *be_path, char *state); +_hidden int libxl__wait_for_backend_deprecated(libxl__gc *gc, + const char *be_path, ...) __attribute__((sentinel)); _hidden int libxl__nic_type(libxl__gc *gc, libxl__device *dev, libxl_nic_type *nictype); diff --git a/tools/libxl/libxl_pci.c b/tools/libxl/libxl_pci.c index 0295e0b..e22852c 100644 --- a/tools/libxl/libxl_pci.c +++ b/tools/libxl/libxl_pci.c @@ -126,7 +126,7 @@ static int libxl__device_pci_add_xenstore(libxl__gc *gc, uint32_t domid, libxl_d return ERROR_FAIL; if (!starting && domtype == LIBXL_DOMAIN_TYPE_PV) { - if (libxl__wait_for_backend(gc, be_path, "4") < 0) + if (libxl__wait_for_backend_deprecated(gc, be_path, "4", (char*)0) < 0) return ERROR_FAIL; } @@ -169,7 +169,8 @@ static int libxl__device_pci_remove_xenstore(libxl__gc *gc, uint32_t domid, libx return ERROR_FAIL; if (domtype == LIBXL_DOMAIN_TYPE_PV) { - if (libxl__wait_for_backend(gc, be_path, "4") < 0) { + if (libxl__wait_for_backend_deprecated(gc, be_path, "4", "6", (char*)0) + < 0) { LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "pci backend at %s is not ready", be_path); return ERROR_FAIL; } @@ -198,7 +199,8 @@ retry_transaction: goto retry_transaction; if (domtype == LIBXL_DOMAIN_TYPE_PV) { - if (libxl__wait_for_backend(gc, be_path, "4") < 0) { + if (libxl__wait_for_backend_deprecated(gc, be_path, "4", "6", (char*)0) + < 0) { LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "pci backend at %s is not ready", be_path); return ERROR_FAIL; } -- 1.7.10.4 ^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough 2014-03-20 11:52 ` Ian Jackson @ 2014-03-20 13:53 ` Pasi Kärkkäinen 2014-03-20 15:28 ` Ian Jackson 2014-03-20 19:34 ` Atom2 2014-03-20 19:32 ` Atom2 1 sibling, 2 replies; 23+ messages in thread From: Pasi Kärkkäinen @ 2014-03-20 13:53 UTC (permalink / raw) To: Ian Jackson; +Cc: Roger Pau Monne, Ian Campbell, xen-devel, Atom2, xen-users On Thu, Mar 20, 2014 at 11:52:57AM +0000, Ian Jackson wrote: > Atom2 writes ("Re: [Xen-devel] [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough"): > > the patch unfortunately doesn't apply to my sources - some comments to > > the reasons why further below. > > Here's a backport. I have compiled but not executed it. > > > Sorry for my delay in answering - this is a resend as the first e-Mail > > with uncompressed attachments did not go through. > > > Just FYI: the version I am using is 4.3.1-r5; I have attached the > > relevant source files referred to by your patches. > > Thanks, but our revision control system enables us to retrieve old > versions very easily :-). So there is not any need to provide us with > these files. > > Having said that, I have no record of 4.3.1-rc5. But I'm pretty sure > the patch below, which is against staging-4.3, will apply to your > tree. It applies cleanly to 4.3.0-rc5 and 4.3.1-rc2, which are my two > guesses as to which version you mean. > Often "r5" style versions refer to Gentoo packaging.. So maybe he actually didn't have a typo in 4.3.1-r5 :) -- Pasi ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough 2014-03-20 13:53 ` Pasi Kärkkäinen @ 2014-03-20 15:28 ` Ian Jackson 2014-03-20 19:34 ` Atom2 1 sibling, 0 replies; 23+ messages in thread From: Ian Jackson @ 2014-03-20 15:28 UTC (permalink / raw) To: Pasi Kärkkäinen Cc: Roger Pau Monne, Ian Campbell, xen-devel, Atom2, xen-users Pasi Kärkkäinen writes ("Re: [Xen-devel] [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough"): > On Thu, Mar 20, 2014 at 11:52:57AM +0000, Ian Jackson wrote: ... > > Having said that, I have no record of 4.3.1-rc5. But I'm pretty sure > > the patch below, which is against staging-4.3, will apply to your > > tree. It applies cleanly to 4.3.0-rc5 and 4.3.1-rc2, which are my two > > guesses as to which version you mean. > > > > Often "r5" style versions refer to Gentoo packaging.. > So maybe he actually didn't have a typo in 4.3.1-r5 :) Ah. I didn't even spot the missing "c". But I bet my 4.3-based patch will apply. Ian. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough 2014-03-20 13:53 ` Pasi Kärkkäinen 2014-03-20 15:28 ` Ian Jackson @ 2014-03-20 19:34 ` Atom2 1 sibling, 0 replies; 23+ messages in thread From: Atom2 @ 2014-03-20 19:34 UTC (permalink / raw) To: Pasi Kärkkäinen, Ian Jackson Cc: Roger Pau Monne, xen-devel, Ian Campbell, xen-users Am 20.03.14 14:53, schrieb Pasi Kärkkäinen: > On Thu, Mar 20, 2014 at 11:52:57AM +0000, Ian Jackson wrote: >> Atom2 writes ("Re: [Xen-devel] [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough"): >>> the patch unfortunately doesn't apply to my sources - some comments to >>> the reasons why further below. >> >> Here's a backport. I have compiled but not executed it. >> >>> Sorry for my delay in answering - this is a resend as the first e-Mail >>> with uncompressed attachments did not go through. >> >>> Just FYI: the version I am using is 4.3.1-r5; I have attached the >>> relevant source files referred to by your patches. >> >> Thanks, but our revision control system enables us to retrieve old >> versions very easily :-). So there is not any need to provide us with >> these files. >> >> Having said that, I have no record of 4.3.1-rc5. But I'm pretty sure >> the patch below, which is against staging-4.3, will apply to your >> tree. It applies cleanly to 4.3.0-rc5 and 4.3.1-rc2, which are my two >> guesses as to which version you mean. >> > > Often "r5" style versions refer to Gentoo packaging.. > So maybe he actually didn't have a typo in 4.3.1-r5 :) Pasi - you were spot on: I use gentoo and the version number was not a typo. > > -- Pasi > ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough 2014-03-20 11:52 ` Ian Jackson 2014-03-20 13:53 ` Pasi Kärkkäinen @ 2014-03-20 19:32 ` Atom2 2014-03-21 18:11 ` Ian Jackson 1 sibling, 1 reply; 23+ messages in thread From: Atom2 @ 2014-03-20 19:32 UTC (permalink / raw) To: Ian Jackson; +Cc: xen-users, xen-devel, Ian Campbell, Roger Pau Monne [-- Attachment #1: Type: text/plain, Size: 8127 bytes --] Am 20.03.14 12:52, schrieb Ian Jackson: > Atom2 writes ("Re: [Xen-devel] [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough"): >> the patch unfortunately doesn't apply to my sources - some comments to >> the reasons why further below. > > Here's a backport. I have compiled but not executed it. It compiled at my end as well, but I am sorry to report that the problem with the 40s delay persits. Attached please find the new xl-output created with xl -vvv create -F domain. This file again contains a few annotations with regards to where the delays happen. To my untrained eye it looks largely identical to the last xl-output with the obvious change of domain-id, addresses, line-numbers for debug output where changes in the sourve have happende and the use of the new function libxl__wait_for_backend_deprecated instead of libxl__wait_for_backend due to your patch, the latter of which I take as proof that your patches have been applied. For other obvious changes I have commented in the file on those lines that I could identify: There are a few new lines which were not there last time and now there's also a new 10s delay which, however, is only visible through xl -vvvv -F domain as the prompt has already returned by the time this new delay happens and this delay therefore is not visible in dom0. I hope that helps. Thanks for your continued support and best regards, Atom2 > >> Sorry for my delay in answering - this is a resend as the first e-Mail >> with uncompressed attachments did not go through. > >> Just FYI: the version I am using is 4.3.1-r5; I have attached the >> relevant source files referred to by your patches. > > Thanks, but our revision control system enables us to retrieve old > versions very easily :-). So there is not any need to provide us with > these files. > > Having said that, I have no record of 4.3.1-rc5. But I'm pretty sure > the patch below, which is against staging-4.3, will apply to your > tree. It applies cleanly to 4.3.0-rc5 and 4.3.1-rc2, which are my two > guesses as to which version you mean. > > Thanks, > Ian. > > From f9df128cd4d4ad6c7ed6ffd9bd8ba0633af78389 Mon Sep 17 00:00:00 2001 > From: Ian Jackson <ian.jackson@eu.citrix.com> > Date: Wed, 19 Mar 2014 15:47:02 +0000 > Subject: [PATCH] libxl: Tolerate backend state "6" on pciback remove > > When shutting down a domain with pci passthrough, it can happen that > the backend has actually shut down (xenbus state 6) before we try to > remove it. When this happens, libxl would time out waiting for the > backend to reach state 4. > > Instead, deal with this by having libxl__wait_for_backend take a list > of suitable states. > > The arrangements are still fundamentally incorrect: > - libxl__wait_for_backend is a slow synchronous function, which is > forbidden; > - There is no way to deal properly with the various xenbus states > that might arise (including erroneous ones). > We will hopefully fix this later, although it's not trivial. For > the moment, rename the function to libxl__wait_for_backend_deprecated. > > Reported-by: Atom2 <ariel.atom2@web2web.at> > Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com> > CC: Atom2 <ariel.atom2@web2web.at> > CC: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> > CC: Roger Pau Monne <roger.pau@citrix.com> > CC: Ian Campbell <Ian.Campbell@citrix.com> > > Backported to 4.3. Conflicts: > tools/libxl/libxl_device.c > tools/libxl/libxl_internal.h > Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com> > --- > tools/libxl/libxl.c | 2 +- > tools/libxl/libxl_device.c | 21 ++++++++++++++------- > tools/libxl/libxl_internal.h | 3 ++- > tools/libxl/libxl_pci.c | 8 +++++--- > 4 files changed, 22 insertions(+), 12 deletions(-) > > diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c > index 3d9543b..c0cc0b7 100644 > --- a/tools/libxl/libxl.c > +++ b/tools/libxl/libxl.c > @@ -2689,7 +2689,7 @@ static void local_device_attach_cb(libxl__egc *egc, libxl__ao_device *aodev) > if (rc < 0) > goto out; > be_path = libxl__device_backend_path(gc, &device); > - rc = libxl__wait_for_backend(gc, be_path, "4"); > + rc = libxl__wait_for_backend_deprecated(gc, be_path, "4", (char*)0); > if (rc < 0) > goto out; > > diff --git a/tools/libxl/libxl_device.c b/tools/libxl/libxl_device.c > index ea845b7..779b38b 100644 > --- a/tools/libxl/libxl_device.c > +++ b/tools/libxl/libxl_device.c > @@ -1094,7 +1094,8 @@ int libxl__wait_for_device_model(libxl__gc *gc, > check_callback, check_callback_userdata); > } > > -int libxl__wait_for_backend(libxl__gc *gc, char *be_path, char *state) > +int libxl__wait_for_backend_deprecated(libxl__gc *gc, const char *be_path, > + ...) > { > libxl_ctx *ctx = libxl__gc_owner(gc); > int watchdog = 100; > @@ -1115,13 +1116,19 @@ int libxl__wait_for_backend(libxl__gc *gc, char *be_path, char *state) > } > goto out; > } else { > - if (!strcmp(p, state)) { > - rc = 0; > - goto out; > - } else { > - usleep(100000); > - watchdog--; > + const char *want; > + va_list al; > + va_start(al,be_path); > + while ((want = va_arg(al, char*))) { > + if (!strcmp(p, want)) { > + va_end(al); > + rc = 0; > + goto out; > + } > } > + va_end(al); > + usleep(100000); > + watchdog--; > } > } > LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "Backend %s not ready", be_path); > diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h > index f051d91..4485c56 100644 > --- a/tools/libxl/libxl_internal.h > +++ b/tools/libxl/libxl_internal.h > @@ -944,7 +944,8 @@ _hidden char *libxl__device_frontend_path(libxl__gc *gc, libxl__device *device); > _hidden int libxl__parse_backend_path(libxl__gc *gc, const char *path, > libxl__device *dev); > _hidden int libxl__device_destroy(libxl__gc *gc, libxl__device *dev); > -_hidden int libxl__wait_for_backend(libxl__gc *gc, char *be_path, char *state); > +_hidden int libxl__wait_for_backend_deprecated(libxl__gc *gc, > + const char *be_path, ...) __attribute__((sentinel)); > _hidden int libxl__nic_type(libxl__gc *gc, libxl__device *dev, > libxl_nic_type *nictype); > > diff --git a/tools/libxl/libxl_pci.c b/tools/libxl/libxl_pci.c > index 0295e0b..e22852c 100644 > --- a/tools/libxl/libxl_pci.c > +++ b/tools/libxl/libxl_pci.c > @@ -126,7 +126,7 @@ static int libxl__device_pci_add_xenstore(libxl__gc *gc, uint32_t domid, libxl_d > return ERROR_FAIL; > > if (!starting && domtype == LIBXL_DOMAIN_TYPE_PV) { > - if (libxl__wait_for_backend(gc, be_path, "4") < 0) > + if (libxl__wait_for_backend_deprecated(gc, be_path, "4", (char*)0) < 0) > return ERROR_FAIL; > } > > @@ -169,7 +169,8 @@ static int libxl__device_pci_remove_xenstore(libxl__gc *gc, uint32_t domid, libx > return ERROR_FAIL; > > if (domtype == LIBXL_DOMAIN_TYPE_PV) { > - if (libxl__wait_for_backend(gc, be_path, "4") < 0) { > + if (libxl__wait_for_backend_deprecated(gc, be_path, "4", "6", (char*)0) > + < 0) { > LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "pci backend at %s is not ready", be_path); > return ERROR_FAIL; > } > @@ -198,7 +199,8 @@ retry_transaction: > goto retry_transaction; > > if (domtype == LIBXL_DOMAIN_TYPE_PV) { > - if (libxl__wait_for_backend(gc, be_path, "4") < 0) { > + if (libxl__wait_for_backend_deprecated(gc, be_path, "4", "6", (char*)0) > + < 0) { > LIBXL__LOG(ctx, LIBXL__LOG_DEBUG, "pci backend at %s is not ready", be_path); > return ERROR_FAIL; > } > [-- Attachment #2: xl-output --] [-- Type: text/plain, Size: 22228 bytes --] Parsing config from 5:voip.9 libxl: debug: libxl_create.c:1230:do_domain_create: ao 0x7fe37ba02850: create: how=(nil) callback=(nil) poller=0x7fe37ba03d70 libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda1 spec.backend=unknown libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk vdev=xvda1, using backend phy libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda2 spec.backend=unknown libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk vdev=xvda2, using backend phy libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda3 spec.backend=unknown libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk vdev=xvda3, using backend phy libxl: debug: libxl_create.c:675:initiate_domain_create: running bootloader libxl: debug: libxl_bootloader.c:327:libxl__bootloader_run: no bootloader configured, using user supplied kernel libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7fe37ba02bd8: deregister unregistered libxl: debug: libxl_x86.c:82:e820_sanitize: Memory: 1048576kB End of RAM: 0x20000 (PFN) Delta: 524288kB, PCI start: 524288kB (0x20000 PFN), Balloon 0kB libxl: debug: libxl_x86.c:201:e820_sanitize: : [0 -> 20000] RAM libxl: debug: libxl_x86.c:201:e820_sanitize: : [20000 -> 20200] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [20200 -> 40000] Unusable libxl: debug: libxl_x86.c:201:e820_sanitize: : [40000 -> 40200] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [40200 -> db9f0] Unusable libxl: debug: libxl_x86.c:201:e820_sanitize: : [db9f0 -> dc0da] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [dc0da -> dc1f9] ACPI NVS libxl: debug: libxl_x86.c:201:e820_sanitize: : [dc1f9 -> dc651] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [dc651 -> dc652] Unusable libxl: debug: libxl_x86.c:201:e820_sanitize: : [dc652 -> dc695] ACPI NVS libxl: debug: libxl_x86.c:201:e820_sanitize: : [dc695 -> dcdba] Unusable libxl: debug: libxl_x86.c:201:e820_sanitize: : [dcdba -> dcff2] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [dcff2 -> dd000] Unusable libxl: debug: libxl_x86.c:201:e820_sanitize: : [dd800 -> dfa00] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [f8000 -> fc000] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [fec00 -> fec01] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [fed00 -> fed04] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [fed1c -> fed20] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [fee00 -> fee01] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [fee01 -> fef00] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [ff000 -> 100000] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [100000 -> 120000] RAM domainbuilder: detail: xc_dom_allocate: cmdline="", features="(null)" libxl: debug: libxl_dom.c:341:libxl__build_pv: pv kernel mapped 0 path /usr/lib/xen/boot/pv-grub-x86_64.gz domainbuilder: detail: xc_dom_kernel_file: filename="/usr/lib/xen/boot/pv-grub-x86_64.gz" domainbuilder: detail: xc_dom_malloc_filemap : 1240 kB domainbuilder: detail: xc_dom_malloc : 15110 kB domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x13631e -> 0xec1ae1 domainbuilder: detail: xc_dom_ramdisk_file: filename="/etc/xen/guests/grub.d/voip.grub" domainbuilder: detail: xc_dom_boot_xen_init: ver 4.3, caps xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 domainbuilder: detail: xc_dom_parse_image: called domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ... domainbuilder: detail: loader probe failed domainbuilder: detail: xc_dom_find_loader: trying Linux bzImage loader ... domainbuilder: detail: xc_dom_probe_bzimage_kernel: kernel is not a bzImage domainbuilder: detail: loader probe failed domainbuilder: detail: xc_dom_find_loader: trying ELF-generic loader ... domainbuilder: detail: loader probe OK xc: detail: elf_parse_binary: phdr: paddr=0x0 memsz=0x9a96e0 xc: detail: elf_parse_binary: memory: 0x0 -> 0x9a96e0 xc: detail: elf_xen_parse: __xen_guest: "GUEST_OS=Mini-OS,XEN_VER=xen-3.0,VIRT_BASE=0x0,ELF_PADDR_OFFSET=0x0,HYPERCALL_PAGE=0x2,LOADER=generic" xc: detail: elf_xen_parse_guest_info: GUEST_OS="Mini-OS" xc: detail: elf_xen_parse_guest_info: XEN_VER="xen-3.0" xc: detail: elf_xen_parse_guest_info: VIRT_BASE="0x0" xc: detail: elf_xen_parse_guest_info: ELF_PADDR_OFFSET="0x0" xc: detail: elf_xen_parse_guest_info: HYPERCALL_PAGE="0x2" xc: detail: elf_xen_parse_guest_info: LOADER="generic" xc: detail: elf_xen_addr_calc_check: addresses: xc: detail: virt_base = 0x0 xc: detail: elf_paddr_offset = 0x0 xc: detail: virt_offset = 0x0 xc: detail: virt_kstart = 0x0 xc: detail: virt_kend = 0x9a96e0 xc: detail: virt_entry = 0x0 xc: detail: p2m_base = 0xffffffffffffffff domainbuilder: detail: xc_dom_parse_elf_kernel: xen-3.0-x86_64: 0x0 -> 0x9a96e0 domainbuilder: detail: xc_dom_mem_init: mem 1024 MB, pages 0x40000 pages, 4k each domainbuilder: detail: xc_dom_mem_init: 0x40000 pages domainbuilder: detail: xc_dom_boot_mem_init: called domainbuilder: detail: x86_compat: guest xen-3.0-x86_64, address size 64 domainbuilder: detail: xc_dom_malloc : 2048 kB domainbuilder: detail: xc_dom_build_image: called domainbuilder: detail: xc_dom_alloc_segment: kernel : 0x0 -> 0x9aa000 (pfn 0x0 + 0x9aa pages) domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x0+0x9aa at 0x7fe377817000 xc: detail: elf_load_binary: phdr 0 at 0x7fe377817000 -> 0x7fe3781c06e0 domainbuilder: detail: xc_dom_alloc_segment: ramdisk : 0x9aa000 -> 0x9ab000 (pfn 0x9aa + 0x1 pages) domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x9aa+0x1 at 0x7fe37b7ce000 domainbuilder: detail: xc_dom_alloc_segment: phys2mach : 0x9ab000 -> 0xbab000 (pfn 0x9ab + 0x200 pages) domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x9ab+0x200 at 0x7fe377617000 domainbuilder: detail: xc_dom_alloc_page : start info : 0xbab000 (pfn 0xbab) domainbuilder: detail: xc_dom_alloc_page : xenstore : 0xbac000 (pfn 0xbac) domainbuilder: detail: xc_dom_alloc_page : console : 0xbad000 (pfn 0xbad) domainbuilder: detail: nr_page_tables: 0x0000ffffffffffff/48: 0x0000000000000000 -> 0x0000ffffffffffff, 1 table(s) domainbuilder: detail: nr_page_tables: 0x0000007fffffffff/39: 0x0000000000000000 -> 0x0000007fffffffff, 1 table(s) domainbuilder: detail: nr_page_tables: 0x000000003fffffff/30: 0x0000000000000000 -> 0x000000003fffffff, 1 table(s) domainbuilder: detail: nr_page_tables: 0x00000000001fffff/21: 0x0000000000000000 -> 0x0000000000ffffff, 8 table(s) domainbuilder: detail: xc_dom_alloc_segment: page tables : 0xbae000 -> 0xbb9000 (pfn 0xbae + 0xb pages) domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0xbae+0xb at 0x7fe37b680000 domainbuilder: detail: xc_dom_alloc_page : boot stack : 0xbb9000 (pfn 0xbb9) domainbuilder: detail: xc_dom_build_image : virt_alloc_end : 0xbba000 domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0x1000000 domainbuilder: detail: xc_dom_boot_image: called domainbuilder: detail: arch_setup_bootearly: doing nothing domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_64 <= matches domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_32p domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32 domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32p domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_64 domainbuilder: detail: xc_dom_update_guest_p2m: dst 64bit, pages 0x40000 domainbuilder: detail: clear_page: pfn 0xbad, mfn 0x71aabe domainbuilder: detail: clear_page: pfn 0xbac, mfn 0x71aabf domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0xbab+0x1 at 0x7fe37b7cb000 domainbuilder: detail: start_info_x86_64: called domainbuilder: detail: setup_hypercall_page: vaddr=0x2000 pfn=0x2 domainbuilder: detail: domain builder memory footprint domainbuilder: detail: allocated domainbuilder: detail: malloc : 17231 kB domainbuilder: detail: anon mmap : 0 bytes domainbuilder: detail: mapped domainbuilder: detail: file mmap : 1241 kB domainbuilder: detail: domU mmap : 11996 kB domainbuilder: detail: arch_setup_bootlate: shared_info: pfn 0x0, mfn 0xdb70f domainbuilder: detail: shared_info_x86_64: called domainbuilder: detail: vcpu_x86_64: called domainbuilder: detail: vcpu_x86_64: cr3: pfn 0xbae mfn 0x71aabd domainbuilder: detail: launch_vm: called, ctxt=0x7fff80e7a330 domainbuilder: detail: xc_dom_release: called libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda1 spec.backend=phy libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7fe37ba04098 wpath=/local/domain/0/backend/vbd/4/51713/state token=3/0: register slotnum=3 libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda2 spec.backend=phy libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda2 spec.backend=phy libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7fe37ba050b8 wpath=/local/domain/0/backend/vbd/4/51714/state token=2/1: register slotnum=2 libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda3 spec.backend=phy libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda3 spec.backend=phy libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7fe37ba048a8 wpath=/local/domain/0/backend/vbd/4/51715/state token=1/2: register slotnum=1 libxl: debug: libxl_create.c:1243:do_domain_create: ao 0x7fe37ba02850: inprogress: poller=0x7fe37ba03d70, flags=i libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fe37ba04098 wpath=/local/domain/0/backend/vbd/4/51713/state token=3/0: event epath=/local/domain/0/backend/vbd/4/51713/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vbd/4/51713/state wanted state 2 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7fe37ba04098 wpath=/local/domain/0/backend/vbd/4/51713/state token=3/0: deregister slotnum=3 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7fe37ba04098: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/block add libxl: debug: libxl_event.c:472:watchfd_callback: watch epath=/local/domain/0/backend/vbd/4/51713/state token=3/0: empty slot libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fe37ba050b8 wpath=/local/domain/0/backend/vbd/4/51714/state token=2/1: event epath=/local/domain/0/backend/vbd/4/51714/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vbd/4/51714/state wanted state 2 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7fe37ba050b8 wpath=/local/domain/0/backend/vbd/4/51714/state token=2/1: deregister slotnum=2 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7fe37ba050b8: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/block add libxl: debug: libxl_event.c:472:watchfd_callback: watch epath=/local/domain/0/backend/vbd/4/51714/state token=2/1: empty slot libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fe37ba048a8 wpath=/local/domain/0/backend/vbd/4/51715/state token=1/2: event epath=/local/domain/0/backend/vbd/4/51715/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vbd/4/51715/state wanted state 2 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7fe37ba048a8 wpath=/local/domain/0/backend/vbd/4/51715/state token=1/2: deregister slotnum=1 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7fe37ba048a8: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/block add libxl: debug: libxl_event.c:472:watchfd_callback: watch epath=/local/domain/0/backend/vbd/4/51715/state token=1/2: empty slot libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7fe37ba091e8 wpath=/local/domain/0/backend/vif/4/0/state token=1/3: register slotnum=1 libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fe37ba091e8 wpath=/local/domain/0/backend/vif/4/0/state token=1/3: event epath=/local/domain/0/backend/vif/4/0/state libxl: debug: libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vif/4/0/state wanted state 2 still waiting state 1 libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fe37ba091e8 wpath=/local/domain/0/backend/vif/4/0/state token=1/3: event epath=/local/domain/0/backend/vif/4/0/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vif/4/0/state wanted state 2 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7fe37ba091e8 wpath=/local/domain/0/backend/vif/4/0/state token=1/3: deregister slotnum=1 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7fe37ba091e8: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/vif-bridge online libxl: error: libxl_pci.c:992:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:09:02.0 libxl: debug: libxl_pci.c:81:libxl__create_pci_backend: Creating pci backend libxl: debug: libxl_event.c:1737:libxl__ao_progress_report: ao 0x7fe37ba02850: progress report: ignored libxl: debug: libxl_event.c:1569:libxl__ao_complete: ao 0x7fe37ba02850: complete, rc=0 libxl: debug: libxl_event.c:1541:libxl__ao__destroy: ao 0x7fe37ba02850: destroy Waiting for domain voip (domid 4) to die [pid 2409] libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7fe37ba007e0 wpath=@releaseDomain token=1/4: register slotnum=1 libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fe37ba007e0 wpath=@releaseDomain token=1/4: event epath=@releaseDomain libxl: debug: libxl.c:1000:domain_death_xswatch_callback: [evg=0x7fe37ba08900:4] from domid=4 nentries=1 rc=1 libxl: debug: libxl.c:1011:domain_death_xswatch_callback: [evg=0x7fe37ba08900:4] got=domaininfos[0] got->domain=4 libxl: debug: libxl.c:1038:domain_death_xswatch_callback: exists shutdown_reported=0 dominf.flags=ffff0020 libxl: debug: libxl.c:1004:domain_death_xswatch_callback: [evg=0] all reported libxl: debug: libxl.c:1068:domain_death_xswatch_callback: domain death search done <NOTE: this is after the start - no further output until shutdown -h now from ssh connection> Domain 4 has shut down, reason code 0 0x0 Action for shutdown reason code 0 is destroy Domain 4 needs to be cleaned up: destroying the domain libxl: debug: libxl.c:1252:libxl_domain_destroy: ao 0x7fe37ba02850: create: how=(nil) callback=(nil) poller=0x7fe37ba03d70 libxl: error: libxl_pci.c:1250:do_pci_remove: xc_domain_irq_permission irq=17 <NOTE: at this point a 10s pause happens> libxl: error: libxl_device.c:1134:libxl__wait_for_backend_deprecated: Backend /local/domain/0/backend/pci/4/0 not ready libxl: debug: libxl_pci.c:204:libxl__device_pci_remove_xenstore: pci backend at /local/domain/0/backend/pci/4/0 is not ready libxl: error: libxl_pci.c:1250:do_pci_remove: xc_domain_irq_permission irq=16 <NOTE: at this point a 10s pause happens> libxl: error: libxl_device.c:1134:libxl__wait_for_backend_deprecated: Backend /local/domain/0/backend/pci/4/0 not ready libxl: debug: libxl_pci.c:174:libxl__device_pci_remove_xenstore: pci backend at /local/domain/0/backend/pci/4/0 is not ready libxl: error: libxl_pci.c:1250:do_pci_remove: xc_domain_irq_permission irq=18 libxl: error: libxl_pci.c:992:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:09:02.0 <NOTE: at this point a 10s pause happens> libxl: error: libxl_device.c:1134:libxl__wait_for_backend_deprecated: Backend /local/domain/0/backend/pci/4/0 not ready libxl: debug: libxl_pci.c:174:libxl__device_pci_remove_xenstore: pci backend at /local/domain/0/backend/pci/4/0 is not ready libxl: error: libxl_pci.c:1250:do_pci_remove: xc_domain_irq_permission irq=23 <NOTE: at this point a 10s pause happens> libxl: error: libxl_device.c:1134:libxl__wait_for_backend_deprecated: Backend /local/domain/0/backend/pci/4/0 not ready libxl: debug: libxl_pci.c:174:libxl__device_pci_remove_xenstore: pci backend at /local/domain/0/backend/pci/4/0 is not ready libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7fe37ba048a8 wpath=/local/domain/0/backend/vbd/4/51713/state token=2/5: register slotnum=2 libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7fe37ba091e8 wpath=/local/domain/0/backend/vbd/4/51714/state token=3/6: register slotnum=3 libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7fe37ba0c298 wpath=/local/domain/0/backend/vbd/4/51715/state token=0/7: register slotnum=0 libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7fe37ba043c8 wpath=/local/domain/0/backend/vif/4/0/state token=19/8: register slotnum=19 libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7fe37ba05408 wpath=/local/domain/0/backend/pci/4/0/state token=18/9: register slotnum=18 libxl: debug: libxl.c:1261:libxl_domain_destroy: ao 0x7fe37ba02850: inprogress: poller=0x7fe37ba03d70, flags=i libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fe37ba048a8 wpath=/local/domain/0/backend/vbd/4/51713/state token=2/5: event epath=/local/domain/0/backend/vbd/4/51713/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vbd/4/51713/state wanted state 6 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7fe37ba048a8 wpath=/local/domain/0/backend/vbd/4/51713/state token=2/5: deregister slotnum=2 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7fe37ba048a8: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/block remove libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fe37ba091e8 wpath=/local/domain/0/backend/vbd/4/51714/state token=3/6: event epath=/local/domain/0/backend/vbd/4/51714/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vbd/4/51714/state wanted state 6 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7fe37ba091e8 wpath=/local/domain/0/backend/vbd/4/51714/state token=3/6: deregister slotnum=3 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7fe37ba091e8: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/block remove libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fe37ba0c298 wpath=/local/domain/0/backend/vbd/4/51715/state token=0/7: event epath=/local/domain/0/backend/vbd/4/51715/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vbd/4/51715/state wanted state 6 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7fe37ba0c298 wpath=/local/domain/0/backend/vbd/4/51715/state token=0/7: deregister slotnum=0 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7fe37ba0c298: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/block remove libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fe37ba043c8 wpath=/local/domain/0/backend/vif/4/0/state token=19/8: event epath=/local/domain/0/backend/vif/4/0/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vif/4/0/state wanted state 6 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7fe37ba043c8 wpath=/local/domain/0/backend/vif/4/0/state token=19/8: deregister slotnum=19 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7fe37ba043c8: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/vif-bridge offline libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fe37ba05408 wpath=/local/domain/0/backend/pci/4/0/state token=18/9: event epath=/local/domain/0/backend/pci/4/0/state libxl: debug: libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/pci/4/0/state wanted state 6 still waiting state 5 <NOTE1: the above line is new compared to the old output file> <NOTE2: at this point a new 10s delay happens - which is only visible here as the prompt in dom0 has already returned> <NOTE3: the next line differs from the old output file> libxl: debug: libxl_event.c:661:devstate_timeout: backend /local/domain/0/backend/pci/4/0/state wanted state 6 timed out libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7fe37ba05408 wpath=/local/domain/0/backend/pci/4/0/state token=18/9: deregister slotnum=18 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7fe37ba05408: deregister unregistered <NOTE: the following 2 lines are new compared to the old output file> libxl: error: libxl_device.c:894:device_backend_callback: unable to remove device with path /local/domain/0/backend/pci/4/0 libxl: error: libxl.c:1452:devices_destroy_cb: libxl__devices_destroy failed for 4 libxl: debug: libxl_event.c:1569:libxl__ao_complete: ao 0x7fe37ba02850: complete, rc=0 libxl: debug: libxl_event.c:1541:libxl__ao__destroy: ao 0x7fe37ba02850: destroy Done. Exiting now libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7fe37ba007e0 wpath=@releaseDomain token=1/4: deregister slotnum=1 xc: debug: hypercall buffer: total allocations:554 total releases:554 xc: debug: hypercall buffer: current allocations:0 maximum allocations:2 xc: debug: hypercall buffer: cache current size:2 xc: debug: hypercall buffer: cache hits:548 misses:2 toobig:4 [-- Attachment #3: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough 2014-03-20 19:32 ` Atom2 @ 2014-03-21 18:11 ` Ian Jackson 2014-03-21 19:39 ` Atom2 0 siblings, 1 reply; 23+ messages in thread From: Ian Jackson @ 2014-03-21 18:11 UTC (permalink / raw) To: Atom2; +Cc: xen-users, xen-devel, Ian Campbell, Roger Pau Monne Atom2 writes ("Re: [Xen-devel] [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough"): > This file again contains a few annotations with regards to where the > delays happen. To my untrained eye it looks largely identical to the > last xl-output with the obvious change of domain-id, addresses, > line-numbers for debug output where changes in the sourve have happende > and the use of the new function libxl__wait_for_backend_deprecated > instead of libxl__wait_for_backend due to your patch, the latter of > which I take as proof that your patches have been applied. Thanks. I'm puzzled now. I have another tiny patch to suggest, which won't fix the problem but will produce more debugging output. Can you run it again with this, on top of the previous patch, please ? Thanks, Ian. diff --git a/tools/libxl/libxl_device.c b/tools/libxl/libxl_device.c index 779b38b..eff452b 100644 --- a/tools/libxl/libxl_device.c +++ b/tools/libxl/libxl_device.c @@ -1131,7 +1131,13 @@ int libxl__wait_for_backend_deprecated(libxl__gc *gc, const char *be_path, watchdog--; } } - LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "Backend %s not ready", be_path); + LOG(ERROR, "Backend %s not ready (state %s)", be_path, p); +{ +const char *fe = libxl__xs_read(gc,0, GCSPRINTF("%s/frontend", be_path)); +const char *fe_state = !fe ? 0 : libxl__xs_read(gc,0, GCSPRINTF("%s/state", fe)); +LOG(ERROR, "FE %s state %s", fe, fe_state); +} + out: return rc; } ^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough 2014-03-21 18:11 ` Ian Jackson @ 2014-03-21 19:39 ` Atom2 2014-04-02 14:44 ` Ian Jackson 0 siblings, 1 reply; 23+ messages in thread From: Atom2 @ 2014-03-21 19:39 UTC (permalink / raw) To: Ian Jackson; +Cc: xen-users, xen-devel, Ian Campbell, Roger Pau Monne [-- Attachment #1: Type: text/plain, Size: 1138 bytes --] Am 21.03.14 19:11, schrieb Ian Jackson: > Thanks. I'm puzzled now. I have another tiny patch to suggest, > which won't fix the problem but will produce more debugging output. Many thanks for the new patch. > > Can you run it again with this, on top of the previous patch, please ? Sure, the new output of xl -vvv create -F domain is again attached to this e-Mail. Thanks again, Atom2 > > Thanks, > Ian. > > diff --git a/tools/libxl/libxl_device.c b/tools/libxl/libxl_device.c > index 779b38b..eff452b 100644 > --- a/tools/libxl/libxl_device.c > +++ b/tools/libxl/libxl_device.c > @@ -1131,7 +1131,13 @@ int libxl__wait_for_backend_deprecated(libxl__gc *gc, const char *be_path, > watchdog--; > } > } > - LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "Backend %s not ready", be_path); > + LOG(ERROR, "Backend %s not ready (state %s)", be_path, p); > +{ > +const char *fe = libxl__xs_read(gc,0, GCSPRINTF("%s/frontend", be_path)); > +const char *fe_state = !fe ? 0 : libxl__xs_read(gc,0, GCSPRINTF("%s/state", fe)); > +LOG(ERROR, "FE %s state %s", fe, fe_state); > +} > + > out: > return rc; > } > > [-- Attachment #2: xl-output --] [-- Type: text/plain, Size: 22450 bytes --] libxl: debug: libxl_create.c:1230:do_domain_create: ao 0x7feea6a15850: create: how=(nil) callback=(nil) poller=0x7feea6a16d70 libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda1 spec.backend=unknown libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk vdev=xvda1, using backend phy libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda2 spec.backend=unknown libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk vdev=xvda2, using backend phy libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda3 spec.backend=unknown libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk vdev=xvda3, using backend phy libxl: debug: libxl_create.c:675:initiate_domain_create: running bootloader libxl: debug: libxl_bootloader.c:327:libxl__bootloader_run: no bootloader configured, using user supplied kernel libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7feea6a15bd8: deregister unregistered libxl: debug: libxl_x86.c:82:e820_sanitize: Memory: 1048576kB End of RAM: 0x20000 (PFN) Delta: 524288kB, PCI start: 524288kB (0x20000 PFN), Balloon 0kB libxl: debug: libxl_x86.c:201:e820_sanitize: : [0 -> 20000] RAM libxl: debug: libxl_x86.c:201:e820_sanitize: : [20000 -> 20200] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [20200 -> 40000] Unusable libxl: debug: libxl_x86.c:201:e820_sanitize: : [40000 -> 40200] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [40200 -> db9f0] Unusable libxl: debug: libxl_x86.c:201:e820_sanitize: : [db9f0 -> dc0da] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [dc0da -> dc1f9] ACPI NVS libxl: debug: libxl_x86.c:201:e820_sanitize: : [dc1f9 -> dc651] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [dc651 -> dc652] Unusable libxl: debug: libxl_x86.c:201:e820_sanitize: : [dc652 -> dc695] ACPI NVS libxl: debug: libxl_x86.c:201:e820_sanitize: : [dc695 -> dcdba] Unusable libxl: debug: libxl_x86.c:201:e820_sanitize: : [dcdba -> dcff2] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [dcff2 -> dd000] Unusable libxl: debug: libxl_x86.c:201:e820_sanitize: : [dd800 -> dfa00] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [f8000 -> fc000] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [fec00 -> fec01] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [fed00 -> fed04] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [fed1c -> fed20] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [fee00 -> fee01] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [fee01 -> fef00] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [ff000 -> 100000] Reserved libxl: debug: libxl_x86.c:201:e820_sanitize: : [100000 -> 120000] RAM domainbuilder: detail: xc_dom_allocate: cmdline="", features="(null)" libxl: debug: libxl_dom.c:341:libxl__build_pv: pv kernel mapped 0 path /usr/lib/xen/boot/pv-grub-x86_64.gz domainbuilder: detail: xc_dom_kernel_file: filename="/usr/lib/xen/boot/pv-grub-x86_64.gz" domainbuilder: detail: xc_dom_malloc_filemap : 1240 kB domainbuilder: detail: xc_dom_malloc : 15110 kB domainbuilder: detail: xc_dom_do_gunzip: unzip ok, 0x13631e -> 0xec1ae1 domainbuilder: detail: xc_dom_ramdisk_file: filename="/etc/xen/guests/grub.d/voip.grub" domainbuilder: detail: xc_dom_boot_xen_init: ver 4.3, caps xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 domainbuilder: detail: xc_dom_parse_image: called domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ... domainbuilder: detail: loader probe failed domainbuilder: detail: xc_dom_find_loader: trying Linux bzImage loader ... domainbuilder: detail: xc_dom_probe_bzimage_kernel: kernel is not a bzImage domainbuilder: detail: loader probe failed domainbuilder: detail: xc_dom_find_loader: trying ELF-generic loader ... domainbuilder: detail: loader probe OK xc: detail: elf_parse_binary: phdr: paddr=0x0 memsz=0x9a96e0 xc: detail: elf_parse_binary: memory: 0x0 -> 0x9a96e0 xc: detail: elf_xen_parse: __xen_guest: "GUEST_OS=Mini-OS,XEN_VER=xen-3.0,VIRT_BASE=0x0,ELF_PADDR_OFFSET=0x0,HYPERCALL_PAGE=0x2,LOADER=generic" xc: detail: elf_xen_parse_guest_info: GUEST_OS="Mini-OS" xc: detail: elf_xen_parse_guest_info: XEN_VER="xen-3.0" xc: detail: elf_xen_parse_guest_info: VIRT_BASE="0x0" xc: detail: elf_xen_parse_guest_info: ELF_PADDR_OFFSET="0x0" xc: detail: elf_xen_parse_guest_info: HYPERCALL_PAGE="0x2" xc: detail: elf_xen_parse_guest_info: LOADER="generic" xc: detail: elf_xen_addr_calc_check: addresses: xc: detail: virt_base = 0x0 xc: detail: elf_paddr_offset = 0x0 xc: detail: virt_offset = 0x0 xc: detail: virt_kstart = 0x0 xc: detail: virt_kend = 0x9a96e0 xc: detail: virt_entry = 0x0 xc: detail: p2m_base = 0xffffffffffffffff domainbuilder: detail: xc_dom_parse_elf_kernel: xen-3.0-x86_64: 0x0 -> 0x9a96e0 domainbuilder: detail: xc_dom_mem_init: mem 1024 MB, pages 0x40000 pages, 4k each domainbuilder: detail: xc_dom_mem_init: 0x40000 pages domainbuilder: detail: xc_dom_boot_mem_init: called domainbuilder: detail: x86_compat: guest xen-3.0-x86_64, address size 64 domainbuilder: detail: xc_dom_malloc : 2048 kB domainbuilder: detail: xc_dom_build_image: called domainbuilder: detail: xc_dom_alloc_segment: kernel : 0x0 -> 0x9aa000 (pfn 0x0 + 0x9aa pages) domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x0+0x9aa at 0x7feea282a000 xc: detail: elf_load_binary: phdr 0 at 0x7feea282a000 -> 0x7feea31d36e0 domainbuilder: detail: xc_dom_alloc_segment: ramdisk : 0x9aa000 -> 0x9ab000 (pfn 0x9aa + 0x1 pages) domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x9aa+0x1 at 0x7feea67e1000 domainbuilder: detail: xc_dom_alloc_segment: phys2mach : 0x9ab000 -> 0xbab000 (pfn 0x9ab + 0x200 pages) domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x9ab+0x200 at 0x7feea262a000 domainbuilder: detail: xc_dom_alloc_page : start info : 0xbab000 (pfn 0xbab) domainbuilder: detail: xc_dom_alloc_page : xenstore : 0xbac000 (pfn 0xbac) domainbuilder: detail: xc_dom_alloc_page : console : 0xbad000 (pfn 0xbad) domainbuilder: detail: nr_page_tables: 0x0000ffffffffffff/48: 0x0000000000000000 -> 0x0000ffffffffffff, 1 table(s) domainbuilder: detail: nr_page_tables: 0x0000007fffffffff/39: 0x0000000000000000 -> 0x0000007fffffffff, 1 table(s) domainbuilder: detail: nr_page_tables: 0x000000003fffffff/30: 0x0000000000000000 -> 0x000000003fffffff, 1 table(s) domainbuilder: detail: nr_page_tables: 0x00000000001fffff/21: 0x0000000000000000 -> 0x0000000000ffffff, 8 table(s) domainbuilder: detail: xc_dom_alloc_segment: page tables : 0xbae000 -> 0xbb9000 (pfn 0xbae + 0xb pages) domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0xbae+0xb at 0x7feea6693000 domainbuilder: detail: xc_dom_alloc_page : boot stack : 0xbb9000 (pfn 0xbb9) domainbuilder: detail: xc_dom_build_image : virt_alloc_end : 0xbba000 domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0x1000000 domainbuilder: detail: xc_dom_boot_image: called domainbuilder: detail: arch_setup_bootearly: doing nothing domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_64 <= matches domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_32p domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32 domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32p domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_64 domainbuilder: detail: xc_dom_update_guest_p2m: dst 64bit, pages 0x40000 domainbuilder: detail: clear_page: pfn 0xbad, mfn 0x7533d8 domainbuilder: detail: clear_page: pfn 0xbac, mfn 0x7533d9 domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0xbab+0x1 at 0x7feea67de000 domainbuilder: detail: start_info_x86_64: called domainbuilder: detail: setup_hypercall_page: vaddr=0x2000 pfn=0x2 domainbuilder: detail: domain builder memory footprint domainbuilder: detail: allocated domainbuilder: detail: malloc : 17231 kB domainbuilder: detail: anon mmap : 0 bytes domainbuilder: detail: mapped domainbuilder: detail: file mmap : 1241 kB domainbuilder: detail: domU mmap : 11996 kB domainbuilder: detail: arch_setup_bootlate: shared_info: pfn 0x0, mfn 0xdb9e8 domainbuilder: detail: shared_info_x86_64: called domainbuilder: detail: vcpu_x86_64: called domainbuilder: detail: vcpu_x86_64: cr3: pfn 0xbae mfn 0x7533d7 domainbuilder: detail: launch_vm: called, ctxt=0x7fffadabe320 domainbuilder: detail: xc_dom_release: called libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda1 spec.backend=phy libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7feea6a17098 wpath=/local/domain/0/backend/vbd/4/51713/state token=3/0: register slotnum=3 libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda2 spec.backend=phy libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda2 spec.backend=phy libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7feea6a180b8 wpath=/local/domain/0/backend/vbd/4/51714/state token=2/1: register slotnum=2 libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda3 spec.backend=phy libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=xvda3 spec.backend=phy libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7feea6a178a8 wpath=/local/domain/0/backend/vbd/4/51715/state token=1/2: register slotnum=1 libxl: debug: libxl_create.c:1243:do_domain_create: ao 0x7feea6a15850: inprogress: poller=0x7feea6a16d70, flags=i libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7feea6a17098 wpath=/local/domain/0/backend/vbd/4/51713/state token=3/0: event epath=/local/domain/0/backend/vbd/4/51713/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vbd/4/51713/state wanted state 2 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7feea6a17098 wpath=/local/domain/0/backend/vbd/4/51713/state token=3/0: deregister slotnum=3 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7feea6a17098: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/block add libxl: debug: libxl_event.c:472:watchfd_callback: watch epath=/local/domain/0/backend/vbd/4/51713/state token=3/0: empty slot libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7feea6a180b8 wpath=/local/domain/0/backend/vbd/4/51714/state token=2/1: event epath=/local/domain/0/backend/vbd/4/51714/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vbd/4/51714/state wanted state 2 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7feea6a180b8 wpath=/local/domain/0/backend/vbd/4/51714/state token=2/1: deregister slotnum=2 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7feea6a180b8: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/block add libxl: debug: libxl_event.c:472:watchfd_callback: watch epath=/local/domain/0/backend/vbd/4/51714/state token=2/1: empty slot libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7feea6a178a8 wpath=/local/domain/0/backend/vbd/4/51715/state token=1/2: event epath=/local/domain/0/backend/vbd/4/51715/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vbd/4/51715/state wanted state 2 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7feea6a178a8 wpath=/local/domain/0/backend/vbd/4/51715/state token=1/2: deregister slotnum=1 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7feea6a178a8: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/block add libxl: debug: libxl_event.c:472:watchfd_callback: watch epath=/local/domain/0/backend/vbd/4/51715/state token=1/2: empty slot libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7feea6a1c1e8 wpath=/local/domain/0/backend/vif/4/0/state token=1/3: register slotnum=1 libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7feea6a1c1e8 wpath=/local/domain/0/backend/vif/4/0/state token=1/3: event epath=/local/domain/0/backend/vif/4/0/state libxl: debug: libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vif/4/0/state wanted state 2 still waiting state 1 libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7feea6a1c1e8 wpath=/local/domain/0/backend/vif/4/0/state token=1/3: event epath=/local/domain/0/backend/vif/4/0/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vif/4/0/state wanted state 2 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7feea6a1c1e8 wpath=/local/domain/0/backend/vif/4/0/state token=1/3: deregister slotnum=1 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7feea6a1c1e8: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/vif-bridge online libxl: error: libxl_pci.c:992:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:09:02.0 libxl: debug: libxl_pci.c:81:libxl__create_pci_backend: Creating pci backend libxl: debug: libxl_event.c:1737:libxl__ao_progress_report: ao 0x7feea6a15850: progress report: ignored libxl: debug: libxl_event.c:1569:libxl__ao_complete: ao 0x7feea6a15850: complete, rc=0 libxl: debug: libxl_event.c:1541:libxl__ao__destroy: ao 0x7feea6a15850: destroy Waiting for domain voip (domid 4) to die [pid 7532] libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7feea6a137e0 wpath=@releaseDomain token=1/4: register slotnum=1 libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7feea6a137e0 wpath=@releaseDomain token=1/4: event epath=@releaseDomain libxl: debug: libxl.c:1000:domain_death_xswatch_callback: [evg=0x7feea6a1b900:4] from domid=4 nentries=1 rc=1 libxl: debug: libxl.c:1011:domain_death_xswatch_callback: [evg=0x7feea6a1b900:4] got=domaininfos[0] got->domain=4 libxl: debug: libxl.c:1038:domain_death_xswatch_callback: exists shutdown_reported=0 dominf.flags=ffff0020 libxl: debug: libxl.c:1004:domain_death_xswatch_callback: [evg=0] all reported libxl: debug: libxl.c:1068:domain_death_xswatch_callback: domain death search done <NOTE: this is after the start - no further output until shutdown -h now from ssh connection> Domain 4 has shut down, reason code 0 0x0 Action for shutdown reason code 0 is destroy Domain 4 needs to be cleaned up: destroying the domain libxl: debug: libxl.c:1252:libxl_domain_destroy: ao 0x7feea6a15850: create: how=(nil) callback=(nil) poller=0x7feea6a16d70 libxl: error: libxl_pci.c:1250:do_pci_remove: xc_domain_irq_permission irq=17 <NOTE: at this point a 10s pause happens> libxl: error: libxl_device.c:1134:libxl__wait_for_backend_deprecated: Backend /local/domain/0/backend/pci/4/0 not ready (state 7) libxl: error: libxl_device.c:1138:libxl__wait_for_backend_deprecated: FE /local/domain/4/device/pci/0 state 6 libxl: debug: libxl_pci.c:204:libxl__device_pci_remove_xenstore: pci backend at /local/domain/0/backend/pci/4/0 is not ready libxl: error: libxl_pci.c:1250:do_pci_remove: xc_domain_irq_permission irq=16 <NOTE: at this point a 10s pause happens> libxl: error: libxl_device.c:1134:libxl__wait_for_backend_deprecated: Backend /local/domain/0/backend/pci/4/0 not ready (state 7) libxl: error: libxl_device.c:1138:libxl__wait_for_backend_deprecated: FE /local/domain/4/device/pci/0 state 6 libxl: debug: libxl_pci.c:174:libxl__device_pci_remove_xenstore: pci backend at /local/domain/0/backend/pci/4/0 is not ready libxl: error: libxl_pci.c:1250:do_pci_remove: xc_domain_irq_permission irq=18 libxl: error: libxl_pci.c:992:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:09:02.0 <NOTE: at this point a 10s pause happens> libxl: error: libxl_device.c:1134:libxl__wait_for_backend_deprecated: Backend /local/domain/0/backend/pci/4/0 not ready (state 7) libxl: error: libxl_device.c:1138:libxl__wait_for_backend_deprecated: FE /local/domain/4/device/pci/0 state 6 libxl: debug: libxl_pci.c:174:libxl__device_pci_remove_xenstore: pci backend at /local/domain/0/backend/pci/4/0 is not ready libxl: error: libxl_pci.c:1250:do_pci_remove: xc_domain_irq_permission irq=23 <NOTE: at this point a 10s pause happens> libxl: error: libxl_device.c:1134:libxl__wait_for_backend_deprecated: Backend /local/domain/0/backend/pci/4/0 not ready (state 7) libxl: error: libxl_device.c:1138:libxl__wait_for_backend_deprecated: FE /local/domain/4/device/pci/0 state 6 libxl: debug: libxl_pci.c:174:libxl__device_pci_remove_xenstore: pci backend at /local/domain/0/backend/pci/4/0 is not ready libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7feea6a178a8 wpath=/local/domain/0/backend/vbd/4/51713/state token=2/5: register slotnum=2 libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7feea6a1c1e8 wpath=/local/domain/0/backend/vbd/4/51714/state token=3/6: register slotnum=3 libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7feea6a1f3d8 wpath=/local/domain/0/backend/vbd/4/51715/state token=0/7: register slotnum=0 libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7feea6a173c8 wpath=/local/domain/0/backend/vif/4/0/state token=19/8: register slotnum=19 libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7feea6a18408 wpath=/local/domain/0/backend/pci/4/0/state token=18/9: register slotnum=18 libxl: debug: libxl.c:1261:libxl_domain_destroy: ao 0x7feea6a15850: inprogress: poller=0x7feea6a16d70, flags=i libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7feea6a178a8 wpath=/local/domain/0/backend/vbd/4/51713/state token=2/5: event epath=/local/domain/0/backend/vbd/4/51713/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vbd/4/51713/state wanted state 6 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7feea6a178a8 wpath=/local/domain/0/backend/vbd/4/51713/state token=2/5: deregister slotnum=2 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7feea6a178a8: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/block remove libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7feea6a1c1e8 wpath=/local/domain/0/backend/vbd/4/51714/state token=3/6: event epath=/local/domain/0/backend/vbd/4/51714/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vbd/4/51714/state wanted state 6 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7feea6a1c1e8 wpath=/local/domain/0/backend/vbd/4/51714/state token=3/6: deregister slotnum=3 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7feea6a1c1e8: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/block remove libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7feea6a1f3d8 wpath=/local/domain/0/backend/vbd/4/51715/state token=0/7: event epath=/local/domain/0/backend/vbd/4/51715/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vbd/4/51715/state wanted state 6 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7feea6a1f3d8 wpath=/local/domain/0/backend/vbd/4/51715/state token=0/7: deregister slotnum=0 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7feea6a1f3d8: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/block remove libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7feea6a173c8 wpath=/local/domain/0/backend/vif/4/0/state token=19/8: event epath=/local/domain/0/backend/vif/4/0/state libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vif/4/0/state wanted state 6 ok libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7feea6a173c8 wpath=/local/domain/0/backend/vif/4/0/state token=19/8: deregister slotnum=19 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7feea6a173c8: deregister unregistered libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/vif-bridge offline libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7feea6a18408 wpath=/local/domain/0/backend/pci/4/0/state token=18/9: event epath=/local/domain/0/backend/pci/4/0/state libxl: debug: libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/pci/4/0/state wanted state 6 still waiting state 5 <NOTE: at this point a 10s pause happens - but prompt in shutdown shell is back> libxl: debug: libxl_event.c:661:devstate_timeout: backend /local/domain/0/backend/pci/4/0/state wanted state 6 timed out libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7feea6a18408 wpath=/local/domain/0/backend/pci/4/0/state token=18/9: deregister slotnum=18 libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7feea6a18408: deregister unregistered libxl: error: libxl_device.c:894:device_backend_callback: unable to remove device with path /local/domain/0/backend/pci/4/0 libxl: error: libxl.c:1452:devices_destroy_cb: libxl__devices_destroy failed for 4 libxl: debug: libxl_event.c:1569:libxl__ao_complete: ao 0x7feea6a15850: complete, rc=0 libxl: debug: libxl_event.c:1541:libxl__ao__destroy: ao 0x7feea6a15850: destroy Done. Exiting now libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7feea6a137e0 wpath=@releaseDomain token=1/4: deregister slotnum=1 xc: debug: hypercall buffer: total allocations:555 total releases:555 xc: debug: hypercall buffer: current allocations:0 maximum allocations:2 xc: debug: hypercall buffer: cache current size:2 xc: debug: hypercall buffer: cache hits:549 misses:2 toobig:4 [-- Attachment #3: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough 2014-03-21 19:39 ` Atom2 @ 2014-04-02 14:44 ` Ian Jackson 2014-04-02 15:17 ` Atom2 0 siblings, 1 reply; 23+ messages in thread From: Ian Jackson @ 2014-04-02 14:44 UTC (permalink / raw) To: Atom2, Konrad Rzeszutek Wilk, Boris Ostrovsky, David Vrabel Cc: xen-users, xen-devel, Ian Campbell, Roger Pau Monne Atom2 writes ("Re: [Xen-devel] [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough"): > Am 21.03.14 19:11, schrieb Ian Jackson: > > Can you run it again with this, on top of the previous patch, please ? > > Sure, the new output of xl -vvv create -F domain is again attached to > this e-Mail. Sorry for the delay replying. I have been ill :-(. > <NOTE: at this point a 10s pause happens> > libxl: error: libxl_device.c:1134:libxl__wait_for_backend_deprecated: Backend /local/domain/0/backend/pci/4/0 not ready (state 7) > libxl: error: libxl_device.c:1138:libxl__wait_for_backend_deprecated: FE /local/domain/4/device/pci/0 state 6 > libxl: debug: libxl_pci.c:204:libxl__device_pci_remove_xenstore: pci backend at /local/domain/0/backend/pci/4/0 is not ready > libxl: error: libxl_pci.c:1250:do_pci_remove: xc_domain_irq_permission irq=16 So the backend here is in state 7 (Reconfiguring), but the frontend is in state 6 (Closed). I think this is a bug in pciback. I looked at drivers/xen/xen-pciback/xenbus.c in Linux 3.13 and found xen_pcibk_frontend_changed which seems to do roughly what I would expect. Has this changed at some point ? Atom, what kernel are you using ? Thanks, Ian. ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough 2014-04-02 14:44 ` Ian Jackson @ 2014-04-02 15:17 ` Atom2 2014-04-18 21:47 ` Atom2 0 siblings, 1 reply; 23+ messages in thread From: Atom2 @ 2014-04-02 15:17 UTC (permalink / raw) To: xen-devel Am 02.04.14 16:44, schrieb Ian Jackson: > Atom2 writes ("Re: [Xen-devel] [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough"): >> Am 21.03.14 19:11, schrieb Ian Jackson: >>> Can you run it again with this, on top of the previous patch, please ? >> >> Sure, the new output of xl -vvv create -F domain is again attached to >> this e-Mail. > > Sorry for the delay replying. I have been ill :-(. Sorry to hear that. Though I noticed your absence from the list I simply assumed that you were off on vacation. In any case good to see you back. > >> <NOTE: at this point a 10s pause happens> >> libxl: error: libxl_device.c:1134:libxl__wait_for_backend_deprecated: Backend /local/domain/0/backend/pci/4/0 not ready (state 7) >> libxl: error: libxl_device.c:1138:libxl__wait_for_backend_deprecated: FE /local/domain/4/device/pci/0 state 6 >> libxl: debug: libxl_pci.c:204:libxl__device_pci_remove_xenstore: pci backend at /local/domain/0/backend/pci/4/0 is not ready >> libxl: error: libxl_pci.c:1250:do_pci_remove: xc_domain_irq_permission irq=16 > > So the backend here is in state 7 (Reconfiguring), but the frontend is > in state 6 (Closed). I think this is a bug in pciback. > > I looked at drivers/xen/xen-pciback/xenbus.c in Linux 3.13 and found > xen_pcibk_frontend_changed which seems to do roughly what I would > expect. > > Has this changed at some point ? > > Atom, what kernel are you using ? All the error messages stem from kernel 3.11.7. In the meantime 3.13.2 became stable for gentoo and I installed that a few days ago. I have not run the debug output yet or timed the shutdown process, but there's still a delay with that kernel and it feels as long as before. I you want, I can clearly provide new debug output or timing information. Thanks Atom2 > > Thanks, > Ian. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel > ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough 2014-04-02 15:17 ` Atom2 @ 2014-04-18 21:47 ` Atom2 2014-04-19 0:12 ` Konrad Rzeszutek Wilk 0 siblings, 1 reply; 23+ messages in thread From: Atom2 @ 2014-04-18 21:47 UTC (permalink / raw) To: xen-devel This is just a (very) gentle ping ... or have I missed out on a reply? Thanks Atom2 Am 02.04.14 17:17, schrieb Atom2: > > Am 02.04.14 16:44, schrieb Ian Jackson: >> Atom2 writes ("Re: [Xen-devel] [Xen-users] substantial shutdown delay >> for PV guests with PCI -passthrough"): >>> Am 21.03.14 19:11, schrieb Ian Jackson: >>>> Can you run it again with this, on top of the previous patch, please ? >>> >>> Sure, the new output of xl -vvv create -F domain is again attached to >>> this e-Mail. >> >> Sorry for the delay replying. I have been ill :-(. > Sorry to hear that. Though I noticed your absence from the list I simply > assumed that you were off on vacation. In any case good to see you back. >> >>> <NOTE: at this point a 10s pause happens> >>> libxl: error: libxl_device.c:1134:libxl__wait_for_backend_deprecated: >>> Backend /local/domain/0/backend/pci/4/0 not ready (state 7) >>> libxl: error: libxl_device.c:1138:libxl__wait_for_backend_deprecated: >>> FE /local/domain/4/device/pci/0 state 6 >>> libxl: debug: libxl_pci.c:204:libxl__device_pci_remove_xenstore: pci >>> backend at /local/domain/0/backend/pci/4/0 is not ready >>> libxl: error: libxl_pci.c:1250:do_pci_remove: >>> xc_domain_irq_permission irq=16 >> >> So the backend here is in state 7 (Reconfiguring), but the frontend is >> in state 6 (Closed). I think this is a bug in pciback. >> >> I looked at drivers/xen/xen-pciback/xenbus.c in Linux 3.13 and found >> xen_pcibk_frontend_changed which seems to do roughly what I would >> expect. >> >> Has this changed at some point ? >> >> Atom, what kernel are you using ? > All the error messages stem from kernel 3.11.7. In the meantime 3.13.2 > became stable for gentoo and I installed that a few days ago. I have not > run the debug output yet or timed the shutdown process, but there's > still a delay with that kernel and it feels as long as before. I you > want, I can clearly provide new debug output or timing information. > > Thanks Atom2 >> >> Thanks, >> Ian. >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> http://lists.xen.org/xen-devel >> > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough 2014-04-18 21:47 ` Atom2 @ 2014-04-19 0:12 ` Konrad Rzeszutek Wilk 2014-04-19 18:59 ` Atom2 0 siblings, 1 reply; 23+ messages in thread From: Konrad Rzeszutek Wilk @ 2014-04-19 0:12 UTC (permalink / raw) To: Atom2; +Cc: xen-devel On Fri, Apr 18, 2014 at 11:47:46PM +0200, Atom2 wrote: > This is just a (very) gentle ping ... or have I missed out on a reply? I ran an PV guest with PCI passthrough this week and it had no trouble - didn't see 10 seconds or so. But I did the shutdown from within the guest (poweroff). What is the kernel you are running as your dom0? Is it the same as frontend? > > Thanks Atom2 > > Am 02.04.14 17:17, schrieb Atom2: > > > >Am 02.04.14 16:44, schrieb Ian Jackson: > >>Atom2 writes ("Re: [Xen-devel] [Xen-users] substantial shutdown delay > >>for PV guests with PCI -passthrough"): > >>>Am 21.03.14 19:11, schrieb Ian Jackson: > >>>>Can you run it again with this, on top of the previous patch, please ? > >>> > >>>Sure, the new output of xl -vvv create -F domain is again attached to > >>>this e-Mail. > >> > >>Sorry for the delay replying. I have been ill :-(. > >Sorry to hear that. Though I noticed your absence from the list I simply > >assumed that you were off on vacation. In any case good to see you back. > >> > >>><NOTE: at this point a 10s pause happens> > >>>libxl: error: libxl_device.c:1134:libxl__wait_for_backend_deprecated: > >>>Backend /local/domain/0/backend/pci/4/0 not ready (state 7) > >>>libxl: error: libxl_device.c:1138:libxl__wait_for_backend_deprecated: > >>>FE /local/domain/4/device/pci/0 state 6 > >>>libxl: debug: libxl_pci.c:204:libxl__device_pci_remove_xenstore: pci > >>>backend at /local/domain/0/backend/pci/4/0 is not ready > >>>libxl: error: libxl_pci.c:1250:do_pci_remove: > >>>xc_domain_irq_permission irq=16 > >> > >>So the backend here is in state 7 (Reconfiguring), but the frontend is > >>in state 6 (Closed). I think this is a bug in pciback. > >> > >>I looked at drivers/xen/xen-pciback/xenbus.c in Linux 3.13 and found > >>xen_pcibk_frontend_changed which seems to do roughly what I would > >>expect. > >> > >>Has this changed at some point ? > >> > >>Atom, what kernel are you using ? > >All the error messages stem from kernel 3.11.7. In the meantime 3.13.2 > >became stable for gentoo and I installed that a few days ago. I have not > >run the debug output yet or timed the shutdown process, but there's > >still a delay with that kernel and it feels as long as before. I you > >want, I can clearly provide new debug output or timing information. > > > >Thanks Atom2 > >> > >>Thanks, > >>Ian. > >> > >>_______________________________________________ > >>Xen-devel mailing list > >>Xen-devel@lists.xen.org > >>http://lists.xen.org/xen-devel > >> > > > >_______________________________________________ > >Xen-devel mailing list > >Xen-devel@lists.xen.org > >http://lists.xen.org/xen-devel > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough 2014-04-19 0:12 ` Konrad Rzeszutek Wilk @ 2014-04-19 18:59 ` Atom2 2014-04-22 10:44 ` George Dunlap 0 siblings, 1 reply; 23+ messages in thread From: Atom2 @ 2014-04-19 18:59 UTC (permalink / raw) To: Konrad Rzeszutek Wilk, Ian Jackson, boris.ostrovsky, david.vrabel Cc: Roger Pau Monne, xen-users, Ian Campbell, xen-devel Hi Konrad, thanks for your quick reply. I have re-added the other recipients that were in the list prior to my reply from 2 April as I just saw that I somehow have managed to drop those guys - which might also explain their silence to my reply. All: sorry for dropping you from my earlier reply. For your convenience I have added my reply from 2 April at the end of this mail. Am 19.04.14 02:12, schrieb Konrad Rzeszutek Wilk: > On Fri, Apr 18, 2014 at 11:47:46PM +0200, Atom2 wrote: >> This is just a (very) gentle ping ... or have I missed out on a reply? > > I ran an PV guest with PCI passthrough this week and it had no trouble - > didn't see 10 seconds or so. But I did the shutdown from within the > guest (poweroff). For me it makes no difference timewise whether I issue a xl shutdown guest from dom0 or whether I issue shutdown -h now from a connection (i.e. ssh or screen or console) to the guest. The main difference being that for the latter the delay is visible whereas for the former, the delay is not so obvious because 'xl shutdown guest' from dom0 due to its asynchronous nature returns immediately even when the guest is still alive. One difference that I have noticed however is that for the shutdown from _within_ the guest (i.e. shutdown -h now) the state of the guest remains 's' in 'xl list' from the time the "system halted" message appears on screen until the prompt returns in dom0 whereas for a shutdown from dom0 with 'xl shutdown guest' the state changes from 's' to 'ps' for a number of seconds before it is finally gone. > > What is the kernel you are running as your dom0? Is it the same as > frontend? Frontend and Backend are both running the same kernel version, albeit obviously with different configurations. The current version of both kernels is 13.3.2-r3 and I am using the gentoo-hardened sources. The same thing also happend with my previous kernel version which was 3.11.7-r2 (also gentoo-hardened sources). Thanks Atom2 ======================================================================== ===== Am 02.04.14 17:17, schrieb Atom2: ======= Am 02.04.14 16:44, schrieb Ian Jackson: > Atom2 writes ("Re: [Xen-devel] [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough"): >> Am 21.03.14 19:11, schrieb Ian Jackson: >>> Can you run it again with this, on top of the previous patch, please ? >> >> Sure, the new output of xl -vvv create -F domain is again attached to >> this e-Mail. > > Sorry for the delay replying. I have been ill . Sorry to hear that. Though I noticed your absence from the list I simply assumed that you were off on vacation. In any case good to see you back. > >> <NOTE: at this point a 10s pause happens> >> libxl: error: libxl_device.c:1134:libxl__wait_for_backend_deprecated: Backend /local/domain/0/backend/pci/4/0 not ready (state 7) >> libxl: error: libxl_device.c:1138:libxl__wait_for_backend_deprecated: FE /local/domain/4/device/pci/0 state 6 >> libxl: debug: libxl_pci.c:204:libxl__device_pci_remove_xenstore: pci backend at /local/domain/0/backend/pci/4/0 is not ready >> libxl: error: libxl_pci.c:1250:do_pci_remove: xc_domain_irq_permission irq=16 > > So the backend here is in state 7 (Reconfiguring), but the frontend is > in state 6 (Closed). I think this is a bug in pciback. > > I looked at drivers/xen/xen-pciback/xenbus.c in Linux 3.13 and found > xen_pcibk_frontend_changed which seems to do roughly what I would > expect. > > Has this changed at some point ? > > Atom, what kernel are you using ? All the error messages stem from kernel 3.11.7. In the meantime 3.13.2 became stable for gentoo and I installed that a few days ago. I have not run the debug output yet or timed the shutdown process, but there's still a delay with that kernel and it feels as long as before. I you want, I can clearly provide new debug output or timing information. Thanks Atom2 > > Thanks, > Ian. ======================================================================= ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough 2014-04-19 18:59 ` Atom2 @ 2014-04-22 10:44 ` George Dunlap 2014-04-22 12:02 ` Atom2 0 siblings, 1 reply; 23+ messages in thread From: George Dunlap @ 2014-04-22 10:44 UTC (permalink / raw) To: Atom2 Cc: Ian Campbell, Ian Jackson, xen-devel@lists.xen.org, David Vrabel, xen-users, Boris Ostrovsky, Roger Pau Monne On Sat, Apr 19, 2014 at 7:59 PM, Atom2 <ariel.atom2@web2web.at> wrote: > Hi Konrad, > thanks for your quick reply. I have re-added the other recipients that were > in the list prior to my reply from 2 April as I just saw that I somehow have > managed to drop those guys - which might also explain their silence to my > reply. > All: sorry for dropping you from my earlier reply. For your convenience I > have added my reply from 2 April at the end of this mail. > > Am 19.04.14 02:12, schrieb Konrad Rzeszutek Wilk: > >> On Fri, Apr 18, 2014 at 11:47:46PM +0200, Atom2 wrote: >>> >>> This is just a (very) gentle ping ... or have I missed out on a reply? >> >> >> I ran an PV guest with PCI passthrough this week and it had no trouble - >> didn't see 10 seconds or so. But I did the shutdown from within the >> guest (poweroff). > > For me it makes no difference timewise whether I issue a > xl shutdown guest > from dom0 or whether I issue > shutdown -h now > from a connection (i.e. ssh or screen or console) to the guest. The main > difference being that for the latter the delay is visible whereas for the > former, the delay is not so obvious because 'xl shutdown guest' from dom0 > due to its asynchronous nature returns immediately even when the guest is > still alive. > > One difference that I have noticed however is that for the shutdown from > _within_ the guest (i.e. shutdown -h now) the state of the guest remains 's' > in 'xl list' from the time the "system halted" message appears on screen > until the prompt returns in dom0 whereas for a shutdown from dom0 with 'xl > shutdown guest' the state changes from 's' to 'ps' for a number of seconds > before it is finally gone. Does it look anything like this? marc.info/?i=<CAFLBxZbOdU=uSwNBVRTB7_7yhPyRShda0asSOvv9J+xfoGxRnA@mail.gmail.com> (the log in question is /var/log/xen/xl-$DOMAINNAME.log) -George ^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough 2014-04-22 10:44 ` George Dunlap @ 2014-04-22 12:02 ` Atom2 0 siblings, 0 replies; 23+ messages in thread From: Atom2 @ 2014-04-22 12:02 UTC (permalink / raw) To: George Dunlap Cc: Ian Campbell, Ian Jackson, xen-devel@lists.xen.org, David Vrabel, xen-users, Boris Ostrovsky, Roger Pau Monne Am 22.04.14 12:44, schrieb George Dunlap: > On Sat, Apr 19, 2014 at 7:59 PM, Atom2 <ariel.atom2@web2web.at> wrote: >> Am 19.04.14 02:12, schrieb Konrad Rzeszutek Wilk: >>> I ran an PV guest with PCI passthrough this week and it had no trouble - >>> didn't see 10 seconds or so. But I did the shutdown from within the >>> guest (poweroff). >> >> For me it makes no difference timewise whether I issue a >> xl shutdown guest >> from dom0 or whether I issue >> shutdown -h now >> from a connection (i.e. ssh or screen or console) to the guest. The main >> difference being that for the latter the delay is visible whereas for the >> former, the delay is not so obvious because 'xl shutdown guest' from dom0 >> due to its asynchronous nature returns immediately even when the guest is >> still alive. >> >> One difference that I have noticed however is that for the shutdown from >> _within_ the guest (i.e. shutdown -h now) the state of the guest remains 's' >> in 'xl list' from the time the "system halted" message appears on screen >> until the prompt returns in dom0 whereas for a shutdown from dom0 with 'xl >> shutdown guest' the state changes from 's' to 'ps' for a number of seconds >> before it is finally gone. > > Does it look anything like this? > > marc.info/?i=<CAFLBxZbOdU=uSwNBVRTB7_7yhPyRShda0asSOvv9J+xfoGxRnA@mail.gmail.com> > > (the log in question is /var/log/xen/xl-$DOMAINNAME.log) > > -George Not really (unless there's a specific command line option required to get your output) - at least to my eye the messages in my log file look rather different: _______________________________________________________________________ Waiting for domain voip (domid 3) to die [pid 2274] Domain 3 has shut down, reason code 0 0x0 Action for shutdown reason code 0 is destroy Domain 3 needs to be cleaned up: destroying the domain libxl: error: libxl_pci.c:1250:do_pci_remove: xc_domain_irq_permission irq=17 libxl: error: libxl_device.c:1134:libxl__wait_for_backend_deprecated: Backend /local/domain/0/backend/pci/3/0 not ready (state 7) libxl: error: libxl_device.c:1138:libxl__wait_for_backend_deprecated: FE /local/domain/3/device/pci/0 state 6 libxl: error: libxl_pci.c:1250:do_pci_remove: xc_domain_irq_permission irq=16 libxl: error: libxl_device.c:1134:libxl__wait_for_backend_deprecated: Backend /local/domain/0/backend/pci/3/0 not ready (state 7) libxl: error: libxl_device.c:1138:libxl__wait_for_backend_deprecated: FE /local/domain/3/device/pci/0 state 6 libxl: error: libxl_pci.c:1250:do_pci_remove: xc_domain_irq_permission irq=23 libxl: error: libxl_device.c:1134:libxl__wait_for_backend_deprecated: Backend /local/domain/0/backend/pci/3/0 not ready (state 7) libxl: error: libxl_device.c:1138:libxl__wait_for_backend_deprecated: FE /local/domain/3/device/pci/0 state 6 libxl: error: libxl_device.c:894:device_backend_callback: unable to remove device with path /local/domain/0/backend/pci/3/0 libxl: error: libxl.c:1452:devices_destroy_cb: libxl__devices_destroy failed for 3 Done. Exiting now Please note: some of my messages may be the result of Ian Jackson's (debug) patches which are still active in my environment. Thanks Atom2 ^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2014-04-22 12:02 UTC | newest]
Thread overview: 23+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <5325B828.1060303@web2web.at>
[not found] ` <1395050430.4122.29.camel@kazak.uk.xensource.com>
[not found] ` <53273B3C.40707@web2web.at>
2014-03-18 10:15 ` [Xen-users] substantial shutdown delay for PV guests with PCI -passthrough Ian Campbell
2014-03-18 13:01 ` Atom2
2014-03-18 15:07 ` Ian Campbell
2014-03-19 0:25 ` Atom2
2014-03-19 11:26 ` Ian Campbell
2014-03-19 13:00 ` Konrad Rzeszutek Wilk
2014-03-19 14:03 ` Atom2
2014-03-19 15:45 ` Ian Jackson
2014-03-20 2:31 ` Atom2
2014-03-20 11:52 ` Ian Jackson
2014-03-20 13:53 ` Pasi Kärkkäinen
2014-03-20 15:28 ` Ian Jackson
2014-03-20 19:34 ` Atom2
2014-03-20 19:32 ` Atom2
2014-03-21 18:11 ` Ian Jackson
2014-03-21 19:39 ` Atom2
2014-04-02 14:44 ` Ian Jackson
2014-04-02 15:17 ` Atom2
2014-04-18 21:47 ` Atom2
2014-04-19 0:12 ` Konrad Rzeszutek Wilk
2014-04-19 18:59 ` Atom2
2014-04-22 10:44 ` George Dunlap
2014-04-22 12:02 ` Atom2
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).