Re: segfault in xl create for HVM with PCI passthrough

[Atom2 <ariel.atom2@web2web.at>]

[-- Attachment #1: Type: text/plain, Size: 5874 bytes --]

Hi Ian,
thanks for your quick reply - please see below.
Am 28.10.14 um 11:59 schrieb Ian Campbell:
> On Mon, 2014-10-27 at 22:25 +0100, Atom2 wrote:
>> Hi guys,
>> I have used XEN for quiet some time and after a steep learning curve I
>> have always been a very happy user! XEN is really a great product.
>> Unfortunately I am now facing a problem that leaves me at loss:
>>
>> Using gentoo as a rolling distribution I recently I upgraded to XEN
>> 4.3.3 (from 4.3.1) and also upgraded the gcc compiler to 4.8.3 (from
>> 4.7.3). Both packages are the latest stable versions available under gentoo.
>>
>> After emerging (that is the re-compilation and installation of XEN 4.3.3
>> on my machine) following a toolchain upgrade to the new gcc I can't
>> start my two HVM FreeBSD virtual machines anymore. Both use PCI
>> passthrough devices and both the motherboard and the processor support
>> VT-d. XEN PV gentoo domUs (without passed through PCI devices) still
>> start up (but are useless for me at the moment as they depend on the
>> services provided by the tow HVM domus).
>>
>> The error when starting manifests itself as follows:
>> # xl create -c pfsense
>> Parsing config from 01:pfsense.1
>> xc: info: VIRTUAL MEMORY ARRANGEMENT:
>>     Loader:        0000000000100000->00000000001c12a4
>>     Modules:       0000000000000000->0000000000000000
>>     TOTAL:         0000000000000000->000000001f800000
>>     ENTRY ADDRESS: 0000000000100000
>> xc: info: PHYSICAL MEMORY ALLOCATION:
>>     4KB PAGES: 0x0000000000000200
>>     2MB PAGES: 0x00000000000000fb
>>     1GB PAGES: 0x0000000000000000
>> Segmentation fault
>> #
>>
>> The domU is in a state of paused for reasons unknown to me and does not
>> use any CPU cycles:
>
> Domains are created paused and then unpaused at the end of the creation
> process, presumably this didn't happen because xl segfaulted first.
I was not aware of that as this pausing/unpausing happens within a very 
short period of time and was never visible to me. But that at least 
explains why the domain is paused ... I again learned something new.
>
> Please can you run the command under gdb and grab a back trace. It would
> also be useful to "xl -vvv create pfsense".
>
First of all attached please find the output of xl -vvv create pfsense. 
I decided to attach a file as most of the output lines are longer than 
80 chars and therefore would most likely be folded by eMail clients.
In terms of the last message before the segfault in my attached file it 
seems to me that the bridge stuff was setup correctly as per the 
following commands:

# brctl show xenbr0
bridge name     bridge id               STP enabled     interfaces
xenbr0          8000.00187d1d7274       no              bond0
                                                         vif2.0
                                                         vif2.0-emu
# ifconfig
<snip>
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
         inet 127.0.0.1  netmask 255.0.0.0
         inet6 ::1  prefixlen 128  scopeid 0x10<host>
         loop  txqueuelen 0  (Local Loopback)
         RX packets 118  bytes 11408 (11.1 KiB)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 118  bytes 11408 (11.1 KiB)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif2.0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
         ether fe:ff:ff:ff:ff:ff  txqueuelen 32  (Ethernet)
         RX packets 0  bytes 0 (0.0 B)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 0  bytes 0 (0.0 B)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif2.0-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
         inet6 fe80::fcff:ffff:feff:ffff  prefixlen 64  scopeid 0x20<link>
         ether fe:ff:ff:ff:ff:ff  txqueuelen 500  (Ethernet)
         RX packets 0  bytes 0 (0.0 B)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 0  bytes 0 (0.0 B)
         TX errors 0  dropped 598 overruns 0  carrier 0  collisions 0

xenbr0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
         inet 192.168.19.2  netmask 255.255.255.0  broadcast 192.168.19.255
         inet6 fe80::218:7dff:fe1d:7274  prefixlen 64  scopeid 0x20<link>
         ether 00:18:7d:1d:72:74  txqueuelen 0  (Ethernet)
         RX packets 58364  bytes 16721913 (15.9 MiB)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 13224  bytes 3090681 (2.9 MiB)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


With regards to gdb: I can certainly run the command under gdb after 
including debug support to the executables - that's no big deal.
I would, however, ask for your advice as to what I need to recompile 
with debugger support? Is xen-tools (which includes xl) sufficient or 
would you think that I also need to include debug support for gcc as the 
library that is mentioned in /var/log/messages (libgcc_s.so.1) seems to 
belong to the gcc package? Or is this library a red herring that just 
works as the catch-all code getting and finally handling the segfault? 
Please advise. Tx.
> [...]
>> pci          = [ '04:00.0', '0a:08.0', '0a:0b.0' ]
>
> You say in $subject that the failure is with PCI, is that because you've
> tried an HVM domain without and it is ok, or is it just that all your
> HVM domains happen to have passthrough enabled?
I haven't tried HVM domains without PCI passthrough (but PV domains w/o 
PCI passthrough and they did not segfault) so far as all my HVM domains 
require PCI devices (either at least a network card for pfsense - in 
actual facts it's more than one that's being passed through - or a SATA 
controller for my second HVM which is used as a storage VM).

If you think that after the gdb stuff it would still be beneficial to go 
down that route, I am sure I can come up with something.
>
> Ian.
>
Again many thanks Atom2

[-- Attachment #2: output --]
[-- Type: text/plain, Size: 7633 bytes --]

Parsing config from pfsense
libxl: debug: libxl_create.c:1243:do_domain_create: ao 0x7fdcfc226ce0: create: how=(nil) callback=(nil) poller=0x7fdcfc227690
libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=sda spec.backend=unknown
libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk vdev=sda, using backend phy
libxl: debug: libxl_create.c:699:initiate_domain_create: running bootloader
libxl: debug: libxl_bootloader.c:321:libxl__bootloader_run: not a PV domain, skipping bootloader
libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7fdcfc228098: deregister unregistered
xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0xc12a4
xc: detail: elf_parse_binary: memory: 0x100000 -> 0x1c12a4
xc: info: VIRTUAL MEMORY ARRANGEMENT:
  Loader:        0000000000100000->00000000001c12a4
  Modules:       0000000000000000->0000000000000000
  TOTAL:         0000000000000000->000000001f800000
  ENTRY ADDRESS: 0000000000100000
xc: info: PHYSICAL MEMORY ALLOCATION:
  4KB PAGES: 0x0000000000000200
  2MB PAGES: 0x00000000000000fb
  1GB PAGES: 0x0000000000000000
xc: detail: elf_load_binary: phdr 0 at 0x7fdcfbe26000 -> 0x7fdcfbede12d
libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=sda spec.backend=phy
libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7fdcfc227398 wpath=/local/domain/0/backend/vbd/2/2048/state token=3/0: register slotnum=3
libxl: debug: libxl_create.c:1256:do_domain_create: ao 0x7fdcfc226ce0: inprogress: poller=0x7fdcfc227690, flags=i
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fdcfc227398 wpath=/local/domain/0/backend/vbd/2/2048/state token=3/0: event epath=/local/domain/0/backend/vbd/2/2048/state
libxl: debug: libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vbd/2/2048/state wanted state 2 still waiting state 1
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fdcfc227398 wpath=/local/domain/0/backend/vbd/2/2048/state token=3/0: event epath=/local/domain/0/backend/vbd/2/2048/state
libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vbd/2/2048/state wanted state 2 ok
libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7fdcfc227398 wpath=/local/domain/0/backend/vbd/2/2048/state token=3/0: deregister slotnum=3
libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7fdcfc227398: deregister unregistered
libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/block add
libxl: debug: libxl_dm.c:1211:libxl__spawn_local_dm: Spawning device-model /usr/lib/xen/bin/qemu-system-i386 with arguments:
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   /usr/lib/xen/bin/qemu-system-i386
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -xen-domid
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   2
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -chardev
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-2,server,nowait
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -mon
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   chardev=libxl-cmd,mode=control
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -name
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   pfsense
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -global
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   isa-fdc.driveA=
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -serial
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   pty
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -nographic
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -vga
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   cirrus
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -global
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   vga.vram_size_mb=8
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -boot
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   order=c
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -device
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   e1000,id=nic0,netdev=net0,mac=00:16:3e:a1:64:01
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -netdev
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   type=tap,id=net0,ifname=vif2.0-emu,script=no,downscript=no
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -M
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   xenfv
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -m
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   504
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -drive
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   file=/etc/xen/guests/disk.d/pfsense.disk,if=scsi,bus=0,unit=0,format=raw,cache=writeback
libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7fdcfc2282d0 wpath=/local/domain/0/device-model/2/state token=3/1: register slotnum=3
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fdcfc2282d0 wpath=/local/domain/0/device-model/2/state token=3/1: event epath=/local/domain/0/device-model/2/state
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fdcfc2282d0 wpath=/local/domain/0/device-model/2/state token=3/1: event epath=/local/domain/0/device-model/2/state
libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7fdcfc2282d0 wpath=/local/domain/0/device-model/2/state token=3/1: deregister slotnum=3
libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7fdcfc2282d0: deregister unregistered
libxl: debug: libxl_qmp.c:707:libxl__qmp_initialize: connected to /var/run/xen/qmp-libxl-2
libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: qmp
libxl: debug: libxl_qmp.c:555:qmp_send_prepare: next qmp command: '{
    "execute": "qmp_capabilities",
    "id": 1
}
'
libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: return
libxl: debug: libxl_qmp.c:555:qmp_send_prepare: next qmp command: '{
    "execute": "query-chardev",
    "id": 2
}
'
libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: return
libxl: debug: libxl_qmp.c:555:qmp_send_prepare: next qmp command: '{
    "execute": "query-vnc",
    "id": 3
}
'
libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: return
libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7fdcfc22b8f8 wpath=/local/domain/0/backend/vif/2/0/state token=3/2: register slotnum=3
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fdcfc22b8f8 wpath=/local/domain/0/backend/vif/2/0/state token=3/2: event epath=/local/domain/0/backend/vif/2/0/state
libxl: debug: libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vif/2/0/state wanted state 2 still waiting state 1
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fdcfc22b8f8 wpath=/local/domain/0/backend/vif/2/0/state token=3/2: event epath=/local/domain/0/backend/vif/2/0/state
libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vif/2/0/state wanted state 2 ok
libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7fdcfc22b8f8 wpath=/local/domain/0/backend/vif/2/0/state token=3/2: deregister slotnum=3
libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7fdcfc22b8f8: deregister unregistered
libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/vif-bridge online
libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/vif-bridge add
Segmentation fault

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel