Re: segfault in xl create for HVM with PCI passthrough

Re: segfault in xl create for HVM with PCI passthrough

[Atom2]

Am 28.10.14 um 17:04 schrieb Ian Campbell:
>>> You say in $subject that the failure is with PCI, is that because you've
>>> tried an HVM domain without and it is ok, or is it just that all your
>>> HVM domains happen to have passthrough enabled?
>> I haven't tried HVM domains without PCI passthrough (but PV domains w/o
>> PCI passthrough and they did not segfault) so far as all my HVM domains
>> require PCI devices (either at least a network card for pfsense - in
>> actual facts it's more than one that's being passed through - or a SATA
>> controller for my second HVM which is used as a storage VM).
>
> The VM doesn't need to be fully functional, it just needs to boot
> without crashing the toolstack. Just running your existing VM with the
> pci line commented out would be useful.
Before re-compiling the xen-tools I made a quick test as you suggested
and commented out the pci line from my config file ... and the boot menu 
showed up (which it did not before when the segfault happened).
I did not boot the pfsense vm any further as this might lead to a change 
in my configuration due to missing devices, but to me this at first 
sight seemed to indicate that is has to do with the PCI passthrough 
functionality.
Although as I did not want to boot the machine (and "xl shutdown" did
not work, not even with -F) I then decided to
	xl destroy pfsense
and that printed a segmentation fault message (in both the shell window
where I started the command from and the console window where the
boot-menu was shown) despite no PCI devices being passed through.

To also check PCI passthrough with a PV domain: I added a pci device to
a config file for a PV domain and started that with
	xl create voip -c
The boot menu appeared without issues. I then also tried
	xl destroy voip
from another window and that issued the following error messages in the
shell window (without using any -vvv option):

# xl destroy voip
libxl: error: libxl_pci.c:1247:do_pci_remove: xc_domain_irq_permission
irq=17
libxl: error: libxl_device.c:1127:libxl__wait_for_backend: Backend
/local/domain/0/backend/pci/4/0 not ready
libxl: error: libxl_pci.c:1247:do_pci_remove: xc_domain_irq_permission
irq=16
libxl: error: libxl_device.c:1127:libxl__wait_for_backend: Backend
/local/domain/0/backend/pci/4/0 not ready
libxl: error: libxl_pci.c:1247:do_pci_remove: xc_domain_irq_permission
irq=23
libxl: error: libxl_device.c:1127:libxl__wait_for_backend: Backend
/local/domain/0/backend/pci/4/0 not ready
Segmentation fault

The "Segmentation fault" message also appeared in both the console
window for the domU and the shell window.

I'll do the gdb stuff later today and provide you with the backtrace.
But that's probably going to be a few hours from now as it requires a
bit or preparation and time to re-compile and I need to do some other
stuff inbetween.

This all seems a bit strange to me at the moement, but I am sure with
your help we will arrive at the grounds of this.

Thanks and regards Atom

segfault in xl create for HVM with PCI passthrough

[Atom2]

Hi guys,
I have used XEN for quiet some time and after a steep learning curve I 
have always been a very happy user! XEN is really a great product. 
Unfortunately I am now facing a problem that leaves me at loss:

Using gentoo as a rolling distribution I recently I upgraded to XEN 
4.3.3 (from 4.3.1) and also upgraded the gcc compiler to 4.8.3 (from 
4.7.3). Both packages are the latest stable versions available under gentoo.

After emerging (that is the re-compilation and installation of XEN 4.3.3 
on my machine) following a toolchain upgrade to the new gcc I can't 
start my two HVM FreeBSD virtual machines anymore. Both use PCI 
passthrough devices and both the motherboard and the processor support 
VT-d. XEN PV gentoo domUs (without passed through PCI devices) still 
start up (but are useless for me at the moment as they depend on the 
services provided by the tow HVM domus).

The error when starting manifests itself as follows:
# xl create -c pfsense
Parsing config from 01:pfsense.1
xc: info: VIRTUAL MEMORY ARRANGEMENT:
   Loader:        0000000000100000->00000000001c12a4
   Modules:       0000000000000000->0000000000000000
   TOTAL:         0000000000000000->000000001f800000
   ENTRY ADDRESS: 0000000000100000
xc: info: PHYSICAL MEMORY ALLOCATION:
   4KB PAGES: 0x0000000000000200
   2MB PAGES: 0x00000000000000fb
   1GB PAGES: 0x0000000000000000
Segmentation fault
#

The domU is in a state of paused for reasons unknown to me and does not 
use any CPU cycles:
# xl list
Name                              ID   Mem VCPUs      State   Time(s)
Domain-0                           0  4094     8     r-----      41.7
pfsense                            1   512     1     --p---       0.0
#

The expected output is actually the boot-menu from FreeBSD (I do use a 
serial console in FreeBSD and that worked for month without any hickup 
before the recent update). It also never entered a paused state ...

/var/log/messages shows the following line related to the segfault:
Oct 27 20:16:22 vm-host kernel: [  458.354314] xl[2906]: segfault at 
7fbc56b93eb0 ip 00007fbc54430b64 sp 00007fbc56b93eb0 error 6 in 
libgcc_s.so.1[7fbc54422000+16000]

If I destroy the paused domU by issuing
# xl destroy pfsense
Segmentation fault
#

An error is again logged in /var/log/messages similar to the start error 
messages as follows:
Oct 27 22:06:59 vm-host kernel: [ 7095.794688] xl[3218]: segfault at 
7f22ced42eb0 ip 00007f22cc5cfb64 sp 00007f22ced42eb0 error 6 in 
libgcc_s.so.1[7f22cc5c1000+16000]

The pfsense config file is pretty simple/basic, nothing fancy in there:
builder       = 'hvm'
cpus          = '2-7'
vcpus         = 1
cpu_weight    = 512
memory        = 512
name          = 'pfsense'
disk          = [ 'phy:/etc/xen/guests/disk.d/pfsense.disk,sda,w' ]
vif           = [ 'mac=00:16:3e:a1:64:01,bridge=xenbr0,model=e1000' ]
on_poweroff  = 'destroy'
on_reboot    = 'restart'
on_crash     = 'restart'
localtime    = 0
boot         = 'c'
vnc          = 0
nographic    = 1
serial       = 'pty'
nx           = 1
pci          = [ '04:00.0', '0a:08.0', '0a:0b.0' ]


In order to rule out any inconsistency after the gcc update I have today 
also emerged the complete world set (including kernel re-compilation) - 
unfortunately to no avail: The error persists. Other than this xl/XEN 
problem the machine operates without any issues.

I'd very much appreciate if somebody could shed some light on this 
issue. In case you require any more information, I am more than happy to 
provide it.

Many thanks in advance and best regards

Atom2

Re: segfault in xl create for HVM with PCI passthrough

[Ian Campbell]

On Mon, 2014-10-27 at 22:25 +0100, Atom2 wrote:
> Hi guys,
> I have used XEN for quiet some time and after a steep learning curve I 
> have always been a very happy user! XEN is really a great product. 
> Unfortunately I am now facing a problem that leaves me at loss:
> 
> Using gentoo as a rolling distribution I recently I upgraded to XEN 
> 4.3.3 (from 4.3.1) and also upgraded the gcc compiler to 4.8.3 (from 
> 4.7.3). Both packages are the latest stable versions available under gentoo.
> 
> After emerging (that is the re-compilation and installation of XEN 4.3.3 
> on my machine) following a toolchain upgrade to the new gcc I can't 
> start my two HVM FreeBSD virtual machines anymore. Both use PCI 
> passthrough devices and both the motherboard and the processor support 
> VT-d. XEN PV gentoo domUs (without passed through PCI devices) still 
> start up (but are useless for me at the moment as they depend on the 
> services provided by the tow HVM domus).
> 
> The error when starting manifests itself as follows:
> # xl create -c pfsense
> Parsing config from 01:pfsense.1
> xc: info: VIRTUAL MEMORY ARRANGEMENT:
>    Loader:        0000000000100000->00000000001c12a4
>    Modules:       0000000000000000->0000000000000000
>    TOTAL:         0000000000000000->000000001f800000
>    ENTRY ADDRESS: 0000000000100000
> xc: info: PHYSICAL MEMORY ALLOCATION:
>    4KB PAGES: 0x0000000000000200
>    2MB PAGES: 0x00000000000000fb
>    1GB PAGES: 0x0000000000000000
> Segmentation fault
> #
> 
> The domU is in a state of paused for reasons unknown to me and does not 
> use any CPU cycles:

Domains are created paused and then unpaused at the end of the creation
process, presumably this didn't happen because xl segfaulted first.

Please can you run the command under gdb and grab a back trace. It would
also be useful to "xl -vvv create pfsense".

[...]
> pci          = [ '04:00.0', '0a:08.0', '0a:0b.0' ]

You say in $subject that the failure is with PCI, is that because you've
tried an HVM domain without and it is ok, or is it just that all your
HVM domains happen to have passthrough enabled?

Ian.

Re: segfault in xl create for HVM with PCI passthrough

[Atom2]

[-- Attachment #1: Type: text/plain, Size: 5874 bytes --]

Hi Ian,
thanks for your quick reply - please see below.
Am 28.10.14 um 11:59 schrieb Ian Campbell:
> On Mon, 2014-10-27 at 22:25 +0100, Atom2 wrote:
>> Hi guys,
>> I have used XEN for quiet some time and after a steep learning curve I
>> have always been a very happy user! XEN is really a great product.
>> Unfortunately I am now facing a problem that leaves me at loss:
>>
>> Using gentoo as a rolling distribution I recently I upgraded to XEN
>> 4.3.3 (from 4.3.1) and also upgraded the gcc compiler to 4.8.3 (from
>> 4.7.3). Both packages are the latest stable versions available under gentoo.
>>
>> After emerging (that is the re-compilation and installation of XEN 4.3.3
>> on my machine) following a toolchain upgrade to the new gcc I can't
>> start my two HVM FreeBSD virtual machines anymore. Both use PCI
>> passthrough devices and both the motherboard and the processor support
>> VT-d. XEN PV gentoo domUs (without passed through PCI devices) still
>> start up (but are useless for me at the moment as they depend on the
>> services provided by the tow HVM domus).
>>
>> The error when starting manifests itself as follows:
>> # xl create -c pfsense
>> Parsing config from 01:pfsense.1
>> xc: info: VIRTUAL MEMORY ARRANGEMENT:
>>     Loader:        0000000000100000->00000000001c12a4
>>     Modules:       0000000000000000->0000000000000000
>>     TOTAL:         0000000000000000->000000001f800000
>>     ENTRY ADDRESS: 0000000000100000
>> xc: info: PHYSICAL MEMORY ALLOCATION:
>>     4KB PAGES: 0x0000000000000200
>>     2MB PAGES: 0x00000000000000fb
>>     1GB PAGES: 0x0000000000000000
>> Segmentation fault
>> #
>>
>> The domU is in a state of paused for reasons unknown to me and does not
>> use any CPU cycles:
>
> Domains are created paused and then unpaused at the end of the creation
> process, presumably this didn't happen because xl segfaulted first.
I was not aware of that as this pausing/unpausing happens within a very 
short period of time and was never visible to me. But that at least 
explains why the domain is paused ... I again learned something new.
>
> Please can you run the command under gdb and grab a back trace. It would
> also be useful to "xl -vvv create pfsense".
>
First of all attached please find the output of xl -vvv create pfsense. 
I decided to attach a file as most of the output lines are longer than 
80 chars and therefore would most likely be folded by eMail clients.
In terms of the last message before the segfault in my attached file it 
seems to me that the bridge stuff was setup correctly as per the 
following commands:

# brctl show xenbr0
bridge name     bridge id               STP enabled     interfaces
xenbr0          8000.00187d1d7274       no              bond0
                                                         vif2.0
                                                         vif2.0-emu
# ifconfig
<snip>
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
         inet 127.0.0.1  netmask 255.0.0.0
         inet6 ::1  prefixlen 128  scopeid 0x10<host>
         loop  txqueuelen 0  (Local Loopback)
         RX packets 118  bytes 11408 (11.1 KiB)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 118  bytes 11408 (11.1 KiB)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif2.0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
         ether fe:ff:ff:ff:ff:ff  txqueuelen 32  (Ethernet)
         RX packets 0  bytes 0 (0.0 B)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 0  bytes 0 (0.0 B)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vif2.0-emu: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
         inet6 fe80::fcff:ffff:feff:ffff  prefixlen 64  scopeid 0x20<link>
         ether fe:ff:ff:ff:ff:ff  txqueuelen 500  (Ethernet)
         RX packets 0  bytes 0 (0.0 B)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 0  bytes 0 (0.0 B)
         TX errors 0  dropped 598 overruns 0  carrier 0  collisions 0

xenbr0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
         inet 192.168.19.2  netmask 255.255.255.0  broadcast 192.168.19.255
         inet6 fe80::218:7dff:fe1d:7274  prefixlen 64  scopeid 0x20<link>
         ether 00:18:7d:1d:72:74  txqueuelen 0  (Ethernet)
         RX packets 58364  bytes 16721913 (15.9 MiB)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 13224  bytes 3090681 (2.9 MiB)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


With regards to gdb: I can certainly run the command under gdb after 
including debug support to the executables - that's no big deal.
I would, however, ask for your advice as to what I need to recompile 
with debugger support? Is xen-tools (which includes xl) sufficient or 
would you think that I also need to include debug support for gcc as the 
library that is mentioned in /var/log/messages (libgcc_s.so.1) seems to 
belong to the gcc package? Or is this library a red herring that just 
works as the catch-all code getting and finally handling the segfault? 
Please advise. Tx.
> [...]
>> pci          = [ '04:00.0', '0a:08.0', '0a:0b.0' ]
>
> You say in $subject that the failure is with PCI, is that because you've
> tried an HVM domain without and it is ok, or is it just that all your
> HVM domains happen to have passthrough enabled?
I haven't tried HVM domains without PCI passthrough (but PV domains w/o 
PCI passthrough and they did not segfault) so far as all my HVM domains 
require PCI devices (either at least a network card for pfsense - in 
actual facts it's more than one that's being passed through - or a SATA 
controller for my second HVM which is used as a storage VM).

If you think that after the gdb stuff it would still be beneficial to go 
down that route, I am sure I can come up with something.
>
> Ian.
>
Again many thanks Atom2

[-- Attachment #2: output --]
[-- Type: text/plain, Size: 7633 bytes --]

Parsing config from pfsense
libxl: debug: libxl_create.c:1243:do_domain_create: ao 0x7fdcfc226ce0: create: how=(nil) callback=(nil) poller=0x7fdcfc227690
libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=sda spec.backend=unknown
libxl: debug: libxl_device.c:296:libxl__device_disk_set_backend: Disk vdev=sda, using backend phy
libxl: debug: libxl_create.c:699:initiate_domain_create: running bootloader
libxl: debug: libxl_bootloader.c:321:libxl__bootloader_run: not a PV domain, skipping bootloader
libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7fdcfc228098: deregister unregistered
xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0xc12a4
xc: detail: elf_parse_binary: memory: 0x100000 -> 0x1c12a4
xc: info: VIRTUAL MEMORY ARRANGEMENT:
  Loader:        0000000000100000->00000000001c12a4
  Modules:       0000000000000000->0000000000000000
  TOTAL:         0000000000000000->000000001f800000
  ENTRY ADDRESS: 0000000000100000
xc: info: PHYSICAL MEMORY ALLOCATION:
  4KB PAGES: 0x0000000000000200
  2MB PAGES: 0x00000000000000fb
  1GB PAGES: 0x0000000000000000
xc: detail: elf_load_binary: phdr 0 at 0x7fdcfbe26000 -> 0x7fdcfbede12d
libxl: debug: libxl_device.c:257:libxl__device_disk_set_backend: Disk vdev=sda spec.backend=phy
libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7fdcfc227398 wpath=/local/domain/0/backend/vbd/2/2048/state token=3/0: register slotnum=3
libxl: debug: libxl_create.c:1256:do_domain_create: ao 0x7fdcfc226ce0: inprogress: poller=0x7fdcfc227690, flags=i
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fdcfc227398 wpath=/local/domain/0/backend/vbd/2/2048/state token=3/0: event epath=/local/domain/0/backend/vbd/2/2048/state
libxl: debug: libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vbd/2/2048/state wanted state 2 still waiting state 1
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fdcfc227398 wpath=/local/domain/0/backend/vbd/2/2048/state token=3/0: event epath=/local/domain/0/backend/vbd/2/2048/state
libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vbd/2/2048/state wanted state 2 ok
libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7fdcfc227398 wpath=/local/domain/0/backend/vbd/2/2048/state token=3/0: deregister slotnum=3
libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7fdcfc227398: deregister unregistered
libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/block add
libxl: debug: libxl_dm.c:1211:libxl__spawn_local_dm: Spawning device-model /usr/lib/xen/bin/qemu-system-i386 with arguments:
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   /usr/lib/xen/bin/qemu-system-i386
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -xen-domid
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   2
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -chardev
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-2,server,nowait
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -mon
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   chardev=libxl-cmd,mode=control
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -name
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   pfsense
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -global
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   isa-fdc.driveA=
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -serial
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   pty
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -nographic
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -vga
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   cirrus
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -global
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   vga.vram_size_mb=8
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -boot
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   order=c
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -device
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   e1000,id=nic0,netdev=net0,mac=00:16:3e:a1:64:01
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -netdev
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   type=tap,id=net0,ifname=vif2.0-emu,script=no,downscript=no
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -M
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   xenfv
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -m
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   504
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   -drive
libxl: debug: libxl_dm.c:1213:libxl__spawn_local_dm:   file=/etc/xen/guests/disk.d/pfsense.disk,if=scsi,bus=0,unit=0,format=raw,cache=writeback
libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7fdcfc2282d0 wpath=/local/domain/0/device-model/2/state token=3/1: register slotnum=3
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fdcfc2282d0 wpath=/local/domain/0/device-model/2/state token=3/1: event epath=/local/domain/0/device-model/2/state
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fdcfc2282d0 wpath=/local/domain/0/device-model/2/state token=3/1: event epath=/local/domain/0/device-model/2/state
libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7fdcfc2282d0 wpath=/local/domain/0/device-model/2/state token=3/1: deregister slotnum=3
libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7fdcfc2282d0: deregister unregistered
libxl: debug: libxl_qmp.c:707:libxl__qmp_initialize: connected to /var/run/xen/qmp-libxl-2
libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: qmp
libxl: debug: libxl_qmp.c:555:qmp_send_prepare: next qmp command: '{
    "execute": "qmp_capabilities",
    "id": 1
}
'
libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: return
libxl: debug: libxl_qmp.c:555:qmp_send_prepare: next qmp command: '{
    "execute": "query-chardev",
    "id": 2
}
'
libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: return
libxl: debug: libxl_qmp.c:555:qmp_send_prepare: next qmp command: '{
    "execute": "query-vnc",
    "id": 3
}
'
libxl: debug: libxl_qmp.c:299:qmp_handle_response: message type: return
libxl: debug: libxl_event.c:559:libxl__ev_xswatch_register: watch w=0x7fdcfc22b8f8 wpath=/local/domain/0/backend/vif/2/0/state token=3/2: register slotnum=3
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fdcfc22b8f8 wpath=/local/domain/0/backend/vif/2/0/state token=3/2: event epath=/local/domain/0/backend/vif/2/0/state
libxl: debug: libxl_event.c:647:devstate_watch_callback: backend /local/domain/0/backend/vif/2/0/state wanted state 2 still waiting state 1
libxl: debug: libxl_event.c:503:watchfd_callback: watch w=0x7fdcfc22b8f8 wpath=/local/domain/0/backend/vif/2/0/state token=3/2: event epath=/local/domain/0/backend/vif/2/0/state
libxl: debug: libxl_event.c:643:devstate_watch_callback: backend /local/domain/0/backend/vif/2/0/state wanted state 2 ok
libxl: debug: libxl_event.c:596:libxl__ev_xswatch_deregister: watch w=0x7fdcfc22b8f8 wpath=/local/domain/0/backend/vif/2/0/state token=3/2: deregister slotnum=3
libxl: debug: libxl_event.c:608:libxl__ev_xswatch_deregister: watch w=0x7fdcfc22b8f8: deregister unregistered
libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/vif-bridge online
libxl: debug: libxl_device.c:959:device_hotplug: calling hotplug script: /etc/xen/scripts/vif-bridge add
Segmentation fault

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: segfault in xl create for HVM with PCI passthrough

[Ian Campbell]

On Tue, 2014-10-28 at 16:39 +0100, Atom2 wrote:
> > Please can you run the command under gdb and grab a back trace. It would
> > also be useful to "xl -vvv create pfsense".
> >
> First of all attached please find the output of xl -vvv create pfsense. 
> I decided to attach a file as most of the output lines are longer than 
> 80 chars and therefore would most likely be folded by eMail clients.

Thanks.

> In terms of the last message before the segfault in my attached file it 
> seems to me that the bridge stuff was setup correctly as per the 
> following commands:

Agreed, it seems the crash happens after that sometime.

> With regards to gdb: I can certainly run the command under gdb after 
> including debug support to the executables - that's no big deal.
> I would, however, ask for your advice as to what I need to recompile 
> with debugger support? Is xen-tools (which includes xl) sufficient

I think just the Xen bits would be sufficient, at least to start with.

>  or 
> would you think that I also need to include debug support for gcc as the 
> library that is mentioned in /var/log/messages (libgcc_s.so.1) seems to 
> belong to the gcc package? Or is this library a red herring that just 
> works as the catch-all code getting and finally handling the segfault? 

I'd recommend ignoring it for now, in the event that the backtrace from
just the xen bits suggests a gcc issue that might change. My money right
now is on it being a xen issue though.

> Please advise. Tx.
> > [...]
> >> pci          = [ '04:00.0', '0a:08.0', '0a:0b.0' ]
> >
> > You say in $subject that the failure is with PCI, is that because you've
> > tried an HVM domain without and it is ok, or is it just that all your
> > HVM domains happen to have passthrough enabled?
> I haven't tried HVM domains without PCI passthrough (but PV domains w/o 
> PCI passthrough and they did not segfault) so far as all my HVM domains 
> require PCI devices (either at least a network card for pfsense - in 
> actual facts it's more than one that's being passed through - or a SATA 
> controller for my second HVM which is used as a storage VM).

The VM doesn't need to be fully functional, it just needs to boot
without crashing the toolstack. Just running your existing VM with the
pci line commented out would be useful.

Ian

Re: segfault in xl create for HVM with PCI passthrough

[Atom2]

[-- Attachment #1: Type: text/plain, Size: 4721 bytes --]

To keep the thread together I am again submitting the relevant parts of 
my last answer (which due to an error on my part originally went out to 
Ian only and I only forward it to the list afterwards which resulted in 
an out-of-thread appeareance) together with the (new) results of my gdb 
excercise. Sorry for any confusion this may(might have) cause(d).

Am 28.10.14 um 17:04 schrieb Ian Campbell:
[...]
>> With regards to gdb: I can certainly run the command under gdb after
>> including debug support to the executables - that's no big deal.
>> I would, however, ask for your advice as to what I need to recompile
>> with debugger support? Is xen-tools (which includes xl) sufficient
>
> I think just the Xen bits would be sufficient, at least to start with.
>
>>   or
>> would you think that I also need to include debug support for gcc as the
>> library that is mentioned in /var/log/messages (libgcc_s.so.1) seems to
>> belong to the gcc package? Or is this library a red herring that just
>> works as the catch-all code getting and finally handling the segfault?
>
> I'd recommend ignoring it for now, in the event that the backtrace from
> just the xen bits suggests a gcc issue that might change. My money right
> now is on it being a xen issue though.
>
After recompiling xen-tools with gdb debug support I started the 
following command:
# gdb --args /usr/sbin/xl create pfsense -c

Please find the command's screen output after its start up to the 
segfault including the output of the bt command after the segfault in 
the attached document named "create".

Furthermore I did the same for the destroy command:
# gdb --args /usr/sbin/xl destroy pfsense

The output of this command is in the attached document named "destroy".

I haven't got much experience with gdb yet so I am unable to interpret 
the outcome of either. Also if there's more/different stuff required, 
please advise me what to do next. Tx.

>>> [...]
>>>> pci          = [ '04:00.0', '0a:08.0', '0a:0b.0' ]
>>>
>>> You say in $subject that the failure is with PCI, is that because you've
>>> tried an HVM domain without and it is ok, or is it just that all your
>>> HVM domains happen to have passthrough enabled?
>> I haven't tried HVM domains without PCI passthrough (but PV domains w/o
>> PCI passthrough and they did not segfault) so far as all my HVM domains
>> require PCI devices (either at least a network card for pfsense - in
>> actual facts it's more than one that's being passed through - or a SATA
>> controller for my second HVM which is used as a storage VM).
>
> The VM doesn't need to be fully functional, it just needs to boot
> without crashing the toolstack. Just running your existing VM with the
> pci line commented out would be useful.
Before re-compiling the xen-tools I made a quick test as you suggested
and commented out the pci line from my config file ... and the boot menu 
showed up (which it did not before when the segfault happened).
I did not boot the pfsense vm any further as this might lead to a change 
in my configuration due to missing devices, but to me this at first 
sight seemed to indicate that is has to do with the PCI passthrough 
functionality.
Although as I did not want to boot the machine (and "xl shutdown" did
not work, not even with -F) I then decided to
     xl destroy pfsense
and that printed a segmentation fault message (in both the shell window
where I started the command from and the console window where the
boot-menu was shown) despite no PCI devices being passed through.

To also check PCI passthrough with a PV domain: I added a pci device to
a config file for a PV domain and started that with
     xl create voip -c
The boot menu appeared without issues. I then also tried
     xl destroy voip
from another window and that issued the following error messages in the
shell window (without using any -vvv option):

# xl destroy voip
libxl: error: libxl_pci.c:1247:do_pci_remove: xc_domain_irq_permission
irq=17
libxl: error: libxl_device.c:1127:libxl__wait_for_backend: Backend
/local/domain/0/backend/pci/4/0 not ready
libxl: error: libxl_pci.c:1247:do_pci_remove: xc_domain_irq_permission
irq=16
libxl: error: libxl_device.c:1127:libxl__wait_for_backend: Backend
/local/domain/0/backend/pci/4/0 not ready
libxl: error: libxl_pci.c:1247:do_pci_remove: xc_domain_irq_permission
irq=23
libxl: error: libxl_device.c:1127:libxl__wait_for_backend: Backend
/local/domain/0/backend/pci/4/0 not ready
Segmentation fault

The "Segmentation fault" message also appeared in both the console
window for the domU and the shell window.

This all seems a bit strange to me at the moement, but I am sure with
your help we will arrive at the grounds of this.

Thanks and regards Atom

[-- Attachment #2: create --]
[-- Type: text/plain, Size: 2863 bytes --]

GNU gdb (Gentoo 7.6.2 p1) 7.6.2
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.gentoo.org/>...
Reading symbols from /usr/sbin/xl...Reading symbols from /usr/lib64/debug/usr/sbin/xl.debug...done.
done.
(gdb) run
Starting program: /usr/sbin/xl create pfsense -c
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff7ffa000
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Parsing config from pfsense
xc: info: VIRTUAL MEMORY ARRANGEMENT:
  Loader:        0000000000100000->00000000001c12a4
  Modules:       0000000000000000->0000000000000000
  TOTAL:         0000000000000000->000000001f800000
  ENTRY ADDRESS: 0000000000100000
xc: info: PHYSICAL MEMORY ALLOCATION:
  4KB PAGES: 0x0000000000000200
  2MB PAGES: 0x00000000000000fb
  1GB PAGES: 0x0000000000000000
[New Thread 0x7ffff7ff5700 (LWP 13464)]
[New Thread 0x7ffff7fe6700 (LWP 13574)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7fe6700 (LWP 13574)]
0x00007ffff5882b64 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libgcc_s.so.1
(gdb) bt
#0  0x00007ffff5882b64 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libgcc_s.so.1
#1  0x00007ffff58835cc in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libgcc_s.so.1
#2  0x00007ffff5883945 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libgcc_s.so.1
#3  0x00007ffff58845c6 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libgcc_s.so.1
#4  0x00007ffff588494c in _Unwind_ForcedUnwind () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libgcc_s.so.1
#5  0x00007ffff731a733 in __pthread_unwind () from /lib64/libpthread.so.0
#6  0x00007ffff7311b49 in sigcancel_handler () from /lib64/libpthread.so.0
#7  <signal handler called>
#8  0x00007ffff731ae4d in read () from /lib64/libpthread.so.0
#9  0x00007ffff6b17b53 in read (__nbytes=16, __buf=0x7fffe80008d0, __fd=18) at /usr/include/bits/unistd.h:44
#10 read_all (fd=18, data=data@entry=0x7fffe80008d0, len=len@entry=16, nonblocking=nonblocking@entry=0) at xs.c:374
#11 0x00007ffff6b17c94 in read_message (h=h@entry=0x555555785670, nonblocking=nonblocking@entry=0) at xs.c:1139
#12 0x00007ffff6b18626 in read_thread (arg=0x555555785670) at xs.c:1211
#13 0x00007ffff731332d in start_thread () from /lib64/libpthread.so.0
#14 0x00007ffff704a19d in clone () from /lib64/libc.so.6
(gdb)

[-- Attachment #3: destroy --]
[-- Type: text/plain, Size: 2431 bytes --]

GNU gdb (Gentoo 7.6.2 p1) 7.6.2
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.gentoo.org/>...
Reading symbols from /usr/sbin/xl...Reading symbols from /usr/lib64/debug/usr/sbin/xl.debug...done.
done.
(gdb) run
Starting program: /usr/sbin/xl destroy pfsense
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff7ffa000
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff7ff6700 (LWP 13639)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7ff6700 (LWP 13639)]
0x00007ffff5882b64 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libgcc_s.so.1
(gdb) bt
#0  0x00007ffff5882b64 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libgcc_s.so.1
#1  0x00007ffff58835cc in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libgcc_s.so.1
#2  0x00007ffff5883945 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libgcc_s.so.1
#3  0x00007ffff58845c6 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libgcc_s.so.1
#4  0x00007ffff588494c in _Unwind_ForcedUnwind () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libgcc_s.so.1
#5  0x00007ffff731a733 in __pthread_unwind () from /lib64/libpthread.so.0
#6  0x00007ffff7311b49 in sigcancel_handler () from /lib64/libpthread.so.0
#7  <signal handler called>
#8  0x00007ffff731ae4d in read () from /lib64/libpthread.so.0
#9  0x00007ffff6b17b53 in read (__nbytes=16, __buf=0x7ffff0000a10, __fd=10) at /usr/include/bits/unistd.h:44
#10 read_all (fd=10, data=data@entry=0x7ffff0000a10, len=len@entry=16, nonblocking=nonblocking@entry=0) at xs.c:374
#11 0x00007ffff6b17c94 in read_message (h=h@entry=0x55555577ed40, nonblocking=nonblocking@entry=0) at xs.c:1139
#12 0x00007ffff6b18626 in read_thread (arg=0x55555577ed40) at xs.c:1211
#13 0x00007ffff731332d in start_thread () from /lib64/libpthread.so.0
#14 0x00007ffff704a19d in clone () from /lib64/libc.so.6
(gdb)

[-- Attachment #4: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: segfault in xl create for HVM with PCI passthrough

[Atom2]

Ian,
apologies for pinging this, but I am not sure whether there's anything 
else over and above the answers in my last message (copied below) that 
you are expecting me to provide before being able to judge where and 
what the issue might be?

Many thanks in advance, Atom2

P.S. In case you again require the attachments to my last message, 
please let me know.

Am 29.10.14 um 01:26 schrieb Atom2:
> To keep the thread together I am again submitting the relevant parts of
> my last answer (which due to an error on my part originally went out to
> Ian only and I only forward it to the list afterwards which resulted in
> an out-of-thread appeareance) together with the (new) results of my gdb
> excercise. Sorry for any confusion this may(might have) cause(d).
>
> Am 28.10.14 um 17:04 schrieb Ian Campbell:
> [...]
>>> With regards to gdb: I can certainly run the command under gdb after
>>> including debug support to the executables - that's no big deal.
>>> I would, however, ask for your advice as to what I need to recompile
>>> with debugger support? Is xen-tools (which includes xl) sufficient
>>
>> I think just the Xen bits would be sufficient, at least to start with.
>>
>>>   or
>>> would you think that I also need to include debug support for gcc as the
>>> library that is mentioned in /var/log/messages (libgcc_s.so.1) seems to
>>> belong to the gcc package? Or is this library a red herring that just
>>> works as the catch-all code getting and finally handling the segfault?
>>
>> I'd recommend ignoring it for now, in the event that the backtrace from
>> just the xen bits suggests a gcc issue that might change. My money right
>> now is on it being a xen issue though.
>>
> After recompiling xen-tools with gdb debug support I started the
> following command:
> # gdb --args /usr/sbin/xl create pfsense -c
>
> Please find the command's screen output after its start up to the
> segfault including the output of the bt command after the segfault in
> the attached document named "create".
>
> Furthermore I did the same for the destroy command:
> # gdb --args /usr/sbin/xl destroy pfsense
>
> The output of this command is in the attached document named "destroy".
>
> I haven't got much experience with gdb yet so I am unable to interpret
> the outcome of either. Also if there's more/different stuff required,
> please advise me what to do next. Tx.
>
>>>> [...]
>>>>> pci          = [ '04:00.0', '0a:08.0', '0a:0b.0' ]
>>>>
>>>> You say in $subject that the failure is with PCI, is that because
>>>> you've
>>>> tried an HVM domain without and it is ok, or is it just that all your
>>>> HVM domains happen to have passthrough enabled?
>>> I haven't tried HVM domains without PCI passthrough (but PV domains w/o
>>> PCI passthrough and they did not segfault) so far as all my HVM domains
>>> require PCI devices (either at least a network card for pfsense - in
>>> actual facts it's more than one that's being passed through - or a SATA
>>> controller for my second HVM which is used as a storage VM).
>>
>> The VM doesn't need to be fully functional, it just needs to boot
>> without crashing the toolstack. Just running your existing VM with the
>> pci line commented out would be useful.
> Before re-compiling the xen-tools I made a quick test as you suggested
> and commented out the pci line from my config file ... and the boot menu
> showed up (which it did not before when the segfault happened).
> I did not boot the pfsense vm any further as this might lead to a change
> in my configuration due to missing devices, but to me this at first
> sight seemed to indicate that is has to do with the PCI passthrough
> functionality.
> Although as I did not want to boot the machine (and "xl shutdown" did
> not work, not even with -F) I then decided to
>      xl destroy pfsense
> and that printed a segmentation fault message (in both the shell window
> where I started the command from and the console window where the
> boot-menu was shown) despite no PCI devices being passed through.
>
> To also check PCI passthrough with a PV domain: I added a pci device to
> a config file for a PV domain and started that with
>      xl create voip -c
> The boot menu appeared without issues. I then also tried
>      xl destroy voip
> from another window and that issued the following error messages in the
> shell window (without using any -vvv option):
>
> # xl destroy voip
> libxl: error: libxl_pci.c:1247:do_pci_remove: xc_domain_irq_permission
> irq=17
> libxl: error: libxl_device.c:1127:libxl__wait_for_backend: Backend
> /local/domain/0/backend/pci/4/0 not ready
> libxl: error: libxl_pci.c:1247:do_pci_remove: xc_domain_irq_permission
> irq=16
> libxl: error: libxl_device.c:1127:libxl__wait_for_backend: Backend
> /local/domain/0/backend/pci/4/0 not ready
> libxl: error: libxl_pci.c:1247:do_pci_remove: xc_domain_irq_permission
> irq=23
> libxl: error: libxl_device.c:1127:libxl__wait_for_backend: Backend
> /local/domain/0/backend/pci/4/0 not ready
> Segmentation fault
>
> The "Segmentation fault" message also appeared in both the console
> window for the domU and the shell window.
>
> This all seems a bit strange to me at the moement, but I am sure with
> your help we will arrive at the grounds of this.
>
> Thanks and regards Atom
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>

Re: segfault in xl create for HVM with PCI passthrough

[Atom2]

[-- Attachment #1: Type: text/plain, Size: 831 bytes --]

Am 28.10.14 um 17:04 schrieb Ian Campbell:
> On Tue, 2014-10-28 at 16:39 +0100, Atom2 wrote:
>>> Please can you run the command under gdb and grab a back trace.
>>>

I have now re-compiled a few more pieces with debugging support, namely 
gcc-8.4.3 and glibc and again run the command
	xl create pfsense -c
under gdb. The new (full) backtrace output is attached to this mail and 
might provide you with some more clues.

BTW the same problem also seems to exist for xen-4.4.1/gcc-4.8.3 and was 
found independent of my report - please see further details and a 
discussion at http://forums.gentoo.org/viewtopic-t-1003746.html and the 
related bug report at https://bugs.gentoo.org/show_bug.cgi?id=528690.

Ian - if you (or anybody else) could add any more insight into this, it 
would be very much appreciated.

Thanks again Atom2

[-- Attachment #2: backtrace-xen-4.3.3-r1 --]
[-- Type: text/plain, Size: 11397 bytes --]

vm-host auto [512] # gdb --args xl create pfsense -c
GNU gdb (Gentoo 7.7.1 p1) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from xl...Reading symbols from /usr/lib64/debug//usr/sbin/xl.debug...done.
done.
(gdb) run
Starting program: /usr/sbin/xl create pfsense -c
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Parsing config from pfsense
xc: info: VIRTUAL MEMORY ARRANGEMENT:
  Loader:        0000000000100000->00000000001c10c4
  Modules:       0000000000000000->0000000000000000
  TOTAL:         0000000000000000->000000001f800000
  ENTRY ADDRESS: 0000000000100000
xc: info: PHYSICAL MEMORY ALLOCATION:
  4KB PAGES: 0x0000000000000200
  2MB PAGES: 0x00000000000000fb
  1GB PAGES: 0x0000000000000000
[New Thread 0x7ffff7ff5700 (LWP 2489)]
[New Thread 0x7ffff7fe6700 (LWP 2601)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7fe6700 (LWP 2601)]
0x00007ffff5892624 in execute_stack_op (op_ptr=0x7ffff7329b83 "w\240\001\006\020\b\002w(\020\t\002w0\020\n\002w8\020\v\003w\300",
    op_end=0x7ffff7329b87 "\020\b\002w(\020\t\002w0\020\n\002w8\020\v\003w\300", context=context@entry=0x7ffff7fe5190,
    initial=initial@entry=0) at /var/tmp/portage/sys-devel/gcc-4.8.3/work/gcc-4.8.3/libgcc/unwind-dw2.c:516
516     /var/tmp/portage/sys-devel/gcc-4.8.3/work/gcc-4.8.3/libgcc/unwind-dw2.c: No such file or directory.
(gdb) bt full
#0  0x00007ffff5892624 in execute_stack_op (
    op_ptr=0x7ffff7329b83 "w\240\001\006\020\b\002w(\020\t\002w0\020\n\002w8\020\v\003w\300",
    op_end=0x7ffff7329b87 "\020\b\002w(\020\t\002w0\020\n\002w8\020\v\003w\300", context=context@entry=0x7ffff7fe5190,
    initial=initial@entry=0) at /var/tmp/portage/sys-devel/gcc-4.8.3/work/gcc-4.8.3/libgcc/unwind-dw2.c:516
        stack = {0 <repeats 44 times>, 140737354027168, 140737312803731, 140737354027184, 140737354027488, 140737340660732,
          140737340663016, 140737354027312, 140737312808747, 140737354027328, 140733193388035, 140737340663560, 352, 10, 167, 220,
          0, 0, 0, 0, 140737354129736}
        stack_elt = <optimized out>
#1  0x00007ffff589308c in uw_update_context_1 (context=context@entry=0x7ffff7fe55a0, fs=fs@entry=0x7ffff7fe52f0)
    at /var/tmp/portage/sys-devel/gcc-4.8.3/work/gcc-4.8.3/libgcc/unwind-dw2.c:1424
        exp = <optimized out>
        len = <optimized out>
        orig_context = {reg = {0x7ffff7fe5698, 0x7ffff7fe56a0, 0x0, 0x7ffff7fe56a8, 0x0, 0x0, 0x7ffff7fe56f0, 0x7ffff7fe5180, 0x0,
            0x0, 0x0, 0x0, 0x7ffff7fe56b0, 0x7ffff7fe56b8, 0x7ffff7fe56c0, 0x7ffff7fe56c8, 0x7ffff7fe56f8, 0x0},
          cfa = 0x7ffff7fe5700, ra = 0x7ffff7322e00 <__restore_rt>, lsda = 0x0, bases = {tbase = 0x0, dbase = 0x0,
            func = 0x7ffff7322dff}, flags = 4611686018427387904, version = 0, args_size = 0, by_value = '\000' <repeats 17 times>}
        cfa = <optimized out>
        i = <optimized out>
        tmp_sp = {ptr = 140737354028800, word = 140737354028800}
#2  0x00007ffff5893405 in uw_update_context (context=context@entry=0x7ffff7fe55a0, fs=fs@entry=0x7ffff7fe52f0)
    at /var/tmp/portage/sys-devel/gcc-4.8.3/work/gcc-4.8.3/libgcc/unwind-dw2.c:1506
No locals.
#3  0x00007ffff5894086 in uw_advance_context (fs=0x7ffff7fe52f0, context=0x7ffff7fe55a0)
    at /var/tmp/portage/sys-devel/gcc-4.8.3/work/gcc-4.8.3/libgcc/unwind-dw2.c:1529
No locals.
#4  _Unwind_ForcedUnwind_Phase2 (exc=exc@entry=0x7ffff7fe6d70, context=context@entry=0x7ffff7fe55a0)
    at /var/tmp/portage/sys-devel/gcc-4.8.3/work/gcc-4.8.3/libgcc/unwind.inc:185
        fs = {regs = {reg = {{loc = {reg = 140737340677076, offset = 140737340677076,
                  exp = 0x7ffff7329bd4 "\003w\220\001\020\002\003w\230\001\020\a\003w\240\001\020\020\003w\250\001"},
                how = REG_SAVED_EXP}, {loc = {reg = 140737340677070, offset = 140737340677070,
                  exp = 0x7ffff7329bce "\003w\210\001\020"}, how = REG_SAVED_EXP}, {loc = {reg = 140737340677082,
                  offset = 140737340677082, exp = 0x7ffff7329bda "\003w\230\001\020\a\003w\240\001\020\020\003w\250\001"},
                how = REG_SAVED_EXP}, {loc = {reg = 140737340677064, offset = 140737340677064,
                  exp = 0x7ffff7329bc8 "\003w\200\001\020\001\003w\210\001\020"}, how = REG_SAVED_EXP}, {loc = {
                  reg = 140737340677052, offset = 140737340677052, exp = 0x7ffff7329bbc "\003", <incomplete sequence \360>},
                how = REG_SAVED_EXP}, {loc = {reg = 140737340677046, offset = 140737340677046,
                  exp = 0x7ffff7329bb6 "\003", <incomplete sequence \350>}, how = REG_SAVED_EXP}, {loc = {reg = 140737340677058,
                  offset = 140737340677058, exp = 0x7ffff7329bc2 "\003", <incomplete sequence \370>}, how = REG_SAVED_EXP}, {
                loc = {reg = 140737340677088, offset = 140737340677088,
                  exp = 0x7ffff7329be0 "\003w\240\001\020\020\003w\250\001"}, how = REG_SAVED_EXP}, {loc = {reg = 140737340677001,
                  offset = 140737340677001, exp = 0x7ffff7329b89 "\002w(\020\t\002w0\020\n\002w8\020\v\003w\300"},
                how = REG_SAVED_EXP}, {loc = {reg = 140737340677006, offset = 140737340677006,
                  exp = 0x7ffff7329b8e "\002w0\020\n\002w8\020\v\003w\300"}, how = REG_SAVED_EXP}, {loc = {reg = 140737340677011,
                  offset = 140737340677011, exp = 0x7ffff7329b93 "\002w8\020\v\003w\300"}, how = REG_SAVED_EXP}, {loc = {
                  reg = 140737340677016, offset = 140737340677016, exp = 0x7ffff7329b98 "\003w\300"}, how = REG_SAVED_EXP}, {
                loc = {reg = 140737340677022, offset = 140737340677022, exp = 0x7ffff7329b9e "\003", <incomplete sequence \310>},
                how = REG_SAVED_EXP}, {loc = {reg = 140737340677028, offset = 140737340677028,
                  exp = 0x7ffff7329ba4 "\003", <incomplete sequence \320>}, how = REG_SAVED_EXP}, {loc = {reg = 140737340677034,
                  offset = 140737340677034, exp = 0x7ffff7329baa "\003", <incomplete sequence \330>}, how = REG_SAVED_EXP}, {
                loc = {reg = 140737340677040, offset = 140737340677040, exp = 0x7ffff7329bb0 "\003", <incomplete sequence \340>},
                how = REG_SAVED_EXP}, {loc = {reg = 140737340677094, offset = 140737340677094,
                  exp = 0x7ffff7329be6 "\003w\250\001"}, how = REG_SAVED_EXP}, {loc = {reg = 0, offset = 0, exp = 0x0},
                how = REG_UNSAVED}}, prev = 0x0, cfa_offset = 0, cfa_reg = 0,
            cfa_exp = 0x7ffff7329b82 "\004w\240\001\006\020\b\002w(\020\t\002w0\020\n\002w8\020\v\003w\300", cfa_how = CFA_EXP},
          pc = 0x7ffff7322dff, personality = 0x0, data_align = -8, code_align = 1, retaddr_column = 16, fde_encoding = 27 '\033',
          lsda_encoding = 255 '\377', saw_z = 1 '\001', signal_frame = 1 '\001', eh_ptr = 0x0}
        action = 10
        stop = 0x7ffff73215e0 <unwind_stop>
        stop_argument = 0x7ffff7fe5d30
        code = <optimized out>
        stop_code = <optimized out>
#5  0x00007ffff589440c in _Unwind_ForcedUnwind (exc=0x7ffff7fe6d70, stop=stop@entry=0x7ffff73215e0 <unwind_stop>,
    stop_argument=0x7ffff7fe5d30) at /var/tmp/portage/sys-devel/gcc-4.8.3/work/gcc-4.8.3/libgcc/unwind.inc:207
        this_context = {reg = {0x7ffff7fe5698, 0x7ffff7fe56a0, 0x0, 0x7ffff7fe56a8, 0x0, 0x0, 0x7ffff7fe56d0, 0x0, 0x0, 0x0, 0x0,
            0x0, 0x7ffff7fe56b0, 0x7ffff7fe56b8, 0x7ffff7fe56c0, 0x7ffff7fe56c8, 0x7ffff7fe56d8, 0x0}, cfa = 0x7ffff7fe56e0,
          ra = 0x7ffff7321773 <__GI___pthread_unwind+83>, lsda = 0x0, bases = {tbase = 0x0, dbase = 0x0,
            func = 0x7ffff58943a0 <_Unwind_ForcedUnwind>}, flags = 4611686018427387904, version = 0, args_size = 0,
          by_value = '\000' <repeats 17 times>}
        cur_context = {reg = {0x7ffff7fe5698, 0x7ffff7fe56a0, 0x0, 0x7ffff7fe56a8, 0x0, 0x0, 0x7ffff7fe56f0, 0x0, 0x0, 0x0, 0x0,
            0x0, 0x7ffff7fe56b0, 0x7ffff7fe56b8, 0x7ffff7fe56c0, 0x7ffff7fe56c8, 0x7ffff7fe56f8, 0x0}, cfa = 0x7ffff7fe5700,
          ra = 0x7ffff7322e00 <__restore_rt>, lsda = 0x0, bases = {tbase = 0x0, dbase = 0x0, func = 0x7ffff7322dff},
          flags = 4611686018427387904, version = 0, args_size = 0, by_value = '\000' <repeats 17 times>}
        code = <optimized out>
#6  0x00007ffff7321773 in __GI___pthread_unwind (buf=<optimized out>) at unwind.c:129
        ibuf = <optimized out>
        self = <optimized out>
#7  0x00007ffff7318b89 in __do_cancel () at ../nptl/pthreadP.h:280
No locals.
#8  sigcancel_handler (sig=<optimized out>, si=<optimized out>, ctx=<optimized out>) at nptl-init.c:214
        si = <optimized out>
        ctx = <optimized out>
        pid = <optimized out>
        oldval = <optimized out>
#9  <signal handler called>
No locals.
#10 0x00007ffff7321e8d in read () at ../sysdeps/unix/syscall-template.S:81
No locals.
#11 0x00007ffff6b247c3 in read (__nbytes=16, __buf=0x7fffe80008d0, __fd=14) at /usr/include/bits/unistd.h:44
No locals.
#12 read_all (fd=14, data=data@entry=0x7fffe80008d0, len=len@entry=16, nonblocking=nonblocking@entry=0) at xs.c:374
        done = <optimized out>
#13 0x00007ffff6b24904 in read_message (h=h@entry=0x555555784280, nonblocking=nonblocking@entry=0) at xs.c:1139
        __cancel_buf = {__cancel_jmp_buf = {{__cancel_jmp_buf = {93824994525824, 1282643245906007851, 1, 140737488341120,
                93824994524064, 140737354032896, 1282643245857773355, 1282645892206696235}, __mask_was_saved = 0}}, __pad = {
            0x7ffff7fe5ee0, 0x0, 0x0, 0x0}}
        __cancel_arg = 0x7fffe80008c0
        __not_first_call = <optimized out>
        msg = 0x7fffe80008c0
        body = 0x0
        saved_errno = 0
        ret = -1
#14 0x00007ffff6b25296 in read_thread (arg=0x555555784280) at xs.c:1211
        h = 0x555555784280
        fd = <optimized out>
#15 0x00007ffff731a36d in start_thread (arg=0x7ffff7fe6700) at pthread_create.c:309
        __res = <optimized out>
        pd = 0x7ffff7fe6700
        now = <optimized out>
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737354032896, 1282643245910202155, 1, 140737488341120, 93824994524064,
                140737354032896, 1282643245897619243, 1282642588654638891}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0,
              0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        robust = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread"
#16 0x00007ffff7052e0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.
(gdb)

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel