segv in osevent_release_nexus with libxl backend to libvirt

[Ian Campbell <Ian.Campbell@citrix.com>]

Hi,

I'm seeing quite a few of these when shutting down domains:

        Program received signal SIGSEGV, Segmentation fault.
        [Switching to Thread 0xb47f9420 (LWP 3322)]
        0xb16d2f20 in osevent_release_nexus (gc=0xb47f88bc, nexi_idle=0x2a0968fc, nexus=0x0) at libxl_event.c:119
        119	libxl_event.c: No such file or directory.
        (gdb) bt
        #0  0xb16d2f20 in osevent_release_nexus (gc=0xb47f88bc, nexi_idle=0x2a0968fc, nexus=0x0) at libxl_event.c:119
        #1  0xb16d3a5c in osevent_hook_pre_release (nexus=0x2a096954, nexi_idle=<optimized out>, ev=0x2a096940, gc=0xb47f88bc) at libxl_event.c:149
        #2  libxl__ev_fd_deregister (gc=0xb47f88bc, ev=0x2a096940) at libxl_event.c:231
        #3  0xb16a47e0 in libxl_ctx_free (ctx=0x2a096888) at libxl.c:166
        #4  0xb171814e in libxlDomainObjPrivateDispose () from /opt/libvirt/lib/libvirt/connection-driver/libvirt_driver_libxl.so
        #5  0xb6c69176 in virObjectUnref () from /opt/libvirt/lib/libvirt.so.0
        #6  0xb17181d2 in libxlDomainObjPrivateFree () from /opt/libvirt/lib/libvirt/connection-driver/libvirt_driver_libxl.so
        #7  0xb6c9c0da in virDomainObjDispose () from /opt/libvirt/lib/libvirt.so.0
        #8  0xb6c69176 in virObjectUnref () from /opt/libvirt/lib/libvirt.so.0
        #9  0xb6c9ca26 in virDomainObjListRemove () from /opt/libvirt/lib/libvirt.so.0
        #10 0xb171c548 in libxlDomainDestroyFlags () from /opt/libvirt/lib/libvirt/connection-driver/libvirt_driver_libxl.so
        #11 0xb171c5fa in libxlDomainDestroy () from /opt/libvirt/lib/libvirt/connection-driver/libvirt_driver_libxl.so
        #12 0xb6d292f8 in virDomainDestroy () from /opt/libvirt/lib/libvirt.so.0
        #13 0x2a019d8a in remoteDispatchDomainDestroy ()
        #14 0x2a019cb8 in remoteDispatchDomainDestroyHelper ()
        #15 0x2a058392 in virNetServerProgramDispatchCall ()
        #16 0x2a057fde in virNetServerProgramDispatch ()
        #17 0x2a052814 in virNetServerProcessMsg ()
        #18 0x2a0528cc in virNetServerHandleJob ()
        #19 0xb6c82714 in virThreadPoolWorker () from /opt/libvirt/lib/libvirt.so.0
        #20 0xb6c82198 in virThreadHelper () from /opt/libvirt/lib/libvirt.so.0
        #21 0xb6b84ebc in start_thread () from /lib/arm-linux-gnueabihf/libpthread.so.0
        #22 0xb6b34a38 in ?? () from /lib/arm-linux-gnueabihf/libc.so.6
        #23 0xb6b34a38 in ?? () from /lib/arm-linux-gnueabihf/libc.so.6
        Backtrace stopped: previous frame identical to this frame (corrupt stack?)
        
This is on ARM but I don't think this appears to be arch specific at
first glance. The bit from virObjectUnref->SEGV appears to be the same
each time, but the leadin can be different:
        (gdb) bt
        #0  0xb16d2f20 in osevent_release_nexus (gc=0xbefff51c, nexi_idle=0x2a09411c, nexus=0x0) at libxl_event.c:119
        #1  0xb16d3a5c in osevent_hook_pre_release (nexus=0x2a094174, nexi_idle=<optimized out>, ev=0x2a094160, gc=0xbefff51c) at libxl_event.c:149
        #2  libxl__ev_fd_deregister (gc=0xbefff51c, ev=0x2a094160) at libxl_event.c:231
        #3  0xb16a47e0 in libxl_ctx_free (ctx=0x2a0940a8) at libxl.c:166
        #4  0xb171814e in libxlDomainObjPrivateDispose () from /opt/libvirt/lib/libvirt/connection-driver/libvirt_driver_libxl.so
        #5  0xb6c69176 in virObjectUnref () from /opt/libvirt/lib/libvirt.so.0
        #6  0xb1717696 in libxlDomainObjTimerEventHookInfoFree () from /opt/libvirt/lib/libvirt/connection-driver/libvirt_driver_libxl.so
        #7  0xb6c3eae4 in virEventPollCleanupTimeouts () from /opt/libvirt/lib/libvirt.so.0
        #8  0xb6c3f0f2 in virEventPollRunOnce () from /opt/libvirt/lib/libvirt.so.0
        #9  0xb6c3d2fc in virEventRunDefaultImpl () from /opt/libvirt/lib/libvirt.so.0
        #10 0x2a05495a in virNetServerRun ()
        #11 0x2a01297c in main ()
        
Perhaps that's just an artefact of the reference counting dropping to to
zero in a different order not really relevant.

I remember that a few releases ago we had some issues of this type, but
I thought they had all be resolved.

I'm running up to date versions of Xen:
        commit 28b4baacd599e8c10e6dac055f6a939bb730fb8a
        Author: Jan Beulich <jbeulich@suse.com>
        Date:   Tue Nov 25 10:08:57 2014 +0100
        
and of libvirt:
        commit 6c79469ccc3245b9572dcaabe8cd620ba3fabfad
        Author: Daniel P. Berrange <berrange@redhat.com>
        Date:   Tue Nov 18 17:53:51 2014 +0000

Any ideas?