XCP: signal -7 (Re: XCP: sr driver question wrt vm-migrate)

[yamamoto@valinux.co.jp (YAMAMOTO Takashi)]

hi,

i got another error on vm-migrate.
"signal -7" in the log seems intersting.  does this ring your bell?

YAMAMOTO Takashi

+ date
Thu Jun 17 21:51:44 JST 2010
+ xe vm-migrate live=true uuid=23ecfa58-aa30-ea6a-f9fe-7cb2a5487592 host=67b8b07b-8c50-4677-a511-beb196ea766f
Lost connection to the server.

/var/log/messages:

Jun 17 21:51:40 s1 ovs-cfg-mod: 00007|cfg_mod|INFO|-port.vif4958.1.vm-uuid=23ecfa58-aa30-ea6a-f9fe-7cb2a5487592
Jun 17 21:51:41 s1 xapi: [ warn|s1|2416799 unix-RPC|VM.pool_migrate R:832813c0722b|hotplug] Warning, deleting 'vif' entry from /xapi/4958/hotplug/vif/1
Jun 17 21:51:41 s1 xapi: [error|s1|90 xal_listen|VM (domid: 4958) device_event = ChangeUncooperative false D:0e953bd99071|event] device_event could not be processed because VM record not in database
Jun 17 21:51:47 s1 xapi: [ warn|s1|2417066 inet_rpc|VM.pool_migrate R:fee54e870a4e|xapi] memory_required_bytes = 1080033280 > memory_static_max = 1073741824; clipping
Jun 17 21:51:57 s1 xenguest: Determined the following parameters from xenstore:
Jun 17 21:51:57 s1 xenguest: vcpu/number:1 vcpu/affinity:0 vcpu/weight:0 vcpu/cap:0 nx: 0 viridian: 1 apic: 1 acpi: 1 pae: 1 acpi_s4: 0 acpi_s3: 0
Jun 17 21:52:43 s1 scripts-vif: Called as "add vif" domid:4959 devid:1 mode:vswitch
Jun 17 21:52:44 s1 scripts-vif: Called as "online vif" domid:4959 devid:1 mode:vswitch
Jun 17 21:52:46 s1 scripts-vif: Adding vif4959.1 to xenbr0 with address fe:ff:ff:ff:ff:ff
Jun 17 21:52:46 s1 ovs-vsctl: Called as br-to-vlan xenbr0
Jun 17 21:52:49 s1 ovs-cfg-mod: 00001|cfg|INFO|using "/etc/ovs-vswitchd.conf" as configuration file, "/etc/.ovs-vswitchd.conf.~lock~" as lock file
Jun 17 21:52:49 s1 ovs-cfg-mod: 00002|cfg_mod|INFO|configuration changes:
Jun 17 21:52:49 s1 ovs-cfg-mod: 00003|cfg_mod|INFO|+bridge.xenbr0.port=vif4959.1
Jun 17 21:52:49 s1 ovs-cfg-mod: 00004|cfg_mod|INFO|+port.vif4959.1.net-uuid=9ca059b1-ac1e-8d3f-ff19-e5e74f7b7392
Jun 17 21:52:49 s1 ovs-cfg-mod: 00005|cfg_mod|INFO|+port.vif4959.1.vif-mac=2e:17:01:b0:05:fb
Jun 17 21:52:49 s1 ovs-cfg-mod: 00006|cfg_mod|INFO|+port.vif4959.1.vif-uuid=271f0001-06ca-c9ca-cabc-dc79f412d925
Jun 17 21:52:49 s1 ovs-cfg-mod: 00007|cfg_mod|INFO|+port.vif4959.1.vm-uuid=23ecfa58-aa30-ea6a-f9fe-7cb2a5487592
Jun 17 21:52:51 s1 xapi: [ info|s1|0 thread_zero||watchdog] received signal -7
Jun 17 21:52:51 s1 xapi: [ info|s1|0 thread_zero||watchdog] xapi watchdog exiting.
Jun 17 21:52:51 s1 xapi: [ info|s1|0 thread_zero||watchdog] Fatal: xapi died with signal -7: not restarting (watchdog never restarts on this signal)
Jun 17 21:55:11 s1 python: PERFMON: caught IOError: (socket error (111, 'Connection refused')) - restarting XAPI session
Jun 17 22:00:02 s1 python: PERFMON: caught socket.error: (111 Connection refused) - restarting XAPI session
Jun 17 22:04:48 s1 python: PERFMON: caught socket.error: (111 Connection refused) - restarting XAPI session
Jun 17 22:09:58 s1 python: PERFMON: caught socket.error: (111 Connection refused) - restarting XAPI session
Jun 17 22:14:52 s1 python: PERFMON: caught socket.error: (111 Connection refused) - restarting XAPI session
Jun 17 22:19:38 s1 python: PERFMON: caught socket.error: (111 Connection refused) - restarting XAPI session

> hi,
> 
> thanks.  i'll take a look at the log if it happens again.
> 
> YAMAMOTO Takashi
> 
>> This is usually the result of a failure earier on. Could you grep through the logs to get the whole trace of what went on? Best thing to do is grep for VM.pool_migrate, then find the task reference (the hex string beginning with 'R:' immediately after the 'VM.pool_migrate') and grep for this string in the logs on both the source and destination machines. 
>> 
>> Have a look  through these, and if it's still not obvious what went wrong, post them to the list and we can have a look.
>> 
>> Cheers,
>> 
>> Jon
>> 
>> 
>> On 16 Jun 2010, at 07:19, YAMAMOTO Takashi wrote:
>> 
>>> hi,
>>> 
>>> after making my sr driver defer the attach operation as you suggested,
>>> i got migration work.  thanks!
>>> 
>>> however, when repeating live migration between two hosts for testing,
>>> i got the following error.  it doesn't seem so reproducable.
>>> do you have any idea?
>>> 
>>> YAMAMOTO Takashi
>>> 
>>> + xe vm-migrate live=true uuid=23ecfa58-aa30-ea6a-f9fe-7cb2a5487592 host=67b8b07b-8c50-4677-a511-beb196ea766f
>>> An error occurred during the migration process.
>>> vm: 23ecfa58-aa30-ea6a-f9fe-7cb2a5487592 (CentOS53x64-1)
>>> source: eea41bdd-d2ce-4a9a-bc51-1ca286320296 (s6)
>>> destination: 67b8b07b-8c50-4677-a511-beb196ea766f (s1)
>>> msg: Caught exception INTERNAL_ERROR: [ Xapi_vm_migrate.Remote_failed("unmarshalling result code from remote") ] at last minute during migration
>>> 
>>>> hi,
>>>> 
>>>> i'll try deferring the attach operation to vdi_activate.
>>>> thanks!
>>>> 
>>>> YAMAMOTO Takashi
>>>> 
>>>>> Yup, vdi activate is the way forward.
>>>>> 
>>>>> If you advertise VDI_ACTIVATE and VDI_DEACTIVATE in the 'get_driver_info' response, xapi will call the following during the start-migrate-shutdown lifecycle:
>>>>> 
>>>>> VM start:
>>>>> 
>>>>> host A: VDI.attach
>>>>> host A: VDI.activate
>>>>> 
>>>>> VM migrate:
>>>>> 
>>>>> host B: VDI.attach
>>>>> 
>>>>>  (VM pauses on host A)
>>>>> 
>>>>> host A: VDI.deactivate
>>>>> host B: VDI.activate
>>>>> 
>>>>>  (VM unpauses on host B)
>>>>> 
>>>>> host A: VDI.detach
>>>>> 
>>>>> VM shutdown:
>>>>> 
>>>>> host B: VDI.deactivate
>>>>> host B: VDI.detach
>>>>> 
>>>>> so the disk is never activated on both hosts at once, but it does still go through a period when it is attached to both hosts at once. So you could, for example, check that the disk *could* be attached on the vdi_attach SMAPI call, and actually attach it properly on the vdi_activate call.
>>>>> 
>>>>> Hope this helps,
>>>>> 
>>>>> Jon
>>>>> 
>>>>> 
>>>>> On 7 Jun 2010, at 09:26, YAMAMOTO Takashi wrote:
>>>>> 
>>>>>> hi,
>>>>>> 
>>>>>> on vm-migrate, xapi attaches a vdi on the migrate-to host
>>>>>> before detaching it on the migrate-from host.
>>>>>> unfortunately it doesn't work for our product, which doesn't
>>>>>> provide a way to attach a volume to multiple hosts at the same time.
>>>>>> is VDI_ACTIVATE something what i can use as a workaround?
>>>>>> or any other suggestions?
>>>>>> 
>>>>>> YAMAMOTO Takashi
>>>>>> 
>>>>>> _______________________________________________
>>>>>> Xen-devel mailing list
>>>>>> Xen-devel@lists.xensource.com
>>>>>> http://lists.xensource.com/xen-devel
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> Xen-devel mailing list
>>>>> Xen-devel@lists.xensource.com
>>>>> http://lists.xensource.com/xen-devel
>>>> 
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xensource.com
>>>> http://lists.xensource.com/xen-devel
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel