Test results for save/restore with upstream pv_ops domU kernels

Test results for save/restore with upstream pv_ops domU kernels

[Pasi Kärkkäinen]

Hello,

I just tried some save/restore tests with Fedora 12 Linux 2.6.31.12 kernels.
The exact Fedora kernel versions are: 2.6.31.12-174.2.3.fc12.i686.PAE and 2.6.31.12-174.2.3.fc12.x86_64.

Dom0 for these tests was CentOS 5.4 (Xen 3.1.2).

- F12 32bit 1vcpu PV guest: 
	save+restore OK, BUG() in guest dmesg after restore [1]

- F12 64bit 1vcpu PV guest:
	save+restore OK, BUG() in guest dmesg after restore [2]

- F12 32bit 2vcpu PV guest:
	save doesn't work, guest stays as "migrating-f12test32" in "xm list" forever and has to be "xm destroy"ed.

- F12 64bit 2vcpu PV guest:
	save doesn't work, guest stays as "migrating-f12test64" in "xm list" forever and has to be "xm destroy"ed.


What's the best way to debug failing "xm save" ? There was no errors in "xm log", or in "xm dmesg".
Also the guest "xm console" doesn't show anything before it dies.

Is it possible some of the save/restore related patches didn't make it to 2.6.31.x stable kernels? 


BUG()s after "xm restore":

[1] F12 32b 1vcpu:

suspending xenstore...
BUG: sleeping function called from invalid context at kernel/mutex.c:94
in_atomic(): 0, irqs_disabled(): 1, pid: 1136, name: kstop/0
Pid: 1136, comm: kstop/0 Tainted: G        W  2.6.31.12-174.2.3.fc12.i686.PAE #1
Call Trace:
 [<c042f3fa>] __might_sleep+0xc6/0xcb
 [<c077a276>] mutex_lock+0x1d/0x3c
 [<c06404a3>] dpm_resume_noirq+0x17/0xe1
 [<c060959d>] xen_suspend+0xc4/0xcf
 [<c046f96c>] stop_cpu+0x75/0xb4
 [<c044d440>] worker_thread+0x13c/0x1bc
 [<c046f8f7>] ? stop_cpu+0x0/0xb4
 [<c0450db9>] ? autoremove_wake_function+0x0/0x34
 [<c044d304>] ? worker_thread+0x0/0x1bc
 [<c0450b0f>] kthread+0x70/0x75
 [<c0450a9f>] ? kthread+0x0/0x75
 [<c0409c07>] kernel_thread_helper+0x7/0x10
eth0: no IPv6 routers present


[2] F12 64b 1vcpu:

suspending xenstore...
trying to map vcpu_info 0 at ffffc9000000b020, mfn 1d9997, offset 32
cpu 0 using vcpu_info at ffffc9000000b020
BUG: sleeping function called from invalid context at kernel/mutex.c:94
in_atomic(): 0, irqs_disabled(): 1, pid: 1112, name: kstop/0
Pid: 1112, comm: kstop/0 Tainted: G        W  2.6.31.12-174.2.3.fc12.x86_64 #1
Call Trace:
 [<ffffffff8104026a>] __might_sleep+0xe6/0xe8
 [<ffffffff8141c844>] mutex_lock+0x22/0x4e
 [<ffffffff812b266a>] dpm_resume_noirq+0x21/0x11f
 [<ffffffff812752a9>] xen_suspend+0xca/0xd1
 [<ffffffff8108c33e>] stop_cpu+0x8c/0xd2
 [<ffffffff810635a0>] worker_thread+0x18a/0x224
 [<ffffffff81067b37>] ? autoremove_wake_function+0x0/0x39
 [<ffffffff8141d6e9>] ? _spin_unlock_irqrestore+0x19/0x1b
 [<ffffffff81063416>] ? worker_thread+0x0/0x224
 [<ffffffff810677b5>] kthread+0x91/0x99
 [<ffffffff81012daa>] child_rip+0xa/0x20
 [<ffffffff81011f97>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff8101271d>] ? retint_restore_args+0x5/0x6
 [<ffffffff81012da0>] ? child_rip+0x0/0x20


-- Pasi

Re: Test results for save/restore with upstream pv_ops domU kernels

[Ian Campbell]

On Thu, 2010-01-28 at 21:25 +0000, Pasi Kärkkäinen wrote:
> Hello,
> 
> I just tried some save/restore tests with Fedora 12 Linux 2.6.31.12 kernels.
> The exact Fedora kernel versions are: 2.6.31.12-174.2.3.fc12.i686.PAE and 2.6.31.12-174.2.3.fc12.x86_64.
> 
> Dom0 for these tests was CentOS 5.4 (Xen 3.1.2).
> 
> - F12 32bit 1vcpu PV guest: 
> 	save+restore OK, BUG() in guest dmesg after restore [1]
> 
> - F12 64bit 1vcpu PV guest:
> 	save+restore OK, BUG() in guest dmesg after restore [2]

I think those are the same underlying bug and are fixed by 
http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=777df2b017ef34b2d1a172bf769582158839a860

> 
> - F12 32bit 2vcpu PV guest:
> 	save doesn't work, guest stays as "migrating-f12test32" in "xm list" forever and has to be "xm destroy"ed.
> 
> - F12 64bit 2vcpu PV guest:
> 	save doesn't work, guest stays as "migrating-f12test64" in "xm list" forever and has to be "xm destroy"ed.
> 
> 
> What's the best way to debug failing "xm save" ? There was no errors in "xm log", or in "xm dmesg".

I think you might see some stuff in /var/log/xen/something but I don't
have any particular tips apart from "add printf/printk".

> Also the guest "xm console" doesn't show anything before it dies.
> 
> Is it possible some of the save/restore related patches didn't make it to 2.6.31.x stable kernels? 

AFAIK they only went into the 2.6.32 stable branch. Unfortunately I
think the 2.6.31 stable series has come to an end now.

Ian.

Re: Test results for save/restore with upstream pv_ops domU kernels

[Pasi Kärkkäinen]

On Fri, Jan 29, 2010 at 10:35:32AM +0000, Ian Campbell wrote:
> On Thu, 2010-01-28 at 21:25 +0000, Pasi Kärkkäinen wrote:
> > Hello,
> > 
> > I just tried some save/restore tests with Fedora 12 Linux 2.6.31.12 kernels.
> > The exact Fedora kernel versions are: 2.6.31.12-174.2.3.fc12.i686.PAE and 2.6.31.12-174.2.3.fc12.x86_64.
> > 
> > Dom0 for these tests was CentOS 5.4 (Xen 3.1.2).
> > 
> > - F12 32bit 1vcpu PV guest: 
> > 	save+restore OK, BUG() in guest dmesg after restore [1]
> > 
> > - F12 64bit 1vcpu PV guest:
> > 	save+restore OK, BUG() in guest dmesg after restore [2]
> 
> I think those are the same underlying bug and are fixed by 
> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=777df2b017ef34b2d1a172bf769582158839a860
> 

Ok. 

There was also this traceback in the beginning of boot, on all 32bit/64bit, 1vcpu2/vcpu guest combinations:

Performance Counters: Core2 events, Intel PMU driver.
------------[ cut here ]------------
WARNING: at arch/x86/kernel/apic/apic.c:247 native_apic_write_dummy+0x32/0x3e() (Not tainted)
Modules linked in:
Pid: 0, comm: swapper Not tainted 2.6.31.12-174.2.3.fc12.i686.PAE #1
Call Trace:
 [<c043db4b>] warn_slowpath_common+0x70/0x87
 [<c041cfb2>] ? native_apic_write_dummy+0x32/0x3e
 [<c043db74>] warn_slowpath_null+0x12/0x15
 [<c041cfb2>] native_apic_write_dummy+0x32/0x3e
 [<c0411e04>] perf_counters_lapic_init+0x30/0x32
 [<c09b3b1b>] init_hw_perf_counters+0x2bc/0x355
 [<c09b3628>] identify_boot_cpu+0x21/0x23
 [<c09b378e>] check_bugs+0xb/0xdc
 [<c047fd73>] ? delayacct_init+0x47/0x4c
 [<c09ab8b4>] start_kernel+0x31c/0x330
 [<c09ab081>] i386_start_kernel+0x70/0x77
 [<c09ae2bb>] xen_start_kernel+0x4b9/0x4c1
 [<c04090a1>] ? syscall_exit+0x1/0x16
---[ end trace a7919e7f17c0a725 ]---

Full boot logs here:
http://pasik.reaktio.net/xen/debug/fedora/


> > 
> > - F12 32bit 2vcpu PV guest:
> > 	save doesn't work, guest stays as "migrating-f12test32" in "xm list" forever and has to be "xm destroy"ed.
> > 
> > - F12 64bit 2vcpu PV guest:
> > 	save doesn't work, guest stays as "migrating-f12test64" in "xm list" forever and has to be "xm destroy"ed.
> > 
> > 
> > What's the best way to debug failing "xm save" ? There was no errors in "xm log", or in "xm dmesg".
> 
> I think you might see some stuff in /var/log/xen/something but I don't
> have any particular tips apart from "add printf/printk".
> 

I'll check /var/log/xen/.

> > Also the guest "xm console" doesn't show anything before it dies.
> > 
> > Is it possible some of the save/restore related patches didn't make it to 2.6.31.x stable kernels? 
> 
> AFAIK they only went into the 2.6.32 stable branch. Unfortunately I
> think the 2.6.31 stable series has come to an end now.
> 

Ok. I'll test 2.6.32.latest aswell.

-- Pasi

Re: Re: Test results for save/restore with upstream pv_ops domU kernels, 2.6.32.7 works OK

[Pasi Kärkkäinen]

On Fri, Jan 29, 2010 at 12:53:38PM +0200, Pasi Kärkkäinen wrote:
> On Fri, Jan 29, 2010 at 10:35:32AM +0000, Ian Campbell wrote:
> > On Thu, 2010-01-28 at 21:25 +0000, Pasi Kärkkäinen wrote:
> > > Hello,
> > > 
> > > I just tried some save/restore tests with Fedora 12 Linux 2.6.31.12 kernels.
> > > The exact Fedora kernel versions are: 2.6.31.12-174.2.3.fc12.i686.PAE and 2.6.31.12-174.2.3.fc12.x86_64.
> > > 
> > > Dom0 for these tests was CentOS 5.4 (Xen 3.1.2).
> > > 
> > > - F12 32bit 1vcpu PV guest: 
> > > 	save+restore OK, BUG() in guest dmesg after restore [1]
> > > 
> > > - F12 64bit 1vcpu PV guest:
> > > 	save+restore OK, BUG() in guest dmesg after restore [2]
> > 
> > I think those are the same underlying bug and are fixed by 
> > http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=777df2b017ef34b2d1a172bf769582158839a860
> > 
> 
> Ok. 
> 
> There was also this traceback in the beginning of boot, on all 32bit/64bit, 1vcpu2/vcpu guest combinations:
> 
> Performance Counters: Core2 events, Intel PMU driver.
> ------------[ cut here ]------------
> WARNING: at arch/x86/kernel/apic/apic.c:247 native_apic_write_dummy+0x32/0x3e() (Not tainted)
> Modules linked in:
> Pid: 0, comm: swapper Not tainted 2.6.31.12-174.2.3.fc12.i686.PAE #1
> Call Trace:
>  [<c043db4b>] warn_slowpath_common+0x70/0x87
>  [<c041cfb2>] ? native_apic_write_dummy+0x32/0x3e
>  [<c043db74>] warn_slowpath_null+0x12/0x15
>  [<c041cfb2>] native_apic_write_dummy+0x32/0x3e
>  [<c0411e04>] perf_counters_lapic_init+0x30/0x32
>  [<c09b3b1b>] init_hw_perf_counters+0x2bc/0x355
>  [<c09b3628>] identify_boot_cpu+0x21/0x23
>  [<c09b378e>] check_bugs+0xb/0xdc
>  [<c047fd73>] ? delayacct_init+0x47/0x4c
>  [<c09ab8b4>] start_kernel+0x31c/0x330
>  [<c09ab081>] i386_start_kernel+0x70/0x77
>  [<c09ae2bb>] xen_start_kernel+0x4b9/0x4c1
>  [<c04090a1>] ? syscall_exit+0x1/0x16
> ---[ end trace a7919e7f17c0a725 ]---
> 
> Full boot logs here:
> http://pasik.reaktio.net/xen/debug/fedora/
> 


This boot-time traceback disappeared when I updated the guest to 2.6.32.7.


> 
> > > 
> > > - F12 32bit 2vcpu PV guest:
> > > 	save doesn't work, guest stays as "migrating-f12test32" in "xm list" forever and has to be "xm destroy"ed.
> > > 
> > > - F12 64bit 2vcpu PV guest:
> > > 	save doesn't work, guest stays as "migrating-f12test64" in "xm list" forever and has to be "xm destroy"ed.
> > > 
> > > 
> > > What's the best way to debug failing "xm save" ? There was no errors in "xm log", or in "xm dmesg".
> > 
> > I think you might see some stuff in /var/log/xen/something but I don't
> > have any particular tips apart from "add printf/printk".
> > 
> 
> I'll check /var/log/xen/.
> 
> > > Also the guest "xm console" doesn't show anything before it dies.
> > > 
> > > Is it possible some of the save/restore related patches didn't make it to 2.6.31.x stable kernels? 
> > 
> > AFAIK they only went into the 2.6.32 stable branch. Unfortunately I
> > think the 2.6.31 stable series has come to an end now.
> > 
> 
> Ok. I'll test 2.6.32.latest aswell.
> 

I grabbed upstream kernel.org Linux 2.6.32.7, and tested the following combinations:

- F12 32bit 1vcpu PV guest
- F12 32bit 2vcpu PV guest
- F12 64bit 1vcpu PV guest
- F12 64bit 2vcpu PV guest

save+restore was successfull for all of the above guests running 2.6.32.7. 
No BUGs or tracebacks anymore.

Any tips for git magic to get all the recent save/restore fixes that went to 2.6.32.x,
so I could send them to Fedora people to apply to F12 kernel? 

Thanks!

-- Pasi

Re: Re: Test results for save/restore with upstream pv_ops domU kernels, 2.6.32.7 works OK

[Andrew Jones]

On 02/01/2010 04:10 PM, Pasi Kärkkäinen wrote:
> On Fri, Jan 29, 2010 at 12:53:38PM +0200, Pasi Kärkkäinen wrote:
>> On Fri, Jan 29, 2010 at 10:35:32AM +0000, Ian Campbell wrote:
>>> On Thu, 2010-01-28 at 21:25 +0000, Pasi Kärkkäinen wrote:
>>>> Hello,
>>>>
>>>> I just tried some save/restore tests with Fedora 12 Linux 2.6.31.12 kernels.
>>>> The exact Fedora kernel versions are: 2.6.31.12-174.2.3.fc12.i686.PAE and 2.6.31.12-174.2.3.fc12.x86_64.
>>>>
>>>> Dom0 for these tests was CentOS 5.4 (Xen 3.1.2).
>>>>
>>>> - F12 32bit 1vcpu PV guest: 
>>>> 	save+restore OK, BUG() in guest dmesg after restore [1]
>>>>
>>>> - F12 64bit 1vcpu PV guest:
>>>> 	save+restore OK, BUG() in guest dmesg after restore [2]
>>>
>>> I think those are the same underlying bug and are fixed by 
>>> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=777df2b017ef34b2d1a172bf769582158839a860
>>>
>>
>> Ok. 
>>
>> There was also this traceback in the beginning of boot, on all 32bit/64bit, 1vcpu2/vcpu guest combinations:
>>
>> Performance Counters: Core2 events, Intel PMU driver.
>> ------------[ cut here ]------------
>> WARNING: at arch/x86/kernel/apic/apic.c:247 native_apic_write_dummy+0x32/0x3e() (Not tainted)
>> Modules linked in:
>> Pid: 0, comm: swapper Not tainted 2.6.31.12-174.2.3.fc12.i686.PAE #1
>> Call Trace:
>>  [<c043db4b>] warn_slowpath_common+0x70/0x87
>>  [<c041cfb2>] ? native_apic_write_dummy+0x32/0x3e
>>  [<c043db74>] warn_slowpath_null+0x12/0x15
>>  [<c041cfb2>] native_apic_write_dummy+0x32/0x3e
>>  [<c0411e04>] perf_counters_lapic_init+0x30/0x32
>>  [<c09b3b1b>] init_hw_perf_counters+0x2bc/0x355
>>  [<c09b3628>] identify_boot_cpu+0x21/0x23
>>  [<c09b378e>] check_bugs+0xb/0xdc
>>  [<c047fd73>] ? delayacct_init+0x47/0x4c
>>  [<c09ab8b4>] start_kernel+0x31c/0x330
>>  [<c09ab081>] i386_start_kernel+0x70/0x77
>>  [<c09ae2bb>] xen_start_kernel+0x4b9/0x4c1
>>  [<c04090a1>] ? syscall_exit+0x1/0x16
>> ---[ end trace a7919e7f17c0a725 ]---
>>
>> Full boot logs here:
>> http://pasik.reaktio.net/xen/debug/fedora/
>>
> 
> 
> This boot-time traceback disappeared when I updated the guest to 2.6.32.7.
> 
> 
>>
>>>>
>>>> - F12 32bit 2vcpu PV guest:
>>>> 	save doesn't work, guest stays as "migrating-f12test32" in "xm list" forever and has to be "xm destroy"ed.
>>>>
>>>> - F12 64bit 2vcpu PV guest:
>>>> 	save doesn't work, guest stays as "migrating-f12test64" in "xm list" forever and has to be "xm destroy"ed.
>>>>
>>>>
>>>> What's the best way to debug failing "xm save" ? There was no errors in "xm log", or in "xm dmesg".
>>>
>>> I think you might see some stuff in /var/log/xen/something but I don't
>>> have any particular tips apart from "add printf/printk".
>>>
>>
>> I'll check /var/log/xen/.
>>
>>>> Also the guest "xm console" doesn't show anything before it dies.
>>>>
>>>> Is it possible some of the save/restore related patches didn't make it to 2.6.31.x stable kernels? 
>>>
>>> AFAIK they only went into the 2.6.32 stable branch. Unfortunately I
>>> think the 2.6.31 stable series has come to an end now.
>>>
>>
>> Ok. I'll test 2.6.32.latest aswell.
>>
> 
> I grabbed upstream kernel.org Linux 2.6.32.7, and tested the following combinations:
> 
> - F12 32bit 1vcpu PV guest
> - F12 32bit 2vcpu PV guest
> - F12 64bit 1vcpu PV guest
> - F12 64bit 2vcpu PV guest
> 
> save+restore was successfull for all of the above guests running 2.6.32.7. 
> No BUGs or tracebacks anymore.
> 
> Any tips for git magic to get all the recent save/restore fixes that went to 2.6.32.x,
> so I could send them to Fedora people to apply to F12 kernel? 
> 

This is great news, but it might a good idea to try 2 or more
save-restore rounds in row first, if you haven't already. In the past
I've seen 1 save/restore work, but then the 2nd round fail. Although,
usually there's some symptom of badness on the 1st round as well.

Thanks,
Andrew

Re: Re: Test results for save/restore with upstream pv_ops domU kernels, 2.6.32.7 works OK

[Pasi Kärkkäinen]

On Mon, Feb 01, 2010 at 04:26:05PM +0100, Andrew Jones wrote:
> On 02/01/2010 04:10 PM, Pasi Kärkkäinen wrote:
> > On Fri, Jan 29, 2010 at 12:53:38PM +0200, Pasi Kärkkäinen wrote:
> >> On Fri, Jan 29, 2010 at 10:35:32AM +0000, Ian Campbell wrote:
> >>> On Thu, 2010-01-28 at 21:25 +0000, Pasi Kärkkäinen wrote:
> >>>> Hello,
> >>>>
> >>>> I just tried some save/restore tests with Fedora 12 Linux 2.6.31.12 kernels.
> >>>> The exact Fedora kernel versions are: 2.6.31.12-174.2.3.fc12.i686.PAE and 2.6.31.12-174.2.3.fc12.x86_64.
> >>>>
> >>>> Dom0 for these tests was CentOS 5.4 (Xen 3.1.2).
> >>>>
> >>>> - F12 32bit 1vcpu PV guest: 
> >>>> 	save+restore OK, BUG() in guest dmesg after restore [1]
> >>>>
> >>>> - F12 64bit 1vcpu PV guest:
> >>>> 	save+restore OK, BUG() in guest dmesg after restore [2]
> >>>
> >>> I think those are the same underlying bug and are fixed by 
> >>> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=777df2b017ef34b2d1a172bf769582158839a860
> >>>
> >>
> >> Ok. 
> >>
> >> There was also this traceback in the beginning of boot, on all 32bit/64bit, 1vcpu2/vcpu guest combinations:
> >>
> >> Performance Counters: Core2 events, Intel PMU driver.
> >> ------------[ cut here ]------------
> >> WARNING: at arch/x86/kernel/apic/apic.c:247 native_apic_write_dummy+0x32/0x3e() (Not tainted)
> >> Modules linked in:
> >> Pid: 0, comm: swapper Not tainted 2.6.31.12-174.2.3.fc12.i686.PAE #1
> >> Call Trace:
> >>  [<c043db4b>] warn_slowpath_common+0x70/0x87
> >>  [<c041cfb2>] ? native_apic_write_dummy+0x32/0x3e
> >>  [<c043db74>] warn_slowpath_null+0x12/0x15
> >>  [<c041cfb2>] native_apic_write_dummy+0x32/0x3e
> >>  [<c0411e04>] perf_counters_lapic_init+0x30/0x32
> >>  [<c09b3b1b>] init_hw_perf_counters+0x2bc/0x355
> >>  [<c09b3628>] identify_boot_cpu+0x21/0x23
> >>  [<c09b378e>] check_bugs+0xb/0xdc
> >>  [<c047fd73>] ? delayacct_init+0x47/0x4c
> >>  [<c09ab8b4>] start_kernel+0x31c/0x330
> >>  [<c09ab081>] i386_start_kernel+0x70/0x77
> >>  [<c09ae2bb>] xen_start_kernel+0x4b9/0x4c1
> >>  [<c04090a1>] ? syscall_exit+0x1/0x16
> >> ---[ end trace a7919e7f17c0a725 ]---
> >>
> >> Full boot logs here:
> >> http://pasik.reaktio.net/xen/debug/fedora/
> >>
> > 
> > 
> > This boot-time traceback disappeared when I updated the guest to 2.6.32.7.
> > 
> > 
> >>
> >>>>
> >>>> - F12 32bit 2vcpu PV guest:
> >>>> 	save doesn't work, guest stays as "migrating-f12test32" in "xm list" forever and has to be "xm destroy"ed.
> >>>>
> >>>> - F12 64bit 2vcpu PV guest:
> >>>> 	save doesn't work, guest stays as "migrating-f12test64" in "xm list" forever and has to be "xm destroy"ed.
> >>>>
> >>>>
> >>>> What's the best way to debug failing "xm save" ? There was no errors in "xm log", or in "xm dmesg".
> >>>
> >>> I think you might see some stuff in /var/log/xen/something but I don't
> >>> have any particular tips apart from "add printf/printk".
> >>>
> >>
> >> I'll check /var/log/xen/.
> >>
> >>>> Also the guest "xm console" doesn't show anything before it dies.
> >>>>
> >>>> Is it possible some of the save/restore related patches didn't make it to 2.6.31.x stable kernels? 
> >>>
> >>> AFAIK they only went into the 2.6.32 stable branch. Unfortunately I
> >>> think the 2.6.31 stable series has come to an end now.
> >>>
> >>
> >> Ok. I'll test 2.6.32.latest aswell.
> >>
> > 
> > I grabbed upstream kernel.org Linux 2.6.32.7, and tested the following combinations:
> > 
> > - F12 32bit 1vcpu PV guest
> > - F12 32bit 2vcpu PV guest
> > - F12 64bit 1vcpu PV guest
> > - F12 64bit 2vcpu PV guest
> > 
> > save+restore was successfull for all of the above guests running 2.6.32.7. 
> > No BUGs or tracebacks anymore.
> > 
> > Any tips for git magic to get all the recent save/restore fixes that went to 2.6.32.x,
> > so I could send them to Fedora people to apply to F12 kernel? 
> > 
> 
> This is great news, but it might a good idea to try 2 or more
> save-restore rounds in row first, if you haven't already. In the past
> I've seen 1 save/restore work, but then the 2nd round fail. Although,
> usually there's some symptom of badness on the 1st round as well.
> 

I forgot to mention that I tried twice with all of the above guests :)
It seems stable.

I just did one more test.. save+restore 5 times in a row, with 4 vcpu PV guest. 
No problems found.

-- Pasi

Re: Test results for save/restore with upstream pv_ops domU kernels, 2.6.33 problems

[Pasi Kärkkäinen]

On Fri, Jan 29, 2010 at 10:35:32AM +0000, Ian Campbell wrote:
> On Thu, 2010-01-28 at 21:25 +0000, Pasi Kärkkäinen wrote:
> > Hello,
> > 
> > I just tried some save/restore tests with Fedora 12 Linux 2.6.31.12 kernels.
> > The exact Fedora kernel versions are: 2.6.31.12-174.2.3.fc12.i686.PAE and 2.6.31.12-174.2.3.fc12.x86_64.
> > 
> > Dom0 for these tests was CentOS 5.4 (Xen 3.1.2).
> > 
> > - F12 32bit 1vcpu PV guest: 
> > 	save+restore OK, BUG() in guest dmesg after restore [1]
> > 
> > - F12 64bit 1vcpu PV guest:
> > 	save+restore OK, BUG() in guest dmesg after restore [2]
> 
> I think those are the same underlying bug and are fixed by 
> http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=777df2b017ef34b2d1a172bf769582158839a860
> 
> > 
> > - F12 32bit 2vcpu PV guest:
> > 	save doesn't work, guest stays as "migrating-f12test32" in "xm list" forever and has to be "xm destroy"ed.
> > 
> > - F12 64bit 2vcpu PV guest:
> > 	save doesn't work, guest stays as "migrating-f12test64" in "xm list" forever and has to be "xm destroy"ed.
> > 
> > 
> > What's the best way to debug failing "xm save" ? There was no errors in "xm log", or in "xm dmesg".
> 
> I think you might see some stuff in /var/log/xen/something but I don't
> have any particular tips apart from "add printf/printk".
> 
> > Also the guest "xm console" doesn't show anything before it dies.
> > 
> > Is it possible some of the save/restore related patches didn't make it to 2.6.31.x stable kernels? 
> 
> AFAIK they only went into the 2.6.32 stable branch. Unfortunately I
> think the 2.6.31 stable series has come to an end now.
> 

Hello,

An user on ##xen just reported that 2.6.32.9 save/restore works OK,
but 2.6.33 fails/crashes.

Some of the patches that went to 2.6.32.x are not in 2.6.33?

-- Pasi

Re: Test results for save/restore with upstream pv_ops domU kernels, 2.6.33 problems

[Ian Campbell]

On Thu, 2010-02-25 at 15:14 +0000, Pasi Kärkkäinen wrote:
> On Fri, Jan 29, 2010 at 10:35:32AM +0000, Ian Campbell wrote:
> > On Thu, 2010-01-28 at 21:25 +0000, Pasi Kärkkäinen wrote:
> > > Hello,
> > > 
> > > I just tried some save/restore tests with Fedora 12 Linux 2.6.31.12 kernels.
> > > The exact Fedora kernel versions are: 2.6.31.12-174.2.3.fc12.i686.PAE and 2.6.31.12-174.2.3.fc12.x86_64.
> > > 
> > > Dom0 for these tests was CentOS 5.4 (Xen 3.1.2).
> > > 
> > > - F12 32bit 1vcpu PV guest: 
> > > 	save+restore OK, BUG() in guest dmesg after restore [1]
> > > 
> > > - F12 64bit 1vcpu PV guest:
> > > 	save+restore OK, BUG() in guest dmesg after restore [2]
> > 
> > I think those are the same underlying bug and are fixed by 
> > http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=777df2b017ef34b2d1a172bf769582158839a860
> > 
> > > 
> > > - F12 32bit 2vcpu PV guest:
> > > 	save doesn't work, guest stays as "migrating-f12test32" in "xm list" forever and has to be "xm destroy"ed.
> > > 
> > > - F12 64bit 2vcpu PV guest:
> > > 	save doesn't work, guest stays as "migrating-f12test64" in "xm list" forever and has to be "xm destroy"ed.
> > > 
> > > 
> > > What's the best way to debug failing "xm save" ? There was no errors in "xm log", or in "xm dmesg".
> > 
> > I think you might see some stuff in /var/log/xen/something but I don't
> > have any particular tips apart from "add printf/printk".
> > 
> > > Also the guest "xm console" doesn't show anything before it dies.
> > > 
> > > Is it possible some of the save/restore related patches didn't make it to 2.6.31.x stable kernels? 
> > 
> > AFAIK they only went into the 2.6.32 stable branch. Unfortunately I
> > think the 2.6.31 stable series has come to an end now.
> > 
> 
> Hello,
> 
> An user on ##xen just reported that 2.6.32.9 save/restore works OK,
> but 2.6.33 fails/crashes.
> 
> Some of the patches that went to 2.6.32.x are not in 2.6.33?

I don't think so -- the acceptance criteria for a stable tree is that
the patch is in mainline already. Must be some new breakage.

Ian.

> 
> -- Pasi
> 

Re: Test results for save/restore with upstream pv_ops domU kernels, 2.6.33 problems

[Pasi Kärkkäinen]

On Thu, Feb 25, 2010 at 03:17:08PM +0000, Ian Campbell wrote:
> On Thu, 2010-02-25 at 15:14 +0000, Pasi Kärkkäinen wrote:
> > On Fri, Jan 29, 2010 at 10:35:32AM +0000, Ian Campbell wrote:
> > > On Thu, 2010-01-28 at 21:25 +0000, Pasi Kärkkäinen wrote:
> > > > Hello,
> > > > 
> > > > I just tried some save/restore tests with Fedora 12 Linux 2.6.31.12 kernels.
> > > > The exact Fedora kernel versions are: 2.6.31.12-174.2.3.fc12.i686.PAE and 2.6.31.12-174.2.3.fc12.x86_64.
> > > > 
> > > > Dom0 for these tests was CentOS 5.4 (Xen 3.1.2).
> > > > 
> > > > - F12 32bit 1vcpu PV guest: 
> > > > 	save+restore OK, BUG() in guest dmesg after restore [1]
> > > > 
> > > > - F12 64bit 1vcpu PV guest:
> > > > 	save+restore OK, BUG() in guest dmesg after restore [2]
> > > 
> > > I think those are the same underlying bug and are fixed by 
> > > http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=commit;h=777df2b017ef34b2d1a172bf769582158839a860
> > > 
> > > > 
> > > > - F12 32bit 2vcpu PV guest:
> > > > 	save doesn't work, guest stays as "migrating-f12test32" in "xm list" forever and has to be "xm destroy"ed.
> > > > 
> > > > - F12 64bit 2vcpu PV guest:
> > > > 	save doesn't work, guest stays as "migrating-f12test64" in "xm list" forever and has to be "xm destroy"ed.
> > > > 
> > > > 
> > > > What's the best way to debug failing "xm save" ? There was no errors in "xm log", or in "xm dmesg".
> > > 
> > > I think you might see some stuff in /var/log/xen/something but I don't
> > > have any particular tips apart from "add printf/printk".
> > > 
> > > > Also the guest "xm console" doesn't show anything before it dies.
> > > > 
> > > > Is it possible some of the save/restore related patches didn't make it to 2.6.31.x stable kernels? 
> > > 
> > > AFAIK they only went into the 2.6.32 stable branch. Unfortunately I
> > > think the 2.6.31 stable series has come to an end now.
> > > 
> > 
> > Hello,
> > 
> > An user on ##xen just reported that 2.6.32.9 save/restore works OK,
> > but 2.6.33 fails/crashes.
> > 
> > Some of the patches that went to 2.6.32.x are not in 2.6.33?
> 
> I don't think so -- the acceptance criteria for a stable tree is that
> the patch is in mainline already. Must be some new breakage.
> 

Ok.. I'll do some testing myself.

-- Pasi

Re: Test results for save/restore with upstream pv_ops domU kernels, 2.6.33 problems

[Pasi Kärkkäinen]

On Thu, Feb 25, 2010 at 05:27:13PM +0200, Pasi Kärkkäinen wrote:
> > > > > 
> > > > > Is it possible some of the save/restore related patches didn't make it to 2.6.31.x stable kernels? 
> > > > 
> > > > AFAIK they only went into the 2.6.32 stable branch. Unfortunately I
> > > > think the 2.6.31 stable series has come to an end now.
> > > > 
> > > 
> > > Hello,
> > > 
> > > An user on ##xen just reported that 2.6.32.9 save/restore works OK,
> > > but 2.6.33 fails/crashes.
> > > 
> > > Some of the patches that went to 2.6.32.x are not in 2.6.33?
> > 
> > I don't think so -- the acceptance criteria for a stable tree is that
> > the patch is in mainline already. Must be some new breakage.
> > 
> 
> Ok.. I'll do some testing myself.
> 

I had to patch 2.6.33 to get it boot without crashing,
(Jeremy sent me a patch to fix it), but after that 
save/restore worked OK for me.

I tried save+restore 10 times in a row for a 
x86_64 4-vcpu 2.6.33 guest with 2 GB of memory,
no problems found on my environment (EL 5.4 dom0).

-- Pasi