[RFT] kvm host suspend/resume

[RFT] kvm host suspend/resume

[Avi Kivity]

I've committed kvm support for host suspend-to-disk (and hopefully 
resume too).  This means you can suspend your machine with running vms, 
and resume it later will all vms returning to the saved state.  This 
should please laptop owners.

The big ironers should also be happy as support for host cpu hotplug was 
also added as a side effect.

Please test this feature!  A few testing cycles on one of my machines 
saw both Linux and Windows guests survive a suspend/resume cycle.

This is available from subversion trunk, or if you don't use subversion, 
from http://people.qumranet.com/avi/kvm-r4341.tar.gz.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [RFT] kvm host suspend/resume

[dormando]

Avi Kivity wrote:
> I've committed kvm support for host suspend-to-disk (and hopefully 
> resume too).  This means you can suspend your machine with running vms, 
> and resume it later will all vms returning to the saved state.  This 
> should please laptop owners.
> 
> The big ironers should also be happy as support for host cpu hotplug was 
> also added as a side effect.
> 
> Please test this feature!  A few testing cycles on one of my machines 
> saw both Linux and Windows guests survive a suspend/resume cycle.
> 
> This is available from subversion trunk, or if you don't use subversion, 
> from http://people.qumranet.com/avi/kvm-r4341.tar.gz.

I pulled r4347 from SVN and built it...

I have a Dell M1210 (Core 2 Duo intel), and tried a few things, all with 
a debian Etch AMD64 guest:

1) chmod'ing /dev/kvm and starting guest as local user
2) starting guest as root
3) starting guest, closing it, then suspend/resuming without unloading 
the modules.
... then suspending/resuming the laptop.

Previously, I could suspend/resume with the modules loaded, but the 
laptop would hardlock as soon as I tried to start up a guest.

This time, in all cases the machine did not come back from suspend, even 
in case #3 where it used to come back fine. I just get a black screen 
and no network activity, so I don't have any interesting troubleshooting 
info. Anything I can do to be of more help?

Thanks,
-Dormando

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [RFT] kvm host suspend/resume

[Avi Kivity]

dormando wrote:
> Avi Kivity wrote:
>> I've committed kvm support for host suspend-to-disk (and hopefully 
>> resume too).  This means you can suspend your machine with running 
>> vms, and resume it later will all vms returning to the saved state.  
>> This should please laptop owners.
>>
>> The big ironers should also be happy as support for host cpu hotplug 
>> was also added as a side effect.
>>
>> Please test this feature!  A few testing cycles on one of my machines 
>> saw both Linux and Windows guests survive a suspend/resume cycle.
>>
>> This is available from subversion trunk, or if you don't use 
>> subversion, from http://people.qumranet.com/avi/kvm-r4341.tar.gz.
>
> I pulled r4347 from SVN and built it...
>
> I have a Dell M1210 (Core 2 Duo intel), and tried a few things, all 
> with a debian Etch AMD64 guest:
>
> 1) chmod'ing /dev/kvm and starting guest as local user
> 2) starting guest as root
> 3) starting guest, closing it, then suspend/resuming without unloading 
> the modules.
> ... then suspending/resuming the laptop.
>
> Previously, I could suspend/resume with the modules loaded, but the 
> laptop would hardlock as soon as I tried to start up a guest.

This is expected: the cpu came up with vmx disabled.

>
> This time, in all cases the machine did not come back from suspend, 
> even in case #3 where it used to come back fine. I just get a black 
> screen and no network activity, so I don't have any interesting 
> troubleshooting info. Anything I can do to be of more help?
>

Sprinkle printk()s in vmx_vcpu_decache(), vmx_hardware_enable(), 
vmx_hardware_disable(), kvm_cpu_hotplug(), kvm_suspend(), and 
kvm_resume().  Please include raw_smp_processor_id() in all outputs, 
then send the log (unfortunately you'll have to copy it by hand: 
netconsole doesn't play well with suspend).

Please start with case #3 (perhaps even case #4 where you load the 
modules but don't start a guest).



-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [RFT] kvm host suspend/resume

[Jeremy Katz]

Avi Kivity wrote:
> I've committed kvm support for host suspend-to-disk (and hopefully 
> resume too).  This means you can suspend your machine with running vms, 
> and resume it later will all vms returning to the saved state.  This 
> should please laptop owners.

So testing with trunk (r4365) + 2.6.20-rc5-ish, I'm able to suspend and 
resume with the modules loaded.  Starting a guest after resume gets the 
following oops in the host; guest is in the grub menu at that point.

kvm: msrs: 1
invalid opcode: 0000 [#1]
SMP
last sysfs file: /class/net/eth0/carrier
Modules linked in: snd_hda_intel snd_hda_codec button kvm_intel kvm 
rfcomm hidp l2cap i915 drm ipt_MASQUERADE iptable_nat nf_nat ipv6 
autofs4 sunrpc bridge dummy nf_conntrack_netbios_ns ipt_REJECT 
nf_conntrack_ipv4 xt_state nf_conntrack nfnetlink xt_tcpudp 
iptable_filter ip_tables x_tables cpufreq_ondemand video sbs i2c_ec dock 
battery asus_acpi backlight ac parport_pc lp parport sr_mod cdrom 
snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device 
snd_pcm_oss snd_mixer_oss hci_usb fw_ohci snd_pcm e100 mii bluetooth 
sdhci ata_piix fw_core ipw3945 mmc_core sg serio_raw i2c_i801 iTCO_wdt 
iTCO_vendor_support snd_timer ieee80211 ieee80211_crypt ata_generic 
i2c_core snd soundcore snd_page_alloc joydev dm_snapshot dm_zero 
dm_mirror dm_mod ahci libata sd_mod scsi_mod ext3 jbd mbcache ehci_hcd 
ohci_hcd uhci_hcd
CPU:    1
EIP:    0060:[<f8d3bc6a>]    Not tainted VLI
EFLAGS: 00010207   (2.6.19-1.2913.fc7 #1)
EIP is at vmx_vcpu_load+0x49/0xc1 [kvm_intel]
eax: db452cbc   ebx: f2f5847c   ecx: 018e2100   edx: c075eb2c
esi: 00000001   edi: db452dc0   ebp: db452cd4   esp: db452cac
ds: 007b   es: 007b   ss: 0068
Process qemu (pid: 21176, ti=db452000 task=e2020030 task.ti=db452000)
Stack: 22222222 22222222 22222222 22222222 144d9000 00000000 db452dc0 
db452cd4
        f2f58484 f2f5847c db452ce4 f8e61ad6 f2f5847c f2f5847c db452cf0 
f8e6322f
        00000001 db452d4c f8d3cc69 c0603728 00000000 00000002 c0603751 
f2f58484
Call Trace:
  [<c04051c9>] show_trace_log_lvl+0x1a/0x2f
  [<c0405279>] show_stack_log_lvl+0x9b/0xa3
  [<c0405415>] show_registers+0x194/0x26a
  [<c0405618>] die+0x12d/0x242
  [<c0605167>] do_trap+0x79/0x91
  [<c0405bbb>] do_invalid_op+0x97/0xa1
  [<c0604f3c>] error_code+0x7c/0x84
  [<f8e61ad6>] vcpu_load+0x3b/0x3f [kvm]
  [<f8e6322f>] kvm_resched+0x20/0x23 [kvm]
  [<f8d3cc69>] kvm_vmx_return+0x1d0/0x1df [kvm_intel]
  [<f8e62139>] kvm_dev_ioctl+0x284/0x117a [kvm]
  [<c0482c36>] do_ioctl+0x22/0x67
  [<c0482ecd>] vfs_ioctl+0x252/0x265
  [<c0482f29>] sys_ioctl+0x49/0x64
  [<c0404058>] syscall_call+0x7/0xb
  =======================
Code: 00 40 39 73 48 89 45 e8 74 07 89 d8 e8 5d ff ff ff 8b 0c b5 80 b3 
71 c0 ba 2c eb 75 c0 8b 43 04 39 04 0a 74 31 89 04 0a 8d 45 e8 <0f> c7 
30 0f 96 c0 84 c0 74 21 8b 45 e8 8b 55 ec c7 04 24 a3 d4
EIP: [<f8d3bc6a>] vmx_vcpu_load+0x49/0xc1 [kvm_intel] SS:ESP 0068:db452cac


Jeremy

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [RFT] kvm host suspend/resume

[Avi Kivity]

[-- Attachment #1: Type: text/plain, Size: 694 bytes --]

Jeremy Katz wrote:
> Avi Kivity wrote:
>> I've committed kvm support for host suspend-to-disk (and hopefully 
>> resume too).  This means you can suspend your machine with running 
>> vms, and resume it later will all vms returning to the saved state.  
>> This should please laptop owners.
>
> So testing with trunk (r4365) + 2.6.20-rc5-ish, I'm able to suspend 
> and resume with the modules loaded.  Starting a guest after resume 
> gets the following oops in the host; guest is in the grub menu at that 
> point.
>

2.6.20-rc6 can't suspend my test machine (regardless of kvm), so this 
possible fix is completely untested.

-- 
error compiling committee.c: too many arguments to function


[-- Attachment #2: hotplug-hotfix.patch --]
[-- Type: text/x-patch, Size: 1768 bytes --]

Index: linux/drivers/kvm/kvm_main.c
===================================================================
--- linux/drivers/kvm/kvm_main.c	(revision 4365)
+++ linux/drivers/kvm/kvm_main.c	(working copy)
@@ -2113,6 +2113,16 @@
 		smp_call_function_single(cpu, kvm_arch_ops->hardware_disable,
 					 NULL, 0, 1);
 		break;
+	}
+	return NOTIFY_OK;
+}
+
+static int kvm_cpu_hotplug2(struct notifier_block *notifier, unsigned long val,
+			   void *v)
+{
+	int cpu = (long)v;
+
+	switch (val) {
 	case CPU_UP_PREPARE:
 		smp_call_function_single(cpu, kvm_arch_ops->hardware_enable,
 					 NULL, 0, 1);
@@ -2126,6 +2136,11 @@
 	.priority = 20, /* must be > scheduler priority */
 };
 
+static struct notifier_block kvm_cpu_notifier2 = {
+	.notifier_call = kvm_cpu_hotplug2,
+	.priority = 2, /* must be < scheduler priority */
+};
+
 static __init void kvm_init_debug(void)
 {
 	struct kvm_stats_debugfs_item *p;
@@ -2197,6 +2212,7 @@
 
 	on_each_cpu(kvm_arch_ops->hardware_enable, 0, 0, 1);
 	r = register_cpu_notifier(&kvm_cpu_notifier);
+	register_cpu_notifier(&kvm_cpu_notifier2);
 	if (r)
 		goto out_free_1;
 	register_reboot_notifier(&kvm_reboot_notifier);
@@ -2226,6 +2242,7 @@
 out_free_2:
 	unregister_reboot_notifier(&kvm_reboot_notifier);
 	unregister_cpu_notifier(&kvm_cpu_notifier);
+	unregister_cpu_notifier(&kvm_cpu_notifier2);
 out_free_1:
 	on_each_cpu(kvm_arch_ops->hardware_disable, 0, 0, 1);
 	kvm_arch_ops->hardware_unsetup();
@@ -2239,6 +2256,7 @@
 	sysdev_class_unregister(&kvm_sysdev_class);
 	unregister_reboot_notifier(&kvm_reboot_notifier);
 	unregister_cpu_notifier(&kvm_cpu_notifier);
+	unregister_cpu_notifier(&kvm_cpu_notifier2);
 	on_each_cpu(kvm_arch_ops->hardware_disable, 0, 0, 1);
 	kvm_arch_ops->hardware_unsetup();
 	kvm_arch_ops = NULL;

[-- Attachment #3: Type: text/plain, Size: 347 bytes --]

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

[-- Attachment #4: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: [RFT] kvm host suspend/resume

[Jeremy Katz]

On Wed, 2007-01-31 at 12:32 +0200, Avi Kivity wrote: 
> Jeremy Katz wrote:
> > Avi Kivity wrote:
> >> I've committed kvm support for host suspend-to-disk (and hopefully 
> >> resume too).  This means you can suspend your machine with running 
> >> vms, and resume it later will all vms returning to the saved state.  
> >> This should please laptop owners.
> >
> > So testing with trunk (r4365) + 2.6.20-rc5-ish, I'm able to suspend 
> > and resume with the modules loaded.  Starting a guest after resume 
> > gets the following oops in the host; guest is in the grub menu at that 
> > point.
> >
> 
> 2.6.20-rc6 can't suspend my test machine (regardless of kvm), so this 
> possible fix is completely untested.

Failure looks basically the same.  Will try to do some more
investigation later today.

kvm: msrs: 1
invalid opcode: 0000 [#1]
SMP 
last sysfs file: /devices/pci0000:00/0000:00:1c.0/0000:02:00.0/cmd
Modules linked in: snd_hda_intel snd_hda_codec button kvm_intel kvm tun
i915 drm ipt_MASQUERADE iptable_nat nf_nat autofs4 hidp rfcomm l2cap
sunrpc bridge dummy nf_conntrack_netbios_ns ipt_REJECT nf_conntrack_ipv4
xt_state nf_conntrack nfnetlink xt_tcpudp iptable_filter ip_tables
x_tables cpufreq_ondemand video sbs i2c_ec dock battery asus_acpi
backlight ac ipv6 parport_pc lp parport snd_seq_dummy snd_seq_oss
snd_seq_midi_event snd_seq snd_seq_device hci_usb snd_pcm_oss sr_mod
cdrom bluetooth snd_mixer_oss fw_ohci e100 mii i2c_i801 snd_pcm iTCO_wdt
iTCO_vendor_support fw_core i2c_core ipw3945 ieee80211 ieee80211_crypt
sdhci snd_timer serio_raw ata_piix snd soundcore snd_page_alloc mmc_core
ata_generic joydev sg dm_snapshot dm_zero dm_mirror dm_mod ahci libata
sd_mod scsi_mod ext3 jbd mbcache ehci_hcd ohci_hcd uhci_hcd
CPU:    1
EIP:    0060:[<f8e2ac6a>]    Not tainted VLI
EFLAGS: 00210207   (2.6.19-1.2913.fc7 #1)
EIP is at vmx_vcpu_load+0x49/0xc1 [kvm_intel]
eax: c3c63d24   ebx: e71d047c   ecx: 018e2100   edx: c075eb48
esi: 00000001   edi: c3c63dc0   ebp: c3c63d3c   esp: c3c63d14
ds: 007b   es: 007b   ss: 0068
Process qemu (pid: 12413, ti=c3c63000 task=e656a030 task.ti=c3c63000)
Stack: 22222222 22222222 22222222 22222222 22311000 00000000 c3c63dc0
c3c63d3c 
       e71d0484 e71d047c c3c63d4c f8e6dae0 f8e6debf fffffffe c3c63f44
f8e6e0c7 
       00000200 bfca8a68 e71d0000 f7fffa80 f7fffa80 f4ac9138 c3c63d98
c0473860 
Call Trace:
[<c04051c9>] show_trace_log_lvl+0x1a/0x2f
[<c0405279>] show_stack_log_lvl+0x9b/0xa3
[<c0405415>] show_registers+0x194/0x26a
[<c0405618>] die+0x12d/0x242
[<c0605167>] do_trap+0x79/0x91
[<c0405bbb>] do_invalid_op+0x97/0xa1
[<c0604f3c>] error_code+0x7c/0x84
[<f8e6dae0>] vcpu_load+0x3b/0x3f [kvm]
[<f8e6e0c7>] kvm_dev_ioctl+0x208/0x117a [kvm]
[<c0482c36>] do_ioctl+0x22/0x67
[<c0482ecd>] vfs_ioctl+0x252/0x265
[<c0482f29>] sys_ioctl+0x49/0x64
[<c0404058>] syscall_call+0x7/0xb
=======================
Code: 00 40 39 73 48 89 45 e8 74 07 89 d8 e8 5d ff ff ff 8b 0c b5 80 b3
71 c0 ba 48 eb 75 c0 8b 43 04 39 04 0a 74 31 89 04 0a 8d 45 e8 <0f> c7
30 0f 96 c0 84 c0 74 21 8b 45 e8 8b 55 ec c7 04 24 a3 c4 
EIP: [<f8e2ac6a>] vmx_vcpu_load+0x49/0xc1 [kvm_intel] SS:ESP
0068:c3c63d14


Jeremy


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [RFT] kvm host suspend/resume

[Avi Kivity]

Jeremy Katz wrote:
> On Wed, 2007-01-31 at 12:32 +0200, Avi Kivity wrote: 
>   
>> Jeremy Katz wrote:
>>     
>>> Avi Kivity wrote:
>>>       
>>>> I've committed kvm support for host suspend-to-disk (and hopefully 
>>>> resume too).  This means you can suspend your machine with running 
>>>> vms, and resume it later will all vms returning to the saved state.  
>>>> This should please laptop owners.
>>>>         
>>> So testing with trunk (r4365) + 2.6.20-rc5-ish, I'm able to suspend 
>>> and resume with the modules loaded.  Starting a guest after resume 
>>> gets the following oops in the host; guest is in the grub menu at that 
>>> point.
>>>
>>>       
>> 2.6.20-rc6 can't suspend my test machine (regardless of kvm), so this 
>> possible fix is completely untested.
>>     
>
> Failure looks basically the same.  Will try to do some more
> investigation later today.
>   

Looks like ->hardware_enable() was not called for some reason.  
Sprinkling printks in the cpu hotplug and resume callbacks will help.


-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [RFT] kvm host suspend/resume

[Jeremy Katz]

[-- Attachment #1: Type: text/plain, Size: 1782 bytes --]

Avi Kivity wrote:
> Jeremy Katz wrote:
>> On Wed, 2007-01-31 at 12:32 +0200, Avi Kivity wrote:  
>>> Jeremy Katz wrote:
>>>> Avi Kivity wrote:
>>>>> I've committed kvm support for host suspend-to-disk (and hopefully 
>>>>> resume too).  This means you can suspend your machine with running 
>>>>> vms, and resume it later will all vms returning to the saved 
>>>>> state.  This should please laptop owners.
>>>>>         
>>>> So testing with trunk (r4365) + 2.6.20-rc5-ish, I'm able to suspend 
>>>> and resume with the modules loaded.  Starting a guest after resume 
>>>> gets the following oops in the host; guest is in the grub menu at 
>>>> that point.
>>>>       
>>> 2.6.20-rc6 can't suspend my test machine (regardless of kvm), so this 
>>> possible fix is completely untested.
>>
>> Failure looks basically the same.  Will try to do some more
>> investigation later today.
> 
> Looks like ->hardware_enable() was not called for some reason.  
> Sprinkling printks in the cpu hotplug and resume callbacks will help.

Multiple things failing here together.

1) With the versioning of Fedora kernels, things were falling back to 
the compat version of smp_call_function_single which then doesn't seem 
to work properly (->hardware_disable() and ->hardware_enable() never get 
called AFAICT)

2) Switching to use the real smp_call_function_single, I get an oops on 
suspend since we're trying to run code on the dead cpu.  Attached patch 
makes it so that we do ->hardware_enable() and ->hardware_disable() when 
the cpu is in a functioning state.  With it, I can suspend/resume 
successfully with no guests running.  With a guest running, it's a bit 
of a toss-up.  But one step at a time :)

Signed-off-by: Jeremy Katz <katzj-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

Jeremy

[-- Attachment #2: kvm-suspend-fix.patch --]
[-- Type: text/x-patch, Size: 679 bytes --]

Index: kvm_main.c
===================================================================
--- kvm_main.c	(revision 4369)
+++ kvm_main.c	(working copy)
@@ -2107,13 +2107,15 @@
 	int cpu = (long)v;
 
 	switch (val) {
-	case CPU_DEAD:
+	case CPU_DOWN_PREPARE:
 	case CPU_UP_CANCELED:
+		printk(KERN_INFO, "kvm: disabling virtualization on CPU%d\n", cpu);
 		decache_vcpus_on_cpu(cpu);
 		smp_call_function_single(cpu, kvm_arch_ops->hardware_disable,
 					 NULL, 0, 1);
 		break;
-	case CPU_UP_PREPARE:
+	case CPU_ONLINE:
+		printk(KERN_INFO, "kvm: enabling virtualization on CPU%d\n", cpu);
 		smp_call_function_single(cpu, kvm_arch_ops->hardware_enable,
 					 NULL, 0, 1);
 		break;

[-- Attachment #3: Type: text/plain, Size: 374 bytes --]

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

[-- Attachment #4: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: [RFT] kvm host suspend/resume

[Avi Kivity]

Jeremy Katz wrote:
> Avi Kivity wrote:
>> Jeremy Katz wrote:
>>> On Wed, 2007-01-31 at 12:32 +0200, Avi Kivity wrote: 
>>>> Jeremy Katz wrote:
>>>>> Avi Kivity wrote:
>>>>>> I've committed kvm support for host suspend-to-disk (and 
>>>>>> hopefully resume too).  This means you can suspend your machine 
>>>>>> with running vms, and resume it later will all vms returning to 
>>>>>> the saved state.  This should please laptop owners.
>>>>>>         
>>>>> So testing with trunk (r4365) + 2.6.20-rc5-ish, I'm able to 
>>>>> suspend and resume with the modules loaded.  Starting a guest 
>>>>> after resume gets the following oops in the host; guest is in the 
>>>>> grub menu at that point.
>>>>>       
>>>> 2.6.20-rc6 can't suspend my test machine (regardless of kvm), so 
>>>> this possible fix is completely untested.
>>>
>>> Failure looks basically the same.  Will try to do some more
>>> investigation later today.
>>
>> Looks like ->hardware_enable() was not called for some reason.  
>> Sprinkling printks in the cpu hotplug and resume callbacks will help.
>
> Multiple things failing here together.
>
> 1) With the versioning of Fedora kernels, things were falling back to 
> the compat version of smp_call_function_single which then doesn't seem 
> to work properly (->hardware_disable() and ->hardware_enable() never 
> get called AFAICT)

Hmm.  It worked for me.  I can't really see anything wrong with it.

>
> 2) Switching to use the real smp_call_function_single, I get an oops 
> on suspend since we're trying to run code on the dead cpu.  Attached 
> patch makes it so that we do ->hardware_enable() and 
> ->hardware_disable() when the cpu is in a functioning state.  With it, 
> I can suspend/resume successfully with no guests running.  With a 
> guest running, it's a bit of a toss-up.  

I think the problem here is the race with the scheduler resuming up 
ahead of kvm.  My previous patch tries to work around it by having one 
callback for suspend and one for resume, each with a different priority 
relative to the scheduler.  Maybe an updated version will help.

> But one step at a time :)

Yes, so I'll apply this :).  Thanks.
>
> Signed-off-by: Jeremy Katz <katzj-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>
> Jeremy
> ------------------------------------------------------------------------
>
> Index: kvm_main.c
> ===================================================================
> --- kvm_main.c	(revision 4369)
> +++ kvm_main.c	(working copy)
> @@ -2107,13 +2107,15 @@
>  	int cpu = (long)v;
>  
>  	switch (val) {
> -	case CPU_DEAD:
> +	case CPU_DOWN_PREPARE:
>  	case CPU_UP_CANCELED:
> +		printk(KERN_INFO, "kvm: disabling virtualization on CPU%d\n", cpu);
>  		decache_vcpus_on_cpu(cpu);
>  		smp_call_function_single(cpu, kvm_arch_ops->hardware_disable,
>  					 NULL, 0, 1);
>  		break;
> -	case CPU_UP_PREPARE:
> +	case CPU_ONLINE:
> +		printk(KERN_INFO, "kvm: enabling virtualization on CPU%d\n", cpu);
>  		smp_call_function_single(cpu, kvm_arch_ops->hardware_enable,
>  					 NULL, 0, 1);
>  		break;
>   


-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642