Suspending with kvm and kvm_loaded

Suspending with kvm and kvm_loaded

[Richard Hughes]

Hi,

I'm getting lots of bug reports about installing kvm and then suspend
breaking. Some background: when the fedora package kvm is installed,
then the kvm and kvm_xxx modules get auto loaded at boot. To get the
machine to suspend again, the modules have to be rmmod'd on suspend and
then modprobe'd on resume.

We can automate the rmmodding and modprobeing adding a simple script in
pm-utils (/etc/pm/sleep.d/) on package install, but to avoid just
bodging the issue I wanted to ask you guys first.

* Is it safe to rmmod kvm and kvm_x before suspend?
* Is there any work on adding suspend and resume hooks into the kvm
driver to support suspend?
* Do you guys know why the kvm module might be stopping suspend?

Thanks for any pointers,

Richard Hughes.



-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: Suspending with kvm and kvm_loaded

[Jeremy Katz]

On Fri, 2007-06-15 at 13:56 +0100, Richard Hughes wrote:
> I'm getting lots of bug reports about installing kvm and then suspend
> breaking. Some background: when the fedora package kvm is installed,
> then the kvm and kvm_xxx modules get auto loaded at boot. To get the
> machine to suspend again, the modules have to be rmmod'd on suspend and
> then modprobe'd on resume.
> 
> We can automate the rmmodding and modprobeing adding a simple script in
> pm-utils (/etc/pm/sleep.d/) on package install, but to avoid just
> bodging the issue I wanted to ask you guys first.

Also could go into the kvm package instead.  Probably shouldn't be done
generally in pm-utils (the mantra of fix the real problem, don't add
more bodges around it ;-)

> * Is it safe to rmmod kvm and kvm_x before suspend?

Relatively.  Although I vaguely remember having it cause an interesting
problem once or twice, but not reproducibly enough to actually figure
out what was going on.

> * Is there any work on adding suspend and resume hooks into the kvm
> driver to support suspend?

The bigger thing is there's a need to add some hooks so that kvm can do
things at the right point in the suspend sequence.  Avi has patches, but
they missed the .22 merge window.  I need to sit down (maybe today) and
refresh the Fedora kvm patch; I was thinking of pushing those changes in
at the same time so that this can go away.  And if that pans out for a
little bit in rawhide, it could easily go out as an F7 update.

Jeremy


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: Suspending with kvm and kvm_loaded

[Richard Hughes]

On Fri, 2007-06-15 at 09:27 -0400, Jeremy Katz wrote:
> On Fri, 2007-06-15 at 13:56 +0100, Richard Hughes wrote:
> > I'm getting lots of bug reports about installing kvm and then suspend
> > breaking. Some background: when the fedora package kvm is installed,
> > then the kvm and kvm_xxx modules get auto loaded at boot. To get the
> > machine to suspend again, the modules have to be rmmod'd on suspend and
> > then modprobe'd on resume.
> > 
> > We can automate the rmmodding and modprobeing adding a simple script in
> > pm-utils (/etc/pm/sleep.d/) on package install, but to avoid just
> > bodging the issue I wanted to ask you guys first.
> 
> Also could go into the kvm package instead.

Ahh yes, that's what I meant, sorry if I was unclear.

> > * Is it safe to rmmod kvm and kvm_x before suspend?
> 
> Relatively.  Although I vaguely remember having it cause an interesting
> problem once or twice, but not reproducibly enough to actually figure
> out what was going on.

Okay, cool.

> > * Is there any work on adding suspend and resume hooks into the kvm
> > driver to support suspend?
> 
> The bigger thing is there's a need to add some hooks so that kvm can do
> things at the right point in the suspend sequence.  Avi has patches, but
> they missed the .22 merge window.  I need to sit down (maybe today) and
> refresh the Fedora kvm patch; I was thinking of pushing those changes in
> at the same time so that this can go away.  And if that pans out for a
> little bit in rawhide, it could easily go out as an F7 update.

Superb. When you push the new patch into rawhide, yell, and I'll get an
army of testers to try and break suspend in new, exotic ways. :-)

Thanks for the super-quick response.

Richard



-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: Suspending with kvm and kvm_loaded

[Avi Kivity]

Richard Hughes wrote:
> Hi,
>
> I'm getting lots of bug reports about installing kvm and then suspend
> breaking. Some background: when the fedora package kvm is installed,
> then the kvm and kvm_xxx modules get auto loaded at boot. To get the
> machine to suspend again, the modules have to be rmmod'd on suspend and
> then modprobe'd on resume.
>
> We can automate the rmmodding and modprobeing adding a simple script in
> pm-utils (/etc/pm/sleep.d/) on package install, but to avoid just
> bodging the issue I wanted to ask you guys first.
>
> * Is it safe to rmmod kvm and kvm_x before suspend?
>   

Yes, although you can't do that if you have any virtual machines running.


-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: Suspending with kvm and kvm_loaded

[Richard Hughes]

On Sat, 2007-06-16 at 10:53 +0300, Avi Kivity wrote:
> Richard Hughes wrote:
> > Hi,
> >
> > I'm getting lots of bug reports about installing kvm and then suspend
> > breaking. Some background: when the fedora package kvm is installed,
> > then the kvm and kvm_xxx modules get auto loaded at boot. To get the
> > machine to suspend again, the modules have to be rmmod'd on suspend and
> > then modprobe'd on resume.
> >
> > We can automate the rmmodding and modprobeing adding a simple script in
> > pm-utils (/etc/pm/sleep.d/) on package install, but to avoid just
> > bodging the issue I wanted to ask you guys first.
> >
> > * Is it safe to rmmod kvm and kvm_x before suspend?
> >   
> 
> Yes, although you can't do that if you have any virtual machines running.

Is there any easy way we can detect if virtual machines are running?

Richard.



-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: Suspending with kvm and kvm_loaded

[Muli Ben-Yehuda]

On Sat, Jun 16, 2007 at 11:29:34AM +0100, Richard Hughes wrote:

> Is there any easy way we can detect if virtual machines are running?

The module use counts will be elevanted and they'll refuse to unload
(or there's a nasty bug somewhere).

Cheers,
Muli

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: Suspending with kvm and kvm_loaded

[Richard Hughes]

On Sat, 2007-06-16 at 15:30 +0300, Muli Ben-Yehuda wrote:
> 
> > Is there any easy way we can detect if virtual machines are running?
> 
> The module use counts will be elevanted and they'll refuse to unload
> (or there's a nasty bug somewhere).

Nahh, that's too brutal. Is there a way to (hypothetically) read:

# cat /sys/module/kvm/parameters/number_of_running_vms
2

Then if the user tries to suspend we can throw up a bubble saying

"Are you sure you want to suspend your computer with a running virtual
machine?"

Richard.



-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: Suspending with kvm and kvm_loaded

[Luca]

On 6/16/07, Richard Hughes <hughsient-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Sat, 2007-06-16 at 15:30 +0300, Muli Ben-Yehuda wrote:
> >
> > > Is there any easy way we can detect if virtual machines are running?
> >
> > The module use counts will be elevanted and they'll refuse to unload
> > (or there's a nasty bug somewhere).
>
> Nahh, that's too brutal. Is there a way to (hypothetically) read:
>
> # cat /sys/module/kvm/parameters/number_of_running_vms
> 2

cat /sys/module/kvm_intel/refcnt (or kvm_amd) will tell you how many
VMs are running.

Luca

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: Suspending with kvm and kvm_loaded

[Muli Ben-Yehuda]

On Sat, Jun 16, 2007 at 07:36:18PM +0200, Luca wrote:
> On 6/16/07, Richard Hughes <hughsient-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >On Sat, 2007-06-16 at 15:30 +0300, Muli Ben-Yehuda wrote:
> >>
> >> > Is there any easy way we can detect if virtual machines are running?
> >>
> >> The module use counts will be elevanted and they'll refuse to unload
> >> (or there's a nasty bug somewhere).
> >
> >Nahh, that's too brutal. Is there a way to (hypothetically) read:
> >
> ># cat /sys/module/kvm/parameters/number_of_running_vms
> >2
> 
> cat /sys/module/kvm_intel/refcnt (or kvm_amd) will tell you how many
> VMs are running.

... were running at the moment when you queried the kernel. Once you
get back the answer, it's irrelevant --- a new VM could've been
launched inbetween reading the refcnt and acting upon it. Really the
only safe way to do this sort of thing is to try and rmmod the module.

Cheers,
Muli

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: Suspending with kvm and kvm_loaded

[Avi Kivity]

Richard Hughes wrote:
> On Sat, 2007-06-16 at 15:30 +0300, Muli Ben-Yehuda wrote:
>   
>>> Is there any easy way we can detect if virtual machines are running?
>>>       
>> The module use counts will be elevanted and they'll refuse to unload
>> (or there's a nasty bug somewhere).
>>     
>
> Nahh, that's too brutal. Is there a way to (hypothetically) read:
>
> # cat /sys/module/kvm/parameters/number_of_running_vms
> 2
>
> Then if the user tries to suspend we can throw up a bubble saying
>
> "Are you sure you want to suspend your computer with a running virtual
> machine?"
>   

Why invest in workarounds when a fix is available?


-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: Suspending with kvm and kvm_loaded

[Avi Kivity]

Richard Hughes wrote:
> On Sun, 2007-06-17 at 10:02 +0300, Avi Kivity wrote:
>   
>>> "Are you sure you want to suspend your computer with a running
>>>       
>> virtual
>>     
>>> machine?"
>>>   
>>>       
>> Why invest in workarounds when a fix is available?
>>     
>
> Totally agree. I was really looking for a proper fix, e.g. use libvirt
> to suspend the virtual machines before the machine is suspended (if any
> are running).
>   

No no no.  The proper fix will have suspend "just working", whether or 
not a virtual machine is running or not.  When you resume, the virtual 
machines continue running as if nothing had happened.

I posted a patchset which does just that, when the F kernel integrates 
it (through inclusion of 2.6.23 or by patching the current kernel), 
Fedora will be able to suspend/resume just fine.


> Does anybod know why libvirt does not detect my running qemu VM (that
> uses KVM acceleration)? I get:
>
> Failed to connect to hypervisor
>
>   

If you start a virtual machine outside libvirt's knowledge, it won't 
pick it up.  You also need to enable the libvirtd service (or however it 
is called).

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: Suspending with kvm and kvm_loaded

[Richard Hughes]

On Sun, 2007-06-17 at 10:02 +0300, Avi Kivity wrote:
> > "Are you sure you want to suspend your computer with a running
> virtual
> > machine?"
> >   
> 
> Why invest in workarounds when a fix is available?

Totally agree. I was really looking for a proper fix, e.g. use libvirt
to suspend the virtual machines before the machine is suspended (if any
are running).

Does anybod know why libvirt does not detect my running qemu VM (that
uses KVM acceleration)? I get:

Failed to connect to hypervisor

Or is livirt XEN specific (!)?

Richard.



-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: Suspending with kvm and kvm_loaded

[Daniel Veillard]

On Sun, Jun 17, 2007 at 10:19:09AM +0100, Richard Hughes wrote:
> On Sun, 2007-06-17 at 11:10 +0300, Avi Kivity wrote:
> > > Does anybod know why libvirt does not detect my running qemu VM (that
> > > uses KVM acceleration)? I get:
> > >
> > If you start a virtual machine outside libvirt's knowledge, it won't 
> > pick it up.  You also need to enable the libvirtd service (or however it 
> > is called).
> 
> Ohh, I see. In which case I can ignore libvirt. Thanks!

  Well if there were a way to discover running KVM instances and
communicate with them, then we wouldn't need to keep a daemon as a
parent process for libvirt access and control. It's a point I would
love to see solved at the QEmu level, but it's not really urgent :-)

Daniel

-- 
Red Hat Virtualization group http://redhat.com/virtualization/
Daniel Veillard      | virtualization library  http://libvirt.org/
veillard-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org  | libxml GNOME XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine  http://rpmfind.net/

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: Suspending with kvm and kvm_loaded

[Avi Kivity]

Daniel Veillard wrote:
>   Well if there were a way to discover running KVM instances and
> communicate with them, then we wouldn't need to keep a daemon as a
> parent process for libvirt access and control. It's a point I would
> love to see solved at the QEmu level, but it's not really urgent :-)
>   

You can set the qemu monitor socket to a unix domain socket in some 
well-known directory, and enumerate virtual machines by readdiring that 
directory and trying to connect.  Of course that only works for qemu 
processes started by libvirt, not random qemu processes.

(and I wish there was an SO_UNLINKONCLOSE option that could racelessly 
garbage collect those dead sockets)

I still think having a daemon is a good idea; it can collect events 
generated by qemu and possibly act on them.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: Suspending with kvm and kvm_loaded

[Richard Hughes]

On Sun, 2007-06-17 at 11:10 +0300, Avi Kivity wrote:
> No no no.  The proper fix will have suspend "just working", whether or 
> not a virtual machine is running or not.  When you resume, the virtual 
> machines continue running as if nothing had happened.

Ahh gotcha. In which case I apologize, I really don't follow kvm
development very closely and didn't know this was even possible or
planned.

> I posted a patchset which does just that, when the F kernel integrates 
> it (through inclusion of 2.6.23 or by patching the current kernel), 
> Fedora will be able to suspend/resume just fine.

I think the latter is most likely, although if you have a link to the
current patch I can push it in the way of davej and see what he says.

> > Does anybod know why libvirt does not detect my running qemu VM (that
> > uses KVM acceleration)? I get:
> >
> If you start a virtual machine outside libvirt's knowledge, it won't 
> pick it up.  You also need to enable the libvirtd service (or however it 
> is called).

Ohh, I see. In which case I can ignore libvirt. Thanks!

Richard.



-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: Suspending with kvm and kvm_loaded

[Richard Hughes]

On Sun, 2007-06-17 at 11:10 +0300, Avi Kivity wrote:
> 
> I posted a patchset which does just that, when the F kernel
> integrates 
> it (through inclusion of 2.6.23 or by patching the current kernel), 
> Fedora will be able to suspend/resume just fine. 

I've tried the current kvm kernel git tree and that failed to suspend
with a running VM. Is there a different branch I should try?

Richard.



-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: Suspending with kvm and kvm_loaded

[Avi Kivity]

Richard Hughes wrote:
> On Sun, 2007-06-17 at 11:10 +0300, Avi Kivity wrote:
>   
>> I posted a patchset which does just that, when the F kernel
>> integrates 
>> it (through inclusion of 2.6.23 or by patching the current kernel), 
>> Fedora will be able to suspend/resume just fine. 
>>     
>
> I've tried the current kvm kernel git tree and that failed to suspend
> with a running VM. Is there a different branch I should try?
>
>   

I've rebased the 'hotplug' branch.  Please try it out.


-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: Suspending with kvm and kvm_loaded

[Jeremy Katz]

On Tue, 2007-06-19 at 11:05 +0300, Avi Kivity wrote:
> Richard Hughes wrote:
> > On Sun, 2007-06-17 at 11:10 +0300, Avi Kivity wrote:
> >> I posted a patchset which does just that, when the F kernel
> >> integrates 
> >> it (through inclusion of 2.6.23 or by patching the current kernel), 
> >> Fedora will be able to suspend/resume just fine. 
> >>     
> > I've tried the current kvm kernel git tree and that failed to suspend
> > with a running VM. Is there a different branch I should try?
>
> I've rebased the 'hotplug' branch.  Please try it out.

I actually extracted the patch set yesterday for the Fedora kernel and
it's in today's rawhide kernel.  Should be a good way to get some extra
testing on it.  In my quick trial run yesterday, it worked quite nicely!

Jeremy


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: Suspending with kvm and kvm_loaded

[Richard Hughes]

On Tue, 2007-06-19 at 08:33 -0400, Jeremy Katz wrote:
> On Tue, 2007-06-19 at 11:05 +0300, Avi Kivity wrote:
> > Richard Hughes wrote:
> > > On Sun, 2007-06-17 at 11:10 +0300, Avi Kivity wrote:
> > >> I posted a patchset which does just that, when the F kernel
> > >> integrates 
> > >> it (through inclusion of 2.6.23 or by patching the current kernel), 
> > >> Fedora will be able to suspend/resume just fine. 
> > >>     
> > > I've tried the current kvm kernel git tree and that failed to suspend
> > > with a running VM. Is there a different branch I should try?
> >
> > I've rebased the 'hotplug' branch.  Please try it out.
> 
> I actually extracted the patch set yesterday for the Fedora kernel and
> it's in today's rawhide kernel.  Should be a good way to get some extra
> testing on it.  In my quick trial run yesterday, it worked quite nicely!

Yes, it works very well for me too. Thanks.

Richard.



-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

Re: Suspending with kvm and kvm_loaded

[Patrick C. F. Ernzer]

[-- Attachment #1.1: Type: text/plain, Size: 1290 bytes --]

Hiho,

--On Tuesday, June 19, 2007 13:41:07 +0100 Richard Hughes 
<hughsient-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> On Tue, 2007-06-19 at 08:33 -0400, Jeremy Katz wrote:
[...]
>> I actually extracted the patch set yesterday for the Fedora kernel and
>> it's in today's rawhide kernel.  Should be a good way to get some extra
>> testing on it.  In my quick trial run yesterday, it worked quite nicely!
>
> Yes, it works very well for me too. Thanks.

Can confirm that in a one-off test this worked here too. Thanks Jeremy.

If needed I can run more tests, for me this is enough as I actually use kvm 
very seldom on the affected machine.

RU

PCFE
-- 
Reg. Adresse:      Red Hat GmbH, Hauptstaetter Str 58, 70178 Stuttgart
Handelsregister:   Amtsgericht Stuttgart HRB 153243
Geschaeftsfuehrer: Alex Pinchev, Charlie Peters, David Owens,
                   Michael Cunningham, Werner Knoblich
------------------------ http://www.redhat.com/ ----------------------
     Patrick C. F. Ernzer          |        Senior Consultant
     Red Hat                       |   Global Professional Services
-----------------------------------+----------------------------------
Do not meddle in the internals of kernels,
for they are subtle and quick to panic.

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 286 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/

[-- Attachment #3: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel