VMs crashing - Ubuntu 8.04, KVM84, libvirt 0.6.1

VMs crashing - Ubuntu 8.04, KVM84, libvirt 0.6.1

[Yann Hamon]

Hello list, 

I am using Ubuntu 8.04 with the kvm84 and libvirt 0.6.1 backports. I am experiencing regular crashes of my Vms - well, more like  "freezes" than actual crashes. This is how it looks like: http://waste.mandragor.org/kvmcpu/munin.html

* The VM keeps running
* The number of LOC IRQs goes down to 0, as well as the number of context switches
* The number of processes gradually grows
* The weird thing: some processes that were running before the crash keep running, but won't be able to fork (as you can see the munin-node continues to work)

This is bothering me a lot and I have absolutely no idea how to fix it or work around it. 
Has this been experienced by anyone else, and/or fixed in a later version of KVM?

-- 
Yann HAMON
Systems Manager
Oxford Archaeology: Exploring the Human Journey
http://thehumanjourney.net


------
Files attached to this email may be in ISO 26300 format (OASIS Open Document Format). If you have difficulty opening them, please visit http://iso26300.info for more information.

Re: VMs crashing - Ubuntu 8.04, KVM84, libvirt 0.6.1

[Lucas Meneghel Rodrigues]

On Fri, 2010-01-22 at 11:26 +0000, Yann Hamon wrote:
> Hello list, 
> 
> I am using Ubuntu 8.04 with the kvm84 and libvirt 0.6.1 backports. I am experiencing regular crashes of my Vms - well, more like  "freezes" than actual crashes. This is how it looks like: http://waste.mandragor.org/kvmcpu/munin.html
> 
> * The VM keeps running
> * The number of LOC IRQs goes down to 0, as well as the number of context switches
> * The number of processes gradually grows
> * The weird thing: some processes that were running before the crash keep running, but won't be able to fork (as you can see the munin-node continues to work)
> 
> This is bothering me a lot and I have absolutely no idea how to fix it or work around it. 
> Has this been experienced by anyone else, and/or fixed in a later version of KVM?

I saw your blog post yesterday speaking about it. Unfortunately I don't
have much experience with kvm under Ubuntu, so you might want to talk to
Dustin, Ubuntu's maintainer of KVM. I saw some replies to your blog post
suggesting you to use Dustin's PPA, have you tried to do that?

I am copying Dustin on the message. As I said, I am not very
knowledgeable with Ubuntu, but perhaps you should try a newer KVM build?
kvm-84 is kinda... old :) A backport of a more current codebase to 8.04
could be a good idea.

Re: VMs crashing - Ubuntu 8.04, KVM84, libvirt 0.6.1

[Yann Hamon]

----- "Lucas Meneghel Rodrigues" <lmr@redhat.com> wrote:

> On Fri, 2010-01-22 at 11:26 +0000, Yann Hamon wrote:
> > Hello list, 
> > 
> > I am using Ubuntu 8.04 with the kvm84 and libvirt 0.6.1 backports. I
> am experiencing regular crashes of my Vms - well, more like  "freezes"
> than actual crashes. This is how it looks like:
> http://waste.mandragor.org/kvmcpu/munin.html
> > 
> > * The VM keeps running
> > * The number of LOC IRQs goes down to 0, as well as the number of
> context switches
> > * The number of processes gradually grows
> > * The weird thing: some processes that were running before the crash
> keep running, but won't be able to fork (as you can see the munin-node
> continues to work)
> > 
> > This is bothering me a lot and I have absolutely no idea how to fix
> it or work around it. 
> > Has this been experienced by anyone else, and/or fixed in a later
> version of KVM?
> 
> I saw your blog post yesterday speaking about it. Unfortunately I don't
> have much experience with kvm under Ubuntu, so you might want to talk to
> Dustin, Ubuntu's maintainer of KVM. I saw some replies to your blog post
> suggesting you to use Dustin's PPA, have you tried to do that?

Yes, it's the one I am using - the package has actually been pushed to backports. I did talk to him, but I think they have problems reproducing the bug (it happens fairly rarely, but about once or twice a week for me as I got 50-60 vms).

> I am copying Dustin on the message. As I said, I am not very
> knowledgeable with Ubuntu, but perhaps you should try a newer KVM
> build?

I think he is aware of the bug, but quite busy with building the virtualisation infrastructure for 10.4 I'm afraid :) We used to be supported customers (pledged for that backport actually :( ) but aren't anymore, due to some disagreement between my boss and one of their commercials, which is why I thought I would give it a try here. Maybe someone else experienced that bug...

> kvm-84 is kinda... old :) A backport of a more current codebase to 8.04
> could be a good idea.

Sure, looking forward to 10.4... Might even uprade to 9.10 if I cant wait til then...


------
Files attached to this email may be in ISO 26300 format (OASIS Open Document Format). If you have difficulty opening them, please visit http://iso26300.info for more information.

Re: VMs crashing - Ubuntu 8.04, KVM84, libvirt 0.6.1

[Dustin Kirkland]

On Fri, Jan 22, 2010 at 7:12 AM, Lucas Meneghel Rodrigues
<lmr@redhat.com> wrote:
> On Fri, 2010-01-22 at 11:26 +0000, Yann Hamon wrote:
>> Hello list,
>>
>> I am using Ubuntu 8.04 with the kvm84 and libvirt 0.6.1 backports. I am experiencing regular crashes of my Vms - well, more like  "freezes" than actual crashes. This is how it looks like: http://waste.mandragor.org/kvmcpu/munin.html
>>
>> * The VM keeps running
>> * The number of LOC IRQs goes down to 0, as well as the number of context switches
>> * The number of processes gradually grows
>> * The weird thing: some processes that were running before the crash keep running, but won't be able to fork (as you can see the munin-node continues to work)
>>
>> This is bothering me a lot and I have absolutely no idea how to fix it or work around it.
>> Has this been experienced by anyone else, and/or fixed in a later version of KVM?
>
> I saw your blog post yesterday speaking about it. Unfortunately I don't
> have much experience with kvm under Ubuntu, so you might want to talk to
> Dustin, Ubuntu's maintainer of KVM. I saw some replies to your blog post
> suggesting you to use Dustin's PPA, have you tried to do that?
>
> I am copying Dustin on the message. As I said, I am not very
> knowledgeable with Ubuntu, but perhaps you should try a newer KVM build?
> kvm-84 is kinda... old :) A backport of a more current codebase to 8.04
> could be a good idea.

Thanks for the pointer, Lucas.

I have worked with Yann to get a better KVM onto Ubuntu 8.04 LTS, more
or less at his request.  At the time, I backported the then-current
kvm-84 from Ubuntu 9.04 (Jaunty) to Ubuntu 8.04 (Hardy), as a stop-gap
measure, replacing the purely ancient kvm-62.  Last I heard from Yann,
kvm-84 (and friends) was working much better than kvm-62.

Ubuntu 9.10 (Karmic) has qemu-kvm-0.11, which seems more stable than
kvm-84, in my experience.  And the under-development Ubuntu 10.04 LTS
(Lucid) currently has qemu-kvm-0.12.2.

Once Ubuntu 10.04 LTS (Lucid) ships in April (or, if I get some down
time between now and then), I will attempt to backport qemu-kvm-0.12.2
to Ubuntu 8.04 (Hardy).

There's more than a few complications with doing this backport, and it
takes considerable effort to do so, as there are several pieces that
must be handled, and thoroughly tested, including at least:
 a) the modern kvm kernel module, and getting it building under DKMS
 b) the modern qemu-kvm userspace, getting it building against the
older toolchain
 c) the modern libvirt library, getting it building against the older toolchain
 d) testing the interoperability of all of the above

My efforts are currently 110% committed against developing Ubuntu
10.04 LTS (Lucid), and the virtualization stack there.  Yann, and
anyone wanting a backported virtualization stack against Ubuntu 8.04
LTS should follow the procedures at:
 * https://help.ubuntu.com/community/UbuntuBackports
and ideally someone from the Ubuntu community with backport experience
can help get a new version building and tested.  If there's someone
volunteering to help with that work, I'll gladly mentor them and guide
the process, sign, and upload packages to the Ubuntu repositories on
their behalf.

Cheers,
:-Dustin

Re: VMs crashing - Ubuntu 8.04, KVM84, libvirt 0.6.1

[Yann Hamon]

> Thanks for the pointer, Lucas.
> 
> I have worked with Yann to get a better KVM onto Ubuntu 8.04 LTS, more
> or less at his request.  At the time, I backported the then-current
> kvm-84 from Ubuntu 9.04 (Jaunty) to Ubuntu 8.04 (Hardy), as a stop-gap
> measure, replacing the purely ancient kvm-62.  Last I heard from Yann,
> kvm-84 (and friends) was working much better than kvm-62.

Hello Dustin, thanks for replying here, and thank you again for the backports. The backports indeed fixes major problems that I was having with kvm62 - however I am experiencing regular crashes and haven't been able to figure out why, til now. This didn't occured during my testing phase, and even now occur fairly rarely (one same VM would crash every month or so) - but becomes an issue as you get more vms. 


> Ubuntu 9.10 (Karmic) has qemu-kvm-0.11, which seems more stable than
> kvm-84, in my experience.  And the under-development Ubuntu 10.04 LTS
> (Lucid) currently has qemu-kvm-0.12.2.
> 
> Once Ubuntu 10.04 LTS (Lucid) ships in April (or, if I get some down
> time between now and then), I will attempt to backport
> qemu-kvm-0.12.2 to Ubuntu 8.04 (Hardy).

I think it's more me who assumed I would have a more stable KVM if I sticked to LTS - KVM being relatively new, I might have been better off upgrading every 6 months. Am readying my hosts for 10.4 as I speak...

> There's more than a few complications with doing this backport, and
> it
> takes considerable effort to do so, as there are several pieces that
> must be handled, and thoroughly tested, including at least:
>  a) the modern kvm kernel module, and getting it building under DKMS
>  b) the modern qemu-kvm userspace, getting it building against the
> older toolchain
>  c) the modern libvirt library, getting it building against the older
> toolchain
>  d) testing the interoperability of all of the above

Indeed. Are you aware of any other production deployment using 8.04 experiencing issues? If not, maybe it's just not worth it.

> My efforts are currently 110% committed against developing Ubuntu
> 10.04 LTS (Lucid), and the virtualization stack there.  Yann, and
> anyone wanting a backported virtualization stack against Ubuntu 8.04
> LTS should follow the procedures at:
>  * https://help.ubuntu.com/community/UbuntuBackports
> and ideally someone from the Ubuntu community with backport
> experience
> can help get a new version building and tested.  If there's someone
> volunteering to help with that work, I'll gladly mentor them and
> guide
> the process, sign, and upload packages to the Ubuntu repositories on
> their behalf.

Oh my KVM hosts don't do much more than KVM, so I'd be fine with any version of Ubuntu, really. Will probably move my dev vms to lucid alpha, and test for a few months - and migrate everything to there after release. From there I might do the 6 months upgrade.
Just thought that if the bug I am having was known and fixed I might get a quick workaround which would have helped me wait :)

Thanks for your reply

-- 
Yann HAMON
Systems Manager
Oxford Archaeology: Exploring the Human Journey
http://thehumanjourney.net


------
Files attached to this email may be in ISO 26300 format (OASIS Open Document Format). If you have difficulty opening them, please visit http://iso26300.info for more information.

Re: VMs crashing - Ubuntu 8.04, KVM84, libvirt 0.6.1

[Dustin Kirkland]

[-- Attachment #1: Type: text/plain, Size: 5101 bytes --]

On Wed, 2010-01-27 at 22:11 +0000, Yann Hamon wrote:
> > Thanks for the pointer, Lucas.
> > 
> > I have worked with Yann to get a better KVM onto Ubuntu 8.04 LTS, more
> > or less at his request.  At the time, I backported the then-current
> > kvm-84 from Ubuntu 9.04 (Jaunty) to Ubuntu 8.04 (Hardy), as a stop-gap
> > measure, replacing the purely ancient kvm-62.  Last I heard from Yann,
> > kvm-84 (and friends) was working much better than kvm-62.
> 
> Hello Dustin, thanks for replying here, and thank you again for the
> backports. The backports indeed fixes major problems that I was having
> with kvm62 - however I am experiencing regular crashes and haven't
> been able to figure out why, til now. This didn't occured during my
> testing phase, and even now occur fairly rarely (one same VM would
> crash every month or so) - but becomes an issue as you get more vms. 

Right, per our former conversations, kvm-84 seemed significantly better.
I'm sorry that it's still not totally stable.

> > Ubuntu 9.10 (Karmic) has qemu-kvm-0.11, which seems more stable than
> > kvm-84, in my experience.  And the under-development Ubuntu 10.04 LTS
> > (Lucid) currently has qemu-kvm-0.12.2.
> > 
> > Once Ubuntu 10.04 LTS (Lucid) ships in April (or, if I get some down
> > time between now and then), I will attempt to backport
> > qemu-kvm-0.12.2 to Ubuntu 8.04 (Hardy).
> 
> I think it's more me who assumed I would have a more stable KVM if I
> sticked to LTS - KVM being relatively new, I might have been better
> off upgrading every 6 months. Am readying my hosts for 10.4 as I
> speak...

Right, so KVM in particular is a bit of a special case, in that it's
evolving so much faster than traditional server projects (take something
like postfix, bind, or ssh).  For most server workloads, I would
generally point you at an Ubuntu LTS for optimum stability.
Unfortunately, KVM was barely 1 year old when Ubuntu 8.04 released in
April of 2008, and has stabilized tremendously in the last 2 years.
Specifically if your key workload is KVM virtual machine hosting, I
would actually recommending upgrading through the non-LTS Ubuntu
releases.

All that said, I think the KVM (linux-2.6.32 + qemu-kvm-0.12.2) story on
Ubuntu 10.04 LTS is going to be very different than KVM (linux-2.6.24 +
kvm-62) story was on Ubuntu 8.04 LTS.  Stability is absolutely Ubuntu's
focus for the 10.04 release.  And I gather from the upstream KVM and
QEMU developers that releases of KVM in linux 2.6.32 and qemu-kvm 0.12.2
are far more stable as well.

> > There's more than a few complications with doing this backport, and
> > it
> > takes considerable effort to do so, as there are several pieces that
> > must be handled, and thoroughly tested, including at least:
> >  a) the modern kvm kernel module, and getting it building under DKMS
> >  b) the modern qemu-kvm userspace, getting it building against the
> > older toolchain
> >  c) the modern libvirt library, getting it building against the older
> > toolchain
> >  d) testing the interoperability of all of the above
> 
> Indeed. Are you aware of any other production deployment using 8.04
> experiencing issues? If not, maybe it's just not worth it.

There were definitely other people who were able to confirm some of the
issues you raised with Hardy's kvm-62.  All of those (that I know of)
were pleased with the backported kvm-84, for those particular issues, at
least.


> > My efforts are currently 110% committed against developing Ubuntu
> > 10.04 LTS (Lucid), and the virtualization stack there.  Yann, and
> > anyone wanting a backported virtualization stack against Ubuntu 8.04
> > LTS should follow the procedures at:
> >  * https://help.ubuntu.com/community/UbuntuBackports
> > and ideally someone from the Ubuntu community with backport
> > experience
> > can help get a new version building and tested.  If there's someone
> > volunteering to help with that work, I'll gladly mentor them and
> > guide
> > the process, sign, and upload packages to the Ubuntu repositories on
> > their behalf.
> 
> Oh my KVM hosts don't do much more than KVM, so I'd be fine with any
> version of Ubuntu, really. Will probably move my dev vms to lucid
> alpha, and test for a few months - and migrate everything to there
> after release.

I think this would be a very wise approach!  Move one or two physical
systems over to Lucid Alpha3 when it releases next week.  File bugs you
find against Lucid Alpha3 and we'll work through those issues, and
upload fixes with great fervor and a much quicker turn-around.


>  From there I might do the 6 months upgrade.
> Just thought that if the bug I am having was known and fixed I might
> get a quick workaround which would have helped me wait :)

Once Ubuntu 10.04 LTS releases, if all of your blocking bugs are fixed,
then I'd stick with that until the next LTS (12.04-ish).  Basically, I'd
only upgrade if there's something in particular that I know is broken in
one release, and is fixed in the next.


:-Dustin

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]