out of memory error leads to unbootable VM

out of memory error leads to unbootable VM

[Ross Boylan]

I had a VM running Linux.  While running that SAS installer the kernel
ran out of memory and killed the installer.  After that, any attempt to
run a binary inside the VM produced "cannot execute binary file".  The
log (inside the VM) also showed messages like
"Buffer I/O error on device hda1, logical block 12243549368".  
hda1 is 86\% full.

I killed the VM and attempted to restart.  However, the virtual BIOS
says the disk is unrecognizeable.

lvscan shows, among other things,
  inactive Original '/dev/turtle/Lenny00' [5.00 GB] inherit
  ACTIVE            '/dev/turtle/SAS92' [33.00 GB] inherit
  inactive Snapshot '/dev/turtle/Lenny01SAS' [3.00 GB] inherit
The active volume has the installation disk; Lenny01SAS was the "disk"
the VM was using; it is a snapshot of Lenny00.  I have not deliberately
deactivated either.

lvdisplay does not give any indication of COW overflow, though that may
be because the volumes are inactive.

Only about 11G of SAS92 are in use.  The SAS installation docs seem to
indicate 1 to 1.5G for a full installation.  Although it looks as if I
had enough room, I suppose a snapshot overflow could account for the
inactive snapshot (and inactive original?).

It probably did run out of virtual RAM, since kvm started
(inadvertently) with the default RAM.

Any ideas what's going on?

Thanks.
Ross Boylan

Re: out of memory error leads to unbootable VM

[Avi Kivity]

Ross Boylan wrote:
> I had a VM running Linux.  While running that SAS installer the kernel
> ran out of memory and killed the installer.  After that, any attempt to
> run a binary inside the VM produced "cannot execute binary file".  The
> log (inside the VM) also showed messages like
> "Buffer I/O error on device hda1, logical block 12243549368".  
> hda1 is 86\% full.
>
> I killed the VM and attempted to restart.  However, the virtual BIOS
> says the disk is unrecognizeable.
>
> lvscan shows, among other things,
>   inactive Original '/dev/turtle/Lenny00' [5.00 GB] inherit
>   ACTIVE            '/dev/turtle/SAS92' [33.00 GB] inherit
>   inactive Snapshot '/dev/turtle/Lenny01SAS' [3.00 GB] inherit
> The active volume has the installation disk; Lenny01SAS was the "disk"
> the VM was using; it is a snapshot of Lenny00.  I have not deliberately
> deactivated either.
>
> lvdisplay does not give any indication of COW overflow, though that may
> be because the volumes are inactive.
>
> Only about 11G of SAS92 are in use.  The SAS installation docs seem to
> indicate 1 to 1.5G for a full installation.  Although it looks as if I
> had enough room, I suppose a snapshot overflow could account for the
> inactive snapshot (and inactive original?).
>
> It probably did run out of virtual RAM, since kvm started
> (inadvertently) with the default RAM.
>
> Any ideas what's going on?
>
>   

Sounds like a guest bug that caused guest disk corruption.  Try with 
more memory (on new volumes to be sure).

-- 
error compiling committee.c: too many arguments to function

Re: out of memory error leads to unbootable VM

[Ross Boylan]

On Tue, 2009-06-02 at 19:27 +0300, Avi Kivity wrote:
> Ross Boylan wrote:
> > I had a VM running Linux.  While running that SAS installer the kernel
> > ran out of memory and killed the installer.  After that, any attempt to
> > run a binary inside the VM produced "cannot execute binary file".  The
> > log (inside the VM) also showed messages like
> > "Buffer I/O error on device hda1, logical block 12243549368".  
> > hda1 is 86\% full.
> >
> > I killed the VM and attempted to restart.  However, the virtual BIOS
> > says the disk is unrecognizeable.
> >
> > lvscan shows, among other things,
> >   inactive Original '/dev/turtle/Lenny00' [5.00 GB] inherit
> >   ACTIVE            '/dev/turtle/SAS92' [33.00 GB] inherit
> >   inactive Snapshot '/dev/turtle/Lenny01SAS' [3.00 GB] inherit
> > The active volume has the installation disk; Lenny01SAS was the "disk"
> > the VM was using; it is a snapshot of Lenny00.  I have not deliberately
> > deactivated either.
> >
> > lvdisplay does not give any indication of COW overflow, though that may
> > be because the volumes are inactive.
> >
> > Only about 11G of SAS92 are in use.  The SAS installation docs seem to
> > indicate 1 to 1.5G for a full installation.  Although it looks as if I
> > had enough room, I suppose a snapshot overflow could account for the
> > inactive snapshot (and inactive original?).
> >
> > It probably did run out of virtual RAM, since kvm started
> > (inadvertently) with the default RAM.
> >
> > Any ideas what's going on?
> >
> >   
> 
> Sounds like a guest bug that caused guest disk corruption.  Try with 
> more memory (on new volumes to be sure).

Isn't Linux (the guest, running Debian Lenny amd64 dual processor) more
robust than that?  

At any rate, I'll try again.  I'm hoping the original (Lenny00) is OK,
so there's less I need to rebuild.

Ross

P.S. Running kvm 72+dfsg-5~lenny1 with a host 2.6.29 kernel.  They guest
was probably 2.6.26, though it might have been 2.6.29.