[Qemu-devel] Crash: When Host HDD is full

[Qemu-devel] Crash: When Host HDD is full

[Alexey Eremenko]

Problem 1:
When Host HDD is full, all guests simply crash. Tried with dynamically
growing .VMDK hard disk.

It shouldn't happen. For example, both VirtualPC and VirtualBox pause
all VMs, and gray their displays when something like that happens.

Host: Fedora 7, 64-bit, Qemu 0.9, AMD Opteron.

Problem 2: Where is Qemu Bugzilla ?

-- 
-Alexey Eremenko "Technologov"

Re: [Qemu-devel] Crash: When Host HDD is full

[Mike Swanson]

On Wednesday 11 July 2007 08:19:48 Alexey Eremenko wrote:
> Problem 1:
> When Host HDD is full, all guests simply crash. Tried with dynamically
> growing .VMDK hard disk.
>
> It shouldn't happen. For example, both VirtualPC and VirtualBox pause
> all VMs, and gray their displays when something like that happens.

IMO, it's a non-issue; you probably shouldn't be letting your filesystems to 
become full in the first place, much less running VMs on it. Besides, most 
filesystems reserve some space for the superuser, now unless there's a 
cross-platform way to figure out just how much space that is, you'd still be 
getting errors despite having 5~10% of the filesystem technically free.

Re: [Qemu-devel] Crash: When Host HDD is full

[Avi Kivity]

Mike Swanson wrote:
> On Wednesday 11 July 2007 08:19:48 Alexey Eremenko wrote:
>   
>> Problem 1:
>> When Host HDD is full, all guests simply crash. Tried with dynamically
>> growing .VMDK hard disk.
>>
>> It shouldn't happen. For example, both VirtualPC and VirtualBox pause
>> all VMs, and gray their displays when something like that happens.
>>     
>
> IMO, it's a non-issue; you probably shouldn't be letting your filesystems to 
> become full in the first place, much less running VMs on it. 

Well, he shouldn't, but neither should qemu crash or corrupt data.

> Besides, most 
> filesystems reserve some space for the superuser, now unless there's a 
> cross-platform way to figure out just how much space that is, you'd still be 
> getting errors despite having 5~10% of the filesystem technically free.
>
>   

Qemu might freeze the guest when it gets -ENOSPC, and say, retry every 
second or wait for user input on the monitor.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Crash: When Host HDD is full

[Paul Brook]

> > Besides, most
> > filesystems reserve some space for the superuser, now unless there's a
> > cross-platform way to figure out just how much space that is, you'd still
> > be getting errors despite having 5~10% of the filesystem technically
> > free.
>
> Qemu might freeze the guest when it gets -ENOSPC, and say, retry every
> second or wait for user input on the monitor.

Better would IMHO be to report an IO error to the guest and allow that to 
decide what to do. If you're bothered about robustness and reliability then 
arbitrarily stopping the guest is not acceptable behaviour. There's no 
guarantee that space will become available in a finite timeframe.

Paul

Re: [Qemu-devel] Crash: When Host HDD is full

[Avi Kivity]

Paul Brook wrote:
>>> Besides, most
>>> filesystems reserve some space for the superuser, now unless there's a
>>> cross-platform way to figure out just how much space that is, you'd still
>>> be getting errors despite having 5~10% of the filesystem technically
>>> free.
>>>       
>> Qemu might freeze the guest when it gets -ENOSPC, and say, retry every
>> second or wait for user input on the monitor.
>>     
>
> Better would IMHO be to report an IO error to the guest and allow that to 
> decide what to do. If you're bothered about robustness and reliability then 
> arbitrarily stopping the guest is not acceptable behaviour. There's no 
> guarantee that space will become available in a finite timeframe.
>   

I've considered that, and I'm not sure.  You will likely get a storm of 
I/O errors on ENOSPC; with several ways for disaster to strike:
- the guest doesn't handle I/O errors well, and keeps writing.  some of 
the writes are overwrites so they hit the disk and data is corrupted
- the guest decides the disk is bad because it has too many errors and 
initiates some recovery procedure

Stopping the guest at least guarantees nothing unexpected happens.  If 
it's part of a managed solution we can output a message to the monitor 
which eventually finds its way to the operator.


-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Crash: When Host HDD is full

[Paul Brook]

> >> Qemu might freeze the guest when it gets -ENOSPC, and say, retry every
> >> second or wait for user input on the monitor.
> >
> > Better would IMHO be to report an IO error to the guest and allow that to
> > decide what to do. If you're bothered about robustness and reliability
> > then arbitrarily stopping the guest is not acceptable behaviour. There's
> > no guarantee that space will become available in a finite timeframe.
>
> I've considered that, and I'm not sure.  You will likely get a storm of
> I/O errors on ENOSPC; with several ways for disaster to strike:
> - the guest doesn't handle I/O errors well, and keeps writing.  some of
> the writes are overwrites so they hit the disk and data is corrupted

If an guest OS ignores IO write errors it's just plain broken.

> - the guest decides the disk is bad because it has too many errors and
> initiates some recovery procedure
>
> Stopping the guest at least guarantees nothing unexpected happens.  If 
> it's part of a managed solution we can output a message to the monitor
> which eventually finds its way to the operator.

I don't buy this argument.  If you don't want "unexpected" things to happen 
then the solution is simple: Make sure you never run out of disk space. 

The fact is that your (virtual) disk *is* broken at this point. The guest OS 
is in a much better position to decide on an appropriate course of action, 
either by retrying or some other recovery mechanism.

There are various error contitions that could be used, for example 
write-protect.

Paul

Re: [Qemu-devel] Crash: When Host HDD is full

[Avi Kivity]

Paul Brook wrote:
>>>> Qemu might freeze the guest when it gets -ENOSPC, and say, retry every
>>>> second or wait for user input on the monitor.
>>>>         
>>> Better would IMHO be to report an IO error to the guest and allow that to
>>> decide what to do. If you're bothered about robustness and reliability
>>> then arbitrarily stopping the guest is not acceptable behaviour. There's
>>> no guarantee that space will become available in a finite timeframe.
>>>       
>> I've considered that, and I'm not sure.  You will likely get a storm of
>> I/O errors on ENOSPC; with several ways for disaster to strike:
>> - the guest doesn't handle I/O errors well, and keeps writing.  some of
>> the writes are overwrites so they hit the disk and data is corrupted
>>     
>
> If an guest OS ignores IO write errors it's just plain broken.
>
>   

Linux 2.4 ignores IO write errors under certain conditions.  Yes, it's 
broken.  But you're making the user suffer for this brokenness even if 
the only thing wrong is a temporary shortage of disk space.

>> - the guest decides the disk is bad because it has too many errors and
>> initiates some recovery procedure
>>
>> Stopping the guest at least guarantees nothing unexpected happens.  If 
>> it's part of a managed solution we can output a message to the monitor
>> which eventually finds its way to the operator.
>>     
>
> I don't buy this argument.  If you don't want "unexpected" things to happen 
> then the solution is simple: Make sure you never run out of disk space. 
>   

That's unrealistic, at least for the casual user running qemu.  A 
managed solution can probably work around this.

Qemu should be more user friendly.

> The fact is that your (virtual) disk *is* broken at this point. The guest OS 
> is in a much better position to decide on an appropriate course of action, 
> either by retrying or some other recovery mechanism.
>
>   

I don't see why it is broken.  The disk contents have not changed since 
after the last successful write.  Once you free some space you can 
continue writing.

Note that a recovery mechanism that involves writing will likely fail as 
well, possibly corrupting the disk in the process.

> There are various error contitions that could be used, for example 
> write-protect.
>   

The guest would most likely be surprised at getting a write-protect 
error on its hard disk, and then the disk *would* be broken.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Crash: When Host HDD is full

[Daniel P. Berrange]

On Thu, Jul 12, 2007 at 07:12:58PM +0300, Avi Kivity wrote:
> Mike Swanson wrote:
> >On Wednesday 11 July 2007 08:19:48 Alexey Eremenko wrote:
> >  
> >>Problem 1:
> >>When Host HDD is full, all guests simply crash. Tried with dynamically
> >>growing .VMDK hard disk.
> >>
> >>It shouldn't happen. For example, both VirtualPC and VirtualBox pause
> >>all VMs, and gray their displays when something like that happens.
> >>    
> >
> >IMO, it's a non-issue; you probably shouldn't be letting your filesystems 
> >to become full in the first place, much less running VMs on it. 
> 
> Well, he shouldn't, but neither should qemu crash or corrupt data.

It should be propagating any IO errors in the host back up to the guest
OS as IDE errors. The guest OS will see these and (in the case of Linux)
take the filesystem offline. There should be no crash or data corruption.
While any outstanding writes will not have been flushed with any reasonable
filesystem it will recover upon reboot, either by fsck, or trivially 
recovering the journal & maintaining filesystem integrity.

Potentially ENOSPC could be handled as a special case by pausing the 
guest, but in general I/O errors can be propagated back to guest for
handling (even if the guest merely takes things ofline).

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: [Qemu-devel] Crash: When Host HDD is full

[Avi Kivity]

Daniel P. Berrange wrote:
> On Thu, Jul 12, 2007 at 07:12:58PM +0300, Avi Kivity wrote:
>   
>> Mike Swanson wrote:
>>     
>>> On Wednesday 11 July 2007 08:19:48 Alexey Eremenko wrote:
>>>  
>>>       
>>>> Problem 1:
>>>> When Host HDD is full, all guests simply crash. Tried with dynamically
>>>> growing .VMDK hard disk.
>>>>
>>>> It shouldn't happen. For example, both VirtualPC and VirtualBox pause
>>>> all VMs, and gray their displays when something like that happens.
>>>>    
>>>>         
>>> IMO, it's a non-issue; you probably shouldn't be letting your filesystems 
>>> to become full in the first place, much less running VMs on it. 
>>>       
>> Well, he shouldn't, but neither should qemu crash or corrupt data.
>>     
>
> It should be propagating any IO errors in the host back up to the guest
> OS as IDE errors. The guest OS will see these and (in the case of Linux)
> take the filesystem offline. There should be no crash or data corruption.
> While any outstanding writes will not have been flushed with any reasonable
> filesystem it will recover upon reboot, either by fsck, or trivially 
> recovering the journal & maintaining filesystem integrity.
>
> Potentially ENOSPC could be handled as a special case by pausing the 
> guest, but in general I/O errors can be propagated back to guest for
> handling (even if the guest merely takes things ofline).
>   

I agree about general I/O errors. However ENOSPC cannot happen for disks 
and should not be converted into an I/O error.



-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Crash: When Host HDD is full

[andrzej zaborowski]

On 12/07/07, Daniel P. Berrange <berrange@redhat.com> wrote:
> On Thu, Jul 12, 2007 at 07:12:58PM +0300, Avi Kivity wrote:
> > Mike Swanson wrote:
> > >On Wednesday 11 July 2007 08:19:48 Alexey Eremenko wrote:
> > >
> > >>Problem 1:
> > >>When Host HDD is full, all guests simply crash. Tried with dynamically
> > >>growing .VMDK hard disk.
> > >>
> > >>It shouldn't happen. For example, both VirtualPC and VirtualBox pause
> > >>all VMs, and gray their displays when something like that happens.
> > >>
> > >
> > >IMO, it's a non-issue; you probably shouldn't be letting your filesystems
> > >to become full in the first place, much less running VMs on it.
> >
> > Well, he shouldn't, but neither should qemu crash or corrupt data.
>
> It should be propagating any IO errors in the host back up to the guest
> OS as IDE errors.

Here's a patch attempting to do that which I posted back when I was
really annoyed at almost losing my data because of QEMU not
propagating the errors:
http://lists.gnu.org/archive/html/qemu-devel/2005-12/msg00275.html . I
agree that this should be the correct behavior but there was no
agreement over this when it was discussed on IRC. (I don't know if the
patch was correct though, it was a bit of black magic for me at the
time)

Cheers,
Andrew

Re: [Qemu-devel] Crash: When Host HDD is full

[Alexey Eremenko]

Pause VMs is the only realistic option. This is the correct option,
because it ensures that guest is still alive.
 Other options might crash the guest, like Windows BSOD. Worse yet is
that guest may think that it's hard disk is bad, and will start
marking it's sectors as bad-blocks.

When the VM is paused, the user may take action to free some disk
space and unpause guest manually.

-- 
-Alexey Eremenko "Technologov"

Re: [Qemu-devel] Crash: When Host HDD is full

[Adam Bolte]

[-- Attachment #1: Type: text/plain, Size: 1142 bytes --]

>From a usability perspective, I also would expect this to be the correct
behaviour. Non-technical users using, say, Windows (in full-screen mode)
on a GNU/Linux desktop need to know that their HDD is full and they must
free up space - not have the guest complain about cryptic HDD errors and
failed writes (assuming the guest OS can even be trusted to do this).

Pausing the VM and notifying the user of no free host storage space is
also the same behaviour as other virtualization products, so this might
already be the expected behaviour from the user's POV.

At the very least, I feel a switch to enable this behaviour would be
appropriate.

-Adam


On Thu, 2007-07-12 at 23:39 +0300, Alexey Eremenko wrote:
> Pause VMs is the only realistic option. This is the correct option,
> because it ensures that guest is still alive.
>  Other options might crash the guest, like Windows BSOD. Worse yet is
> that guest may think that it's hard disk is bad, and will start
> marking it's sectors as bad-blocks.
> 
> When the VM is paused, the user may take action to free some disk
> space and unpause guest manually.
> 

[-- Attachment #2: Type: text/html, Size: 1667 bytes --]

Re: [Qemu-devel] Crash: When Host HDD is full

[Andreas Färber]

Am 19.07.2007 um 09:25 schrieb Adam Bolte:

> >From a usability perspective, I also would expect this to be the  
> correct
> behaviour. Non-technical users using, say, Windows (in full-screen  
> mode)
> on a GNU/Linux desktop need to know that their HDD is full and they  
> must
> free up space - not have the guest complain about cryptic HDD  
> errors and
> failed writes (assuming the guest OS can even be trusted to do this).
>
> Pausing the VM and notifying the user of no free host storage space is
> also the same behaviour as other virtualization products, so this  
> might
> already be the expected behaviour from the user's POV.
>
> At the very least, I feel a switch to enable this behaviour would be
> appropriate.

Having developed applications for memory-limited devices I agree that  
an application must handle any cases, whether assumed common or not,  
including out-of-memory situations. So qemu should always detect this  
and handle it in some way.

Regarding the user's point of view described above, I concur that a  
switch would be a good compromise. However given that someone using  
CVS head is likely to read the mailing list and all other users  
upgrading should read the release notes, this option should be the  
default or otherwise exactly those users the feature is targeted at  
will not use the switch! It's also more frontend-friendly not needing  
an additional switch for mainstream users.

Andreas

Re: [Qemu-devel] Crash: When Host HDD is full

[Alexey Eremenko]

On 7/19/07, Andreas Färber <andreas.faerber@web.de> wrote:
>
>
> Regarding the user's point of view described above, I concur that a
> switch would be a good compromise. However given that someone using
> CVS head is likely to read the mailing list and all other users
> upgrading should read the release notes, this option should be the
> default or otherwise exactly those users the feature is targeted at
> will not use the switch! It's also more frontend-friendly not needing
> an additional switch for mainstream users.

Agreed. This bug is critical in both senses - from developer's point
of view - that's a crash - something should be prevented at all costs,
and from user point of view, where user should not need to run after
the crash/exception number and read tons of logs to understand why
Qemu crash or his guest crashed.

As Adam wrote, other virtualization vendors already provide a fix for
such a situation, so why re-invert the wheel ? Let's just follow them,
and pause the VM, with displaying a warning message to the user.

-- 
-Alexey Eremenko "Technologov"