What happens to pages that failed to be written to disk?

What happens to pages that failed to be written to disk?

[Martin Jambor]

Hi,

I have tried to find out how filesystems are supposed to handle the
situation when an asynchronous writeout of a page fails and so had a
look at the ext2 code. All I have found is that for example
mpage_end_io_write sets the Error flag of both the page and its
mapping... and that is about it. What happens to these pages later on?
Does the memory manager attempt to write them again? Is the error
somehow signalled to anyone? Do filesystems try to relocate the data
from bad blocks of the device?

TIA

Martin Jambor

Re: What happens to pages that failed to be written to disk?

[Andrew Morton]

Martin Jambor <jamborm@gmail.com> wrote:
>
> Hi,
> 
> I have tried to find out how filesystems are supposed to handle the
> situation when an asynchronous writeout of a page fails and so had a
> look at the ext2 code. All I have found is that for example
> mpage_end_io_write sets the Error flag of both the page and its
> mapping... and that is about it.

yup.

> What happens to these pages later on?

They get reclaimed without being written.

> Does the memory manager attempt to write them again?

No.  Because if the error persists the machine will run out of memory in a
fatal manner.

> Is the error
> somehow signalled to anyone?

Yes, it's propagated into the file's address_space for a later
fsync()/fdatasync()/msync() to detect.

> Do filesystems try to relocate the data
> from bad blocks of the device?

Nope.  Disks will do that internally.  If a disk gets a write I/O error
it's generally dead.

Nowadays I/O errors generally arise from storage networks when something in
the network is disrupted.  Possibly our behaviour is wrong in that case,
although it could be argued that the retry logic should be implemented at
the storage layer and not the VFS layer.  The out-of-memory problem is a
killer.

Re: What happens to pages that failed to be written to disk?

[Szakacsits Szabolcs]

On Thu, 28 Jul 2005, Andrew Morton wrote:
> Martin Jambor <jamborm@gmail.com> wrote:
> >
> > Do filesystems try to relocate the data from bad blocks of the
> > device?

Only Windows NTFS, not others AFAIK (most filesytems can mark them during
mkfs, that's all).

> Nope.  Disks will do that internally.  If a disk gets a write I/O error
> it's generally dead.

That's what I thought also for over a decade (that they are basically dead
soon) so originally I disabled NTFS resizing support for such disks (the
tool is quite widely used since it's the only free, open source NTFS
resizer).

However over the last three years users convinced me that it's quite ok
having a few bad sectors (not only in the remapping area). I investigated
several "dead" disks and the pattern was that they had maximum a dozen bad
sectors, the disk quality didn't degrade in time and users sweared they
never had any problem (crash, data loss, corruption, whatever).

So recently I added optional support for working with such "dead" disks
and this is a bit frustrating since if users don't run badblocks in 
such cases when creating the Linux filesystems then they will have a
hopefully not too spectacular encounter with these sectors sooner or
later.

Of course I don't say that all disks having bad sectors are totally ok and
never will die. Only that that it seems some subset of them seems to keep
working perfectly fine (the issue looks similar to the "having a few dead
pixel" problem on some LCD's and laptops).

	Szaka

Re: What happens to pages that failed to be written to disk?

[Bryan Henderson]

>On Thu, 28 Jul 2005, Andrew Morton wrote:
>> Martin Jambor <jamborm@gmail.com> wrote:
>> >
>> > Do filesystems try to relocate the data from bad blocks of the
>> > device?
>
>Only Windows NTFS, not others AFAIK (most filesytems can mark them during
>mkfs, that's all).
>
>> Nope.  Disks will do that internally.  If a disk gets a write I/O error
>> it's generally dead.
>
>That's what I thought also for over a decade (that they are basically 
dead
>soon) so originally I disabled NTFS resizing support for such disks (the
>tool is quite widely used since it's the only free, open source NTFS
>resizer).
>
>However over the last three years users convinced me that it's quite ok
>having a few bad sectors

There's a common misunderstanding in this area.  First of all, Andrew and 
Szakacsits are talking about different things:  Szakacsits is saying that 
you don't have to throw away your whole disk because of one media error (a 
spot on the disk that won't hold data).  Andrew is saying that if you get 
an error when writing, the disk is dead, and the reasoning goes that if it 
were just a media error, the write wouldn't have failed -- the disk would 
have relocated the sector somewhere else and succeeded.

Szakacsits is right.  Andrew is too, but for a different reason.

A normal disk doesn't give you a write error when a media error prevents 
writing the data.  The disk doesn't know that the data it wrote did not 
get actually stored.  It's not going to wait for the disk to come around 
again and try to read it back to verify.  And even if it did, a lot of 
media errors cause the data to disappear after a short while, so that 
wouldn't help much.  So if a write fails, it isn't because of a media 
error; i.e. can't be fixed by relocation.  The write fails because the 
whole drive is broken.  The disk won't turn, a wire is broken, etc.

(The drive relocates a bad sector when you write to it after a previously 
failed read.  I.e. after data has already been lost).

As Andrew pointed out, write errors are becoming much more common these 
days because of network storage.  The write fails because the disk isn't 
plugged in, the network switch isn't properly configured, the storage 
server isn't up and running yet, and a bunch of other fairly common 
problems.

What makes this really interesting in relation to the question about what 
to do with these failed writes is not just that they're so common, but 
that they're all easily repairable.  If you had a few megabytes stuck in 
your cache because the storage server isn't up yet, it would be nice if 
the system could just write them out a few seconds later when the problem 
is resolved.  Or if they're stuck because the drive isn't properly plugged 
in, it would be nice if you could tell an operator to either plug it in or 
explicitly delete the file.  But the memory management issue is a major 
stumbling block.

Re: What happens to pages that failed to be written to disk?

[Martin Jambor]

Hi and thanks for all answers.

On 7/28/05, Andrew Morton <akpm@osdl.org> wrote:
> > Is the error
> > somehow signalled to anyone?
> 
> Yes, it's propagated into the file's address_space for a later
> fsync()/fdatasync()/msync() to detect.

I see, so a subsequent sync, fsync or umount fail with an error even
when the writing that failed was not initiated because of them?

> > Do filesystems try to relocate the data
> > from bad blocks of the device?
> 
> Nope.  Disks will do that internally.  If a disk gets a write I/O error
> it's generally dead.

I am not interested in what happens in HW, I strive to write a filesystem :-) 
Anyway, I see that a write error probably does not happen because of
bad blocks anyway but because something even worse happened and
therefore there is no point in it even though our filesystem would be
able to relocate stuff fairly easily. Am I right?

Thanks again,

Martin

Re: What happens to pages that failed to be written to disk?

[Anton Altaparmakov]

On Sat, 30 Jul 2005, Martin Jambor wrote:
> Hi and thanks for all answers.
> 
> On 7/28/05, Andrew Morton <akpm@osdl.org> wrote:
> > > Is the error
> > > somehow signalled to anyone?
> > 
> > Yes, it's propagated into the file's address_space for a later
> > fsync()/fdatasync()/msync() to detect.
> 
> I see, so a subsequent sync, fsync or umount fail with an error even
> when the writing that failed was not initiated because of them?
> 
> > > Do filesystems try to relocate the data
> > > from bad blocks of the device?
> > 
> > Nope.  Disks will do that internally.  If a disk gets a write I/O error
> > it's generally dead.
> 
> I am not interested in what happens in HW, I strive to write a filesystem :-) 
> Anyway, I see that a write error probably does not happen because of
> bad blocks anyway but because something even worse happened and
> therefore there is no point in it even though our filesystem would be
> able to relocate stuff fairly easily. Am I right?

It can and does happen due to bad blocks.  The drive only relocates up to 
a certain number (until its spare blocks run out) then defetive blocks 
start appearing on disk.  We (NTFS developers) have seen a lot of users 
with disks with bad blocks on them.  And Windows NTFS does relocation so 
when a write fails it relocates the data block and repeats the write.  So 
if your filesystem can do the same then it is definitely worth doing.

The argument that once you start seing bad sectors the disk is going to be 
dead soon is wrong as many examples in the field have shown.  As Szaka 
said it is just like dead pixels on LCD screens.  The drive can develop a 
certain number and then stop and then work happily for many years.

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/