filesystem corruption (ReiserFS, 2.6.6): regions replaced by \000 bytes

filesystem corruption (ReiserFS, 2.6.6): regions replaced by \000 bytes

[David Madore]

Hi folks.

I'm afraid this bug-report will be rather worthless as it is, because
the bug has proven remarkable elusive and has defeated all my attempts
to track it down to a precise test case or set of circumstances.  But
since it seems important, I thought it might be worth a post anyway.
Any help is appreciated in clarifying the circumstances which trigger
the problem, or generally in making this report more useful.

The bottom line: I've experienced file corruption, of the following
nature: consecutive regions (all, it seems, aligned on 256-byte
boundaries, and typically around 1kb or 2kb in length) of seemingly
random files are replaced by null bytes.  The filesystem is ReiserFS
(but as nearly all my filesystems are ReiserFS anyway, I cannot
conclusively say that the bug is indeed in ReiserFS).  The problem has
occurred with kernel versions Debian-packaged 2.6.6-1-686-smp and
home-compiled 2.6.6-mm2 (SMP also).  The hardware is an Intel
bi-PII450 (with 256MB RAM) using aic7xxx as low-level disk driver; and
I have good reasons to think that the hardware is sound (and the
memory banks in particular).  Distribution is Debian Sarge (with a few
unstable packages as well).  System load is moderate.  The affected
files were typically corrupted during operation of Debian "apt-get":
either while updating the apt-cache (which became corrupted) or while
extracting packages (which randomly corrupted newly extracted files).

That's about all I can say for sure.  I have tried to reproduce the
problem by stress-testing the filesystem (creating large numbers of
small files, or small numbers of large files, containing ARC4 streams
produced by <URL: ftp://quatramaran.ens.fr/pub/madore/misc/arc4gen.c >
and checking them afterward against the same procedural generator) --
but to no avail: even by trying heavily concurrent access I have not
been able to reproduce a single occurrence of the bug).  Maybe apt-get
has a specific way of writing files (but I can't really think how;
might it use mmap() in some way? that doesn't sound plausible) which
makes it trigger the bug.

I also have a UP box (Intel PIII600 with 384MB RAM and IDE disks)
which, even though it has a nearly identical setup and is used in a
roughly equal way, has not experienced any kind of corruption; so
maybe the bug is SMP-specific (which would explain it going more or
less unnoticed) -- on the other hand, a friend of mine has mentioned
having observed similar problems on a UP box with 2.6.5 installed
under very heavy load on ReiserFS, but I can't say more here.  I'm
really sorry that all this is very fuzzy.

Any suggestions (either on how to fix the problem or to work around
it, or on how to reproduce it experimentally) are welcome.  A friendly
pat on the back would also be welcome. :-)

-- 
     David A. Madore
    (david.madore@ens.fr,
     http://www.eleves.ens.fr:8080/home/madore/ )

Re: filesystem corruption (ReiserFS, 2.6.6): regions replaced by \000 bytes

[Chris Mason]

On Fri, 2004-05-28 at 08:28, David Madore wrote:
> Hi folks.
> 
> I'm afraid this bug-report will be rather worthless as it is, because
> the bug has proven remarkable elusive and has defeated all my attempts
> to track it down to a precise test case or set of circumstances.  But
> since it seems important, I thought it might be worth a post anyway.
> Any help is appreciated in clarifying the circumstances which trigger
> the problem, or generally in making this report more useful.
> 
> The bottom line: I've experienced file corruption, of the following
> nature: consecutive regions (all, it seems, aligned on 256-byte
> boundaries, and typically around 1kb or 2kb in length) of seemingly
> random files are replaced by null bytes.  

The good news is that we tracked this one down recently.  2.6.7-rc1
shouldn't do this anymore.

-chris

Re: filesystem corruption (ReiserFS, 2.6.6): regions replaced by \000 bytes

[Lenar Lõhmus]

Hi,

Uh, seems I've experienced this problem too. But since there were
disk-full condition and at the same time electricity went away ... I thought
I had really bad luck (needed to apt-get --reinstall whole debian).

Glad to hear it's found and fixed now.

Lenar

Chris Mason wrote:

>The good news is that we tracked this one down recently.  2.6.7-rc1
>shouldn't do this anymore.
>
>-chris
>
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>  
>

Re: filesystem corruption (ReiserFS, 2.6.6): regions replaced by \000 bytes

[Tomas Szepe]

On May-28 2004, Fri, 08:46 -0400
Chris Mason <mason@suse.com> wrote:

> > The bottom line: I've experienced file corruption, of the following
> > nature: consecutive regions (all, it seems, aligned on 256-byte
> > boundaries, and typically around 1kb or 2kb in length) of seemingly
> > random files are replaced by null bytes.  
> 
> The good news is that we tracked this one down recently.  2.6.7-rc1
> shouldn't do this anymore.

So did this only affect SMP machines?

-- 
Tomas Szepe <szepe@pinerecords.com>

Re: filesystem corruption (ReiserFS, 2.6.6): regions replaced by \000 bytes

[Chris Mason]

On Fri, 2004-05-28 at 12:24, Tomas Szepe wrote:
> On May-28 2004, Fri, 08:46 -0400
> Chris Mason <mason@suse.com> wrote:
> 
> > > The bottom line: I've experienced file corruption, of the following
> > > nature: consecutive regions (all, it seems, aligned on 256-byte
> > > boundaries, and typically around 1kb or 2kb in length) of seemingly
> > > random files are replaced by null bytes.  
> > 
> > The good news is that we tracked this one down recently.  2.6.7-rc1
> > shouldn't do this anymore.
> 
> So did this only affect SMP machines?

No, if you slept in the right spot you could hit it on UP.

-chris

Re: filesystem corruption (ReiserFS, 2.6.6): regions replaced by \000 bytes

[Pat]

On Friday 28 May 2004 11:29, Chris Mason wrote:
> On Fri, 2004-05-28 at 12:24, Tomas Szepe wrote:
> > On May-28 2004, Fri, 08:46 -0400
> >
> > Chris Mason <mason@suse.com> wrote:
> > > > The bottom line: I've experienced file corruption, of the
> > > > following nature: consecutive regions (all, it seems, aligned
> > > > on 256-byte boundaries, and typically around 1kb or 2kb in
> > > > length) of seemingly random files are replaced by null bytes.
> > >
> > > The good news is that we tracked this one down recently. 
> > > 2.6.7-rc1 shouldn't do this anymore.
> >
> > So did this only affect SMP machines?
>
> No, if you slept in the right spot you could hit it on UP.

I saw this once when using 2.6.6, it was messing up the filesystem 
structures as well (ext2 & ext3), replacing mostly with nulls, some 
with random letters and numbers, for 4-6 character lengths, and not on 
any nice boundaries.  Since I stopped trying to use the ATI framebuffer 
driver (this is on a 21164A alpha, 164LX motherboard, ATI Mach64 CT 
video), it seems to have stopped.  Also, I noticed that the framebuffer 
driver didn't work so well.

2.6.7-rc1 seems to have a compile error for alpha as well, I'll post 
that in a separate email.

Pat
-- 
PLUG Vice President                   --  http://plug.purdue.org
The Computer Refuge                   --  http://computer-refuge.org
Purdue University Research Computing  --  http://www.itap.purdue.edu/rcs

Re: filesystem corruption (ReiserFS, 2.6.6): regions replaced by \000 bytes

[Chris Mason]

On Fri, 2004-05-28 at 12:42, Pat wrote:
> On Friday 28 May 2004 11:29, Chris Mason wrote:
> > On Fri, 2004-05-28 at 12:24, Tomas Szepe wrote:
> > > On May-28 2004, Fri, 08:46 -0400
> > >
> > > Chris Mason <mason@suse.com> wrote:
> > > > > The bottom line: I've experienced file corruption, of the
> > > > > following nature: consecutive regions (all, it seems, aligned
> > > > > on 256-byte boundaries, and typically around 1kb or 2kb in
> > > > > length) of seemingly random files are replaced by null bytes.
> > > >
> > > > The good news is that we tracked this one down recently. 
> > > > 2.6.7-rc1 shouldn't do this anymore.
> > >
> > > So did this only affect SMP machines?
> >
> > No, if you slept in the right spot you could hit it on UP.
> 
> I saw this once when using 2.6.6, it was messing up the filesystem 
> structures as well (ext2 & ext3), replacing mostly with nulls, some 
> with random letters and numbers, for 4-6 character lengths, and not on 
> any nice boundaries.  Since I stopped trying to use the ATI framebuffer 
> driver (this is on a 21164A alpha, 164LX motherboard, ATI Mach64 CT 
> video), it seems to have stopped.  Also, I noticed that the framebuffer 
> driver didn't work so well.
> 
This is different.   The reiserfs bug can only trigger data corruptions
in reiserfs, and won't trigger metadata problems.

-chris

Re: filesystem corruption (ReiserFS, 2.6.6): regions replaced by \000 bytes

[Tomas Szepe]

On May-28 2004, Fri, 12:29 -0400
Chris Mason <mason@suse.com> wrote:

> On Fri, 2004-05-28 at 12:24, Tomas Szepe wrote:
> > On May-28 2004, Fri, 08:46 -0400
> > Chris Mason <mason@suse.com> wrote:
> > 
> > > > The bottom line: I've experienced file corruption, of the following
> > > > nature: consecutive regions (all, it seems, aligned on 256-byte
> > > > boundaries, and typically around 1kb or 2kb in length) of seemingly
> > > > random files are replaced by null bytes.  
> > > 
> > > The good news is that we tracked this one down recently.  2.6.7-rc1
> > > shouldn't do this anymore.
> > 
> > So did this only affect SMP machines?
> 
> No, if you slept in the right spot you could hit it on UP.

Uh oh.  Any idea about when the bug was introduced?

-- 
Tomas Szepe <szepe@pinerecords.com>

Re: filesystem corruption (ReiserFS, 2.6.6): regions replaced by \000 bytes

[Chris Mason]

On Fri, 2004-05-28 at 12:45, Tomas Szepe wrote:

[ reiserfs data corruption bug ]

> > > So did this only affect SMP machines?
> > 
> > No, if you slept in the right spot you could hit it on UP.
> 
> Uh oh.  Any idea about when the bug was introduced?

2.6.6, with the reiserfs data=ordered patches.

-chris

Re: filesystem corruption (ReiserFS, 2.6.6): regions replaced by \000 bytes

[Steven Cole]

On May 28, 2004, at 10:45 AM, Tomas Szepe wrote:

> On May-28 2004, Fri, 12:29 -0400
> Chris Mason <mason@suse.com> wrote:
>
>> On Fri, 2004-05-28 at 12:24, Tomas Szepe wrote:
>>> On May-28 2004, Fri, 08:46 -0400
>>> Chris Mason <mason@suse.com> wrote:
>>>
>>>>> The bottom line: I've experienced file corruption, of the following
>>>>> nature: consecutive regions (all, it seems, aligned on 256-byte
>>>>> boundaries, and typically around 1kb or 2kb in length) of seemingly
>>>>> random files are replaced by null bytes.
>>>>
>>>> The good news is that we tracked this one down recently.  2.6.7-rc1
>>>> shouldn't do this anymore.
>>>
>>> So did this only affect SMP machines?
>>
>> No, if you slept in the right spot you could hit it on UP.
>
> Uh oh.  Any idea about when the bug was introduced?
>

As far as I know, I was the first to publicly complain about
the bug, first to Bitmover, (I was hitting it when using bk)
who then figured out that it was a kernel bug, hence the
long "1352 NUL bytes at end of page" thread.

I first noticed the bug around April 15, doing almost nightly
kernel updates and builds.  The bug was rather hard to hit
reliably though, so it could have been there for some time
earlier.

Steven

Re: filesystem corruption (ReiserFS, 2.6.6): regions replaced by \000 bytes

[Lenar Lõhmus]

Tomas Szepe wrote:

>On May-28 2004, Fri, 08:46 -0400
>Chris Mason <mason@suse.com> wrote:
>
>  
>
>>>The bottom line: I've experienced file corruption, of the following
>>>nature: consecutive regions (all, it seems, aligned on 256-byte
>>>boundaries, and typically around 1kb or 2kb in length) of seemingly
>>>random files are replaced by null bytes.  
>>>      
>>>
>>The good news is that we tracked this one down recently.  2.6.7-rc1
>>shouldn't do this anymore.
>>    
>>
>
>So did this only affect SMP machines?
>
>  
>
No, it's UP here. And I think it happened first with 2.6.6-rc2-mm2.

Lenar

I would like to see ReiserFS V3 enter a feature freeze real soon.

[Hans Reiser]

While I like and appreciate the data journaling stuff, and I think it 
should go in, real soon now I think we should avoid adding new features 
to V3.  Let the mission critical server folks have a reiserfs version 
that only gets bug fixes added to it, and let V4 be for those who want 
excitement.

Are there any things which Chris and Jeff think should go in besides 
data journaling/ordering and bitmap algorithm changes?

Also, I would like to see some serious benchmarks of the bitmap 
algorithm changes before they go in.  They seem nice in theory, and some 
users liked them for their uses, but that does not make a serious 
scientific study.  Such a study has a high chance of making them even 
better.;-)

zam, I view you as the block allocator maintainer, please review that 
bitmap code from Chris.

Chris and Jeff, can you propose a benchmarking plan for the bitmap code?

Hans


Chris Mason wrote:

>On Fri, 2004-05-28 at 09:27, Vladimir Saveliev wrote:
>
>  
>
>>>The good news is that we tracked this one down recently.  2.6.7-rc1
>>>shouldn't do this anymore.
>>>
>>>      
>>>
>>Just out of curiosity, what was it?
>>    
>>
>
>It was a bug in reiserfs_file_write caused by the data=ordered changes. 
>I was dropping i_sem before changing i_size, and the result was that
>writepage was zeroing out bytes it thought was past the end of the file.
>
>See the recent thread with "1352 NUL bytes at the end of a page?" in the
>subject.  The funny part was that we started getting bug reports for
>this on the suse kernel just a few days before Steven Cole reported it,
>so I was able to overlap some of the bug hunting/testing.
>
>-chris
>
>
>
>
>
>  
>

Re: I would like to see ReiserFS V3 enter a feature freeze real soon.

[Chris Mason]

On Mon, 2004-05-31 at 12:48, Hans Reiser wrote:
> While I like and appreciate the data journaling stuff, and I think it 
> should go in, real soon now I think we should avoid adding new features 
> to V3.  Let the mission critical server folks have a reiserfs version 
> that only gets bug fixes added to it, and let V4 be for those who want 
> excitement.
> 
> Are there any things which Chris and Jeff think should go in besides 
> data journaling/ordering and bitmap algorithm changes?
> 

We've got io error fault tolerance that needs to go in after the barrier
code has stabilized.  I can't promise that I'll never making another
change in there, but my goal is to keep them to a minimum.

> Also, I would like to see some serious benchmarks of the bitmap 
> algorithm changes before they go in.  They seem nice in theory, and some 
> users liked them for their uses, but that does not make a serious 
> scientific study.  Such a study has a high chance of making them even 
> better.;-)
> 

Some benchmarks have been posted on reiserfs-list, but I'd love to
coordinate with you on getting some mongo numbers.  I can spout off a
long list of places where the code does better then the original v3
allocator, but that's because those were the ones I was trying to fix
;-)

> zam, I view you as the block allocator maintainer, please review that 
> bitmap code from Chris.
> 
> Chris and Jeff, can you propose a benchmarking plan for the bitmap code?

A good start would be to just rebenchmark against v4.

-chris

Re: I would like to see ReiserFS V3 enter a feature freeze real soon.

[Hans Reiser]

Chris Mason wrote:

>On Mon, 2004-05-31 at 12:48, Hans Reiser wrote:
>  
>
>>While I like and appreciate the data journaling stuff, and I think it 
>>should go in, real soon now I think we should avoid adding new features 
>>to V3.  Let the mission critical server folks have a reiserfs version 
>>that only gets bug fixes added to it, and let V4 be for those who want 
>>excitement.
>>
>>Are there any things which Chris and Jeff think should go in besides 
>>data journaling/ordering and bitmap algorithm changes?
>>
>>    
>>
>
>We've got io error fault tolerance that needs to go in after the barrier
>code has stabilized. 
>
Ok, this qualifies as bug fixing or close enough.;-)

> I can't promise that I'll never making another
>change in there, but my goal is to keep them to a minimum.
>
>  
>
>>Also, I would like to see some serious benchmarks of the bitmap 
>>algorithm changes before they go in.  They seem nice in theory, and some 
>>users liked them for their uses, but that does not make a serious 
>>scientific study.  Such a study has a high chance of making them even 
>>better.;-)
>>
>>    
>>
>
>Some benchmarks have been posted on reiserfs-list, but I'd love to
>coordinate with you on getting some mongo numbers. 
>
Ok.

> I can spout off a
>long list of places where the code does better then the original v3
>allocator, but that's because those were the ones I was trying to fix
>;-)
>
>  
>
>>zam, I view you as the block allocator maintainer, please review that 
>>bitmap code from Chris.
>>
>>Chris and Jeff, can you propose a benchmarking plan for the bitmap code?
>>    
>>
>
>A good start would be to just rebenchmark against v4.
>  
>
V4 performance is not at a stable point at the moment I think, I have 
not been monitoring things closely due to trying to earn bucks 
consulting, and performance did not get tested every week, but there 
have been reports of performance decreasing and no reports of anyone 
investigating it, so I need to....

Elena, please compose a URL consisting of past benchmarks of various V4 
snapshots and send it to me.  (I did not read the last one you sent, 
sorry about that, so include the contents of that one also).

If the objective is to determine if the algorithm is good, then we 
should test it with only the algorithm in question changed.

I would be quite happy to add the algorithm to V4 (or Chris and Jeff can 
do that) and test it on vs. off.

Zam, place that on your todo list and send the todo list to me....

>-chris
>
>
>
>
>  
>

Re: I would like to see ReiserFS V3 enter a feature freeze real soon.

[Chris Mason]

On Tue, 2004-06-01 at 13:02, Hans Reiser wrote:

> > I can't promise that I'll never making another
> >change in there, but my goal is to keep them to a minimum.
> >
> >  
> >
> >>Also, I would like to see some serious benchmarks of the bitmap 
> >>algorithm changes before they go in.  They seem nice in theory, and some 
> >>users liked them for their uses, but that does not make a serious 
> >>scientific study.  Such a study has a high chance of making them even 
> >>better.;-)
> >>
> >>    
> >>
> >
> >Some benchmarks have been posted on reiserfs-list, but I'd love to
> >coordinate with you on getting some mongo numbers. 
> >
> Ok.

> >A good start would be to just rebenchmark against v4.
> >  
> >
> V4 performance is not at a stable point at the moment I think, I have 
> not been monitoring things closely due to trying to earn bucks 
> consulting, and performance did not get tested every week, but there 
> have been reports of performance decreasing and no reports of anyone 
> investigating it, so I need to....
> 

Sure, since v4 is being done again -mm right now (right?) you can just
benchmark against a few of the new options.  mount -o alloc=skip_busy
will give you the old allocator.

> Elena, please compose a URL consisting of past benchmarks of various V4 
> snapshots and send it to me.  (I did not read the last one you sent, 
> sorry about that, so include the contents of that one also).
> 
> If the objective is to determine if the algorithm is good, then we 
> should test it with only the algorithm in question changed.
> 
> I would be quite happy to add the algorithm to V4 (or Chris and Jeff can 
> do that) and test it on vs. off.

The algorithm has a few key components, but v4 doesn't need most of it. 
The part to inherit packing localities down the chain would be most
interesting in v4.

The rest approximates things v4 should already be good at, like grouping
some of the metadata near the data blocks.

-chris