Re: 2.5.12 severe ext3 filesystem corruption warning!

Re: 2.5.12 severe ext3 filesystem corruption warning!

[Andries.Brouwer]

>> 2.5.12, serious ext3 filesystem corrupting behavior

I have had problems with 2.5.10 (first few blocks of the root
filesystem overwritten) and then went back to 2.5.8 that I had
used for a while already, but then also noticed corruption there.
Back at 2.4.17 today..

In my case the problem was almost certainly the IDE code.
More in particular, the 2.5.8 corruption happened on four
different occasions, on two different disks, hanging off
an HPT366 that is without problems on 2.4*. Three of the
four times there were messages like

Apr 29 15:26:00 kernel: hde: task_out_intr: status=0x51 { DriveReady SeekComplete Error }
Apr 29 15:26:00 kernel: hde: task_out_intr: error=0x04 { DriveStatusError }

May  2 01:21:23 kernel: hdf: status error: status=0x50 { DriveReady SeekComplete }
May  2 01:21:23 kernel: hdf: no DRQ after issuing WRITE
May  2 01:21:37 kernel: hdf: task_out_intr: status=0x51 { DriveReady SeekComplete Error }
May  2 01:21:37 kernel: hdf: task_out_intr: error=0x04 { DriveStatusError }

Each time some data was written at a wrong address on disk.
Now these are ext2 filesystems, so I noticed.
Elsewhere I have ext3 and reiserfs, but journalling does not
protect against IDE drivers that write stuff to the wrong disk block.

Andries

Re: 2.5.12 severe ext3 filesystem corruption warning!

[Martin Dalecki]

Użytkownik Andries.Brouwer@cwi.nl napisał:
>>>2.5.12, serious ext3 filesystem corrupting behavior
>>
> 
> I have had problems with 2.5.10 (first few blocks of the root
> filesystem overwritten) and then went back to 2.5.8 that I had
> used for a while already, but then also noticed corruption there.
> Back at 2.4.17 today..
> 
> In my case the problem was almost certainly the IDE code.
> More in particular, the 2.5.8 corruption happened on four
> different occasions, on two different disks, hanging off
> an HPT366 that is without problems on 2.4*. Three of the
> four times there were messages like

Hmm... let me assume that you are using UDMA on all those drives.
Since you have apparently a system with quite a lot of
different simultanecousy active drives in them it could be very well possible
that the code that determined the do_request drive selection strategy
was the cause of your problems. It could very well be that
the recent changes with respect to this actually could have cured
this. (2.5.8 is the time around where die PADAM_ tags got introduced
there.

> Apr 29 15:26:00 kernel: hde: task_out_intr: status=0x51 { DriveReady SeekComplete Error }
> Apr 29 15:26:00 kernel: hde: task_out_intr: error=0x04 { DriveStatusError }
> 
> May  2 01:21:23 kernel: hdf: status error: status=0x50 { DriveReady SeekComplete }
> May  2 01:21:23 kernel: hdf: no DRQ after issuing WRITE
> May  2 01:21:37 kernel: hdf: task_out_intr: status=0x51 { DriveReady SeekComplete Error }
> May  2 01:21:37 kernel: hdf: task_out_intr: error=0x04 { DriveStatusError }
> 
> Each time some data was written at a wrong address on disk.
> Now these are ext2 filesystems, so I noticed.
> Elsewhere I have ext3 and reiserfs, but journalling does not
> protect against IDE drivers that write stuff to the wrong disk block.

Re: 2.5.12 severe ext3 filesystem corruption warning!

[Andrew Morton]

Andries.Brouwer@cwi.nl wrote:
> 
> >> 2.5.12, serious ext3 filesystem corrupting behavior
> 
> I have had problems with 2.5.10 (first few blocks of the root
> filesystem overwritten) and then went back to 2.5.8 that I had
> used for a while already, but then also noticed corruption there.
> Back at 2.4.17 today..
> 
> In my case the problem was almost certainly the IDE code.

hmm.  Maybe.  As Al says, the fact that it concerned
mapped files is deeply fishy.

Daniel, could you please check your logs for IDE
errors and "buffer layer errors" and also tell us
(as much as you can remember) the names of all the
affected files, and what application would have
written them.

Thanks.

-

Re: 2.5.12 severe ext3 filesystem corruption warning!

[Daniel Pittman]

On Thu, 02 May 2002, Andrew Morton wrote:
> Andries.Brouwer@cwi.nl wrote:
>> 
>> >> 2.5.12, serious ext3 filesystem corrupting behavior
>> 
>> I have had problems with 2.5.10 (first few blocks of the root
>> filesystem overwritten) and then went back to 2.5.8 that I had
>> used for a while already, but then also noticed corruption there.
>> Back at 2.4.17 today..
>> 
>> In my case the problem was almost certainly the IDE code.

For what it's worth, I was aware of the risks with the new IDE code and
have been monitoring this myself. Your notes on it are worth it. :)

> hmm.  Maybe.  As Al says, the fact that it concerned
> mapped files is deeply fishy.
> 
> Daniel, could you please check your logs for IDE
> errors and "buffer layer errors" and 

Sure. I scanned the full set of logs for the time and found no instances
of anything reporting an IDE error or a buffer layer error. Sorry.

> also tell us (as much as you can remember) the names of all the
> affected files, and what application would have written them.

The list of applications is as I mentioned in my other mail on the
topic. I should have read ahead more. The file list, to my knowledge,
is:

~/.gkrellm/user_config                          (gkrellm)
~/.galeon/bookmarks.xbel                        (galeon)
~/.galeon/mozilla/galeon/prefs.js               (mozilla in galeon)
~./newsrc.eld                                   (XEmacs/gnus)

Also, a number of items in ~/Mail/, all written by XEmacs/Gnus. Around
half a dozen groups were effected -- every single one that had mail
added to it during while running under the broken kernel.

The files, specifically, were:

~/Mail/<group>/.overview
~/Mail/<group>/.marks
~/Mail/<group>/[1-9]+

~/News/<group>.SCORE

The last item are the individual email messages, stored in one file per
message, with the INN-style .overview file containing the relevant
cached headers from the messages.

The .marks file contains a Lisp expression defining what has been seen,
flagged, etc, in Gnus for that group. It's updated when you look at
messages in the summary buffer (seen) as well as other things.

Finally, the SCORE files are a killfile on steroids; things add and
subtract from the score and, based on the numeric result, you can act
automatically on the messages.

The only corrupted SCORE files I found were the ones that use
"temporary" scores -- things that are reduced gradually over time.

This is the one case that lets me absolutely confirm that it's only
files that are *written* that were corrupted -- the SCORE files that
were used in a read-only fashion are fine and dandy.


So, the mail group metadata and the mail messages themselves were all
corrupted if written at the time.


Oh, and finally, I forgot to include the dumpe2fs info for the
filesystems before, my bad. They are included, from the mounted
filesystems. :)

        Daniel

] dumpe2fs -h /dev/discs/disc0/part7 # /home/daniel
dumpe2fs 1.27 (8-Mar-2002)
Filesystem volume name:   daniel
Last mounted on:          <not available>
Filesystem UUID:          e93e49be-05d5-4ae5-8ac7-a8883b9b607f
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal filetype needs_recovery sparse_super
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              1921024
Block count:              7680944
Reserved block count:     0
Free blocks:              6063023
Free inodes:              1678311
First block:              0
Block size:               2048
Fragment size:            2048
Blocks per group:         16384
Fragments per group:      16384
Inodes per group:         4096
Inode blocks per group:   256
Last mount time:          Thu May  2 18:13:59 2002
Last write time:          Thu May  2 18:13:59 2002
Mount count:              1
Maximum mount count:      37
Last checked:             Thu May  2 18:13:49 2002
Check interval:           15552000 (6 months)
Next check after:         Tue Oct 29 19:13:49 2002
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:		  128
Journal UUID:             <none>
Journal inode:            8
Journal device:	          0x0000
First orphan inode:       1253429

] dumpe2fs -h /dev/discs/disc0/part6 # /
dumpe2fs 1.27 (8-Mar-2002)
Filesystem volume name:   root
Last mounted on:          <not available>
Filesystem UUID:          8fb25561-51cc-4fd6-b0cd-0cb2e22a7b07
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal filetype needs_recovery sparse_super
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              1921024
Block count:              7680944
Reserved block count:     384047
Free blocks:              5540410
Free inodes:              1676136
First block:              0
Block size:               2048
Fragment size:            2048
Blocks per group:         16384
Fragments per group:      16384
Inodes per group:         4096
Inode blocks per group:   256
Last mount time:          Fri May  3 04:12:09 2002
Last write time:          Fri May  3 04:12:09 2002
Mount count:              1
Maximum mount count:      30
Last checked:             Fri May  3 04:12:03 2002
Check interval:           15552000 (6 months)
Next check after:         Wed Oct 30 05:12:03 2002
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:		  128
Journal UUID:             <none>
Journal inode:            8
Journal device:	          0x0000
First orphan inode:       381271

-- 
I am, as I said, inspired by the biological phenomena in which chemical
forces are used in repetitious fashion to produce all kinds of weird
effects (one of which is the author).
        -- Richard Feynman, _There's Plenty of Room at the Bottom_

Re: 2.5.12 severe ext3 filesystem corruption warning!

[Milton Miller]

Maybe this patch would make a difference ??

milton

===== mm/filemap.c 1.81 vs edited =====
--- 1.81/mm/filemap.c	Mon Apr 29 17:18:54 2002
+++ edited/mm/filemap.c	Fri May  3 20:47:35 2002
@@ -287,11 +287,11 @@
 	clean_list_pages(mapping, &mapping->io_pages, start);
 	clean_list_pages(mapping, &mapping->dirty_pages, start);
 	do {
-		unlocked |= truncate_list_pages(mapping,
+		unlocked = truncate_list_pages(mapping,
 				&mapping->io_pages, start, &partial);
 		unlocked |= truncate_list_pages(mapping,
 				&mapping->dirty_pages, start, &partial);
-		unlocked = truncate_list_pages(mapping,
+		unlocked |= truncate_list_pages(mapping,
 				&mapping->clean_pages, start, &partial);
 		unlocked |= truncate_list_pages(mapping,
 				&mapping->locked_pages, start, &partial);

Re: 2.5.12 severe ext3 filesystem corruption warning!

[Andrew Morton]

Milton Miller wrote:
> 
> Maybe this patch would make a difference ??

Well it certainly won't hurt.  Thanks.

-

2.5.12 severe ext3 filesystem corruption warning!

[Daniel Pittman]

I gave the 2.5.12 kernel a shot on my workstation tonight and found an
*extremely* serious ext3 filesystem corrupting behavior.

The only files that were mangled were files that had been modified while
running, so it looks like this is an issue with finding the contents of
modified files to write, not with random data dropping onto the disk.

That said, however, the contents were rather random -- blocks from
.overview were placed into a number of other files, chunks of
.newsrc.eld written to the gkrellm configuration file and the like.

Unmodified files, however, don't seem to have been touched. This can't
be perfect, of course, as something I never look at may have been
destroyed, but it seems to be reliably.

The system is a mobile P4, 512MB RAM, IDE disk. Only one filesystem
seems badly effected, my home directory, which is ext3 and fully
data-journaled.

I couldn't find corruption on the root filesystem[1] but there isn't
much of that which is actively written at runtime. Nothing notable in
/var seems broken, thankfully, so no panic. :)

Anyway, I don't know if anyone else is seeing the problems but this is a
first worrying datapoint with the kernel...

        Daniel

Footnotes: 
[1]  Contains everything else, including /usr and /var

-- 
20+ years as a vegetarian and the guy who steals my credit card
orders $6,000 worth of chicken parts: proof that the most powerful
force in the universe is Irony.
        -- David Weinberger, _JOHO_ (2000-03-20)

Re: 2.5.12 severe ext3 filesystem corruption warning!

[Andrew Morton]

Daniel Pittman wrote:
> 
> I gave the 2.5.12 kernel a shot on my workstation tonight and found an
> *extremely* serious ext3 filesystem corrupting behavior.

A few things..

Are your other filesystems using journalled data as well?

Are you sure that all kernel files were recompiled?  If,
for example, you had some 2.5.11 objects in the link, that
would be bad.

Do you know whether the bad data is actually on-disk, or
could it be just in-RAM?  ie: was the data still bad after
a reboot?

What blocksize is that filesystem using?  The output of
`dumpe2fs -h /dev/whatever' will tell you this.

Can you please force an fsck against that filesystem,
see what it says?

Thanks. 

-

Re: 2.5.12 severe ext3 filesystem corruption warning!

[Alexander Viro]

On Thu, 2 May 2002, Andrew Morton wrote:
> A few things..
> 
[snip]

Andrew, judging by the filenames he'd mentioned, I suspect that he runs
innd.  I.e. one of the very few programs heavily using truncate().  And
no, it doesn't promise anything good - last time we had crap in truncate/mmap
interaction it was a hell to fix.

I suspect that you had screwed the truncate exclusion warranties up.  If
_any_ IO happens in the area currently manipulated by ->truncate() - you
are screwed and results would look pretty much like the things mentioned
in bug report.

Re: 2.5.12 severe ext3 filesystem corruption warning!

[Andrew Morton]

Alexander Viro wrote:
> 
> On Thu, 2 May 2002, Andrew Morton wrote:
> > A few things..
> >
> [snip]
> 
> Andrew, judging by the filenames he'd mentioned, I suspect that he runs
> innd.  I.e. one of the very few programs heavily using truncate().  And
> no, it doesn't promise anything good - last time we had crap in truncate/mmap
> interaction it was a hell to fix.
> 
> I suspect that you had screwed the truncate exclusion warranties up.  If
> _any_ IO happens in the area currently manipulated by ->truncate() - you
> are screwed and results would look pretty much like the things mentioned
> in bug report.

OK, thanks.  That's something to go on.

The only change which comes to mind is discard_buffer() - it's
no longer clearing BH_Uptodate.  Because for the partial page
at the end of the mapping, buffers outside i_size were zeroed
in truncate_partial_page().  But with (presumed) 4k blocksize,
it can't be that.

ext3_writepage() can hold a reference against the buffers
after the page has come unlocked, so a try_to_free in the
right window will fail, which will leave the page floating
about on the page LRU, not mapped into any address_space,
clean, not uptodate, but with uptodate buffers.  Nobody
but the VM should ever find that page, but it does make more
sense to mark that buffer not uptodate.  This would only
happen on super-heavy SMP loads.

So hmm.

There's a small SMP-only race in truncate_list_pages:
the test for PageWriteback happens against an unlocked
page.  There's a three-instruction window where writeback
could start up.  That won't be it, but I'll fix that.

-

Re: 2.5.12 severe ext3 filesystem corruption warning!

[Daniel Pittman]

On Thu, 02 May 2002, Andrew Morton wrote:
> Alexander Viro wrote:
>> On Thu, 2 May 2002, Andrew Morton wrote:
>> > A few things..

[...]

>> I suspect that you had screwed the truncate exclusion warranties up. 
>> If _any_ IO happens in the area currently manipulated by ->truncate()
>> - you are screwed and results would look pretty much like the things
>> mentioned in bug report.
> 
> OK, thanks.  That's something to go on.
> 
> The only change which comes to mind is discard_buffer() - it's
> no longer clearing BH_Uptodate.  Because for the partial page
> at the end of the mapping, buffers outside i_size were zeroed
> in truncate_partial_page().  But with (presumed) 4k blocksize,
> it can't be that.
> 
> ext3_writepage() can hold a reference against the buffers
> after the page has come unlocked, so a try_to_free in the
> right window will fail, which will leave the page floating
> about on the page LRU, not mapped into any address_space,
> clean, not uptodate, but with uptodate buffers.  Nobody
> but the VM should ever find that page, but it does make more
> sense to mark that buffer not uptodate.  This would only
> happen on super-heavy SMP loads.

Oh, it's a laptop, so SMP is right out of the picture. ;)
        Daniel

-- 
Many of my favorite shamans are rock stars. They probably don't even know
they're shamans but they know how to get to ecstasy and back, and how to take
others with them. They may not have a license, but they know how how to drive.
        -- Gabrielle Roth, _Maps to Ecstasy_

Re: 2.5.12 severe ext3 filesystem corruption warning!

[Daniel Pittman]

On Thu, 02 May 2002, Andrew Morton wrote:
> Daniel Pittman wrote:
>> 
>> I gave the 2.5.12 kernel a shot on my workstation tonight and found
>> an *extremely* serious ext3 filesystem corrupting behavior.
> 
> A few things..
> 
> Are your other filesystems using journalled data as well?

Yes:

] mount
/dev/discs/disc0/part6 on / type ext3 (rw,data=journal)
none on /proc type proc (rw)
none on /proc/bus/usb type usbdevfs (rw)
/dev/discs/disc0/part7 on /home/daniel type ext3 (rw,data=journal)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
anu.rimspace.net:/music on /music type nfs (rw,noexec,nosuid,nodev,rsize=8192,wsize=8192,nfsvers=3,bg,soft,addr=210.23.138.21)

> Are you sure that all kernel files were recompiled?  If,
> for example, you had some 2.5.11 objects in the link, that
> would be bad.

Yes. The kernel was build with clean, oldconfig, dep and then building
the image and modules. 

> Do you know whether the bad data is actually on-disk, or
> could it be just in-RAM?  ie: was the data still bad after
> a reboot?

I only found it after the reboot; being the cautious sort I rebooted as
soon as it showed odd data issues, back to the known stable 2.5.6 and
did a full fsck run. That showed no issues, which was nice, but I found
the data corruption.

> What blocksize is that filesystem using?  The output of
> `dumpe2fs -h /dev/whatever' will tell you this.

2k blocks, as I found that 4k was giving me over 40% wasted space last
time I bothered to check. I have most of the files in this filesystem
being ~2K in size.

Full output of the command is attached below.

> Can you please force an fsck against that filesystem,
> see what it says?

I did. The filesystem itself was clean -- not even a warning. Just data
corruption, not metadata.

On Thu, 2 May 2002, Alexander Viro wrote:
> On Thu, 2 May 2002, Andrew Morton wrote:
>> A few things..
>> 
> [snip]
> 
> Andrew, judging by the filenames he'd mentioned, I suspect that he
> runs innd. I.e. one of the very few programs heavily using truncate().

Not quite. I found the corruption in three places:

* my XEmacs and Gnus mail spool.
* gkrellm configuration files.
* galeon/mozilla bookmarks and preferences.

XEmacs uses the mode (O_WRONLY | O_TRUNC) when it writes out files, so
everything that was written there would have been truncated before
output. There shouldn't have been any mmap() of the files, though.

It's also worth noting that XEmacs would fsync() the file handle after
writing the content, for what that's worth.


I found corruption in the galeon bookmarks file, which seemed to start
writing XML half way through the actual data and to add a block of
around 2K NULL bytes half way through the content.

I also found corruption in the mozilla prefs.js file from the same
application[1]. That was simply a truncated file -- no NULL bytes or
anything, just a file that cut off half way through a single expression
like:

user_pref("wallet.capture

This /may/ be the logical break between two write(2) calls or a partly
completed write, though.

> And no, it doesn't promise anything good - last time we had crap in
> truncate/mmap interaction it was a hell to fix.
> 
> I suspect that you had screwed the truncate exclusion warranties up. 
> If _any_ IO happens in the area currently manipulated by ->truncate()
> - you are screwed and results would look pretty much like the things
> mentioned in bug report.

Ick. Er, good luck. I am quite happy to provide more information and
even to install a scratch disk and try to get it to fail the same way if
you wish.

I can't use the same kernel build, though, as I removed the binary
kernel. I can build an equivalent one, of course, but it's not quite the
same. Sorry about that -- I did it before thinking about it. :/

Let me know if there is any more information that would be useful, other
than the corrupted files. I killed those, also, without thinking. :(

        Daniel

Footnotes: 
[1]  Galeon is a GNOME front-end for Mozilla, in case you didn't know.

-- 
The English have the most rigid code of immorality in the world.
        -- Malcom Bradbury, _Eating People is Wrong_

Re: 2.5.12 severe ext3 filesystem corruption warning!

[Andrew Morton]

Daniel, this is good stuff.  Thanks.

Daniel Pittman wrote:
> 
> ...
> 
> Not quite. I found the corruption in three places:
> 
> * my XEmacs and Gnus mail spool.
> * gkrellm configuration files.
> * galeon/mozilla bookmarks and preferences.

So all of those files were written to by their application
during the failed session.

Question on the table is: did the kernel write the wrong
stuff into those files, or did it forget to write the
right stuff?

Is the incorrect content within those files recognisable
at all?  If so, what is it?

> XEmacs uses the mode (O_WRONLY | O_TRUNC) when it writes out files, so
> everything that was written there would have been truncated before
> output. There shouldn't have been any mmap() of the files, though.
> 
> It's also worth noting that XEmacs would fsync() the file handle after
> writing the content, for what that's worth.
> 
> I found corruption in the galeon bookmarks file, which seemed to start
> writing XML half way through the actual data and to add a block of
> around 2K NULL bytes half way through the content.
> 
> I also found corruption in the mozilla prefs.js file from the same
> application[1]. That was simply a truncated file -- no NULL bytes or
> anything, just a file that cut off half way through a single expression
> like:
> 
> user_pref("wallet.capture
> 
> This /may/ be the logical break between two write(2) calls or a partly
> completed write, though.

That sounds like metadata corruption.  Are you sure that
the file doesn't have a chunk of invisible nulls in it,
after the text?  Because if it got chopped off in this
manner, e2fsck should have detected something.

> > And no, it doesn't promise anything good - last time we had crap in
> > truncate/mmap interaction it was a hell to fix.
> >
> > I suspect that you had screwed the truncate exclusion warranties up.
> > If _any_ IO happens in the area currently manipulated by ->truncate()
> > - you are screwed and results would look pretty much like the things
> > mentioned in bug report.
> 
> Ick. Er, good luck. I am quite happy to provide more information and
> even to install a scratch disk and try to get it to fail the same way if
> you wish.

It would be good if you had the time to do that.
 

-

Re: 2.5.12 severe ext3 filesystem corruption warning!

[Daniel Pittman]

On Thu, 02 May 2002, Andrew Morton wrote:
> Daniel, this is good stuff.  Thanks.
> Daniel Pittman wrote:
>> 
>> ...
>> 
>> Not quite. I found the corruption in three places:
>> 
>> * my XEmacs and Gnus mail spool.
>> * gkrellm configuration files.
>> * galeon/mozilla bookmarks and preferences.
> 
> So all of those files were written to by their application
> during the failed session.

Yes. Files that they accessed in a read-only fashion, and untouched
files, seem entirely unharmed.

> Question on the table is: did the kernel write the wrong
> stuff into those files, or did it forget to write the
> right stuff?

Wrong stuff.

> Is the incorrect content within those files recognisable
> at all?  If so, what is it?

Yes. There are three things that I noted as incorrect in the files.

Many of them, but not all, featured large blocks of NULL (0) bytes, on
the order of 2K worth of them. No exacting counts available but they did
fill a page in an 80x30 terminal, generally.

Several of the files featured the "right" content, just in the wrong
location. For example, the galeon 'bookmarks.xbel' file started about a
third of the way into it's correct content.

Only one of the files seemed to be gratuitously truncated, the mozilla
prefs.js file.


The remaining file content seemed to be randomly swizzled. I found part
of a .overview file in .newsrc.eld, for example, and part of the
.newsrc.eld elsewhere. Also, 'active' got written to a mail folder...


There were *no* cases where the file content was unrecognizable with the
exception of the NULL bytes. In *every* case the content was from a
*completely* different file.

This included files in different directories getting content swapped, so
a file in ~ got content from ~/Mail/<group>/, so it's not just within
the same directory.


I can't think of a single verifiable case where a file got the wrong
data from another process, though. That's not a "didn't", just a "can't
remember seeing", though, so don't take that as gospel.

[...]

>> I also found corruption in the mozilla prefs.js file from the same
>> application[1]. That was simply a truncated file -- no NULL bytes or
>> anything, just a file that cut off half way through a single
>> expression like:
>> 
>> user_pref("wallet.capture
>> 
>> This /may/ be the logical break between two write(2) calls or a
>> partly completed write, though.
> 
> That sounds like metadata corruption. Are you sure that the file
> doesn't have a chunk of invisible nulls in it, after the text? 

Yes. I used an editor that displays the NULL byte correctly in files to
view it when looking for damage and there was *no* NULL content in this
file.

None of the NULL bytes were at the end of the file, always in the
middle of the data. 

> Because if it got chopped off in this manner, e2fsck should have
> detected something.

Yup. I carefully watched the forced fsck and no errors. This is:
] fsck.ext3 -V
e2fsck 1.27 (8-Mar-2002)
	Using EXT2FS Library version 1.27, 8-Mar-2002

>> > And no, it doesn't promise anything good - last time we had crap in
>> > truncate/mmap interaction it was a hell to fix.
>> >
>> > I suspect that you had screwed the truncate exclusion warranties
>> > up. If _any_ IO happens in the area currently manipulated by
>> > ->truncate() - you are screwed and results would look pretty much
>> > like the things mentioned in bug report.
>> 
>> Ick. Er, good luck. I am quite happy to provide more information and
>> even to install a scratch disk and try to get it to fail the same way
>> if you wish.
> 
> It would be good if you had the time to do that.

Sure. I will have a run at reproducing it in an hour or two, then post
the results to the list under this thread, with a direct CC. :)

        Daniel

-- 
Nobody objects to a woman being a good writer or sculptor or geneticist if at
the same time she manages to be a good wife, good mother, good-looking,
well-groomed and unaggressive.
        -- Leslie M. McIntyre