Catastrophe with mailboxes on ReiserFS

Catastrophe with mailboxes on ReiserFS

[Robert Tiismus]

Hi!
	I have been using Reiserfs on mail server for two months. Today
a _Very Bad Thing_ (VBT) happened. I would appreciate if someone would give a
helpful comment about IT (VBT) being related with the incident. The
result of the mess was that some blocks of user1 mailbox were placed in
the middle of user2 mailbox, and some blocks from user2 mailbox were
palced into user3 mailbox. The result was surprisingly good: all three
mailboxes were readable by pine/elm/pop3. I speculate that this happened
during previous malfunction during disk access. I had added some devices to
server and did not close the case to check if the server does not
malfunction with the new hardware. I think that when closing the side
panel, I misaligned the scsi cable from the connector and disk access
malfunctioned. Then, after rebooting the system, everything seemed ok, besides
this unfortunate inident with three mailboxes :(
Is it possible, that VBT happened during following reboot?

System is Debian GNU/Linux 2.4.18 (Woody). ReiserFS version 3.6.25

The fstab line looks like this:

/dev/sda2	/	 reiserfs	defaults	0	0

Best wishes,

Robert Tiismus

Re: Catastrophe with mailboxes on ReiserFS

[Oleg Drokin]

Hello!

On Tue, Oct 08, 2002 at 11:55:42PM +0300, Robert Tiismus wrote:
> malfunction with the new hardware. I think that when closing the side
> panel, I misaligned the scsi cable from the connector and disk access
> malfunctioned. Then, after rebooting the system, everything seemed ok, besides
> this unfortunate inident with three mailboxes :(

Behaviour of hardware is unpredicatble on bad wiring and that kind of stuff.

> Is it possible, that VBT happened during following reboot?

If the wiring was already ok, then this is very unlikely.

Also you probably want to run reiserfsck on that disk to make sure
no other damage happened.

Bye,
    Oleg

Re: Catastrophe with mailboxes on ReiserFS

[Robert Tiismus]

On Wed, 9 Oct 2002, Oleg Drokin wrote:

> Hello!
>
> On Tue, Oct 08, 2002 at 11:55:42PM +0300, Robert Tiismus wrote:
> > malfunction with the new hardware. I think that when closing the side
> > panel, I misaligned the scsi cable from the connector and disk access
> > malfunctioned. Then, after rebooting the system, everything seemed ok, besides
> > this unfortunate inident with three mailboxes :(
>
> Behaviour of hardware is unpredicatble on bad wiring and that kind of stuff.
>
> > Is it possible, that VBT happened during following reboot?
>
> If the wiring was already ok, then this is very unlikely.
>
> Also you probably want to run reiserfsck on that disk to make sure
> no other damage happened.
Thank you. Reiserfsck said that all is ok. It's just that I have seen nothing
similar happening with other filesystems. I would prefer disappearing
data to leaking data. Am I completely wrong, when I say that because of
'tail packing' in ReiserFS, it can leak information with more
probability than, lets say ext2 or ufs or...? Will notails option give
me more secure FS? I have to assure company management, and myself, that this
incident happened not because I put ReiserFS on new server. It could
have happened also with other filesystems :)

Robert Tiismus

Re: Catastrophe with mailboxes on ReiserFS

[Oleg Drokin]

Hello!

On Wed, Oct 09, 2002 at 11:06:36AM +0300, Robert Tiismus wrote:

> > Also you probably want to run reiserfsck on that disk to make sure
> > no other damage happened.
> Thank you. Reiserfsck said that all is ok. It's just that I have seen nothing

This is good.

> similar happening with other filesystems. I would prefer disappearing
> data to leaking data. Am I completely wrong, when I say that because of
> 'tail packing' in ReiserFS, it can leak information with more
> probability than, lets say ext2 or ufs or...? Will notails option give

No, tail packing should have no effect on leakign data.
Anyway tail packing is only takes effect for files less than 16K in size.

> me more secure FS? I have to assure company management, and myself, that this

notails will have no effect on that.

> incident happened not because I put ReiserFS on new server. It could
> have happened also with other filesystems :)

Indeed it seems that you've got either some metadata altered (block pointers)
or the block content swapped somehow, both of which could happen on any FS
with absolutely same results.
BTW, I just remembered that until you apply Chris' Mason "data logging" patches,
there is a certain window where system crash would lead to "deleted" 
data appearing at the end of files that were appended before the crash.
(that's it, metadata is already updated and list newly allocated blocknumbers,
but old content od those blocks is still intact since crash prevented system
from putting new content in there), but since you've got other file's data
in the middle of file, this is not the case.

Bye,
    Oleg
> 
> Robert Tiismus
> 

Re: Catastrophe with mailboxes on ReiserFS

[Dieter Nützel]

Am Mittwoch, 9. Oktober 2002 10:18 schrieb Oleg Drokin:
> Hello!
>
> On Wed, Oct 09, 2002 at 11:06:36AM +0300, Robert Tiismus wrote:
> > > Also you probably want to run reiserfsck on that disk to make sure
> > > no other damage happened.
> >
> > Thank you. Reiserfsck said that all is ok. It's just that I have seen
> > nothing
>
> This is good.
>
> > similar happening with other filesystems. I would prefer disappearing
> > data to leaking data. Am I completely wrong, when I say that because of
> > 'tail packing' in ReiserFS, it can leak information with more
> > probability than, lets say ext2 or ufs or...? Will notails option give
>
> No, tail packing should have no effect on leakign data.
> Anyway tail packing is only takes effect for files less than 16K in size.
>
> > me more secure FS? I have to assure company management, and myself, that
> > this
>
> notails will have no effect on that.
>
> > incident happened not because I put ReiserFS on new server. It could
> > have happened also with other filesystems :)
>
> Indeed it seems that you've got either some metadata altered (block
> pointers) or the block content swapped somehow, both of which could happen
> on any FS with absolutely same results.
> BTW, I just remembered that until you apply Chris' Mason "data logging"
> patches, there is a certain window where system crash would lead to
> "deleted" data appearing at the end of files that were appended before the
> crash. (that's it, metadata is already updated and list newly allocated
> blocknumbers, but old content od those blocks is still intact since crash
> prevented system from putting new content in there), but since you've got
> other file's data in the middle of file, this is not the case.

Oleg, I "see" it from time to time during kernel and DRI development (X server 
crashes), too. This times without Chris's stuff. 2.4.19-ck5 (latest ReiserFS 
patches for 2.4.19) and now with 2.5.40-ac3.

Most times "messages", "localmessages", kernel .o.*, dep files, etc. are 
broken (wrong stuff or mixed stuff).

Latest my ".q3a/baseq3/q3key" file (DRI SMP tests) was broken (completely 
other stuff in it) even though it only should opened for reading.

-Dieter

PS I send you a compressed copy of my "messages" file in private.

Re: Catastrophe with mailboxes on ReiserFS

[Oleg Drokin]

Hello!

On Wed, Oct 09, 2002 at 04:02:34PM +0200, Dieter N?tzel wrote:

> > prevented system from putting new content in there), but since you've got
> > other file's data in the middle of file, this is not the case.
> Oleg, I "see" it from time to time during kernel and DRI development (X server 
> crashes), too. This times without Chris's stuff. 2.4.19-ck5 (latest ReiserFS 
> patches for 2.4.19) and now with 2.5.40-ac3.

What's "it"? Is it means what I describe, or "it" means wrong stuff in the
middle of files?
> 
> Most times "messages", "localmessages", kernel .o.*, dep files, etc. are 
> broken (wrong stuff or mixed stuff).

These are recently modified stuff.

> Latest my ".q3a/baseq3/q3key" file (DRI SMP tests) was broken (completely 
> other stuff in it) even though it only should opened for reading.

ls -la .q3a/baseq3/q3key?

> PS I send you a compressed copy of my "messages" file in private.

It only confirms my version, note how whole block of zeroes
was inserted after emergency sync (perhaps during crash).

Bye,
    Oleg

Re: Catastrophe with mailboxes on ReiserFS

[Dieter Nützel]

Am Mittwoch, 9. Oktober 2002 16:09 schrieb Oleg Drokin:
> Hello!
>
> On Wed, Oct 09, 2002 at 04:02:34PM +0200, Dieter N?tzel wrote:
> > > prevented system from putting new content in there), but since you've
> > > got other file's data in the middle of file, this is not the case.
> >
> > Oleg, I "see" it from time to time during kernel and DRI development (X
> > server crashes), too. This times without Chris's stuff. 2.4.19-ck5
> > (latest ReiserFS patches for 2.4.19) and now with 2.5.40-ac3.
>
> What's "it"? Is it means what I describe, or "it" means wrong stuff in the
> middle of files?

Wrong stuff in the middle (appended) _AND_ sometimes whole broken files (if 
the files are smaller).

> > Most times "messages", "localmessages", kernel .o.*, dep files, etc. are
> > broken (wrong stuff or mixed stuff).
>
> These are recently modified stuff.

Yes.

> > Latest my ".q3a/baseq3/q3key" file (DRI SMP tests) was broken (completely
> > other stuff in it) even though it only should opened for reading.
>
> ls -la .q3a/baseq3/q3key?

Sorry, I had a copy of it elsewhere and deleted it ;-(

> > PS I send you a compressed copy of my "messages" file in private.
>
> It only confirms my version, note how whole block of zeroes
> was inserted after emergency sync (perhaps during crash).

"whole block of zeroes" would be fine but it isn't sometimes...
Sometimes it is stuff from other (deleted?) files.

When I get the "next" crash during kernel compilation I "save" some files for 
you.

-Dieter

Re: Catastrophe with mailboxes on ReiserFS

[Oleg Drokin]

Hello!

On Wed, Oct 09, 2002 at 04:20:15PM +0200, Dieter N?tzel wrote:
> > What's "it"? Is it means what I describe, or "it" means wrong stuff in the
> > middle of files?
> Wrong stuff in the middle (appended) _AND_ sometimes whole broken files (if 
> the files are smaller).

If the damaged part is still within appended datam, it is still possible that
not all of data happened to flush.

> > It only confirms my version, note how whole block of zeroes
> > was inserted after emergency sync (perhaps during crash).
> "whole block of zeroes" would be fine but it isn't sometimes...
> Sometimes it is stuff from other (deleted?) files.

Well, it depend on what was in newly allocated blocks, it might be zeroes for
empty blocks or some deleted data for previously used blocks.

> When I get the "next" crash during kernel compilation I "save" some files for 
> you.

Hm. This is not necessary, I know what's happening in your case it seems.

Bye,
    Oleg

Re: Catastrophe with mailboxes on ReiserFS

[Dieter Nützel]

Am Mittwoch, 9. Oktober 2002 16:09 schrieb Oleg Drokin:
> Hello!
>
> On Wed, Oct 09, 2002 at 04:02:34PM +0200, Dieter N?tzel wrote:
> > > prevented system from putting new content in there), but since you've
> > > got other file's data in the middle of file, this is not the case.
> >
> > Oleg, I "see" it from time to time during kernel and DRI development (X
> > server crashes), too. This times without Chris's stuff. 2.4.19-ck5
> > (latest ReiserFS patches for 2.4.19) and now with 2.5.40-ac3.
>
> What's "it"? Is it means what I describe, or "it" means wrong stuff in the
> middle of files?
>
> > Most times "messages", "localmessages", kernel .o.*, dep files, etc. are
> > broken (wrong stuff or mixed stuff).
>
> These are recently modified stuff.
>
> > Latest my ".q3a/baseq3/q3key" file (DRI SMP tests) was broken (completely
> > other stuff in it) even though it only should opened for reading.
>
> ls -la .q3a/baseq3/q3key?

OK, read it wrong. You mean "ls" NOT "less" ;-)

It is VERY, VERY small...

ls -la .q3a/baseq3/q3key
-rw-r--r--    1 nuetzel  users         167 Okt  9 05:47 .q3a/baseq3/q3key

:-)))

Regards,
	Dieter

Re: Catastrophe with mailboxes on ReiserFS

[Oleg Drokin]

Hello!

On Wed, Oct 09, 2002 at 04:23:31PM +0200, Dieter N?tzel wrote:

> > > Latest my ".q3a/baseq3/q3key" file (DRI SMP tests) was broken (completely
> > > other stuff in it) even though it only should opened for reading.
> > ls -la .q3a/baseq3/q3key?
> OK, read it wrong. You mean "ls" NOT "less" ;-)
> It is VERY, VERY small...
> ls -la .q3a/baseq3/q3key
> -rw-r--r--    1 nuetzel  users         167 Okt  9 05:47 .q3a/baseq3/q3key

Looks like recently modified for me ;)

Bye, 
    Oleg

Re: Catastrophe with mailboxes on ReiserFS

[Dieter Nützel]

Am Mittwoch, 9. Oktober 2002 16:32 schrieb Oleg Drokin:
> Hello!
>
> On Wed, Oct 09, 2002 at 04:23:31PM +0200, Dieter N?tzel wrote:
> > > > Latest my ".q3a/baseq3/q3key" file (DRI SMP tests) was broken
> > > > (completely other stuff in it) even though it only should opened for
> > > > reading.
> > >
> > > ls -la .q3a/baseq3/q3key?
> >
> > OK, read it wrong. You mean "ls" NOT "less" ;-)
> > It is VERY, VERY small...
> > ls -la .q3a/baseq3/q3key
> > -rw-r--r--    1 nuetzel  users         167 Okt  9 05:47 .q3a/baseq3/q3key
>
> Looks like recently modified for me ;)

Yes, copied over...;-)

But you are sure with "cp -a original".
And Q3A seems to "modify" it ;-(
Truly bad coding style...

So NOT ReiserFS to blame?!

Thanks,
	Dieter

Re: Catastrophe with mailboxes on ReiserFS

[Oleg Drokin]

Hello!

On Wed, Oct 09, 2002 at 05:15:20PM +0200, Dieter N?tzel wrote:

> But you are sure with "cp -a original".
> And Q3A seems to "modify" it ;-(
> Truly bad coding style...
> So NOT ReiserFS to blame?!

There is a reason thato blame reiserfs too, unfortunately.
But Cris' patches should change that.

Bye,
    Oleg

Re: Catastrophe with mailboxes on ReiserFS

[Dieter Nützel]

Am Mittwoch, 9. Oktober 2002 17:19 schrieb Oleg Drokin:
> Hello!
>
> On Wed, Oct 09, 2002 at 05:15:20PM +0200, Dieter N?tzel wrote:
> > But you are sure with "cp -a original".
> > And Q3A seems to "modify" it ;-(
> > Truly bad coding style...
> > So NOT ReiserFS to blame?!
>
> There is a reason thato blame reiserfs too, unfortunately.
> But Cris' patches should change that.

Yes, to "close" this thread, "data-logging" should be my friend ;-)
Hopefully soon for 2.5.41+.

Regards,
	Dieter

Re: Catastrophe with mailboxes on ReiserFS

[Dieter Nützel]

Am Mittwoch, 9. Oktober 2002 16:02 schrieb Dieter Nützel:
> Am Mittwoch, 9. Oktober 2002 10:18 schrieb Oleg Drokin:
> > Hello!
> >
> > On Wed, Oct 09, 2002 at 11:06:36AM +0300, Robert Tiismus wrote:
> > > > Also you probably want to run reiserfsck on that disk to make sure
> > > > no other damage happened.
> > >
> > > Thank you. Reiserfsck said that all is ok. It's just that I have seen
> > > nothing
> >
> > This is good.
> >
> > > similar happening with other filesystems. I would prefer disappearing
> > > data to leaking data. Am I completely wrong, when I say that because of
> > > 'tail packing' in ReiserFS, it can leak information with more
> > > probability than, lets say ext2 or ufs or...? Will notails option give
> >
> > No, tail packing should have no effect on leakign data.
> > Anyway tail packing is only takes effect for files less than 16K in size.
> >
> > > me more secure FS? I have to assure company management, and myself,
> > > that this
> >
> > notails will have no effect on that.
> >
> > > incident happened not because I put ReiserFS on new server. It could
> > > have happened also with other filesystems :)
> >
> > Indeed it seems that you've got either some metadata altered (block
> > pointers) or the block content swapped somehow, both of which could
> > happen on any FS with absolutely same results.
> > BTW, I just remembered that until you apply Chris' Mason "data logging"
> > patches, there is a certain window where system crash would lead to
> > "deleted" data appearing at the end of files that were appended before
> > the crash. (that's it, metadata is already updated and list newly
> > allocated blocknumbers, but old content od those blocks is still intact
> > since crash prevented system from putting new content in there), but
> > since you've got other file's data in the middle of file, this is not the
> > case.
>
> Oleg, I "see" it from time to time during kernel and DRI development (X
> server crashes), too. This times without Chris's stuff. 2.4.19-ck5 (latest
> ReiserFS patches for 2.4.19) and now with 2.5.40-ac3.
>
> Most times "messages", "localmessages", kernel .o.*, dep files, etc. are
> broken (wrong stuff or mixed stuff).
>
> Latest my ".q3a/baseq3/q3key" file (DRI SMP tests) was broken (completely
> other stuff in it) even though it only should opened for reading.

Oh, forgotten my mount options:

/dev/sda3 on / type reiserfs (rw,noatime,notail)
/dev/sda2 on /tmp type reiserfs (rw,notail)
/dev/sda5 on /var type reiserfs (rw,notail)
/dev/sda6 on /home type reiserfs (rw,notail)
/dev/sda7 on /usr type reiserfs (rw,notail)
/dev/sda8 on /opt type reiserfs (rw,notail)
/dev/sdb1 on /Pakete type reiserfs (rw,notail)
/dev/sdb5 on /database/db1 type reiserfs (rw,notail)
/dev/sdb6 on /database/db2 type reiserfs (rw,notail)
/dev/sdb7 on /database/db3 type reiserfs (rw,notail)
/dev/sdb8 on /database/db4 type reiserfs (rw,notail)

-Dieter

Re: Catastrophe with mailboxes on ReiserFS

[Matthias Andree]

Oleg Drokin <green@namesys.com> writes:

> BTW, I just remembered that until you apply Chris' Mason "data logging" patches,
> there is a certain window where system crash would lead to "deleted" 
> data appearing at the end of files that were appended before the
> crash.

I'd like to see these patches merged. I hurt myself several times with
reiserfs, when the computer locked up during heavy file system activity,
with NUL blocks in files, files mixed up and the like. I've never seen
this on any data=ordered or data=journal file system, regardless of ext3
or reiserfs.

Of course, write caches must be turned of or write barrier patches be
applied to be safe in case of a power blackout.

-- 
Matthias Andree

Re: Catastrophe with mailboxes on ReiserFS

[Newsmail]

What are those write barrier patches? and where can I find them? I was also 
hurted by several 'cross linked' files, described in here. is there a way 
to deactivate write caching for just one filesystem?
regards, greg

ps: please somebody let me know where I can find documentation about those 
chris mason patches, or at least what is dara=ordered or data=journal 
options. everybody talks about it, nobody explains. or did I miss something?

>I'd like to see these patches merged. I hurt myself several times with
>reiserfs, when the computer locked up during heavy file system activity,
>with NUL blocks in files, files mixed up and the like. I've never seen
>this on any data=ordered or data=journal file system, regardless of ext3
>or reiserfs.
>
>Of course, write caches must be turned of or write barrier patches be
>applied to be safe in case of a power blackout.
>
>--
>Matthias Andree

Re: Catastrophe with mailboxes on ReiserFS

[Matthias Andree]

On Thu, 10 Oct 2002, Newsmail wrote:

> What are those write barrier patches? and where can I find them? I was also 

I don't know. Some vendor kernels have them (SuSE), but I'm not sure
where they are available ATM.

> hurted by several 'cross linked' files, described in here. is there a way 
> to deactivate write caching for just one filesystem?

No. You can only disable it per drive.

> ps: please somebody let me know where I can find documentation about those 
> chris mason patches, or at least what is dara=ordered or data=journal 
> options. everybody talks about it, nobody explains. or did I miss something?

You can safely consult ext3 documentation on these options, reiserfs
should behave the same in respect to these options.

-- 
Matthias Andree

Re: Catastrophe with mailboxes on ReiserFS

[Hans Reiser]

Oleg Drokin wrote:

>Hello!
>
>On Tue, Oct 08, 2002 at 11:55:42PM +0300, Robert Tiismus wrote:
>  
>
>>malfunction with the new hardware. I think that when closing the side
>>panel, I misaligned the scsi cable from the connector and disk access
>>malfunctioned. Then, after rebooting the system, everything seemed ok, besides
>>this unfortunate inident with three mailboxes :(
>>    
>>
>
>Behaviour of hardware is unpredicatble on bad wiring and that kind of stuff.
>
>  
>
>>Is it possible, that VBT happened during following reboot?
>>    
>>
>
>If the wiring was already ok, then this is very unlikely.
>
>Also you probably want to run reiserfsck on that disk to make sure
>no other damage happened.
>
>Bye,
>    Oleg
>
>
>  
>
Robert's problem sounds like the disadvantage of metadata journaling in 
action.  He can avoid this in the future by using write ordering and 
chris's datalogging patches, or v4 when that ships.

Hans