How to set a reiserfs partition to get an occasional fsck?

How to set a reiserfs partition to get an occasional fsck?

[Michael James]

While the new logging filesystems are a great improvement
 my experience is that they can't survive forever in the real world
 without an occasional rebuild or fsck.

The suse and other lists have warnings by people burnt by reiserfs.
I haven't (yet) lost any data but have had some scary times.

This hasn't been bugs in reiserfs (3.6) itself
 as most instability was tracked to (very marginally) flakey RAM.

However while the glitches were caused by corrupt RAM
 they left me with faults in the filesystem,
 faults that persisted across reboots.

These included un-list-able and un-cat-able files.
ie: read or ask the size of that file
 and it's bye-bye to that terminal.
It made the whole system unuseable
 as processes "trod on the cracks" and hung.
Backups? Hah, not with that file in the partition.

So I think a lot of bad press stems from the misconception
 that any filesystem can avoid bitrot forever without an fsck.
But this is painful to do by hand, I have to boot a rescue system
 and run reiserfsck by hand, to do the root and system partitions.

How can I get back the old behaviour an fsck happening
 during reboot every x reboots or y days?

Or,  how can I trigger an "fsck reboot"?

TIA, michaelj

PS:	I've just realized I can do it by adding an fsck
	  into the linuxrc script of a cooked initrd image.
	That would give me an "fsck boot" option in grub.
	Comments?

-- 
Michael James                         michael.james@csiro.au
System Administrator                    voice:  02 6246 5040
CSIRO Bioinformatics Facility             fax:  02 6246 5166

Re: How to set a reiserfs partition to get an occasional fsck?

[Bennett Todd]

[-- Attachment #1: Type: text/plain, Size: 1412 bytes --]

2004-01-29T23:15:10 Michael James:
> PS:	I've just realized I can do it by adding an fsck
> 	  into the linuxrc script of a cooked initrd image.

I'm sure there are many approaches that can be used, but it so
happens that I can specifically comment on that one, having recently
gotten started playing with initrds.

They're so easy, so fun!

If I wanted to make a "fsck boot", here's how _I_ would do it.

	dd if=/dev/zero bs=1024k count=4 of=initrd.fs
	mke2fs initrd.fs
	mount -o loop initrd.fs /mnt
	# copy a full busybox setup onto /mnt
	# copy the statically-linked fsck binary I want to /mnt
	sync
	umount /mnt
	gzip -9 <initrd.fs >initrd.img

That's a rescue disk. If you want it to be completely automated, add
this etc/init.d/rcS, mode 0755:

	#!/bin/sh
	fsck command I want
	reboot

I've got a statically linked busybox (along with a lot of other
components, all statically linked against uClibc) available from
<URL:http://bent.latency.net/bent/>. For rescue disk creation you'll
want to unpack elsewhere and delete docs, and var/lib/bpm (where the
complete package sources are installed).

To boot such an initrd, besides specifying your kernel and initrd,
you want to add "rw" and "root=/dev/ram0" to your boot options.

But I don't think I'd actually respond this way to the problem
you've got. RAM is so cheap, run memtest86 to find out for sure
which bits are marginal, and replace 'em.

-Bennett

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

RE: How to set a reiserfs partition to get an occasional fsck?

[Burnes, James]

Michael,

If this is a production machine (and it sounds like it), it would be
much easier to test your RAM before putting it into production.  In
either case you shouldn't need to boot from a floppy.  I'd just go to
single user mode and do it there or boot from a live Cd (Knoppix or
something) or (eek!) install a very small ext2 root.

I also was bitten by RAM-induced instability (hey, a new acronym - RII)
when using Reiser.  It makes you wonder how many of the SIMM modules
and/or motherboards are marginal or are using marginal contact
materials.

Case in point: 

I had installed Slackware 9.1 on one of my lab servers that had been
running OpenBSD.  Now OpenBSD is an excellent secure OS, but nobody
would delude themselves that the kernel is much of a stress of cpu and
memory -- especially running as only a firewall.

I wiped OpenBSD and installed Slack with reiser everywhere.  Everything
was fine until did a kernel recompile.  The machine cratered so hard
that rebuilding the filesystem was almost pointless.  It was a real
mess.

Of course I bitched and moaned and cursed the day that Hans Reiser was
born.
Damn !@#$@#$@.  Can't write a file system to save his life! No wonder
that @#$@#$ wasn't #@$#@$@#.  ;-) 

Fortunately, I didn't lose much.

I took a deep breath, composed myself and realized I really liked
Reiserfs and wondered if it wasn't something wrong with my machine.
First I suspected that something was wrong with the BIOS because I had
been having strange interrupt problems on my ethernet card. Upgrading
the BIOS firmware fixed the ethernet card, but kernel compiles were
still crashing.

After days of spare time work on this machine I finally opened it up and
looked around.  Nothing obvious, so I threw up my hands and just
re-seated the SIMM modules.

One more time re-installing Slackware and I did a kernel compile.  No
problems.  Completely stable.  I wrote a script that did nothing but run
kernel compiles for 5 hours straight.  Still stable.  Everything fine.
So strange.  The SIMM modules seemed to be tightly seated.  Maybe
corrosion?

During my diagnostic work I had tried a Knoppix live rescue CD.  I was
so impressed with Knoppix (and Debian) that I switched.  (someone should
make a switch campaign for Debian like the Apple switch campaign ;-).  I
must say that the convenience of apt-get has made it possible to
experiment with so many different software configurations in my lab that
I'm about 100% more productive.

So in actuality what sometimes seems like a pain-in-the-ass can actually
be a blessing in disguise.

Thank you, Han Reiser for introducing me to Knoppix/Debian (and a fine
filesystem ;-)  Goodbye Slackware.  You were a fine friend for 10 years.
Some day I'll come back and visit. ;-)

Hans and crew: I'm still not sure what kind of memory operations crash
out the marginal SIMMs.  Have you isolated them?  Could you reproduce
it?  It would be interesting if you could write a small utility that
stresses RAM in the same way that Reiser stresses it.  That way there
would be no suprises and it could be run as a pre-install package before
Reiser is installed.  Something like:

YOU ARE ABOUT TO INSTALL REISERFS!  THIS IS A RELIABLE AND HIGH
PERFORMANCE ACCELERATED FILE SYSTEM.  IT DEPENDS ON HIGH QUALITY RAM
CORRECTLY SEATED ON YOUR MOTHERBOARD.  WOULD YOU LIKE TO TEST YOUR RAM
MODULES BEFORE YOU INSTALL IT? (Y/N) Y<cr>

TESTING: PATTERN TEST 1....PATTERN TEST 2.....DMA TEST.....etc

Your RAM passed the stress testing -- proceed.

Or 

ERROR: Detected periodic faults in CPU mediated block transfers.
       Detected periodic faults in microcoded block transfers.
       Detected periodic faults in memory to I/O DMA transfers.

** Your system memory is faulty.  Suggest you re-seat the modules, clean
module connectors or replace your memory with higher quality modules. **

This way there would be no suprises.  Reiser (and especially Reiser4)
uses deep magick.  That comes at a price. I remember a test of various
journaling file systems a while back.  They measured a lot of things,
but seemed to really focus on speed vs CPU consumed.

Reiser3 and 4 consumed more CPU than the average file system so it was
rated low in the standings.  But what was really interesting was what
they didn't mention.  Over in the corner of the chart Reiser had
completed nearly all the tests before any other file system. ;-)

Reiser, the accelerated file system.  It comes at a price.  I wouldn't
use it on a PDA ;-)

Have a good day,

j.burnes


> -----Original Message-----
> From: Michael James [mailto:Michael.James@csiro.au]
> Sent: Thursday, January 29, 2004 10:15 PM
> To: reiserfs-list@namesys.com
> Subject: How to set a reiserfs partition to get an occasional fsck?
> 
> While the new logging filesystems are a great improvement
>  my experience is that they can't survive forever in the real world
>  without an occasional rebuild or fsck.
> 
> The suse and other lists have warnings by people burnt by reiserfs.
> I haven't (yet) lost any data but have had some scary times.
> 
> This hasn't been bugs in reiserfs (3.6) itself
>  as most instability was tracked to (very marginally) flakey RAM.
> 
> However while the glitches were caused by corrupt RAM
>  they left me with faults in the filesystem,
>  faults that persisted across reboots.
> 
> These included un-list-able and un-cat-able files.
> ie: read or ask the size of that file
>  and it's bye-bye to that terminal.
> It made the whole system unuseable
>  as processes "trod on the cracks" and hung.
> Backups? Hah, not with that file in the partition.
> 
> So I think a lot of bad press stems from the misconception
>  that any filesystem can avoid bitrot forever without an fsck.
> But this is painful to do by hand, I have to boot a rescue system
>  and run reiserfsck by hand, to do the root and system partitions.
> 
> How can I get back the old behaviour an fsck happening
>  during reboot every x reboots or y days?
> 
> Or,  how can I trigger an "fsck reboot"?
> 
> TIA, michaelj
> 
> PS:	I've just realized I can do it by adding an fsck
> 	  into the linuxrc script of a cooked initrd image.
> 	That would give me an "fsck boot" option in grub.
> 	Comments?
> 
> --
> Michael James                         michael.james@csiro.au
> System Administrator                    voice:  02 6246 5040
> CSIRO Bioinformatics Facility             fax:  02 6246 5166

RE: How to set a reiserfs partition to get an occasional fsck?

[Stewart Smith]

[-- Attachment #1: Type: text/plain, Size: 817 bytes --]

On Sat, 2004-01-31 at 03:31, Burnes, James wrote:
> If this is a production machine (and it sounds like it), it would be
> much easier to test your RAM before putting it into production.  In

or buy ERROR CORRECTING RAM!!!! The trend in more recent years towards a
lack of error correction is just biting people *real* hard.

The corruption you see could purely be cosmic rays going through your
ram banks and flipping bits - people used to see this happening over 20
years ago, and ram densities have kind of increased since then.

The probability of this happening with 2GB ram these days is actually
quite high. Just hope it isn't a bit you care about.

but yeah, anywhere important - use ECC ram, or you may end up with such
strangeness - well, at least you'll be able to detect such strangeness!

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

RE: How to set a reiserfs partition to get an occasional fsck?

[Lamont R. Peterson]

[-- Attachment #1: Type: text/plain, Size: 4520 bytes --]

On Fri, 2004-01-30 at 09:31, Burnes, James wrote:
[SNIP]
> After days of spare time work on this machine I finally opened it up and
> looked around.  Nothing obvious, so I threw up my hands and just
> re-seated the SIMM modules.

SIMMs??  Really?  I have a couple of old machines at home that only have
SIMM slots, but otherwise, I have not seen SIMMs in a long long time. 
You meant DIMMs, right?

> One more time re-installing Slackware and I did a kernel compile.  No
> problems.  Completely stable.  I wrote a script that did nothing but run
> kernel compiles for 5 hours straight.  Still stable.  Everything fine.
> So strange.  The SIMM modules seemed to be tightly seated.  Maybe
> corrosion?

Memory problems could rightfully be blamed for many problems that are
encountered.

[SNIP]
> Hans and crew: I'm still not sure what kind of memory operations crash
> out the marginal SIMMs.  Have you isolated them?  Could you reproduce
> it?  It would be interesting if you could write a small utility that
> stresses RAM in the same way that Reiser stresses it.  That way there
> would be no suprises and it could be run as a pre-install package before
> Reiser is installed.  Something like:

Reading your story above, I would not jump to the conclusion that the
load using ReiserFS places on the CPU would manifest consistent
crashes.  I think you hit on the real problem, your RAM.  Remember,
kernel compilation causes the CPU to work hard and requires lots of
RAM.  I do not believe that ReiserFS being involved contributed
*ANYTHING* to cause the crashes that you experienced.

Frankly, I am frustrated when people imply that ReiserFS is faulty
because they were using that filesystem when they experienced a
problem.  This happens far too much out there.  ReiserFS is not perfect,
nor is it the destabilizing component that many people try to make it
out to be.

Please, do not take this personally.  I do not believe it was your
intention to spread such innuendo.

> YOU ARE ABOUT TO INSTALL REISERFS!  THIS IS A RELIABLE AND HIGH
> PERFORMANCE ACCELERATED FILE SYSTEM.  IT DEPENDS ON HIGH QUALITY RAM
> CORRECTLY SEATED ON YOUR MOTHERBOARD.  WOULD YOU LIKE TO TEST YOUR RAM
> MODULES BEFORE YOU INSTALL IT? (Y/N) Y<cr>
> 
> TESTING: PATTERN TEST 1....PATTERN TEST 2.....DMA TEST.....etc
> 
> Your RAM passed the stress testing -- proceed.
> 
> Or 
> 
> ERROR: Detected periodic faults in CPU mediated block transfers.
>        Detected periodic faults in microcoded block transfers.
>        Detected periodic faults in memory to I/O DMA transfers.
> 
> ** Your system memory is faulty.  Suggest you re-seat the modules, clean
> module connectors or replace your memory with higher quality modules. **

This would definitely be a mistake.  Even if there was any truth to it
as far as ReiserFS is concerned (which I think there is not), there is
no good reason to add something into the process that would scare off
those who are not "in-the-know" about memory problems.

> This way there would be no suprises.  Reiser (and especially Reiser4)
> uses deep magick.  That comes at a price. I remember a test of various
> journaling file systems a while back.  They measured a lot of things,
> but seemed to really focus on speed vs CPU consumed.
> 
> Reiser3 and 4 consumed more CPU than the average file system so it was
> rated low in the standings.  But what was really interesting was what
> they didn't mention.  Over in the corner of the chart Reiser had
> completed nearly all the tests before any other file system. ;-)

Very good point.  There seems to be a tendency to impugn ReiserFS
unduly.  For me, I would be perfectly happy to trade such a small
percentage of compute cycles (which no one uses all the time, anyway)
for such a large improvement in overall system performance. 

> Reiser, the accelerated file system.  It comes at a price.  I wouldn't
> use it on a PDA ;-)

It is not that much CPU time, but on a PDA the filesystem performance
will be affected more by RAM speed (there is no hard drive) than CPU
overhead (once again, no one uses all those cycles all the time). 
However, I would probably not use ReiserFS on a PDA because of the
amount of the already limited storage space that would be used up by
it's fixed overhead.  Just like I would not create a 20MB /boot
partition using ReiserFS.
-- 
Lamont Peterson <lamont@gurulabs.com>
Senior Instructor
Guru Labs <http://www.gurulabs.com/>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]