reiserfs on removable media

reiserfs on removable media

[Zygo Blaxell]

I have a 120GB reiserfs in a portable disk enclosure with USB2.0 and
IEEE1394 interfaces.  Unfortunately the current Linux USB and firewire
drivers in 2.4.21 still have nasty issues, with the result that I've had
too many crashes to count while working out how to get the device drivers
to talk to this disk reliably (probably 50 or more crashes so far).

Obviously these problems aren't reiserfs's fault, nor can reiserfs do
anything about these problems, but it's nice to see that reiserfs survives
as well as it does.

Two reiserfs improvements come to mind:

- There is a tendency for files that were being grown at crash time to
contain invalid data.  It seems that the inodes are being updated before
the data blocks they refer to are written.  It would be nice if the inode
writes were deferred (or at least made invisible) until after the data
blocks were written.  I'd rather lose my data than possibly have random
garbage masquerading as my data.

- If the device is detached while a filesystem is mounted, reiserfs gets a
whole lot of I/O errors (or worse) and immediately oopses.  It would be
nice if reiserfs would handle this a bit more gracefully--it should at
least let me kill processes with open files and umount the filesystem.
OTOH many other things also oops with with current USB/firewire/scsi device
driver stack too.  :-P

Otherwise, this particular reiserfs has survived all of the crashes so
far, even under the heavy I/O loads that seem to trigger the crashes.
Cool.

Re: reiserfs on removable media

[Andreas Dilger]

On Jul 02, 2003  14:23 -0400, Zygo Blaxell wrote:
> Two reiserfs improvements come to mind:
> 
> - There is a tendency for files that were being grown at crash time to
> contain invalid data.  It seems that the inodes are being updated before
> the data blocks they refer to are written.  It would be nice if the inode
> writes were deferred (or at least made invisible) until after the data
> blocks were written.  I'd rather lose my data than possibly have random
> garbage masquerading as my data.

This is called ordered data mode, and exists on ext3 and also reiserfs with
Chris Mason's patches.  Under normal usage it shouldn't change performance
compared to writeback data mode (which is what reiserfs does by default).

> - If the device is detached while a filesystem is mounted, reiserfs gets a
> whole lot of I/O errors (or worse) and immediately oopses.  It would be
> nice if reiserfs would handle this a bit more gracefully--it should at
> least let me kill processes with open files and umount the filesystem.
> OTOH many other things also oops with with current USB/firewire/scsi device
> driver stack too.  :-P

Well, if something oopses you are pretty much stuck w.r.t. killing the
process and unmounting the fs.  So fix the oopses and the rest should
come around as a result.  Of course, the reiserfs folks can do a lot
more with a specific oops report than just "it immediately oopses".  ;-)

Not much you can do about the IO errors (i.e. working as designed).
That's going to happen if you remove your device while writing to it.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: reiserfs on removable media

[Hans Reiser]

Andreas Dilger wrote:

>On Jul 02, 2003  14:23 -0400, Zygo Blaxell wrote:
>  
>
>>Two reiserfs improvements come to mind:
>>
>>- There is a tendency for files that were being grown at crash time to
>>contain invalid data.  It seems that the inodes are being updated before
>>the data blocks they refer to are written.  It would be nice if the inode
>>writes were deferred (or at least made invisible) until after the data
>>blocks were written.  I'd rather lose my data than possibly have random
>>garbage masquerading as my data.
>>    
>>
>
>This is called ordered data mode, and exists on ext3 and also reiserfs with
>Chris Mason's patches.  Under normal usage it shouldn't change performance
>compared to writeback data mode (which is what reiserfs does by default).
>
It had some impact, I forget exactly how much, maybe Chris can 
resuscitate his benchmark of it?

Remind me about removable media around January, and we'll write some 
code for reiser4 to make it more graceful for it (somehow prompt the 
user to insert disk, etc.)

-- 
Hans

Re: reiserfs on removable media

[Chris Mason]

On Wed, 2003-07-02 at 14:53, Hans Reiser wrote:

> >This is called ordered data mode, and exists on ext3 and also reiserfs with
> >Chris Mason's patches.  Under normal usage it shouldn't change performance
> >compared to writeback data mode (which is what reiserfs does by default).
> >
> It had some impact, I forget exactly how much, maybe Chris can 
> resuscitate his benchmark of it?
> 

The major cost of data=ordered is that dirty blocks are flushed every 5
seconds instead of every 30.  The journal header patch in my
experimental data logging directory changes things so that only new
bytes in the file are done in data=ordered mode (either adding a new
block or appending onto the end of the file).

This helps a lot in the file rewrite tests.

-chris

Re: reiserfs on removable media

[Dieter Nützel]

Am Mittwoch, 2. Juli 2003 20:59 schrieb Chris Mason:
> On Wed, 2003-07-02 at 14:53, Hans Reiser wrote:
> > >This is called ordered data mode, and exists on ext3 and also reiserfs
> > > with Chris Mason's patches.  Under normal usage it shouldn't change
> > > performance compared to writeback data mode (which is what reiserfs
> > > does by default).

Chris,

I thought data=ordered is the "new" default with your patch?

> > It had some impact, I forget exactly how much, maybe Chris can
> > resuscitate his benchmark of it?
>
> The major cost of data=ordered is that dirty blocks are flushed every 5
> seconds instead of every 30.  The journal header patch in my
> experimental data logging directory changes things so that only new
> bytes in the file are done in data=ordered mode (either adding a new
> block or appending onto the end of the file).
>
> This helps a lot in the file rewrite tests.

What's faster than with your patches? ordered|journal|writeback?

I thought is order: writeback < ordered < journal ;-)

Thanks,
	Dieter

Re: reiserfs on removable media

[Chris Mason]

On Wed, 2003-07-02 at 15:08, Dieter Nützel wrote:
> Am Mittwoch, 2. Juli 2003 20:59 schrieb Chris Mason:
> > On Wed, 2003-07-02 at 14:53, Hans Reiser wrote:
> > > >This is called ordered data mode, and exists on ext3 and also reiserfs
> > > > with Chris Mason's patches.  Under normal usage it shouldn't change
> > > > performance compared to writeback data mode (which is what reiserfs
> > > > does by default).
> 
> Chris,
> 
> I thought data=ordered is the "new" default with your patch?
> 
It is.

> > > It had some impact, I forget exactly how much, maybe Chris can
> > > resuscitate his benchmark of it?
> >
> > The major cost of data=ordered is that dirty blocks are flushed every 5
> > seconds instead of every 30.  The journal header patch in my
> > experimental data logging directory changes things so that only new
> > bytes in the file are done in data=ordered mode (either adding a new
> > block or appending onto the end of the file).
> >
> > This helps a lot in the file rewrite tests.
> 
> What's faster than with your patches? ordered|journal|writeback?
> 
> I thought is order: writeback < ordered < journal ;-)

Usually ;-)  ordered is faster in a few rare benchmarks because it helps
keeps the number of dirty buffers lower and generally sends the dirty
buffers to the disk in a big flood.

journal is faster for some fsync heavy benchmarks.

For practical desktop usage, data=ordered and writeback are usually
close to each other.

-chris

Re: reiserfs on removable media

[Zygo Blaxell]

On Wed, 02 Jul 2003 14:53:39 -0400, Hans Reiser wrote:
> Remind me about removable media around January, and we'll write some
> code for reiser4 to make it more graceful for it (somehow prompt the
> user to insert disk, etc.)

Ow!  Ow!  Ow!  Kernel prompting the user...  Ow!  ;-)

Now, "kernel notifying an automounter daemon process, which talks to the
user in user-space" is somewhere in the realm of possibility...

Actually to be clear there are two topics here:  removable _media_ and
removable _drives_.  e.g. a typical IDE disk is non-removable media in a
non-removable drive.

A floppy disk in a typical floppy drive is removable media in a
non-removable drive, but a floppy disk in a USB floppy drive is removable
media in a removable drive.  Practically speaking there's not much
difference--if the drive was removed, you'd have to assume the media was
removed as well, if only because there's no way to receive media-change
notifications if the drive isn't connected.

The USB drive I wrote about earlier is a desktop non-removable IDE disk in
a removable drive.

The difference is subtle but it does allow for some interesting stuff to
happen in the block device layer.  The hard drive has a serial number,
which (in theory) could be queried by the USB storage layer and used as a
unique identifier for the drive.  This could e.g. suspend all read/write
requests to the drive while it is disconnected, and resume said requests
when it is reconnected.  That's all I'd need for my laptop setup
(assuming I don't connect the drive somewhere else, that is)...and it
doesn't require changing one line of reiserfs.

Recipe for reiserfs oops on "removable" disks (was: Re: reiserfs on removable media)

[Zygo Blaxell]

On Wed, 02 Jul 2003 14:45:44 -0400, Andreas Dilger wrote:
> On Jul 02, 2003  14:23 -0400, Zygo Blaxell wrote:

> This is called ordered data mode, and exists on ext3 and also reiserfs
> with Chris Mason's patches.

Ah, thank you, I had forgotten that the feature had a name, and that ext3
can be configured to have the same behavior.  ;-)

> Well, if something oopses you are pretty much stuck w.r.t. killing the
> process and unmounting the fs.  So fix the oopses and the rest should
> come around as a result.  Of course, the reiserfs folks can do a lot
> more with a specific oops report than just "it immediately oopses".  ;-)

But it _does_ immediately oops.  Actually that's not true, it BUG()s
first.  And depending on the device driver chain it may also oops in
other places.

I guess I forgot that "the oops I've seen about 20 times this morning"
isn't really useful information to other people.  ;-)

You can get a similar oops without any special hardware using the network
block device.  Here's the recipe:

Create a large-file on machine A.  Run 'nbd-server 11111 large-file' on machine
A.  The file has to be big enough that mkreiserfs can create a filesystem
on it.

Run the following on machine B:

	nbd-client A 11111 /dev/nbd/0
	mkreiserfs /dev/nbd/0
	mount /dev/nbd/0 /test
	ls -l / > /test/some-data

Then on machine A:

	killall nbd-server

Then do something on machine B with the /test filesystem, and watch the
fireworks.  It looks something like this:

NBD: receive - sock=-1040559660 at buf=-1047896328, size=16 returned 0
.                                                                               
NBD: Recv control failed.(result 0)                                             
req should never be null                                                        
nbd: shutting down socket                                                       
nbd: queue cleared                                                              
Kernel call returned.Closing: que, sock, done                                   
NBD, minor 0: Request when not-ready.                                           
NBD, minor 0: Request when not-ready.                                           
NBD, minor 0: Request when not-ready.                                           
NBD, minor 0: Request when not-ready.                                           
NBD, minor 0: Request when not-ready.                                           
NBD, minor 0: Request when not-ready.                                           
NBD, minor 0: Request when not-ready.                                           
NBD, minor 0: Request when not-ready.                                           
NBD, minor 0: Request when not-ready.                                           
journal-601, buffer write failed              
[the rest is filtered through ksymoops]
kernel BUG at prints.c:334!                                                     
invalid operand: 0000                                                           
CPU:    0                                                                       
EIP:    0010:[<c01a4fa9>]    Not tainted                                        
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00000282                                                                
eax: 00000024   ebx: c02f2460   ecx: c034cd2c   edx: 00000000                   
esi: c133e000   edi: c133e000   ebp: 00000004   esp: c1fedec8                   
ds: 0018   es: 0018   ss: 0018                                                  
Process kupdated (pid: 6, stackpage=c1fed000)                                   
Stack: c02f035a c0403740 c02f2460 c1fedeec c28540cc 00000000 c01af619 c133e000  
       c02f2460 00000000 00000000 00000010 00000000 c2854100 c28540f4 00000005  
       00000000 c1307814 c01b313e c133e000 c28540cc 00000001 c1fedf80 c133e000  
Call Trace:    [<c01af619>] [<c01b313e>] [<c01b230f>] [<c01a24be>] [<c014b508>] 
  [<c014a267>] [<c014a736>] [<c0107420>]                                        
Code: 0f 0b 4e 01 60 03 2f c0 68 40 37 40 c0 85 f6 74 16 31 c0 66               


>>EIP; c01a4fa9 <reiserfs_panic+29/60>   <=====

>>ebx; c02f2460 <MAX_KEY+1000/4b24>
>>ecx; c034cd2c <log_wait+0/10>
>>esi; c133e000 <_end+ef7548/3047d548>
>>edi; c133e000 <_end+ef7548/3047d548>
>>esp; c1fedec8 <_end+1ba7410/3047d548>

Trace; c01af619 <flush_commit_list+2e5/3e4>
Trace; c01b313e <do_journal_end+7ea/ad0>
Trace; c01b230f <flush_old_commits+137/150>
Trace; c01a24be <reiserfs_write_super+5e/bc>
Trace; c014b508 <sync_supers+14c/1c8>
Trace; c014a267 <sync_old_buffers+5b/140>
Trace; c014a736 <kupdate+1ea/20c>
Trace; c0107420 <arch_kernel_thread+28/38>

Code;  c01a4fa9 <reiserfs_panic+29/60>
00000000 <_EIP>:
Code;  c01a4fa9 <reiserfs_panic+29/60>   <=====
   0:   0f 0b                     ud2a      <=====
Code;  c01a4fab <reiserfs_panic+2b/60>
   2:   4e                        dec    %esi
Code;  c01a4fac <reiserfs_panic+2c/60>
   3:   01 60 03                  add    %esp,0x3(%eax)
Code;  c01a4faf <reiserfs_panic+2f/60>
   6:   2f                        das    
Code;  c01a4fb0 <reiserfs_panic+30/60>
   7:   c0 68 40 37               shrb   $0x37,0x40(%eax)
Code;  c01a4fb4 <reiserfs_panic+34/60>
   b:   40                        inc    %eax
Code;  c01a4fb5 <reiserfs_panic+35/60>
   c:   c0 85 f6 74 16 31 c0      rolb   $0xc0,0x311674f6(%ebp)
Code;  c01a4fbc <reiserfs_panic+3c/60>
  13:   66                        data16


> Not much you can do about the IO errors (i.e. working as designed).
> That's going to happen if you remove your device while writing to it.

I don't mind I/O errors as long as the journalling semantics are preserved
and enough of the kernel is intact to umount and mount again later.

Re: reiserfs on removable media

[Oleg Drokin]

Hello!

On Wed, Jul 02, 2003 at 02:23:13PM -0400, Zygo Blaxell wrote:

> - If the device is detached while a filesystem is mounted, reiserfs gets a
> whole lot of I/O errors (or worse) and immediately oopses.  It would be
> nice if reiserfs would handle this a bit more gracefully--it should at
> least let me kill processes with open files and umount the filesystem.
> OTOH many other things also oops with with current USB/firewire/scsi device
> driver stack too.  :-P

Write errors to data areas are not mostly "safe".
It's write errors into journal area that kill the thing.
Jeff Mahoney of SuSE have the patch that remounts the FS R/O in
case of such an event (I think he even posted some preliminary patches
here), it is what you most probably need in this case.

Bye,
    Oleg