NFS setup for a reiserfs-based /home

NFS setup for a reiserfs-based /home

[Javier Marcet]

[-- Attachment #1: Type: text/plain, Size: 1772 bytes --]

Hi reiser users and developers :),
I am in the process of deploying linux in a whole computer lab on my
university. I've already setup a mail and ssh server which is working
pretty well. All the partitions on the server machine are reiserfs,
since I was free to install whatever I wanted and reiser is the fs I've
had been more satisfies with, especially since 2.4.20-pre6 - it is
working way better than it did on 2.4.18 -.

So far so good. The truth is I don't have that much experience with NFS,
and I want to have /home on this server machines for all the student
accounts. Initially it is only going to be used by the CS students and
those who want to try linux. Later on, when it's been proved it's
working reliably, I've been promised we would be able to erase the
Exchange server completely (I just can't see the time to get rid of that
***) and use Linux for all mail purposes.

As I said, I'm a newbie on NFS, and I've been reading quite a lot about
the interactions of it with journalling file systems. Namely, last thing
I remeber is I should put the journal on a different volume, if at all
possible, when using reiser - I wonder if it is enough a different
partition, or it'll really benefit from splitting it between two
different physical HDs -.

That's my question then. What do you suggest me to get the most out of
reiserfs (with which I'm really happy) for a server storing /home.
Even if at the beginning it'll be small (only a few students), it'll
most probably will be deployed campus-wide later on, hence I'd much
prefer to set it up in a scalable way - storage space and speed-wise -
from day one.

Thanks in advance for any hint :)


-- 
Javier Marcet <jmarcet@pobox.com>
Saint Louis University, Madrid Campus

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: NFS setup for a reiserfs-based /home

[Oleg Drokin]

Hello!

On Tue, Sep 24, 2002 at 11:57:56PM +0200, Javier Marcet wrote:
 
> As I said, I'm a newbie on NFS, and I've been reading quite a lot about
> the interactions of it with journalling file systems. Namely, last thing
> I remeber is I should put the journal on a different volume, if at all

No, this is not required for NFS (And never was).
There is nothing special about running NFS off reiserfs (v3.6 on-disk format)
filesystems. (in case of kernel nfsd, there are some difficulties with
userspace nfsd, so it should be avoided).

> That's my question then. What do you suggest me to get the most out of
> reiserfs (with which I'm really happy) for a server storing /home.
> Even if at the beginning it'll be small (only a few students), it'll
> most probably will be deployed campus-wide later on, hence I'd much
> prefer to set it up in a scalable way - storage space and speed-wise -
> from day one.

You may want to put your /home volume on some kind of RAID1 device to
increase data safeness from hardware errors and to increase access (read) speed.
(but this is storage space ineffective of course).

Bye,
    Oleg

Re: NFS setup for a reiserfs-based /home

[Dieter Nützel]

Am Mittwoch, 25. September 2002 09:40 schrieb Oleg Drokin:
> Hello!
>
> On Tue, Sep 24, 2002 at 11:57:56PM +0200, Javier Marcet wrote:
> > As I said, I'm a newbie on NFS, and I've been reading quite a lot about
> > the interactions of it with journalling file systems. Namely, last thing
> > I remeber is I should put the journal on a different volume, if at all
>
> No, this is not required for NFS (And never was).
> There is nothing special about running NFS off reiserfs (v3.6 on-disk
> format) filesystems. (in case of kernel nfsd, there are some difficulties
> with userspace nfsd, so it should be avoided).

Excuse me but this is not clear for me.
Oleg, you told him to avoid knfsd?
Then I have to say fix what ever is needed.
Knfsd is so much superior.

> > That's my question then. What do you suggest me to get the most out of
> > reiserfs (with which I'm really happy) for a server storing /home.
> > Even if at the beginning it'll be small (only a few students), it'll
> > most probably will be deployed campus-wide later on, hence I'd much
> > prefer to set it up in a scalable way - storage space and speed-wise -
> > from day one.
>
> You may want to put your /home volume on some kind of RAID1 device to
> increase data safeness from hardware errors and to increase access (read)
> speed. (but this is storage space ineffective of course).

I'll suggest RAID5 (4 or more disks plus 1 or more spare) for all your 
storage.

Regards,
	Dieter

-- 
Dieter Nützel
Graduate Student, Computer Science

University of Hamburg
Department of Computer Science
@home: Dieter.Nuetzel at hamburg.de (replace at with @)

Re: NFS setup for a reiserfs-based /home

[Oleg Drokin]

Hello!

On Wed, Sep 25, 2002 at 03:33:19PM +0200, Dieter N?tzel wrote:
> > > As I said, I'm a newbie on NFS, and I've been reading quite a lot about
> > > the interactions of it with journalling file systems. Namely, last thing
> > > I remeber is I should put the journal on a different volume, if at all
> > No, this is not required for NFS (And never was).
> > There is nothing special about running NFS off reiserfs (v3.6 on-disk
> > format) filesystems. (in case of kernel nfsd, there are some difficulties
> > with userspace nfsd, so it should be avoided).
> Excuse me but this is not clear for me.
> Oleg, you told him to avoid knfsd?

No I said to avoid unfsd (I already made that clear in separate private email)

> > You may want to put your /home volume on some kind of RAID1 device to
> > increase data safeness from hardware errors and to increase access (read)
> > speed. (but this is storage space ineffective of course).
> I'll suggest RAID5 (4 or more disks plus 1 or more spare) for all your 
> storage.

RAID5 is CPU hungry on writes, also writes are slower compared to RAID1.

Bye,
    Oleg

Re: NFS setup for a reiserfs-based /home

[Matthias Urlichs]

Hi,

Oleg Drokin:
> > I'll suggest RAID5 (4 or more disks plus 1 or more spare) for all your 
> > storage.
> 
> RAID5 is CPU hungry on writes, also writes are slower compared to RAID1.
> 
... on the other hand, you need fewer disks than with RAID1.  ;-)

Anyway, reasonably fast hardware RAID controllers do exist, so this isn't
necessarily a problem.

-- 
Matthias Urlichs     |     noris network AG     |     http://smurf.noris.de/

Re: NFS setup for a reiserfs-based /home

[Oleg Drokin]

Hello!

On Wed, Sep 25, 2002 at 03:46:27PM +0200, Matthias Urlichs wrote:

> > > I'll suggest RAID5 (4 or more disks plus 1 or more spare) for all your 
> > > storage.
> > RAID5 is CPU hungry on writes, also writes are slower compared to RAID1.
> ... on the other hand, you need fewer disks than with RAID1.  ;-)

On the third hand, in RAID1 setup with N disks I won't mind failure of
N-1 disks. ;)

> Anyway, reasonably fast hardware RAID controllers do exist, so this isn't
> necessarily a problem.

This is just matter of truest, if you trust the binary closed source firmware
from such a device, that it will handle failures and this kind of stuff
correctly, go for it.
With Linux' softraid paranoid people can verify their own worst assumptions by
just reading source code ;)

Bye,
    Oleg

Re: NFS setup for a reiserfs-based /home

[Matthias Urlichs]

Hi,

Oleg Drokin:
> This is just matter of truest, if you trust the binary closed source firmware
> from such a device, that it will handle failures and this kind of stuff
> correctly, go for it.
> With Linux' softraid paranoid people can verify their own worst assumptions by
> just reading source code ;)
> 
Well, Linux' softraid doesn't handle some things at all well, either...
as I noticed the hard way. A bug report to the softraid people will be
written, and it will NOT be pretty.

-- 
Matthias Urlichs     |     noris network AG     |     http://smurf.noris.de/

Re: NFS setup for a reiserfs-based /home

[Oleg Drokin]

Hello!

On Wed, Sep 25, 2002 at 04:09:57PM +0200, Matthias Urlichs wrote:
> > This is just matter of truest, if you trust the binary closed source firmware
> > from such a device, that it will handle failures and this kind of stuff
> > correctly, go for it.
> > With Linux' softraid paranoid people can verify their own worst assumptions by
> > just reading source code ;)
> Well, Linux' softraid doesn't handle some things at all well, either...

Well, same goes for SCSI layer and so on, but if you really-really need that
to be fixed, you can fix it ;)

Bye,
    Oleg

Re: NFS setup for a reiserfs-based /home

[bscott]

On Wed, 25 Sep 2002, at 5:37pm, green@namesys.com wrote:
> RAID5 is CPU hungry on writes, also writes are slower compared to RAID1.

  With RAID controllers that include dedicated hardware for the XOR
computations available for less than $300 US, the performance difference
usually does not matter.  About the only time performance matters is for
things like transaction journals on large databases, which are pretty much
write-only and very often a bottleneck.

On Wed, 25 Sep 2002, at 5:51pm, green@namesys.com wrote:
> On the third hand, in RAID1 setup with N disks I won't mind failure of N-1
> disks. ;)

  You can have hot spares or N+2 redundancy for other RAID levels as well.

>> Anyway, reasonably fast hardware RAID controllers do exist, so this
>> isn't necessarily a problem.
> 
> This is just matter of truest, if you trust the binary closed source
> firmware from such a device, that it will handle failures and this kind of
> stuff correctly, go for it.

  On the other hand, software RAID is pretty weak in comparison to hardware
RAID.  You spend host resources (CPU, memory, bus bandwidth) doing RAID
stuff that is better done on a separate controller.  Some kinds of device
failures tend to bring down the entire system.  And you have to worry about
boot-time, before the kernel RAID is active.

  With a good hardware RAID controller, the OS just sees a big disk.  INT13
support is available, so GRUB and LILO don't care, either.  You free up CPU,
RAM, and bus bandwidth.  You can isolate device and bus failures from the
host.  You can have dedicated, battery-backed, write-back cache RAM on the
controller, which greatly improves performance.

  If you can afford it, hardware is *definitely* the way to go.

  Of course, a lot of Linux users are also looking to do things on the
cheap.  You cannot beat software RAID and a bunch of IDE disks for cheap
storage.  :-)

> With Linux' softraid paranoid people can verify their own worst
> assumptions by just reading source code ;)

  Verify, yes.  Fix?  Well, I looked at the 2.2 SCSI layer once.  I still
have nightmares about it.  :-)

-- 
Ben Scott <bscott@ntisys.com>
| The opinions expressed in this message are those of the author and do not |
| necessarily represent the views or policy of any other person, entity or  |
| organization.  All information is provided without warranty of any kind.  |

Re: NFS setup for a reiserfs-based /home

[Hans Reiser]

Matthias Urlichs wrote:

>Hi,
>
>Oleg Drokin:
>  
>
>>This is just matter of truest, if you trust the binary closed source firmware
>>from such a device, that it will handle failures and this kind of stuff
>>correctly, go for it.
>>With Linux' softraid paranoid people can verify their own worst assumptions by
>>just reading source code ;)
>>
>>    
>>
>Well, Linux' softraid doesn't handle some things at all well, either...
>as I noticed the hard way. A bug report to the softraid people will be
>written, and it will NOT be pretty.
>
>  
>
Yes, I believe this.  RAID is risky whether closed or open, there are a 
lot of not so well done implementations.  With hardware RAID the thing 
to worry about is, does it eventually write its caches out to disk. 
 Yes, there are some that do not, and then the battery fails and 6 month 
old data goes poof, resulting in a support call in which we tell a 
sysadmin the sad story of how fsck does not work if there is nothing on 
disk, and no it is not reiserfs it is the hardware RAID that made it go 
poof, and then they go back to the large established vendor and sure 
enough do confirm that it is a known bug....

Hans

Re: NFS setup for a reiserfs-based /home

[Hans Reiser]

Somebody wrote:

>> Hans,
>>
>> which HW RAID controllers would you advice?
>>
>> Cheers,
>>
>  
>
Unfortunately I just don't know.  We can't afford them at Namesys.....;-)

I am going to test some of Philippe's RAID equipment for Reiser4 (his 
company sponsors Reiser4 with a very substantial service contract), I 
should probably add the following test.

Try putting putting some data into the FS that is small enough to fit 
into the RAID card NVRAM, letting it go idle for an hour, yanking the 
power cord out, and then putting another RAID card which is the same 
brand card in as a replacement for your card while the machine is off, 
rebooting, and seeing what happens to your data.  If I understand 
properly from a few users, there exist some cards/RAID units for which 
the data will be non-existent on the disks.  I would not use such cards 
myself.  

I think it would be useful if the identity of such cards became known. 
 Notice how I avoid saying any names because I want to see it reproduced 
independently first before I say any names.  These users I know about 
found out by equipment failing, and one should always be careful about 
reproducing the cause for such events before repeating the story to others.

RAID controllers going kaput are not any rarer than any other PC cards 
going kaput, so....  You probably all know people who didn't backup a 
RAID device, and the controller scribbled garbage to disk, and the 
source code repository or other mission critical datastore.....

Hans