Reiserfs with Samba vs NetApp Filer

Reiserfs with Samba vs NetApp Filer

[darren]

Hi all,

I was just wondering if I should spend $$ (my company's of cos) on a
NetApp filer...

OR

I just get one of the Dell 3650 doing nothing in the corner with RAID5
and run Samba as a fileserver?

Anyone got any benchmarks to compare the two?

Is NetApp as good as its advertised?

Any comments on this setup is welcomed

Regards
Darren

Reiserfs and recovering from the format action...

[Fabrizio Morbini]

Hi, if I format a reiserfs partition there are methods for recover it?

Fabrizio.

Re: Reiserfs and recovering from the format action...

[Oleg Drokin]

Hello!

On Thu, Oct 10, 2002 at 09:01:40AM +0200, Fabrizio Morbini wrote:
> Hi, if I format a reiserfs partition there are methods for recover it?

http://namesys.com/support.html

Bye,
    Oleg

Re: Reiserfs with Samba vs NetApp Filer

[Philippe Gramoullé]

Hi,

On Thu, 10 Oct 2002 09:48:34 +0800
"darren" <teodarren@myrealbox.com> wrote:

  |  Hi all,
  |  
  |  I was just wondering if I should spend $$ (my company's of cos) on a
  |  NetApp filer...
  |  
  |  OR
  |  
  |  I just get one of the Dell 3650 doing nothing in the corner with RAID5
  |  and run Samba as a fileserver?
  |  
  |  Anyone got any benchmarks to compare the two?
  |  
  |  Is NetApp as good as its advertised?

We switched from NetApp to DELL ( PowerVault 210S ) mainly because of the $$ :o)

NetApp are really good but really expensive as well.So it depend how much value
you give to your data.

Netapp as well have tons of features that  linux boxes +PV210S can't do :Clustering
volume copy, out out the box snapshots,etc..

  |  
  |  Any comments on this setup is welcomed

Now i think DELL realease their 220S with 14 disks in a shelf so that would have the bang
for your bucks. On the otherside, NetApp comes with a very nice , proprietary , OS that's quite
simple to use.

So best price/performance => DELL
If you have unvaluable data => NetApp (+ backup :o)

  |  
  |  Regards
  |  Darren
  |  


Philippe.

Re: Reiserfs with Samba vs NetApp Filer

[Hans Reiser]

Philippe Gramoullé wrote:

>Hi,
>
>On Thu, 10 Oct 2002 09:48:34 +0800
>"darren" <teodarren@myrealbox.com> wrote:
>
>  |  Hi all,
>  |  
>  |  I was just wondering if I should spend $$ (my company's of cos) on a
>  |  NetApp filer...
>  |  
>  |  OR
>  |  
>  |  I just get one of the Dell 3650 doing nothing in the corner with RAID5
>  |  and run Samba as a fileserver?
>  |  
>  |  Anyone got any benchmarks to compare the two?
>  |  
>  |  Is NetApp as good as its advertised?
>
>We switched from NetApp to DELL ( PowerVault 210S ) mainly because of the $$ :o)
>
>NetApp are really good but really expensive as well.So it depend how much value
>you give to your data.
>
I would say that WAFL is an absolutely superb filesystem for the purpose 
of RAID NFS fileserving.  Reiser4 may give it some serious competition 
after some time has passed and we have time for such things as carefully 
benchmarking NFS, etc., but WAFL is a great filesystem for NFS RAID as 
it is right now.  That said, mp3.com saved $22 million dollars by using 
Linux+reiserfs instead of Netapps and Veritas.  They may be good, but 
reiser3 is cheaper.  A lot cheaper.  Do the math, and you'll buy Linux 
instead I think.;-)

Nobody has done reiser3 vs. WAFL benchmarks.

>
>Netapp as well have tons of features that  linux boxes +PV210S can't do :Clustering
>volume copy, out out the box snapshots,etc..
>
Linux has snapshots if you use lvm, but it is true that  Netapp makes a 
good product.

>
>  |  
>  |  Any comments on this setup is welcomed
>
>Now i think DELL realease their 220S with 14 disks in a shelf so that would have the bang
>for your bucks. On the otherside, NetApp comes with a very nice , proprietary , OS that's quite
>simple to use.
>
>So best price/performance => DELL
>If you have unvaluable data => NetApp (+ backup :o)
>
>  |  
>  |  Regards
>  |  Darren
>  |  
>
>
>Philippe.
>
>
>  
>

Re: Reiserfs with Samba vs NetApp Filer

[Philippe Gramoullé]

Hi,

Yes, that's what i meant by "not *out of the box* snapshots".

We quite never played with LVM as it adds another software layer and i didn't feel
it was mature enough at the time we put everything in production but given the benefits
we might want to give it a try.

BTW, Hans, what you would recommand : LVM or EVMS ?

Thanks,

Philippe


On Thu, 10 Oct 2002 21:31:29 +0400
Hans Reiser <reiser@namesys.com> wrote:

  |  >Netapp as well have tons of features that  linux boxes +PV210S can't do :Clustering
  |   >volume copy, out out the box snapshots,etc..
  |   >
  |   Linux has snapshots if you use lvm, but it is true that  Netapp makes a 
  |   good product.

Re: Reiserfs with Samba vs NetApp Filer

[Hans Reiser]

Philippe Gramoullé wrote:

>Hi,
>
>Yes, that's what i meant by "not *out of the box* snapshots".
>
>We quite never played with LVM as it adds another software layer and i didn't feel
>it was mature enough at the time we put everything in production but given the benefits
>we might want to give it a try.
>
>BTW, Hans, what you would recommand : LVM or EVMS ?
>
>Thanks,
>
>Philippe
>
>
>On Thu, 10 Oct 2002 21:31:29 +0400
>Hans Reiser <reiser@namesys.com> wrote:
>
>  |  >Netapp as well have tons of features that  linux boxes +PV210S can't do :Clustering
>  |   >volume copy, out out the box snapshots,etc..
>  |   >
>  |   Linux has snapshots if you use lvm, but it is true that  Netapp makes a 
>  |   good product.
>
>
>  
>
I lack the expertise to advise on this.  There has been a lot of 
controversy on this.  Keep an eye out for lvm2 to come out.

Hans

Re: Reiserfs with Samba vs NetApp Filer

[Dieter Nützel]

Am Donnerstag, 10. Oktober 2002 20:02 schrieb Hans Reiser:
> Philippe Gramoullé wrote:
> >Hi,
> >
> >Yes, that's what i meant by "not *out of the box* snapshots".
> >
> >We quite never played with LVM as it adds another software layer and i
> > didn't feel it was mature enough at the time we put everything in
> > production but given the benefits we might want to give it a try.
> >
> >BTW, Hans, what you would recommand : LVM or EVMS ?
> >
> >Thanks,
> >
> >Philippe
> >
> >
> >On Thu, 10 Oct 2002 21:31:29 +0400
> >
> >Hans Reiser <reiser@namesys.com> wrote:
> >  |  >Netapp as well have tons of features that  linux boxes +PV210S can't
> >  |  > do :Clustering
> >  |  >
> >  |   >volume copy, out out the box snapshots,etc..
> >  |
> >  |   Linux has snapshots if you use lvm, but it is true that  Netapp
> >  | makes a good product.
>
> I lack the expertise to advise on this.  There has been a lot of
> controversy on this.  Keep an eye out for lvm2 to come out.

Read about it on LKML.
LVM is dropped with 2.5.41+.
LVM2 comes under the "hood" of EMVS.

-Dieter

Re: Reiserfs with Samba vs NetApp Filer

[Dieter Nützel]

Am Donnerstag, 10. Oktober 2002 20:57 schrieb Dieter Nützel:

> Read about it on LKML.
> LVM is dropped with 2.5.41+.
> LVM2 comes under the "hood" of EMVS.

Of course EVMS ;-)

-Dieter

Re: Reiserfs with Samba vs NetApp Filer

[Lars Marowsky-Bree]

On 2002-10-10T19:53:38,
   Philippe Gramoullé <philippe.gramoulle@mmania.com> said:

> Yes, that's what i meant by "not *out of the box* snapshots".

Well, LVM functionality comes out of the box with all modern distributions.

> We quite never played with LVM as it adds another software layer and i
> didn't feel it was mature enough at the time we put everything in production
> but given the benefits we might want to give it a try.

It is very mature in my experience.

> BTW, Hans, what you would recommand : LVM or EVMS ?

With 2.4, go with LVM1. The jury is still out whether the successor will be
EVMS or LVM2 in 2.5/2.6, but I personally have decided to quite like EVMS.


Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
Principal Squirrel
Research and Development, SuSE Linux AG
 
``Immortality is an adequate definition of high availability for me.''
	--- Gregory F. Pfister

Re: Reiserfs with Samba vs NetApp Filer

[Philippe Gramoullé]

Hi,

On Fri, 11 Oct 2002 12:40:41 +0200
Lars Marowsky-Bree <lmb@suse.de> wrote:

  |  On 2002-10-10T19:53:38,
  |     Philippe Gramoullé <philippe.gramoulle@mmania.com> said:
  |  
  |  > Yes, that's what i meant by "not *out of the box* snapshots".
  |  
  |  Well, LVM functionality comes out of the box with all modern distributions.

Ok let me rephrase it: First i didn't want to start a thread on pros/cons of LVM.

And i only wanted to say that LVM (though part of all modern distributions) doesn't
come installed out of the box,at least not on Debain , which we use mainly.
Whereas snapshots on Netapp comes out of the box with DATAONTAP.

But LVM is there of course if one needs snapshot functionnality.

  |  
  |  > We quite never played with LVM as it adds another software layer and i
  |  > didn't feel it was mature enough at the time we put everything in production
  |  > but given the benefits we might want to give it a try.
  |  
  |  It is very mature in my experience.

Sure it is : We _do_ use it on databases servers , but mostly for volume management
but not for the snapshot features. I wanted to say that given the complexity of our setup
we didn't feel the need to add another software layer : keeping everything in sync is already
not that easy. But sure i'd definitely recommand LVM :o)

  |  
  |  > BTW, Hans, what you would recommand : LVM or EVMS ?
  |  
  |  With 2.4, go with LVM1. The jury is still out whether the successor will be
  |  EVMS or LVM2 in 2.5/2.6, but I personally have decided to quite like EVMS.

Thanks for the input.

  |  
  |  
  |  Sincerely,
  |      Lars Marowsky-Brée <lmb@suse.de>
  |  

Philippe
Lycos Europe
Paris NOC manager.

Re: Reiserfs with Samba vs NetApp Filer

[Adrian Phillips]

>>>>> "Philippe" == Philippe Gramoullé <philippe.gramoulle@mmania.com> writes:

    Philippe> And i only wanted to say that LVM (though part of all
    Philippe> modern distributions) doesn't come installed out of the
    Philippe> box,at least not on Debain , which we use mainly.

Does not one of the numerous 2.4 Debian kernel-images support LVM (I
always build my own) ? Certainly theLVM v1 tools are part of the
distribution. Or do you have another meaning for out of the box ?

    Philippe>   | It is very mature in my experience.

It has been for me. I've been using LVM and reiserfs together using
2.2 for 18 months now with almost no problems (a couple of buglets in
reiserfs which caused little in the way of problems), and more
recently 2.4 (although the 2.4.19 is causing some grief on what seems
to be memory management).

    Lars>   | With 2.4, go with LVM1. The jury is still out
    Lars> whether the successor will be | EVMS or LVM2 in 2.5/2.6,
    Lars> but I personally have decided to quite like EVMS.

Yes, I've been testing EVMS at home (although a bit behind with the
latest) and am impressed stability wise. I haven't looked at LVM2 yet
though.

Sincerely,

Adrian Phillips

-- 
Your mouse has moved.
Windows NT must be restarted for the change to take effect.
Reboot now?  [OK]

Re: Reiserfs with Samba vs NetApp Filer

[Philippe Gramoullé]

On 11 Oct 2002 14:34:40 +0200
Adrian Phillips <a.phillips@met.no> wrote:

  |  Or do you have another meaning for out of the box ?

I was thinking : already setup LVM partitions & kernel support.
(though it may be possible indeed that Debian ships LVM ready kernel)
IIRC, Mandrake offer LVM partitioning at install time.

This said, no need to go on with this thread to know what is out of the box
snapshots or not:

let me rephrase it: "Linux offers out of the box snapshots"

Philippe

Re: Reiserfs with Samba vs NetApp Filer

[Hans Reiser]

Russell Coker wrote:

>On Thu, 10 Oct 2002 19:31, Hans Reiser wrote:
>  
>
>>I would say that WAFL is an absolutely superb filesystem for the purpose
>>of RAID NFS fileserving.  Reiser4 may give it some serious competition
>>after some time has passed and we have time for such things as carefully
>>benchmarking NFS, etc., but WAFL is a great filesystem for NFS RAID as
>>it is right now.  That said, mp3.com saved $22 million dollars by using
>>Linux+reiserfs instead of Netapps and Veritas.  They may be good, but
>>reiser3 is cheaper.  A lot cheaper.  Do the math, and you'll buy Linux
>>instead I think.;-)
>>
>>Nobody has done reiser3 vs. WAFL benchmarks.
>>    
>>
>
>One thing that should be stated here is that you can use some of the money you 
>save to buy more powerful hardware.  If ReiserFS is running on drives that 
>have a higher rotational speed and there are more drives in the RAID then it 
>should beat WAFL on most tests regardless of issues of file system 
>efficiency.
>
except that WAFL (at least it was in 1996 when I last benchmarked one) 
is memory bandwidth bound, not disk or CPU bound, when configured with 
14 disk drives.  They do a nice job of zero-copy optimizing from what I 
understand.

So, in a minor correction, instead of buying a WAFL server, you can buy 
3 reiserfs servers, and this may or may not overcome the memory 
bandwidth limitation, but it will certainly do a nicer job with the disk 
space limitation.;-)

I should also mention that actually most of these fileservers are disk 
space bound not performance bound for most of their users, at least that 
was true when I was a sysadmin back in the old days....

Hans

Re: Reiserfs with Samba vs NetApp Filer

[Russell Coker]

On Fri, 11 Oct 2002 22:00, Hans Reiser wrote:
> except that WAFL (at least it was in 1996 when I last benchmarked one)
> is memory bandwidth bound, not disk or CPU bound, when configured with
> 14 disk drives.  They do a nice job of zero-copy optimizing from what I
> understand.

Hmm, I wonder what the difference in memory bandwidth between a P4 and a 
NetApp device is.  Maybe the P4 has 3 times the memory bandwidth of the 
NetApp...

Also most people seem to run into a limit of PCI bus bandwidth first.  Then 
after surmounting that by using two PCI buses with hardware RAID controllers 
and running software RAID across them the next issue becomes cable tangle.

S-ATA should solve the cable issue.  PCI-X should solve the PCI bottleneck.  
Then it'll be interesting to see what the bottleneck is.

> I should also mention that actually most of these fileservers are disk
> space bound not performance bound for most of their users, at least that
> was true when I was a sysadmin back in the old days....

I think that's about to change (if it hasn't already).  3ware sells RAID 
controllers that do 8 or 12 drives.  200G ATA drives are now on sale, and 
320G drives will be available soon.

12 * 320G should solve the disk space issue for most people...

-- 
http://www.coker.com.au/bonnie++/     Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/       Postal SMTP/POP benchmark
http://www.coker.com.au/projects.html Projects I am working on
http://www.coker.com.au/~russell/     My home page

Re: Reiserfs with Samba vs NetApp Filer

[Hans Reiser]

Russell Coker wrote:

>  
>
>>I should also mention that actually most of these fileservers are disk
>>space bound not performance bound for most of their users, at least that
>>was true when I was a sysadmin back in the old days....
>>    
>>
>
>I think that's about to change (if it hasn't already).  3ware sells RAID 
>controllers that do 8 or 12 drives.  200G ATA drives are now on sale, and 
>320G drives will be available soon.
>
>12 * 320G should solve the disk space issue for most people...
>
>  
>
I don't think users will ever have enough disk space....

Re: Reiserfs with Samba vs NetApp Filer

[Russell Coker]

On Sat, 12 Oct 2002 15:00, Ragnar Kjørstad wrote:
> > I think that's about to change (if it hasn't already).  3ware sells RAID
> > controllers that do 8 or 12 drives.  200G ATA drives are now on sale, and
> > 320G drives will be available soon.
> >
> > 12 * 320G should solve the disk space issue for most people...
>
> The disk-space / cabeling-issue is mostly an IDE-problem anyway. If you
> go SCSI/FC there is practically no limit on how much storage you can put
> on a single server. (performance-issues aside)
>
> Sure, IDE is less expensive, but when comparing to NetApp there is
> plenty of room to go for SCSI/FC - so diskspace should defenitively not
> be an argument for going for NetApp.

Actually the problem then becomes case space.  On the linux IDE arrays list 
there are a few people having 2 or more 3ware cards, they have 16 or more IDE 
drives in a system.  With 24 inch cables they can get by with regular IDE 
cabling (and S-ATA will solve this properly soon).  3ware cards allow support 
for enough disks.  The problem then becomes what to house the disks in and 
how to supply enough power.

Here's the URL for subscribing to the IDE arrays list, I think that many 
people here will be interested (Ragnar - you might want to make 3ware not be 
the only IDE RAID hardware that's supported on the list).
http://lists.math.uh.edu/cgi-bin/mj_wwwusr

-- 
http://www.coker.com.au/bonnie++/     Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/       Postal SMTP/POP benchmark
http://www.coker.com.au/projects.html Projects I am working on
http://www.coker.com.au/~russell/     My home page

Re: Reiserfs with Samba vs NetApp Filer

[Heinz-Josef Claes]

Am Fre, 2002-10-11 um 12.54 schrieb Philippe Gramoullé:
> 
> Hi,
> 
> On Fri, 11 Oct 2002 12:40:41 +0200
> Lars Marowsky-Bree <lmb@suse.de> wrote:
> 
>   |  On 2002-10-10T19:53:38,
>   |     Philippe Gramoullé <philippe.gramoulle@mmania.com> said:
>   |  
>   |  > Yes, that's what i meant by "not *out of the box* snapshots".
>   |  
>   |  Well, LVM functionality comes out of the box with all modern distributions.
> 
> Ok let me rephrase it: First i didn't want to start a thread on pros/cons of LVM.
> 
> And i only wanted to say that LVM (though part of all modern distributions) doesn't
> come installed out of the box,at least not on Debain , which we use mainly.
> Whereas snapshots on Netapp comes out of the box with DATAONTAP.
> 
> But LVM is there of course if one needs snapshot functionnality.
> 
Some years ago I was responsible for buying some filers for a very
critical environment (a dealing room), so I got some experiences (also
with errors).

The snapshot functionnality of the filers makes a snapshot on the block
layer on a big filesystem in a minute or so. The mainly needed overhead
are the used blocks between the snapshot (or after the snapshot). This
is not comparable to a snapshot of EMC, which is simply a copy of the
hole disk to another one (with RAID under it and so on ...).

Is LVM comparable to netapp or to EMC snapshot?

I was very impressed by the netapp snapshot because you can make a
snapshot very fast (mainly the inodes are copied) and also online during
the day (which we did). The users could themselfes restore there data
via normal filesystem operations. We had nearly no need anymore to
restore data from tapes.
(This was the reason why I wrote storeBackup, a simple tool which has
some disadvantages and also some advantages compared to a filer.)

Hope this clearyfies the functionality of the Netapp snapshot, which I
think is patented!

-- 
Heinz-Josef Claes			hjclaes@web.de
project: http://sourceforge.net/projects/storebackup
	-> snapshot-like backup to another disk

Re: Reiserfs with Samba vs NetApp Filer

[Adrian Phillips]

>>>>> "Russell" == Russell Coker <bofh@coker.com.au> writes:

    Russell> Actually the problem then becomes case space.  On the
    Russell> linux IDE arrays list there are a few people having 2 or
    Russell> more 3ware cards, they have 16 or more IDE drives in a
    Russell> system.  With 24 inch cables they can get by with regular
    Russell> IDE cabling (and S-ATA will solve this properly soon).
    Russell> 3ware cards allow support for enough disks.  The problem
    Russell> then becomes what to house the disks in and how to supply
    Russell> enough power.

This may be a bit OT for this list by now. We've just had delivered a
28 disk solution (without all the disks yet but approx 5TB). There is
a cabinet which can handle 32 disks I believe but don't ask me about
power for it. We've got a 810W failover power supply with
this. Cabling is supplied by 3ware, they have up to 36", out of
spec. but of decent enough quality to work (I hope).

Anyway, there are limits to IDE til SATA turns up. My boss is looking
at a 50TB solution and (PATA) IDE currently can't handle this.

Sincerely,

Adrian Phillips

-- 
Your mouse has moved.
Windows NT must be restarted for the change to take effect.
Reboot now?  [OK]

Re: Reiserfs with Samba vs NetApp Filer

[Hans Reiser]

Russell Coker wrote:

>On Sat, 12 Oct 2002 15:00, Ragnar Kjørstad wrote:
>  
>
>>>I think that's about to change (if it hasn't already).  3ware sells RAID
>>>controllers that do 8 or 12 drives.  200G ATA drives are now on sale, and
>>>320G drives will be available soon.
>>>
>>>12 * 320G should solve the disk space issue for most people...
>>>      
>>>
>>The disk-space / cabeling-issue is mostly an IDE-problem anyway. If you
>>go SCSI/FC there is practically no limit on how much storage you can put
>>on a single server. (performance-issues aside)
>>
>>Sure, IDE is less expensive, but when comparing to NetApp there is
>>plenty of room to go for SCSI/FC - so diskspace should defenitively not
>>be an argument for going for NetApp.
>>    
>>
>
>Actually the problem then becomes case space.  On the linux IDE arrays list 
>there are a few people having 2 or more 3ware cards, they have 16 or more IDE 
>drives in a system.  With 24 inch cables they can get by with regular IDE 
>cabling (and S-ATA will solve this properly soon).  3ware cards allow support 
>for enough disks.  The problem then becomes what to house the disks in and 
>how to supply enough power.
>
>Here's the URL for subscribing to the IDE arrays list, I think that many 
>people here will be interested (Ragnar - you might want to make 3ware not be 
>the only IDE RAID hardware that's supported on the list).
>http://lists.math.uh.edu/cgi-bin/mj_wwwusr
>
>  
>
When I referred to disk space bound, I meant that users rarely have as 
many gigabytes as they need given the dollars that they are willing to 
spend.  One can always buy three servers if case space is limited.

Someday not too long from now, it will look like one filesystem even 
though it is in multiple cases.  Whether that is in reiser5 or reiser6 
depends on what sponsors fund first.

Hans

Re: Reiserfs with Samba vs NetApp Filer

[Andreas Dilger]

On Oct 13, 2002  01:55 +0400, Hans Reiser wrote:
> Someday not too long from now, it will look like one filesystem even 
> though it is in multiple cases.  Whether that is in reiser5 or reiser6 
> depends on what sponsors fund first.

Hans,
you should take a look at Lustre - www.lustre.org.  We are basically
already developing what you are suggesting - a distributed filesystem
which is built atop two or more local filesystems.  The aggregate
throughput of N lustre storage servers is basically N times the
throughput of a single server (clients communicate directly with the
storage targets, so the cross-sectional bandwidth in perfectly
scalable on a switched network).

Like Intermezzo, Lustre can be stacked on top of journaling local
filesystems, so it would be possible to use reiserfs for both the
metadata and storage targets.

We are deploying on a 1000-node cluster early next year, and expect
total throughput around 4GB/s (we have already made a limited test
at 1.4GB/s) with 90TB of storage - on a 2.4 kernel.  Because we are
using multiple separate filesystems, we are not hampered by the
2TB block device limit, and we get all sorts of parallelisms that
are not possible with a single large server.

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/

Re: Reiserfs with Samba vs NetApp Filer

[The Amazing Dragon]

> From: Russell Coker <bofh@coker.com.au>
> On Fri, 11 Oct 2002 22:00, Hans Reiser wrote:
> > except that WAFL (at least it was in 1996 when I last benchmarked one)
> > is memory bandwidth bound, not disk or CPU bound, when configured with
> > 14 disk drives.  They do a nice job of zero-copy optimizing from what I
> > understand.
> 
> Hmm, I wonder what the difference in memory bandwidth between a P4 and a 
> NetApp device is.  Maybe the P4 has 3 times the memory bandwidth of the 
> NetApp...

As of early last year, the low-end NetApps used a P3; so a factor of 4 on
their processor bus. They were using a combination of PCI bus(es), so the
I/O wasn't that much better on the low-end.

> Also most people seem to run into a limit of PCI bus bandwidth first.  Then 
> after surmounting that by using two PCI buses with hardware RAID controllers 
> and running software RAID across them the next issue becomes cable tangle.
> 
> S-ATA should solve the cable issue.  PCI-X should solve the PCI bottleneck.  
> Then it'll be interesting to see what the bottleneck is.

<cough>  <cough>         (about PCI-X) Yeah riiiight.

I believe PCI-X is still 64-bits wide (I haven't seen anything mention
this so I assume it is unchanged) and up to 133MHz, so theoretical
bandwidth of 1066MBps. So 7 Serial ATA drives, perhaps 10 drives if you
take into account the real performance of drives.

Then you've got to add network, gigabit ethernet is 250MBps (total, full
duplex). There is ten gigabit coming in the near future which half duplex
is 1250MBps, I guess you're going to need a second PCI-X bus just to
handle /most/ of the network bandwidth.

What astonishes me is that 66MHz 64-bit PCI is *still* confined
exclusively to high end motherboards, when even the low-end motherboards
are struggling against the bandwidth monster.

> > I should also mention that actually most of these fileservers are disk
> > space bound not performance bound for most of their users, at least that
> > was true when I was a sysadmin back in the old days....
> 
> I think that's about to change (if it hasn't already).  3ware sells RAID 
> controllers that do 8 or 12 drives.  200G ATA drives are now on sale, and 
> 320G drives will be available soon.
> 
> 12 * 320G should solve the disk space issue for most people...

For now, but I'm sure either games or something else will expand the
storage space used by people. People are now storing zillions of songs on
hard drives because compression (MPEG-1 layer 3) and storage are up to
the challenge, quite possibly it will be high resolution video tommorrow.

Then there is the challenge of backups. I kind of like the idea of a
DVD-R drive that takes whole spools of disks, but those are still
expensive and are still losing the race with hard drives.


-- 
|\__/|\__/|\______          --=> 8-) EHM <=--          ______/|\__/|\__/|
\    |    |       | EHeM@cs.pdx.edu      PGP 8881EF59 |       |    |    /
  \   \   | ______| -O #include <stddisclaimer.h> O-  |______ |   /   /
    \___\_|/82 04 A1 3C C7 B1 37 2A   E3 6E 84 DA 97 4C 40 E6\|_/___/

Re: Reiserfs with Samba vs NetApp Filer

[Russell Coker]

On Sun, 13 Oct 2002 01:02, The Amazing Dragon (Elliott Mitchell) wrote:
> > S-ATA should solve the cable issue.  PCI-X should solve the PCI
> > bottleneck. Then it'll be interesting to see what the bottleneck is.
>
> <cough>  <cough>         (about PCI-X) Yeah riiiight.
>
> I believe PCI-X is still 64-bits wide (I haven't seen anything mention
> this so I assume it is unchanged) and up to 133MHz, so theoretical
> bandwidth of 1066MBps. So 7 Serial ATA drives, perhaps 10 drives if you
> take into account the real performance of drives.

Currently people who are using 3ware RAID controllers are finding that PCI is 
a serious bottleneck.  PCI-X is apparently twice as fast as PCI which helps.  
Then you can have multiple PCI-X buses (as high end motherboards now have 
multiple PCI buses).

> Then you've got to add network, gigabit ethernet is 250MBps (total, full
> duplex). There is ten gigabit coming in the near future which half duplex
> is 1250MBps, I guess you're going to need a second PCI-X bus just to
> handle /most/ of the network bandwidth.

I still haven't heard of anyone getting more than 50% throughput out of 1Gb 
Ethernet, I doubt that a PC will use any portion of 10Gb Ethernet at any time 
in the near future.

> What astonishes me is that 66MHz 64-bit PCI is *still* confined
> exclusively to high end motherboards, when even the low-end motherboards
> are struggling against the bandwidth monster.

Yes, that sucks!  It seems that they think that we only want performance for 
graphics (which AGP satisfies).  Maybe we need a RAID controller that plugs 
into an AGP port?  ;)

-- 
http://www.coker.com.au/bonnie++/     Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/       Postal SMTP/POP benchmark
http://www.coker.com.au/projects.html Projects I am working on
http://www.coker.com.au/~russell/     My home page

Re: Reiserfs with Samba vs NetApp Filer

[Hans Reiser]

Andreas Dilger wrote:

>On Oct 13, 2002  01:55 +0400, Hans Reiser wrote:
>  
>
>>Someday not too long from now, it will look like one filesystem even 
>>though it is in multiple cases.  Whether that is in reiser5 or reiser6 
>>depends on what sponsors fund first.
>>    
>>
>
>Hans,
>you should take a look at Lustre - www.lustre.org.  We are basically
>already developing what you are suggesting - a distributed filesystem
>which is built atop two or more local filesystems.  The aggregate
>throughput of N lustre storage servers is basically N times the
>throughput of a single server (clients communicate directly with the
>storage targets, so the cross-sectional bandwidth in perfectly
>scalable on a switched network).
>
>Like Intermezzo, Lustre can be stacked on top of journaling local
>filesystems, so it would be possible to use reiserfs for both the
>metadata and storage targets.
>
>We are deploying on a 1000-node cluster early next year, and expect
>total throughput around 4GB/s (we have already made a limited test
>at 1.4GB/s) with 90TB of storage - on a 2.4 kernel.  Because we are
>using multiple separate filesystems, we are not hampered by the
>2TB block device limit, and we get all sorts of parallelisms that
>are not possible with a single large server.
>
>Cheers, Andreas
>--
>Andreas Dilger
>http://www-mddsp.enel.ucalgary.ca/People/adilger/
>http://sourceforge.net/projects/ext2resize/
>
>
>
>  
>
I really don't understand what is the advantage of object based disk 
storage.  It seems like its main effect is to prevent people from coming 
up with optimizations the drive manufacturer did not think of.  I don't 
at all understand these supposed metadata advantages.  We are lucky that 
we don't have in disk drives the sort of innovation inhibiting 
separation of compilers and CPUs that our compatriots in the language 
design business suffer from.  The more smarts that go into the drive, 
the more our field will ossify, unless they work closely with FS authors.

I enjoyed reading about your project, thanks for the URL.

Hans

Re: Reiserfs with Samba vs NetApp Filer

[Hans Reiser]

Ragnar Kjørstad wrote:

>On Sun, Oct 13, 2002 at 01:27:32AM +0200, Russell Coker wrote:
>  
>
>>>Then you've got to add network, gigabit ethernet is 250MBps (total, full
>>>duplex). There is ten gigabit coming in the near future which half duplex
>>>is 1250MBps, I guess you're going to need a second PCI-X bus just to
>>>handle /most/ of the network bandwidth.
>>>      
>>>
>>I still haven't heard of anyone getting more than 50% throughput out of 1Gb 
>>Ethernet, I doubt that a PC will use any portion of 10Gb Ethernet at any time 
>>in the near future.
>>    
>>
>
>We've messured over 70 MB/s throughput on our NAS-server (NFS). 
>That is 70MB/s in data-transfer (bonnie++ numbers), so when you take
>NFS, UDP, IP and ethernet-overhead into account you get an even higher
>network throughput.
>
>I haven't tested raw ethernet-throughput, but I would assume that to be
>even higher.
>
>
>  
>
Is that a bigstorage.com NAS?

Re: Reiserfs with Samba vs NetApp Filer

[Andreas Dilger]

On Oct 13, 2002  05:23 +0400, Hans Reiser wrote:
> Andreas Dilger wrote:
> >On Oct 13, 2002  01:55 +0400, Hans Reiser wrote:
> >>Someday not too long from now, it will look like one filesystem even 
> >>though it is in multiple cases.  Whether that is in reiser5 or reiser6 
> >>depends on what sponsors fund first.
> >
> >you should take a look at Lustre - www.lustre.org.  We are basically
> >already developing what you are suggesting - a distributed filesystem
> >which is built atop two or more local filesystems.  The aggregate
> >throughput of N lustre storage servers is basically N times the
> >throughput of a single server (clients communicate directly with the
> >storage targets, so the cross-sectional bandwidth in perfectly
> >scalable on a switched network).
> >
> >Like Intermezzo, Lustre can be stacked on top of journaling local
> >filesystems, so it would be possible to use reiserfs for both the
> >metadata and storage targets.
> >
> >We are deploying on a 1000-node cluster early next year, and expect
> >total throughput around 4GB/s (we have already made a limited test
> >at 1.4GB/s) with 90TB of storage - on a 2.4 kernel.  Because we are
> >using multiple separate filesystems, we are not hampered by the
> >2TB block device limit, and we get all sorts of parallelisms that
> >are not possible with a single large server.
>
> I really don't understand what is the advantage of object based disk 
> storage.  It seems like its main effect is to prevent people from coming 
> up with optimizations the drive manufacturer did not think of.  I don't 
> at all understand these supposed metadata advantages.  We are lucky that 
> we don't have in disk drives the sort of innovation inhibiting 
> separation of compilers and CPUs that our compatriots in the language 
> design business suffer from.  The more smarts that go into the drive, 
> the more our field will ossify, unless they work closely with FS authors.

While it is _possible_ that you have object based storage on a drive,
the reality is that the object storage targets (OSTs henceforth) are
in practise really large storage systems, like a IBM Shark, or a Linux
box with a few TB of disk and RAID and LVM, and most importantly have a
regular filesystem like ext3 or reiserfs on top of all that storage to
do all of the real storage management.

The benefit of an object based netowrk protocol like Lustre is that the
client is free from all of the details of file and block allocation, and
the OST filesystem can do all of this.  Since each OST has an independent
filesystem, it can handle all of the locking/threading for block and
inode allocation locally.  It can also do this in any way it sees fit,
so it actually allows for MORE innovation at the OST filesystem level
than other distributed filesystems.

The Lustre network protocol could be considered akin to a network
version of the Linux VFS - the Lustre client (like a Linux process)
is doing I/O on a file, but the Lustre OST (like a Linux filesystem)
is free to implement the details of storing data within that file as
it sees fit.  Similarly, the metadata server (MDS) is free to store
filenames, EA data, etc however it wants.

Lustre, like the VFS, needs locking to ensure multiple processes do
not do conflicting things.  The Lustre locking code actually is only
doing per-node locking, and trusts the Linux VFS to do the right thing
internally, so we leverage as much of Al Viro's work in this complex
area as we possibly can ;-).

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/

Re: Reiserfs with Samba vs NetApp Filer

[The Amazing Dragon]

> From: Russell Coker <bofh@coker.com.au>
> On Sun, 13 Oct 2002 01:02, The Amazing Dragon (Elliott Mitchell) wrote:
> > Then you've got to add network, gigabit ethernet is 250MBps (total, full
> > duplex). There is ten gigabit coming in the near future which half duplex
> > is 1250MBps, I guess you're going to need a second PCI-X bus just to
> > handle /most/ of the network bandwidth.
> 
> I still haven't heard of anyone getting more than 50% throughput out of 1Gb 
> Ethernet, I doubt that a PC will use any portion of 10Gb Ethernet at any time 
> in the near future.

Depending upon the machine and task, that is not surprising. For low end
hardware 50% utilization is impressive since with PCI doing 133MBps, your
bus is *completly* used. Higher end hardware should do better (and then
you start running into the speed of the network code).

I've been pondering how well doing cache-coherency in software over
(gigabit/multiple-100) ethernet would work. That would likely be able to
severely chew on even 10G ether (but this is wandering away from
storage).

> > What astonishes me is that 66MHz 64-bit PCI is *still* confined
> > exclusively to high end motherboards, when even the low-end motherboards
> > are struggling against the bandwidth monster.
> 
> Yes, that sucks!  It seems that they think that we only want performance for 
> graphics (which AGP satisfies).  Maybe we need a RAID controller that plugs 
> into an AGP port?  ;)

I've pondered that idea myself. AGP is plain 66MHz/32bit, 1x was single
data rate (266MB/s), 2x went for double data rate (both clock edges,
533MB/s), 4x went for quadruple data rate (1066MB/s), 8x multiplies the
clock by 8 (2166MB/s). So 8x will beat PCI-X by a factor of two, but then
how many motherboards have you seen that feature dual AGP slots (I'd been
wondering why nobody had done that with everyone dreaming of dual head
and nVidia's GeForce 3 only had a single output)?

Interestingly even as AGP has gotten faster, the original need (textures)
has been alleviated because the graphics chips have gotten so damn fast
even AGP8x isn't anywhere near enough to satisfy the bandwidth demands
for textures (Radeon 9700 has 128MB of memory with bandwidth of 20GB/s).
If OpenGL 2.1 was to introduce a display list processor, it could even
return to the point where plain vanilla 33MHz/32bit PCI would be quite
usable again.


-- 
|\__/|\__/|\______          --=> 8-) EHM <=--          ______/|\__/|\__/|
\    |    |       | EHeM@cs.pdx.edu      PGP 8881EF59 |       |    |    /
  \   \   | ______| -O #include <stddisclaimer.h> O-  |______ |   /   /
    \___\_|/82 04 A1 3C C7 B1 37 2A   E3 6E 84 DA 97 4C 40 E6\|_/___/

Re: Reiserfs with Samba vs NetApp Filer

[Hans Reiser]

Andreas Dilger wrote:

>On Oct 13, 2002  05:23 +0400, Hans Reiser wrote:
>  
>
>>Andreas Dilger wrote:
>>    
>>
>>>On Oct 13, 2002  01:55 +0400, Hans Reiser wrote:
>>>      
>>>
>>>>Someday not too long from now, it will look like one filesystem even 
>>>>though it is in multiple cases.  Whether that is in reiser5 or reiser6 
>>>>depends on what sponsors fund first.
>>>>        
>>>>
>>>you should take a look at Lustre - www.lustre.org.  We are basically
>>>already developing what you are suggesting - a distributed filesystem
>>>which is built atop two or more local filesystems.  The aggregate
>>>throughput of N lustre storage servers is basically N times the
>>>throughput of a single server (clients communicate directly with the
>>>storage targets, so the cross-sectional bandwidth in perfectly
>>>scalable on a switched network).
>>>
>>>Like Intermezzo, Lustre can be stacked on top of journaling local
>>>filesystems, so it would be possible to use reiserfs for both the
>>>metadata and storage targets.
>>>
>>>We are deploying on a 1000-node cluster early next year, and expect
>>>total throughput around 4GB/s (we have already made a limited test
>>>at 1.4GB/s) with 90TB of storage - on a 2.4 kernel.  Because we are
>>>using multiple separate filesystems, we are not hampered by the
>>>2TB block device limit, and we get all sorts of parallelisms that
>>>are not possible with a single large server.
>>>      
>>>
>>I really don't understand what is the advantage of object based disk 
>>storage.  It seems like its main effect is to prevent people from coming 
>>up with optimizations the drive manufacturer did not think of.  I don't 
>>at all understand these supposed metadata advantages.  We are lucky that 
>>we don't have in disk drives the sort of innovation inhibiting 
>>separation of compilers and CPUs that our compatriots in the language 
>>design business suffer from.  The more smarts that go into the drive, 
>>the more our field will ossify, unless they work closely with FS authors.
>>    
>>
>
>While it is _possible_ that you have object based storage on a drive,
>the reality is that the object storage targets (OSTs henceforth) are
>in practise really large storage systems, like a IBM Shark, or a Linux
>box with a few TB of disk and RAID and LVM, and most importantly have a
>regular filesystem like ext3 or reiserfs on top of all that storage to
>do all of the real storage management.
>
>The benefit of an object based netowrk protocol like Lustre is that the
>client is free from all of the details of file and block allocation, and
>the OST filesystem can do all of this.  Since each OST has an independent
>filesystem, it can handle all of the locking/threading for block and
>inode allocation locally.  It can also do this in any way it sees fit,
>so it actually allows for MORE innovation at the OST filesystem level
>than other distributed filesystems.
>
>The Lustre network protocol could be considered akin to a network
>version of the Linux VFS - the Lustre client (like a Linux process)
>is doing I/O on a file, but the Lustre OST (like a Linux filesystem)
>is free to implement the details of storing data within that file as
>it sees fit.  Similarly, the metadata server (MDS) is free to store
>filenames, EA data, etc however it wants.
>
>Lustre, like the VFS, needs locking to ensure multiple processes do
>not do conflicting things.  The Lustre locking code actually is only
>doing per-node locking, and trusts the Linux VFS to do the right thing
>internally, so we leverage as much of Al Viro's work in this complex
>area as we possibly can ;-).
>
>Cheers, Andreas
>--
>Andreas Dilger
>http://www-mddsp.enel.ucalgary.ca/People/adilger/
>http://sourceforge.net/projects/ext2resize/
>
>
>
>  
>
Ok, this makes some sense.  How does Seagate feel about that view, I am 
curious?

Hans

Re: Reiserfs with Samba vs NetApp Filer

[Andreas Dilger]

On Oct 13, 2002  17:48 +0400, Hans Reiser wrote:
> Andreas Dilger wrote:
> >While it is _possible_ that you have object based storage on a drive,
> >the reality is that the object storage targets (OSTs henceforth) are
> >in practise really large storage systems, like a IBM Shark, or a Linux
> >box with a few TB of disk and RAID and LVM, and most importantly have a
> >regular filesystem like ext3 or reiserfs on top of all that storage to
> >do all of the real storage management.
> >
> >The benefit of an object based netowrk protocol like Lustre is that the
> >client is free from all of the details of file and block allocation, and
> >the OST filesystem can do all of this.  Since each OST has an independent
> >filesystem, it can handle all of the locking/threading for block and
> >inode allocation locally.  It can also do this in any way it sees fit,
> >so it actually allows for MORE innovation at the OST filesystem level
> >than other distributed filesystems.
> >
> >The Lustre network protocol could be considered akin to a network
> >version of the Linux VFS - the Lustre client (like a Linux process)
> >is doing I/O on a file, but the Lustre OST (like a Linux filesystem)
> >is free to implement the details of storing data within that file as
> >it sees fit.  Similarly, the metadata server (MDS) is free to store
> >filenames, EA data, etc however it wants.
> >
> >Lustre, like the VFS, needs locking to ensure multiple processes do
> >not do conflicting things.  The Lustre locking code actually is only
> >doing per-node locking, and trusts the Linux VFS to do the right thing
> >internally, so we leverage as much of Al Viro's work in this complex
> >area as we possibly can ;-).
>
> Ok, this makes some sense.  How does Seagate feel about that view, I am 
> curious?

It is irrelevant how they feel, since they stopped funding our project
in 2000 after we had barely completed the initial prototype.  Even so,
(back when we were looking at "smart" individual disks that presented
an OBD interface over ethernet), the disks themselves would have been
running Linux internally, with the block and inode allocation half of
the filesystem running there.

Since it was a lot of additional effort to maintain the split filesystem
code ourselves, and also less flexible, we implemented an "obdfilter"
OBD driver, which stack on top of a regular Linux filesystem at the VFS
layer, and basically replaces the file I/O syscall interface with a
network RPC layer.

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/

Re: Reiserfs with Samba vs NetApp Filer

[Hans Reiser]

Ragnar Kjørstad wrote:

>
>I think there the linux filesystems possible implement two very distinct
>features:
>- an (transactional?) object store (like the reiserfs tree)
>- and special objects; like directories, "inodes", EAs and so on
>
>I know that in some filesystems the distinction is not so clear (e.g. in
>ext2 where inodes have their specific place in the object-store).
>
>But my point is; would it not make sense to split up the filesystem-code
>(for the filesystems that would allow it) in the two parts mentioned
>above? Lustre is not alone in wanting a more low-level interface to the
>disk - I think this is very simular to the interface that squid uses, is
>it not?
>
>
>
>  
>

I was about to suggest to Andreas that his implementation on top of 
reiserfs is likely to be very inefficient unless he does something 
similar to our squid for reiserfs code (sponsored by someone who didn't 
continue sponsoring in the typical dotcom "more than nine months, no 
investor wants to wait that long for a return, give me a shorter 
timeline than that" fashion).

Andreas, very briefly, you want to find things in reiserfs by their 
keys, and you have no need for any directory entries at all, and having 
those directory entries will add substantial overhead.  Your metadata 
manager can probably track references, and if so there is no need for 
the FS to track it.

Unless you cut out directories you might even find that ext3 performs 
better because you can do lookups by inodes efficiently, and we can only 
do lookups by keys efficiently.

Hans

Reiserfs with Samba vs NetApp Filer (purely performance)

[darren]

Hi all,

Thanks for all the very informative bits on this thread.

In my usage, I intend to implement this fileserver to complement another
server that runs a very intensive small files read/write operation.

These files often run into 200,000 or more but are usually in the range
of 100kb or less.

Hence, space is not a big issue to me, but performance is. (there's no
point splitting out the IO from the main server if it runs slower!!).

I will be using an spare server to do this job: Dell Xeon 2GHz with 5 x
72Gb SCSI hdd (I got on board RAID but have not decided on how to
configure yet -help here are welcome too!).

My options are: Win2k (NTFS with CIFS), Linux (Reiserfs with Samba) or
Netapp 870.

What do you guys think?

Thanx again
Darren

Re: Reiserfs with Samba vs NetApp Filer

[Andreas Dilger]

On Oct 13, 2002  20:36 +0400, Hans Reiser wrote:
> Andreas, very briefly, you want to find things in reiserfs by their 
> keys, and you have no need for any directory entries at all, and having 
> those directory entries will add substantial overhead.  Your metadata 
> manager can probably track references, and if so there is no need for 
> the FS to track it.

On the storage target (OST), we don't really need directories at all.
The obdfilter uses a 64-bit object number (generated by the OST itself)
to refer to "files" it is storing, so as long as the reiserfs keys fit
into 64 bits, having a very efficient lookup method would be ideal.

The metadata server (MDS) of course must handle directories as a normal
filesystem does, because it is in charge of presenting a normal filesystem
heirarchy to applications.

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/

Re: Reiserfs with Samba vs NetApp Filer

[Hans Reiser]

Andreas Dilger wrote:

>On Oct 13, 2002  20:36 +0400, Hans Reiser wrote:
>  
>
>>Andreas, very briefly, you want to find things in reiserfs by their 
>>keys, and you have no need for any directory entries at all, and having 
>>those directory entries will add substantial overhead.  Your metadata 
>>manager can probably track references, and if so there is no need for 
>>the FS to track it.
>>    
>>
>
>On the storage target (OST), we don't really need directories at all.
>The obdfilter uses a 64-bit object number (generated by the OST itself)
>to refer to "files" it is storing, so as long as the reiserfs keys fit
>into 64 bits, having a very efficient lookup method would be ideal.
>
>The metadata server (MDS) of course must handle directories as a normal
>filesystem does, because it is in charge of presenting a normal filesystem
>heirarchy to applications.
>
>Cheers, Andreas
>--
>Andreas Dilger
>http://www-mddsp.enel.ucalgary.ca/People/adilger/
>http://sourceforge.net/projects/ext2resize/
>
>
>
>  
>
Well, if you decide you want to do a serious integration of reiserfs and 
your product, let us know.  Using directories on the OST would not be a 
serious integration.

Hans

Re: Reiserfs with Samba vs NetApp Filer

[Russell Coker]

On Mon, 14 Oct 2002 19:59, Chris Wedgwood wrote:
> On Sat, Oct 12, 2002 at 04:02:03PM -0700, The Amazing Dragon wrote:
> > I believe PCI-X is still 64-bits wide (I haven't seen anything
> > mention this so I assume it is unchanged) and up to 133MHz, so
> > theoretical bandwidth of 1066MBps. So 7 Serial ATA drives, perhaps
> > 10 drives if you take into account the real performance of drives.
>
> Bus width is at best a very naive metric.
>
> At present, you can't get 100MB/s sustained read off any IDE drive I'm
> aware off, not even for linear-read-out.  You *may* get 50MB/s on a
> high-end SCSI disk, most IDE drives seem to top out around 30MB/s.

In my tests IDE drives topped out at 30MB/s for sustained reads from the start 
of the drive two years ago, one year ago they were approaching 40MB/s.  I 
haven't tested the latest drives, but I expect them to be faster...  Maybe 
the recently released IDE drives will sustain 50MB/s for the first few dozen 
gigs at least...

> The other thing is, how many people need 1G/s+ disk IO?  Chances are
> if you need this, you might be convinced to use real-hardware instead
> of a PC full of IDE drives.

The real question is "how many people need such linear performance with no 
seeks?".  Probably almost none.

> > What astonishes me is that 66MHz 64-bit PCI is *still* confined
> > exclusively to high end motherboards, when even the low-end
> > motherboards are struggling against the bandwidth monster.
>
> Nope.  Some cheap MB's are 64/66, the Dual Athlon boards for example.

Cheap == single CPU.

In a quick web search of some companies I buy from, one sells a single 
dual-CPU motherboard which costs three times as much as the single-CPU 
motherboards (which have become amazingly cheap).  However it'll end up 
costing more because with the lack of choice I'll probably be forced to spend 
more on other things.

> P.S. FWIW, SGI demonstrated 7GB/s through the filesystem over five
>      years ago.

Yes, and still no-one makes PC hardware that compares.  :(

-- 
http://www.coker.com.au/bonnie++/     Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/       Postal SMTP/POP benchmark
http://www.coker.com.au/projects.html Projects I am working on
http://www.coker.com.au/~russell/     My home page

Re: Reiserfs with Samba vs NetApp Filer

[Dieter Nützel]

Am Montag, 14. Oktober 2002 22:52 schrieb Russell Coker:
> On Mon, 14 Oct 2002 19:59, Chris Wedgwood wrote:
> > On Sat, Oct 12, 2002 at 04:02:03PM -0700, The Amazing Dragon wrote:

[-]

> > At present, you can't get 100MB/s sustained read off any IDE drive I'm
> > aware off, not even for linear-read-out.  You *may* get 50MB/s on a
> > high-end SCSI disk, most IDE drives seem to top out around 30MB/s.
>
> In my tests IDE drives topped out at 30MB/s for sustained reads from the
> start of the drive two years ago, one year ago they were approaching
> 40MB/s.  I haven't tested the latest drives, but I expect them to be
> faster...  Maybe the recently released IDE drives will sustain 50MB/s for
> the first few dozen gigs at least...

Very true, today.
Maxtor, WD, IBM all aroung 45 MB/sec.
W'll see what the next round bring (Maxtor 200-320 GB).

But SCSI is there ~80 MB/sec (Maxtor, IBM) and Fujitsu is at 104/111 MB/sec 
;-)

> > > What astonishes me is that 66MHz 64-bit PCI is *still* confined
> > > exclusively to high end motherboards, when even the low-end
> > > motherboards are struggling against the bandwidth monster.
> >
> > Nope.  Some cheap MB's are 64/66, the Dual Athlon boards for example.
>
> Cheap == single CPU.
>
> In a quick web search of some companies I buy from, one sells a single
> dual-CPU motherboard which costs three times as much as the single-CPU
> motherboards (which have become amazingly cheap).  However it'll end up
> costing more because with the lack of choice I'll probably be forced to
> spend more on other things.

Dual Athlon MBs (760MPX, all with 64/66) are aroung 300 ¤ (EUR) here in 
Germany.

Regards,
	Dieter

Re: Reiserfs with Samba vs NetApp Filer

[Russell Coker]

On Tue, 15 Oct 2002 00:58, Chris Wedgwood wrote:
> On Mon, Oct 14, 2002 at 10:52:09PM +0200, Russell Coker wrote:
> > In my tests IDE drives topped out at 30MB/s for sustained reads from
> > the start of the drive two years ago, one year ago they were
> > approaching 40MB/s.
>
> It's a little higher than I expected.  How much does it vary from
> end-to-end?

See the following graph:
http://www.coker.com.au/~russell/hardware/46g.png

This shows testing a single 46G drive, two drives on different buses at the 
same time, and two drives on the same bus at the same time.  zcav (part of 
Bonnie++) was used to perform the tests.

> THat said, It's still a far cry from 100MB/s.

True.  S-ATA only needs it's speed for RAID attachments.

> > The real question is "how many people need such linear performance
> > with no seeks?".  Probably almost none.
>
> Across a decent fs, provided the layout isn't too bad, seeks really
> don't hurt that much for large reads.  With lots of spindles, it gets
> even better.

But how many people need such large reads?

The most disk intensive programs I use are Squid and GCC, Squid has an average 
file size of ~12K, and I suspect that the source files used by GCC aren't 
much bigger...

> > Cheap == single CPU.
>
> So put a single CPU in it then; or use some other MB.  64/66PCI isn't
> that hard to get or expensive.

Sure I can afford it if I really want it.  However the original discussion was 
about the lamentable fact that all the cheapest motherboards don't have it.  
So far no-one has presented any evidence of the <E100 motherbords having it.

> > In a quick web search of some companies I buy from, one sells a
> > single dual-CPU motherboard which costs three times as much as the
> > single-CPU motherboards (which have become amazingly cheap).
>
> The MB cost is a small part of the overall cost, especially if you're
> looking at 8x320GB drives.

True, but that's not what we were discussing.

-- 
http://www.coker.com.au/bonnie++/     Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/       Postal SMTP/POP benchmark
http://www.coker.com.au/projects.html Projects I am working on
http://www.coker.com.au/~russell/     My home page

Re: Reiserfs with Samba vs NetApp Filer

[Hans Reiser]

Russell Coker wrote:

>See the following graph:
>http://www.coker.com.au/~russell/hardware/46g.png
>
>This shows testing a single 46G drive, two drives on different buses at the 
>same time, and two drives on the same bus at the same time.  zcav (part of 
>Bonnie++) was used to perform the tests.
>
>  
>
>  
>
I am surprised that separating them onto different buses has so little 
effect.  It looks like most of the bottleneck for large reads off two 
drives is not the IDE bus, but something else (maybe CPU or memory 
bandwidth).

Hans

Re: Reiserfs with Samba vs NetApp Filer

[Hans Reiser]

Russell Coker wrote:

>See the following graph:
>http://www.coker.com.au/~russell/hardware/46g.png
>
>This shows testing a single 46G drive, two drives on different buses at the 
>same time, and two drives on the same bus at the same time.  zcav (part of 
>Bonnie++) was used to perform the tests.
>
>  
>

How many CPUs in this test?

Re: Reiserfs with Samba vs NetApp Filer

[Russell Coker]

On Tue, 15 Oct 2002 15:42, Hans Reiser wrote:
> Russell Coker wrote:
> >See the following graph:
> >http://www.coker.com.au/~russell/hardware/46g.png
> >
> >This shows testing a single 46G drive, two drives on different buses at
> > the same time, and two drives on the same bus at the same time.  zcav
> > (part of Bonnie++) was used to perform the tests.
>
> I am surprised that separating them onto different buses has so little
> effect.  It looks like most of the bottleneck for large reads off two
> drives is not the IDE bus, but something else (maybe CPU or memory
> bandwidth).

I was surprised too.  Especially as it's an ATA-66 bus (the bus was expected 
to be a bottleneck).

Only a single CPU.

I would like to do more research on this matter and write a magazine article 
(I already have a magazine wanting to publish it).  All I need is suitable 
access to the latest hardware to perform my tests (tests on old hardware 
while still being interesting research doesn't sell magazines).

-- 
http://www.coker.com.au/bonnie++/     Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/       Postal SMTP/POP benchmark
http://www.coker.com.au/projects.html Projects I am working on
http://www.coker.com.au/~russell/     My home page

Re: Reiserfs with Samba vs NetApp Filer (purely performance)

[Hans Reiser]

darren wrote:

>Hi all,
>
>Thanks for all the very informative bits on this thread.
>
>In my usage, I intend to implement this fileserver to complement another
>server that runs a very intensive small files read/write operation.
>
>These files often run into 200,000 or more but are usually in the range
>of 100kb or less.
>
>Hence, space is not a big issue to me, but performance is. (there's no
>point splitting out the IO from the main server if it runs slower!!).
>
>I will be using an spare server to do this job: Dell Xeon 2GHz with 5 x
>72Gb SCSI hdd (I got on board RAID but have not decided on how to
>configure yet -help here are welcome too!).
>
>My options are: Win2k (NTFS with CIFS), Linux (Reiserfs with Samba) or
>Netapp 870.
>
>What do you guys think?
>
>Thanx again
>Darren
>
>
>
>  
>
Turn off tails if you want good performance for V3.

Re: Reiserfs with Samba vs NetApp Filer (purely performance)

[Richard Sharpe]

On Sun, 15 Dec 2002, Hans Reiser wrote:

> darren wrote:
> 
> >Hi all,
> >
> >Thanks for all the very informative bits on this thread.
> >
> >In my usage, I intend to implement this fileserver to complement another
> >server that runs a very intensive small files read/write operation.
> >
> >These files often run into 200,000 or more but are usually in the range
> >of 100kb or less.
> >
> >Hence, space is not a big issue to me, but performance is. (there's no
> >point splitting out the IO from the main server if it runs slower!!).
> >
> >I will be using an spare server to do this job: Dell Xeon 2GHz with 5 x
> >72Gb SCSI hdd (I got on board RAID but have not decided on how to
> >configure yet -help here are welcome too!).
> >
> >My options are: Win2k (NTFS with CIFS), Linux (Reiserfs with Samba) or
> >Netapp 870.

If those are your choices, then I would rank them:

  NetApp F870, Linux with ReiserFS and Samba, Win2K

John Terpstra has done some testing with a modified cifs_bm which confirms 
that Linux with EXT2/3 or ReiserFS outperforms Win2K on the same hardware.

However, my testing of the NetApp suggests it will outperform the other 
two choices.

If you choose to use Samba, you will want to make sure that the sendfile 
stuff is implemented. Unfortunately, the directory indexing in Reiser does 
not help that much a lot of the time because Samba is forced to do 
directory scans because it has to implement case-independent 
lookups/searches.

Regards
-----
Richard Sharpe, rsharpe[at]ns.aus.com, rsharpe[at]samba.org, 
sharpe[at]ethereal.com, http://www.richardsharpe.com

Re: Reiserfs with Samba vs NetApp Filer (purely performance)

[Ragnar Kjørstad]

On Sat, Dec 14, 2002 at 02:34:06PM -0800, Richard Sharpe wrote:
> If those are your choices, then I would rank them:
> 
>   NetApp F870, Linux with ReiserFS and Samba, Win2K
> 
> John Terpstra has done some testing with a modified cifs_bm which confirms 
> that Linux with EXT2/3 or ReiserFS outperforms Win2K on the same hardware.
> 
> However, my testing of the NetApp suggests it will outperform the other 
> two choices.

What kind of benchmarks, and what were the results?

> If you choose to use Samba, you will want to make sure that the sendfile 
> stuff is implemented. Unfortunately, the directory indexing in Reiser does 
> not help that much a lot of the time because Samba is forced to do 
> directory scans because it has to implement case-independent 
> lookups/searches.

Isn't that an configuration-option?

As a sidenote, I think it wouldn't be too hard to make reiserfs
optionally case-insensitive, but still able to use indexes. Perhaps it
would even be possible to have two virtual directories pr traditional
directory? one that has a regular index and one that has a
case-insensitive index. Of course this is not relevant for the
right-here right-now choice of fileservers, but....


-- 
Ragnar Kjørstad

Re: Reiserfs with Samba vs NetApp Filer (purely performance)

[Hans Reiser]

Ragnar Kjørstad wrote:

>On Sat, Dec 14, 2002 at 02:34:06PM -0800, Richard Sharpe wrote:
>  
>
>>If those are your choices, then I would rank them:
>>
>>  NetApp F870, Linux with ReiserFS and Samba, Win2K
>>
>>John Terpstra has done some testing with a modified cifs_bm which confirms 
>>that Linux with EXT2/3 or ReiserFS outperforms Win2K on the same hardware.
>>
>>However, my testing of the NetApp suggests it will outperform the other 
>>two choices.
>>    
>>
>
>What kind of benchmarks, and what were the results?
>
>  
>
>>If you choose to use Samba, you will want to make sure that the sendfile 
>>stuff is implemented. Unfortunately, the directory indexing in Reiser does 
>>not help that much a lot of the time because Samba is forced to do 
>>directory scans because it has to implement case-independent 
>>lookups/searches.
>>    
>>
>
>Isn't that an configuration-option?
>
>As a sidenote, I think it wouldn't be too hard to make reiserfs
>optionally case-insensitive, but still able to use indexes. Perhaps it
>would even be possible to have two virtual directories pr traditional
>directory? one that has a regular index and one that has a
>case-insensitive index. Of course this is not relevant for the
>right-here right-now choice of fileservers, but....
>
>
>  
>
There is quite a lot that could be done to optimize for Samba, but 
nobody is sponsoring it unfortunately.  Seems like some money could be 
made by a clever appliance vendor....

Re: Reiserfs with Samba vs NetApp Filer (purely performance)

[Richard Sharpe]

On Sun, 15 Dec 2002, Ragnar Kjørstad wrote:

> On Sat, Dec 14, 2002 at 02:34:06PM -0800, Richard Sharpe wrote:
> > If those are your choices, then I would rank them:
> > 
> >   NetApp F870, Linux with ReiserFS and Samba, Win2K
> > 
> > John Terpstra has done some testing with a modified cifs_bm which confirms 
> > that Linux with EXT2/3 or ReiserFS outperforms Win2K on the same hardware.
> > 
> > However, my testing of the NetApp suggests it will outperform the other 
> > two choices.
> 
> What kind of benchmarks, and what were the results?

I was using cifs_bm, in both cases. We presented results at CIFS2002, and 
you can find the papers on the web. Also look for John Terpstra's paper at 
Sambaxp in Germany in April 2002. We tested a bunch of things, but we did 
not focus on comparative performance. cifs_bm is based around the 
NetBench-like tests from smbtorture. 

There is a CIFS benchmarking effort going on as part of SNIA, as well. We 
are working on something like SPECsfs for CIFS.
 
> > If you choose to use Samba, you will want to make sure that the sendfile 
> > stuff is implemented. Unfortunately, the directory indexing in Reiser does 
> > not help that much a lot of the time because Samba is forced to do 
> > directory scans because it has to implement case-independent 
> > lookups/searches.
> 
> Isn't that an configuration-option?

Well, yes it is, but if you want to be more like a Windows server, then 
you need to do it correctly.

> As a sidenote, I think it wouldn't be too hard to make reiserfs
> optionally case-insensitive, but still able to use indexes. Perhaps it
> would even be possible to have two virtual directories pr traditional
> directory? one that has a regular index and one that has a
> case-insensitive index. Of course this is not relevant for the
> right-here right-now choice of fileservers, but....

The namecache is system-wide. So, you will have to do it in the name cache 
code. One of the guys who worked for Quantum seems to have done some mods 
for XFS for that, and they added a lookup entry in the dcache ops table, 
it seems. This would allow a loadable file system to hook that entry and 
provide case independent lookups. There are some curly problems with 
negative entries. You would also want to make it a process settable 
feature, esp if you run NFS as well (guess who wants to do that :-)

Finally, if your goal is semantic equivalence with Windows, you will want 
two indexes in ReiserFS or EXT2/3 with the dirindex stuff anyway, since 
Windows actually has two name spaces (long names and 8.3 names) and you 
have to search both name spaces for at least some operations, and to 
ensure that you do the correct thing in the presence of names that differ 
only by the case of some characters, you must create 8.3 names on file 
creation (which is pretty much what NetApp do).

Regards
-----
Richard Sharpe, rsharpe[at]ns.aus.com, rsharpe[at]samba.org, 
sharpe[at]ethereal.com, http://www.richardsharpe.com

Re: Reiserfs with Samba vs NetApp Filer (purely performance)

[Richard Sharpe]

On Sun, 15 Dec 2002, Hans Reiser wrote:

> Ragnar Kjørstad wrote:
> 
> >>If you choose to use Samba, you will want to make sure that the sendfile 
> >>stuff is implemented. Unfortunately, the directory indexing in Reiser does 
> >>not help that much a lot of the time because Samba is forced to do 
> >>directory scans because it has to implement case-independent 
> >>lookups/searches.
> >>    
> >>
> >
> >Isn't that an configuration-option?
> >
> >As a sidenote, I think it wouldn't be too hard to make reiserfs
> >optionally case-insensitive, but still able to use indexes. Perhaps it
> >would even be possible to have two virtual directories pr traditional
> >directory? one that has a regular index and one that has a
> >case-insensitive index. Of course this is not relevant for the
> >right-here right-now choice of fileservers, but....
> >
> >
> >  
> >
> There is quite a lot that could be done to optimize for Samba, but 
> nobody is sponsoring it unfortunately.  Seems like some money could be 
> made by a clever appliance vendor....

The issues are complex. Samba still needs some work to make it perfect for 
appliance vendors, for example, although Andrew is talking about an NTFS 
layer which will make things better.

Those same appliance vendors will need/want file system changes, esp if 
they are trying to support NFS and CIFS off of the same box, and they tend 
to want to push many of the clever bits into the file system and keep them 
propietary :-) This is certainly true of the three vendors that I have any 
experience with.

However, I expect some patches for XFS will become available in the not 
too distant future.

Regards
-----
Richard Sharpe, rsharpe[at]ns.aus.com, rsharpe[at]samba.org, 
sharpe[at]ethereal.com, http://www.richardsharpe.com