ReiserFS v3 + millions of files?

ReiserFS v3 + millions of files?

[Dan Oglesby]

Greetings...

Long time ReiserFS user, first time I've had a problem.  Signed up for the 
mailing list last week, and was surprised to see so little traffic (might be 
a good thing?).

I'm running Red Hat 7.3 using a Red Hat 2.4.20 kernel.  The system has a 
RAID-5 array via 3Ware 7500 controller, and three Western Digital 120GB 
"Special Edition" hard drives.  The array is one filesystem, ReiserFS.  The 
operating system, swap, and other files are stored on a hard drive that is on 
the primary IDE controller off of the motherboard.

The system is a single board computer, with a P4 3.06 GHz hyperthreaded 
processor (kernel is SMP enabled), 512MB of RAM, and contains a mix of 
ReiserFS and EXT2 filesystems on the primary drive (ReiserFS only on the 
array).  No NFS.

The array is used to store what will basically amount to more than one million 
files with an average size of sixty kilobytes.

During simulations for file writes, I'm seeing write performance begin to drop 
dramatically after 800,000 files have been stored on the filesystem.

The filesystem is being mounted with the following options:  
defaults,notail,noatime,nodiratime

The filesystem was created with default options, basically a "mkreiserfs /dev/
sda1".

Is this behavior I should expect from ReiserFS v3?

This week I will be switching from a Red Hat kernel to a vanilla kernel (from 
kernel.org), first the latest 2.4 kernel, then the latest 2.6 kernel.  After 
that...  I dunno.

Help?

--Dan Oglesby

Re: ReiserFS v3 + millions of files?

[Hans Reiser]

Dan Oglesby wrote:

>Greetings...
>
>Long time ReiserFS user, first time I've had a problem.  Signed up for the 
>mailing list last week, and was surprised to see so little traffic (might be 
>a good thing?).
>
>I'm running Red Hat 7.3 using a Red Hat 2.4.20 kernel.  The system has a 
>RAID-5 array via 3Ware 7500 controller, and three Western Digital 120GB 
>"Special Edition" hard drives.  The array is one filesystem, ReiserFS.  The 
>operating system, swap, and other files are stored on a hard drive that is on 
>the primary IDE controller off of the motherboard.
>
>The system is a single board computer, with a P4 3.06 GHz hyperthreaded 
>processor (kernel is SMP enabled), 512MB of RAM, and contains a mix of 
>ReiserFS and EXT2 filesystems on the primary drive (ReiserFS only on the 
>array).  No NFS.
>
>The array is used to store what will basically amount to more than one million 
>files with an average size of sixty kilobytes.
>
>During simulations for file writes, I'm seeing write performance begin to drop 
>dramatically after 800,000 files have been stored on the filesystem.
>
>The filesystem is being mounted with the following options:  
>defaults,notail,noatime,nodiratime
>
>The filesystem was created with default options, basically a "mkreiserfs /dev/
>sda1".
>
>Is this behavior I should expect from ReiserFS v3?
>
>This week I will be switching from a Red Hat kernel to a vanilla kernel (from 
>kernel.org), first the latest 2.4 kernel, then the latest 2.6 kernel.  After 
>that...  I dunno.
>
>Help?
>
>--Dan Oglesby
>
>
>
>  
>
what size directories?

-- 
Hans

Re: ReiserFS v3 + millions of files?

[Hans Reiser]

I have unsubstantiated suspicions about VFS/dcache scalability.  It 
would be interesting to try to determine such things as how much of ram 
is consumed by dcache, etc.

Mr. Demidov, how long until sys_reiser4() can be used to create a 
million files sized 60k using lnodes?  It would be an interesting 
benchmarking experiment.....

sys_reiser4() sidesteps dcache....

Hans

Dan Oglesby wrote:

>Greetings...
>
>Long time ReiserFS user, first time I've had a problem.  Signed up for the 
>mailing list last week, and was surprised to see so little traffic (might be 
>a good thing?).
>
>I'm running Red Hat 7.3 using a Red Hat 2.4.20 kernel.  The system has a 
>RAID-5 array via 3Ware 7500 controller, and three Western Digital 120GB 
>"Special Edition" hard drives.  The array is one filesystem, ReiserFS.  The 
>operating system, swap, and other files are stored on a hard drive that is on 
>the primary IDE controller off of the motherboard.
>
>The system is a single board computer, with a P4 3.06 GHz hyperthreaded 
>processor (kernel is SMP enabled), 512MB of RAM, and contains a mix of 
>ReiserFS and EXT2 filesystems on the primary drive (ReiserFS only on the 
>array).  No NFS.
>
>The array is used to store what will basically amount to more than one million 
>files with an average size of sixty kilobytes.
>
>During simulations for file writes, I'm seeing write performance begin to drop 
>dramatically after 800,000 files have been stored on the filesystem.
>
>The filesystem is being mounted with the following options:  
>defaults,notail,noatime,nodiratime
>
>The filesystem was created with default options, basically a "mkreiserfs /dev/
>sda1".
>
>Is this behavior I should expect from ReiserFS v3?
>
>This week I will be switching from a Red Hat kernel to a vanilla kernel (from 
>kernel.org), first the latest 2.4 kernel, then the latest 2.6 kernel.  After 
>that...  I dunno.
>
>Help?
>
>--Dan Oglesby
>
>
>
>  
>


-- 
Hans

Re: ReiserFS v3 + millions of files?

[Nikita Danilov]

Dan Oglesby writes:
 > Greetings...
 > 
 > Long time ReiserFS user, first time I've had a problem.  Signed up for the 
 > mailing list last week, and was surprised to see so little traffic (might be 
 > a good thing?).
 > 
 > I'm running Red Hat 7.3 using a Red Hat 2.4.20 kernel.  The system has a 
 > RAID-5 array via 3Ware 7500 controller, and three Western Digital 120GB 
 > "Special Edition" hard drives.  The array is one filesystem, ReiserFS.  The 
 > operating system, swap, and other files are stored on a hard drive that is on 
 > the primary IDE controller off of the motherboard.
 > 
 > The system is a single board computer, with a P4 3.06 GHz hyperthreaded 
 > processor (kernel is SMP enabled), 512MB of RAM, and contains a mix of 
 > ReiserFS and EXT2 filesystems on the primary drive (ReiserFS only on the 
 > array).  No NFS.
 > 
 > The array is used to store what will basically amount to more than one million 
 > files with an average size of sixty kilobytes.
 > 
 > During simulations for file writes, I'm seeing write performance begin to drop 
 > dramatically after 800,000 files have been stored on the filesystem.

Reiserfs (both v3 and v4) stores directory entries (names within
directory) sorted by a hash of the file name. If files are created in
the "random" order, that is, if hashes of names aren't more or less
monotonic, reiserfs will have to modify the same block many times during
insertion of large number of files. As blocks with names stop fitting
into memory this means that the same block has to be fetched many times
from the disk.

To confirm that this is really what the cause of your problem, please
answer following questions:

1. are all your files in the same directory?
2. how names of files are generated?
3. what hash are using on reiserfs (default is r5).

Reiserfs has another problem due to its limited capability of handling
hash collisions in file names. Reiser4 scales much better in this
respect, and generally, works fine with scores of millions of files in
one directory.

 > 
 > The filesystem is being mounted with the following options:  
 > defaults,notail,noatime,nodiratime
 > 
 > The filesystem was created with default options, basically a "mkreiserfs /dev/
 > sda1".
 > 
 > Is this behavior I should expect from ReiserFS v3?
 > 
 > This week I will be switching from a Red Hat kernel to a vanilla kernel (from 
 > kernel.org), first the latest 2.4 kernel, then the latest 2.6 kernel.  After 
 > that...  I dunno.
 > 
 > Help?
 > 
 > --Dan Oglesby
 > 

Nikita.

Object Oriented FS

[darren]

Hi All,

Any comments about the Panasas ActiveScale File System (PanFS) by Panasas?

It is supposed to be a "object oriented" FS which allows very high
throughput by scaling

Re: Object Oriented FS

[Hans Reiser]

darren wrote:

> allows very high
>throughput by scaling
>
>
>
>  
>
Do you know what that  means?  (Seriously, I don't....)

-- 
Hans

RE: Object Oriented FS

[Mike Young]

Hans,

Do you believe this is just a marketing convention to refer to a file-system
as an object oriented one?  Bear in mind that Microsoft is supposed to
enrich their NTFS with object support from SQL Server in their next OS.  The
code name for this is Longhorn.

I really can't tell what this means to a file system in terms of support.
It sounds like it's intended to provide support for really large numbers of
files in a single directory vs. nested ones.  But I'm just not sure.  Do you
see a limitation, going forward, with Reiser4 and large numbers of files?

Thanks,

Mike

-----Original Message-----
From: Hans Reiser [mailto:reiser@namesys.com] 
Sent: Monday, October 27, 2003 10:23 AM
To: darren
Cc: reiserfs-list@namesys.com
Subject: Re: Object Oriented FS

darren wrote:

> allows very high
>throughput by scaling
>
>
>
>  
>
Do you know what that  means?  (Seriously, I don't....)

-- 
Hans

Re: Object Oriented FS

[Hans Reiser]

Mike Young wrote:

>Hans,
>
>Do you believe this is just a marketing convention to refer to a file-system
>as an object oriented one? 
>
No.  Also, note that they aren't saying the fs is object oriented, they 
are saying the storage system is not a block device but an object server.

> Bear in mind that Microsoft is supposed to
>enrich their NTFS with object support from SQL Server in their next OS.
>
Reiser4 and MS are talking about different layers than Panassas when we 
talk about being object oriented.  MS objects and Reiser4 plugins might 
be similar notions, I am not an NTFS expert.....

>  The
>code name for this is Longhorn.
>
>I really can't tell what this means to a file system in terms of support.
>It sounds like it's intended to provide support for really large numbers of
>files in a single directory
>
Uh, reiser4 does that without objects having anything to do with it.  
Large numbers of files in a single directory means one should use trees 
not linear searches.....

> vs. nested ones.  But I'm just not sure.  Do you
>see a limitation, going forward, with Reiser4 and large numbers of files?
>  
>
We have trouble with more than 2^60 files in the filesystem. 

;-)

An interesting question though is at how many files in a directory being 
read does dcache start to croak and sys_reiser4 becomes a performance 
necessity....?  Maybe in a few weeks Mr. Demidov can measure that.

>Thanks,
>
>Mike
>
>-----Original Message-----
>From: Hans Reiser [mailto:reiser@namesys.com] 
>Sent: Monday, October 27, 2003 10:23 AM
>To: darren
>Cc: reiserfs-list@namesys.com
>Subject: Re: Object Oriented FS
>
>darren wrote:
>
>  
>
>>allows very high
>>throughput by scaling
>>
>>
>>
>> 
>>
>>    
>>
>Do you know what that  means?  (Seriously, I don't....)
>
>  
>


-- 
Hans

Re: Object Oriented FS

[Andreas Dilger]

On Oct 27, 2003  21:23 +0300, Hans Reiser wrote:
> darren wrote:
> > allows very high throughput by scaling
>
> Do you know what that  means?  (Seriously, I don't....)

Panasas (from what little I have seen of it) looks to be very similar to
Lustre.  Clients do not access disks directly, but rather have a mapping
layer between offsets and actual data on disk (similar to LVM and VM
abstractions of block devices/memory).  Once client has this mapping (very
small in lustre, on the order of a few hundred bytes at most), it
can do IO directly to one or more storage devices hence scaling of
throughput proportional to number of storage targets.

In Lustre at least, the client knows nothing about the physical layout of
blocks in the file on disk, but rather just accesses one or more objects
via object identifyer, offset, length so the actual layout of the on disk
data can change.

Lustre is GPL, don't know about Panasas.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: Object Oriented FS

[Steven Cole]

On Monday 27 October 2003 11:56 am, Andreas Dilger wrote:
> On Oct 27, 2003  21:23 +0300, Hans Reiser wrote:
> > darren wrote:
> > > allows very high throughput by scaling
> >
> > Do you know what that  means?  (Seriously, I don't....)
>
> Panasas (from what little I have seen of it) looks to be very similar to
> Lustre.  Clients do not access disks directly, but rather have a mapping
> layer between offsets and actual data on disk (similar to LVM and VM
> abstractions of block devices/memory).  Once client has this mapping (very
> small in lustre, on the order of a few hundred bytes at most), it
> can do IO directly to one or more storage devices hence scaling of
> throughput proportional to number of storage targets.
>
> In Lustre at least, the client knows nothing about the physical layout of
> blocks in the file on disk, but rather just accesses one or more objects
> via object identifyer, offset, length so the actual layout of the on disk
> data can change.
>
> Lustre is GPL, don't know about Panasas.
>
> Cheers, Andreas

Reading the following, it appears that something _should_ be GPL...

From http://www.panasas.com/object_storage_arch.html

"Panasas is committed to drive the development of standards for the 
object-based storage. Through the ANSI T10 committee, Panasas is working with 
industry leaders including EMC, IBM, Intel, and Seagate to ensure that 
open-source, reference implementations of object-based storage designs are 
available and driven through an open standards process."

From http://www.panasas.com/directflow.html

"For the operating system on client server nodes to directly read and write 
data objects, an installable file system must be deployed. The DirectFLOW 
client installs as a local file system under Linux and is transparent to the 
operating system and applications above it. This installable file system 
manages the striping of data objects across StorageBlades on a per-object 
basis, and unlike standard RAID arrays, has the ability to optimize data 
layout and RAID level by each object. DirectFLOW thereby enables 
high-performance simultaneous data access for large numbers of clients to 
many files."

The fairly large .pdf files available for download seem to be short (2-page), 
prettied-up versions of the information available at the above URLs.

Steven

Re: Object Oriented FS

[lrc1]

Similarly, any comments on Frigate (see

    http://fmg-www.cs.ucla.edu/project-members/tek/papers/coots97.html

) or the old Pick operating system?

Leo.

-----------------------------------------------------------------
University of St Andrews Webmail: http://webmail.st-andrews.ac.uk

Re: ReiserFS v3 + millions of files?

[Todd Lyons]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dan Oglesby wanted us to know:

>The array is used to store what will basically amount to more than one million 
>files with an average size of sixty kilobytes.

If I had to make an educated guess, you're doing something similar to
what I'm doing.  That's all you'll get out of me for now :-)

>During simulations for file writes, I'm seeing write performance begin to drop 
>dramatically after 800,000 files have been stored on the filesystem.

Hmmm, I did some tests that I went up to 300,000 files with an average
size of 20Kbytes.  The end application will be targeted at 6.5 million
files with the same average 20Kbyte size.

For my tests, I compared EXT3, XFS, reiser(tails), and reiser(notail).
The results are at http://www.mrball.net/file%20creation%20tests.html

After reading your post, I'm going to repartition and repeat my tests
with much larger filecounts.

>The filesystem is being mounted with the following options:  
>defaults,notail,noatime,nodiratime

I'll repeat with noatime and nodiratime, but my simple little benchmark
opens(creates), writes, and closes.  Very little affect by my
guestimation will be felt by those two.  Maybe I'm wrong though.

>The filesystem was created with default options, basically a "mkreiserfs /dev/
>sda1".

I made it a point of using slow IDE drives for my tests.  They were done
on a 5400 RPM IDE drive.
- -- 
Blue skies...	Todd 	Public key: http://www.mrball.net/todd.asc
 Favourite shell:  bash, though I also like 'init=/usr/bin/emacs'
                                                --Andrew Tridgell
Linux kernel 2.4.22-10mm.2mdk   1 user,  load average: 0.00, 0.03, 0.06
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: http://www.mrball.net/todd.asc

iD8DBQE/n1hBIBT1264ScBURAhc7AKDkwHfhMXdbkEN14d2FVrwsb1A7YgCgymmN
bFouT/2wnIEA6N3fd4u6q3Q=
=xoll
-----END PGP SIGNATURE-----

Re: ReiserFS v3 + millions of files?

[Hans Reiser]

Todd Lyons wrote:

>
>
> For my tests, I compared EXT3, XFS, reiser(tails), and reiser(notail).
> The results are at http://www.mrball.net/file%20creation%20tests.html
>
>
Is ext3 using htree?

-- 
Hans

Re: ReiserFS v3 + millions of files?

[Todd Lyons]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Reiser wanted us to know:

>>For my tests, I compared EXT3, XFS, reiser(tails), and reiser(notail).
>>The results are at http://www.mrball.net/file%20creation%20tests.html
>Is ext3 using htree?

You know, I don't know the answer to that.  I would guess that it's the
default compilation options, so if you say that it uses that by default,
I'll believe you.  I tried grepping the binary but didn't find anything
that indicated type of structure.

I found this in namei.c.  Dunno if it helps:
 *  Directory entry file type support and forward compatibility hooks
 *      for B-tree directories by Theodore Ts'o (tytso@mit.edu), 1998

To me that doesn't really tell anything though.

Anyway, I fired off an instance of my test program that created 3
million files and it finished in about 26 minutes.  But then I found
that I had filled the partition so I shortened the file size a little
and restarted it.  It was running when I left and I'll analyze the logs
tomorrow.  BTW, that was on a machine with 5400 RPM 40 GB HD and 128
Megs of RAM (trying to starve the system for resources to get rid of
effects of caching and buffering).
- -- 
Blue skies...	Todd 	  Public key: http://www.mrball.net/todd.asc
    They dont need to adjust their pricing, they just need to 
   lobby for new laws to protect their flawed business models. 
              Oh wait, they just did that.         --Dan Hollis
Linux kernel 2.4.22-10mm.2mdk   1 user,  load average: 0.02, 0.01, 0.00
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: http://www.mrball.net/todd.asc

iD8DBQE/oKn4IBT1264ScBURAgDzAJwKxhDmsx4eEsgZk5uQtT3G0yproACfVb4o
LlfB7Z4tO3cNQJo4AqOkl6g=
=zWHs
-----END PGP SIGNATURE-----

Re: ReiserFS v3 + millions of files?

[Andreas Dilger]

On Oct 29, 2003  22:04 -0800, Todd Lyons wrote:
> Hans Reiser wrote:
> > Is ext3 using htree?
> 
> You know, I don't know the answer to that.  I would guess that it's the
> default compilation options, so if you say that it uses that by default,
> I'll believe you.  I tried grepping the binary but didn't find anything
> that indicated type of structure.

Unlikely.  The htree (also called indexed directory) code is not in any
common kernel, although we have been using it extensively for Lustre for
many months without problems.

In the past we did some comparisons between ext3+htree and reiserfs for
creating 10M files in a single directory and for local filesystems both
started at around 25k-30k/s but ext3 started to taper off after about
1M files to under 10k/s (this was on a very large/fast SCSI RAID array)
while reiserfs kept pretty steady at 25k/s for the whole 10M files.

Ext3 was a lot worse initially, but I wrote a patch which avoided reading
empty inode table blocks from disk while it was allocating inodes.  That
patch made it into 2.6 I think, but isn't in stock 2.4.  What also makes
a big difference is the size of your journal, because you can quickly
dirty a lot of blocks as you insert filenames into the directory when
there are a lot of directory blocks.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: ReiserFS v3 + millions of files?

[Dan Oglesby]

> Dan Oglesby writes:
>  > On Monday 27 October 2003 5:42 am, you wrote:
>  > > Dan Oglesby writes:
>  > >  > Greetings...
>  > >  >
>  > >  > Long time ReiserFS user, first time I've had a problem.  Signed up for
>  > >  > the mailing list last week, and was surprised to see so little traffic
>  > >  > (might be a good thing?).
>  > >  >
>  > >  > I'm running Red Hat 7.3 using a Red Hat 2.4.20 kernel.  The system has a
>  > >  > RAID-5 array via 3Ware 7500 controller, and three Western Digital 120GB
>  > >  > "Special Edition" hard drives.  The array is one filesystem, ReiserFS. 
>  > >  > The operating system, swap, and other files are stored on a hard drive
>  > >  > that is on the primary IDE controller off of the motherboard.
>  > >  >
>  > >  > The system is a single board computer, with a P4 3.06 GHz hyperthreaded
>  > >  > processor (kernel is SMP enabled), 512MB of RAM, and contains a mix of
>  > >  > ReiserFS and EXT2 filesystems on the primary drive (ReiserFS only on the
>  > >  > array).  No NFS.
>  > >  >
>  > >  > The array is used to store what will basically amount to more than one
>  > >  > million files with an average size of sixty kilobytes.
>  > >  >
>  > >  > During simulations for file writes, I'm seeing write performance begin
>  > >  > to drop dramatically after 800,000 files have been stored on the
>  > >  > filesystem.
>  > >
>  > > Reiserfs (both v3 and v4) stores directory entries (names within
>  > > directory) sorted by a hash of the file name. If files are created in
>  > > the "random" order, that is, if hashes of names aren't more or less
>  > > monotonic, reiserfs will have to modify the same block many times during
>  > > insertion of large number of files. As blocks with names stop fitting
>  > > into memory this means that the same block has to be fetched many times
>  > > from the disk.
>  > >
>  > > To confirm that this is really what the cause of your problem, please
>  > > answer following questions:
>  > >
>  > > 1. are all your files in the same directory?
>  > 
>  > For now, yes.
>  > 
>  > > 2. how names of files are generated?
>  > 
>  > Filenames are very long, based on site ID, internal codes, date and time.  A 
>  > generic example would be:
> 
> [I would prefer to have this discussion continued on the our mailing
> list (Reiserfs mail-list <Reiserfs-List@Namesys.COM>), so if you don't
> object, please CC reply there.]

Sorry about that.  I must have missed the headers on one of my replies.

> 
>  > 
>  > 		siteID.1234.4321.20031028123456789.ext
> 
> I wasn't precise enough in forming the question. How file names are
> changing through time? That is, how these sequences of digits above
> depend on time? I can guess that "20031028" is a date, but what about
> other parts?
> 

Basically, the siteID doesn't change, and I don't believe the next two 
sections ("1234.4321" in the example above) don't change much either. 
The only part of the filename that changes constantly is the date/time 
section, just before the end.

> Can you modify file name patterns so that name's initial prefix
> increases (in lexicographical order) through time? Like this:
> 
> YYYYMMDDHHmmss.sequential-no.siteID.rest.ext
> 
> this should get rid of, or at least alleviate the problem you have
> described.
> 

I don't see why that would be a problem.  I'll present this option to 
the developers today, and will hopefully be testing very soon.

>  > 
>  > Most files are that size or several characters longer.
>  > 	
>  > > 3. what hash are using on reiserfs (default is r5).
>  > >
>  > 
>  > Default (r5).
>  > 
>  > > Reiserfs has another problem due to its limited capability of handling
>  > > hash collisions in file names. Reiser4 scales much better in this
>  > > respect, and generally, works fine with scores of millions of files in
>  > > one directory.
>  > >
>  > 
>  > Sounds like you guys/gals are going to have another person willing to run the 
>  > current incarnation of Reiser4 on a test machine in the very near future.
> 
> Reiser4 is not yet ready for production. Only use it to manipulate data
> that can be recovered by other mean

I know Reiser4 isn't ready for production, but I wouldn't mind checking 
out the performance differences on a test machine.

Thanks for the info...  I'll see what happens when I apply your theory 
to one of our test machines.

--Dan