what do you do that stresses your filesystem?

what do you do that stresses your filesystem?

[Hans Reiser]

We were discussing how to optimize reiser4 best, and came to realize 
that us developers did not have a good enough intuition for what users 
do that stresses their filesystem enough that they care about its 
performance.

If you just do edits of files it probably does not matter too much what 
fs you use.

Booting the machine seems like one activity that many users end up 
waiting on the FS for.  Yes?

Starting up complex and big applications like xemacs and mozilla would 
be another.  Yes?

Others?

Hans

PS

reiser4 performance is up a lot recently, and within two weeks I think 
cp -r will have been optimized as much as is worth doing.  cp -r 
accesses files in readdir order, and that does indeed seem worth 
optimizing, but soon we will need to optimize more sophisticated access 
patterns than that.....

Hans

Re: what do you do that stresses your filesystem?

[Anders Widman]

> We were discussing how to optimize reiser4 best, and came to realize 
> that us developers did not have a good enough intuition for what users 
> do that stresses their filesystem enough that they care about its 
> performance.

> If you just do edits of files it probably does not matter too much what 
> fs you use.

depends  on  how  large  edits  =)..  Like  video-editing requires big
r/w performance.

> Booting the machine seems like one activity that many users end up
> waiting on the FS for.  Yes?

Hopefully we do not have to reboot to often... :)

> Starting up complex and big applications like xemacs and mozilla would 
> be another.  Yes?

True.. Many X-applications are reading much from disk...

> Others?

File-serving with big directories to multiple users... with both reads
and writes over Samba or NFS.

//Anders


> Hans

> PS

> reiser4 performance is up a lot recently, and within two weeks I think 
> cp -r will have been optimized as much as is worth doing.  cp -r 
> accesses files in readdir order, and that does indeed seem worth 
> optimizing, but soon we will need to optimize more sophisticated access 
> patterns than that.....

> Hans

Re: what do you do that stresses your filesystem?

[Hans Reiser]

Anders Widman wrote:

>  
>
>>Booting the machine seems like one activity that many users end up
>>waiting on the FS for.  Yes?
>>    
>>
>
>Hopefully we do not have to reboot to often... :)
>
>  
>
buying the seagate hard drives with fluid ballbearings (and fanless 
heatsinks) reduces the need to halt at bedtime by a lot.....

Hans

Re: what do you do that stresses your filesystem?

[Ed Tomlinson]

On December 23, 2002 06:45 am, Hans Reiser wrote:
> Anders Widman wrote:
> >>Booting the machine seems like one activity that many users end up
> >>waiting on the FS for.  Yes?
> >
> >Hopefully we do not have to reboot to often... :)
>
> buying the seagate hard drives with fluid ballbearings (and fanless
> heatsinks) reduces the need to halt at bedtime by a lot.....

Various (pulls, clones) bk operations using the 2.5 kernel tree.

Ed Tomlinson

Re: what do you do that stresses your filesystem?

[Hans Reiser]

Ed Tomlinson wrote:

>On December 23, 2002 06:45 am, Hans Reiser wrote:
>  
>
>>Anders Widman wrote:
>>    
>>
>>>>Booting the machine seems like one activity that many users end up
>>>>waiting on the FS for.  Yes?
>>>>        
>>>>
>>>Hopefully we do not have to reboot to often... :)
>>>      
>>>
>>buying the seagate hard drives with fluid ballbearings (and fanless
>>heatsinks) reduces the need to halt at bedtime by a lot.....
>>    
>>
>
>Various (pulls, clones) bk operations using the 2.5 kernel tree.
>
>Ed Tomlinson
>
>
>
>  
>
Isn't that CPU and memory bound?

Re: what do you do that stresses your filesystem?

[Dieter Nützel]

Am Montag, 23. Dezember 2002 12:37 schrieb Anders Widman:
> > We were discussing how to optimize reiser4 best, and came to realize
> > that us developers did not have a good enough intuition for what users
> > do that stresses their filesystem enough that they care about its
> > performance.
> >
> > If you just do edits of files it probably does not matter too much what
> > fs you use.
>
> depends  on  how  large  edits  =)..  Like  video-editing requires big
> r/w performance.

Like Andrew Morten suggest all the time (deadline IO tests):

Running (heavy) constant (streaming) IO in the background during "normal" IO 
(20-40 parallel gcc/g++ compilers (make -j20) in my case).

WMs e.g. KDE 3.1 isn't a case any longer...;-)

Give your "workers" a break.
			Merry Christmas!

-- 
Dieter Nützel
Graduate Student, Computer Science

University of Hamburg
Department of Computer Science
@home: Dieter.Nuetzel at hamburg.de (replace at with @)

new reiserfs4 snapshots?

[Ookhoi]

Hans Reiser wrote (ao):
> We were discussing how to optimize reiser4 best, and came to realize
> that us developers did not have a good enough intuition for what users
> do that stresses their filesystem enough that they care about its
> performance.
> 
> If you just do edits of files it probably does not matter too much
> what fs you use.
> 
> Booting the machine seems like one activity that many users end up
> waiting on the FS for.  Yes?
> 
> Starting up complex and big applications like xemacs and mozilla would
> be another.  Yes?
> 
> Others?

find?
(un)tar sources/backups/etc?
mail? (mbox/maildir; large mailboxes)

> Hans
> 
> PS
> 
> reiser4 performance is up a lot recently, and within two weeks I think
> cp -r will have been optimized as much as is worth doing.  cp -r
> accesses files in readdir order, and that does indeed seem worth
> optimizing, but soon we will need to optimize more sophisticated
> access patterns than that.....

Can you please make new reiserfs4 snapshots available for testing? The
last one is from December 4.

I tried to pull the linux kernel bk and the reiserfs ones, got a kernel
that compiled and booted, but the fs went bellyup pretty fast. I don't
think a report is useful as the bk pulls are moving targets. Am I right?

Also I'm not sure I did it the right way as I couldn't find it
documented. Can you please point me to some documentation or provide
quick instructions in a reply?

Thank you and the team for these great filesystems.

Re: what do you do that stresses your filesystem?

[Hendrik Visage]

On Mon, Dec 23, 2002 at 02:28:32PM +0300, Hans Reiser wrote:
> We were discussing how to optimize reiser4 best, and came to realize 
> that us developers did not have a good enough intuition for what users 
> do that stresses their filesystem enough that they care about its 
> performance.


> Starting up complex and big applications like xemacs and mozilla would 
> be another.  Yes?

kernel/mozilla/KDE3 recompiles :)

Seriously though, those are massive packages that causes lots of I/Os.
The Lunar/Sorcerer/SourceMage solution was a tmpfs to extract the
sources to, and compile in the tmpfs. Thus, it would be interesting to 
compare the two scenarios with each other.

Another slowness (especially with reiserfs3 without the notails option) 
is "dd if=/dev/zero of=/swapfile bs=1k count=1024k"

Hendrik

Re: what do you do that stresses your filesystem?

[Oleg Drokin]

Hello!

On Mon, Dec 23, 2002 at 02:03:57PM +0200, Hendrik Visage wrote:

> Another slowness (especially with reiserfs3 without the notails option) 
> is "dd if=/dev/zero of=/swapfile bs=1k count=1024k"

Well, I'd say that artrifical limiting of amount of data to be transferred
is a cheating ;).
Reiser4 is not vulnerable to this, though.

Bye,
    Oleg

Re: what do you do that stresses your filesystem?

[Hans Reiser]

Oleg Drokin wrote:

>Hello!
>
>On Mon, Dec 23, 2002 at 02:03:57PM +0200, Hendrik Visage wrote:
>
>  
>
>>Another slowness (especially with reiserfs3 without the notails option) 
>>is "dd if=/dev/zero of=/swapfile bs=1k count=1024k"
>>    
>>
>
>Well, I'd say that artrifical limiting of amount of data to be transferred
>is a cheating ;).
>Reiser4 is not vulnerable to this, though.
>
>Bye,
>    Oleg
>
>
>  
>
can you say more about performance of v3 and v4 at this task?

I suppose I shouldn't ask Hendrik how often he does this in the course 
of his usual usage patterns....;-)

Re: what do you do that stresses your filesystem?

[Hendrik Visage]

On Mon, Dec 23, 2002 at 03:12:49PM +0300, Hans Reiser wrote:
> >>Another slowness (especially with reiserfs3 without the notails option) 
> >>is "dd if=/dev/zero of=/swapfile bs=1k count=1024k"
> >>   
> >>
> >
> >Well, I'd say that artrifical limiting of amount of data to be transferred
> >is a cheating ;).

Well... It's not that people needs unlimited swapspace ;^)

> I suppose I shouldn't ask Hendrik how often he does this in the course 
> of his usual usage patterns....;-)

It does add a "too long" wait during installation processes, which happens
about once in a fortnight, (testing some stuff, fresh installs easier ;^)

Hendrik

Re: what do you do that stresses your filesystem?

[Oleg Drokin]

Hello!

On Mon, Dec 23, 2002 at 02:21:10PM +0200, Hendrik Visage wrote:
> > >>Another slowness (especially with reiserfs3 without the notails option) 
> > >>is "dd if=/dev/zero of=/swapfile bs=1k count=1024k"
> > >Well, I'd say that artrifical limiting of amount of data to be transferred
> > >is a cheating ;).
> Well... It's not that people needs unlimited swapspace ;^)

I mean limiting of blocksize.

Bye,
    Oleg

Re: what do you do that stresses your filesystem?

[Hendrik Visage]

On Mon, Dec 23, 2002 at 03:26:40PM +0300, Oleg Drokin wrote:
> Hello!
> 
> On Mon, Dec 23, 2002 at 02:21:10PM +0200, Hendrik Visage wrote:
> > > >>Another slowness (especially with reiserfs3 without the notails option) 
> > > >>is "dd if=/dev/zero of=/swapfile bs=1k count=1024k"
> > > >Well, I'd say that artrifical limiting of amount of data to be transferred
> > > >is a cheating ;).
> > Well... It's not that people needs unlimited swapspace ;^)
> 
> I mean limiting of blocksize.

The only reason for the bs= option is to have a defined size for the
swapfile, ie. bs*count=swapsize. I could've changed the line to
"dd if=/dev/zero of=/swapfile bs=1024k count=1k"

In any case, it's just to make sure, if however, I knew that a specific
set of ibs/obs/bs options would improve speed, I'd use them.

Hendrik

Re: what do you do that stresses your filesystem?

[Oleg Drokin]

Hello!

On Mon, Dec 23, 2002 at 02:37:18PM +0200, Hendrik Visage wrote:
> > > > >>Another slowness (especially with reiserfs3 without the notails option) 
> > > > >>is "dd if=/dev/zero of=/swapfile bs=1k count=1024k"
> > > > >Well, I'd say that artrifical limiting of amount of data to be transferred
> > > > >is a cheating ;).
> > > Well... It's not that people needs unlimited swapspace ;^)
> > I mean limiting of blocksize.
> The only reason for the bs= option is to have a defined size for the
> swapfile, ie. bs*count=swapsize. I could've changed the line to
> "dd if=/dev/zero of=/swapfile bs=1024k count=1k"

My measurement show that I save 2+ seconds of system time by doing this.
Should be even more of savings after reiserfs_file_write() will be accepted
into main kernel. Almost no effect on real time for me, though. (~30 sec)

Bye,
    Oleg

Re: what do you do that stresses your filesystem?

[Eric Whiting]

Hans Reiser wrote:

> >>Another slowness (especially with reiserfs3 without the notails option)
> >>is "dd if=/dev/zero of=/swapfile bs=1k count=1024k"
>
> I suppose I shouldn't ask Hendrik how often he does this in the course
> of his usual usage patterns....;-)

The above dd command (with different values of bs) is one of the first tests I
do when I bring up a RAID array -- under solaris9 or linux. It is a good way to
measure streaming throughput. 

As for other ways to artificially stress the filesystem I sometimes run multiple
concurrent 'while(true);do bonnie -s N;done' loops with different values for N. 

As far as real life stressing of the filesystem -- I think kernel compiles and
just letting the users hit the filesystem are the main high stress fs loads. My
filesystems get a lot of rsync activity as well. 

eric

Re: what do you do that stresses your filesystem?

[Russell Coker]

On Mon, 23 Dec 2002 12:28, Hans Reiser wrote:
> We were discussing how to optimize reiser4 best, and came to realize
> that us developers did not have a good enough intuition for what users
> do that stresses their filesystem enough that they care about its
> performance.
>
> Booting the machine seems like one activity that many users end up
> waiting on the FS for.  Yes?

Yes, although there are other factors than the file system which determine 
boot time.  Different types of init scripts can probably improve performance 
more than anything you could do to a file system.

Also if you have a rack-mount server then it'll probably take 2 minutes or 
more to go through the BIOS checks so FS delays will seem like nothing by 
comparison.

> Starting up complex and big applications like xemacs and mozilla would
> be another.  Yes?

Yes, particularly KDE.  Although I don't do it often enough to care so much.

> Others?

Mail serving with Maildir storage.  Rsync of gigebytes of files.  Kernel 
compiles.  Debian package installation.

Also there are some issues where Linux does not perform well, such as typing 
"sync" on a machine that has a heavy write load (which can take an indefinate 
amount of time).

-- 
http://www.coker.com.au/bonnie++/     Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/       Postal SMTP/POP benchmark
http://www.coker.com.au/projects.html Projects I am working on
http://www.coker.com.au/~russell/     My home page

Re: what do you do that stresses your filesystem?

[Alexander G. M. Smith]

Russell Coker wrote on Mon, 23 Dec 2002 14:21:52 +0100:
> Mail serving with Maildir storage.

Same sort of thing here.  Thousands of e-mail and news files in
a directory, so displaying them in a GUI directory display window
can be slow.  Particularly if it is also getting attributes to
display (from, to, subject, date, thread).  This is under BeOS/BFS
and can get a bit slow if the attributes end up in disk sectors
far away from the inode for the file (but speed is good when block
size (== inode size) is boosted up enough so that the attributes
all fit in the inode).

So, directory reading speed plus attribute reading speed are
important to me.

- Alex

Re: what do you do that stresses your filesystem?

[Hans Reiser]

Russell Coker wrote:

>On Mon, 23 Dec 2002 12:28, Hans Reiser wrote:
>  
>
>>We were discussing how to optimize reiser4 best, and came to realize
>>that us developers did not have a good enough intuition for what users
>>do that stresses their filesystem enough that they care about its
>>performance.
>>
>>Booting the machine seems like one activity that many users end up
>>waiting on the FS for.  Yes?
>>    
>>
>
>Yes, although there are other factors than the file system which determine 
>boot time.  Different types of init scripts can probably improve performance 
>more than anything you could do to a file system.
>
>Also if you have a rack-mount server then it'll probably take 2 minutes or 
>more to go through the BIOS checks so FS delays will seem like nothing by 
>comparison.
>
I mostly meant waiting for the fs to read all the binaries that get 
executed at startup, not just the journal replay time.

Re: what do you do that stresses your filesystem?

[Ookhoi]

Hans Reiser wrote (ao):
> I mostly meant waiting for the fs to read all the binaries that get
> executed at startup, not just the journal replay time.

So you run redhat eh ;-)

Can you please make a reiserfs4 snapshot available for testing before
you all enjoy a nice Christmas?

Re: what do you do that stresses your filesystem?

[Oleg Drokin]

Hello!

On Tue, Dec 24, 2002 at 11:15:27AM +0100, Ookhoi wrote:

> Can you please make a reiserfs4 snapshot available for testing before
> you all enjoy a nice Christmas?

Russian (hm, should I say Greek Orthodox here?) Christmas is after New Year,
anyway ;)

We will release new snapshot before New Year, I think.

Bye,
    Oleg

Re: what do you do that stresses your filesystem?

[Ookhoi]

Oleg Drokin wrote (ao):
> On Tue, Dec 24, 2002 at 11:15:27AM +0100, Ookhoi wrote:
> > Can you please make a reiserfs4 snapshot available for testing
> > before you all enjoy a nice Christmas?
> 
> Russian (hm, should I say Greek Orthodox here?) Christmas is after New
> Year, anyway ;)

Owh, I'm sorry :-]  Guess I need to read up on that one.

> We will release new snapshot before New Year, I think.

Great, thanks!

Re: what do you do that stresses your filesystem?

[Hans Reiser]

Ookhoi wrote:

>Hans Reiser wrote (ao):
>  
>
>>I mostly meant waiting for the fs to read all the binaries that get
>>executed at startup, not just the journal replay time.
>>    
>>
>
>So you run redhat eh ;-)
>
>Can you please make a reiserfs4 snapshot available for testing before
>you all enjoy a nice Christmas?
>
>
>  
>
I pushed a little more on this... you'll get something before western Xmas.

Re: what do you do that stresses your filesystem?

[Russell Coker]

On Tue, 24 Dec 2002 10:26, Hans Reiser wrote:
> >Also if you have a rack-mount server then it'll probably take 2 minutes or
> >more to go through the BIOS checks so FS delays will seem like nothing by
> >comparison.
>
> I mostly meant waiting for the fs to read all the binaries that get
> executed at startup, not just the journal replay time.

The time taken to read the binaries is probably about 2 seconds on one of my 
servers.  The time taken to go through the BIOS part of the booting is a 
minimum of 30s and up to 5m!

If you get a rack-mount server from any of the major vendors you can only boot 
it once a year if you want to get close to five-nines.  Reboot every 10 
months and the BIOS time alone will take you down to four-nines.

-- 
http://www.coker.com.au/bonnie++/     Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/       Postal SMTP/POP benchmark
http://www.coker.com.au/projects.html Projects I am working on
http://www.coker.com.au/~russell/     My home page

Re: what do you do that stresses your filesystem?

[bscott]

On Mon, 23 Dec 2002, at 2:28pm, reiser@namesys.com wrote:
> We were discussing how to optimize reiser4 best, and came to realize that
> us developers did not have a good enough intuition for what users do that
> stresses their filesystem enough that they care about its performance.

  Define "users".

  Even a moderately large mailserver will have many simultaneous opens,
reads, rewrites, creates, unlinks, and so on happening at once.  Depending
on the type of mailbox storage format in use, the patterns are slightly
different.  Traditional BSD mbox does a lot of sequential file reading, does
appends on new messages, and typically rewrites the whole file when deleted
messages are purged.  The maildir and mh formats, on the other hand, have
many smaller files.  Reading the mail index involves lots of open/read-one-
-or-two-blocks/close operations in sequence.  New messages creates files;
purging old ones deletes them.

-- 
Ben Scott <bscott@ntisys.com>
| The opinions expressed in this message are those of the author and do not |
| necessarily represent the views or policy of any other person, entity or  |
| organization.  All information is provided without warranty of any kind.  |

Re: what do you do that stresses your filesystem?

[Ross Vandegrift]

On Mon, Dec 23, 2002 at 02:28:32PM +0300, Hans Reiser wrote:
> Booting the machine seems like one activity that many users end up 
> waiting on the FS for.  Yes?

Not really.  I've been running ReiserFS for at least three or four
years.  I haven't seen a machine have to fsck in forever ::-)

On machines with Linux md RAID arrays that need to be remirrored, 
I do end up waiting a bit (10-30 seconds) due to read starvation, but
this problem is so much better than it used to be it's hardly worth
mentioning.

> Starting up complex and big applications like xemacs and mozilla would 
> be another.  Yes?

Not really - I've found that apps like mozilla are usually loaded from
disk exactly once per reboot, and then stay cached in memory until the
box is restarted.  Most of the startup time comes from the app doing
lots of work itself - subsequent starts show no disk activity at all.

> Others?

As a long time user of ReiserFS, I don't really wait on the filesystem
anymore.  With Ext2, I used to wait on large deletes, but that's clearly
not an issue anymore.  I've noticed that when I need a performance
upgrade, a faster disk is the best solution.

-- 
Ross Vandegrift
ross@willow.seitz.com

A Pope has a Water Cannon.                               It is a Water Cannon.
He fires Holy-Water from it.                        It is a Holy-Water Cannon.
He Blesses it.                                 It is a Holy Holy-Water Cannon.
He Blesses the Hell out of it.          It is a Wholly Holy Holy-Water Cannon.
He has it pierced.                It is a Holey Wholly Holy Holy-Water Cannon.
He makes it official.       It is a Canon Holey Wholly Holy Holy-Water Cannon.
Batman and Robin arrive.                                       He shoots them.

Re: what do you do that stresses your filesystem?

[Russell Coker]

On Mon, 23 Dec 2002 16:56, Ross Vandegrift wrote:
> On machines with Linux md RAID arrays that need to be remirrored,
> I do end up waiting a bit (10-30 seconds) due to read starvation, but
> this problem is so much better than it used to be it's hardly worth
> mentioning.

One thing I've done before is configure the boot scripts to set the RAID 
re-sync speed very low before fsck/mount time and then set it back later.  It 
didn't seem to do as much good as I thought it would though.

-- 
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/    Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page

Re: what do you do that stresses your filesystem?

[Chris Haynes]

Hans,

Thanks for asking!

I'm about to deploy an on-line transaction-based service.. All
service-specific software is in 1000+ Java classes. The OS is SuSE
7.3, and Java version 1.4

The heavist uses of the file store are:

During development:
+ javadoc

During service deployment / update:
+ rsync
+ When using a tripwire-like Java program which checks the SHA digest
of all deployable files against zipped JARs

During production operation:
+ atomic, synchronized writes to multiple files (typically 3 - 4 files
in different directories, the first is the creation of a new 4 kb
file, others are usually updates of existing files - growing typically
1kb per update). This files system is mounted on a RAID-1 pair.
+ rsync
+ Successive reading of all files in a directory sub-tree (up to 10M
files)
in filestore-defined order (i.e. the program makes no demands or
assumptions about the order - it uses the order supplied by Java's
File.files()).


The greatest performance concern I have is with the file writes. As
these are atomic transactions, I use a separate thread for each file's
write (to give the kernel's escalator a chance to work), and require
that the write operations be individually hardware-synchronized  using
Java's FileDescriptor.sync() method. I then use a counter to detect
when all threads have reported that their files have been written -
this indicating successful commitment of the transaction.

I handle read-and write-locking in the application.

Usually, there are no lock conflicts, so there can be many concurrent
transaction commitments. I use a thread pool of  50 threads to handle
the individual file writes (the 50 being a guess at the likely point
of diminishing returns).

My expectation/hope is that, so long as there are enough threads
available in this pool, all transactions will be completed within one
disk rotation period(regardless of the number of concurrent
transactions or number of files per transaction and the fact that I'm
using software RAID-1). I've not yet been able to validate this
(theoretically or practically).

I would *really* like  to be able to group all the file writes for a
transaction  into a single logical  API call and have the kernel/file
system report successful completion of all data and metadata aspects
of the transaction using a single application thread.

HTH

Chris Haynes

Re: what do you do that stresses your filesystem?

[Hans Reiser]

Chris Haynes, maybe as we bring atomic transactions into use we should 
work with you to see if they address your needs.

Chris Mason, maybe you might want to comment on what he is doing with 
threads and how it interacts with V3 journal commits?  (I bet you are on 
vacation though?)

Hans


Chris Haynes wrote:

>Hans,
>
>Thanks for asking!
>
>I'm about to deploy an on-line transaction-based service.. All
>service-specific software is in 1000+ Java classes. The OS is SuSE
>7.3, and Java version 1.4
>
>The heavist uses of the file store are:
>
>During development:
>+ javadoc
>
>During service deployment / update:
>+ rsync
>+ When using a tripwire-like Java program which checks the SHA digest
>of all deployable files against zipped JARs
>
>During production operation:
>+ atomic, synchronized writes to multiple files (typically 3 - 4 files
>in different directories, the first is the creation of a new 4 kb
>file, others are usually updates of existing files - growing typically
>1kb per update). This files system is mounted on a RAID-1 pair.
>+ rsync
>+ Successive reading of all files in a directory sub-tree (up to 10M
>files)
>in filestore-defined order (i.e. the program makes no demands or
>assumptions about the order - it uses the order supplied by Java's
>File.files()).
>
>
>The greatest performance concern I have is with the file writes. As
>these are atomic transactions, I use a separate thread for each file's
>write (to give the kernel's escalator a chance to work), and require
>that the write operations be individually hardware-synchronized  using
>Java's FileDescriptor.sync() method. I then use a counter to detect
>when all threads have reported that their files have been written -
>this indicating successful commitment of the transaction.
>
>I handle read-and write-locking in the application.
>
>Usually, there are no lock conflicts, so there can be many concurrent
>transaction commitments. I use a thread pool of  50 threads to handle
>the individual file writes (the 50 being a guess at the likely point
>of diminishing returns).
>
>My expectation/hope is that, so long as there are enough threads
>available in this pool, all transactions will be completed within one
>disk rotation period(regardless of the number of concurrent
>transactions or number of files per transaction and the fact that I'm
>using software RAID-1). I've not yet been able to validate this
>(theoretically or practically).
>
>I would *really* like  to be able to group all the file writes for a
>transaction  into a single logical  API call and have the kernel/file
>system report successful completion of all data and metadata aspects
>of the transaction using a single application thread.
>
>HTH
>
>Chris Haynes
>
>
>
>
>  
>

Re: what do you do that stresses your filesystem?

[Chris Mason]

On Tue, 2002-12-24 at 04:36, Hans Reiser wrote:

> Chris Haynes wrote:
> >
> >I'm about to deploy an on-line transaction-based service.. All
> >service-specific software is in 1000+ Java classes. The OS is SuSE
> >7.3, and Java version 1.4

You'll get better performance from the suse update kernels (or anything
with the data logging patches.

> >
> >The heavist uses of the file store are:
> >
> >During development:
> >+ javadoc
> >
> >During service deployment / update:
> >+ rsync
> >+ When using a tripwire-like Java program which checks the SHA digest
> >of all deployable files against zipped JARs
> >
> >During production operation:
> >+ atomic, synchronized writes to multiple files (typically 3 - 4 files
> >in different directories, the first is the creation of a new 4 kb
> >file, others are usually updates of existing files - growing typically
> >1kb per update). This files system is mounted on a RAID-1 pair.

Is this the only type of write being done?  If so, mounting with
data=journal will give you a pretty big boost.  Regardless of which data
mode you use, the io pattern you want to try for looks like this:

write(file1)
write(file2)
write(fileX) ...
fsync(fileX)
fsync(fileX-1) ...
fsync(file1)

This allows for the biggest transaction before the fsync comes in and
forces a commit.  By doing an fsync on the newest file first, you
greatly increase the chances that all the transactions for all the old
files will be committed by the fsync(fileX).  When this happens, the
rest of the fsyncs just trigger writes on the data blocks.

If you are using data=journal, the rest of the fsyncs become noops,
since the fsync(fileX) will also commit all the data blocks of all the
previous writes.

If there are other types of writes going on (large non-synchronous
writes), data=journal will hurt performance because it involves writing
all the data blocks twice.  In this case your best bet will be a
dedicated logging device.

If the synchronous writes for a single transaction are the only writes
to the FS, and you are doing writes to many different files, you can
also just do all the writes/creates, and then run sync().

This will get all the data block writes scheduled at once, and then
write to the log.

> >+ rsync
> >+ Successive reading of all files in a directory sub-tree (up to 10M
> >files)
> >in filestore-defined order (i.e. the program makes no demands or
> >assumptions about the order - it uses the order supplied by Java's
> >File.files()).
> >
> >
> >The greatest performance concern I have is with the file writes. As
> >these are atomic transactions, I use a separate thread for each file's
> >write (to give the kernel's escalator a chance to work), and require
> >that the write operations be individually hardware-synchronized  using
> >Java's FileDescriptor.sync() method. I then use a counter to detect
> >when all threads have reported that their files have been written -
> >this indicating successful commitment of the transaction.
> >
> >I handle read-and write-locking in the application.
> >
> >Usually, there are no lock conflicts, so there can be many concurrent
> >transaction commitments. I use a thread pool of  50 threads to handle
> >the individual file writes (the 50 being a guess at the likely point
> >of diminishing returns).
> >
> >My expectation/hope is that, so long as there are enough threads
> >available in this pool, all transactions will be completed within one
> >disk rotation period(regardless of the number of concurrent
> >transactions or number of files per transaction and the fact that I'm
> >using software RAID-1). I've not yet been able to validate this
> >(theoretically or practically).

It won't happen, you've got a chance in data=journal mode, but otherwise
there will be seeks to and from the log area as you write and wait on
the data blocks and the log blocks.

> >
> >I would *really* like  to be able to group all the file writes for a
> >transaction  into a single logical  API call and have the kernel/file
> >system report successful completion of all data and metadata aspects
> >of the transaction using a single application thread.
> >

The kernel doesn't have this right now, but the aio code in 2.5.x is
close.  

-chris

Re: what do you do that stresses your filesystem?

[Matthew Johnson]

Some others I can think of...

Running Wine and winex applications

Running games, such as ut2003. I don't mean card games. Running game servers 
too...

Starting large Window managers like KDE.

Mat

On Monday 23 December 2002 03:28 am, Hans Reiser wrote:
> We were discussing how to optimize reiser4 best, and came to realize
> that us developers did not have a good enough intuition for what users
> do that stresses their filesystem enough that they care about its
> performance.
>
> If you just do edits of files it probably does not matter too much what
> fs you use.
>
> Booting the machine seems like one activity that many users end up
> waiting on the FS for.  Yes?
>
> Starting up complex and big applications like xemacs and mozilla would
> be another.  Yes?
>
> Others?
>
> Hans
>
> PS
>
> reiser4 performance is up a lot recently, and within two weeks I think
> cp -r will have been optimized as much as is worth doing.  cp -r
> accesses files in readdir order, and that does indeed seem worth
> optimizing, but soon we will need to optimize more sophisticated access
> patterns than that.....
>
> Hans

Re: what do you do that stresses your filesystem?

[Manuel Krause]

On 12/23/2002 12:28 PM, Hans Reiser wrote:
> We were discussing how to optimize reiser4 best, and came to realize 
> that us developers did not have a good enough intuition for what users 
> do that stresses their filesystem enough that they care about its 
> performance.
> 
> If you just do edits of files it probably does not matter too much what 
> fs you use.
> 
> Booting the machine seems like one activity that many users end up 
> waiting on the FS for.  Yes?
> 
> Starting up complex and big applications like xemacs and mozilla would 
> be another.  Yes?
> 
> Others?

Anyone already mentioned VMware sessions? ;-)

Running VMware on a Linux host would be an example. The application 
itself is no problem for the fs. But maybe the case when the VMware's 
guest OS is under high memory usage and disk i/o and VMware swaps out to 
a file in /tmp on the host OS. Usually this happens at the same time 
when Linux swaps out the "unneeded" things, on here: parts of KDE and 
Netscape7+.
So it is no fs only stress test at all. I can suggest running SpeedDisk 
on your fragmented Win98 disks from within VMware to have this scenario. 
(Then, switching to a mail composer window under Linux and waiting for 
the cursor ... takes ... some time...)

Regards,

Manuel

> 
> Hans
> 
> PS
> 
> reiser4 performance is up a lot recently, and within two weeks I think 
> cp -r will have been optimized as much as is worth doing.  cp -r 
> accesses files in readdir order, and that does indeed seem worth 
> optimizing, but soon we will need to optimize more sophisticated access 
> patterns than that.....
> 
> Hans

Re: what do you do that stresses your filesystem?

[Andrew Clausen]

On Mon, Dec 23, 2002 at 02:28:32PM +0300, Hans Reiser wrote:
> We were discussing how to optimize reiser4 best, and came to realize 
> that us developers did not have a good enough intuition for what users 
> do that stresses their filesystem enough that they care about its 
> performance.

Something else you might want to do is measure "commit latency"...
the time between file operations being made by userland, and those
operations becoming durable (i.e. getting committed into the log).

If you can benchmark this, it might give users more confidence.
(When I clicked save, it did, actually, save!)

This is kind of related to real-time stuff, which is another thing
you might want to test.  Eg: audio/video playback... can reiserfs
sustain IO bandwidth well, without falling behind occasionally?
(i.e. maintaining low latency)

XFS has (had?  I think it's only the irix port) an ioctl to tell
the OS "I want to read 300kb/s with maximum 0.1ms latency on fd 3".
(The ioctl() would fail if Irix couldn't guarantee that)

I'm not saying this is useful for everyone, just it should help
you understand the performance better...

Cheers,
Andrew

Re: what do you do that stresses your filesystem?

[Brian Tinsley]

I'm late replying to this one, but I've been on vacation :)

Any possible optimizations for the following would be beneficial:
  * interaction with the Linux NFS server
  * SMP machines
  * apps like grep, awk, and sed

Any optimizations for the following application characteristics would be 
beneficial:

  * Our applications perform multi-threaded streaming I/O 
(network-to-disk and vice versa) with read/write block sizes varying 
from 16KB to 64KB; we have files ranging in size from 32KB to 500MB+ 
(this top end will likely grow into several GB in the near future).
  * Our directory structure is very broad, but not very deep and files 
are stored in leaf nodes only.
  * When a file is created and completely written to disk, it is often 
re-read in its entirety at least two times within a couple of minutes.
  * Our database app performs lots of block-based random I/O and fsyncs 
within a dozen or so large files.



Hans Reiser wrote:

> We were discussing how to optimize reiser4 best, and came to realize 
> that us developers did not have a good enough intuition for what users 
> do that stresses their filesystem enough that they care about its 
> performance.
>
> If you just do edits of files it probably does not matter too much 
> what fs you use.
>
> Booting the machine seems like one activity that many users end up 
> waiting on the FS for.  Yes?
>
> Starting up complex and big applications like xemacs and mozilla would 
> be another.  Yes?
>
> Others?
>
> Hans
>
> PS
>
> reiser4 performance is up a lot recently, and within two weeks I think 
> cp -r will have been optimized as much as is worth doing.  cp -r 
> accesses files in readdir order, and that does indeed seem worth 
> optimizing, but soon we will need to optimize more sophisticated 
> access patterns than that.....
>
> Hans


-- 

-[========================]-
-[      Brian Tinsley     ]-
-[ Chief Systems Engineer ]-
-[        Emageon         ]-
-[========================]-

Re: what do you do that stresses your filesystem?

[Hans Reiser]

Brian Tinsley wrote:

> I'm late replying to this one, but I've been on vacation :)
>
> Any possible optimizations for the following would be beneficial:
>  * interaction with the Linux NFS server 

Yes, agreed, especially 0 copy stuff.

>
>  * SMP machines 

already addressed by reiser4 (but not tested and benchmarked yet)

>
>  * apps like grep, awk, and sed 

reasonable.

>
>
> Any optimizations for the following application characteristics would 
> be beneficial:
>
>  * Our applications perform multi-threaded streaming I/O 
> (network-to-disk and vice versa) with read/write block sizes varying 
> from 16KB to 64KB; we have files ranging in size from 32KB to 500MB+ 
> (this top end will likely grow into several GB in the near future). 

hopefully allocate on flush will optimize that well enough.  how many 
threads?

>
>  * Our directory structure is very broad, but not very deep and files 
> are stored in leaf nodes only. 

hopefully reiserfs already does this one well.

>
>  * When a file is created and completely written to disk, it is often 
> re-read in its entirety at least two times within a couple of minutes. 

Hopefully current lru algorithms handle this.

>
>  * Our database app performs lots of block-based random I/O and fsyncs 
> within a dozen or so large files. 

This we have not given a lot of attention to, and probably some serious 
study of it is desirable.  I hope that we do reasonably well at this, 
but I can't say that I know that we do.

Thanks Brian, this was a good list.

>
>
>
>
> Hans Reiser wrote:
>
>> We were discussing how to optimize reiser4 best, and came to realize 
>> that us developers did not have a good enough intuition for what 
>> users do that stresses their filesystem enough that they care about 
>> its performance.
>>
>> If you just do edits of files it probably does not matter too much 
>> what fs you use.
>>
>> Booting the machine seems like one activity that many users end up 
>> waiting on the FS for.  Yes?
>>
>> Starting up complex and big applications like xemacs and mozilla 
>> would be another.  Yes?
>>
>> Others?
>>
>> Hans
>>
>> PS
>>
>> reiser4 performance is up a lot recently, and within two weeks I 
>> think cp -r will have been optimized as much as is worth doing.  cp 
>> -r accesses files in readdir order, and that does indeed seem worth 
>> optimizing, but soon we will need to optimize more sophisticated 
>> access patterns than that.....
>>
>> Hans
>
>
>


-- 
Hans

Legato (was: what do you do that stresses your filesystem?

[Hendrik Visage]

On Sun, Jan 05, 2003 at 11:17:05AM +0300, Hans Reiser wrote:
> Brian Tinsley wrote:
> 
> >I'm late replying to this one, but I've been on vacation :)

And I've just got the following request/issue put to me:

 Support by Legato Networker for backups. (www.legato.com)
 (At present it's only supported by Legato on S/390 platforms :()

It's also an interesting thing to see the impact of the traversing
of the filesystem during backups.

Re: Legato (was: what do you do that stresses your filesystem?

[Brian Tinsley]

Hendrik Visage wrote:

>It's also an interesting thing to see the impact of the traversing
>of the filesystem during backups.
>  
>

This is something that is hurting us badly right now! When our backup 
product starts scanning half terabyte filesystems with millions of 
files, it can essentially bring the machine to its knees (i.e., system 
time stays in the 90% range).


-- 

-[========================]-
-[      Brian Tinsley     ]-
-[ Chief Systems Engineer ]-
-[        Emageon         ]-
-[========================]-

Re: Legato (was: what do you do that stresses your filesystem?

[Hans Reiser]

Hendrik Visage wrote:

>On Sun, Jan 05, 2003 at 11:17:05AM +0300, Hans Reiser wrote:
>  
>
>>Brian Tinsley wrote:
>>
>>    
>>
>>>I'm late replying to this one, but I've been on vacation :)
>>>      
>>>
>
>And I've just got the following request/issue put to me:
>
> Support by Legato Networker for backups. (www.legato.com)
> (At present it's only supported by Legato on S/390 platforms :()
>
>It's also an interesting thing to see the impact of the traversing
>of the filesystem during backups.
>
>
>
>
>  
>
Please contact legato and ask them why they don't support it.  Frankly I 
think it is pretty strange to support it only for the S/390, I can't 
imagine the software knowign the difference between reiserfs on the 
S/390 and reiserfs on Intel.

-- 
Hans

Re: what do you do that stresses your filesystem?

[Brian Tinsley]

>
>
>>
>>  * SMP machines 
>
>
> already addressed by reiser4 (but not tested and benchmarked yet) 


I'm working on getting set up to do just this. I've got Mongo and both 
standard SCSI and fibre channel storage, but haven't had enough free 
time to cobble together a full 2.5.x system.

>>
>>  * Our applications perform multi-threaded streaming I/O 
>> (network-to-disk and vice versa) with read/write block sizes varying 
>> from 16KB to 64KB; we have files ranging in size from 32KB to 500MB+ 
>> (this top end will likely grow into several GB in the near future). 
>
>
> hopefully allocate on flush will optimize that well enough.  how many 
> threads? 

Given a "normal" system load, I would say 10 to 20 concurrent threads 
performing disk I/O would be a reasonable number.

>>
>>  * Our directory structure is very broad, but not very deep and files 
>> are stored in leaf nodes only. 
>
>
> hopefully reiserfs already does this one well. 

Definitely! This is one of the reasons I chose to use it :)

>>
>>  * Our database app performs lots of block-based random I/O and 
>> fsyncs within a dozen or so large files. 
>
>
> This we have not given a lot of attention to, and probably some 
> serious study of it is desirable.  I hope that we do reasonably well 
> at this, but I can't say that I know that we do.

Our database transaction times are consistently very good, but I've 
never performed any hardcore analysis either.



-- 

-[========================]-
-[      Brian Tinsley     ]-
-[ Chief Systems Engineer ]-
-[        Emageon         ]-
-[========================]-

Re: what do you do that stresses your filesystem?

[Hans Reiser]

Brian Tinsley wrote:

>>
>>
>>>
>>>  * SMP machines 
>>
>>
>>
>> already addressed by reiser4 (but not tested and benchmarked yet) 
>
>
>
> I'm working on getting set up to do just this. I've got Mongo and both 
> standard SCSI and fibre channel storage, but haven't had enough free 
> time to cobble together a full 2.5.x system. 

Ok, but be aware that reiser4 is several weeks away from being ready for 
prime time.  Our performance needs more tweaks, and bugs remain.

>
>
>>>
>>>  * Our applications perform multi-threaded streaming I/O 
>>> (network-to-disk and vice versa) with read/write block sizes varying 
>>> from 16KB to 64KB; we have files ranging in size from 32KB to 500MB+ 
>>> (this top end will likely grow into several GB in the near future). 
>>
>>
>>
>> hopefully allocate on flush will optimize that well enough.  how many 
>> threads? 
>
>
> Given a "normal" system load, I would say 10 to 20 concurrent threads 
> performing disk I/O would be a reasonable number. 

Divide the amount of ram flushed by the number of threads, and you'll 
get the size of each threads batch that gets optimized all together at 
flush time.  Probably you could benefit from some special purpose 
tweaking of the FS (to monitor and adjust the size of the batches), but 
hopefully it does not perform too bad as it is.


-- 
Hans

Re: what do you do that stresses your filesystem?

[Philipp Gühring]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

> Booting the machine seems like one activity that many users end up
> waiting on the FS for.  Yes?

Yes, I think this is the "Linux factor". Seeing how long booting needs. 
(Although I think that timeouts like the SCSI initialisation or non-existing 
DHCP servers need far more time than optimizing the filesystem could ever 
give)

> Starting up complex and big applications like xemacs and mozilla would
> be another.  Yes?

Yes, especially have a look at KDE. 
(They have a longstanding performance discussion with all the shared 
libraries, ...)

> Others?

I am currently thinking about multi-dimensional region search 
(Sedgewick, "Algorithms in C", chapter 26 in the german edition), and how 
something similar could be implemented on ReiserFS.
I am currently developing a Location-Based-Service application on top of my 
IVI::DB (XML-ReiserFS database), and having 3-dimensional region searches is 
the primary service, my application has to provide.

At first, I wanted to implement Sedgewick's algorithm completely in RAM, but 
now the requirements changed, and I have to do it in the filesystem, and I 
calculate with 30000 queries per minute.

Many greetings,
- -- 
~ Philipp Gühring              p.guehring@futureware.at
~ http://www.livingxml.net/       ICQ UIN: 6588261
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE+GEjRlqQ+F+0wB3oRApf/AJ4g2pnacYjVfkv0tROaM8gOn5UWIgCfevfI
uvOuZSRMt3lrv7wj8x2MIAU=
=lY5a
-----END PGP SIGNATURE-----

Re: what do you do that stresses your filesystem?

[Zygo Blaxell]

In article <3E06F360.7000708@namesys.com>,
Hans Reiser  <reiser@namesys.com> wrote:
>We were discussing how to optimize reiser4 best, and came to realize 
>that us developers did not have a good enough intuition for what users 
>do that stresses their filesystem enough that they care about its 
>performance.

>Booting the machine seems like one activity that many users end up 
>waiting on the FS for.  Yes?

>Starting up complex and big applications like xemacs and mozilla would 
>be another.  Yes?

Not for me.  I reboot only for upgrades, crashes, power failures, and
dead hardware replacement.  I try to avoid complex and big applications
for many reasons including startup time, but startup time is relatively
unimportant.

>Others?

Lots of rsync's, cp -al's, rm -rf's, sometimes involving millions of ~12K
files at a time.  apt-get dist-upgrade on Debian.  Big C/C++ compiles.
Often a number of these at the same time on the same machine.

I store multi-gigabyte collections of files about 12K in size each.
This isn't usually a performance issue as the files are normally accessed
at the rate of a few hundred per hour; however, relatively infrequently
I need to copy one of these collections to another machine, or process
all of the files at once, or transfer the collection to or from tape.

Often I want to _avoid_ putting stress on my filesystem.  I want to do
full system backups, but I don't really care how long they take, and I
don't want any other users of the machine to be inconvenienced by them.
Obviously if the machine is otherwise idle, I want the backups to be fast.

My favorite toy benchmark application involves creating a bootable CD-ROM
from a chroot Debian system.  This involves four phases:

	1.  Build a compressed initrd (mostly CPU bound)

	2.  Scan the chroot filesystem for identical files and
	hardlinking them together (mostly FS bound)

	3.  mkzftree --parallelism 16 (CPU and FS bound)

	4.  mkisofs (mostly FS bound)

All input and output is on the same filesystem.

Phase 2 is interesting because the other filesystems (xfs, ext[23], tmpfs)
are all more than twice as slow as Reiser.  This phase represents about 
one third of the total time.  It's a perl script that runs 'find', sorts
the results by size, then reads all files of identical size sequentially
to generate sha1sums, then replaces files with identical sha1sums with
hardlinks to one of the files.

Phase 3 is interesting because the distribution of system and real time
varies wildly depending on filesystem (and not, as one might expect, on
things like whether the machine has one or two CPU's or whether a disk
array is used or a single disk).  reiser finished in 30% less real time
than the other filesystems despite being rather heavy in CPU system time.

Reiser is the fastest filesystem at each of these phases individually,
and overall reiser is about 40% faster than xfs or ext[23] over a wide
variety of system configurations.  tmpfs was so slow that I didn't bother
letting it finish (I killed it after 90 minutes on a machine that runs
the benchmark on reiser in 22 minutes).

-- 
Zygo Blaxell (Laptop) <zblaxell@feedme.hungrycats.org>
GPG = D13D 6651 F446 9787 600B AD1E CCF3 6F93 2823 44AD