Reiserfs concurrent write problems

Reiserfs concurrent write problems

[Bruce Guenter]

[-- Attachment #1: Type: text/plain, Size: 691 bytes --]

Greetings.

I have been running a series of custom benchmarks to see what
filesystems perform best for a maildir-based mail server.

	http://untroubled.org/benchmarking/2004-04/

So far, reiserfs has by far the best reading rate, but has serious
problems when more than one writer is trying to fsync concurrently.
Also, adding an external journal actually slows down the procedure
instead of improving performance.  Are there any configuration changes
or patches I should be trying to improve write performance?

Thanks.
-- 
Bruce Guenter <bruceg@em.ca> http://em.ca/~bruceg/ http://untroubled.org/
OpenPGP key: 699980E8 / D0B7 C8DD 365D A395 29DA  2E2A E96F B2DC 6999 80E8

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Reiserfs concurrent write problems

[Chris Mason]

On Mon, 2004-04-26 at 13:26, Bruce Guenter wrote:
> Greetings.
> 
> I have been running a series of custom benchmarks to see what
> filesystems perform best for a maildir-based mail server.
> 
> 	http://untroubled.org/benchmarking/2004-04/
> 
> So far, reiserfs has by far the best reading rate, but has serious
> problems when more than one writer is trying to fsync concurrently.
> Also, adding an external journal actually slows down the procedure
> instead of improving performance.  Are there any configuration changes
> or patches I should be trying to improve write performance?

Please try 2.6.6-rc2-mm2, which has new block allocator patches and
other speedups.

The default mount options should work best for you, but this might work
too:

mount -o alloc=skip_busy:dirid_groups /dev/xxx /mnt

Depending on the file size, 

mount -o notail can make a huge difference too.

-chris

Re: Reiserfs concurrent write problems

[Bruce Guenter]

[-- Attachment #1: Type: text/plain, Size: 699 bytes --]

On Mon, Apr 26, 2004 at 01:36:11PM -0400, Chris Mason wrote:
> Please try 2.6.6-rc2-mm2, which has new block allocator patches and
> other speedups.

Thanks.  I'm compiling this now.

> The default mount options should work best for you, but this might work
> too:
> 
> mount -o alloc=skip_busy:dirid_groups /dev/xxx /mnt

I'll try that.

> Depending on the file size, 
> 
> mount -o notail can make a huge difference too.

They are mostly small files, averaging about 12.5kB.  I am already using
	mount -o noatime,notail

-- 
Bruce Guenter <bruceg@em.ca> http://em.ca/~bruceg/ http://untroubled.org/
OpenPGP key: 699980E8 / D0B7 C8DD 365D A395 29DA  2E2A E96F B2DC 6999 80E8

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Reiserfs concurrent write problems

[Bruce Guenter]

[-- Attachment #1: Type: text/plain, Size: 720 bytes --]

On Mon, Apr 26, 2004 at 01:36:11PM -0400, Chris Mason wrote:
> Please try 2.6.6-rc2-mm2, which has new block allocator patches and
> other speedups.
> 
> The default mount options should work best for you, but this might work
> too:
> 
> mount -o alloc=skip_busy:dirid_groups /dev/xxx /mnt

Using the new kernel and the above mount options boosted the write
performance of ReiserFS, almost to the level of XFS.  At the highest
concurrency level, the reading performance dropped, but it's still much
faster than the other FSs.

Thanks for your help.
-- 
Bruce Guenter <bruceg@em.ca> http://em.ca/~bruceg/ http://untroubled.org/
OpenPGP key: 699980E8 / D0B7 C8DD 365D A395 29DA  2E2A E96F B2DC 6999 80E8

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Reiserfs concurrent write problems

[Chris Mason]

On Tue, 2004-04-27 at 12:15, Bruce Guenter wrote:
> On Mon, Apr 26, 2004 at 01:36:11PM -0400, Chris Mason wrote:
> > Please try 2.6.6-rc2-mm2, which has new block allocator patches and
> > other speedups.
> > 
> > The default mount options should work best for you, but this might work
> > too:
> > 
> > mount -o alloc=skip_busy:dirid_groups /dev/xxx /mnt
> 
> Using the new kernel and the above mount options boosted the write
> performance of ReiserFS, almost to the level of XFS.  At the highest
> concurrency level, the reading performance dropped, but it's still much
> faster than the other FSs.
> 
> Thanks for your help.

Good to hear, could I trouble you to ping me (or better yet
reiserfs-list) when you have the graphs updated?

Other notes about 2.6.6-rc2-mm2, it has some ext3 optimizations as well,
you might want to rebenchmark there.  reiserfs defaults to -o
data=ordered in the 2.6.6-rc2 (and -mm), you should get slightly better
numbers with -o data=writeback.

-chris

Re: Reiserfs concurrent write problems

[Bruce Guenter]

[-- Attachment #1: Type: text/plain, Size: 917 bytes --]

On Tue, Apr 27, 2004 at 02:00:41PM -0400, Chris Mason wrote:
> Good to hear, could I trouble you to ping me (or better yet
> reiserfs-list) when you have the graphs updated?

For sure.

> Other notes about 2.6.6-rc2-mm2, it has some ext3 optimizations as well,
> you might want to rebenchmark there.

I am re-running all of the benchmarks, actually, as well as doing some
tweaking to the XFS options.

> reiserfs defaults to -o
> data=ordered in the 2.6.6-rc2 (and -mm), you should get slightly better
> numbers with -o data=writeback.

I will try that, as well as data=journal (which I understand is one of
the new features in the -mm patches).  The production system will not
run data=writeback, but it is still a useful number.

Thanks.
-- 
Bruce Guenter <bruceg@em.ca> http://em.ca/~bruceg/ http://untroubled.org/
OpenPGP key: 699980E8 / D0B7 C8DD 365D A395 29DA  2E2A E96F B2DC 6999 80E8

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Reiserfs concurrent write problems

[Chris Mason]

On Tue, 2004-04-27 at 16:15, Bruce Guenter wrote:
> On Tue, Apr 27, 2004 at 02:00:41PM -0400, Chris Mason wrote:
> > Good to hear, could I trouble you to ping me (or better yet
> > reiserfs-list) when you have the graphs updated?
> 
> For sure.
> 
> > Other notes about 2.6.6-rc2-mm2, it has some ext3 optimizations as well,
> > you might want to rebenchmark there.
> 
> I am re-running all of the benchmarks, actually, as well as doing some
> tweaking to the XFS options.
> 
> > reiserfs defaults to -o
> > data=ordered in the 2.6.6-rc2 (and -mm), you should get slightly better
> > numbers with -o data=writeback.
> 
> I will try that, as well as data=journal (which I understand is one of
> the new features in the -mm patches).  The production system will not
> run data=writeback, but it is still a useful number.

-mm doesn't have data=journal for reiserfs, but I do have a patch
available.  I'll send you an update tomorrow morning.

-chris

Re: Reiserfs concurrent write problems

[Chris Mason]

On Tue, 2004-04-27 at 16:15, Bruce Guenter wrote:
> On Tue, Apr 27, 2004 at 02:00:41PM -0400, Chris Mason wrote:
> > Good to hear, could I trouble you to ping me (or better yet
> > reiserfs-list) when you have the graphs updated?
> 
> For sure.
> 
> > Other notes about 2.6.6-rc2-mm2, it has some ext3 optimizations as well,
> > you might want to rebenchmark there.
> 
> I am re-running all of the benchmarks, actually, as well as doing some
> tweaking to the XFS options.

Ok, after a quick read of the rest of your page, here are a few ideas:

reiserfs doesn't need as large a log as ext3 in order to get good
numbers for fsync heavy workloads.  This is good because reiserfs also
pins the logged buffers more then ext3 does, so a 132mb log on reiserfs
is likely to tie up that much ram in pinned metadata.  You might want to
experiment a little with the log size.

I'm a little confused about how an external log slows things down,
please let me know if you see those results with the new patches.

Your sequence is this:

write(tmp file)
fsync(tmp file)
rename(tmp file, somedir/new file)
fsync(somedir)

That last fsync holds the directory semaphore during the log commit,
which hurts concurrency on that directory badly.  Both reiserfs and ext3
can get away with:
rename(tmp file, somedir/new file)
fsync(tmp file fd)

(where tmp file fd has been kept open from the initial write to tmp
file)

It should be much faster.

In reiserfs data=journal mode, you get strict ordering of all the
transactions.  This means you can safely do this:

write(temp file)
rename(temp file, somedir/new file)
fsync(temp file fd)

In data=journal mode and still be safe, as long as the same process does
all three steps above.  This is because you know the rename will either
be in the same atomic unit as the last data block in the file, or it
will be in a later atomic unit then the last data block in the file. 
Either way, the rename can't happen first.

This should hold true for ext3 as well, but you should check with the
ext3 developers before basing any products on it.

-chris

Re: Reiserfs concurrent write problems

[Bruce Guenter]

[-- Attachment #1: Type: text/plain, Size: 2297 bytes --]

On Wed, Apr 28, 2004 at 07:48:22PM -0400, Chris Mason wrote:
> reiserfs doesn't need as large a log as ext3 in order to get good
> numbers for fsync heavy workloads.  This is good because reiserfs also
> pins the logged buffers more then ext3 does, so a 132mb log on reiserfs
> is likely to tie up that much ram in pinned metadata.  You might want to
> experiment a little with the log size.

Noted.  Having extra RAM tied up is likely to only show up in the read
performance.

> I'm a little confused about how an external log slows things down,
> please let me know if you see those results with the new patches.

With 2.6.6-rc2-mm2, having an external log makes it considerably faster, up
to 16-concurrency level, where it is somewhat slower.

I'd post the new results, but ext3 is behaving very weird under this
kernel -- doing a single transaction takes up to 950ms to complete,
which is more than an order of magnitude too big.

> Your sequence is this:
> 
> write(tmp file)
> fsync(tmp file)
> rename(tmp file, somedir/new file)
> fsync(somedir)
> 
> That last fsync holds the directory semaphore during the log commit,
> which hurts concurrency on that directory badly.

Fortunately, there is relatively little concurrency on a per-directory
level.  There are so many directories in play that there is little
chance that two processes (either reading or writing) will be in
contention for the same directory.

> Both reiserfs and ext3
> can get away with:
> rename(tmp file, somedir/new file)
> fsync(tmp file fd)
> 
> It should be much faster.

I'll add a deferred-fsync mode to test this.

> In reiserfs data=journal mode, you get strict ordering of all the
> transactions.

What about in data=ordered mode?

> This means you can safely do this:
> write(temp file)
> rename(temp file, somedir/new file)
> fsync(temp file fd)

How is that different from the above?

> This should hold true for ext3 as well, but you should check with the
> ext3 developers before basing any products on it.

I will double check on the ext3 list to see if they order the commits
this way.
-- 
Bruce Guenter <bruceg@em.ca> http://em.ca/~bruceg/ http://untroubled.org/
OpenPGP key: 699980E8 / D0B7 C8DD 365D A395 29DA  2E2A E96F B2DC 6999 80E8

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Reiserfs concurrent write problems

[Chris Mason]

On Thu, 2004-04-29 at 15:04, Bruce Guenter wrote:
> On Wed, Apr 28, 2004 at 07:48:22PM -0400, Chris Mason wrote:
> > reiserfs doesn't need as large a log as ext3 in order to get good
> > numbers for fsync heavy workloads.  This is good because reiserfs also
> > pins the logged buffers more then ext3 does, so a 132mb log on reiserfs
> > is likely to tie up that much ram in pinned metadata.  You might want to
> > experiment a little with the log size.
> 
> Noted.  Having extra RAM tied up is likely to only show up in the read
> performance.
> 
> > I'm a little confused about how an external log slows things down,
> > please let me know if you see those results with the new patches.
> 
> With 2.6.6-rc2-mm2, having an external log makes it considerably faster, up
> to 16-concurrency level, where it is somewhat slower.

Strange stuff, it should always be faster.  I'll have to try and
reproduce.

> 
> I'd post the new results, but ext3 is behaving very weird under this
> kernel -- doing a single transaction takes up to 950ms to complete,
> which is more than an order of magnitude too big.
> 
Ouch, Andrew will want to know about that.  

> > Your sequence is this:
> > 
> > write(tmp file)
> > fsync(tmp file)
> > rename(tmp file, somedir/new file)
> > fsync(somedir)
> > 
> > That last fsync holds the directory semaphore during the log commit,
> > which hurts concurrency on that directory badly.
> 
> Fortunately, there is relatively little concurrency on a per-directory
> level.  There are so many directories in play that there is little
> chance that two processes (either reading or writing) will be in
> contention for the same directory.
> 
> > Both reiserfs and ext3
> > can get away with:
> > rename(tmp file, somedir/new file)
> > fsync(tmp file fd)
> > 
> > It should be much faster.
> 
> I'll add a deferred-fsync mode to test this.
> 
> > In reiserfs data=journal mode, you get strict ordering of all the
> > transactions.
> 
> What about in data=ordered mode?
> 
It won't work in the face of io errors.

> > This means you can safely do this:
> > write(temp file)
> > rename(temp file, somedir/new file)
> > fsync(temp file fd)
> 
> How is that different from the above?
> 
There's no fsync after the first write.  Just to be explicit:

data=writeback|ordered			data=journal
write(temp file)			write(temp file)
fsync(temp file fd)
rename(tempfile, somedir/newfile)	rename(tempfile, somedir/newfile)
fsync(temp file fd)			fsync(temp file fd)

-chris

Re: Reiserfs concurrent write problems

[Bruce Guenter]

[-- Attachment #1: Type: text/plain, Size: 517 bytes --]

On Thu, Apr 29, 2004 at 03:19:25PM -0400, Chris Mason wrote:
> On Thu, 2004-04-29 at 15:04, Bruce Guenter wrote:
> > > In reiserfs data=journal mode, you get strict ordering of all the
> > > transactions.
> > 
> > What about in data=ordered mode?
>
> It won't work in the face of io errors.

What kind of I/O errors would cause what kind of problems?
-- 
Bruce Guenter <bruceg@em.ca> http://em.ca/~bruceg/ http://untroubled.org/
OpenPGP key: 699980E8 / D0B7 C8DD 365D A395 29DA  2E2A E96F B2DC 6999 80E8

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Reiserfs concurrent write problems

[Chris Mason]

On Thu, 2004-04-29 at 16:08, Bruce Guenter wrote:
> On Thu, Apr 29, 2004 at 03:19:25PM -0400, Chris Mason wrote:
> > On Thu, 2004-04-29 at 15:04, Bruce Guenter wrote:
> > > > In reiserfs data=journal mode, you get strict ordering of all the
> > > > transactions.
> > > 
> > > What about in data=ordered mode?
> >
> > It won't work in the face of io errors.
> 
> What kind of I/O errors would cause what kind of problems?

In data=ordered mode if you get io errors on the data blocks, you might
not find out about them until the fsync.  If the fsync is after the
rename, you'll rename a bad file into the delivered directory.

-chris

Re: Reiserfs concurrent write problems

[Bruce Guenter]

[-- Attachment #1: Type: text/plain, Size: 480 bytes --]

On Tue, Apr 27, 2004 at 02:00:41PM -0400, Chris Mason wrote:
> Good to hear, could I trouble you to ping me (or better yet
> reiserfs-list) when you have the graphs updated?

I've updated the benchmarking web page to include the new results from
the 2.6.6-rc2-mm2 kernel:
	http://untroubled.org/benchmarking/2004-04/
-- 
Bruce Guenter <bruceg@em.ca> http://em.ca/~bruceg/ http://untroubled.org/
OpenPGP key: 699980E8 / D0B7 C8DD 365D A395 29DA  2E2A E96F B2DC 6999 80E8

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Reiserfs concurrent write problems

[Chris Mason]

On Fri, 2004-04-30 at 15:54, Bruce Guenter wrote:
> On Tue, Apr 27, 2004 at 02:00:41PM -0400, Chris Mason wrote:
> > Good to hear, could I trouble you to ping me (or better yet
> > reiserfs-list) when you have the graphs updated?
> 
> I've updated the benchmarking web page to include the new results from
> the 2.6.6-rc2-mm2 kernel:
> 	http://untroubled.org/benchmarking/2004-04/

Thanks.  One thing I didn't notice before:

Filesystem partition: 72GB on a software RAID-1 array of Fujitsu SCSI
15kRPM disks. 

Journal partition: 400MB on a software RAID-1 array of Quantum Atlas 10K
SCSI disks. 

So the external log is on slower disks?  That might explain why it is
sometimes slower to have reiserfs on the external log.

-chris

Re: Reiserfs concurrent write problems

[Bruce Guenter]

[-- Attachment #1: Type: text/plain, Size: 762 bytes --]

On Fri, Apr 30, 2004 at 04:24:10PM -0400, Chris Mason wrote:
> Thanks.  One thing I didn't notice before:
> 
> Filesystem partition: 72GB on a software RAID-1 array of Fujitsu SCSI
> 15kRPM disks. 
> Journal partition: 400MB on a software RAID-1 array of Quantum Atlas 10K
> SCSI disks. 
> 
> So the external log is on slower disks?  That might explain why it is
> sometimes slower to have reiserfs on the external log.

Yes, the journal is on a slower array.  That's what was available when
building the system.  They're still not *SLOW* disks, though, at 10kRPM
and 32MB/sec transfer rate.
-- 
Bruce Guenter <bruceg@em.ca> http://em.ca/~bruceg/ http://untroubled.org/
OpenPGP key: 699980E8 / D0B7 C8DD 365D A395 29DA  2E2A E96F B2DC 6999 80E8

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Reiserfs concurrent write problems

[Dieter Nützel]

Am Freitag, 30. April 2004 23:44 schrieb Bruce Guenter:
> On Fri, Apr 30, 2004 at 04:24:10PM -0400, Chris Mason wrote:
> > Thanks.  One thing I didn't notice before:
> >
> > Filesystem partition: 72GB on a software RAID-1 array of Fujitsu SCSI
> > 15kRPM disks.
> > Journal partition: 400MB on a software RAID-1 array of Quantum Atlas 10K
> > SCSI disks.
> >
> > So the external log is on slower disks?  That might explain why it is
> > sometimes slower to have reiserfs on the external log.
>
> Yes, the journal is on a slower array.  That's what was available when
> building the system.  They're still not *SLOW* disks, though, at 10kRPM
> and 32MB/sec transfer rate.

I have the former Fujitsu, too.
So the Atlas are "very" slow disks. ;-)

One of my Fujitsu MASxxxx:

SunWave1 /home/nuetzel# hdparm -tT /dev/sdb2

/dev/sdb2:
 Timing buffer-cache reads:   1064 MB in  2.00 seconds = 531.55 MB/sec
 Timing buffered disk reads:  214 MB in  3.00 seconds =  71.30 MB/sec

Cheers,
	Dieter

Re: Reiserfs concurrent write problems

[Paul Wagland]

[-- Attachment #1: Type: text/plain, Size: 974 bytes --]


On Apr 30, 2004, at 21:54, Bruce Guenter wrote:

> On Tue, Apr 27, 2004 at 02:00:41PM -0400, Chris Mason wrote:
>> Good to hear, could I trouble you to ping me (or better yet
>> reiserfs-list) when you have the graphs updated?
>
> I've updated the benchmarking web page to include the new results from
> the 2.6.6-rc2-mm2 kernel:
> 	http://untroubled.org/benchmarking/2004-04/

One interesting thing that I noticed is that all of the other 
filesystems have had major performance regressions on most of the tests 
as compared to the gentoo kernel.

I.e.
                2.6.6-rc2-mm2   Gentoo-2.6.5
ext2 1wr del:  70.537          45.716
ext2 16wr del: 647.679         580.760
xfs 1wr del:   14.388          16.216
xfs 16wr del:  308.173         122.987

I am very curious as to how the Gentoo kernel does against the stock 
kernel, i.e. whether or not this is a mainline regression, or whether 
Gentoo has an added patch or two that helps things out somehow.

Cheers,
Paul

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 186 bytes --]

Re: Reiserfs concurrent write problems

[Chris Mason]

On Sat, 2004-05-01 at 05:26, Paul Wagland wrote:
> On Apr 30, 2004, at 21:54, Bruce Guenter wrote:
> 
> > On Tue, Apr 27, 2004 at 02:00:41PM -0400, Chris Mason wrote:
> >> Good to hear, could I trouble you to ping me (or better yet
> >> reiserfs-list) when you have the graphs updated?
> >
> > I've updated the benchmarking web page to include the new results from
> > the 2.6.6-rc2-mm2 kernel:
> > 	http://untroubled.org/benchmarking/2004-04/
> 
> One interesting thing that I noticed is that all of the other 
> filesystems have had major performance regressions on most of the tests 
> as compared to the gentoo kernel.
> 
It's a good point.  The jump between 2.6.5 and 2.6.6-rc2-mm2 is pretty
big.  There were fundamental changes in filesystem writeback, scheduler
changes and a few others.  The reiserfs logging changes and new block
allocator do make -mm2 the best for this kind of benchmark on reiserfs,
but maybe not for the other filesystems.

It might be good to get a benchmarks against 2.6.6-rc2, which has many
of the writeback changes from -mm.  If those are causing problems, we'll
want to know sooner rather then later.

There are also some latency fixes in -mm2, which might slow down the
delivery procs in order to give better latencies to concurrent readers. 
Notice the read times are much better in -mm.

Any time you've got a multiprocess benchmark like this, it helps to
gather some kind of information about latencies and fairness, especially
for the delivery rate graphs.

The XFS reading time vs 16 writers is a latency bug of some kind. 
Probably easily solvable, I'm sure they would appreciate a heads up on
it.

-chris