Re: Reiserfs concurrent write problems

[Bruce Guenter <lists-reiserfs-list@bruce-guenter.dyndns.org>]

[-- Attachment #1: Type: text/plain, Size: 2297 bytes --]

On Wed, Apr 28, 2004 at 07:48:22PM -0400, Chris Mason wrote:
> reiserfs doesn't need as large a log as ext3 in order to get good
> numbers for fsync heavy workloads.  This is good because reiserfs also
> pins the logged buffers more then ext3 does, so a 132mb log on reiserfs
> is likely to tie up that much ram in pinned metadata.  You might want to
> experiment a little with the log size.

Noted.  Having extra RAM tied up is likely to only show up in the read
performance.

> I'm a little confused about how an external log slows things down,
> please let me know if you see those results with the new patches.

With 2.6.6-rc2-mm2, having an external log makes it considerably faster, up
to 16-concurrency level, where it is somewhat slower.

I'd post the new results, but ext3 is behaving very weird under this
kernel -- doing a single transaction takes up to 950ms to complete,
which is more than an order of magnitude too big.

> Your sequence is this:
> 
> write(tmp file)
> fsync(tmp file)
> rename(tmp file, somedir/new file)
> fsync(somedir)
> 
> That last fsync holds the directory semaphore during the log commit,
> which hurts concurrency on that directory badly.

Fortunately, there is relatively little concurrency on a per-directory
level.  There are so many directories in play that there is little
chance that two processes (either reading or writing) will be in
contention for the same directory.

> Both reiserfs and ext3
> can get away with:
> rename(tmp file, somedir/new file)
> fsync(tmp file fd)
> 
> It should be much faster.

I'll add a deferred-fsync mode to test this.

> In reiserfs data=journal mode, you get strict ordering of all the
> transactions.

What about in data=ordered mode?

> This means you can safely do this:
> write(temp file)
> rename(temp file, somedir/new file)
> fsync(temp file fd)

How is that different from the above?

> This should hold true for ext3 as well, but you should check with the
> ext3 developers before basing any products on it.

I will double check on the ext3 list to see if they order the commits
this way.
-- 
Bruce Guenter <bruceg@em.ca> http://em.ca/~bruceg/ http://untroubled.org/
OpenPGP key: 699980E8 / D0B7 C8DD 365D A395 29DA  2E2A E96F B2DC 6999 80E8

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]