Re: [PATCH RFC] O_DIRECT reads and writes without i_sem

[Chris Mason <mason@suse.com>]

On Mon, 2004-11-01 at 16:08 +0000, Christoph Hellwig wrote:
> On Mon, Nov 01, 2004 at 10:32:07AM -0500, Chris Mason wrote:
> > Hello everyone,
> > 
> > Right now, O_DIRECT reads and writes on regular files have to take i_sem
> > while reading file metadata in order to make sure we don't race with
> > hole filling.
> > 
> > This patch tries to get around that by avoiding i_sem when we are doing
> > an O_DIRECT read or write inside of i_size.  Yet another rw semaphore is
> > added to struct inode to protect against holes being filled during the
> > O_DIRECT.  direct-io.c gets another special case to be aware of the
> > locking.
> > 
> > This has only been lightly tested, I'm posting here for general comments
> > before I go much further.  I'm rounding up some hardware with enough
> > disks to benchmark it properly.  

Looks like there's a long line to get to the big test machines here, so
I did some more local performance tests.  I've got two scsi drives
striped into a lvm2 target (64k stripe size).  FS is reiser.  I did
testing on writes because more of the io is done under i_sem for writes,
so it should be easiest to show the benefit there.

The test is a 16k random write test on a 15GB file.  50MB of io is done
by each random writer, and everyone is writing to the same preallocated
file.

1 random writer              = .64MB/s
2 random writers (unpatched) = .71MB/s
2 random writers (patched)   = 1.02MB/s

In the patched kernel, the two writer procs finished at roughly the same
time (< 1sec difference).  In the unpatched kernel, writer #1 finished
in about 1/2 the time as writer #2.

-chris