Re: 2.4.17: pwrite destroys block I/O throughput

[Andrew Morton <akpm@zip.com.au>]

"Jeffrey W. Baker" wrote:
> 
> Hi there,
> 
> I've never heard of pwrite and pread before, but htdig apparently makes
> very heavy use of it.

pwrite() is nice.  There's nothing special about it from a kernel
point of view.  It's equivalent to lseek+write to lower layers.

> Is linux's pwrite() just horribly broken?  Is htdig the only program
> that uses it?

Anything which does lots of discontiguous writes can do this.
Probably the recent shortening of the request queue made
it a little worse, but without the ability to perform
write merging at the buffercache LRU list level, we don't
really have a fix.

The reason why it makes your *read* throughput so bad is
that the writes are asynchronous.  So htdig can cheerfully
fill the request queue with 128 writes (and 128 seeks!) but
processes which are doing reads cannot do this asynchronously
(apart from readhead, which doesn't help much here).

So the readers get stuck on a queue behind 127 write seeks.
Eventually their read hits the head of the queue and gets
serviced.  Then they request another read.  And they go
to the back of the queue (or maybe the middle, if they get
lucky  - depends what block they're trying to read).

> Here's a little snapshot of htdig's syscalls, strace -s 0:
> 
> pwrite(6, ""..., 8192, 20717568)        = 8192
> pread(6, ""..., 8192, 138395648)        = 8192
> pwrite(6, ""..., 8192, 127918080)       = 8192
> ...
>

ug.  So we do have a real-world case.
 
> It's seeking all over the place.  Maybe pwrite/pread bypass the elevator
> and proper I/O scheduling.

Nope.  It's just a pathological case.

You'll get much, much better behaviour with

	http://www.zip.com.au/~akpm/linux/2.4/2.4.18-pre7/read-latency2.patch

Because it

a) boosts the priority of readers and
b) Increases the request queue size a lot, so write merges will
   be more common.

Long-term, the only fix for this is to perform the write-merging
at a much higher level - to give it visibility of all the writable
data in the machine.

-