Re: O_DIRECT performance impact on 2.4.18 (was: Re: [PATCH] 2.5.14IDE 56)

Andrea Arcangeli
2002-05-14T00:21:41Z

On Tue, May 14, 2002 at 09:58:31AM +1000, Lincoln Dale wrote:
> At 01:19 PM 13/05/2002 +0200, Andrea Arcangeli wrote:
> >> so the above is:
> >> blocks = 64K, bs=8k means 64000 x 8192-byte read()s = 524288000 bytes
> >> blocks = 5K, bs=1m means 5000 x 1048576-byte read()s = 5242880000 bytes
> >
> >if the program is doing only what shown in the main loop, then you're
> >reading 10 times more data with O_DIRECT, that was my point in saying
> >64k*8k == 5k * 1M / 10, but I assume you took it into account (otherwise
> >it means O_DIRECT is just 5 times faster than buffered-I/O for you)
>
> no- count them up above.
> without O_DIRECT i was doing: 64000 x 8192 = 524288000 bytes
> with O_DIRECT i was doing: 5000 x 1048576 = 524288000 bytes.
>
> ie. same amount of data.

I don't mind the 1024/1000 difference, in this context let's assume
k=K,m=M and g=G, I only mind the 1 order of magintude difference. python
tells me:

w/o O_DIRECT 64000 * 8192 = 524288000
w/ O_DIRECT 5000 * 1048576 = 5242880000
^ note the additional zero

they're both _bytes_, it's apples against apples.

You instead wrote:

without O_DIRECT i was doing: 64000 x 8192 = 524288000
with O_DIRECT i was doing: 5000 x 1048576 = 524288000

in the above you're losing a 0 at the end in the O_DIRECT case.

> regardless, the "mbyte/sec" is calculated at the very end.

So I assume the 1 order of magnitude difference didn't affect the
benchmark results anyways. That's quite expectable otherwise as said in
the earlier email O_DIRECT would be just running 5 times faster than
non-O_DIRECT :).

> >Also I would suggest to measure the time taken by the whole workload, not
> >only
> >the time for read/write syscalls.
>
> the time taken for the whole workload _is_ calculated at the very end of
> the workload. that way and "readahead" doesn't have an unfair advantage.
> i also have gettimeofday() calls to measure latency on a per-read or
> per-write basis

Ok fine, by only reading the pseudocode of the main loop it wasn't
obvious (I mistaken the latency accounting for the global throughput
accounting). thanks for the clarification.

> ...
> >One thing I would also recommend is to write a threaded version of the
> >program,
> >that reads or writes to all the /dev/sd disks simultaneously, first w/
> >O_DIRECT
> >then w/o O_DIRECT. The reason is that currently you aren't using all the
> >disks
> >at once with O_DIRECT due the lack of async-io
>
> i've thought about doing that - shame that there isn't an async version of
> read(). the hard part is that i want to keep the disks at roughly the same
> "block" at the same time, so there will still need to be some
> syncronization of threads.
> otherwise, basic SCSI id priority means that it won't be fair across all
> disks. (remeber that i'm also attempting to measure latency)
>
> ...
> >Since you "stripe" by hand in all disks you do a different workload than my
> >previous benchs and you definitely want to keep all the harddisk running
> >at the
> >same time. I would also suggest to benchmark a single disk, to see if there
> >is
> >still such a big performance difference (again: including the cost outside
> >the
> >syscalls too).
>
> a single-disk means we hit the performance limits of a single disk
> spindle. ie. around 45mbyte/sec sustained throughput.
> if you don't think there's any real overhead in the MD driver, i'll just
> use that instead for now. (raid-0)

I think raid0 is a good start to make all disks running at the same time
for O_DIRECT too (only make sure to use a buffer large nr_PV*512k or
nr_PV*1M to allow the generation of large dma transactions to each
disk). The overhead of raid-0 shouldn't be noticeable, it should be
minimal compared to the overhead of the 1k bhs.

Andrea