2.4.13-pre5-aa1 O_DIRECT drastic HIGHMEM performance hit

2.4.13-pre5-aa1 O_DIRECT drastic HIGHMEM performance hit

[Marvin Justice]

We're looking into setting up an HD video editing solution with the
following hardware:

dual 1GHz P3, 2GB RAM
Serverworks HeSL based board
dual U160 Adaptec SCSI card
5 68gb Fujitsu drives on each SCSI channel

The 10 drives are software raid0 striped with a 68k chunk size. For an
initial benchmark I got a copy of lmdd and modified it to open files
using O_DIRECT (also hacked it to work on W2K with FILE_FLAG_NO_BUFFER).
Here are the average results in MB/sec for 2.4.13-pre5-aa1 streaming
chunks of data 4MB at a time to RAM where they are immediately
discarded:

                     w           r
no O_DIRECT         136         132
O_DIRECT            111          96

I experimented around a bit and discovered that increasing
KIO_MAX_ATOMIC_IO from 512 to 4096 gave a significant performance boost
for the O_DIRECT read case:
                     w           r
no O_DIRECT         117         124
O_DIRECT            111         165


Next I recompiled without HIGHMEM support and was quite suprised
(numbers are with KIO_MAX_ATOMIC_IO=4096)

                     w           r
no O_DIRECT         138         125
O_DIRECT            221         182

XFS shows similar behavior: (XFS barfs if you change KIO_MAX_ATOMIC_IO
from 512)

                           w         r
O_DIRECT w/ HIGHMEM       114       146
O_DIRECT w/o HIGHMEM      218       248


(Incidentally, W2k reads and writes at a smooth 255 MB/sec on identical
hardware.) 

Are we stuck with a low mem configuration or are there workarounds that
would allow us to stick with the initial 2GB of RAM and still get ~200
MB/sec.
 


--------------------------
Marvin Justice
Software Developer
BOXX Technologies, Inc.
www.boxxtech.com
mjustice@boxxtech.com
(V) (512)225-6318
(F) (512)835-0434

Re: 2.4.13-pre5-aa1 O_DIRECT drastic HIGHMEM performance hit

[Alan Cox]

> Are we stuck with a low mem configuration or are there workarounds that
> would allow us to stick with the initial 2GB of RAM and still get ~200
> MB/sec.

Jens has patches for doing direct DMA I/O from highmem pages. That should
basically nullify that hit. Doing that mainstream is a 2.5 thing really

Re: 2.4.13-pre5-aa1 O_DIRECT drastic HIGHMEM performance hit

[Andrea Arcangeli]

On Thu, Oct 25, 2001 at 11:57:08AM -0500, Marvin Justice wrote:
> Are we stuck with a low mem configuration or are there workarounds that
> would allow us to stick with the initial 2GB of RAM and still get ~200
> MB/sec.

the problem you seen isn't specific to O_DIRECT. This should solve your
problem (not just for O_DIRECT):

	ftp://ftp.us.kernel.org/pub/linux/kernel/people/people/axboe/patches/2.4.13-pre4/block-highmem-all-17.bz2

It's a long time that I want to include the anti-bounce bits, but I am
still looking into the vm. The uglier part is that I suspect we'll also
need to make a 2G bounce option for some hardware combination. I hope
I'm wrong :).

Andrea