mmap() call blocks on Disk I/O?

[George Wang <gwang@litespeedtech.com>]

Hi,

I am trouble shooting some disk I/O performance issues and have an
interesting finding.

The project is to improve event-driven web server performance in disk
bound situation by avoiding blocking on disk I/O.
We use separate threads to do disk I/O tasks, so we expect the main
thread wont be badly affected by disk I/O, but it still is.
I use strace to locate the sys-calls that block the main server thread
when server is doing disk I/O intensive tasks.

The finding is mmap() calls. we map files in 128KB memory blocks in the
main thread.

When disk I/O is low, mmap() usually takes 0.00002 second to complete,
when the disk getting busy, it takes about 0.01 second to
complete,  if there are multiple disk I/O tasks making the disk seeks
back and forth. mmap could take 0.1 second and up, the worst number we
get is 1.4 seconds.

Noticed that there is a patch to improve the multi-thread mmap()
performance, it should not be the same issue as we have met, I believe.

Just want to report this issue and hope it is helpful to improve the
kernel.

OpenSuse 11.0 with kernel 2.6.22.19-0.2-default, dual Intel Xeon 5410
(8-cores) with 8GB memory.

Thanks,
George Wang