Re: Higher than expected disk write(2) latency

[Martin Sustrik <sustrik@fastmq.com>]

Hi all,

> Which disk scheduler are you using - some of the disk schedulers
> intentionally delay writes to try and get better block merging.

We've run the test with different I/O schedulers. We've found out that 
poor performance we've seen before was due to the short AIO queue size. 
Now, when queue size is big enough to hold all the write requests in the 
test, the results are much better. Have a look here for the code of the 
test and the results in form of graphs:

http://www.zeromq.org/results:aio

There are still at least two problems I see:

1. Enqueueing of writes seems to block every now and then although the 
size of AIO queue is large enough to hold all the requests in the test.

2. We've observed that messages are batched for writes by approximately 
30 per batch. The latency impact of messages 2-30 is almost zero, 
however, the impact of the first message is much higher than expected. I 
would expect latency of one disk revolving (8.3 ms) + optionally latency 
of head movement (say 10 us). What we are seing are latencies of 30-60 ms.

Martin