Re: Slow sync in Interbench: anticipatory starves writes?

[Con Kolivas <kernel@kolivas.org>]

On Sun, 14 Aug 2005 09:06, Indan Zupancic wrote:
> Hello,

Hi.

Interesting find. I'm forwarding this on to lkml and Nick so they can peruse 
your findings and see what needs to be done.

Thanks,
Con

> Short version:
> With the anticipatory IO scheduler it's possible to cause extreme long
> delays so that sync() calls can take minutes. Isn't this a potential DOS?
>
> Long version:
> Running interbench took often much longer than expected. As the early
> versions were buggy I thought it was an interbench bug. But now, with
> interbench 0.29, the problem is still there. So I tried pinning it down
> and found out that it isn't an interbench bug at all, but that sync() is
> taking so long.
>
> All test results and data can be found at http://nul.nu/~indan/synctest/
> If no interval is given then it's 2 seconds.
>
> The testcase I used most is running a modified interbench which only
> tests X with the options -w Compile -c -t 10, so that only the X
> simulation under the Compile load is run without causing extra hd
> activity. All changes can be found in the file named "diff". This seemed
> the minimal test which reliably caused sync to be slow often. It may be
> that the full interbench run causes worse conditions, as some normal runs
> took 53 minutes (see 2.6.13-rc6-hg.log1).
>
> test1/ and test2/ dirs have earlier tests without sync timing. test3/ has
> sync timing added and is probably the most useful. Normally the X+Compile
> simulation takes about 15 to 20 seconds, but often 30 seconds or more,
> and once in a while more than one minute. test1/ was slowest with almost
> 4 minutes.
>
> At the end of the Compile test interbench stops the burn, write and read
> thread, in that order. Because stopping the write thread causes a sync
> which takes very long, the read thread keeps running, as can be seen in
> the vmstat output. A good log is
> http://nul.nu/~indan/synctest/test3/log9. There can also be seen that
> while interbench is waiting for the sync, almost no writing happens at
> all.
>
> Another good log with also meminfo output is
> http://nul.nu/~indan/synctest/test3/log2. There can be seen that even
> when there is no dirty or writeback data sync still doesn't return but
> keeps going.
>
> While writing this mail I tried other IO schedulers, and none have slow
> syncs except anticipatory. The others don't take more than one second for
> the sync.
>
> Funny that a cpu scheduler test finds IO scheduler problems. :-)
>
> Greetings,
>
> Indan