Re: Linux 2.6.29

[Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>]

> 
> Linus Torvalds wrote:
> > This obviously starts the merge window for 2.6.30, although as usual, I'll 
> > probably wait a day or two before I start actively merging. I do that in 
> > order to hopefully result in people testing the final plain 2.6.29 a bit 
> > more before all the crazy changes start up again.
> 
> I know this has been discussed before:
> 
> [129401.996244] INFO: task updatedb.mlocat:31092 blocked for more than 
> 480 seconds.
> [129402.084667] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
> disables this message.
> [129402.179331] updatedb.mloc D 0000000000000000     0 31092  31091
> [129402.179335]  ffff8805ffa1d900 0000000000000082 ffff8803ff5688a8 
> 0000000000001000
> [129402.179338]  ffffffff806cc000 ffffffff806cc000 ffffffff806d3e80 
> ffffffff806d3e80
> [129402.179341]  ffffffff806cfe40 ffffffff806d3e80 ffff8801fb9f87e0 
> 000000000000ffff
> [129402.179343] Call Trace:
> [129402.179353]  [<ffffffff802d3ff0>] sync_buffer+0x0/0x50
> [129402.179358]  [<ffffffff80493a50>] io_schedule+0x20/0x30
> [129402.179360]  [<ffffffff802d402b>] sync_buffer+0x3b/0x50
> [129402.179362]  [<ffffffff80493d2f>] __wait_on_bit+0x4f/0x80
> [129402.179364]  [<ffffffff802d3ff0>] sync_buffer+0x0/0x50
> [129402.179366]  [<ffffffff80493dda>] out_of_line_wait_on_bit+0x7a/0xa0
> [129402.179369]  [<ffffffff80252730>] wake_bit_function+0x0/0x30
> [129402.179396]  [<ffffffffa0264346>] ext3_find_entry+0xf6/0x610 [ext3]
> [129402.179399]  [<ffffffff802d3453>] __find_get_block+0x83/0x170
> [129402.179403]  [<ffffffff802c4a90>] ifind_fast+0x50/0xa0
> [129402.179405]  [<ffffffff802c5874>] iget_locked+0x44/0x180
> [129402.179412]  [<ffffffffa0266435>] ext3_lookup+0x55/0x100 [ext3]
> [129402.179415]  [<ffffffff802c32a7>] d_alloc+0x127/0x1c0
> [129402.179417]  [<ffffffff802ba2a7>] do_lookup+0x1b7/0x250
> [129402.179419]  [<ffffffff802bc51d>] __link_path_walk+0x76d/0xd60
> [129402.179421]  [<ffffffff802ba17f>] do_lookup+0x8f/0x250
> [129402.179424]  [<ffffffff802c8b37>] mntput_no_expire+0x27/0x150
> [129402.179426]  [<ffffffff802bcb64>] path_walk+0x54/0xb0
> [129402.179428]  [<ffffffff802bfd10>] filldir+0x0/0xf0
> [129402.179430]  [<ffffffff802bcc8a>] do_path_lookup+0x7a/0x150
> [129402.179432]  [<ffffffff802bbb55>] getname+0xe5/0x1f0
> [129402.179434]  [<ffffffff802bd8d4>] user_path_at+0x44/0x80
> [129402.179437]  [<ffffffff802b53b5>] cp_new_stat+0xe5/0x100
> [129402.179440]  [<ffffffff802b56d0>] vfs_lstat_fd+0x20/0x60
> [129402.179442]  [<ffffffff802b5737>] sys_newlstat+0x27/0x50
> [129402.179445]  [<ffffffff8020c35b>] system_call_fastpath+0x16/0x1b
> Consensus seems to be something with large memory machines, lots of 
> dirty pages and a long writeout time due to ext3.
> 
> At the moment this the largest "usabillity" issue in the serversetup I'm 
> working with. Can there be done something to "autotune" it .. or perhaps 
> even fix it? .. or is it just to shift to xfs or wait for ext4?
> 

Hi Jesper,

What you are seeing looks awefully like the bug I have spent some time
to try to figure out in this bugzilla thread :

[Bug 12309] Large I/O operations result in slow performance and high
            iowait times
http://bugzilla.kernel.org/show_bug.cgi?id=12309

I created a fio test case out of a lttng trace to reproduce the problem
and created a patch to try to account the pages used by the i/o elevator
in the vm page count used to calculate memory pressure. Basically, the
behavior I was seeing is a constant increase of memory usage when doing
a dd-like write to disk until the memory fills up, which is indeed
wrong. The patch I posted in that thread seems to cause other problems
though, so probably we should teach kjournald to do better.

Here is the patch attempt :
http://bugzilla.kernel.org/attachment.cgi?id=20172

Here is the fio test case :
http://bugzilla.kernel.org/attachment.cgi?id=19894

My findings were this (I hope other people with deeper knowledge of
block layer/vm interaction can correct me) :

- Upon heavy and long disk writes, the pages used to back the buffers
  continuously increase as if there was no memory pressure at all.
  Therefore, I suspect they are held in a nowhere land that's unaccounted
  for at the vm layer (not part of memory pressure). That would seem to
  be the I/O elevator.

Can you give a try at the dd and fio test cases pointed out in the
bugzilla entry ? You may also want to see if my patch helps to partially
solve your problem. Another hint is to try to use the cgroups to
restrict you heavy I/O processes to a limited amount of memory;
although it does not solve the core of the problem, it made it disappear
for me. And of course trying to get a LTTng trace to get your head
around the problem can be very efficient. It's available as a git tree
over 2.6.29, and includes VFS, block I/O layer and vm instrumentation,
which helps looking at their interaction. All information is at
http://www.lttng.org.

Hoping this helps,

Mathieu

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68