Re: 2.6.23-rc6-mm1 -- mkfs stuck in 'D'

Fengguang Wu
2007-09-23T01:21:06Z

On Sat, Sep 22, 2007 at 03:16:22PM +0200, Peter Zijlstra wrote:
> On Sat, 22 Sep 2007 09:55:09 +0800 Fengguang Wu <wfg@mail.ustc.edu.cn>
> wrote:
>
> > --- linux-2.6.22.orig/mm/page-writeback.c
> > +++ linux-2.6.22/mm/page-writeback.c
> > @@ -426,6 +426,14 @@ static void balance_dirty_pages(struct a
> > bdi_nr_writeback = bdi_stat(bdi, BDI_WRITEBACK);
> > }
> >
> > + printk(KERN_DEBUG "balance_dirty_pages written %lu %lu congested %d limits %lu %lu %lu %lu %lu %ld\n",
> > + pages_written,
> > + write_chunk - wbc.nr_to_write,
> > + bdi_write_congested(bdi),
> > + background_thresh, dirty_thresh,
> > + bdi_thresh, bdi_nr_reclaimable, bdi_nr_writeback,
> > + bdi_thresh - bdi_nr_reclaimable - bdi_nr_writeback);
> > +
> > if (bdi_nr_reclaimable + bdi_nr_writeback <= bdi_thresh)
> > break;
> > if (pages_written >= write_chunk)
> >
>
> > [ 1305.361511] balance_dirty_pages written 0 0 congested 0 limits 48869 195477 5801 5760 288 -247
>
> <snip long series of mostly identical lines>
>
> Could you perhaps instrument the writeback_inodes() path to see why
> nothing is written out? - the attached patch would be a nice start.

Curiously the lockup problem disappeared after upgrading to 2.6.23-rc6-mm1.
(need to watch it in a longer time window).

Anyway here's the output of your patch:
sb_locked 0
sb_empty 97011

> Most peculiar. It seems writeback_inodes() doesn't even attempt to
> write out stuff. Nor are outstanding writeback pages completed.

Still true. Another problem is that balance_dirty_pages() is being called even
when there are only 54 dirty pages. That could slow down writers unnecessarily.

balance_dirty_pages() should not be entered at all with small nr_dirty.

Look at these lines:
[ 197.471619] balance_dirty_pages for tar written 405 405 congested 0 global 196554 54 403 196097 bdi 0 0 398 -398
[ 197.472196] balance_dirty_pages for tar written 405 0 congested 0 global 196554 54 372 196128 bdi 0 0 380 -380
[ 197.472893] balance_dirty_pages for tar written 405 0 congested 0 global 196554 54 372 196128 bdi 23 0 369 -346
[ 197.473158] balance_dirty_pages for tar written 405 0 congested 0 global 196554 54 372 196128 bdi 23 0 366 -343
[ 197.473403] balance_dirty_pages for tar written 405 0 congested 0 global 196554 54 372 196128 bdi 23 0 365 -342
[ 197.473674] balance_dirty_pages for tar written 405 0 congested 0 global 196554 54 372 196128 bdi 23 0 364 -341
[ 197.474265] balance_dirty_pages for tar written 405 0 congested 0 global 196554 54 372 196128 bdi 23 0 362 -339
[ 197.475440] balance_dirty_pages for tar written 405 0 congested 0 global 196554 54 341 196159 bdi 47 0 327 -280
[ 197.476970] balance_dirty_pages for tar written 405 0 congested 0 global 196546 54 279 196213 bdi 95 0 279 -184
[ 197.477773] balance_dirty_pages for tar written 405 0 congested 0 global 196546 54 248 196244 bdi 95 0 255 -160
[ 197.479463] balance_dirty_pages for tar written 405 0 congested 0 global 196546 54 217 196275 bdi 143 0 210 -67
[ 197.479656] balance_dirty_pages for tar written 405 0 congested 0 global 196546 54 217 196275 bdi 143 0 209 -66
[ 197.481159] balance_dirty_pages for tar written 405 0 congested 0 global 196546 54 155 196337 bdi 167 0 163 4

The trace messages are generated by the following code:

--- linux-2.6.23-rc6-mm1.orig/mm/page-writeback.c
+++ linux-2.6.23-rc6-mm1/mm/page-writeback.c
@@ -421,6 +421,18 @@ static void balance_dirty_pages(struct a
bdi_nr_writeback = bdi_stat(bdi, BDI_WRITEBACK);
}

+ printk(KERN_DEBUG "balance_dirty_pages for %s written %lu %lu congested %d "
+ "global %lu %lu %lu %ld bdi %lu %lu %lu %ld\n",
+ current->comm,
+ pages_written, write_chunk - wbc.nr_to_write,
+ bdi_write_congested(bdi),
+ dirty_thresh,
+ global_dirty_unstable_pages(), global_page_state(NR_WRITEBACK),
+ dirty_thresh -
+ global_dirty_unstable_pages() - global_page_state(NR_WRITEBACK),
+ bdi_thresh, bdi_nr_reclaimable, bdi_nr_writeback,
+ bdi_thresh - bdi_nr_reclaimable - bdi_nr_writeback);
+
if (bdi_nr_reclaimable + bdi_nr_writeback <= bdi_thresh)
break;
if (pages_written >= write_chunk)