Re: hunting an IO hang

[Chris Mason <chris.mason@oracle.com>]

Excerpts from Mel Gorman's message of 2011-01-17 05:27:44 -0500:
> On Sun, Jan 16, 2011 at 09:41:41PM -0500, Chris Mason wrote:
> > Excerpts from Andrew Morton's message of 2011-01-16 21:30:00 -0500:
> > > (lots of cc's added)
> > > 
> > > On Sun, 16 Jan 2011 21:07:40 -0500 Chris Mason <chris.mason@oracle.com> wrote:
> > > 
> > > > Excerpts from Linus Torvalds's message of 2011-01-16 20:53:04 -0500:
> > > > > .. except I actually didn't add Andrew to the cc after all.
> > > > > 
> > > > > NOW I did.
> > > > > 
> > > > > Oh, and if you can repeat this and bisect it, it would obviously be
> > > > > great. But that sounds rather painful.
> > > > 
> > > > Ok, so I've got 3 different problems in 3 totally different areas.
> > > > I'm running w/kvm, but this VM is very stable with 2.6.37.  Running
> > > > Linus' current git it goes boom in exotic ways, this time it was only on
> > > > ext3, btrfs code never loaded.
> > > > 
> > > > Linus, if you're planning on rc1 tonight I'll send my pull request out
> > > > the door.  Otherwise I'd prefer to fix this and send my pull after
> > > > actually getting a long btrfs run on the current code.
> > > > 
> > > > Next up, CONFIG_DEBUG*, always an adventure on rc1 kernels ;)
> > > > 
> > > > WARNING: at lib/list_debug.c:57 list_del+0xc0/0xed()
> > > > Hardware name: Bochs
> > > > list_del corruption. next->prev should be ffffea000010cde0, but was ffff88007cff6bc8
> > > > Modules linked in:
> > > > Pid: 524, comm: kswapd0 Not tainted 2.6.37-josef+ #180
> > > > Call Trace:
> > > >  [<ffffffff8106ec94>] ? warn_slowpath_common+0x85/0x9d
> > > >  [<ffffffff8106ed4f>] ? warn_slowpath_fmt+0x46/0x48
> > > >  [<ffffffff81263d6c>] ? list_del+0xc0/0xed
> > > >  [<ffffffff81106d9d>] ? migrate_pages+0x26f/0x357
> > > >  [<ffffffff81100e18>] ? compaction_alloc+0x0/0x2dc
> > > >  [<ffffffff8110150d>] ? compact_zone+0x391/0x5c4
> > > >  [<ffffffff81101905>] ? compact_zone_order+0xc2/0xd1
> > > >  [<ffffffff815c321e>] ? _raw_spin_unlock+0xe/0x10
> > > >  [<ffffffff810dc446>] ? kswapd+0x5c8/0x88f
> > > >  [<ffffffff810dbe7e>] ? kswapd+0x0/0x88f
> > > >  [<ffffffff81089ce8>] ? kthread+0x82/0x8a
> > > >  [<ffffffff810347d4>] ? kernel_thread_helper+0x4/0x10
> > > >  [<ffffffff81089c66>] ? kthread+0x0/0x8a
> > > >  [<ffffffff810347d0>] ? kernel_thread_helper+0x0/0x10
> > > > ---[ end trace 5c6b7933d16b301f ]---
> > > 
> > > uh-oh.  Does disabling CONFIG_COMPACTION make this go away (requires
> > > disabling CONFIG_TRANSPARENT_HUGEPAGE first).
> > 
> > We'll see.  I gave THP this same run of tests back in November, it
> > passed without any problems (after fixing the related btrfs migration
> > bug).  All of the crashes I've seen this weekend had this in the
> > .config:
> > 
> 
> I can't find the reset of the thread on any mailing list and am trying
> to reproduce the problem locally. What workload were you running?

I'm running a very basic IO stress test:

http://oss.oracle.com/~mason/stress.sh

The command line is stress.sh -n 50 -c /mnt/linux-2.6 /mnt

Which starts 50 processes that do cp -a /mnt/linux-2.6
/mnt/stress/$$.  Then they verify the result was correct and then they
delete it, forever in a loop.  In this case my linux-2.6 directory is a
full git tree with sources checked out.  No obj files though.

This was my crash from an overnight run with CONFIG_COMPACTION off:

# CONFIG_COMPACTION is not set
CONFIG_MIGRATION=y
CONFIG_DEBUG_PAGEALLOC=y
CONFIG_DEBUG_SLAB=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_SPINLOCK_SLEEP=y
CONFIG_DEBUG_VM=y
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_LIST=y
CONFIG_DEBUG_PAGEALLOC=y

I do have an NFS mount active during the run, but it isn't part of the
test at all.

I've also managed to get all the procs in the system stuck waiting for
IO requests.  It is possible these are two different bugs.  This
list_del oops hits faster if I run the test with a good deal of memory
pressure via an external memory hog.

------------[ cut here ]------------
WARNING: at lib/list_debug.c:54 list_del+0x97/0xed()
Hardware name: Bochs
list_del corruption. prev->next should be ffffea000116d478, but was ffffea00014ba2c8
Modules linked in: btrfs lzo_compress
Pid: 524, comm: kswapd0 Not tainted 2.6.37-josef+ #182
Call Trace:
 [<ffffffff8106edc1>] ? warn_slowpath_common+0x85/0x9d
 [<ffffffff8106ee7c>] ? warn_slowpath_fmt+0x46/0x48
 [<ffffffff81262e27>] ? list_del+0x97/0xed
 [<ffffffff810dbc59>] ? putback_lru_pages+0x7c/0x1eb
 [<ffffffff810dc070>] ? shrink_inactive_list+0x2a8/0x342
 [<ffffffff810dc676>] ? shrink_zone+0x327/0x3d6
 [<ffffffff8119a17a>] ? nfs_access_cache_shrinker+0x179/0x1a0
 [<ffffffff815c302e>] ? _raw_spin_unlock+0xe/0x10
 [<ffffffff810d2055>] ? zone_watermark_ok_safe+0xa9/0xb8
 [<ffffffff810dd26c>] ? kswapd+0x509/0x876
 [<ffffffff810dcd63>] ? kswapd+0x0/0x876
 [<ffffffff81089e40>] ? kthread+0x82/0x8a
 [<ffffffff810347d4>] ? kernel_thread_helper+0x4/0x10
 [<ffffffff81089dbe>] ? kthread+0x0/0x8a
 [<ffffffff810347d0>] ? kernel_thread_helper+0x0/0x10

-chris

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>