Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

[Thomas Gleixner <tglx@linutronix.de>]

On Thu, 12 Jul 2012, Mike Galbraith wrote:
> On Thu, 2012-07-12 at 10:44 +0200, Mike Galbraith wrote: 
> > On Thu, 2012-07-12 at 07:47 +0200, Mike Galbraith wrote: 
> > > Greetings,
> > > 
> > > I'm chasing btrfs critters in an enterprise 3.0-rt kernel, and just
> > > checked to see if they're alive in virgin latest/greatest rt kernel.  
> > > 
> > > Both are indeed alive and well, ie I didn't break it, nor did the
> > > zillion patches in enterprise base kernel, so others may have an
> > > opportunity to meet these critters up close and personal as well.
> > 
> > 3.2-rt both explodes and deadlocks as well.  3.0-rt (virgin I mean) does
> > neither, so with enough re-integrate investment, it might be bisectable.
> 
> Nope, virgin 3.0-rt just didn't feel like it at the time.  Booted it
> again to run hefty test over lunch, it didn't survive 1 xfstests 006,
> much less hundreds.
> 
> crash> bt
> PID: 7604   TASK: ffff880174238b20  CPU: 0   COMMAND: "btrfs-worker-0"
>  #0 [ffff88017455d9c8] machine_kexec at ffffffff81025794
>  #1 [ffff88017455da28] crash_kexec at ffffffff8109781d
>  #2 [ffff88017455daf8] panic at ffffffff814a0661
>  #3 [ffff88017455db78] __try_to_take_rt_mutex at ffffffff81086d2f
>  #4 [ffff88017455dbc8] rt_spin_lock_slowlock at ffffffff814a2670
>  #5 [ffff88017455dca8] rt_spin_lock at ffffffff814a2db9
>  #6 [ffff88017455dcb8] schedule_bio at ffffffff81243133
>  #7 [ffff88017455dcf8] btrfs_map_bio at ffffffff812477be
>  #8 [ffff88017455dd68] __btree_submit_bio_done at ffffffff812152f6
>  #9 [ffff88017455dd78] run_one_async_done at ffffffff812148fa
> #10 [ffff88017455dd98] run_ordered_completions at ffffffff812493e8
> #11 [ffff88017455ddd8] worker_loop at ffffffff81249dc9
> #12 [ffff88017455de88] kthread at ffffffff81070266
> #13 [ffff88017455df48] kernel_thread_helper at ffffffff814a9be4
> crash> struct rt_mutex 0xffff880174530108
> struct rt_mutex {
>   wait_lock = {
>     raw_lock = {
>       slock = 7966
>     }
>   }, 
>   wait_list = {
>     node_list = {
>       next = 0xffff880175ecc970, 
>       prev = 0xffff880175ecc970
>     }, 
>     rawlock = 0xffff880175ecc968, 

Pointer into lala land again.

rawlock points to ...968 and the node_list to ...970.

struct rt_mutex {
        raw_spinlock_t          wait_lock;
        struct plist_head       wait_list;

The raw_lock pointer of the plist_head is initialized in
__rt_mutex_init() so it points to wait_lock. 

Can you check the offset of wait_list vs. the rt_mutex itself?

I wouldn't be surprised if it's exactly 8 bytes. And then this thing
looks like a copied lock with stale pointers to hell. Eew.

Thanks,

	tglx