Re: bcache_gc: BUG: soft lockup

[Johannes Thumshirn <jthumshirn@suse.de>]

[ +cc Kent ]

On Wed, Jan 27, 2016 at 02:57:25PM +0000, Yannis Aribaud wrote:
> Hi,
> 
> After several weeks using the 4.2.6 kernel + patches from Ewheeler we just ran into a crash again.
> This time the kernel was still running and the server was responsive but not able to do any IO on the bcache devices.
> 
> [696983.683498] bcache_writebac D ffffffff810643df     0  5741      2 0x00000000
> [696983.683505]  ffff88103d01f180 0000000000000046 ffff88107842d000 ffffffff811a95cd
> [696983.683510]  0000000000000000 ffff8810388c4000 ffff88103d01f180 0000000000000001
> [696983.683514]  ffff882034ae0c10 0000000000000000 ffff882034ae0000 ffffffff8139601e
> [696983.683518] Call Trace:
> [696983.683530]  [<ffffffff811a95cd>] ? blk_queue_bio+0x262/0x279
> [696983.683539]  [<ffffffff8139601e>] ? schedule+0x6b/0x78
> [696983.683553]  [<ffffffffa032ce9b>] ? closure_sync+0x66/0x91 [bcache]
> [696983.683563]  [<ffffffffa033c89f>] ? bch_writeback_thread+0x622/0x6b5 [bcache]
> [696983.683569]  [<ffffffff8100265c>] ? __switch_to+0x1de/0x3f7
> [696983.683578]  [<ffffffffa033c89f>] ? bch_writeback_thread+0x622/0x6b5 [bcache]
> [696983.683586]  [<ffffffffa033c27d>] ? write_dirty_finish+0x1bf/0x1bf [bcache]
> [696983.683594]  [<ffffffff810589d6>] ? kthread+0x99/0xa1
> [696983.683598]  [<ffffffff8105893d>] ? kthread_parkme+0x16/0x16
> [696983.683603]  [<ffffffff813986df>] ? ret_from_fork+0x3f/0x70
> [696983.683607]  [<ffffffff8105893d>] ? kthread_parkme+0x16/0x16
> 
> Don't know if this help.
> Unfortunately I thing that we will rollback and stop using Bcache unless this is really fixed :/
> 

Hi Yannis,

Do you have a machine with a bcache setup running where you can reproduce the
error? Or do you know a method to reproduce the error?

What I'd be interested in is which locks are held when it locks up (you can
acquire this information with SysRq+d or echo d > /proc/sysrq-trigger.

Kent, do you have an idea what's happening here?

> Regards,
> 
> 7 décembre 2015 11:35 "Yannis Aribaud" <bugs@d6bell.net> a écrit:
> > Hi everyone,
> > 
> > It's been one week I'm using a 4.2.6 kernel merged with the Bcache patches from Ewheeler and no
> > signs of any kind of trouble I had before.
> > Thus it seems your patches fix my soft lockup issue.
> > It's currently running on one of my ceph nodes, I will certainly push it on the others during the
> > next weeks.
> > 
> > It would be great to merge thoses patches upstream since it seems that using Bcache in production
> > requires those fixes.
> > 
> > Anyway, thanks to all of you for your time, advices and work on Bcache. I'll keep you updated.
> > 
> > Regards,
> > -- 
> > Open is better
> -- 
> Open is better

-- 
Johannes Thumshirn                                          Storage
jthumshirn@suse.de                                +49 911 74053 689
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850