[GIT PULL] bcache fixes for 3.11

linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed

* [GIT PULL] bcache fixes for 3.11
@ 2013-08-14 22:59 Kent Overstreet
  2013-08-15  6:43 ` Stefan Priebe - Profihost AG
  0 siblings, 1 reply; 28+ messages in thread
From: Kent Overstreet @ 2013-08-14 22:59 UTC (permalink / raw)
  To: Jens Axboe; +Cc: linux-kernel, linux-bcache

Jens, here's the latest bcache fixes. Some urgent stuff in here:


The following changes since commit 79826c35eb99cd3c0873b8396f45fa26c87fb0b0:

  bcache: Allocation kthread fixes (2013-07-12 00:22:49 -0700)

are available in the git repository at:

  git://evilpiepirate.org/~kent/linux-bcache.git bcache-for-3.11

for you to fetch changes up to 0434a516e99ee51ac2d0dfa71b2f56c89ac5db05:

  bcache: Fix a flush/fua performance bug (2013-08-14 15:44:51 -0700)

----------------------------------------------------------------
Gabriel de Perthuis (1):
      bcache: Strip endline when writing the label through sysfs

Geert Uytterhoeven (1):
      bcache: Correct printf()-style format length modifier

Kent Overstreet (4):
      bcache: Fix a dumb journal discard bug
      bcache: Fix for when no journal entries are found
      bcache: Fix a writeback performance regression
      bcache: Fix a flush/fua performance bug

 drivers/md/bcache/bcache.h    |  7 +++----
 drivers/md/bcache/btree.c     |  2 +-
 drivers/md/bcache/journal.c   | 33 ++++++++++++++++++++-------------
 drivers/md/bcache/sysfs.c     |  9 +++++++--
 drivers/md/bcache/util.c      | 11 ++++++++++-
 drivers/md/bcache/util.h      | 12 +++++++++---
 drivers/md/bcache/writeback.c | 37 ++++++++++++++++++-------------------
 7 files changed, 68 insertions(+), 43 deletions(-)

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [GIT PULL] bcache fixes for 3.11
  2013-08-14 22:59 [GIT PULL] bcache fixes for 3.11 Kent Overstreet
@ 2013-08-15  6:43 ` Stefan Priebe - Profihost AG
  2013-08-15  7:07   ` Kent Overstreet
  2013-08-16 10:11   ` Stefan Priebe - Profihost AG
  0 siblings, 2 replies; 28+ messages in thread
From: Stefan Priebe - Profihost AG @ 2013-08-15  6:43 UTC (permalink / raw)
  To: Kent Overstreet; +Cc: Jens Axboe, linux-kernel, linux-bcache

Am 15.08.2013 00:59, schrieb Kent Overstreet:
> Jens, here's the latest bcache fixes. Some urgent stuff in here:
> 
> 
> The following changes since commit 79826c35eb99cd3c0873b8396f45fa26c87fb0b0:
> 
>   bcache: Allocation kthread fixes (2013-07-12 00:22:49 -0700)
> 
> are available in the git repository at:
> 
>   git://evilpiepirate.org/~kent/linux-bcache.git bcache-for-3.11
> 
> for you to fetch changes up to 0434a516e99ee51ac2d0dfa71b2f56c89ac5db05:
> 
>   bcache: Fix a flush/fua performance bug (2013-08-14 15:44:51 -0700)
> 
> ----------------------------------------------------------------
> Gabriel de Perthuis (1):
>       bcache: Strip endline when writing the label through sysfs
> 
> Geert Uytterhoeven (1):
>       bcache: Correct printf()-style format length modifier
> 
> Kent Overstreet (4):
>       bcache: Fix a dumb journal discard bug
>       bcache: Fix for when no journal entries are found
>       bcache: Fix a writeback performance regression
>       bcache: Fix a flush/fua performance bug
> 
>  drivers/md/bcache/bcache.h    |  7 +++----
>  drivers/md/bcache/btree.c     |  2 +-
>  drivers/md/bcache/journal.c   | 33 ++++++++++++++++++++-------------
>  drivers/md/bcache/sysfs.c     |  9 +++++++--
>  drivers/md/bcache/util.c      | 11 ++++++++++-
>  drivers/md/bcache/util.h      | 12 +++++++++---
>  drivers/md/bcache/writeback.c | 37 ++++++++++++++++++-------------------
>  7 files changed, 68 insertions(+), 43 deletions(-)
> --

As 3.10 is long term stable release. You might need CC the stable list
for patches which have to go to 3.10 as well.

At least this one should go to 3.10:
> Gabriel de Perthuis (1):
>       bcache: Strip endline when writing the label through sysfs

Thanks!

Greets,
Stefan


^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [GIT PULL] bcache fixes for 3.11
  2013-08-15  6:43 ` Stefan Priebe - Profihost AG
@ 2013-08-15  7:07   ` Kent Overstreet
  2013-08-16 10:11   ` Stefan Priebe - Profihost AG
  1 sibling, 0 replies; 28+ messages in thread
From: Kent Overstreet @ 2013-08-15  7:07 UTC (permalink / raw)
  To: Stefan Priebe - Profihost AG; +Cc: Jens Axboe, linux-kernel, linux-bcache

On Thu, Aug 15, 2013 at 08:43:22AM +0200, Stefan Priebe - Profihost AG wrote:
> Am 15.08.2013 00:59, schrieb Kent Overstreet:
> > Jens, here's the latest bcache fixes. Some urgent stuff in here:
> > 
> > 
> > The following changes since commit 79826c35eb99cd3c0873b8396f45fa26c87fb0b0:
> > 
> >   bcache: Allocation kthread fixes (2013-07-12 00:22:49 -0700)
> > 
> > are available in the git repository at:
> > 
> >   git://evilpiepirate.org/~kent/linux-bcache.git bcache-for-3.11
> > 
> > for you to fetch changes up to 0434a516e99ee51ac2d0dfa71b2f56c89ac5db05:
> > 
> >   bcache: Fix a flush/fua performance bug (2013-08-14 15:44:51 -0700)
> > 
> > ----------------------------------------------------------------
> > Gabriel de Perthuis (1):
> >       bcache: Strip endline when writing the label through sysfs
> > 
> > Geert Uytterhoeven (1):
> >       bcache: Correct printf()-style format length modifier
> > 
> > Kent Overstreet (4):
> >       bcache: Fix a dumb journal discard bug
> >       bcache: Fix for when no journal entries are found
> >       bcache: Fix a writeback performance regression
> >       bcache: Fix a flush/fua performance bug
> > 
> >  drivers/md/bcache/bcache.h    |  7 +++----
> >  drivers/md/bcache/btree.c     |  2 +-
> >  drivers/md/bcache/journal.c   | 33 ++++++++++++++++++++-------------
> >  drivers/md/bcache/sysfs.c     |  9 +++++++--
> >  drivers/md/bcache/util.c      | 11 ++++++++++-
> >  drivers/md/bcache/util.h      | 12 +++++++++---
> >  drivers/md/bcache/writeback.c | 37 ++++++++++++++++++-------------------
> >  7 files changed, 68 insertions(+), 43 deletions(-)
> > --
> 
> As 3.10 is long term stable release. You might need CC the stable list
> for patches which have to go to 3.10 as well.
> 
> At least this one should go to 3.10:
> > Gabriel de Perthuis (1):
> >       bcache: Strip endline when writing the label through sysfs

That patch is fixing a bug in code that was merged for 3.11 - the rest
are marked for stable :)

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: [GIT PULL] bcache fixes for 3.11
  2013-08-15  6:43 ` Stefan Priebe - Profihost AG
  2013-08-15  7:07   ` Kent Overstreet
@ 2013-08-16 10:11   ` Stefan Priebe - Profihost AG
  2013-08-18 22:09     ` bcache: Fix a writeback performance regression Stefan Priebe
  1 sibling, 1 reply; 28+ messages in thread
From: Stefan Priebe - Profihost AG @ 2013-08-16 10:11 UTC (permalink / raw)
  To: Kent Overstreet; +Cc: Jens Axboe, linux-kernel, linux-bcache

Hi,

bcache: Fix a writeback performance regression

this one results in 3.10 into hung tasks in bcache_writeback read_dirty.

Stefan
Am 15.08.2013 08:43, schrieb Stefan Priebe - Profihost AG:
> Am 15.08.2013 00:59, schrieb Kent Overstreet:
>> Jens, here's the latest bcache fixes. Some urgent stuff in here:
>>
>>
>> The following changes since commit 79826c35eb99cd3c0873b8396f45fa26c87fb0b0:
>>
>>   bcache: Allocation kthread fixes (2013-07-12 00:22:49 -0700)
>>
>> are available in the git repository at:
>>
>>   git://evilpiepirate.org/~kent/linux-bcache.git bcache-for-3.11
>>
>> for you to fetch changes up to 0434a516e99ee51ac2d0dfa71b2f56c89ac5db05:
>>
>>   bcache: Fix a flush/fua performance bug (2013-08-14 15:44:51 -0700)
>>
>> ----------------------------------------------------------------
>> Gabriel de Perthuis (1):
>>       bcache: Strip endline when writing the label through sysfs
>>
>> Geert Uytterhoeven (1):
>>       bcache: Correct printf()-style format length modifier
>>
>> Kent Overstreet (4):
>>       bcache: Fix a dumb journal discard bug
>>       bcache: Fix for when no journal entries are found
>>       bcache: Fix a writeback performance regression
>>       bcache: Fix a flush/fua performance bug
>>
>>  drivers/md/bcache/bcache.h    |  7 +++----
>>  drivers/md/bcache/btree.c     |  2 +-
>>  drivers/md/bcache/journal.c   | 33 ++++++++++++++++++++-------------
>>  drivers/md/bcache/sysfs.c     |  9 +++++++--
>>  drivers/md/bcache/util.c      | 11 ++++++++++-
>>  drivers/md/bcache/util.h      | 12 +++++++++---
>>  drivers/md/bcache/writeback.c | 37 ++++++++++++++++++-------------------
>>  7 files changed, 68 insertions(+), 43 deletions(-)
>> --
> 
> As 3.10 is long term stable release. You might need CC the stable list
> for patches which have to go to 3.10 as well.
> 
> At least this one should go to 3.10:
>> Gabriel de Perthuis (1):
>>       bcache: Strip endline when writing the label through sysfs
> 
> Thanks!
> 
> Greets,
> Stefan
> 

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-16 10:11   ` Stefan Priebe - Profihost AG
@ 2013-08-18 22:09     ` Stefan Priebe
  2013-08-19 22:27       ` Kent Overstreet
  0 siblings, 1 reply; 28+ messages in thread
From: Stefan Priebe @ 2013-08-18 22:09 UTC (permalink / raw)
  To: Kent Overstreet; +Cc: Jens Axboe, linux-kernel, linux-bcache


Vanilla 3.10.7 + bcache: Fix a writeback performance regression

http://pastebin.com/raw.php?i=LXZk4cMH

Stefan

Am 16.08.2013 12:11, schrieb Stefan Priebe - Profihost AG:
> Hi,
>
> bcache: Fix a writeback performance regression
>
> this one results in 3.10 into hung tasks in bcache_writeback read_dirty.
>
> Stefan
> Am 15.08.2013 08:43, schrieb Stefan Priebe - Profihost AG:
>> Am 15.08.2013 00:59, schrieb Kent Overstreet:
>>> Jens, here's the latest bcache fixes. Some urgent stuff in here:
>>>
>>>
>>> The following changes since commit 79826c35eb99cd3c0873b8396f45fa26c87fb0b0:
>>>
>>>    bcache: Allocation kthread fixes (2013-07-12 00:22:49 -0700)
>>>
>>> are available in the git repository at:
>>>
>>>    git://evilpiepirate.org/~kent/linux-bcache.git bcache-for-3.11
>>>
>>> for you to fetch changes up to 0434a516e99ee51ac2d0dfa71b2f56c89ac5db05:
>>>
>>>    bcache: Fix a flush/fua performance bug (2013-08-14 15:44:51 -0700)
>>>
>>> ----------------------------------------------------------------
>>> Gabriel de Perthuis (1):
>>>        bcache: Strip endline when writing the label through sysfs
>>>
>>> Geert Uytterhoeven (1):
>>>        bcache: Correct printf()-style format length modifier
>>>
>>> Kent Overstreet (4):
>>>        bcache: Fix a dumb journal discard bug
>>>        bcache: Fix for when no journal entries are found
>>>        bcache: Fix a writeback performance regression
>>>        bcache: Fix a flush/fua performance bug
>>>
>>>   drivers/md/bcache/bcache.h    |  7 +++----
>>>   drivers/md/bcache/btree.c     |  2 +-
>>>   drivers/md/bcache/journal.c   | 33 ++++++++++++++++++++-------------
>>>   drivers/md/bcache/sysfs.c     |  9 +++++++--
>>>   drivers/md/bcache/util.c      | 11 ++++++++++-
>>>   drivers/md/bcache/util.h      | 12 +++++++++---
>>>   drivers/md/bcache/writeback.c | 37 ++++++++++++++++++-------------------
>>>   7 files changed, 68 insertions(+), 43 deletions(-)
>>> --
>>
>> As 3.10 is long term stable release. You might need CC the stable list
>> for patches which have to go to 3.10 as well.
>>
>> At least this one should go to 3.10:
>>> Gabriel de Perthuis (1):
>>>        bcache: Strip endline when writing the label through sysfs
>>
>> Thanks!
>>
>> Greets,
>> Stefan
>>

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-18 22:09     ` bcache: Fix a writeback performance regression Stefan Priebe
@ 2013-08-19 22:27       ` Kent Overstreet
  2013-08-20  8:01         ` Stefan Priebe - Profihost AG
  0 siblings, 1 reply; 28+ messages in thread
From: Kent Overstreet @ 2013-08-19 22:27 UTC (permalink / raw)
  To: Stefan Priebe; +Cc: Jens Axboe, linux-kernel, linux-bcache

On Mon, Aug 19, 2013 at 12:09:24AM +0200, Stefan Priebe wrote:
> 
> Vanilla 3.10.7 + bcache: Fix a writeback performance regression
> 
> http://pastebin.com/raw.php?i=LXZk4cMH

Whoops, at first I thought this was the same bug as one I'd already been
chasing down that had been a harmless bug - turns out I didn't look
closely enough at the backtrace.

What happened is background writeback is deadlocking, because for some
reason the workqueue it's running out of is a singlethreaded workqueue,
so as soon as it decides to queue enough writeback bios that it has to
sleep on that semaphore (which often won't happen due to the PD
controller based ratelimiting) - boom, deadlock.

Here's the fixup patch I just tested and am applying:


>From 0af68de350e05e43fd093b36dcb0fe8aa838fabf Mon Sep 17 00:00:00 2001
From: Kent Overstreet <kmo@daterainc.com>
Date: Mon, 19 Aug 2013 15:26:22 -0700
Subject: [PATCH] bcache: Fix a writeback deadlock

Signed-off-by: Kent Overstreet <kmo@daterainc.com>

diff --git a/drivers/md/bcache/writeback.c b/drivers/md/bcache/writeback.c
index f88c62e..27ac519 100644
--- a/drivers/md/bcache/writeback.c
+++ b/drivers/md/bcache/writeback.c
@@ -351,7 +351,7 @@ static void write_dirty(struct closure *cl)
 
 	closure_bio_submit(&io->bio, cl, &io->dc->disk);
 
-	continue_at(cl, write_dirty_finish, dirty_wq);
+	continue_at(cl, write_dirty_finish, system_wq);
 }
 
 static void read_dirty_endio(struct bio *bio, int error)
@@ -371,7 +371,7 @@ static void read_dirty_submit(struct closure *cl)
 
 	closure_bio_submit(&io->bio, cl, &io->dc->disk);
 
-	continue_at(cl, write_dirty, dirty_wq);
+	continue_at(cl, write_dirty, system_wq);
 }
 
 static void read_dirty(struct closure *cl)
@@ -512,7 +512,7 @@ void bch_writeback_exit(void)
 
 int __init bch_writeback_init(void)
 {
-	dirty_wq = create_singlethread_workqueue("bcache_writeback");
+	dirty_wq = create_workqueue("bcache_writeback");
 	if (!dirty_wq)
 		return -ENOMEM;
 

^ permalink raw reply related	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-19 22:27       ` Kent Overstreet
@ 2013-08-20  8:01         ` Stefan Priebe - Profihost AG
  2013-08-20  8:07           ` Stefan Priebe - Profihost AG
  0 siblings, 1 reply; 28+ messages in thread
From: Stefan Priebe - Profihost AG @ 2013-08-20  8:01 UTC (permalink / raw)
  To: Kent Overstreet; +Cc: Jens Axboe, linux-kernel, linux-bcache

Am 20.08.2013 00:27, schrieb Kent Overstreet:
> On Mon, Aug 19, 2013 at 12:09:24AM +0200, Stefan Priebe wrote:
>>
>> Vanilla 3.10.7 + bcache: Fix a writeback performance regression
>>
>> http://pastebin.com/raw.php?i=LXZk4cMH
> 
> Whoops, at first I thought this was the same bug as one I'd already been
> chasing down that had been a harmless bug - turns out I didn't look
> closely enough at the backtrace.
> 
> What happened is background writeback is deadlocking, because for some
> reason the workqueue it's running out of is a singlethreaded workqueue,
> so as soon as it decides to queue enough writeback bios that it has to
> sleep on that semaphore (which often won't happen due to the PD
> controller based ratelimiting) - boom, deadlock.
> 
> Here's the fixup patch I just tested and am applying:

thanks, this works fine. The only issue i still have, is that the
dirty_date value starts at 0 and goes to a negative value after reboot.

Greets,
Stefan

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-20  8:01         ` Stefan Priebe - Profihost AG
@ 2013-08-20  8:07           ` Stefan Priebe - Profihost AG
  2013-08-21 21:10             ` Kent Overstreet
  2013-08-21 23:47             ` Kent Overstreet
  0 siblings, 2 replies; 28+ messages in thread
From: Stefan Priebe - Profihost AG @ 2013-08-20  8:07 UTC (permalink / raw)
  To: Kent Overstreet; +Cc: Jens Axboe, linux-kernel, linux-bcache

Am 20.08.2013 10:01, schrieb Stefan Priebe - Profihost AG:
> Am 20.08.2013 00:27, schrieb Kent Overstreet:
>> On Mon, Aug 19, 2013 at 12:09:24AM +0200, Stefan Priebe wrote:
>>>
>>> Vanilla 3.10.7 + bcache: Fix a writeback performance regression
>>>
>>> http://pastebin.com/raw.php?i=LXZk4cMH
>>
>> Whoops, at first I thought this was the same bug as one I'd already been
>> chasing down that had been a harmless bug - turns out I didn't look
>> closely enough at the backtrace.
>>
>> What happened is background writeback is deadlocking, because for some
>> reason the workqueue it's running out of is a singlethreaded workqueue,
>> so as soon as it decides to queue enough writeback bios that it has to
>> sleep on that semaphore (which often won't happen due to the PD
>> controller based ratelimiting) - boom, deadlock.
>>
>> Here's the fixup patch I just tested and am applying:

Oh i'm now seeing very high CPU spikes of kworker... i don't see if i
remove bcache: Fix a writeback performance regression.

Stefan

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-20  8:07           ` Stefan Priebe - Profihost AG
@ 2013-08-21 21:10             ` Kent Overstreet
  2013-08-21 23:47             ` Kent Overstreet
  1 sibling, 0 replies; 28+ messages in thread
From: Kent Overstreet @ 2013-08-21 21:10 UTC (permalink / raw)
  To: Stefan Priebe - Profihost AG; +Cc: Jens Axboe, linux-kernel, linux-bcache

On Tue, Aug 20, 2013 at 10:07:45AM +0200, Stefan Priebe - Profihost AG wrote:
> Am 20.08.2013 10:01, schrieb Stefan Priebe - Profihost AG:
> > Am 20.08.2013 00:27, schrieb Kent Overstreet:
> >> On Mon, Aug 19, 2013 at 12:09:24AM +0200, Stefan Priebe wrote:
> >>>
> >>> Vanilla 3.10.7 + bcache: Fix a writeback performance regression
> >>>
> >>> http://pastebin.com/raw.php?i=LXZk4cMH
> >>
> >> Whoops, at first I thought this was the same bug as one I'd already been
> >> chasing down that had been a harmless bug - turns out I didn't look
> >> closely enough at the backtrace.
> >>
> >> What happened is background writeback is deadlocking, because for some
> >> reason the workqueue it's running out of is a singlethreaded workqueue,
> >> so as soon as it decides to queue enough writeback bios that it has to
> >> sleep on that semaphore (which often won't happen due to the PD
> >> controller based ratelimiting) - boom, deadlock.
> >>
> >> Here's the fixup patch I just tested and am applying:
> 
> Oh i'm now seeing very high CPU spikes of kworker... i don't see if i
> remove bcache: Fix a writeback performance regression.

*swears*

I just saw that this morning, but I assumed it was a bug in my stripe
aware scan code that isn't in this branch yet - thanks for letting me
know that's not the case.

I shall work on a fix for the fix...

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-20  8:07           ` Stefan Priebe - Profihost AG
  2013-08-21 21:10             ` Kent Overstreet
@ 2013-08-21 23:47             ` Kent Overstreet
  2013-08-22  5:25               ` Stefan Priebe
  2013-08-22  5:27               ` Stefan Priebe
  1 sibling, 2 replies; 28+ messages in thread
From: Kent Overstreet @ 2013-08-21 23:47 UTC (permalink / raw)
  To: Stefan Priebe - Profihost AG; +Cc: Jens Axboe, linux-kernel, linux-bcache

On Tue, Aug 20, 2013 at 10:07:45AM +0200, Stefan Priebe - Profihost AG wrote:
> Am 20.08.2013 10:01, schrieb Stefan Priebe - Profihost AG:
> > Am 20.08.2013 00:27, schrieb Kent Overstreet:
> >> On Mon, Aug 19, 2013 at 12:09:24AM +0200, Stefan Priebe wrote:
> >>>
> >>> Vanilla 3.10.7 + bcache: Fix a writeback performance regression
> >>>
> >>> http://pastebin.com/raw.php?i=LXZk4cMH
> >>
> >> Whoops, at first I thought this was the same bug as one I'd already been
> >> chasing down that had been a harmless bug - turns out I didn't look
> >> closely enough at the backtrace.
> >>
> >> What happened is background writeback is deadlocking, because for some
> >> reason the workqueue it's running out of is a singlethreaded workqueue,
> >> so as soon as it decides to queue enough writeback bios that it has to
> >> sleep on that semaphore (which often won't happen due to the PD
> >> controller based ratelimiting) - boom, deadlock.
> >>
> >> Here's the fixup patch I just tested and am applying:
> 
> Oh i'm now seeing very high CPU spikes of kworker... i don't see if i
> remove bcache: Fix a writeback performance regression.

Are you able to reproduce it? I'm not having any luck reproducing it...

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-21 23:47             ` Kent Overstreet
@ 2013-08-22  5:25               ` Stefan Priebe
  2013-08-22  5:27               ` Stefan Priebe
  1 sibling, 0 replies; 28+ messages in thread
From: Stefan Priebe @ 2013-08-22  5:25 UTC (permalink / raw)
  To: Kent Overstreet; +Cc: Jens Axboe, linux-kernel, linux-bcache

Am 22.08.2013 01:47, schrieb Kent Overstreet:
> On Tue, Aug 20, 2013 at 10:07:45AM +0200, Stefan Priebe - Profihost AG wrote:
>> Am 20.08.2013 10:01, schrieb Stefan Priebe - Profihost AG:
>>> Am 20.08.2013 00:27, schrieb Kent Overstreet:
>>>> On Mon, Aug 19, 2013 at 12:09:24AM +0200, Stefan Priebe wrote:
>>>>>
>>>>> Vanilla 3.10.7 + bcache: Fix a writeback performance regression
>>>>>
>>>>> http://pastebin.com/raw.php?i=LXZk4cMH
>>>>
>>>> Whoops, at first I thought this was the same bug as one I'd already been
>>>> chasing down that had been a harmless bug - turns out I didn't look
>>>> closely enough at the backtrace.
>>>>
>>>> What happened is background writeback is deadlocking, because for some
>>>> reason the workqueue it's running out of is a singlethreaded workqueue,
>>>> so as soon as it decides to queue enough writeback bios that it has to
>>>> sleep on that semaphore (which often won't happen due to the PD
>>>> controller based ratelimiting) - boom, deadlock.
>>>>
>>>> Here's the fixup patch I just tested and am applying:
>>
>> Oh i'm now seeing very high CPU spikes of kworker... i don't see if i
>> remove bcache: Fix a writeback performance regression.
>
> Are you able to reproduce it? I'm not having any luck reproducing it...

Hi,

yes but only on production system it seems it happens when there is a 
big bunch in dirty and having heavy 4K random I/O.

Stefan

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-21 23:47             ` Kent Overstreet
  2013-08-22  5:25               ` Stefan Priebe
@ 2013-08-22  5:27               ` Stefan Priebe
  2013-08-22  5:43                 ` Kent Overstreet
  1 sibling, 1 reply; 28+ messages in thread
From: Stefan Priebe @ 2013-08-22  5:27 UTC (permalink / raw)
  To: Kent Overstreet; +Cc: Jens Axboe, linux-kernel, linux-bcache

today i had this one:

2013-08-22 06:28:55     [<ffffffff815247f4>] schedule+0x24/0x70
2013-08-22 06:28:55     [<ffffffff815248c7>] io_schedule+0x87/0xd0
2013-08-22 06:28:55     [<ffffffff810e6099>] sleep_on_page+0x9/0x10
2013-08-22 06:28:55     [<ffffffff81522dc7>] __wait_on_bit+0x57/0x80
2013-08-22 06:28:55     [<ffffffff810e5bbc>] ? find_get_pages_tag+0xcc/0x180
2013-08-22 06:28:55     [<ffffffff810e62ae>] wait_on_page_bit+0x6e/0x80
2013-08-22 06:28:55     [<ffffffff81070af0>] ? 
autoremove_wake_function+0x40/0x40
2013-08-22 06:28:55     [<ffffffff810f2660>] ? pagevec_lookup_tag+0x20/0x30
2013-08-22 06:28:55     [<ffffffff810e67ef>] 
filemap_fdatawait_range+0x10f/0x1b0
2013-08-22 06:28:55     [<ffffffff810e69b0>] 
filemap_write_and_wait_range+0x90/0xa0
2013-08-22 06:28:55     [<ffffffff811ef9c3>] xfs_file_fsync+0x63/0x1e0
2013-08-22 06:28:55     [<ffffffff8116d7c3>] vfs_fsync_range+0x23/0x30
2013-08-22 06:28:55     [<ffffffff8116d7e7>] vfs_fsync+0x17/0x20
2013-08-22 06:28:55     [<ffffffff8116da1c>] do_fsync+0x3c/0x60
2013-08-22 06:28:55     [<ffffffff8116da4e>] SyS_fdatasync+0xe/0x20
2013-08-22 06:28:55     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
2013-08-22 06:28:54     [<ffffffff815247f4>] schedule+0x24/0x70
2013-08-22 06:28:54     [<ffffffff815248c7>] io_schedule+0x87/0xd0
2013-08-22 06:28:54     [<ffffffff810e6099>] sleep_on_page+0x9/0x10
2013-08-22 06:28:54     [<ffffffff81522dc7>] __wait_on_bit+0x57/0x80
2013-08-22 06:28:54     [<ffffffff810e5bbc>] ? find_get_pages_tag+0xcc/0x180
2013-08-22 06:28:54     [<ffffffff810e62ae>] wait_on_page_bit+0x6e/0x80
2013-08-22 06:28:54     [<ffffffff81070af0>] ? 
autoremove_wake_function+0x40/0x40
2013-08-22 06:28:54     [<ffffffff810f2660>] ? pagevec_lookup_tag+0x20/0x30
2013-08-22 06:28:54     [<ffffffff810e67ef>] 
filemap_fdatawait_range+0x10f/0x1b0
2013-08-22 06:28:54     [<ffffffff810e69b0>] 
filemap_write_and_wait_range+0x90/0xa0
2013-08-22 06:28:54     [<ffffffff811ef9c3>] xfs_file_fsync+0x63/0x1e0
2013-08-22 06:28:54     [<ffffffff8116d7c3>] vfs_fsync_range+0x23/0x30
2013-08-22 06:28:54     [<ffffffff8116d7e7>] vfs_fsync+0x17/0x20
2013-08-22 06:28:54     [<ffffffff8116da1c>] do_fsync+0x3c/0x60
2013-08-22 06:28:54     [<ffffffff8116da6b>] SyS_fsync+0xb/0x10
2013-08-22 06:28:54     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
2013-08-22 06:28:54     INFO: task ceph-osd:3520 blocked for more than 
120 seconds.
2013-08-22 06:28:54     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-08-22 06:28:54     ceph-osd D ffffffff8160f760 0 3520 1 0x00000004
2013-08-22 06:28:54     ffff880c381d5c78 0000000000000086 
ffff880c381d5fd8 0000000000012c80
2013-08-22 06:28:54     ffff880c381d4010 0000000000012c80 
0000000000012c80 0000000000012c80
2013-08-22 06:28:54     ffff880c381d5fd8 0000000000012c80 
ffff880c39100000 ffff880c48dd0000
2013-08-22 06:28:54     Call Trace:
2013-08-22 06:28:54     [<ffffffff810e6090>] ? __lock_page+0x70/0x70
2013-08-22 06:28:53     [<ffffffff810e5bbc>] ? find_get_pages_tag+0xcc/0x180
2013-08-22 06:28:53     [<ffffffff810e62ae>] wait_on_page_bit+0x6e/0x80
2013-08-22 06:28:53     [<ffffffff81070af0>] ? 
autoremove_wake_function+0x40/0x40
2013-08-22 06:28:53     [<ffffffff810f2660>] ? pagevec_lookup_tag+0x20/0x30
2013-08-22 06:28:53     [<ffffffff810e67ef>] 
filemap_fdatawait_range+0x10f/0x1b0
2013-08-22 06:28:53     [<ffffffff815247f4>] ? schedule+0x24/0x70
2013-08-22 06:28:53     [<ffffffff81522b3d>] ? schedule_timeout+0x16d/0x200
2013-08-22 06:28:53     [<ffffffff81068e9a>] ? 
__queue_delayed_work+0xaa/0x1a0
2013-08-22 06:28:53     [<ffffffff81068cf9>] ? 
try_to_grab_pending+0x109/0x190
2013-08-22 06:28:53     [<ffffffff810e68b3>] filemap_fdatawait+0x23/0x30
2013-08-22 06:28:53     [<ffffffff81166e35>] wait_sb_inodes+0xb5/0x110
2013-08-22 06:28:53     [<ffffffff81167a1c>] sync_inodes_sb+0x9c/0xd0
2013-08-22 06:28:53     [<ffffffff8116dc66>] __sync_filesystem+0x16/0x50
2013-08-22 06:28:53     [<ffffffff8116dce3>] sync_filesystem+0x43/0x60
2013-08-22 06:28:53     [<ffffffff8116dd50>] SyS_syncfs+0x50/0x90
2013-08-22 06:28:53     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
2013-08-22 06:28:53     INFO: task ceph-osd:3516 blocked for more than 
120 seconds.
2013-08-22 06:28:53     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-08-22 06:28:53     ceph-osd D ffffffff8160f760 0 3516 1 0x00000004
2013-08-22 06:28:53     ffff880c383d3c78 0000000000000086 
ffff880c383d3fd8 0000000000012c80
2013-08-22 06:28:53     ffff880c383d2010 0000000000012c80 
0000000000012c80 0000000000012c80
2013-08-22 06:28:53     ffff880c383d3fd8 0000000000012c80 
ffff880c428018e0 ffff880c48dd0000
2013-08-22 06:28:53     Call Trace:
2013-08-22 06:28:53     [<ffffffff810e6090>] ? __lock_page+0x70/0x70
2013-08-22 06:28:52     [<ffffffff81219983>] ? xfs_bmbt_get_all+0x13/0x20
2013-08-22 06:28:52     [<ffffffff81208a14>] xfs_attr_set_int+0x3a4/0x4f0
2013-08-22 06:28:52     [<ffffffff81208be1>] xfs_attr_set+0x81/0x90
2013-08-22 06:28:52     [<ffffffff81201462>] xfs_xattr_set+0x42/0x60
2013-08-22 06:28:52     [<ffffffff8116257b>] generic_setxattr+0x9b/0xb0
2013-08-22 06:28:52     [<ffffffff8116376f>] __vfs_setxattr_noperm+0x5f/0xe0
2013-08-22 06:28:52     [<ffffffff811638a4>] vfs_setxattr+0xb4/0xc0
2013-08-22 06:28:52     [<ffffffff8116396e>] setxattr+0xbe/0x200
2013-08-22 06:28:52     [<ffffffff81143261>] ? __sb_start_write+0x71/0x110
2013-08-22 06:28:52     [<ffffffff8115ce51>] ? 
__mnt_want_write_file+0x61/0x80
2013-08-22 06:28:52     [<ffffffff81163b56>] SyS_fsetxattr+0xa6/0xc0
2013-08-22 06:28:52     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
2013-08-22 06:28:52     INFO: task ceph-osd:3508 blocked for more than 
120 seconds.
2013-08-22 06:28:52     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-08-22 06:28:52     ceph-osd D ffffffff8160f760 0 3508 1 0x00000004
2013-08-22 06:28:52     ffff880c2c1bfc28 0000000000000086 
ffff880c2c1bffd8 0000000000012c80
2013-08-22 06:28:52     ffff880c2c1be010 0000000000012c80 
0000000000012c80 0000000000012c80
2013-08-22 06:28:52     ffff880c2c1bffd8 0000000000012c80 
ffff880c43eae380 ffff880c48dd4aa0
2013-08-22 06:28:52     Call Trace:
2013-08-22 06:28:52     [<ffffffff810e6090>] ? __lock_page+0x70/0x70
2013-08-22 06:28:52     [<ffffffff815247f4>] schedule+0x24/0x70
2013-08-22 06:28:52     [<ffffffff815248c7>] io_schedule+0x87/0xd0
2013-08-22 06:28:52     [<ffffffff810e6099>] sleep_on_page+0x9/0x10
2013-08-22 06:28:52     [<ffffffff81522dc7>] __wait_on_bit+0x57/0x80
2013-08-22 06:28:51     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
2013-08-22 06:28:51     INFO: task ceph-osd:3467 blocked for more than 
120 seconds.
2013-08-22 06:28:51     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-08-22 06:28:51     ceph-osd D ffffffff8160f760 0 3467 1 0x00000004
2013-08-22 06:28:51     ffff880c375657e8 0000000000000086 
ffff880c37565fd8 0000000000012c80
2013-08-22 06:28:51     ffff880c37564010 0000000000012c80 
0000000000012c80 0000000000012c80
2013-08-22 06:28:51     ffff880c37565fd8 0000000000012c80 
ffff880c3850caa0 ffff880c48dd4aa0
2013-08-22 06:28:51     Call Trace:
2013-08-22 06:28:51     [<ffffffff81245efe>] ? xlog_bdstrat+0x1e/0x60
2013-08-22 06:28:51     [<ffffffff815247f4>] schedule+0x24/0x70
2013-08-22 06:28:51     [<ffffffff81522b3d>] schedule_timeout+0x16d/0x200
2013-08-22 06:28:51     [<ffffffff8124653d>] ? xlog_sync+0x2ed/0x490
2013-08-22 06:28:51     [<ffffffff81523b7a>] __down+0x6a/0xa0
2013-08-22 06:28:51     [<ffffffff811eb747>] ? _xfs_buf_find+0x107/0x290
2013-08-22 06:28:51     [<ffffffff810757ac>] down+0x3c/0x50
2013-08-22 06:28:51     [<ffffffff811eb070>] xfs_buf_lock+0x40/0xe0
2013-08-22 06:28:51     [<ffffffff811eb747>] _xfs_buf_find+0x107/0x290
2013-08-22 06:28:51     [<ffffffff811ebb1f>] xfs_buf_get_map+0x2f/0x180
2013-08-22 06:28:51     [<ffffffff8124dc68>] 
xfs_trans_get_buf_map+0xc8/0x160
2013-08-22 06:28:51     [<ffffffff81224093>] xfs_da_get_buf+0xb3/0xe0
2013-08-22 06:28:51     [<ffffffff81209b20>] 
xfs_attr3_leaf_to_node+0xb0/0x240
2013-08-22 06:28:51     [<ffffffff8120d12b>] ? 
xfs_attr3_leaf_add+0x15b/0x1c0
2013-08-22 06:28:51     [<ffffffff8120843c>] 
xfs_attr_leaf_addname+0x14c/0x380
2013-08-22 06:28:51     [<ffffffff81243f68>] ? xfs_trans_add_item+0x28/0x70
2013-08-22 06:28:50     [<ffffffff810e8420>] ? mempool_alloc_slab+0x10/0x20
2013-08-22 06:28:50     [<ffffffff815239c2>] ? down_read+0x12/0x20
2013-08-22 06:28:50     [<ffffffffa010c38a>] request_write+0x8a/0x380 
[bcache]
2013-08-22 06:28:50     [<ffffffffa010c8fb>] 
cached_dev_make_request+0x27b/0x350 [bcache]
2013-08-22 06:28:50     [<ffffffff81281982>] generic_make_request+0xc2/0x100
2013-08-22 06:28:50     [<ffffffff81281a27>] submit_bio+0x67/0x130
2013-08-22 06:28:50     [<ffffffff811e823f>] xfs_submit_ioend_bio+0x2f/0x40
2013-08-22 06:28:50     [<ffffffff811e8314>] xfs_submit_ioend+0xc4/0x130
2013-08-22 06:28:50     [<ffffffff811e98bb>] xfs_vm_writepage+0x2eb/0x590
2013-08-22 06:28:50     [<ffffffff810ef752>] __writepage+0x12/0x40
2013-08-22 06:28:50     [<ffffffff810f090b>] write_cache_pages+0x23b/0x4e0
2013-08-22 06:28:50     [<ffffffff81524437>] ? __schedule+0x3d7/0x6d0
2013-08-22 06:28:50     [<ffffffff810ef740>] ? set_page_dirty+0x60/0x60
2013-08-22 06:28:50     [<ffffffff810f0bfc>] generic_writepages+0x4c/0x70
2013-08-22 06:28:50     [<ffffffff811e810f>] xfs_vm_writepages+0x4f/0x60
2013-08-22 06:28:50     [<ffffffff810f0c3b>] do_writepages+0x1b/0x40
2013-08-22 06:28:50     [<ffffffff810e6911>] 
__filemap_fdatawrite_range+0x51/0x60
2013-08-22 06:28:50     [<ffffffff81097cf0>] ? futex_wake+0x110/0x120
2013-08-22 06:28:50     [<ffffffff810e699a>] 
filemap_write_and_wait_range+0x7a/0xa0
2013-08-22 06:28:50     [<ffffffff811ef9c3>] xfs_file_fsync+0x63/0x1e0
2013-08-22 06:28:50     [<ffffffff8116d7c3>] vfs_fsync_range+0x23/0x30
2013-08-22 06:28:50     [<ffffffff8116d7e7>] vfs_fsync+0x17/0x20
2013-08-22 06:28:50     [<ffffffff8116da1c>] do_fsync+0x3c/0x60
2013-08-22 06:28:50     [<ffffffff8116da4e>] SyS_fdatasync+0xe/0x20
2013-08-22 06:28:49     [<ffffffff810ef740>] ? set_page_dirty+0x60/0x60
2013-08-22 06:28:49     [<ffffffff810f0bfc>] generic_writepages+0x4c/0x70
2013-08-22 06:28:49     [<ffffffff811639a3>] ? setxattr+0xf3/0x200
2013-08-22 06:28:49     [<ffffffff811e810f>] xfs_vm_writepages+0x4f/0x60
2013-08-22 06:28:49     [<ffffffff810f0c3b>] do_writepages+0x1b/0x40
2013-08-22 06:28:49     [<ffffffff810e6911>] 
__filemap_fdatawrite_range+0x51/0x60
2013-08-22 06:28:49     [<ffffffff810e699a>] 
filemap_write_and_wait_range+0x7a/0xa0
2013-08-22 06:28:49     [<ffffffff811ef9c3>] xfs_file_fsync+0x63/0x1e0
2013-08-22 06:28:49     [<ffffffff8116d7c3>] vfs_fsync_range+0x23/0x30
2013-08-22 06:28:49     [<ffffffff8116d7e7>] vfs_fsync+0x17/0x20
2013-08-22 06:28:49     [<ffffffff8116da1c>] do_fsync+0x3c/0x60
2013-08-22 06:28:49     [<ffffffff8116da6b>] SyS_fsync+0xb/0x10
2013-08-22 06:28:49     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
2013-08-22 06:28:49     INFO: task ceph-osd:3483 blocked for more than 
120 seconds.
2013-08-22 06:28:49     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-08-22 06:28:49     ceph-osd D ffffffff8160f760 0 3483 1 0x00000004
2013-08-22 06:28:49     ffff880c2c0d1910 0000000000000086 
ffff880c2c0d1fd8 0000000000012c80
2013-08-22 06:28:49     ffff880c2c0d0010 0000000000012c80 
0000000000012c80 0000000000012c80
2013-08-22 06:28:49     ffff880c2c0d1fd8 0000000000012c80 
ffff880c3a14caa0 ffff880c48dd31c0
2013-08-22 06:28:49     Call Trace:
2013-08-22 06:28:49     [<ffffffff815247f4>] schedule+0x24/0x70
2013-08-22 06:28:49     [<ffffffff81525cfd>] 
rwsem_down_read_failed+0x9d/0xe5
2013-08-22 06:28:49     [<ffffffff812aa024>] 
call_rwsem_down_read_failed+0x14/0x30
2013-08-22 06:28:48     [<ffffffff81143261>] ? __sb_start_write+0x71/0x110
2013-08-22 06:28:48     [<ffffffff8115ce51>] ? 
__mnt_want_write_file+0x61/0x80
2013-08-22 06:28:48     [<ffffffff81163b56>] SyS_fsetxattr+0xa6/0xc0
2013-08-22 06:28:48     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
2013-08-22 06:28:48     INFO: task ceph-osd:3482 blocked for more than 
120 seconds.
2013-08-22 06:28:48     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-08-22 06:28:48     ceph-osd D ffffffff8160f760 0 3482 1 0x00000004
2013-08-22 06:28:48     ffff880c2c0cf910 0000000000000086 
ffff880c2c0cffd8 0000000000012c80
2013-08-22 06:28:48     ffff880c2c0ce010 0000000000012c80 
0000000000012c80 0000000000012c80
2013-08-22 06:28:48     ffff880c2c0cffd8 0000000000012c80 
ffff880c47768000 ffff880c48dd0000
2013-08-22 06:28:48     Call Trace:
2013-08-22 06:28:48     [<ffffffff815247f4>] schedule+0x24/0x70
2013-08-22 06:28:48     [<ffffffff81525cfd>] 
rwsem_down_read_failed+0x9d/0xe5
2013-08-22 06:28:48     [<ffffffff812aa024>] 
call_rwsem_down_read_failed+0x14/0x30
2013-08-22 06:28:48     [<ffffffff810e8420>] ? mempool_alloc_slab+0x10/0x20
2013-08-22 06:28:48     [<ffffffff815239c2>] ? down_read+0x12/0x20
2013-08-22 06:28:48     [<ffffffffa010c38a>] request_write+0x8a/0x380 
[bcache]
2013-08-22 06:28:48     [<ffffffffa010c8fb>] 
cached_dev_make_request+0x27b/0x350 [bcache]
2013-08-22 06:28:48     [<ffffffff81281982>] generic_make_request+0xc2/0x100
2013-08-22 06:28:48     [<ffffffff81281a27>] submit_bio+0x67/0x130
2013-08-22 06:28:48     [<ffffffff811e823f>] xfs_submit_ioend_bio+0x2f/0x40
2013-08-22 06:28:48     [<ffffffff811e833e>] xfs_submit_ioend+0xee/0x130
2013-08-22 06:28:48     [<ffffffff811e98bb>] xfs_vm_writepage+0x2eb/0x590
2013-08-22 06:28:48     [<ffffffff810ef752>] __writepage+0x12/0x40
2013-08-22 06:28:48     [<ffffffff810f090b>] write_cache_pages+0x23b/0x4e0
2013-08-22 06:28:47     [<ffffffff812469b6>] _xfs_log_force+0x116/0x250
2013-08-22 06:28:47     [<ffffffff8124836a>] xfs_log_force+0x2a/0x90
2013-08-22 06:28:47     [<ffffffff811eb747>] ? _xfs_buf_find+0x107/0x290
2013-08-22 06:28:47     [<ffffffff811eb0a3>] xfs_buf_lock+0x73/0xe0
2013-08-22 06:28:47     [<ffffffff811eb747>] _xfs_buf_find+0x107/0x290
2013-08-22 06:28:47     [<ffffffff811ebb1f>] xfs_buf_get_map+0x2f/0x180
2013-08-22 06:28:47     [<ffffffff8124dc68>] 
xfs_trans_get_buf_map+0xc8/0x160
2013-08-22 06:28:47     [<ffffffff81224093>] xfs_da_get_buf+0xb3/0xe0
2013-08-22 06:28:47     [<ffffffff81209b20>] 
xfs_attr3_leaf_to_node+0xb0/0x240
2013-08-22 06:28:47     [<ffffffff8120d12b>] ? 
xfs_attr3_leaf_add+0x15b/0x1c0
2013-08-22 06:28:47     [<ffffffff8120843c>] 
xfs_attr_leaf_addname+0x14c/0x380
2013-08-22 06:28:47     [<ffffffff81243f68>] ? xfs_trans_add_item+0x28/0x70
2013-08-22 06:28:47     [<ffffffff81219983>] ? xfs_bmbt_get_all+0x13/0x20
2013-08-22 06:28:47     [<ffffffff81208a14>] xfs_attr_set_int+0x3a4/0x4f0
2013-08-22 06:28:47     [<ffffffff81208be1>] xfs_attr_set+0x81/0x90
2013-08-22 06:28:47     [<ffffffff81201462>] xfs_xattr_set+0x42/0x60
2013-08-22 06:28:47     [<ffffffff8116257b>] generic_setxattr+0x9b/0xb0
2013-08-22 06:28:47     [<ffffffff8116376f>] __vfs_setxattr_noperm+0x5f/0xe0
2013-08-22 06:28:47     [<ffffffff811638a4>] vfs_setxattr+0xb4/0xc0
2013-08-22 06:28:47     [<ffffffff8116396e>] setxattr+0xbe/0x200
2013-08-22 06:28:47     [<ffffffff811ec1ca>] xfs_buf_iorequest+0x4a/0xa0
2013-08-22 06:28:47     [<ffffffff81245efe>] xlog_bdstrat+0x1e/0x60
2013-08-22 06:28:47     [<ffffffff8124653d>] xlog_sync+0x2ed/0x490
2013-08-22 06:28:47     [<ffffffff8124686b>] 
xlog_state_release_iclog+0x8b/0xc0
2013-08-22 06:28:46     [<ffffffff8116dd50>] SyS_syncfs+0x50/0x90
2013-08-22 06:28:46     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
2013-08-22 06:28:46     INFO: task ceph-osd:3481 blocked for more than 
120 seconds.
2013-08-22 06:28:46     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-08-22 06:28:46     ceph-osd D ffffffff8160f760 0 3481 1 0x00000004
2013-08-22 06:28:46     ffff880c2c0cd500 0000000000000086 
ffff880c2c0cdfd8 0000000000012c80
2013-08-22 06:28:46     ffff880c2c0cc010 0000000000012c80 
0000000000012c80 0000000000012c80
2013-08-22 06:28:46     ffff880c2c0cdfd8 0000000000012c80 
ffff880c4776b1c0 ffff880c48dd4aa0
2013-08-22 06:28:46     Call Trace:
2013-08-22 06:28:46     [<ffffffff8124e5b6>] ? xfs_trans_brelse+0xb6/0x100
2013-08-22 06:28:46     [<ffffffff815247f4>] schedule+0x24/0x70
2013-08-22 06:28:46     [<ffffffff81525cfd>] 
rwsem_down_read_failed+0x9d/0xe5
2013-08-22 06:28:46     [<ffffffff812aa024>] 
call_rwsem_down_read_failed+0x14/0x30
2013-08-22 06:28:46     [<ffffffff810e8420>] ? mempool_alloc_slab+0x10/0x20
2013-08-22 06:28:46     [<ffffffff815239c2>] ? down_read+0x12/0x20
2013-08-22 06:28:46     [<ffffffffa010c38a>] request_write+0x8a/0x380 
[bcache]
2013-08-22 06:28:46     [<ffffffffa010c8fb>] 
cached_dev_make_request+0x27b/0x350 [bcache]
2013-08-22 06:28:46     [<ffffffff81281982>] generic_make_request+0xc2/0x100
2013-08-22 06:28:46     [<ffffffff81281a27>] submit_bio+0x67/0x130
2013-08-22 06:28:46     [<ffffffff811eabf0>] xfs_buf_ioapply_map+0x180/0x1d0
2013-08-22 06:28:46     [<ffffffff811ead47>] _xfs_buf_ioapply+0x107/0x150
2013-08-22 06:28:46     [<ffffffff81245efe>] ? xlog_bdstrat+0x1e/0x60
2013-08-22 06:28:45     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-08-22 06:28:45     ceph-osd D ffffffff8160f760 0 3462 1 0x00000004
2013-08-22 06:28:45     ffff880c375cdc28 0000000000000086 
ffff880c375cdfd8 0000000000012c80
2013-08-22 06:28:45     ffff880c375cc010 0000000000012c80 
0000000000012c80 0000000000012c80
2013-08-22 06:28:45     ffff880c375cdfd8 0000000000012c80 
ffff880c46318000 ffff880c48dd6380
2013-08-22 06:28:45     Call Trace:
2013-08-22 06:28:45     [<ffffffff810e6090>] ? __lock_page+0x70/0x70
2013-08-22 06:28:45     [<ffffffff815247f4>] schedule+0x24/0x70
2013-08-22 06:28:45     [<ffffffff815248c7>] io_schedule+0x87/0xd0
2013-08-22 06:28:45     [<ffffffff810e6099>] sleep_on_page+0x9/0x10
2013-08-22 06:28:45     [<ffffffff81522dc7>] __wait_on_bit+0x57/0x80
2013-08-22 06:28:45     [<ffffffff810e5bbc>] ? find_get_pages_tag+0xcc/0x180
2013-08-22 06:28:45     [<ffffffff810e62ae>] wait_on_page_bit+0x6e/0x80
2013-08-22 06:28:45     [<ffffffff81070af0>] ? 
autoremove_wake_function+0x40/0x40
2013-08-22 06:28:45     [<ffffffff810f2660>] ? pagevec_lookup_tag+0x20/0x30
2013-08-22 06:28:45     [<ffffffff810e67ef>] 
filemap_fdatawait_range+0x10f/0x1b0
2013-08-22 06:28:45     [<ffffffff815247f4>] ? schedule+0x24/0x70
2013-08-22 06:28:45     [<ffffffff81522b3d>] ? schedule_timeout+0x16d/0x200
2013-08-22 06:28:45     [<ffffffff81068e9a>] ? 
__queue_delayed_work+0xaa/0x1a0
2013-08-22 06:28:45     [<ffffffff810e68b3>] filemap_fdatawait+0x23/0x30
2013-08-22 06:28:45     [<ffffffff81166e35>] wait_sb_inodes+0xb5/0x110
2013-08-22 06:28:45     [<ffffffff81167a1c>] sync_inodes_sb+0x9c/0xd0
2013-08-22 06:28:45     [<ffffffff8116dc66>] __sync_filesystem+0x16/0x50
2013-08-22 06:28:45     [<ffffffff8116dce3>] sync_filesystem+0x43/0x60
2013-08-22 06:28:44     [<ffffffffa010c8fb>] 
cached_dev_make_request+0x27b/0x350 [bcache]
2013-08-22 06:28:44     [<ffffffff81281982>] generic_make_request+0xc2/0x100
2013-08-22 06:28:44     [<ffffffff81281a27>] submit_bio+0x67/0x130
2013-08-22 06:28:44     [<ffffffff811e823f>] xfs_submit_ioend_bio+0x2f/0x40
2013-08-22 06:28:44     [<ffffffff811e833e>] xfs_submit_ioend+0xee/0x130
2013-08-22 06:28:44     [<ffffffff811e98bb>] xfs_vm_writepage+0x2eb/0x590
2013-08-22 06:28:44     [<ffffffff810ef752>] __writepage+0x12/0x40
2013-08-22 06:28:44     [<ffffffff810f090b>] write_cache_pages+0x23b/0x4e0
2013-08-22 06:28:44     [<ffffffff81524437>] ? __schedule+0x3d7/0x6d0
2013-08-22 06:28:44     [<ffffffff810ef740>] ? set_page_dirty+0x60/0x60
2013-08-22 06:28:44     [<ffffffff810f0bfc>] generic_writepages+0x4c/0x70
2013-08-22 06:28:44     [<ffffffff811e810f>] xfs_vm_writepages+0x4f/0x60
2013-08-22 06:28:44     [<ffffffff810f0c3b>] do_writepages+0x1b/0x40
2013-08-22 06:28:44     [<ffffffff810e6911>] 
__filemap_fdatawrite_range+0x51/0x60
2013-08-22 06:28:44     [<ffffffff81097cf0>] ? futex_wake+0x110/0x120
2013-08-22 06:28:44     [<ffffffff810e699a>] 
filemap_write_and_wait_range+0x7a/0xa0
2013-08-22 06:28:44     [<ffffffff811ef9c3>] xfs_file_fsync+0x63/0x1e0
2013-08-22 06:28:44     [<ffffffff8116d7c3>] vfs_fsync_range+0x23/0x30
2013-08-22 06:28:44     [<ffffffff8116d7e7>] vfs_fsync+0x17/0x20
2013-08-22 06:28:44     [<ffffffff8116da1c>] do_fsync+0x3c/0x60
2013-08-22 06:28:44     [<ffffffff8116da4e>] SyS_fdatasync+0xe/0x20
2013-08-22 06:28:44     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
2013-08-22 06:28:44     INFO: task ceph-osd:3462 blocked for more than 
120 seconds.
2013-08-22 06:28:43     INFO: task ceph-osd:24130 blocked for more than 
120 seconds.
2013-08-22 06:28:43     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-08-22 06:28:43     ceph-osd D ffffffff8160f760 0 24130 1 0x00000004
2013-08-22 06:28:43     ffff880c27fe98d8 0000000000000082 
ffff880c27fe9fd8 0000000000012c80
2013-08-22 06:28:43     ffff880c27fe8010 0000000000012c80 
0000000000012c80 0000000000012c80
2013-08-22 06:28:43     ffff880c27fe9fd8 0000000000012c80 
ffff880c215e6380 ffff880c48decaa0
2013-08-22 06:28:43     Call Trace:
2013-08-22 06:28:43     [<ffffffff81170430>] ? __wait_on_buffer+0x30/0x30
2013-08-22 06:28:43     [<ffffffff815247f4>] schedule+0x24/0x70
2013-08-22 06:28:43     [<ffffffff815248c7>] io_schedule+0x87/0xd0
2013-08-22 06:28:43     [<ffffffff81170439>] sleep_on_buffer+0x9/0x10
2013-08-22 06:28:43     [<ffffffff81522dc7>] __wait_on_bit+0x57/0x80
2013-08-22 06:28:43     [<ffffffff81170430>] ? __wait_on_buffer+0x30/0x30
2013-08-22 06:28:43     [<ffffffff81522e63>] 
out_of_line_wait_on_bit+0x73/0x90
2013-08-22 06:28:43     [<ffffffff81070af0>] ? 
autoremove_wake_function+0x40/0x40
2013-08-22 06:28:43     [<ffffffff8116f64b>] ? submit_bh+0xb/0x10
2013-08-22 06:28:43     [<ffffffff81170426>] __wait_on_buffer+0x26/0x30
2013-08-22 06:28:43     [<ffffffff81172803>] __block_write_begin+0x383/0x530
2013-08-22 06:28:43     [<ffffffff811e9050>] ? 
xfs_get_blocks_direct+0x20/0x20
2013-08-22 06:28:43     [<ffffffff810e6d0f>] ? 
grab_cache_page_write_begin+0x9f/0xd0
2013-08-22 06:28:43     [<ffffffff811e802f>] xfs_vm_write_begin+0x5f/0xd0
2013-08-22 06:28:43     [<ffffffff810e55ba>] 
generic_perform_write+0xca/0x200
2013-08-22 06:28:43     [<ffffffff810e574f>] 
generic_file_buffered_write+0x5f/0x90
2013-08-22 06:28:43     [<ffffffff811f0275>] 
xfs_file_buffered_aio_write+0xf5/0x180
2013-08-22 06:28:43     [<ffffffff811f03ce>] xfs_file_aio_write+0xce/0x150
2013-08-22 06:28:43     [<ffffffff8114ac8d>] ? path_put+0x1d/0x30
2013-08-22 06:28:43     [<ffffffff81140788>] do_sync_readv_writev+0x68/0xa0
2013-08-22 06:28:43     [<ffffffff81142052>] do_readv_writev+0xf2/0x2e0
2013-08-22 06:28:43     [<ffffffff811f0300>] ? 
xfs_file_buffered_aio_write+0x180/0x180
2013-08-22 06:28:43     [<ffffffff811407c0>] ? 
do_sync_readv_writev+0xa0/0xa0
2013-08-22 06:28:43     [<ffffffff8114227e>] vfs_writev+0x3e/0x60
2013-08-22 06:28:43     [<ffffffff811423ca>] SyS_writev+0x5a/0xc0
2013-08-22 06:28:43     [<ffffffff811410d3>] ? SyS_lseek+0x53/0x80
2013-08-22 06:28:43     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
2013-08-22 06:28:43     INFO: task ceph-osd:24134 blocked for more than 
120 seconds.
2013-08-22 06:28:43     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-08-22 06:28:43     ceph-osd D ffffffff8160f760 0 24134 1 0x00000004
2013-08-22 06:28:43     ffff880c23d31708 0000000000000082 
ffff880c23d31fd8 0000000000012c80
2013-08-22 06:28:43     ffff880c23d30010 0000000000012c80 
0000000000012c80 0000000000012c80
2013-08-22 06:28:43     ffff880c23d31fd8 0000000000012c80 
ffff880c361e0000 ffff880c48dd4aa0
2013-08-22 06:28:43     Call Trace:
2013-08-22 06:28:43     [<ffffffff815247f4>] schedule+0x24/0x70
2013-08-22 06:28:43     [<ffffffff81524ae9>] 
schedule_preempt_disabled+0x9/0x10
2013-08-22 06:28:43     [<ffffffff81523664>] 
__mutex_lock_slowpath+0x194/0x240
2013-08-22 06:28:43     [<ffffffff81522e9e>] mutex_lock+0x1e/0x40
2013-08-22 06:28:43     [<ffffffffa00f891b>] 
bch_bucket_alloc_set+0x3b/0x80 [bcache]
2013-08-22 06:28:43     [<ffffffffa010ac1c>] 
bch_alloc_sectors+0x12c/0x4e0 [bcache]
2013-08-22 06:28:43     [<ffffffffa0104e15>] ? 
bch_generic_make_request_hack+0x25/0xc0 [bcache]
2013-08-22 06:28:43     [<ffffffffa0105013>] ? 
__bch_submit_bbio+0x73/0x80 [bcache]
2013-08-22 06:28:43     [<ffffffffa010bc98>] 
bch_insert_data_loop+0xf8/0x610 [bcache]
2013-08-22 06:28:43     [<ffffffffa010a335>] ? 
bch_get_congested+0x25/0x70 [bcache]
2013-08-22 06:28:43     [<ffffffffa010c1cd>] bch_insert_data+0x1d/0x20 
[bcache]
2013-08-22 06:28:43     [<ffffffffa0116d13>] closure_queue+0x43/0x60 
[bcache]
2013-08-22 06:28:43     [<ffffffffa010c49e>] request_write+0x19e/0x380 
[bcache]

Am 22.08.2013 01:47, schrieb Kent Overstreet:
> On Tue, Aug 20, 2013 at 10:07:45AM +0200, Stefan Priebe - Profihost AG wrote:
>> Am 20.08.2013 10:01, schrieb Stefan Priebe - Profihost AG:
>>> Am 20.08.2013 00:27, schrieb Kent Overstreet:
>>>> On Mon, Aug 19, 2013 at 12:09:24AM +0200, Stefan Priebe wrote:
>>>>>
>>>>> Vanilla 3.10.7 + bcache: Fix a writeback performance regression
>>>>>
>>>>> http://pastebin.com/raw.php?i=LXZk4cMH
>>>>
>>>> Whoops, at first I thought this was the same bug as one I'd already been
>>>> chasing down that had been a harmless bug - turns out I didn't look
>>>> closely enough at the backtrace.
>>>>
>>>> What happened is background writeback is deadlocking, because for some
>>>> reason the workqueue it's running out of is a singlethreaded workqueue,
>>>> so as soon as it decides to queue enough writeback bios that it has to
>>>> sleep on that semaphore (which often won't happen due to the PD
>>>> controller based ratelimiting) - boom, deadlock.
>>>>
>>>> Here's the fixup patch I just tested and am applying:
>>
>> Oh i'm now seeing very high CPU spikes of kworker... i don't see if i
>> remove bcache: Fix a writeback performance regression.
>
> Are you able to reproduce it? I'm not having any luck reproducing it...
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-22  5:27               ` Stefan Priebe
@ 2013-08-22  5:43                 ` Kent Overstreet
  2013-08-22  5:59                   ` Stefan Priebe
  0 siblings, 1 reply; 28+ messages in thread
From: Kent Overstreet @ 2013-08-22  5:43 UTC (permalink / raw)
  To: Stefan Priebe; +Cc: Jens Axboe, linux-kernel, linux-bcache

On Thu, Aug 22, 2013 at 07:27:12AM +0200, Stefan Priebe wrote:
> today i had this one:

Heh, I finally tracked it down earlier today. Turned out to be a rather
embarassing bug... schedule_timeout() is not the same as
schedule_timeout_interruptible().

I'll get the fix out in the morning, for the moment an easy workaround
is to set writeback_percent to 0.

> 
> 2013-08-22 06:28:55     [<ffffffff815247f4>] schedule+0x24/0x70
> 2013-08-22 06:28:55     [<ffffffff815248c7>] io_schedule+0x87/0xd0
> 2013-08-22 06:28:55     [<ffffffff810e6099>] sleep_on_page+0x9/0x10
> 2013-08-22 06:28:55     [<ffffffff81522dc7>] __wait_on_bit+0x57/0x80
> 2013-08-22 06:28:55     [<ffffffff810e5bbc>] ? find_get_pages_tag+0xcc/0x180
> 2013-08-22 06:28:55     [<ffffffff810e62ae>] wait_on_page_bit+0x6e/0x80
> 2013-08-22 06:28:55     [<ffffffff81070af0>] ?
> autoremove_wake_function+0x40/0x40
> 2013-08-22 06:28:55     [<ffffffff810f2660>] ? pagevec_lookup_tag+0x20/0x30
> 2013-08-22 06:28:55     [<ffffffff810e67ef>]
> filemap_fdatawait_range+0x10f/0x1b0
> 2013-08-22 06:28:55     [<ffffffff810e69b0>]
> filemap_write_and_wait_range+0x90/0xa0
> 2013-08-22 06:28:55     [<ffffffff811ef9c3>] xfs_file_fsync+0x63/0x1e0
> 2013-08-22 06:28:55     [<ffffffff8116d7c3>] vfs_fsync_range+0x23/0x30
> 2013-08-22 06:28:55     [<ffffffff8116d7e7>] vfs_fsync+0x17/0x20
> 2013-08-22 06:28:55     [<ffffffff8116da1c>] do_fsync+0x3c/0x60
> 2013-08-22 06:28:55     [<ffffffff8116da4e>] SyS_fdatasync+0xe/0x20
> 2013-08-22 06:28:55     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
> 2013-08-22 06:28:54     [<ffffffff815247f4>] schedule+0x24/0x70
> 2013-08-22 06:28:54     [<ffffffff815248c7>] io_schedule+0x87/0xd0
> 2013-08-22 06:28:54     [<ffffffff810e6099>] sleep_on_page+0x9/0x10
> 2013-08-22 06:28:54     [<ffffffff81522dc7>] __wait_on_bit+0x57/0x80
> 2013-08-22 06:28:54     [<ffffffff810e5bbc>] ? find_get_pages_tag+0xcc/0x180
> 2013-08-22 06:28:54     [<ffffffff810e62ae>] wait_on_page_bit+0x6e/0x80
> 2013-08-22 06:28:54     [<ffffffff81070af0>] ?
> autoremove_wake_function+0x40/0x40
> 2013-08-22 06:28:54     [<ffffffff810f2660>] ? pagevec_lookup_tag+0x20/0x30
> 2013-08-22 06:28:54     [<ffffffff810e67ef>]
> filemap_fdatawait_range+0x10f/0x1b0
> 2013-08-22 06:28:54     [<ffffffff810e69b0>]
> filemap_write_and_wait_range+0x90/0xa0
> 2013-08-22 06:28:54     [<ffffffff811ef9c3>] xfs_file_fsync+0x63/0x1e0
> 2013-08-22 06:28:54     [<ffffffff8116d7c3>] vfs_fsync_range+0x23/0x30
> 2013-08-22 06:28:54     [<ffffffff8116d7e7>] vfs_fsync+0x17/0x20
> 2013-08-22 06:28:54     [<ffffffff8116da1c>] do_fsync+0x3c/0x60
> 2013-08-22 06:28:54     [<ffffffff8116da6b>] SyS_fsync+0xb/0x10
> 2013-08-22 06:28:54     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
> 2013-08-22 06:28:54     INFO: task ceph-osd:3520 blocked for more
> than 120 seconds.
> 2013-08-22 06:28:54     "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> 2013-08-22 06:28:54     ceph-osd D ffffffff8160f760 0 3520 1 0x00000004
> 2013-08-22 06:28:54     ffff880c381d5c78 0000000000000086
> ffff880c381d5fd8 0000000000012c80
> 2013-08-22 06:28:54     ffff880c381d4010 0000000000012c80
> 0000000000012c80 0000000000012c80
> 2013-08-22 06:28:54     ffff880c381d5fd8 0000000000012c80
> ffff880c39100000 ffff880c48dd0000
> 2013-08-22 06:28:54     Call Trace:
> 2013-08-22 06:28:54     [<ffffffff810e6090>] ? __lock_page+0x70/0x70
> 2013-08-22 06:28:53     [<ffffffff810e5bbc>] ? find_get_pages_tag+0xcc/0x180
> 2013-08-22 06:28:53     [<ffffffff810e62ae>] wait_on_page_bit+0x6e/0x80
> 2013-08-22 06:28:53     [<ffffffff81070af0>] ?
> autoremove_wake_function+0x40/0x40
> 2013-08-22 06:28:53     [<ffffffff810f2660>] ? pagevec_lookup_tag+0x20/0x30
> 2013-08-22 06:28:53     [<ffffffff810e67ef>]
> filemap_fdatawait_range+0x10f/0x1b0
> 2013-08-22 06:28:53     [<ffffffff815247f4>] ? schedule+0x24/0x70
> 2013-08-22 06:28:53     [<ffffffff81522b3d>] ? schedule_timeout+0x16d/0x200
> 2013-08-22 06:28:53     [<ffffffff81068e9a>] ?
> __queue_delayed_work+0xaa/0x1a0
> 2013-08-22 06:28:53     [<ffffffff81068cf9>] ?
> try_to_grab_pending+0x109/0x190
> 2013-08-22 06:28:53     [<ffffffff810e68b3>] filemap_fdatawait+0x23/0x30
> 2013-08-22 06:28:53     [<ffffffff81166e35>] wait_sb_inodes+0xb5/0x110
> 2013-08-22 06:28:53     [<ffffffff81167a1c>] sync_inodes_sb+0x9c/0xd0
> 2013-08-22 06:28:53     [<ffffffff8116dc66>] __sync_filesystem+0x16/0x50
> 2013-08-22 06:28:53     [<ffffffff8116dce3>] sync_filesystem+0x43/0x60
> 2013-08-22 06:28:53     [<ffffffff8116dd50>] SyS_syncfs+0x50/0x90
> 2013-08-22 06:28:53     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
> 2013-08-22 06:28:53     INFO: task ceph-osd:3516 blocked for more
> than 120 seconds.
> 2013-08-22 06:28:53     "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> 2013-08-22 06:28:53     ceph-osd D ffffffff8160f760 0 3516 1 0x00000004
> 2013-08-22 06:28:53     ffff880c383d3c78 0000000000000086
> ffff880c383d3fd8 0000000000012c80
> 2013-08-22 06:28:53     ffff880c383d2010 0000000000012c80
> 0000000000012c80 0000000000012c80
> 2013-08-22 06:28:53     ffff880c383d3fd8 0000000000012c80
> ffff880c428018e0 ffff880c48dd0000
> 2013-08-22 06:28:53     Call Trace:
> 2013-08-22 06:28:53     [<ffffffff810e6090>] ? __lock_page+0x70/0x70
> 2013-08-22 06:28:52     [<ffffffff81219983>] ? xfs_bmbt_get_all+0x13/0x20
> 2013-08-22 06:28:52     [<ffffffff81208a14>] xfs_attr_set_int+0x3a4/0x4f0
> 2013-08-22 06:28:52     [<ffffffff81208be1>] xfs_attr_set+0x81/0x90
> 2013-08-22 06:28:52     [<ffffffff81201462>] xfs_xattr_set+0x42/0x60
> 2013-08-22 06:28:52     [<ffffffff8116257b>] generic_setxattr+0x9b/0xb0
> 2013-08-22 06:28:52     [<ffffffff8116376f>] __vfs_setxattr_noperm+0x5f/0xe0
> 2013-08-22 06:28:52     [<ffffffff811638a4>] vfs_setxattr+0xb4/0xc0
> 2013-08-22 06:28:52     [<ffffffff8116396e>] setxattr+0xbe/0x200
> 2013-08-22 06:28:52     [<ffffffff81143261>] ? __sb_start_write+0x71/0x110
> 2013-08-22 06:28:52     [<ffffffff8115ce51>] ?
> __mnt_want_write_file+0x61/0x80
> 2013-08-22 06:28:52     [<ffffffff81163b56>] SyS_fsetxattr+0xa6/0xc0
> 2013-08-22 06:28:52     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
> 2013-08-22 06:28:52     INFO: task ceph-osd:3508 blocked for more
> than 120 seconds.
> 2013-08-22 06:28:52     "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> 2013-08-22 06:28:52     ceph-osd D ffffffff8160f760 0 3508 1 0x00000004
> 2013-08-22 06:28:52     ffff880c2c1bfc28 0000000000000086
> ffff880c2c1bffd8 0000000000012c80
> 2013-08-22 06:28:52     ffff880c2c1be010 0000000000012c80
> 0000000000012c80 0000000000012c80
> 2013-08-22 06:28:52     ffff880c2c1bffd8 0000000000012c80
> ffff880c43eae380 ffff880c48dd4aa0
> 2013-08-22 06:28:52     Call Trace:
> 2013-08-22 06:28:52     [<ffffffff810e6090>] ? __lock_page+0x70/0x70
> 2013-08-22 06:28:52     [<ffffffff815247f4>] schedule+0x24/0x70
> 2013-08-22 06:28:52     [<ffffffff815248c7>] io_schedule+0x87/0xd0
> 2013-08-22 06:28:52     [<ffffffff810e6099>] sleep_on_page+0x9/0x10
> 2013-08-22 06:28:52     [<ffffffff81522dc7>] __wait_on_bit+0x57/0x80
> 2013-08-22 06:28:51     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
> 2013-08-22 06:28:51     INFO: task ceph-osd:3467 blocked for more
> than 120 seconds.
> 2013-08-22 06:28:51     "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> 2013-08-22 06:28:51     ceph-osd D ffffffff8160f760 0 3467 1 0x00000004
> 2013-08-22 06:28:51     ffff880c375657e8 0000000000000086
> ffff880c37565fd8 0000000000012c80
> 2013-08-22 06:28:51     ffff880c37564010 0000000000012c80
> 0000000000012c80 0000000000012c80
> 2013-08-22 06:28:51     ffff880c37565fd8 0000000000012c80
> ffff880c3850caa0 ffff880c48dd4aa0
> 2013-08-22 06:28:51     Call Trace:
> 2013-08-22 06:28:51     [<ffffffff81245efe>] ? xlog_bdstrat+0x1e/0x60
> 2013-08-22 06:28:51     [<ffffffff815247f4>] schedule+0x24/0x70
> 2013-08-22 06:28:51     [<ffffffff81522b3d>] schedule_timeout+0x16d/0x200
> 2013-08-22 06:28:51     [<ffffffff8124653d>] ? xlog_sync+0x2ed/0x490
> 2013-08-22 06:28:51     [<ffffffff81523b7a>] __down+0x6a/0xa0
> 2013-08-22 06:28:51     [<ffffffff811eb747>] ? _xfs_buf_find+0x107/0x290
> 2013-08-22 06:28:51     [<ffffffff810757ac>] down+0x3c/0x50
> 2013-08-22 06:28:51     [<ffffffff811eb070>] xfs_buf_lock+0x40/0xe0
> 2013-08-22 06:28:51     [<ffffffff811eb747>] _xfs_buf_find+0x107/0x290
> 2013-08-22 06:28:51     [<ffffffff811ebb1f>] xfs_buf_get_map+0x2f/0x180
> 2013-08-22 06:28:51     [<ffffffff8124dc68>]
> xfs_trans_get_buf_map+0xc8/0x160
> 2013-08-22 06:28:51     [<ffffffff81224093>] xfs_da_get_buf+0xb3/0xe0
> 2013-08-22 06:28:51     [<ffffffff81209b20>]
> xfs_attr3_leaf_to_node+0xb0/0x240
> 2013-08-22 06:28:51     [<ffffffff8120d12b>] ?
> xfs_attr3_leaf_add+0x15b/0x1c0
> 2013-08-22 06:28:51     [<ffffffff8120843c>]
> xfs_attr_leaf_addname+0x14c/0x380
> 2013-08-22 06:28:51     [<ffffffff81243f68>] ? xfs_trans_add_item+0x28/0x70
> 2013-08-22 06:28:50     [<ffffffff810e8420>] ? mempool_alloc_slab+0x10/0x20
> 2013-08-22 06:28:50     [<ffffffff815239c2>] ? down_read+0x12/0x20
> 2013-08-22 06:28:50     [<ffffffffa010c38a>]
> request_write+0x8a/0x380 [bcache]
> 2013-08-22 06:28:50     [<ffffffffa010c8fb>]
> cached_dev_make_request+0x27b/0x350 [bcache]
> 2013-08-22 06:28:50     [<ffffffff81281982>] generic_make_request+0xc2/0x100
> 2013-08-22 06:28:50     [<ffffffff81281a27>] submit_bio+0x67/0x130
> 2013-08-22 06:28:50     [<ffffffff811e823f>] xfs_submit_ioend_bio+0x2f/0x40
> 2013-08-22 06:28:50     [<ffffffff811e8314>] xfs_submit_ioend+0xc4/0x130
> 2013-08-22 06:28:50     [<ffffffff811e98bb>] xfs_vm_writepage+0x2eb/0x590
> 2013-08-22 06:28:50     [<ffffffff810ef752>] __writepage+0x12/0x40
> 2013-08-22 06:28:50     [<ffffffff810f090b>] write_cache_pages+0x23b/0x4e0
> 2013-08-22 06:28:50     [<ffffffff81524437>] ? __schedule+0x3d7/0x6d0
> 2013-08-22 06:28:50     [<ffffffff810ef740>] ? set_page_dirty+0x60/0x60
> 2013-08-22 06:28:50     [<ffffffff810f0bfc>] generic_writepages+0x4c/0x70
> 2013-08-22 06:28:50     [<ffffffff811e810f>] xfs_vm_writepages+0x4f/0x60
> 2013-08-22 06:28:50     [<ffffffff810f0c3b>] do_writepages+0x1b/0x40
> 2013-08-22 06:28:50     [<ffffffff810e6911>]
> __filemap_fdatawrite_range+0x51/0x60
> 2013-08-22 06:28:50     [<ffffffff81097cf0>] ? futex_wake+0x110/0x120
> 2013-08-22 06:28:50     [<ffffffff810e699a>]
> filemap_write_and_wait_range+0x7a/0xa0
> 2013-08-22 06:28:50     [<ffffffff811ef9c3>] xfs_file_fsync+0x63/0x1e0
> 2013-08-22 06:28:50     [<ffffffff8116d7c3>] vfs_fsync_range+0x23/0x30
> 2013-08-22 06:28:50     [<ffffffff8116d7e7>] vfs_fsync+0x17/0x20
> 2013-08-22 06:28:50     [<ffffffff8116da1c>] do_fsync+0x3c/0x60
> 2013-08-22 06:28:50     [<ffffffff8116da4e>] SyS_fdatasync+0xe/0x20
> 2013-08-22 06:28:49     [<ffffffff810ef740>] ? set_page_dirty+0x60/0x60
> 2013-08-22 06:28:49     [<ffffffff810f0bfc>] generic_writepages+0x4c/0x70
> 2013-08-22 06:28:49     [<ffffffff811639a3>] ? setxattr+0xf3/0x200
> 2013-08-22 06:28:49     [<ffffffff811e810f>] xfs_vm_writepages+0x4f/0x60
> 2013-08-22 06:28:49     [<ffffffff810f0c3b>] do_writepages+0x1b/0x40
> 2013-08-22 06:28:49     [<ffffffff810e6911>]
> __filemap_fdatawrite_range+0x51/0x60
> 2013-08-22 06:28:49     [<ffffffff810e699a>]
> filemap_write_and_wait_range+0x7a/0xa0
> 2013-08-22 06:28:49     [<ffffffff811ef9c3>] xfs_file_fsync+0x63/0x1e0
> 2013-08-22 06:28:49     [<ffffffff8116d7c3>] vfs_fsync_range+0x23/0x30
> 2013-08-22 06:28:49     [<ffffffff8116d7e7>] vfs_fsync+0x17/0x20
> 2013-08-22 06:28:49     [<ffffffff8116da1c>] do_fsync+0x3c/0x60
> 2013-08-22 06:28:49     [<ffffffff8116da6b>] SyS_fsync+0xb/0x10
> 2013-08-22 06:28:49     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
> 2013-08-22 06:28:49     INFO: task ceph-osd:3483 blocked for more
> than 120 seconds.
> 2013-08-22 06:28:49     "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> 2013-08-22 06:28:49     ceph-osd D ffffffff8160f760 0 3483 1 0x00000004
> 2013-08-22 06:28:49     ffff880c2c0d1910 0000000000000086
> ffff880c2c0d1fd8 0000000000012c80
> 2013-08-22 06:28:49     ffff880c2c0d0010 0000000000012c80
> 0000000000012c80 0000000000012c80
> 2013-08-22 06:28:49     ffff880c2c0d1fd8 0000000000012c80
> ffff880c3a14caa0 ffff880c48dd31c0
> 2013-08-22 06:28:49     Call Trace:
> 2013-08-22 06:28:49     [<ffffffff815247f4>] schedule+0x24/0x70
> 2013-08-22 06:28:49     [<ffffffff81525cfd>]
> rwsem_down_read_failed+0x9d/0xe5
> 2013-08-22 06:28:49     [<ffffffff812aa024>]
> call_rwsem_down_read_failed+0x14/0x30
> 2013-08-22 06:28:48     [<ffffffff81143261>] ? __sb_start_write+0x71/0x110
> 2013-08-22 06:28:48     [<ffffffff8115ce51>] ?
> __mnt_want_write_file+0x61/0x80
> 2013-08-22 06:28:48     [<ffffffff81163b56>] SyS_fsetxattr+0xa6/0xc0
> 2013-08-22 06:28:48     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
> 2013-08-22 06:28:48     INFO: task ceph-osd:3482 blocked for more
> than 120 seconds.
> 2013-08-22 06:28:48     "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> 2013-08-22 06:28:48     ceph-osd D ffffffff8160f760 0 3482 1 0x00000004
> 2013-08-22 06:28:48     ffff880c2c0cf910 0000000000000086
> ffff880c2c0cffd8 0000000000012c80
> 2013-08-22 06:28:48     ffff880c2c0ce010 0000000000012c80
> 0000000000012c80 0000000000012c80
> 2013-08-22 06:28:48     ffff880c2c0cffd8 0000000000012c80
> ffff880c47768000 ffff880c48dd0000
> 2013-08-22 06:28:48     Call Trace:
> 2013-08-22 06:28:48     [<ffffffff815247f4>] schedule+0x24/0x70
> 2013-08-22 06:28:48     [<ffffffff81525cfd>]
> rwsem_down_read_failed+0x9d/0xe5
> 2013-08-22 06:28:48     [<ffffffff812aa024>]
> call_rwsem_down_read_failed+0x14/0x30
> 2013-08-22 06:28:48     [<ffffffff810e8420>] ? mempool_alloc_slab+0x10/0x20
> 2013-08-22 06:28:48     [<ffffffff815239c2>] ? down_read+0x12/0x20
> 2013-08-22 06:28:48     [<ffffffffa010c38a>]
> request_write+0x8a/0x380 [bcache]
> 2013-08-22 06:28:48     [<ffffffffa010c8fb>]
> cached_dev_make_request+0x27b/0x350 [bcache]
> 2013-08-22 06:28:48     [<ffffffff81281982>] generic_make_request+0xc2/0x100
> 2013-08-22 06:28:48     [<ffffffff81281a27>] submit_bio+0x67/0x130
> 2013-08-22 06:28:48     [<ffffffff811e823f>] xfs_submit_ioend_bio+0x2f/0x40
> 2013-08-22 06:28:48     [<ffffffff811e833e>] xfs_submit_ioend+0xee/0x130
> 2013-08-22 06:28:48     [<ffffffff811e98bb>] xfs_vm_writepage+0x2eb/0x590
> 2013-08-22 06:28:48     [<ffffffff810ef752>] __writepage+0x12/0x40
> 2013-08-22 06:28:48     [<ffffffff810f090b>] write_cache_pages+0x23b/0x4e0
> 2013-08-22 06:28:47     [<ffffffff812469b6>] _xfs_log_force+0x116/0x250
> 2013-08-22 06:28:47     [<ffffffff8124836a>] xfs_log_force+0x2a/0x90
> 2013-08-22 06:28:47     [<ffffffff811eb747>] ? _xfs_buf_find+0x107/0x290
> 2013-08-22 06:28:47     [<ffffffff811eb0a3>] xfs_buf_lock+0x73/0xe0
> 2013-08-22 06:28:47     [<ffffffff811eb747>] _xfs_buf_find+0x107/0x290
> 2013-08-22 06:28:47     [<ffffffff811ebb1f>] xfs_buf_get_map+0x2f/0x180
> 2013-08-22 06:28:47     [<ffffffff8124dc68>]
> xfs_trans_get_buf_map+0xc8/0x160
> 2013-08-22 06:28:47     [<ffffffff81224093>] xfs_da_get_buf+0xb3/0xe0
> 2013-08-22 06:28:47     [<ffffffff81209b20>]
> xfs_attr3_leaf_to_node+0xb0/0x240
> 2013-08-22 06:28:47     [<ffffffff8120d12b>] ?
> xfs_attr3_leaf_add+0x15b/0x1c0
> 2013-08-22 06:28:47     [<ffffffff8120843c>]
> xfs_attr_leaf_addname+0x14c/0x380
> 2013-08-22 06:28:47     [<ffffffff81243f68>] ? xfs_trans_add_item+0x28/0x70
> 2013-08-22 06:28:47     [<ffffffff81219983>] ? xfs_bmbt_get_all+0x13/0x20
> 2013-08-22 06:28:47     [<ffffffff81208a14>] xfs_attr_set_int+0x3a4/0x4f0
> 2013-08-22 06:28:47     [<ffffffff81208be1>] xfs_attr_set+0x81/0x90
> 2013-08-22 06:28:47     [<ffffffff81201462>] xfs_xattr_set+0x42/0x60
> 2013-08-22 06:28:47     [<ffffffff8116257b>] generic_setxattr+0x9b/0xb0
> 2013-08-22 06:28:47     [<ffffffff8116376f>] __vfs_setxattr_noperm+0x5f/0xe0
> 2013-08-22 06:28:47     [<ffffffff811638a4>] vfs_setxattr+0xb4/0xc0
> 2013-08-22 06:28:47     [<ffffffff8116396e>] setxattr+0xbe/0x200
> 2013-08-22 06:28:47     [<ffffffff811ec1ca>] xfs_buf_iorequest+0x4a/0xa0
> 2013-08-22 06:28:47     [<ffffffff81245efe>] xlog_bdstrat+0x1e/0x60
> 2013-08-22 06:28:47     [<ffffffff8124653d>] xlog_sync+0x2ed/0x490
> 2013-08-22 06:28:47     [<ffffffff8124686b>]
> xlog_state_release_iclog+0x8b/0xc0
> 2013-08-22 06:28:46     [<ffffffff8116dd50>] SyS_syncfs+0x50/0x90
> 2013-08-22 06:28:46     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
> 2013-08-22 06:28:46     INFO: task ceph-osd:3481 blocked for more
> than 120 seconds.
> 2013-08-22 06:28:46     "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> 2013-08-22 06:28:46     ceph-osd D ffffffff8160f760 0 3481 1 0x00000004
> 2013-08-22 06:28:46     ffff880c2c0cd500 0000000000000086
> ffff880c2c0cdfd8 0000000000012c80
> 2013-08-22 06:28:46     ffff880c2c0cc010 0000000000012c80
> 0000000000012c80 0000000000012c80
> 2013-08-22 06:28:46     ffff880c2c0cdfd8 0000000000012c80
> ffff880c4776b1c0 ffff880c48dd4aa0
> 2013-08-22 06:28:46     Call Trace:
> 2013-08-22 06:28:46     [<ffffffff8124e5b6>] ? xfs_trans_brelse+0xb6/0x100
> 2013-08-22 06:28:46     [<ffffffff815247f4>] schedule+0x24/0x70
> 2013-08-22 06:28:46     [<ffffffff81525cfd>]
> rwsem_down_read_failed+0x9d/0xe5
> 2013-08-22 06:28:46     [<ffffffff812aa024>]
> call_rwsem_down_read_failed+0x14/0x30
> 2013-08-22 06:28:46     [<ffffffff810e8420>] ? mempool_alloc_slab+0x10/0x20
> 2013-08-22 06:28:46     [<ffffffff815239c2>] ? down_read+0x12/0x20
> 2013-08-22 06:28:46     [<ffffffffa010c38a>]
> request_write+0x8a/0x380 [bcache]
> 2013-08-22 06:28:46     [<ffffffffa010c8fb>]
> cached_dev_make_request+0x27b/0x350 [bcache]
> 2013-08-22 06:28:46     [<ffffffff81281982>] generic_make_request+0xc2/0x100
> 2013-08-22 06:28:46     [<ffffffff81281a27>] submit_bio+0x67/0x130
> 2013-08-22 06:28:46     [<ffffffff811eabf0>] xfs_buf_ioapply_map+0x180/0x1d0
> 2013-08-22 06:28:46     [<ffffffff811ead47>] _xfs_buf_ioapply+0x107/0x150
> 2013-08-22 06:28:46     [<ffffffff81245efe>] ? xlog_bdstrat+0x1e/0x60
> 2013-08-22 06:28:45     "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> 2013-08-22 06:28:45     ceph-osd D ffffffff8160f760 0 3462 1 0x00000004
> 2013-08-22 06:28:45     ffff880c375cdc28 0000000000000086
> ffff880c375cdfd8 0000000000012c80
> 2013-08-22 06:28:45     ffff880c375cc010 0000000000012c80
> 0000000000012c80 0000000000012c80
> 2013-08-22 06:28:45     ffff880c375cdfd8 0000000000012c80
> ffff880c46318000 ffff880c48dd6380
> 2013-08-22 06:28:45     Call Trace:
> 2013-08-22 06:28:45     [<ffffffff810e6090>] ? __lock_page+0x70/0x70
> 2013-08-22 06:28:45     [<ffffffff815247f4>] schedule+0x24/0x70
> 2013-08-22 06:28:45     [<ffffffff815248c7>] io_schedule+0x87/0xd0
> 2013-08-22 06:28:45     [<ffffffff810e6099>] sleep_on_page+0x9/0x10
> 2013-08-22 06:28:45     [<ffffffff81522dc7>] __wait_on_bit+0x57/0x80
> 2013-08-22 06:28:45     [<ffffffff810e5bbc>] ? find_get_pages_tag+0xcc/0x180
> 2013-08-22 06:28:45     [<ffffffff810e62ae>] wait_on_page_bit+0x6e/0x80
> 2013-08-22 06:28:45     [<ffffffff81070af0>] ?
> autoremove_wake_function+0x40/0x40
> 2013-08-22 06:28:45     [<ffffffff810f2660>] ? pagevec_lookup_tag+0x20/0x30
> 2013-08-22 06:28:45     [<ffffffff810e67ef>]
> filemap_fdatawait_range+0x10f/0x1b0
> 2013-08-22 06:28:45     [<ffffffff815247f4>] ? schedule+0x24/0x70
> 2013-08-22 06:28:45     [<ffffffff81522b3d>] ? schedule_timeout+0x16d/0x200
> 2013-08-22 06:28:45     [<ffffffff81068e9a>] ?
> __queue_delayed_work+0xaa/0x1a0
> 2013-08-22 06:28:45     [<ffffffff810e68b3>] filemap_fdatawait+0x23/0x30
> 2013-08-22 06:28:45     [<ffffffff81166e35>] wait_sb_inodes+0xb5/0x110
> 2013-08-22 06:28:45     [<ffffffff81167a1c>] sync_inodes_sb+0x9c/0xd0
> 2013-08-22 06:28:45     [<ffffffff8116dc66>] __sync_filesystem+0x16/0x50
> 2013-08-22 06:28:45     [<ffffffff8116dce3>] sync_filesystem+0x43/0x60
> 2013-08-22 06:28:44     [<ffffffffa010c8fb>]
> cached_dev_make_request+0x27b/0x350 [bcache]
> 2013-08-22 06:28:44     [<ffffffff81281982>] generic_make_request+0xc2/0x100
> 2013-08-22 06:28:44     [<ffffffff81281a27>] submit_bio+0x67/0x130
> 2013-08-22 06:28:44     [<ffffffff811e823f>] xfs_submit_ioend_bio+0x2f/0x40
> 2013-08-22 06:28:44     [<ffffffff811e833e>] xfs_submit_ioend+0xee/0x130
> 2013-08-22 06:28:44     [<ffffffff811e98bb>] xfs_vm_writepage+0x2eb/0x590
> 2013-08-22 06:28:44     [<ffffffff810ef752>] __writepage+0x12/0x40
> 2013-08-22 06:28:44     [<ffffffff810f090b>] write_cache_pages+0x23b/0x4e0
> 2013-08-22 06:28:44     [<ffffffff81524437>] ? __schedule+0x3d7/0x6d0
> 2013-08-22 06:28:44     [<ffffffff810ef740>] ? set_page_dirty+0x60/0x60
> 2013-08-22 06:28:44     [<ffffffff810f0bfc>] generic_writepages+0x4c/0x70
> 2013-08-22 06:28:44     [<ffffffff811e810f>] xfs_vm_writepages+0x4f/0x60
> 2013-08-22 06:28:44     [<ffffffff810f0c3b>] do_writepages+0x1b/0x40
> 2013-08-22 06:28:44     [<ffffffff810e6911>]
> __filemap_fdatawrite_range+0x51/0x60
> 2013-08-22 06:28:44     [<ffffffff81097cf0>] ? futex_wake+0x110/0x120
> 2013-08-22 06:28:44     [<ffffffff810e699a>]
> filemap_write_and_wait_range+0x7a/0xa0
> 2013-08-22 06:28:44     [<ffffffff811ef9c3>] xfs_file_fsync+0x63/0x1e0
> 2013-08-22 06:28:44     [<ffffffff8116d7c3>] vfs_fsync_range+0x23/0x30
> 2013-08-22 06:28:44     [<ffffffff8116d7e7>] vfs_fsync+0x17/0x20
> 2013-08-22 06:28:44     [<ffffffff8116da1c>] do_fsync+0x3c/0x60
> 2013-08-22 06:28:44     [<ffffffff8116da4e>] SyS_fdatasync+0xe/0x20
> 2013-08-22 06:28:44     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
> 2013-08-22 06:28:44     INFO: task ceph-osd:3462 blocked for more
> than 120 seconds.
> 2013-08-22 06:28:43     INFO: task ceph-osd:24130 blocked for more
> than 120 seconds.
> 2013-08-22 06:28:43     "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> 2013-08-22 06:28:43     ceph-osd D ffffffff8160f760 0 24130 1 0x00000004
> 2013-08-22 06:28:43     ffff880c27fe98d8 0000000000000082
> ffff880c27fe9fd8 0000000000012c80
> 2013-08-22 06:28:43     ffff880c27fe8010 0000000000012c80
> 0000000000012c80 0000000000012c80
> 2013-08-22 06:28:43     ffff880c27fe9fd8 0000000000012c80
> ffff880c215e6380 ffff880c48decaa0
> 2013-08-22 06:28:43     Call Trace:
> 2013-08-22 06:28:43     [<ffffffff81170430>] ? __wait_on_buffer+0x30/0x30
> 2013-08-22 06:28:43     [<ffffffff815247f4>] schedule+0x24/0x70
> 2013-08-22 06:28:43     [<ffffffff815248c7>] io_schedule+0x87/0xd0
> 2013-08-22 06:28:43     [<ffffffff81170439>] sleep_on_buffer+0x9/0x10
> 2013-08-22 06:28:43     [<ffffffff81522dc7>] __wait_on_bit+0x57/0x80
> 2013-08-22 06:28:43     [<ffffffff81170430>] ? __wait_on_buffer+0x30/0x30
> 2013-08-22 06:28:43     [<ffffffff81522e63>]
> out_of_line_wait_on_bit+0x73/0x90
> 2013-08-22 06:28:43     [<ffffffff81070af0>] ?
> autoremove_wake_function+0x40/0x40
> 2013-08-22 06:28:43     [<ffffffff8116f64b>] ? submit_bh+0xb/0x10
> 2013-08-22 06:28:43     [<ffffffff81170426>] __wait_on_buffer+0x26/0x30
> 2013-08-22 06:28:43     [<ffffffff81172803>] __block_write_begin+0x383/0x530
> 2013-08-22 06:28:43     [<ffffffff811e9050>] ?
> xfs_get_blocks_direct+0x20/0x20
> 2013-08-22 06:28:43     [<ffffffff810e6d0f>] ?
> grab_cache_page_write_begin+0x9f/0xd0
> 2013-08-22 06:28:43     [<ffffffff811e802f>] xfs_vm_write_begin+0x5f/0xd0
> 2013-08-22 06:28:43     [<ffffffff810e55ba>]
> generic_perform_write+0xca/0x200
> 2013-08-22 06:28:43     [<ffffffff810e574f>]
> generic_file_buffered_write+0x5f/0x90
> 2013-08-22 06:28:43     [<ffffffff811f0275>]
> xfs_file_buffered_aio_write+0xf5/0x180
> 2013-08-22 06:28:43     [<ffffffff811f03ce>] xfs_file_aio_write+0xce/0x150
> 2013-08-22 06:28:43     [<ffffffff8114ac8d>] ? path_put+0x1d/0x30
> 2013-08-22 06:28:43     [<ffffffff81140788>] do_sync_readv_writev+0x68/0xa0
> 2013-08-22 06:28:43     [<ffffffff81142052>] do_readv_writev+0xf2/0x2e0
> 2013-08-22 06:28:43     [<ffffffff811f0300>] ?
> xfs_file_buffered_aio_write+0x180/0x180
> 2013-08-22 06:28:43     [<ffffffff811407c0>] ?
> do_sync_readv_writev+0xa0/0xa0
> 2013-08-22 06:28:43     [<ffffffff8114227e>] vfs_writev+0x3e/0x60
> 2013-08-22 06:28:43     [<ffffffff811423ca>] SyS_writev+0x5a/0xc0
> 2013-08-22 06:28:43     [<ffffffff811410d3>] ? SyS_lseek+0x53/0x80
> 2013-08-22 06:28:43     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
> 2013-08-22 06:28:43     INFO: task ceph-osd:24134 blocked for more
> than 120 seconds.
> 2013-08-22 06:28:43     "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> 2013-08-22 06:28:43     ceph-osd D ffffffff8160f760 0 24134 1 0x00000004
> 2013-08-22 06:28:43     ffff880c23d31708 0000000000000082
> ffff880c23d31fd8 0000000000012c80
> 2013-08-22 06:28:43     ffff880c23d30010 0000000000012c80
> 0000000000012c80 0000000000012c80
> 2013-08-22 06:28:43     ffff880c23d31fd8 0000000000012c80
> ffff880c361e0000 ffff880c48dd4aa0
> 2013-08-22 06:28:43     Call Trace:
> 2013-08-22 06:28:43     [<ffffffff815247f4>] schedule+0x24/0x70
> 2013-08-22 06:28:43     [<ffffffff81524ae9>]
> schedule_preempt_disabled+0x9/0x10
> 2013-08-22 06:28:43     [<ffffffff81523664>]
> __mutex_lock_slowpath+0x194/0x240
> 2013-08-22 06:28:43     [<ffffffff81522e9e>] mutex_lock+0x1e/0x40
> 2013-08-22 06:28:43     [<ffffffffa00f891b>]
> bch_bucket_alloc_set+0x3b/0x80 [bcache]
> 2013-08-22 06:28:43     [<ffffffffa010ac1c>]
> bch_alloc_sectors+0x12c/0x4e0 [bcache]
> 2013-08-22 06:28:43     [<ffffffffa0104e15>] ?
> bch_generic_make_request_hack+0x25/0xc0 [bcache]
> 2013-08-22 06:28:43     [<ffffffffa0105013>] ?
> __bch_submit_bbio+0x73/0x80 [bcache]
> 2013-08-22 06:28:43     [<ffffffffa010bc98>]
> bch_insert_data_loop+0xf8/0x610 [bcache]
> 2013-08-22 06:28:43     [<ffffffffa010a335>] ?
> bch_get_congested+0x25/0x70 [bcache]
> 2013-08-22 06:28:43     [<ffffffffa010c1cd>]
> bch_insert_data+0x1d/0x20 [bcache]
> 2013-08-22 06:28:43     [<ffffffffa0116d13>] closure_queue+0x43/0x60
> [bcache]
> 2013-08-22 06:28:43     [<ffffffffa010c49e>]
> request_write+0x19e/0x380 [bcache]
> 
> Am 22.08.2013 01:47, schrieb Kent Overstreet:
> >On Tue, Aug 20, 2013 at 10:07:45AM +0200, Stefan Priebe - Profihost AG wrote:
> >>Am 20.08.2013 10:01, schrieb Stefan Priebe - Profihost AG:
> >>>Am 20.08.2013 00:27, schrieb Kent Overstreet:
> >>>>On Mon, Aug 19, 2013 at 12:09:24AM +0200, Stefan Priebe wrote:
> >>>>>
> >>>>>Vanilla 3.10.7 + bcache: Fix a writeback performance regression
> >>>>>
> >>>>>http://pastebin.com/raw.php?i=LXZk4cMH
> >>>>
> >>>>Whoops, at first I thought this was the same bug as one I'd already been
> >>>>chasing down that had been a harmless bug - turns out I didn't look
> >>>>closely enough at the backtrace.
> >>>>
> >>>>What happened is background writeback is deadlocking, because for some
> >>>>reason the workqueue it's running out of is a singlethreaded workqueue,
> >>>>so as soon as it decides to queue enough writeback bios that it has to
> >>>>sleep on that semaphore (which often won't happen due to the PD
> >>>>controller based ratelimiting) - boom, deadlock.
> >>>>
> >>>>Here's the fixup patch I just tested and am applying:
> >>
> >>Oh i'm now seeing very high CPU spikes of kworker... i don't see if i
> >>remove bcache: Fix a writeback performance regression.
> >
> >Are you able to reproduce it? I'm not having any luck reproducing it...
> >--
> >To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> >the body of a message to majordomo@vger.kernel.org
> >More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-22  5:43                 ` Kent Overstreet
@ 2013-08-22  5:59                   ` Stefan Priebe
  2013-08-22  6:02                     ` Kent Overstreet
  0 siblings, 1 reply; 28+ messages in thread
From: Stefan Priebe @ 2013-08-22  5:59 UTC (permalink / raw)
  To: Kent Overstreet; +Cc: Jens Axboe, linux-kernel, linux-bcache


 >schedule_timeout() is not the same as
 >schedule_timeout_interruptible().

just search and replace? So i can try on my own.

Stefan

Am 22.08.2013 07:43, schrieb Kent Overstreet:
> On Thu, Aug 22, 2013 at 07:27:12AM +0200, Stefan Priebe wrote:
>> today i had this one:
>
> Heh, I finally tracked it down earlier today. Turned out to be a rather
> embarassing bug... schedule_timeout() is not the same as
> schedule_timeout_interruptible().
>
> I'll get the fix out in the morning, for the moment an easy workaround
> is to set writeback_percent to 0.
>
>>
>> 2013-08-22 06:28:55     [<ffffffff815247f4>] schedule+0x24/0x70
>> 2013-08-22 06:28:55     [<ffffffff815248c7>] io_schedule+0x87/0xd0
>> 2013-08-22 06:28:55     [<ffffffff810e6099>] sleep_on_page+0x9/0x10
>> 2013-08-22 06:28:55     [<ffffffff81522dc7>] __wait_on_bit+0x57/0x80
>> 2013-08-22 06:28:55     [<ffffffff810e5bbc>] ? find_get_pages_tag+0xcc/0x180
>> 2013-08-22 06:28:55     [<ffffffff810e62ae>] wait_on_page_bit+0x6e/0x80
>> 2013-08-22 06:28:55     [<ffffffff81070af0>] ?
>> autoremove_wake_function+0x40/0x40
>> 2013-08-22 06:28:55     [<ffffffff810f2660>] ? pagevec_lookup_tag+0x20/0x30
>> 2013-08-22 06:28:55     [<ffffffff810e67ef>]
>> filemap_fdatawait_range+0x10f/0x1b0
>> 2013-08-22 06:28:55     [<ffffffff810e69b0>]
>> filemap_write_and_wait_range+0x90/0xa0
>> 2013-08-22 06:28:55     [<ffffffff811ef9c3>] xfs_file_fsync+0x63/0x1e0
>> 2013-08-22 06:28:55     [<ffffffff8116d7c3>] vfs_fsync_range+0x23/0x30
>> 2013-08-22 06:28:55     [<ffffffff8116d7e7>] vfs_fsync+0x17/0x20
>> 2013-08-22 06:28:55     [<ffffffff8116da1c>] do_fsync+0x3c/0x60
>> 2013-08-22 06:28:55     [<ffffffff8116da4e>] SyS_fdatasync+0xe/0x20
>> 2013-08-22 06:28:55     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
>> 2013-08-22 06:28:54     [<ffffffff815247f4>] schedule+0x24/0x70
>> 2013-08-22 06:28:54     [<ffffffff815248c7>] io_schedule+0x87/0xd0
>> 2013-08-22 06:28:54     [<ffffffff810e6099>] sleep_on_page+0x9/0x10
>> 2013-08-22 06:28:54     [<ffffffff81522dc7>] __wait_on_bit+0x57/0x80
>> 2013-08-22 06:28:54     [<ffffffff810e5bbc>] ? find_get_pages_tag+0xcc/0x180
>> 2013-08-22 06:28:54     [<ffffffff810e62ae>] wait_on_page_bit+0x6e/0x80
>> 2013-08-22 06:28:54     [<ffffffff81070af0>] ?
>> autoremove_wake_function+0x40/0x40
>> 2013-08-22 06:28:54     [<ffffffff810f2660>] ? pagevec_lookup_tag+0x20/0x30
>> 2013-08-22 06:28:54     [<ffffffff810e67ef>]
>> filemap_fdatawait_range+0x10f/0x1b0
>> 2013-08-22 06:28:54     [<ffffffff810e69b0>]
>> filemap_write_and_wait_range+0x90/0xa0
>> 2013-08-22 06:28:54     [<ffffffff811ef9c3>] xfs_file_fsync+0x63/0x1e0
>> 2013-08-22 06:28:54     [<ffffffff8116d7c3>] vfs_fsync_range+0x23/0x30
>> 2013-08-22 06:28:54     [<ffffffff8116d7e7>] vfs_fsync+0x17/0x20
>> 2013-08-22 06:28:54     [<ffffffff8116da1c>] do_fsync+0x3c/0x60
>> 2013-08-22 06:28:54     [<ffffffff8116da6b>] SyS_fsync+0xb/0x10
>> 2013-08-22 06:28:54     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
>> 2013-08-22 06:28:54     INFO: task ceph-osd:3520 blocked for more
>> than 120 seconds.
>> 2013-08-22 06:28:54     "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> 2013-08-22 06:28:54     ceph-osd D ffffffff8160f760 0 3520 1 0x00000004
>> 2013-08-22 06:28:54     ffff880c381d5c78 0000000000000086
>> ffff880c381d5fd8 0000000000012c80
>> 2013-08-22 06:28:54     ffff880c381d4010 0000000000012c80
>> 0000000000012c80 0000000000012c80
>> 2013-08-22 06:28:54     ffff880c381d5fd8 0000000000012c80
>> ffff880c39100000 ffff880c48dd0000
>> 2013-08-22 06:28:54     Call Trace:
>> 2013-08-22 06:28:54     [<ffffffff810e6090>] ? __lock_page+0x70/0x70
>> 2013-08-22 06:28:53     [<ffffffff810e5bbc>] ? find_get_pages_tag+0xcc/0x180
>> 2013-08-22 06:28:53     [<ffffffff810e62ae>] wait_on_page_bit+0x6e/0x80
>> 2013-08-22 06:28:53     [<ffffffff81070af0>] ?
>> autoremove_wake_function+0x40/0x40
>> 2013-08-22 06:28:53     [<ffffffff810f2660>] ? pagevec_lookup_tag+0x20/0x30
>> 2013-08-22 06:28:53     [<ffffffff810e67ef>]
>> filemap_fdatawait_range+0x10f/0x1b0
>> 2013-08-22 06:28:53     [<ffffffff815247f4>] ? schedule+0x24/0x70
>> 2013-08-22 06:28:53     [<ffffffff81522b3d>] ? schedule_timeout+0x16d/0x200
>> 2013-08-22 06:28:53     [<ffffffff81068e9a>] ?
>> __queue_delayed_work+0xaa/0x1a0
>> 2013-08-22 06:28:53     [<ffffffff81068cf9>] ?
>> try_to_grab_pending+0x109/0x190
>> 2013-08-22 06:28:53     [<ffffffff810e68b3>] filemap_fdatawait+0x23/0x30
>> 2013-08-22 06:28:53     [<ffffffff81166e35>] wait_sb_inodes+0xb5/0x110
>> 2013-08-22 06:28:53     [<ffffffff81167a1c>] sync_inodes_sb+0x9c/0xd0
>> 2013-08-22 06:28:53     [<ffffffff8116dc66>] __sync_filesystem+0x16/0x50
>> 2013-08-22 06:28:53     [<ffffffff8116dce3>] sync_filesystem+0x43/0x60
>> 2013-08-22 06:28:53     [<ffffffff8116dd50>] SyS_syncfs+0x50/0x90
>> 2013-08-22 06:28:53     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
>> 2013-08-22 06:28:53     INFO: task ceph-osd:3516 blocked for more
>> than 120 seconds.
>> 2013-08-22 06:28:53     "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> 2013-08-22 06:28:53     ceph-osd D ffffffff8160f760 0 3516 1 0x00000004
>> 2013-08-22 06:28:53     ffff880c383d3c78 0000000000000086
>> ffff880c383d3fd8 0000000000012c80
>> 2013-08-22 06:28:53     ffff880c383d2010 0000000000012c80
>> 0000000000012c80 0000000000012c80
>> 2013-08-22 06:28:53     ffff880c383d3fd8 0000000000012c80
>> ffff880c428018e0 ffff880c48dd0000
>> 2013-08-22 06:28:53     Call Trace:
>> 2013-08-22 06:28:53     [<ffffffff810e6090>] ? __lock_page+0x70/0x70
>> 2013-08-22 06:28:52     [<ffffffff81219983>] ? xfs_bmbt_get_all+0x13/0x20
>> 2013-08-22 06:28:52     [<ffffffff81208a14>] xfs_attr_set_int+0x3a4/0x4f0
>> 2013-08-22 06:28:52     [<ffffffff81208be1>] xfs_attr_set+0x81/0x90
>> 2013-08-22 06:28:52     [<ffffffff81201462>] xfs_xattr_set+0x42/0x60
>> 2013-08-22 06:28:52     [<ffffffff8116257b>] generic_setxattr+0x9b/0xb0
>> 2013-08-22 06:28:52     [<ffffffff8116376f>] __vfs_setxattr_noperm+0x5f/0xe0
>> 2013-08-22 06:28:52     [<ffffffff811638a4>] vfs_setxattr+0xb4/0xc0
>> 2013-08-22 06:28:52     [<ffffffff8116396e>] setxattr+0xbe/0x200
>> 2013-08-22 06:28:52     [<ffffffff81143261>] ? __sb_start_write+0x71/0x110
>> 2013-08-22 06:28:52     [<ffffffff8115ce51>] ?
>> __mnt_want_write_file+0x61/0x80
>> 2013-08-22 06:28:52     [<ffffffff81163b56>] SyS_fsetxattr+0xa6/0xc0
>> 2013-08-22 06:28:52     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
>> 2013-08-22 06:28:52     INFO: task ceph-osd:3508 blocked for more
>> than 120 seconds.
>> 2013-08-22 06:28:52     "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> 2013-08-22 06:28:52     ceph-osd D ffffffff8160f760 0 3508 1 0x00000004
>> 2013-08-22 06:28:52     ffff880c2c1bfc28 0000000000000086
>> ffff880c2c1bffd8 0000000000012c80
>> 2013-08-22 06:28:52     ffff880c2c1be010 0000000000012c80
>> 0000000000012c80 0000000000012c80
>> 2013-08-22 06:28:52     ffff880c2c1bffd8 0000000000012c80
>> ffff880c43eae380 ffff880c48dd4aa0
>> 2013-08-22 06:28:52     Call Trace:
>> 2013-08-22 06:28:52     [<ffffffff810e6090>] ? __lock_page+0x70/0x70
>> 2013-08-22 06:28:52     [<ffffffff815247f4>] schedule+0x24/0x70
>> 2013-08-22 06:28:52     [<ffffffff815248c7>] io_schedule+0x87/0xd0
>> 2013-08-22 06:28:52     [<ffffffff810e6099>] sleep_on_page+0x9/0x10
>> 2013-08-22 06:28:52     [<ffffffff81522dc7>] __wait_on_bit+0x57/0x80
>> 2013-08-22 06:28:51     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
>> 2013-08-22 06:28:51     INFO: task ceph-osd:3467 blocked for more
>> than 120 seconds.
>> 2013-08-22 06:28:51     "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> 2013-08-22 06:28:51     ceph-osd D ffffffff8160f760 0 3467 1 0x00000004
>> 2013-08-22 06:28:51     ffff880c375657e8 0000000000000086
>> ffff880c37565fd8 0000000000012c80
>> 2013-08-22 06:28:51     ffff880c37564010 0000000000012c80
>> 0000000000012c80 0000000000012c80
>> 2013-08-22 06:28:51     ffff880c37565fd8 0000000000012c80
>> ffff880c3850caa0 ffff880c48dd4aa0
>> 2013-08-22 06:28:51     Call Trace:
>> 2013-08-22 06:28:51     [<ffffffff81245efe>] ? xlog_bdstrat+0x1e/0x60
>> 2013-08-22 06:28:51     [<ffffffff815247f4>] schedule+0x24/0x70
>> 2013-08-22 06:28:51     [<ffffffff81522b3d>] schedule_timeout+0x16d/0x200
>> 2013-08-22 06:28:51     [<ffffffff8124653d>] ? xlog_sync+0x2ed/0x490
>> 2013-08-22 06:28:51     [<ffffffff81523b7a>] __down+0x6a/0xa0
>> 2013-08-22 06:28:51     [<ffffffff811eb747>] ? _xfs_buf_find+0x107/0x290
>> 2013-08-22 06:28:51     [<ffffffff810757ac>] down+0x3c/0x50
>> 2013-08-22 06:28:51     [<ffffffff811eb070>] xfs_buf_lock+0x40/0xe0
>> 2013-08-22 06:28:51     [<ffffffff811eb747>] _xfs_buf_find+0x107/0x290
>> 2013-08-22 06:28:51     [<ffffffff811ebb1f>] xfs_buf_get_map+0x2f/0x180
>> 2013-08-22 06:28:51     [<ffffffff8124dc68>]
>> xfs_trans_get_buf_map+0xc8/0x160
>> 2013-08-22 06:28:51     [<ffffffff81224093>] xfs_da_get_buf+0xb3/0xe0
>> 2013-08-22 06:28:51     [<ffffffff81209b20>]
>> xfs_attr3_leaf_to_node+0xb0/0x240
>> 2013-08-22 06:28:51     [<ffffffff8120d12b>] ?
>> xfs_attr3_leaf_add+0x15b/0x1c0
>> 2013-08-22 06:28:51     [<ffffffff8120843c>]
>> xfs_attr_leaf_addname+0x14c/0x380
>> 2013-08-22 06:28:51     [<ffffffff81243f68>] ? xfs_trans_add_item+0x28/0x70
>> 2013-08-22 06:28:50     [<ffffffff810e8420>] ? mempool_alloc_slab+0x10/0x20
>> 2013-08-22 06:28:50     [<ffffffff815239c2>] ? down_read+0x12/0x20
>> 2013-08-22 06:28:50     [<ffffffffa010c38a>]
>> request_write+0x8a/0x380 [bcache]
>> 2013-08-22 06:28:50     [<ffffffffa010c8fb>]
>> cached_dev_make_request+0x27b/0x350 [bcache]
>> 2013-08-22 06:28:50     [<ffffffff81281982>] generic_make_request+0xc2/0x100
>> 2013-08-22 06:28:50     [<ffffffff81281a27>] submit_bio+0x67/0x130
>> 2013-08-22 06:28:50     [<ffffffff811e823f>] xfs_submit_ioend_bio+0x2f/0x40
>> 2013-08-22 06:28:50     [<ffffffff811e8314>] xfs_submit_ioend+0xc4/0x130
>> 2013-08-22 06:28:50     [<ffffffff811e98bb>] xfs_vm_writepage+0x2eb/0x590
>> 2013-08-22 06:28:50     [<ffffffff810ef752>] __writepage+0x12/0x40
>> 2013-08-22 06:28:50     [<ffffffff810f090b>] write_cache_pages+0x23b/0x4e0
>> 2013-08-22 06:28:50     [<ffffffff81524437>] ? __schedule+0x3d7/0x6d0
>> 2013-08-22 06:28:50     [<ffffffff810ef740>] ? set_page_dirty+0x60/0x60
>> 2013-08-22 06:28:50     [<ffffffff810f0bfc>] generic_writepages+0x4c/0x70
>> 2013-08-22 06:28:50     [<ffffffff811e810f>] xfs_vm_writepages+0x4f/0x60
>> 2013-08-22 06:28:50     [<ffffffff810f0c3b>] do_writepages+0x1b/0x40
>> 2013-08-22 06:28:50     [<ffffffff810e6911>]
>> __filemap_fdatawrite_range+0x51/0x60
>> 2013-08-22 06:28:50     [<ffffffff81097cf0>] ? futex_wake+0x110/0x120
>> 2013-08-22 06:28:50     [<ffffffff810e699a>]
>> filemap_write_and_wait_range+0x7a/0xa0
>> 2013-08-22 06:28:50     [<ffffffff811ef9c3>] xfs_file_fsync+0x63/0x1e0
>> 2013-08-22 06:28:50     [<ffffffff8116d7c3>] vfs_fsync_range+0x23/0x30
>> 2013-08-22 06:28:50     [<ffffffff8116d7e7>] vfs_fsync+0x17/0x20
>> 2013-08-22 06:28:50     [<ffffffff8116da1c>] do_fsync+0x3c/0x60
>> 2013-08-22 06:28:50     [<ffffffff8116da4e>] SyS_fdatasync+0xe/0x20
>> 2013-08-22 06:28:49     [<ffffffff810ef740>] ? set_page_dirty+0x60/0x60
>> 2013-08-22 06:28:49     [<ffffffff810f0bfc>] generic_writepages+0x4c/0x70
>> 2013-08-22 06:28:49     [<ffffffff811639a3>] ? setxattr+0xf3/0x200
>> 2013-08-22 06:28:49     [<ffffffff811e810f>] xfs_vm_writepages+0x4f/0x60
>> 2013-08-22 06:28:49     [<ffffffff810f0c3b>] do_writepages+0x1b/0x40
>> 2013-08-22 06:28:49     [<ffffffff810e6911>]
>> __filemap_fdatawrite_range+0x51/0x60
>> 2013-08-22 06:28:49     [<ffffffff810e699a>]
>> filemap_write_and_wait_range+0x7a/0xa0
>> 2013-08-22 06:28:49     [<ffffffff811ef9c3>] xfs_file_fsync+0x63/0x1e0
>> 2013-08-22 06:28:49     [<ffffffff8116d7c3>] vfs_fsync_range+0x23/0x30
>> 2013-08-22 06:28:49     [<ffffffff8116d7e7>] vfs_fsync+0x17/0x20
>> 2013-08-22 06:28:49     [<ffffffff8116da1c>] do_fsync+0x3c/0x60
>> 2013-08-22 06:28:49     [<ffffffff8116da6b>] SyS_fsync+0xb/0x10
>> 2013-08-22 06:28:49     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
>> 2013-08-22 06:28:49     INFO: task ceph-osd:3483 blocked for more
>> than 120 seconds.
>> 2013-08-22 06:28:49     "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> 2013-08-22 06:28:49     ceph-osd D ffffffff8160f760 0 3483 1 0x00000004
>> 2013-08-22 06:28:49     ffff880c2c0d1910 0000000000000086
>> ffff880c2c0d1fd8 0000000000012c80
>> 2013-08-22 06:28:49     ffff880c2c0d0010 0000000000012c80
>> 0000000000012c80 0000000000012c80
>> 2013-08-22 06:28:49     ffff880c2c0d1fd8 0000000000012c80
>> ffff880c3a14caa0 ffff880c48dd31c0
>> 2013-08-22 06:28:49     Call Trace:
>> 2013-08-22 06:28:49     [<ffffffff815247f4>] schedule+0x24/0x70
>> 2013-08-22 06:28:49     [<ffffffff81525cfd>]
>> rwsem_down_read_failed+0x9d/0xe5
>> 2013-08-22 06:28:49     [<ffffffff812aa024>]
>> call_rwsem_down_read_failed+0x14/0x30
>> 2013-08-22 06:28:48     [<ffffffff81143261>] ? __sb_start_write+0x71/0x110
>> 2013-08-22 06:28:48     [<ffffffff8115ce51>] ?
>> __mnt_want_write_file+0x61/0x80
>> 2013-08-22 06:28:48     [<ffffffff81163b56>] SyS_fsetxattr+0xa6/0xc0
>> 2013-08-22 06:28:48     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
>> 2013-08-22 06:28:48     INFO: task ceph-osd:3482 blocked for more
>> than 120 seconds.
>> 2013-08-22 06:28:48     "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> 2013-08-22 06:28:48     ceph-osd D ffffffff8160f760 0 3482 1 0x00000004
>> 2013-08-22 06:28:48     ffff880c2c0cf910 0000000000000086
>> ffff880c2c0cffd8 0000000000012c80
>> 2013-08-22 06:28:48     ffff880c2c0ce010 0000000000012c80
>> 0000000000012c80 0000000000012c80
>> 2013-08-22 06:28:48     ffff880c2c0cffd8 0000000000012c80
>> ffff880c47768000 ffff880c48dd0000
>> 2013-08-22 06:28:48     Call Trace:
>> 2013-08-22 06:28:48     [<ffffffff815247f4>] schedule+0x24/0x70
>> 2013-08-22 06:28:48     [<ffffffff81525cfd>]
>> rwsem_down_read_failed+0x9d/0xe5
>> 2013-08-22 06:28:48     [<ffffffff812aa024>]
>> call_rwsem_down_read_failed+0x14/0x30
>> 2013-08-22 06:28:48     [<ffffffff810e8420>] ? mempool_alloc_slab+0x10/0x20
>> 2013-08-22 06:28:48     [<ffffffff815239c2>] ? down_read+0x12/0x20
>> 2013-08-22 06:28:48     [<ffffffffa010c38a>]
>> request_write+0x8a/0x380 [bcache]
>> 2013-08-22 06:28:48     [<ffffffffa010c8fb>]
>> cached_dev_make_request+0x27b/0x350 [bcache]
>> 2013-08-22 06:28:48     [<ffffffff81281982>] generic_make_request+0xc2/0x100
>> 2013-08-22 06:28:48     [<ffffffff81281a27>] submit_bio+0x67/0x130
>> 2013-08-22 06:28:48     [<ffffffff811e823f>] xfs_submit_ioend_bio+0x2f/0x40
>> 2013-08-22 06:28:48     [<ffffffff811e833e>] xfs_submit_ioend+0xee/0x130
>> 2013-08-22 06:28:48     [<ffffffff811e98bb>] xfs_vm_writepage+0x2eb/0x590
>> 2013-08-22 06:28:48     [<ffffffff810ef752>] __writepage+0x12/0x40
>> 2013-08-22 06:28:48     [<ffffffff810f090b>] write_cache_pages+0x23b/0x4e0
>> 2013-08-22 06:28:47     [<ffffffff812469b6>] _xfs_log_force+0x116/0x250
>> 2013-08-22 06:28:47     [<ffffffff8124836a>] xfs_log_force+0x2a/0x90
>> 2013-08-22 06:28:47     [<ffffffff811eb747>] ? _xfs_buf_find+0x107/0x290
>> 2013-08-22 06:28:47     [<ffffffff811eb0a3>] xfs_buf_lock+0x73/0xe0
>> 2013-08-22 06:28:47     [<ffffffff811eb747>] _xfs_buf_find+0x107/0x290
>> 2013-08-22 06:28:47     [<ffffffff811ebb1f>] xfs_buf_get_map+0x2f/0x180
>> 2013-08-22 06:28:47     [<ffffffff8124dc68>]
>> xfs_trans_get_buf_map+0xc8/0x160
>> 2013-08-22 06:28:47     [<ffffffff81224093>] xfs_da_get_buf+0xb3/0xe0
>> 2013-08-22 06:28:47     [<ffffffff81209b20>]
>> xfs_attr3_leaf_to_node+0xb0/0x240
>> 2013-08-22 06:28:47     [<ffffffff8120d12b>] ?
>> xfs_attr3_leaf_add+0x15b/0x1c0
>> 2013-08-22 06:28:47     [<ffffffff8120843c>]
>> xfs_attr_leaf_addname+0x14c/0x380
>> 2013-08-22 06:28:47     [<ffffffff81243f68>] ? xfs_trans_add_item+0x28/0x70
>> 2013-08-22 06:28:47     [<ffffffff81219983>] ? xfs_bmbt_get_all+0x13/0x20
>> 2013-08-22 06:28:47     [<ffffffff81208a14>] xfs_attr_set_int+0x3a4/0x4f0
>> 2013-08-22 06:28:47     [<ffffffff81208be1>] xfs_attr_set+0x81/0x90
>> 2013-08-22 06:28:47     [<ffffffff81201462>] xfs_xattr_set+0x42/0x60
>> 2013-08-22 06:28:47     [<ffffffff8116257b>] generic_setxattr+0x9b/0xb0
>> 2013-08-22 06:28:47     [<ffffffff8116376f>] __vfs_setxattr_noperm+0x5f/0xe0
>> 2013-08-22 06:28:47     [<ffffffff811638a4>] vfs_setxattr+0xb4/0xc0
>> 2013-08-22 06:28:47     [<ffffffff8116396e>] setxattr+0xbe/0x200
>> 2013-08-22 06:28:47     [<ffffffff811ec1ca>] xfs_buf_iorequest+0x4a/0xa0
>> 2013-08-22 06:28:47     [<ffffffff81245efe>] xlog_bdstrat+0x1e/0x60
>> 2013-08-22 06:28:47     [<ffffffff8124653d>] xlog_sync+0x2ed/0x490
>> 2013-08-22 06:28:47     [<ffffffff8124686b>]
>> xlog_state_release_iclog+0x8b/0xc0
>> 2013-08-22 06:28:46     [<ffffffff8116dd50>] SyS_syncfs+0x50/0x90
>> 2013-08-22 06:28:46     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
>> 2013-08-22 06:28:46     INFO: task ceph-osd:3481 blocked for more
>> than 120 seconds.
>> 2013-08-22 06:28:46     "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> 2013-08-22 06:28:46     ceph-osd D ffffffff8160f760 0 3481 1 0x00000004
>> 2013-08-22 06:28:46     ffff880c2c0cd500 0000000000000086
>> ffff880c2c0cdfd8 0000000000012c80
>> 2013-08-22 06:28:46     ffff880c2c0cc010 0000000000012c80
>> 0000000000012c80 0000000000012c80
>> 2013-08-22 06:28:46     ffff880c2c0cdfd8 0000000000012c80
>> ffff880c4776b1c0 ffff880c48dd4aa0
>> 2013-08-22 06:28:46     Call Trace:
>> 2013-08-22 06:28:46     [<ffffffff8124e5b6>] ? xfs_trans_brelse+0xb6/0x100
>> 2013-08-22 06:28:46     [<ffffffff815247f4>] schedule+0x24/0x70
>> 2013-08-22 06:28:46     [<ffffffff81525cfd>]
>> rwsem_down_read_failed+0x9d/0xe5
>> 2013-08-22 06:28:46     [<ffffffff812aa024>]
>> call_rwsem_down_read_failed+0x14/0x30
>> 2013-08-22 06:28:46     [<ffffffff810e8420>] ? mempool_alloc_slab+0x10/0x20
>> 2013-08-22 06:28:46     [<ffffffff815239c2>] ? down_read+0x12/0x20
>> 2013-08-22 06:28:46     [<ffffffffa010c38a>]
>> request_write+0x8a/0x380 [bcache]
>> 2013-08-22 06:28:46     [<ffffffffa010c8fb>]
>> cached_dev_make_request+0x27b/0x350 [bcache]
>> 2013-08-22 06:28:46     [<ffffffff81281982>] generic_make_request+0xc2/0x100
>> 2013-08-22 06:28:46     [<ffffffff81281a27>] submit_bio+0x67/0x130
>> 2013-08-22 06:28:46     [<ffffffff811eabf0>] xfs_buf_ioapply_map+0x180/0x1d0
>> 2013-08-22 06:28:46     [<ffffffff811ead47>] _xfs_buf_ioapply+0x107/0x150
>> 2013-08-22 06:28:46     [<ffffffff81245efe>] ? xlog_bdstrat+0x1e/0x60
>> 2013-08-22 06:28:45     "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> 2013-08-22 06:28:45     ceph-osd D ffffffff8160f760 0 3462 1 0x00000004
>> 2013-08-22 06:28:45     ffff880c375cdc28 0000000000000086
>> ffff880c375cdfd8 0000000000012c80
>> 2013-08-22 06:28:45     ffff880c375cc010 0000000000012c80
>> 0000000000012c80 0000000000012c80
>> 2013-08-22 06:28:45     ffff880c375cdfd8 0000000000012c80
>> ffff880c46318000 ffff880c48dd6380
>> 2013-08-22 06:28:45     Call Trace:
>> 2013-08-22 06:28:45     [<ffffffff810e6090>] ? __lock_page+0x70/0x70
>> 2013-08-22 06:28:45     [<ffffffff815247f4>] schedule+0x24/0x70
>> 2013-08-22 06:28:45     [<ffffffff815248c7>] io_schedule+0x87/0xd0
>> 2013-08-22 06:28:45     [<ffffffff810e6099>] sleep_on_page+0x9/0x10
>> 2013-08-22 06:28:45     [<ffffffff81522dc7>] __wait_on_bit+0x57/0x80
>> 2013-08-22 06:28:45     [<ffffffff810e5bbc>] ? find_get_pages_tag+0xcc/0x180
>> 2013-08-22 06:28:45     [<ffffffff810e62ae>] wait_on_page_bit+0x6e/0x80
>> 2013-08-22 06:28:45     [<ffffffff81070af0>] ?
>> autoremove_wake_function+0x40/0x40
>> 2013-08-22 06:28:45     [<ffffffff810f2660>] ? pagevec_lookup_tag+0x20/0x30
>> 2013-08-22 06:28:45     [<ffffffff810e67ef>]
>> filemap_fdatawait_range+0x10f/0x1b0
>> 2013-08-22 06:28:45     [<ffffffff815247f4>] ? schedule+0x24/0x70
>> 2013-08-22 06:28:45     [<ffffffff81522b3d>] ? schedule_timeout+0x16d/0x200
>> 2013-08-22 06:28:45     [<ffffffff81068e9a>] ?
>> __queue_delayed_work+0xaa/0x1a0
>> 2013-08-22 06:28:45     [<ffffffff810e68b3>] filemap_fdatawait+0x23/0x30
>> 2013-08-22 06:28:45     [<ffffffff81166e35>] wait_sb_inodes+0xb5/0x110
>> 2013-08-22 06:28:45     [<ffffffff81167a1c>] sync_inodes_sb+0x9c/0xd0
>> 2013-08-22 06:28:45     [<ffffffff8116dc66>] __sync_filesystem+0x16/0x50
>> 2013-08-22 06:28:45     [<ffffffff8116dce3>] sync_filesystem+0x43/0x60
>> 2013-08-22 06:28:44     [<ffffffffa010c8fb>]
>> cached_dev_make_request+0x27b/0x350 [bcache]
>> 2013-08-22 06:28:44     [<ffffffff81281982>] generic_make_request+0xc2/0x100
>> 2013-08-22 06:28:44     [<ffffffff81281a27>] submit_bio+0x67/0x130
>> 2013-08-22 06:28:44     [<ffffffff811e823f>] xfs_submit_ioend_bio+0x2f/0x40
>> 2013-08-22 06:28:44     [<ffffffff811e833e>] xfs_submit_ioend+0xee/0x130
>> 2013-08-22 06:28:44     [<ffffffff811e98bb>] xfs_vm_writepage+0x2eb/0x590
>> 2013-08-22 06:28:44     [<ffffffff810ef752>] __writepage+0x12/0x40
>> 2013-08-22 06:28:44     [<ffffffff810f090b>] write_cache_pages+0x23b/0x4e0
>> 2013-08-22 06:28:44     [<ffffffff81524437>] ? __schedule+0x3d7/0x6d0
>> 2013-08-22 06:28:44     [<ffffffff810ef740>] ? set_page_dirty+0x60/0x60
>> 2013-08-22 06:28:44     [<ffffffff810f0bfc>] generic_writepages+0x4c/0x70
>> 2013-08-22 06:28:44     [<ffffffff811e810f>] xfs_vm_writepages+0x4f/0x60
>> 2013-08-22 06:28:44     [<ffffffff810f0c3b>] do_writepages+0x1b/0x40
>> 2013-08-22 06:28:44     [<ffffffff810e6911>]
>> __filemap_fdatawrite_range+0x51/0x60
>> 2013-08-22 06:28:44     [<ffffffff81097cf0>] ? futex_wake+0x110/0x120
>> 2013-08-22 06:28:44     [<ffffffff810e699a>]
>> filemap_write_and_wait_range+0x7a/0xa0
>> 2013-08-22 06:28:44     [<ffffffff811ef9c3>] xfs_file_fsync+0x63/0x1e0
>> 2013-08-22 06:28:44     [<ffffffff8116d7c3>] vfs_fsync_range+0x23/0x30
>> 2013-08-22 06:28:44     [<ffffffff8116d7e7>] vfs_fsync+0x17/0x20
>> 2013-08-22 06:28:44     [<ffffffff8116da1c>] do_fsync+0x3c/0x60
>> 2013-08-22 06:28:44     [<ffffffff8116da4e>] SyS_fdatasync+0xe/0x20
>> 2013-08-22 06:28:44     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
>> 2013-08-22 06:28:44     INFO: task ceph-osd:3462 blocked for more
>> than 120 seconds.
>> 2013-08-22 06:28:43     INFO: task ceph-osd:24130 blocked for more
>> than 120 seconds.
>> 2013-08-22 06:28:43     "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> 2013-08-22 06:28:43     ceph-osd D ffffffff8160f760 0 24130 1 0x00000004
>> 2013-08-22 06:28:43     ffff880c27fe98d8 0000000000000082
>> ffff880c27fe9fd8 0000000000012c80
>> 2013-08-22 06:28:43     ffff880c27fe8010 0000000000012c80
>> 0000000000012c80 0000000000012c80
>> 2013-08-22 06:28:43     ffff880c27fe9fd8 0000000000012c80
>> ffff880c215e6380 ffff880c48decaa0
>> 2013-08-22 06:28:43     Call Trace:
>> 2013-08-22 06:28:43     [<ffffffff81170430>] ? __wait_on_buffer+0x30/0x30
>> 2013-08-22 06:28:43     [<ffffffff815247f4>] schedule+0x24/0x70
>> 2013-08-22 06:28:43     [<ffffffff815248c7>] io_schedule+0x87/0xd0
>> 2013-08-22 06:28:43     [<ffffffff81170439>] sleep_on_buffer+0x9/0x10
>> 2013-08-22 06:28:43     [<ffffffff81522dc7>] __wait_on_bit+0x57/0x80
>> 2013-08-22 06:28:43     [<ffffffff81170430>] ? __wait_on_buffer+0x30/0x30
>> 2013-08-22 06:28:43     [<ffffffff81522e63>]
>> out_of_line_wait_on_bit+0x73/0x90
>> 2013-08-22 06:28:43     [<ffffffff81070af0>] ?
>> autoremove_wake_function+0x40/0x40
>> 2013-08-22 06:28:43     [<ffffffff8116f64b>] ? submit_bh+0xb/0x10
>> 2013-08-22 06:28:43     [<ffffffff81170426>] __wait_on_buffer+0x26/0x30
>> 2013-08-22 06:28:43     [<ffffffff81172803>] __block_write_begin+0x383/0x530
>> 2013-08-22 06:28:43     [<ffffffff811e9050>] ?
>> xfs_get_blocks_direct+0x20/0x20
>> 2013-08-22 06:28:43     [<ffffffff810e6d0f>] ?
>> grab_cache_page_write_begin+0x9f/0xd0
>> 2013-08-22 06:28:43     [<ffffffff811e802f>] xfs_vm_write_begin+0x5f/0xd0
>> 2013-08-22 06:28:43     [<ffffffff810e55ba>]
>> generic_perform_write+0xca/0x200
>> 2013-08-22 06:28:43     [<ffffffff810e574f>]
>> generic_file_buffered_write+0x5f/0x90
>> 2013-08-22 06:28:43     [<ffffffff811f0275>]
>> xfs_file_buffered_aio_write+0xf5/0x180
>> 2013-08-22 06:28:43     [<ffffffff811f03ce>] xfs_file_aio_write+0xce/0x150
>> 2013-08-22 06:28:43     [<ffffffff8114ac8d>] ? path_put+0x1d/0x30
>> 2013-08-22 06:28:43     [<ffffffff81140788>] do_sync_readv_writev+0x68/0xa0
>> 2013-08-22 06:28:43     [<ffffffff81142052>] do_readv_writev+0xf2/0x2e0
>> 2013-08-22 06:28:43     [<ffffffff811f0300>] ?
>> xfs_file_buffered_aio_write+0x180/0x180
>> 2013-08-22 06:28:43     [<ffffffff811407c0>] ?
>> do_sync_readv_writev+0xa0/0xa0
>> 2013-08-22 06:28:43     [<ffffffff8114227e>] vfs_writev+0x3e/0x60
>> 2013-08-22 06:28:43     [<ffffffff811423ca>] SyS_writev+0x5a/0xc0
>> 2013-08-22 06:28:43     [<ffffffff811410d3>] ? SyS_lseek+0x53/0x80
>> 2013-08-22 06:28:43     [<ffffffff81526a92>] system_call_fastpath+0x16/0x1b
>> 2013-08-22 06:28:43     INFO: task ceph-osd:24134 blocked for more
>> than 120 seconds.
>> 2013-08-22 06:28:43     "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> 2013-08-22 06:28:43     ceph-osd D ffffffff8160f760 0 24134 1 0x00000004
>> 2013-08-22 06:28:43     ffff880c23d31708 0000000000000082
>> ffff880c23d31fd8 0000000000012c80
>> 2013-08-22 06:28:43     ffff880c23d30010 0000000000012c80
>> 0000000000012c80 0000000000012c80
>> 2013-08-22 06:28:43     ffff880c23d31fd8 0000000000012c80
>> ffff880c361e0000 ffff880c48dd4aa0
>> 2013-08-22 06:28:43     Call Trace:
>> 2013-08-22 06:28:43     [<ffffffff815247f4>] schedule+0x24/0x70
>> 2013-08-22 06:28:43     [<ffffffff81524ae9>]
>> schedule_preempt_disabled+0x9/0x10
>> 2013-08-22 06:28:43     [<ffffffff81523664>]
>> __mutex_lock_slowpath+0x194/0x240
>> 2013-08-22 06:28:43     [<ffffffff81522e9e>] mutex_lock+0x1e/0x40
>> 2013-08-22 06:28:43     [<ffffffffa00f891b>]
>> bch_bucket_alloc_set+0x3b/0x80 [bcache]
>> 2013-08-22 06:28:43     [<ffffffffa010ac1c>]
>> bch_alloc_sectors+0x12c/0x4e0 [bcache]
>> 2013-08-22 06:28:43     [<ffffffffa0104e15>] ?
>> bch_generic_make_request_hack+0x25/0xc0 [bcache]
>> 2013-08-22 06:28:43     [<ffffffffa0105013>] ?
>> __bch_submit_bbio+0x73/0x80 [bcache]
>> 2013-08-22 06:28:43     [<ffffffffa010bc98>]
>> bch_insert_data_loop+0xf8/0x610 [bcache]
>> 2013-08-22 06:28:43     [<ffffffffa010a335>] ?
>> bch_get_congested+0x25/0x70 [bcache]
>> 2013-08-22 06:28:43     [<ffffffffa010c1cd>]
>> bch_insert_data+0x1d/0x20 [bcache]
>> 2013-08-22 06:28:43     [<ffffffffa0116d13>] closure_queue+0x43/0x60
>> [bcache]
>> 2013-08-22 06:28:43     [<ffffffffa010c49e>]
>> request_write+0x19e/0x380 [bcache]
>>
>> Am 22.08.2013 01:47, schrieb Kent Overstreet:
>>> On Tue, Aug 20, 2013 at 10:07:45AM +0200, Stefan Priebe - Profihost AG wrote:
>>>> Am 20.08.2013 10:01, schrieb Stefan Priebe - Profihost AG:
>>>>> Am 20.08.2013 00:27, schrieb Kent Overstreet:
>>>>>> On Mon, Aug 19, 2013 at 12:09:24AM +0200, Stefan Priebe wrote:
>>>>>>>
>>>>>>> Vanilla 3.10.7 + bcache: Fix a writeback performance regression
>>>>>>>
>>>>>>> http://pastebin.com/raw.php?i=LXZk4cMH
>>>>>>
>>>>>> Whoops, at first I thought this was the same bug as one I'd already been
>>>>>> chasing down that had been a harmless bug - turns out I didn't look
>>>>>> closely enough at the backtrace.
>>>>>>
>>>>>> What happened is background writeback is deadlocking, because for some
>>>>>> reason the workqueue it's running out of is a singlethreaded workqueue,
>>>>>> so as soon as it decides to queue enough writeback bios that it has to
>>>>>> sleep on that semaphore (which often won't happen due to the PD
>>>>>> controller based ratelimiting) - boom, deadlock.
>>>>>>
>>>>>> Here's the fixup patch I just tested and am applying:
>>>>
>>>> Oh i'm now seeing very high CPU spikes of kworker... i don't see if i
>>>> remove bcache: Fix a writeback performance regression.
>>>
>>> Are you able to reproduce it? I'm not having any luck reproducing it...
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-22  5:59                   ` Stefan Priebe
@ 2013-08-22  6:02                     ` Kent Overstreet
  2013-08-22  7:32                       ` Stefan Priebe - Profihost AG
  0 siblings, 1 reply; 28+ messages in thread
From: Kent Overstreet @ 2013-08-22  6:02 UTC (permalink / raw)
  To: Stefan Priebe; +Cc: Jens Axboe, linux-kernel, linux-bcache

On Thu, Aug 22, 2013 at 07:59:04AM +0200, Stefan Priebe wrote:
> 
> >schedule_timeout() is not the same as
> >schedule_timeout_interruptible().
> 
> just search and replace? So i can try on my own.

The one in read_dirty(), line ~330

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-22  6:02                     ` Kent Overstreet
@ 2013-08-22  7:32                       ` Stefan Priebe - Profihost AG
  2013-08-22  7:42                         ` Kent Overstreet
  2013-08-26 19:21                         ` Stefan Priebe
  0 siblings, 2 replies; 28+ messages in thread
From: Stefan Priebe - Profihost AG @ 2013-08-22  7:32 UTC (permalink / raw)
  To: Kent Overstreet; +Cc: Jens Axboe, linux-kernel, linux-bcache

great!

Everything seems to work fine now! Except read_dirty always going to
negative values after a reboot.

Stefan

Am 22.08.2013 08:02, schrieb Kent Overstreet:
> On Thu, Aug 22, 2013 at 07:59:04AM +0200, Stefan Priebe wrote:
>>
>>> schedule_timeout() is not the same as
>>> schedule_timeout_interruptible().
>>
>> just search and replace? So i can try on my own.
> 
> The one in read_dirty(), line ~330
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-22  7:32                       ` Stefan Priebe - Profihost AG
@ 2013-08-22  7:42                         ` Kent Overstreet
  2013-08-26 19:21                         ` Stefan Priebe
  1 sibling, 0 replies; 28+ messages in thread
From: Kent Overstreet @ 2013-08-22  7:42 UTC (permalink / raw)
  To: Stefan Priebe - Profihost AG; +Cc: Jens Axboe, linux-kernel, linux-bcache

On Thu, Aug 22, 2013 at 09:32:52AM +0200, Stefan Priebe - Profihost AG wrote:
> great!
> 
> Everything seems to work fine now! Except read_dirty always going to
> negative values after a reboot.

That one's fixed in 3.11 :)

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-22  7:32                       ` Stefan Priebe - Profihost AG
  2013-08-22  7:42                         ` Kent Overstreet
@ 2013-08-26 19:21                         ` Stefan Priebe
  2013-08-28  3:37                           ` kernel neophyte
  1 sibling, 1 reply; 28+ messages in thread
From: Stefan Priebe @ 2013-08-26 19:21 UTC (permalink / raw)
  To: Kent Overstreet; +Cc: Jens Axboe, linux-kernel, linux-bcache

Hi Kent,

a new one under 3.10:

2013-08-26 21:05:30     INFO: task ceph-osd:8939 blocked for more than 
120 seconds.
2013-08-26 21:05:30     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-08-26 21:05:30     ceph-osd D ffffffff8160f760 0 8939 1 0x00000000
2013-08-26 21:05:30     ffff880bd9a61820 0000000000000086 
ffff880bd9a61fd8 0000000000012c80
2013-08-26 21:05:30     ffff880bd9a60010 0000000000012c80 
0000000000012c80 0000000000012c80
2013-08-26 21:05:30     ffff880bd9a61fd8 0000000000012c80 
ffff880bd7abb1c0 ffff880c48dee380
2013-08-26 21:05:29     [<ffffffff811e8128>] xfs_vm_readpages+0x18/0x20
2013-08-26 21:05:29     [<ffffffff810f1da3>] read_pages+0x43/0x100
2013-08-26 21:05:29     [<ffffffff810e64df>] ? __page_cache_alloc+0x9f/0xc0
2013-08-26 21:05:29     [<ffffffff810f1fab>] 
__do_page_cache_readahead+0x14b/0x160
2013-08-26 21:05:29     [<ffffffff810f1fdc>] ra_submit+0x1c/0x20
2013-08-26 21:05:29     [<ffffffff810f22a5>] ondemand_readahead+0x115/0x240
2013-08-26 21:05:29     [<ffffffff810f249e>] 
page_cache_sync_readahead+0x2e/0x40
2013-08-26 21:05:29     [<ffffffff810e7a5e>] T.1003+0x33e/0x430
2013-08-26 21:05:29     [<ffffffff810e7c30>] 
generic_file_aio_read+0xe0/0x220
2013-08-26 21:05:29     [<ffffffff811efd0a>] xfs_file_aio_read+0x15a/0x2a0
2013-08-26 21:05:29     [<ffffffff8114095a>] do_sync_read+0x7a/0xb0
2013-08-26 21:05:29     [<ffffffff81140df1>] vfs_read+0xb1/0x130
2013-08-26 21:05:29     [<ffffffff81141757>] SyS_pread64+0x97/0xa0
2013-08-26 21:05:29     [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b
2013-08-26 21:05:29     INFO: task ceph-osd:8896 blocked for more than 
120 seconds.
2013-08-26 21:05:29     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-08-26 21:05:29     ceph-osd D ffffffff8160f760 0 8896 1 0x00000000
2013-08-26 21:05:29     ffff880bd9a9dc28 0000000000000086 
ffff880bd9a9dfd8 0000000000012c80
2013-08-26 21:05:29     ffff880bd9a9c010 0000000000012c80 
0000000000012c80 0000000000012c80
2013-08-26 21:05:29     ffff880bd9a9dfd8 0000000000012c80 
ffff880bd79298e0 ffff880c48dd6380
2013-08-26 21:05:29     Call Trace:
2013-08-26 21:05:29     [<ffffffff810e60f0>] ? __lock_page+0x70/0x70
2013-08-26 21:05:28     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-08-26 21:05:28     ceph-osd D ffffffff8160f760 0 8798 1 0x00000000
2013-08-26 21:05:28     ffff880bd998b740 0000000000000086 
ffff880bd998bfd8 0000000000012c80
2013-08-26 21:05:28     ffff880bd998a010 0000000000012c80 
0000000000012c80 0000000000012c80
2013-08-26 21:05:28     ffff880bd998bfd8 0000000000012c80 
ffff880bd99c0000 ffff880c48dd6380
2013-08-26 21:05:28     Call Trace:
2013-08-26 21:05:28     [<ffffffff81524874>] schedule+0x24/0x70
2013-08-26 21:05:28     [<ffffffff81525d7d>] 
rwsem_down_read_failed+0x9d/0xe5
2013-08-26 21:05:28     [<ffffffff812aa0a4>] 
call_rwsem_down_read_failed+0x14/0x30
2013-08-26 21:05:28     [<ffffffff81523a42>] ? down_read+0x12/0x20
2013-08-26 21:05:28     [<ffffffffa01ead62>] btree_read_async+0xa2/0x1a0 
[bcache]
2013-08-26 21:05:28     [<ffffffffa01f7b73>] closure_queue+0x43/0x60 
[bcache]
2013-08-26 21:05:28     [<ffffffffa01eb903>] T.1050+0x63/0x70 [bcache]
2013-08-26 21:05:28     [<ffffffffa01ed6ea>] 
cached_dev_make_request+0x20a/0x350 [bcache]
2013-08-26 21:05:28     [<ffffffff81281952>] generic_make_request+0xc2/0x100
2013-08-26 21:05:28     [<ffffffff812819f7>] submit_bio+0x67/0x130
2013-08-26 21:05:28     [<ffffffff8117b240>] do_mpage_readpage+0x2a0/0x660
2013-08-26 21:05:28     [<ffffffff81101cee>] ? 
__inc_zone_page_state+0x2e/0x30
2013-08-26 21:05:28     [<ffffffff810e6581>] ? 
add_to_page_cache_locked+0x81/0x110
2013-08-26 21:05:28     [<ffffffff8117b76a>] mpage_readpages+0xfa/0x150
2013-08-26 21:05:28     [<ffffffff811e90c0>] ? 
xfs_get_blocks_direct+0x20/0x20
2013-08-26 21:05:28     [<ffffffff811e90c0>] ? 
xfs_get_blocks_direct+0x20/0x20
2013-08-26 21:05:28     [<ffffffff810eeda3>] ? 
__alloc_pages_nodemask+0x123/0x240
2013-08-26 21:05:27     [<ffffffff8115bf93>] ? __close_fd+0x43/0x80
2013-08-26 21:05:27     [<ffffffff8116dabe>] SyS_fdatasync+0xe/0x20
2013-08-26 21:05:27     [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b
2013-08-26 21:05:27     INFO: task ceph-osd:8746 blocked for more than 
120 seconds.
2013-08-26 21:05:27     "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
2013-08-26 21:05:27     ceph-osd D ffffffff8160f760 0 8746 1 0x00000000
2013-08-26 21:05:27     ffff880c2dde1d38 0000000000000086 
ffff880c2dde1fd8 0000000000012c80
2013-08-26 21:05:27     ffff880c2dde0010 0000000000012c80 
0000000000012c80 0000000000012c80
2013-08-26 21:05:27     ffff880c2dde1fd8 0000000000012c80 
ffff880c239eb1c0 ffff880c48dd6380
2013-08-26 21:05:27     Call Trace:
2013-08-26 21:05:27     [<ffffffff81524874>] schedule+0x24/0x70
2013-08-26 21:05:27     [<ffffffff81522bbd>] schedule_timeout+0x16d/0x200
2013-08-26 21:05:27     [<ffffffff81068eba>] ? 
__queue_delayed_work+0xaa/0x1a0
2013-08-26 21:05:27     [<ffffffff81068d19>] ? 
try_to_grab_pending+0x109/0x190
2013-08-26 21:05:27     [<ffffffff815251e5>] wait_for_completion+0x95/0x110
2013-08-26 21:05:27     [<ffffffff8107fd00>] ? try_to_wake_up+0x2a0/0x2a0
2013-08-26 21:05:27     [<ffffffff81167797>] ? bdi_queue_work+0x77/0xc0
2013-08-26 21:05:27     [<ffffffff81167863>] 
writeback_inodes_sb_nr+0x83/0xb0
2013-08-26 21:05:27     [<ffffffff811678ea>] writeback_inodes_sb+0x5a/0x70
2013-08-26 21:05:27     [<ffffffff8116dd0a>] __sync_filesystem+0x4a/0x50
2013-08-26 21:05:27     [<ffffffff8116dd42>] sync_filesystem+0x32/0x60
2013-08-26 21:05:27     [<ffffffff8116ddc0>] SyS_syncfs+0x50/0x90
2013-08-26 21:05:27     [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b
2013-08-26 21:05:27     INFO: task ceph-osd:8798 blocked for more than 
120 seconds.

Stefan

Am 22.08.2013 09:32, schrieb Stefan Priebe - Profihost AG:
> great!
>
> Everything seems to work fine now! Except read_dirty always going to
> negative values after a reboot.
>
> Stefan
>
> Am 22.08.2013 08:02, schrieb Kent Overstreet:
>> On Thu, Aug 22, 2013 at 07:59:04AM +0200, Stefan Priebe wrote:
>>>
>>>> schedule_timeout() is not the same as
>>>> schedule_timeout_interruptible().
>>>
>>> just search and replace? So i can try on my own.
>>
>> The one in read_dirty(), line ~330
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-26 19:21                         ` Stefan Priebe
@ 2013-08-28  3:37                           ` kernel neophyte
  2013-08-28  6:05                             ` Kent Overstreet
  0 siblings, 1 reply; 28+ messages in thread
From: kernel neophyte @ 2013-08-28  3:37 UTC (permalink / raw)
  To: Stefan Priebe; +Cc: Kent Overstreet, Jens Axboe, linux-kernel, linux-bcache

Hi Kent,

I am still seeing deadlock:

[  930.169234] bcache: bch_cached_dev_attach() Caching sda3 as bcache0
on set 06496904-10ca-489a-ae75-68c6a07d3db1
[ 2522.956188] INFO: task bcache_writebac:2058 blocked for more than
120 seconds.
[ 2522.956199] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 2522.956205] bcache_writebac D ffffffff81813a60     0  2058      2 0x00000000
[ 2522.956212]  ffff881c39c1fcd8 0000000000000046 0000000000000001
0000000000000001
[ 2522.956218]  ffff882f8b56b320 ffff881c39c1ffd8 ffff881c39c1ffd8
ffff881c39c1ffd8
[ 2522.956222]  ffff882fa6aeb320 ffff882f8b56b320 ffff882f8b56b320
ffff882f8b56b320
[ 2522.956227] Call Trace:
[ 2522.956241]  [<ffffffff816be299>] schedule+0x29/0x70
[ 2522.956247]  [<ffffffff816befb5>] rwsem_down_write_failed+0xf5/0x1a0
[ 2522.956255]  [<ffffffff81332c93>] call_rwsem_down_write_failed+0x13/0x20
[ 2522.956261]  [<ffffffff816bcc41>] ? down_write+0x31/0x40
[ 2522.956269]  [<ffffffff8151b782>] bch_writeback_thread+0x62/0x7f0
[ 2522.956278]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
[ 2522.956284]  [<ffffffff8151b720>] ? write_dirty+0xc0/0xc0
[ 2522.956290]  [<ffffffff81069f40>] kthread+0xc0/0xd0
[ 2522.956294]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2522.956302]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
[ 2522.956306]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2522.956313] INFO: task kworker/u64:1:2433 blocked for more than 120 seconds.
[ 2522.956317] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 2522.956322] kworker/u64:1   D ffffffff81813a60     0  2433      2 0x00000000
[ 2522.956332] Workqueue: writeback bdi_writeback_workfn (flush-252:0)
[ 2522.956336]  ffff882f899d1560 0000000000000046 ffffffff81120635
0000000000000100
[ 2522.956340]  ffff88219b488000 ffff882f899d1fd8 ffff882f899d1fd8
ffff882f899d1fd8
[ 2522.956345]  ffff882fa6aeb320 ffff88219b488000 ffffffff8131ad55
ffff88219b488000
[ 2522.956349] Call Trace:
[ 2522.956359]  [<ffffffff81120635>] ? mempool_alloc_slab+0x15/0x20
[ 2522.956367]  [<ffffffff8131ad55>] ? blk_throtl_bio+0x3b5/0x540
[ 2522.956372]  [<ffffffff816be299>] schedule+0x29/0x70
[ 2522.956378]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
[ 2522.956383]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
[ 2522.956388]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
[ 2522.956394]  [<ffffffff8151fe32>] cached_dev_make_request+0x752/0xf00
[ 2522.956400]  [<ffffffff81301af8>] ? generic_make_request_checks+0x1e8/0x3a0
[ 2522.956405]  [<ffffffff81301d7a>] generic_make_request+0xca/0x100
[ 2522.956409]  [<ffffffff81301e29>] submit_bio+0x79/0x160
[ 2522.956417]  [<ffffffff811b1710>] ? bio_alloc_bioset+0xa0/0x1d0
[ 2522.956421]  [<ffffffff811abd3f>] _submit_bh+0x13f/0x200
[ 2522.956425]  [<ffffffff811abe10>] submit_bh+0x10/0x20
[ 2522.956430]  [<ffffffff811af6b8>] __block_write_full_page+0x1d8/0x360
[ 2522.956439]  [<ffffffff8145fc0b>] ? scsi_request_fn+0xbb/0x530
[ 2522.956444]  [<ffffffff811ad3c0>] ? end_buffer_async_read+0x130/0x130
[ 2522.956449]  [<ffffffff811b2c40>] ? I_BDEV+0x10/0x10
[ 2522.956454]  [<ffffffff811b2c40>] ? I_BDEV+0x10/0x10
[ 2522.956459]  [<ffffffff811af90a>] block_write_full_page_endio+0xca/0x100
[ 2522.956464]  [<ffffffff811af955>] block_write_full_page+0x15/0x20
[ 2522.956469]  [<ffffffff811b3608>] blkdev_writepage+0x18/0x20
[ 2522.956473]  [<ffffffff81127d37>] __writepage+0x17/0x40
[ 2522.956477]  [<ffffffff8112820e>] write_cache_pages+0x20e/0x460
[ 2522.956481]  [<ffffffff81127d20>] ? set_page_dirty_lock+0x60/0x60
[ 2522.956486]  [<ffffffff811ac251>] ? __set_page_dirty+0x71/0xc0
[ 2522.956490]  [<ffffffff811284aa>] generic_writepages+0x4a/0x70
[ 2522.956495]  [<ffffffff81129cd0>] do_writepages+0x20/0x40
[ 2522.956501]  [<ffffffff811a36f5>] __writeback_single_inode+0x45/0x280
[ 2522.956507]  [<ffffffff811269e2>] ? __alloc_pages_nodemask+0x152/0x9b0
[ 2522.956513]  [<ffffffff811a495d>] writeback_sb_inodes+0x19d/0x3c0
[ 2522.956518]  [<ffffffff811a4c1e>] __writeback_inodes_wb+0x9e/0xd0
[ 2522.956523]  [<ffffffff811a4ecb>] wb_writeback+0x27b/0x320
[ 2522.956529]  [<ffffffff81196228>] ? get_nr_dirty_inodes+0x58/0x80
[ 2522.956534]  [<ffffffff811a500f>] wb_check_old_data_flush+0x9f/0xb0
[ 2522.956539]  [<ffffffff811a67e1>] wb_do_writeback+0x151/0x1d0
[ 2522.956547]  [<ffffffff8106539f>] ? set_worker_desc+0x6f/0x80
[ 2522.956552]  [<ffffffff811a68da>] bdi_writeback_workfn+0x7a/0x200
[ 2522.956557]  [<ffffffff810624d4>] process_one_work+0x174/0x490
[ 2522.956562]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
[ 2522.956567]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
[ 2522.956571]  [<ffffffff81069f40>] kthread+0xc0/0xd0
[ 2522.956575]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2522.956581]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
[ 2522.956585]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2522.956588] INFO: task iozone:2435 blocked for more than 120 seconds.
[ 2522.956593] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 2522.956597] iozone          D 0000000000000001     0  2435   1545 0x00000000
[ 2522.956602]  ffff882a75671978 0000000000000082 ffff882a75671938
ffffffff81301d7a
[ 2522.956606]  ffff882f88dc1990 ffff882a75671fd8 ffff882a75671fd8
ffff882a75671fd8
[ 2522.956611]  ffff882f8ec91990 ffff882f88dc1990 ffff882a75671978
ffff882fbf2539f8
[ 2522.956615] Call Trace:
[ 2522.956619]  [<ffffffff81301d7a>] ? generic_make_request+0xca/0x100
[ 2522.956625]  [<ffffffff816be299>] schedule+0x29/0x70
[ 2522.956630]  [<ffffffff816be36f>] io_schedule+0x8f/0xd0
[ 2522.956636]  [<ffffffff811b797c>] do_blockdev_direct_IO+0x1a7c/0x1fb0
[ 2522.956645]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
[ 2522.956652]  [<ffffffff811b7f05>] __blockdev_direct_IO+0x55/0x60
[ 2522.956658]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
[ 2522.956662]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
[ 2522.956669]  [<ffffffffa0263349>] ext2_direct_IO+0x79/0xe0 [ext2]
[ 2522.956674]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
[ 2522.956680]  [<ffffffff8104ade6>] ? current_fs_time+0x16/0x60
[ 2522.956685]  [<ffffffff8111f126>] generic_file_direct_write+0xc6/0x180
[ 2522.956690]  [<ffffffff8111f4bd>] __generic_file_aio_write+0x2dd/0x3b0
[ 2522.956696]  [<ffffffff8111f5f9>] generic_file_aio_write+0x69/0xd0
[ 2522.956702]  [<ffffffff8117b88a>] do_sync_write+0x7a/0xb0
[ 2522.956706]  [<ffffffff811bb108>] ? fsnotify+0x1f8/0x2b0
[ 2522.956712]  [<ffffffff8117c63e>] vfs_write+0xce/0x1e0
[ 2522.956716]  [<ffffffff8117cb22>] SyS_write+0x52/0xa0
[ 2522.956722]  [<ffffffff816c7a02>] system_call_fastpath+0x16/0x1b
[ 2522.956726] INFO: task kworker/2:1:2470 blocked for more than 120 seconds.
[ 2522.956730] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 2522.956735] kworker/2:1     D ffffffff81813d40     0  2470      2 0x00000000
[ 2522.956743] Workqueue: bcache bch_data_insert_keys
[ 2522.956745]  ffff882a5b0c9628 0000000000000046 ffff882a5b0c9648
0000004000000000
[ 2522.956749]  ffff882f8ec91990 ffff882a5b0c9fd8 ffff882a5b0c9fd8
ffff882a5b0c9fd8
[ 2522.956754]  ffff882fa6489990 ffff882f8ec91990 ffff882f8ec91990
ffff8828dba00d98
[ 2522.956758] Call Trace:
[ 2522.956763]  [<ffffffff816be299>] schedule+0x29/0x70
[ 2522.956769]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
[ 2522.956774]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
[ 2522.956779]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
[ 2522.956783]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
[ 2522.956791]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
[ 2522.956799]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
[ 2522.956805]  [<ffffffff81076828>] ? resched_task+0x68/0x70
[ 2522.956810]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
[ 2522.956816]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
[ 2522.956822]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
[ 2522.956827]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
[ 2522.956833]  [<ffffffff81126e9c>] __alloc_pages_nodemask+0x60c/0x9b0
[ 2522.956843]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
[ 2522.956848]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
[ 2522.956852]  [<ffffffff8150ebb3>] mca_data_alloc+0x73/0x1d0
[ 2522.956857]  [<ffffffff8150ee5a>] mca_bucket_alloc+0x14a/0x1f0
[ 2522.956861]  [<ffffffff81511020>] mca_alloc+0x360/0x470
[ 2522.956866]  [<ffffffff81517031>] ? __bch_bset_search+0x1d1/0x480
[ 2522.956871]  [<ffffffff8151183e>] bch_btree_node_get+0x10e/0x280
[ 2522.956875]  [<ffffffff81511c02>] bch_btree_map_nodes_recurse+0xe2/0x170
[ 2522.956880]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
[ 2522.956886]  [<ffffffff811b51ea>] ? dio_bio_end_io+0x5a/0x90
[ 2522.956890]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
[ 2522.956895]  [<ffffffff81514dce>] __bch_btree_map_nodes+0x13e/0x1c0
[ 2522.956899]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
[ 2522.956905]  [<ffffffff8151ac6f>] ? bch_journal+0x42f/0x4b0
[ 2522.956909]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
[ 2522.956915]  [<ffffffff8151e1be>] bch_data_insert_keys+0x3e/0x160
[ 2522.956920]  [<ffffffff810624d4>] process_one_work+0x174/0x490
[ 2522.956925]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
[ 2522.956930]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
[ 2522.956933]  [<ffffffff81069f40>] kthread+0xc0/0xd0
[ 2522.956938]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2522.956943]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
[ 2522.956947]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2522.956951] INFO: task kworker/3:2:2471 blocked for more than 120 seconds.
[ 2522.956955] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 2522.956959] kworker/3:2     D ffffffff81813a60     0  2471      2 0x00000000
[ 2522.956966] Workqueue: events update_writeback_rate
[ 2522.956968]  ffff882a5d125cd0 0000000000000046 0000000000000003
ffff882a5d125d68
[ 2522.956972]  ffff882f8ec94cb0 ffff882a5d125fd8 ffff882a5d125fd8
ffff882a5d125fd8
[ 2522.956977]  ffff882fa6aeb320 ffff882f8ec94cb0 0000000200000003
ffff882f8ec94cb0
[ 2522.956981] Call Trace:
[ 2522.956987]  [<ffffffff816be299>] schedule+0x29/0x70
[ 2522.956992]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
[ 2522.956997]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
[ 2522.957002]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
[ 2522.957007]  [<ffffffff8151b010>] update_writeback_rate+0x30/0x230
[ 2522.957011]  [<ffffffff810624d4>] process_one_work+0x174/0x490
[ 2522.957017]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
[ 2522.957021]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
[ 2522.957025]  [<ffffffff81069f40>] kthread+0xc0/0xd0
[ 2522.957030]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2522.957035]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
[ 2522.957039]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2522.957048] INFO: task kworker/u64:2:2540 blocked for more than 120 seconds.
[ 2522.957052] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 2522.957056] kworker/u64:2   D ffffffff81813a60     0  2540      2 0x00000000
[ 2522.957062] Workqueue: bch_btree_io btree_node_write_work
[ 2522.957064]  ffff882a4fdc9778 0000000000000046 ffff882a4fdc9738
ffff882f86e3be58
[ 2522.957069]  ffff882fa4bc1990 ffff882a4fdc9fd8 ffff882a4fdc9fd8
ffff882a4fdc9fd8
[ 2522.957073]  ffff882fa6aeb320 ffff882fa4bc1990 ffff882f86e3be00
ffff8828dba00d98
[ 2522.957078] Call Trace:
[ 2522.957083]  [<ffffffff816be299>] schedule+0x29/0x70
[ 2522.957088]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
[ 2522.957092]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
[ 2522.957102]  [<ffffffff81484090>] ? ata_scsiop_mode_sense+0x380/0x380
[ 2522.957107]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
[ 2522.957111]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
[ 2522.957116]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
[ 2522.957121]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
[ 2522.957126]  [<ffffffff81076828>] ? resched_task+0x68/0x70
[ 2522.957130]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
[ 2522.957135]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
[ 2522.957140]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
[ 2522.957146]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
[ 2522.957152]  [<ffffffff81126e9c>] __alloc_pages_nodemask+0x60c/0x9b0
[ 2522.957158]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
[ 2522.957163]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
[ 2522.957168]  [<ffffffff81517fc8>] __btree_sort+0x48/0x230
[ 2522.957173]  [<ffffffff8151765c>] ? __bch_btree_iter_init+0x7c/0xc0
[ 2522.957178]  [<ffffffff81518301>] bch_btree_sort_partial+0x101/0x120
[ 2522.957182]  [<ffffffff8150f000>] ? __btree_node_write_done+0x100/0x100
[ 2522.957187]  [<ffffffff81518468>] bch_btree_sort_lazy+0x68/0x90
[ 2522.957191]  [<ffffffff815109ba>] bch_btree_node_write+0x36a/0x4a0
[ 2522.957196]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
[ 2522.957201]  [<ffffffff8106212c>] ? pwq_activate_delayed_work+0x4c/0xb0
[ 2522.957205]  [<ffffffff81511477>] btree_node_write_work+0x57/0x80
[ 2522.957210]  [<ffffffff810624d4>] process_one_work+0x174/0x490
[ 2522.957215]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
[ 2522.957219]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
[ 2522.957223]  [<ffffffff81069f40>] kthread+0xc0/0xd0
[ 2522.957228]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2522.957233]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
[ 2522.957237]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0


I can easily reproduce this, please let me know if you require any
info/test any patch.

Thanks,
Neo

On Mon, Aug 26, 2013 at 12:21 PM, Stefan Priebe <s.priebe@profihost.ag> wrote:
> Hi Kent,
>
> a new one under 3.10:
>
> 2013-08-26 21:05:30     INFO: task ceph-osd:8939 blocked for more than 120
> seconds.
> 2013-08-26 21:05:30     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> 2013-08-26 21:05:30     ceph-osd D ffffffff8160f760 0 8939 1 0x00000000
> 2013-08-26 21:05:30     ffff880bd9a61820 0000000000000086 ffff880bd9a61fd8
> 0000000000012c80
> 2013-08-26 21:05:30     ffff880bd9a60010 0000000000012c80 0000000000012c80
> 0000000000012c80
> 2013-08-26 21:05:30     ffff880bd9a61fd8 0000000000012c80 ffff880bd7abb1c0
> ffff880c48dee380
> 2013-08-26 21:05:29     [<ffffffff811e8128>] xfs_vm_readpages+0x18/0x20
> 2013-08-26 21:05:29     [<ffffffff810f1da3>] read_pages+0x43/0x100
> 2013-08-26 21:05:29     [<ffffffff810e64df>] ? __page_cache_alloc+0x9f/0xc0
> 2013-08-26 21:05:29     [<ffffffff810f1fab>]
> __do_page_cache_readahead+0x14b/0x160
> 2013-08-26 21:05:29     [<ffffffff810f1fdc>] ra_submit+0x1c/0x20
> 2013-08-26 21:05:29     [<ffffffff810f22a5>] ondemand_readahead+0x115/0x240
> 2013-08-26 21:05:29     [<ffffffff810f249e>]
> page_cache_sync_readahead+0x2e/0x40
> 2013-08-26 21:05:29     [<ffffffff810e7a5e>] T.1003+0x33e/0x430
> 2013-08-26 21:05:29     [<ffffffff810e7c30>]
> generic_file_aio_read+0xe0/0x220
> 2013-08-26 21:05:29     [<ffffffff811efd0a>] xfs_file_aio_read+0x15a/0x2a0
> 2013-08-26 21:05:29     [<ffffffff8114095a>] do_sync_read+0x7a/0xb0
> 2013-08-26 21:05:29     [<ffffffff81140df1>] vfs_read+0xb1/0x130
> 2013-08-26 21:05:29     [<ffffffff81141757>] SyS_pread64+0x97/0xa0
> 2013-08-26 21:05:29     [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b
> 2013-08-26 21:05:29     INFO: task ceph-osd:8896 blocked for more than 120
> seconds.
> 2013-08-26 21:05:29     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> 2013-08-26 21:05:29     ceph-osd D ffffffff8160f760 0 8896 1 0x00000000
> 2013-08-26 21:05:29     ffff880bd9a9dc28 0000000000000086 ffff880bd9a9dfd8
> 0000000000012c80
> 2013-08-26 21:05:29     ffff880bd9a9c010 0000000000012c80 0000000000012c80
> 0000000000012c80
> 2013-08-26 21:05:29     ffff880bd9a9dfd8 0000000000012c80 ffff880bd79298e0
> ffff880c48dd6380
> 2013-08-26 21:05:29     Call Trace:
> 2013-08-26 21:05:29     [<ffffffff810e60f0>] ? __lock_page+0x70/0x70
> 2013-08-26 21:05:28     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> 2013-08-26 21:05:28     ceph-osd D ffffffff8160f760 0 8798 1 0x00000000
> 2013-08-26 21:05:28     ffff880bd998b740 0000000000000086 ffff880bd998bfd8
> 0000000000012c80
> 2013-08-26 21:05:28     ffff880bd998a010 0000000000012c80 0000000000012c80
> 0000000000012c80
> 2013-08-26 21:05:28     ffff880bd998bfd8 0000000000012c80 ffff880bd99c0000
> ffff880c48dd6380
> 2013-08-26 21:05:28     Call Trace:
> 2013-08-26 21:05:28     [<ffffffff81524874>] schedule+0x24/0x70
> 2013-08-26 21:05:28     [<ffffffff81525d7d>]
> rwsem_down_read_failed+0x9d/0xe5
> 2013-08-26 21:05:28     [<ffffffff812aa0a4>]
> call_rwsem_down_read_failed+0x14/0x30
> 2013-08-26 21:05:28     [<ffffffff81523a42>] ? down_read+0x12/0x20
> 2013-08-26 21:05:28     [<ffffffffa01ead62>] btree_read_async+0xa2/0x1a0
> [bcache]
> 2013-08-26 21:05:28     [<ffffffffa01f7b73>] closure_queue+0x43/0x60
> [bcache]
> 2013-08-26 21:05:28     [<ffffffffa01eb903>] T.1050+0x63/0x70 [bcache]
> 2013-08-26 21:05:28     [<ffffffffa01ed6ea>]
> cached_dev_make_request+0x20a/0x350 [bcache]
> 2013-08-26 21:05:28     [<ffffffff81281952>] generic_make_request+0xc2/0x100
> 2013-08-26 21:05:28     [<ffffffff812819f7>] submit_bio+0x67/0x130
> 2013-08-26 21:05:28     [<ffffffff8117b240>] do_mpage_readpage+0x2a0/0x660
> 2013-08-26 21:05:28     [<ffffffff81101cee>] ?
> __inc_zone_page_state+0x2e/0x30
> 2013-08-26 21:05:28     [<ffffffff810e6581>] ?
> add_to_page_cache_locked+0x81/0x110
> 2013-08-26 21:05:28     [<ffffffff8117b76a>] mpage_readpages+0xfa/0x150
> 2013-08-26 21:05:28     [<ffffffff811e90c0>] ?
> xfs_get_blocks_direct+0x20/0x20
> 2013-08-26 21:05:28     [<ffffffff811e90c0>] ?
> xfs_get_blocks_direct+0x20/0x20
> 2013-08-26 21:05:28     [<ffffffff810eeda3>] ?
> __alloc_pages_nodemask+0x123/0x240
> 2013-08-26 21:05:27     [<ffffffff8115bf93>] ? __close_fd+0x43/0x80
> 2013-08-26 21:05:27     [<ffffffff8116dabe>] SyS_fdatasync+0xe/0x20
> 2013-08-26 21:05:27     [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b
> 2013-08-26 21:05:27     INFO: task ceph-osd:8746 blocked for more than 120
> seconds.
> 2013-08-26 21:05:27     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> 2013-08-26 21:05:27     ceph-osd D ffffffff8160f760 0 8746 1 0x00000000
> 2013-08-26 21:05:27     ffff880c2dde1d38 0000000000000086 ffff880c2dde1fd8
> 0000000000012c80
> 2013-08-26 21:05:27     ffff880c2dde0010 0000000000012c80 0000000000012c80
> 0000000000012c80
> 2013-08-26 21:05:27     ffff880c2dde1fd8 0000000000012c80 ffff880c239eb1c0
> ffff880c48dd6380
> 2013-08-26 21:05:27     Call Trace:
> 2013-08-26 21:05:27     [<ffffffff81524874>] schedule+0x24/0x70
> 2013-08-26 21:05:27     [<ffffffff81522bbd>] schedule_timeout+0x16d/0x200
> 2013-08-26 21:05:27     [<ffffffff81068eba>] ?
> __queue_delayed_work+0xaa/0x1a0
> 2013-08-26 21:05:27     [<ffffffff81068d19>] ?
> try_to_grab_pending+0x109/0x190
> 2013-08-26 21:05:27     [<ffffffff815251e5>] wait_for_completion+0x95/0x110
> 2013-08-26 21:05:27     [<ffffffff8107fd00>] ? try_to_wake_up+0x2a0/0x2a0
> 2013-08-26 21:05:27     [<ffffffff81167797>] ? bdi_queue_work+0x77/0xc0
> 2013-08-26 21:05:27     [<ffffffff81167863>]
> writeback_inodes_sb_nr+0x83/0xb0
> 2013-08-26 21:05:27     [<ffffffff811678ea>] writeback_inodes_sb+0x5a/0x70
> 2013-08-26 21:05:27     [<ffffffff8116dd0a>] __sync_filesystem+0x4a/0x50
> 2013-08-26 21:05:27     [<ffffffff8116dd42>] sync_filesystem+0x32/0x60
> 2013-08-26 21:05:27     [<ffffffff8116ddc0>] SyS_syncfs+0x50/0x90
> 2013-08-26 21:05:27     [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b
> 2013-08-26 21:05:27     INFO: task ceph-osd:8798 blocked for more than 120
> seconds.
>
> Stefan
>
> Am 22.08.2013 09:32, schrieb Stefan Priebe - Profihost AG:
>
>> great!
>>
>> Everything seems to work fine now! Except read_dirty always going to
>> negative values after a reboot.
>>
>> Stefan
>>
>> Am 22.08.2013 08:02, schrieb Kent Overstreet:
>>>
>>> On Thu, Aug 22, 2013 at 07:59:04AM +0200, Stefan Priebe wrote:
>>>>
>>>>
>>>>> schedule_timeout() is not the same as
>>>>> schedule_timeout_interruptible().
>>>>
>>>>
>>>> just search and replace? So i can try on my own.
>>>
>>>
>>> The one in read_dirty(), line ~330
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-bcache"
>>> in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-28  3:37                           ` kernel neophyte
@ 2013-08-28  6:05                             ` Kent Overstreet
  2013-08-28 18:12                               ` kernel neophyte
  0 siblings, 1 reply; 28+ messages in thread
From: Kent Overstreet @ 2013-08-28  6:05 UTC (permalink / raw)
  To: kernel neophyte; +Cc: Stefan Priebe, Jens Axboe, linux-kernel, linux-bcache

On Tue, Aug 27, 2013 at 08:37:54PM -0700, kernel neophyte wrote:
> Hi Kent,
> 
> I am still seeing deadlock:

Seeing blk_throtl_bio (what asshole misspells words in their function
names, so when you go to grep for them you grep for the wrong thing?) in
the backtrace - is that what it takes to hit it?

> 
> [  930.169234] bcache: bch_cached_dev_attach() Caching sda3 as bcache0
> on set 06496904-10ca-489a-ae75-68c6a07d3db1
> [ 2522.956188] INFO: task bcache_writebac:2058 blocked for more than
> 120 seconds.
> [ 2522.956199] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 2522.956205] bcache_writebac D ffffffff81813a60     0  2058      2 0x00000000
> [ 2522.956212]  ffff881c39c1fcd8 0000000000000046 0000000000000001
> 0000000000000001
> [ 2522.956218]  ffff882f8b56b320 ffff881c39c1ffd8 ffff881c39c1ffd8
> ffff881c39c1ffd8
> [ 2522.956222]  ffff882fa6aeb320 ffff882f8b56b320 ffff882f8b56b320
> ffff882f8b56b320
> [ 2522.956227] Call Trace:
> [ 2522.956241]  [<ffffffff816be299>] schedule+0x29/0x70
> [ 2522.956247]  [<ffffffff816befb5>] rwsem_down_write_failed+0xf5/0x1a0
> [ 2522.956255]  [<ffffffff81332c93>] call_rwsem_down_write_failed+0x13/0x20
> [ 2522.956261]  [<ffffffff816bcc41>] ? down_write+0x31/0x40
> [ 2522.956269]  [<ffffffff8151b782>] bch_writeback_thread+0x62/0x7f0
> [ 2522.956278]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
> [ 2522.956284]  [<ffffffff8151b720>] ? write_dirty+0xc0/0xc0
> [ 2522.956290]  [<ffffffff81069f40>] kthread+0xc0/0xd0
> [ 2522.956294]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2522.956302]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
> [ 2522.956306]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2522.956313] INFO: task kworker/u64:1:2433 blocked for more than 120 seconds.
> [ 2522.956317] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 2522.956322] kworker/u64:1   D ffffffff81813a60     0  2433      2 0x00000000
> [ 2522.956332] Workqueue: writeback bdi_writeback_workfn (flush-252:0)
> [ 2522.956336]  ffff882f899d1560 0000000000000046 ffffffff81120635
> 0000000000000100
> [ 2522.956340]  ffff88219b488000 ffff882f899d1fd8 ffff882f899d1fd8
> ffff882f899d1fd8
> [ 2522.956345]  ffff882fa6aeb320 ffff88219b488000 ffffffff8131ad55
> ffff88219b488000
> [ 2522.956349] Call Trace:
> [ 2522.956359]  [<ffffffff81120635>] ? mempool_alloc_slab+0x15/0x20
> [ 2522.956367]  [<ffffffff8131ad55>] ? blk_throtl_bio+0x3b5/0x540
> [ 2522.956372]  [<ffffffff816be299>] schedule+0x29/0x70
> [ 2522.956378]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
> [ 2522.956383]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
> [ 2522.956388]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
> [ 2522.956394]  [<ffffffff8151fe32>] cached_dev_make_request+0x752/0xf00
> [ 2522.956400]  [<ffffffff81301af8>] ? generic_make_request_checks+0x1e8/0x3a0
> [ 2522.956405]  [<ffffffff81301d7a>] generic_make_request+0xca/0x100
> [ 2522.956409]  [<ffffffff81301e29>] submit_bio+0x79/0x160
> [ 2522.956417]  [<ffffffff811b1710>] ? bio_alloc_bioset+0xa0/0x1d0
> [ 2522.956421]  [<ffffffff811abd3f>] _submit_bh+0x13f/0x200
> [ 2522.956425]  [<ffffffff811abe10>] submit_bh+0x10/0x20
> [ 2522.956430]  [<ffffffff811af6b8>] __block_write_full_page+0x1d8/0x360
> [ 2522.956439]  [<ffffffff8145fc0b>] ? scsi_request_fn+0xbb/0x530
> [ 2522.956444]  [<ffffffff811ad3c0>] ? end_buffer_async_read+0x130/0x130
> [ 2522.956449]  [<ffffffff811b2c40>] ? I_BDEV+0x10/0x10
> [ 2522.956454]  [<ffffffff811b2c40>] ? I_BDEV+0x10/0x10
> [ 2522.956459]  [<ffffffff811af90a>] block_write_full_page_endio+0xca/0x100
> [ 2522.956464]  [<ffffffff811af955>] block_write_full_page+0x15/0x20
> [ 2522.956469]  [<ffffffff811b3608>] blkdev_writepage+0x18/0x20
> [ 2522.956473]  [<ffffffff81127d37>] __writepage+0x17/0x40
> [ 2522.956477]  [<ffffffff8112820e>] write_cache_pages+0x20e/0x460
> [ 2522.956481]  [<ffffffff81127d20>] ? set_page_dirty_lock+0x60/0x60
> [ 2522.956486]  [<ffffffff811ac251>] ? __set_page_dirty+0x71/0xc0
> [ 2522.956490]  [<ffffffff811284aa>] generic_writepages+0x4a/0x70
> [ 2522.956495]  [<ffffffff81129cd0>] do_writepages+0x20/0x40
> [ 2522.956501]  [<ffffffff811a36f5>] __writeback_single_inode+0x45/0x280
> [ 2522.956507]  [<ffffffff811269e2>] ? __alloc_pages_nodemask+0x152/0x9b0
> [ 2522.956513]  [<ffffffff811a495d>] writeback_sb_inodes+0x19d/0x3c0
> [ 2522.956518]  [<ffffffff811a4c1e>] __writeback_inodes_wb+0x9e/0xd0
> [ 2522.956523]  [<ffffffff811a4ecb>] wb_writeback+0x27b/0x320
> [ 2522.956529]  [<ffffffff81196228>] ? get_nr_dirty_inodes+0x58/0x80
> [ 2522.956534]  [<ffffffff811a500f>] wb_check_old_data_flush+0x9f/0xb0
> [ 2522.956539]  [<ffffffff811a67e1>] wb_do_writeback+0x151/0x1d0
> [ 2522.956547]  [<ffffffff8106539f>] ? set_worker_desc+0x6f/0x80
> [ 2522.956552]  [<ffffffff811a68da>] bdi_writeback_workfn+0x7a/0x200
> [ 2522.956557]  [<ffffffff810624d4>] process_one_work+0x174/0x490
> [ 2522.956562]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
> [ 2522.956567]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
> [ 2522.956571]  [<ffffffff81069f40>] kthread+0xc0/0xd0
> [ 2522.956575]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2522.956581]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
> [ 2522.956585]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2522.956588] INFO: task iozone:2435 blocked for more than 120 seconds.
> [ 2522.956593] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 2522.956597] iozone          D 0000000000000001     0  2435   1545 0x00000000
> [ 2522.956602]  ffff882a75671978 0000000000000082 ffff882a75671938
> ffffffff81301d7a
> [ 2522.956606]  ffff882f88dc1990 ffff882a75671fd8 ffff882a75671fd8
> ffff882a75671fd8
> [ 2522.956611]  ffff882f8ec91990 ffff882f88dc1990 ffff882a75671978
> ffff882fbf2539f8
> [ 2522.956615] Call Trace:
> [ 2522.956619]  [<ffffffff81301d7a>] ? generic_make_request+0xca/0x100
> [ 2522.956625]  [<ffffffff816be299>] schedule+0x29/0x70
> [ 2522.956630]  [<ffffffff816be36f>] io_schedule+0x8f/0xd0
> [ 2522.956636]  [<ffffffff811b797c>] do_blockdev_direct_IO+0x1a7c/0x1fb0
> [ 2522.956645]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
> [ 2522.956652]  [<ffffffff811b7f05>] __blockdev_direct_IO+0x55/0x60
> [ 2522.956658]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
> [ 2522.956662]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
> [ 2522.956669]  [<ffffffffa0263349>] ext2_direct_IO+0x79/0xe0 [ext2]
> [ 2522.956674]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
> [ 2522.956680]  [<ffffffff8104ade6>] ? current_fs_time+0x16/0x60
> [ 2522.956685]  [<ffffffff8111f126>] generic_file_direct_write+0xc6/0x180
> [ 2522.956690]  [<ffffffff8111f4bd>] __generic_file_aio_write+0x2dd/0x3b0
> [ 2522.956696]  [<ffffffff8111f5f9>] generic_file_aio_write+0x69/0xd0
> [ 2522.956702]  [<ffffffff8117b88a>] do_sync_write+0x7a/0xb0
> [ 2522.956706]  [<ffffffff811bb108>] ? fsnotify+0x1f8/0x2b0
> [ 2522.956712]  [<ffffffff8117c63e>] vfs_write+0xce/0x1e0
> [ 2522.956716]  [<ffffffff8117cb22>] SyS_write+0x52/0xa0
> [ 2522.956722]  [<ffffffff816c7a02>] system_call_fastpath+0x16/0x1b
> [ 2522.956726] INFO: task kworker/2:1:2470 blocked for more than 120 seconds.
> [ 2522.956730] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 2522.956735] kworker/2:1     D ffffffff81813d40     0  2470      2 0x00000000
> [ 2522.956743] Workqueue: bcache bch_data_insert_keys
> [ 2522.956745]  ffff882a5b0c9628 0000000000000046 ffff882a5b0c9648
> 0000004000000000
> [ 2522.956749]  ffff882f8ec91990 ffff882a5b0c9fd8 ffff882a5b0c9fd8
> ffff882a5b0c9fd8
> [ 2522.956754]  ffff882fa6489990 ffff882f8ec91990 ffff882f8ec91990
> ffff8828dba00d98
> [ 2522.956758] Call Trace:
> [ 2522.956763]  [<ffffffff816be299>] schedule+0x29/0x70
> [ 2522.956769]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
> [ 2522.956774]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
> [ 2522.956779]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
> [ 2522.956783]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
> [ 2522.956791]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
> [ 2522.956799]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
> [ 2522.956805]  [<ffffffff81076828>] ? resched_task+0x68/0x70
> [ 2522.956810]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
> [ 2522.956816]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
> [ 2522.956822]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
> [ 2522.956827]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
> [ 2522.956833]  [<ffffffff81126e9c>] __alloc_pages_nodemask+0x60c/0x9b0
> [ 2522.956843]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
> [ 2522.956848]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
> [ 2522.956852]  [<ffffffff8150ebb3>] mca_data_alloc+0x73/0x1d0
> [ 2522.956857]  [<ffffffff8150ee5a>] mca_bucket_alloc+0x14a/0x1f0
> [ 2522.956861]  [<ffffffff81511020>] mca_alloc+0x360/0x470
> [ 2522.956866]  [<ffffffff81517031>] ? __bch_bset_search+0x1d1/0x480
> [ 2522.956871]  [<ffffffff8151183e>] bch_btree_node_get+0x10e/0x280
> [ 2522.956875]  [<ffffffff81511c02>] bch_btree_map_nodes_recurse+0xe2/0x170
> [ 2522.956880]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
> [ 2522.956886]  [<ffffffff811b51ea>] ? dio_bio_end_io+0x5a/0x90
> [ 2522.956890]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
> [ 2522.956895]  [<ffffffff81514dce>] __bch_btree_map_nodes+0x13e/0x1c0
> [ 2522.956899]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
> [ 2522.956905]  [<ffffffff8151ac6f>] ? bch_journal+0x42f/0x4b0
> [ 2522.956909]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
> [ 2522.956915]  [<ffffffff8151e1be>] bch_data_insert_keys+0x3e/0x160
> [ 2522.956920]  [<ffffffff810624d4>] process_one_work+0x174/0x490
> [ 2522.956925]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
> [ 2522.956930]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
> [ 2522.956933]  [<ffffffff81069f40>] kthread+0xc0/0xd0
> [ 2522.956938]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2522.956943]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
> [ 2522.956947]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2522.956951] INFO: task kworker/3:2:2471 blocked for more than 120 seconds.
> [ 2522.956955] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 2522.956959] kworker/3:2     D ffffffff81813a60     0  2471      2 0x00000000
> [ 2522.956966] Workqueue: events update_writeback_rate
> [ 2522.956968]  ffff882a5d125cd0 0000000000000046 0000000000000003
> ffff882a5d125d68
> [ 2522.956972]  ffff882f8ec94cb0 ffff882a5d125fd8 ffff882a5d125fd8
> ffff882a5d125fd8
> [ 2522.956977]  ffff882fa6aeb320 ffff882f8ec94cb0 0000000200000003
> ffff882f8ec94cb0
> [ 2522.956981] Call Trace:
> [ 2522.956987]  [<ffffffff816be299>] schedule+0x29/0x70
> [ 2522.956992]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
> [ 2522.956997]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
> [ 2522.957002]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
> [ 2522.957007]  [<ffffffff8151b010>] update_writeback_rate+0x30/0x230
> [ 2522.957011]  [<ffffffff810624d4>] process_one_work+0x174/0x490
> [ 2522.957017]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
> [ 2522.957021]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
> [ 2522.957025]  [<ffffffff81069f40>] kthread+0xc0/0xd0
> [ 2522.957030]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2522.957035]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
> [ 2522.957039]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2522.957048] INFO: task kworker/u64:2:2540 blocked for more than 120 seconds.
> [ 2522.957052] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 2522.957056] kworker/u64:2   D ffffffff81813a60     0  2540      2 0x00000000
> [ 2522.957062] Workqueue: bch_btree_io btree_node_write_work
> [ 2522.957064]  ffff882a4fdc9778 0000000000000046 ffff882a4fdc9738
> ffff882f86e3be58
> [ 2522.957069]  ffff882fa4bc1990 ffff882a4fdc9fd8 ffff882a4fdc9fd8
> ffff882a4fdc9fd8
> [ 2522.957073]  ffff882fa6aeb320 ffff882fa4bc1990 ffff882f86e3be00
> ffff8828dba00d98
> [ 2522.957078] Call Trace:
> [ 2522.957083]  [<ffffffff816be299>] schedule+0x29/0x70
> [ 2522.957088]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
> [ 2522.957092]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
> [ 2522.957102]  [<ffffffff81484090>] ? ata_scsiop_mode_sense+0x380/0x380
> [ 2522.957107]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
> [ 2522.957111]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
> [ 2522.957116]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
> [ 2522.957121]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
> [ 2522.957126]  [<ffffffff81076828>] ? resched_task+0x68/0x70
> [ 2522.957130]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
> [ 2522.957135]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
> [ 2522.957140]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
> [ 2522.957146]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
> [ 2522.957152]  [<ffffffff81126e9c>] __alloc_pages_nodemask+0x60c/0x9b0
> [ 2522.957158]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
> [ 2522.957163]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
> [ 2522.957168]  [<ffffffff81517fc8>] __btree_sort+0x48/0x230
> [ 2522.957173]  [<ffffffff8151765c>] ? __bch_btree_iter_init+0x7c/0xc0
> [ 2522.957178]  [<ffffffff81518301>] bch_btree_sort_partial+0x101/0x120
> [ 2522.957182]  [<ffffffff8150f000>] ? __btree_node_write_done+0x100/0x100
> [ 2522.957187]  [<ffffffff81518468>] bch_btree_sort_lazy+0x68/0x90
> [ 2522.957191]  [<ffffffff815109ba>] bch_btree_node_write+0x36a/0x4a0
> [ 2522.957196]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
> [ 2522.957201]  [<ffffffff8106212c>] ? pwq_activate_delayed_work+0x4c/0xb0
> [ 2522.957205]  [<ffffffff81511477>] btree_node_write_work+0x57/0x80
> [ 2522.957210]  [<ffffffff810624d4>] process_one_work+0x174/0x490
> [ 2522.957215]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
> [ 2522.957219]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
> [ 2522.957223]  [<ffffffff81069f40>] kthread+0xc0/0xd0
> [ 2522.957228]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2522.957233]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
> [ 2522.957237]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> 
> 
> I can easily reproduce this, please let me know if you require any
> info/test any patch.
> 
> Thanks,
> Neo
> 
> On Mon, Aug 26, 2013 at 12:21 PM, Stefan Priebe <s.priebe@profihost.ag> wrote:
> > Hi Kent,
> >
> > a new one under 3.10:
> >
> > 2013-08-26 21:05:30     INFO: task ceph-osd:8939 blocked for more than 120
> > seconds.
> > 2013-08-26 21:05:30     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> > disables this message.
> > 2013-08-26 21:05:30     ceph-osd D ffffffff8160f760 0 8939 1 0x00000000
> > 2013-08-26 21:05:30     ffff880bd9a61820 0000000000000086 ffff880bd9a61fd8
> > 0000000000012c80
> > 2013-08-26 21:05:30     ffff880bd9a60010 0000000000012c80 0000000000012c80
> > 0000000000012c80
> > 2013-08-26 21:05:30     ffff880bd9a61fd8 0000000000012c80 ffff880bd7abb1c0
> > ffff880c48dee380
> > 2013-08-26 21:05:29     [<ffffffff811e8128>] xfs_vm_readpages+0x18/0x20
> > 2013-08-26 21:05:29     [<ffffffff810f1da3>] read_pages+0x43/0x100
> > 2013-08-26 21:05:29     [<ffffffff810e64df>] ? __page_cache_alloc+0x9f/0xc0
> > 2013-08-26 21:05:29     [<ffffffff810f1fab>]
> > __do_page_cache_readahead+0x14b/0x160
> > 2013-08-26 21:05:29     [<ffffffff810f1fdc>] ra_submit+0x1c/0x20
> > 2013-08-26 21:05:29     [<ffffffff810f22a5>] ondemand_readahead+0x115/0x240
> > 2013-08-26 21:05:29     [<ffffffff810f249e>]
> > page_cache_sync_readahead+0x2e/0x40
> > 2013-08-26 21:05:29     [<ffffffff810e7a5e>] T.1003+0x33e/0x430
> > 2013-08-26 21:05:29     [<ffffffff810e7c30>]
> > generic_file_aio_read+0xe0/0x220
> > 2013-08-26 21:05:29     [<ffffffff811efd0a>] xfs_file_aio_read+0x15a/0x2a0
> > 2013-08-26 21:05:29     [<ffffffff8114095a>] do_sync_read+0x7a/0xb0
> > 2013-08-26 21:05:29     [<ffffffff81140df1>] vfs_read+0xb1/0x130
> > 2013-08-26 21:05:29     [<ffffffff81141757>] SyS_pread64+0x97/0xa0
> > 2013-08-26 21:05:29     [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b
> > 2013-08-26 21:05:29     INFO: task ceph-osd:8896 blocked for more than 120
> > seconds.
> > 2013-08-26 21:05:29     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> > disables this message.
> > 2013-08-26 21:05:29     ceph-osd D ffffffff8160f760 0 8896 1 0x00000000
> > 2013-08-26 21:05:29     ffff880bd9a9dc28 0000000000000086 ffff880bd9a9dfd8
> > 0000000000012c80
> > 2013-08-26 21:05:29     ffff880bd9a9c010 0000000000012c80 0000000000012c80
> > 0000000000012c80
> > 2013-08-26 21:05:29     ffff880bd9a9dfd8 0000000000012c80 ffff880bd79298e0
> > ffff880c48dd6380
> > 2013-08-26 21:05:29     Call Trace:
> > 2013-08-26 21:05:29     [<ffffffff810e60f0>] ? __lock_page+0x70/0x70
> > 2013-08-26 21:05:28     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> > disables this message.
> > 2013-08-26 21:05:28     ceph-osd D ffffffff8160f760 0 8798 1 0x00000000
> > 2013-08-26 21:05:28     ffff880bd998b740 0000000000000086 ffff880bd998bfd8
> > 0000000000012c80
> > 2013-08-26 21:05:28     ffff880bd998a010 0000000000012c80 0000000000012c80
> > 0000000000012c80
> > 2013-08-26 21:05:28     ffff880bd998bfd8 0000000000012c80 ffff880bd99c0000
> > ffff880c48dd6380
> > 2013-08-26 21:05:28     Call Trace:
> > 2013-08-26 21:05:28     [<ffffffff81524874>] schedule+0x24/0x70
> > 2013-08-26 21:05:28     [<ffffffff81525d7d>]
> > rwsem_down_read_failed+0x9d/0xe5
> > 2013-08-26 21:05:28     [<ffffffff812aa0a4>]
> > call_rwsem_down_read_failed+0x14/0x30
> > 2013-08-26 21:05:28     [<ffffffff81523a42>] ? down_read+0x12/0x20
> > 2013-08-26 21:05:28     [<ffffffffa01ead62>] btree_read_async+0xa2/0x1a0
> > [bcache]
> > 2013-08-26 21:05:28     [<ffffffffa01f7b73>] closure_queue+0x43/0x60
> > [bcache]
> > 2013-08-26 21:05:28     [<ffffffffa01eb903>] T.1050+0x63/0x70 [bcache]
> > 2013-08-26 21:05:28     [<ffffffffa01ed6ea>]
> > cached_dev_make_request+0x20a/0x350 [bcache]
> > 2013-08-26 21:05:28     [<ffffffff81281952>] generic_make_request+0xc2/0x100
> > 2013-08-26 21:05:28     [<ffffffff812819f7>] submit_bio+0x67/0x130
> > 2013-08-26 21:05:28     [<ffffffff8117b240>] do_mpage_readpage+0x2a0/0x660
> > 2013-08-26 21:05:28     [<ffffffff81101cee>] ?
> > __inc_zone_page_state+0x2e/0x30
> > 2013-08-26 21:05:28     [<ffffffff810e6581>] ?
> > add_to_page_cache_locked+0x81/0x110
> > 2013-08-26 21:05:28     [<ffffffff8117b76a>] mpage_readpages+0xfa/0x150
> > 2013-08-26 21:05:28     [<ffffffff811e90c0>] ?
> > xfs_get_blocks_direct+0x20/0x20
> > 2013-08-26 21:05:28     [<ffffffff811e90c0>] ?
> > xfs_get_blocks_direct+0x20/0x20
> > 2013-08-26 21:05:28     [<ffffffff810eeda3>] ?
> > __alloc_pages_nodemask+0x123/0x240
> > 2013-08-26 21:05:27     [<ffffffff8115bf93>] ? __close_fd+0x43/0x80
> > 2013-08-26 21:05:27     [<ffffffff8116dabe>] SyS_fdatasync+0xe/0x20
> > 2013-08-26 21:05:27     [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b
> > 2013-08-26 21:05:27     INFO: task ceph-osd:8746 blocked for more than 120
> > seconds.
> > 2013-08-26 21:05:27     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> > disables this message.
> > 2013-08-26 21:05:27     ceph-osd D ffffffff8160f760 0 8746 1 0x00000000
> > 2013-08-26 21:05:27     ffff880c2dde1d38 0000000000000086 ffff880c2dde1fd8
> > 0000000000012c80
> > 2013-08-26 21:05:27     ffff880c2dde0010 0000000000012c80 0000000000012c80
> > 0000000000012c80
> > 2013-08-26 21:05:27     ffff880c2dde1fd8 0000000000012c80 ffff880c239eb1c0
> > ffff880c48dd6380
> > 2013-08-26 21:05:27     Call Trace:
> > 2013-08-26 21:05:27     [<ffffffff81524874>] schedule+0x24/0x70
> > 2013-08-26 21:05:27     [<ffffffff81522bbd>] schedule_timeout+0x16d/0x200
> > 2013-08-26 21:05:27     [<ffffffff81068eba>] ?
> > __queue_delayed_work+0xaa/0x1a0
> > 2013-08-26 21:05:27     [<ffffffff81068d19>] ?
> > try_to_grab_pending+0x109/0x190
> > 2013-08-26 21:05:27     [<ffffffff815251e5>] wait_for_completion+0x95/0x110
> > 2013-08-26 21:05:27     [<ffffffff8107fd00>] ? try_to_wake_up+0x2a0/0x2a0
> > 2013-08-26 21:05:27     [<ffffffff81167797>] ? bdi_queue_work+0x77/0xc0
> > 2013-08-26 21:05:27     [<ffffffff81167863>]
> > writeback_inodes_sb_nr+0x83/0xb0
> > 2013-08-26 21:05:27     [<ffffffff811678ea>] writeback_inodes_sb+0x5a/0x70
> > 2013-08-26 21:05:27     [<ffffffff8116dd0a>] __sync_filesystem+0x4a/0x50
> > 2013-08-26 21:05:27     [<ffffffff8116dd42>] sync_filesystem+0x32/0x60
> > 2013-08-26 21:05:27     [<ffffffff8116ddc0>] SyS_syncfs+0x50/0x90
> > 2013-08-26 21:05:27     [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b
> > 2013-08-26 21:05:27     INFO: task ceph-osd:8798 blocked for more than 120
> > seconds.
> >
> > Stefan
> >
> > Am 22.08.2013 09:32, schrieb Stefan Priebe - Profihost AG:
> >
> >> great!
> >>
> >> Everything seems to work fine now! Except read_dirty always going to
> >> negative values after a reboot.
> >>
> >> Stefan
> >>
> >> Am 22.08.2013 08:02, schrieb Kent Overstreet:
> >>>
> >>> On Thu, Aug 22, 2013 at 07:59:04AM +0200, Stefan Priebe wrote:
> >>>>
> >>>>
> >>>>> schedule_timeout() is not the same as
> >>>>> schedule_timeout_interruptible().
> >>>>
> >>>>
> >>>> just search and replace? So i can try on my own.
> >>>
> >>>
> >>> The one in read_dirty(), line ~330
> >>> --
> >>> To unsubscribe from this list: send the line "unsubscribe linux-bcache"
> >>> in
> >>> the body of a message to majordomo@vger.kernel.org
> >>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>>
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-28  6:05                             ` Kent Overstreet
@ 2013-08-28 18:12                               ` kernel neophyte
  2013-08-28 18:38                                 ` Stefan Priebe - Profihost AG
  0 siblings, 1 reply; 28+ messages in thread
From: kernel neophyte @ 2013-08-28 18:12 UTC (permalink / raw)
  To: Kent Overstreet; +Cc: Stefan Priebe, Jens Axboe, linux-kernel, linux-bcache

On Tue, Aug 27, 2013 at 11:05 PM, Kent Overstreet <kmo@daterainc.com> wrote:
> On Tue, Aug 27, 2013 at 08:37:54PM -0700, kernel neophyte wrote:
>> Hi Kent,
>>
>> I am still seeing deadlock:
>
> Seeing blk_throtl_bio (what asshole misspells words in their function
> names, so when you go to grep for them you grep for the wrong thing?) in
> the backtrace - is that what it takes to hit it?
>

I don't think so,
I had another deadlock:

[ 1179.193990] bio: create slab <bio-2> at 2
[ 1179.194491] bcache: run_cache_set() invalidating existing data
[ 1179.212308] bcache: register_cache() registered cache device rxd1
[ 1203.846220] bcache: register_bdev() registered backing device sda3
[ 1227.780088] bcache: bch_cached_dev_attach() Caching sda3 as bcache0
on set a079dda3-27fd-4b1c-a616-d46110bfae6c
[ 2162.497950] INFO: task kworker/0:0:4 blocked for more than 120 seconds.
[ 2162.497961] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 2162.497967] kworker/0:0     D 0000000000000000     0     4      2 0x00000000
[ 2162.497982] Workqueue: events write_dirty_finish
[ 2162.497986]  ffff882fa6ac9b30 0000000000000046 ffffffff81511b8c
ffffffff815137c0
[ 2162.497992]  ffff882fa6a84cb0 ffff882fa6ac9fd8 ffff882fa6ac9fd8
ffff882fa6ac9fd8
[ 2162.497996]  ffff882f78139990 ffff882fa6a84cb0 ffffffff815117a1
ffff882fa6a84cb0
[ 2162.498001] Call Trace:
[ 2162.498010]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
[ 2162.498015]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
[ 2162.498019]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
[ 2162.498028]  [<ffffffff816be299>] schedule+0x29/0x70
[ 2162.498034]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
[ 2162.498043]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
[ 2162.498048]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
[ 2162.498052]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
[ 2162.498057]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
[ 2162.498066]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
[ 2162.498072]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
[ 2162.498076]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
[ 2162.498082]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
[ 2162.498087]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
[ 2162.498093]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
[ 2162.498100]  [<ffffffff810624d4>] process_one_work+0x174/0x490
[ 2162.498105]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
[ 2162.498110]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
[ 2162.498115]  [<ffffffff81069f40>] kthread+0xc0/0xd0
[ 2162.498120]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2162.498127]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
[ 2162.498131]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2162.498145] INFO: task bcache_allocato:1766 blocked for more than
120 seconds.
[ 2162.498149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 2162.498154] bcache_allocato D 0000000000000001     0  1766      2 0x00000000
[ 2162.498158]  ffff8822630f3d88 0000000000000046 ffff8822630f3da8
ffffffff810808ad
[ 2162.498163]  ffff882f891de640 ffff8822630f3fd8 ffff8822630f3fd8
ffff8822630f3fd8
[ 2162.498167]  ffff882f81a91990 ffff882f891de640 ffff882f88b33320
ffff8801fad20d98
[ 2162.498172] Call Trace:
[ 2162.498177]  [<ffffffff810808ad>] ? dequeue_task_fair+0x2cd/0x530
[ 2162.498182]  [<ffffffff816be299>] schedule+0x29/0x70
[ 2162.498187]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
[ 2162.498192]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
[ 2162.498197]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
[ 2162.498202]  [<ffffffff8150cdbf>] bch_allocator_thread+0x10f/0xe20
[ 2162.498207]  [<ffffffff8150ccb0>] ? bch_bucket_add_unused+0xe0/0xe0
[ 2162.498211]  [<ffffffff81069f40>] kthread+0xc0/0xd0
[ 2162.498215]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2162.498220]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
[ 2162.498224]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2162.498228] INFO: task bcache_writebac:1771 blocked for more than
120 seconds.
[ 2162.498233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 2162.498237] bcache_writebac D 0000000000000000     0  1771      2 0x00000000
[ 2162.498241]  ffff88197d2adc58 0000000000000046 ffff88197d2add48
ffffffff811269e2
[ 2162.498245]  ffff882f891d8000 ffff88197d2adfd8 ffff88197d2adfd8
ffff88197d2adfd8
[ 2162.498250]  ffff882f81a91990 ffff882f891d8000 ffff88197d2adc68
ffff882f89550ba0
[ 2162.498254] Call Trace:
[ 2162.498264]  [<ffffffff811269e2>] ? __alloc_pages_nodemask+0x152/0x9b0
[ 2162.498269]  [<ffffffff816be299>] schedule+0x29/0x70
[ 2162.498274]  [<ffffffff816bc015>] schedule_timeout+0x1e5/0x250
[ 2162.498280]  [<ffffffff810799bf>] ? try_to_wake_up+0x24f/0x2b0
[ 2162.498285]  [<ffffffff816bcd09>] __down_common+0x8e/0xe3
[ 2162.498293]  [<ffffffff81160600>] ? alloc_pages_current+0x90/0x170
[ 2162.498298]  [<ffffffff816bcdd1>] __down+0x1d/0x1f
[ 2162.498304]  [<ffffffff8106f351>] down+0x41/0x50
[ 2162.498310]  [<ffffffff8151ba7d>] bch_writeback_thread+0x35d/0x7f0
[ 2162.498315]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
[ 2162.498321]  [<ffffffff8151b720>] ? write_dirty+0xc0/0xc0
[ 2162.498325]  [<ffffffff81069f40>] kthread+0xc0/0xd0
[ 2162.498329]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2162.498334]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
[ 2162.498338]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2162.498342] INFO: task kworker/0:2:1789 blocked for more than 120 seconds.
[ 2162.498346] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 2162.498351] kworker/0:2     D ffffffff81813a60     0  1789      2 0x00000000
[ 2162.498357] Workqueue: events write_dirty_finish
[ 2162.498360]  ffff88277062fb30 0000000000000046 ffffffff81511b8c
ffffffff815137c0
[ 2162.498364]  ffff882fa6786640 ffff88277062ffd8 ffff88277062ffd8
ffff88277062ffd8
[ 2162.498368]  ffffffff81c10440 ffff882fa6786640 ffffffff815117a1
ffff882fa6786640
[ 2162.498373] Call Trace:
[ 2162.498378]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
[ 2162.498382]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
[ 2162.498387]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
[ 2162.498392]  [<ffffffff816be299>] schedule+0x29/0x70
[ 2162.498397]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
[ 2162.498402]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
[ 2162.498407]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
[ 2162.498411]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
[ 2162.498416]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
[ 2162.498420]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
[ 2162.498425]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
[ 2162.498430]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
[ 2162.498435]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
[ 2162.498439]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
[ 2162.498444]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
[ 2162.498449]  [<ffffffff810624d4>] process_one_work+0x174/0x490
[ 2162.498454]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
[ 2162.498459]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
[ 2162.498463]  [<ffffffff81069f40>] kthread+0xc0/0xd0
[ 2162.498467]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2162.498472]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
[ 2162.498476]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2162.498480] INFO: task iozone:2052 blocked for more than 120 seconds.
[ 2162.498484] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 2162.498488] iozone          D 0000000000000001     0  2052   1944 0x00000000
[ 2162.498492]  ffff882f7e2d3978 0000000000000082 ffff882f7e2d3938
ffffffff81301d7a
[ 2162.498497]  ffff882f8a116640 ffff882f7e2d3fd8 ffff882f7e2d3fd8
ffff882f7e2d3fd8
[ 2162.498501]  ffff882f88b33320 ffff882f8a116640 ffff882f7e2d3978
ffff882fbf2339f8
[ 2162.498505] Call Trace:
[ 2162.498511]  [<ffffffff81301d7a>] ? generic_make_request+0xca/0x100
[ 2162.498517]  [<ffffffff816be299>] schedule+0x29/0x70
[ 2162.498522]  [<ffffffff816be36f>] io_schedule+0x8f/0xd0
[ 2162.498530]  [<ffffffff811b797c>] do_blockdev_direct_IO+0x1a7c/0x1fb0
[ 2162.498539]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
[ 2162.498546]  [<ffffffff811b7f05>] __blockdev_direct_IO+0x55/0x60
[ 2162.498552]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
[ 2162.498556]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
[ 2162.498562]  [<ffffffffa0274349>] ext2_direct_IO+0x79/0xe0 [ext2]
[ 2162.498567]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
[ 2162.498572]  [<ffffffff8107e775>] ? set_next_entity+0xa5/0xc0
[ 2162.498579]  [<ffffffff8104ade6>] ? current_fs_time+0x16/0x60
[ 2162.498585]  [<ffffffff8111f126>] generic_file_direct_write+0xc6/0x180
[ 2162.498590]  [<ffffffff8111f4bd>] __generic_file_aio_write+0x2dd/0x3b0
[ 2162.498595]  [<ffffffff816bc364>] ? mutex_unlock+0x14/0x20
[ 2162.498600]  [<ffffffff8111f5f9>] generic_file_aio_write+0x69/0xd0
[ 2162.498607]  [<ffffffff8117b88a>] do_sync_write+0x7a/0xb0
[ 2162.498612]  [<ffffffff8117c63e>] vfs_write+0xce/0x1e0
[ 2162.498617]  [<ffffffff8117bba0>] ? generic_file_llseek_size+0x110/0x110
[ 2162.498621]  [<ffffffff8117cb22>] SyS_write+0x52/0xa0
[ 2162.498627]  [<ffffffff816c7a02>] system_call_fastpath+0x16/0x1b
[ 2162.498630] INFO: task kworker/0:1:2293 blocked for more than 120 seconds.
[ 2162.498635] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 2162.498639] kworker/0:1     D ffffffff81813a60     0  2293      2 0x00000000
[ 2162.498645] Workqueue: events write_dirty_finish
[ 2162.498647]  ffff882f8a05bb30 0000000000000046 ffffffff81511b8c
ffffffff815137c0
[ 2162.498652]  ffff882f8a163320 ffff882f8a05bfd8 ffff882f8a05bfd8
ffff882f8a05bfd8
[ 2162.498656]  ffffffff81c10440 ffff882f8a163320 ffffffff815117a1
ffff882f8a163320
[ 2162.498661] Call Trace:
[ 2162.498665]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
[ 2162.498669]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
[ 2162.498674]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
[ 2162.498679]  [<ffffffff816be299>] schedule+0x29/0x70
[ 2162.498684]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
[ 2162.498689]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
[ 2162.498693]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
[ 2162.498698]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
[ 2162.498702]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
[ 2162.498707]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
[ 2162.498712]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
[ 2162.498716]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
[ 2162.498721]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
[ 2162.498726]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
[ 2162.498731]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
[ 2162.498735]  [<ffffffff810624d4>] process_one_work+0x174/0x490
[ 2162.498740]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
[ 2162.498745]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
[ 2162.498749]  [<ffffffff81069f40>] kthread+0xc0/0xd0
[ 2162.498753]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2162.498758]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
[ 2162.498762]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2162.498766] INFO: task kworker/0:3:2534 blocked for more than 120 seconds.
[ 2162.498772] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 2162.498777] kworker/0:3     D ffffffff81813a60     0  2534      2 0x00000000
[ 2162.498783] Workqueue: events write_dirty_finish
[ 2162.498785]  ffff88237d0bdb30 0000000000000046 ffffffff81511b8c
ffffffff815137c0
[ 2162.498790]  ffff882fa66fe640 ffff88237d0bdfd8 ffff88237d0bdfd8
ffff88237d0bdfd8
[ 2162.498794]  ffffffff81c10440 ffff882fa66fe640 ffffffff815117a1
ffff882fa66fe640
[ 2162.498799] Call Trace:
[ 2162.498803]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
[ 2162.498807]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
[ 2162.498812]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
[ 2162.498817]  [<ffffffff816be299>] schedule+0x29/0x70
[ 2162.498822]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
[ 2162.498826]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
[ 2162.498831]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
[ 2162.498836]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
[ 2162.498840]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
[ 2162.498845]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
[ 2162.498849]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
[ 2162.498854]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
[ 2162.498859]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
[ 2162.498864]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
[ 2162.498868]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
[ 2162.498873]  [<ffffffff810624d4>] process_one_work+0x174/0x490
[ 2162.498878]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
[ 2162.498882]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
[ 2162.498886]  [<ffffffff81069f40>] kthread+0xc0/0xd0
[ 2162.498890]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2162.498896]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
[ 2162.498900]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2162.498903] INFO: task kworker/0:4:2537 blocked for more than 120 seconds.
[ 2162.498908] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 2162.498914] kworker/0:4     D 0000000000000000     0  2537      2 0x00000000
[ 2162.498920] Workqueue: events write_dirty_finish
[ 2162.498922]  ffff882f76d8fb30 0000000000000046 ffffffff81511b8c
ffffffff815137c0
[ 2162.498926]  ffff882f88b30000 ffff882f76d8ffd8 ffff882f76d8ffd8
ffff882f76d8ffd8
[ 2162.498931]  ffff882f78beb320 ffff882f88b30000 ffffffff815117a1
ffff882f88b30000
[ 2162.498935] Call Trace:
[ 2162.498940]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
[ 2162.498944]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
[ 2162.498948]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
[ 2162.498953]  [<ffffffff816be299>] schedule+0x29/0x70
[ 2162.498958]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
[ 2162.498963]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
[ 2162.498968]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
[ 2162.498972]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
[ 2162.498977]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
[ 2162.498981]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
[ 2162.498986]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
[ 2162.498990]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
[ 2162.498995]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
[ 2162.499000]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
[ 2162.499004]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
[ 2162.499009]  [<ffffffff810624d4>] process_one_work+0x174/0x490
[ 2162.499014]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
[ 2162.499019]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
[ 2162.499023]  [<ffffffff81069f40>] kthread+0xc0/0xd0
[ 2162.499027]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2162.499032]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
[ 2162.499036]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2162.499039] INFO: task kworker/1:1:2544 blocked for more than 120 seconds.
[ 2162.499044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 2162.499050] kworker/1:1     D ffffffff81813d40     0  2544      2 0x00000000
[ 2162.499057] Workqueue: bcache bch_data_insert_keys
[ 2162.499059]  ffff882fa677b3f8 0000000000000046 ffff882fa677b418
ffff882f8a709000
[ 2162.499063]  ffff882f88b33320 ffff882fa677bfd8 ffff882fa677bfd8
ffff882fa677bfd8
[ 2162.499068]  ffff882fa6471990 ffff882f88b33320 ffff882f88b33320
ffff8801fad20d98
[ 2162.499072] Call Trace:
[ 2162.499077]  [<ffffffff816be299>] schedule+0x29/0x70
[ 2162.499082]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
[ 2162.499087]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
[ 2162.499092]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
[ 2162.499096]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
[ 2162.499102]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
[ 2162.499109]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
[ 2162.499113]  [<ffffffff81076828>] ? resched_task+0x68/0x70
[ 2162.499118]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
[ 2162.499126]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
[ 2162.499131]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
[ 2162.499137]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
[ 2162.499143]  [<ffffffff81126e9c>] __alloc_pages_nodemask+0x60c/0x9b0
[ 2162.499149]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
[ 2162.499155]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
[ 2162.499159]  [<ffffffff8150ebb3>] mca_data_alloc+0x73/0x1d0
[ 2162.499163]  [<ffffffff81510f37>] mca_alloc+0x277/0x470
[ 2162.499168]  [<ffffffff81511d1c>] bch_btree_node_alloc+0x8c/0x1c0
[ 2162.499173]  [<ffffffff81513020>] btree_split+0x110/0x5c0
[ 2162.499179]  [<ffffffff81457d43>] ? scsi_pool_alloc_command+0x33/0x80
[ 2162.499184]  [<ffffffff81515fc7>] ? bch_keylist_pop_front+0x47/0x50
[ 2162.499188]  [<ffffffff8150fed6>] ? bch_btree_insert_keys+0x56/0x250
[ 2162.499193]  [<ffffffff81082def>] ? update_group_power+0x13f/0x220
[ 2162.499198]  [<ffffffff81513582>] bch_btree_insert_node+0xb2/0x2f0
[ 2162.499202]  [<ffffffff815137e8>] btree_insert_fn+0x28/0x50
[ 2162.499207]  [<ffffffff81511b8c>] bch_btree_map_nodes_recurse+0x6c/0x170
[ 2162.499211]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
[ 2162.499216]  [<ffffffff816bcc26>] ? down_write+0x16/0x40
[ 2162.499221]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
[ 2162.499225]  [<ffffffff81511c30>] bch_btree_map_nodes_recurse+0x110/0x170
[ 2162.499230]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
[ 2162.499235]  [<ffffffff811b51ea>] ? dio_bio_end_io+0x5a/0x90
[ 2162.499239]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
[ 2162.499244]  [<ffffffff81514dce>] __bch_btree_map_nodes+0x13e/0x1c0
[ 2162.499248]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
[ 2162.499254]  [<ffffffff8151ac6f>] ? bch_journal+0x42f/0x4b0
[ 2162.499258]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
[ 2162.499264]  [<ffffffff8151e1be>] bch_data_insert_keys+0x3e/0x160
[ 2162.499268]  [<ffffffff810624d4>] process_one_work+0x174/0x490
[ 2162.499273]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
[ 2162.499278]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
[ 2162.499282]  [<ffffffff81069f40>] kthread+0xc0/0xd0
[ 2162.499286]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2162.499292]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
[ 2162.499296]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2162.499299] INFO: task kworker/0:5:2552 blocked for more than 120 seconds.
[ 2162.499304] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 2162.499310] kworker/0:5     D 0000000000000000     0  2552      2 0x00000000
[ 2162.499411] Workqueue: events write_dirty_finish
[ 2162.499413]  ffff88226314db30 0000000000000046 ffffffff81511b8c
ffffffff815137c0
[ 2162.499418]  ffff882fa6781990 ffff88226314dfd8 ffff88226314dfd8
ffff88226314dfd8
[ 2162.499422]  ffff882fa6a84cb0 ffff882fa6781990 ffffffff815117a1
ffff882fa6781990
[ 2162.499427] Call Trace:
[ 2162.499431]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
[ 2162.499435]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
[ 2162.499440]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
[ 2162.499445]  [<ffffffff816be299>] schedule+0x29/0x70
[ 2162.499450]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
[ 2162.499455]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
[ 2162.499460]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
[ 2162.499464]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
[ 2162.499468]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
[ 2162.499473]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
[ 2162.499478]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
[ 2162.499483]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
[ 2162.499488]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
[ 2162.499492]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
[ 2162.499497]  [<ffffffff810624d4>] process_one_work+0x174/0x490
[ 2162.499502]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
[ 2162.499507]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
[ 2162.499511]  [<ffffffff81069f40>] kthread+0xc0/0xd0
[ 2162.499515]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
[ 2162.499520]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
[ 2162.499524]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0



>> [  930.169234] bcache: bch_cached_dev_attach() Caching sda3 as bcache0
>> on set 06496904-10ca-489a-ae75-68c6a07d3db1
>> [ 2522.956188] INFO: task bcache_writebac:2058 blocked for more than
>> 120 seconds.
>> [ 2522.956199] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [ 2522.956205] bcache_writebac D ffffffff81813a60     0  2058      2 0x00000000
>> [ 2522.956212]  ffff881c39c1fcd8 0000000000000046 0000000000000001
>> 0000000000000001
>> [ 2522.956218]  ffff882f8b56b320 ffff881c39c1ffd8 ffff881c39c1ffd8
>> ffff881c39c1ffd8
>> [ 2522.956222]  ffff882fa6aeb320 ffff882f8b56b320 ffff882f8b56b320
>> ffff882f8b56b320
>> [ 2522.956227] Call Trace:
>> [ 2522.956241]  [<ffffffff816be299>] schedule+0x29/0x70
>> [ 2522.956247]  [<ffffffff816befb5>] rwsem_down_write_failed+0xf5/0x1a0
>> [ 2522.956255]  [<ffffffff81332c93>] call_rwsem_down_write_failed+0x13/0x20
>> [ 2522.956261]  [<ffffffff816bcc41>] ? down_write+0x31/0x40
>> [ 2522.956269]  [<ffffffff8151b782>] bch_writeback_thread+0x62/0x7f0
>> [ 2522.956278]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
>> [ 2522.956284]  [<ffffffff8151b720>] ? write_dirty+0xc0/0xc0
>> [ 2522.956290]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>> [ 2522.956294]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2522.956302]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>> [ 2522.956306]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2522.956313] INFO: task kworker/u64:1:2433 blocked for more than 120 seconds.
>> [ 2522.956317] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [ 2522.956322] kworker/u64:1   D ffffffff81813a60     0  2433      2 0x00000000
>> [ 2522.956332] Workqueue: writeback bdi_writeback_workfn (flush-252:0)
>> [ 2522.956336]  ffff882f899d1560 0000000000000046 ffffffff81120635
>> 0000000000000100
>> [ 2522.956340]  ffff88219b488000 ffff882f899d1fd8 ffff882f899d1fd8
>> ffff882f899d1fd8
>> [ 2522.956345]  ffff882fa6aeb320 ffff88219b488000 ffffffff8131ad55
>> ffff88219b488000
>> [ 2522.956349] Call Trace:
>> [ 2522.956359]  [<ffffffff81120635>] ? mempool_alloc_slab+0x15/0x20
>> [ 2522.956367]  [<ffffffff8131ad55>] ? blk_throtl_bio+0x3b5/0x540
>> [ 2522.956372]  [<ffffffff816be299>] schedule+0x29/0x70
>> [ 2522.956378]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>> [ 2522.956383]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
>> [ 2522.956388]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>> [ 2522.956394]  [<ffffffff8151fe32>] cached_dev_make_request+0x752/0xf00
>> [ 2522.956400]  [<ffffffff81301af8>] ? generic_make_request_checks+0x1e8/0x3a0
>> [ 2522.956405]  [<ffffffff81301d7a>] generic_make_request+0xca/0x100
>> [ 2522.956409]  [<ffffffff81301e29>] submit_bio+0x79/0x160
>> [ 2522.956417]  [<ffffffff811b1710>] ? bio_alloc_bioset+0xa0/0x1d0
>> [ 2522.956421]  [<ffffffff811abd3f>] _submit_bh+0x13f/0x200
>> [ 2522.956425]  [<ffffffff811abe10>] submit_bh+0x10/0x20
>> [ 2522.956430]  [<ffffffff811af6b8>] __block_write_full_page+0x1d8/0x360
>> [ 2522.956439]  [<ffffffff8145fc0b>] ? scsi_request_fn+0xbb/0x530
>> [ 2522.956444]  [<ffffffff811ad3c0>] ? end_buffer_async_read+0x130/0x130
>> [ 2522.956449]  [<ffffffff811b2c40>] ? I_BDEV+0x10/0x10
>> [ 2522.956454]  [<ffffffff811b2c40>] ? I_BDEV+0x10/0x10
>> [ 2522.956459]  [<ffffffff811af90a>] block_write_full_page_endio+0xca/0x100
>> [ 2522.956464]  [<ffffffff811af955>] block_write_full_page+0x15/0x20
>> [ 2522.956469]  [<ffffffff811b3608>] blkdev_writepage+0x18/0x20
>> [ 2522.956473]  [<ffffffff81127d37>] __writepage+0x17/0x40
>> [ 2522.956477]  [<ffffffff8112820e>] write_cache_pages+0x20e/0x460
>> [ 2522.956481]  [<ffffffff81127d20>] ? set_page_dirty_lock+0x60/0x60
>> [ 2522.956486]  [<ffffffff811ac251>] ? __set_page_dirty+0x71/0xc0
>> [ 2522.956490]  [<ffffffff811284aa>] generic_writepages+0x4a/0x70
>> [ 2522.956495]  [<ffffffff81129cd0>] do_writepages+0x20/0x40
>> [ 2522.956501]  [<ffffffff811a36f5>] __writeback_single_inode+0x45/0x280
>> [ 2522.956507]  [<ffffffff811269e2>] ? __alloc_pages_nodemask+0x152/0x9b0
>> [ 2522.956513]  [<ffffffff811a495d>] writeback_sb_inodes+0x19d/0x3c0
>> [ 2522.956518]  [<ffffffff811a4c1e>] __writeback_inodes_wb+0x9e/0xd0
>> [ 2522.956523]  [<ffffffff811a4ecb>] wb_writeback+0x27b/0x320
>> [ 2522.956529]  [<ffffffff81196228>] ? get_nr_dirty_inodes+0x58/0x80
>> [ 2522.956534]  [<ffffffff811a500f>] wb_check_old_data_flush+0x9f/0xb0
>> [ 2522.956539]  [<ffffffff811a67e1>] wb_do_writeback+0x151/0x1d0
>> [ 2522.956547]  [<ffffffff8106539f>] ? set_worker_desc+0x6f/0x80
>> [ 2522.956552]  [<ffffffff811a68da>] bdi_writeback_workfn+0x7a/0x200
>> [ 2522.956557]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>> [ 2522.956562]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>> [ 2522.956567]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>> [ 2522.956571]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>> [ 2522.956575]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2522.956581]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>> [ 2522.956585]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2522.956588] INFO: task iozone:2435 blocked for more than 120 seconds.
>> [ 2522.956593] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [ 2522.956597] iozone          D 0000000000000001     0  2435   1545 0x00000000
>> [ 2522.956602]  ffff882a75671978 0000000000000082 ffff882a75671938
>> ffffffff81301d7a
>> [ 2522.956606]  ffff882f88dc1990 ffff882a75671fd8 ffff882a75671fd8
>> ffff882a75671fd8
>> [ 2522.956611]  ffff882f8ec91990 ffff882f88dc1990 ffff882a75671978
>> ffff882fbf2539f8
>> [ 2522.956615] Call Trace:
>> [ 2522.956619]  [<ffffffff81301d7a>] ? generic_make_request+0xca/0x100
>> [ 2522.956625]  [<ffffffff816be299>] schedule+0x29/0x70
>> [ 2522.956630]  [<ffffffff816be36f>] io_schedule+0x8f/0xd0
>> [ 2522.956636]  [<ffffffff811b797c>] do_blockdev_direct_IO+0x1a7c/0x1fb0
>> [ 2522.956645]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
>> [ 2522.956652]  [<ffffffff811b7f05>] __blockdev_direct_IO+0x55/0x60
>> [ 2522.956658]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
>> [ 2522.956662]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>> [ 2522.956669]  [<ffffffffa0263349>] ext2_direct_IO+0x79/0xe0 [ext2]
>> [ 2522.956674]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
>> [ 2522.956680]  [<ffffffff8104ade6>] ? current_fs_time+0x16/0x60
>> [ 2522.956685]  [<ffffffff8111f126>] generic_file_direct_write+0xc6/0x180
>> [ 2522.956690]  [<ffffffff8111f4bd>] __generic_file_aio_write+0x2dd/0x3b0
>> [ 2522.956696]  [<ffffffff8111f5f9>] generic_file_aio_write+0x69/0xd0
>> [ 2522.956702]  [<ffffffff8117b88a>] do_sync_write+0x7a/0xb0
>> [ 2522.956706]  [<ffffffff811bb108>] ? fsnotify+0x1f8/0x2b0
>> [ 2522.956712]  [<ffffffff8117c63e>] vfs_write+0xce/0x1e0
>> [ 2522.956716]  [<ffffffff8117cb22>] SyS_write+0x52/0xa0
>> [ 2522.956722]  [<ffffffff816c7a02>] system_call_fastpath+0x16/0x1b
>> [ 2522.956726] INFO: task kworker/2:1:2470 blocked for more than 120 seconds.
>> [ 2522.956730] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [ 2522.956735] kworker/2:1     D ffffffff81813d40     0  2470      2 0x00000000
>> [ 2522.956743] Workqueue: bcache bch_data_insert_keys
>> [ 2522.956745]  ffff882a5b0c9628 0000000000000046 ffff882a5b0c9648
>> 0000004000000000
>> [ 2522.956749]  ffff882f8ec91990 ffff882a5b0c9fd8 ffff882a5b0c9fd8
>> ffff882a5b0c9fd8
>> [ 2522.956754]  ffff882fa6489990 ffff882f8ec91990 ffff882f8ec91990
>> ffff8828dba00d98
>> [ 2522.956758] Call Trace:
>> [ 2522.956763]  [<ffffffff816be299>] schedule+0x29/0x70
>> [ 2522.956769]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
>> [ 2522.956774]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
>> [ 2522.956779]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>> [ 2522.956783]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
>> [ 2522.956791]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
>> [ 2522.956799]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
>> [ 2522.956805]  [<ffffffff81076828>] ? resched_task+0x68/0x70
>> [ 2522.956810]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
>> [ 2522.956816]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
>> [ 2522.956822]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
>> [ 2522.956827]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
>> [ 2522.956833]  [<ffffffff81126e9c>] __alloc_pages_nodemask+0x60c/0x9b0
>> [ 2522.956843]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
>> [ 2522.956848]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
>> [ 2522.956852]  [<ffffffff8150ebb3>] mca_data_alloc+0x73/0x1d0
>> [ 2522.956857]  [<ffffffff8150ee5a>] mca_bucket_alloc+0x14a/0x1f0
>> [ 2522.956861]  [<ffffffff81511020>] mca_alloc+0x360/0x470
>> [ 2522.956866]  [<ffffffff81517031>] ? __bch_bset_search+0x1d1/0x480
>> [ 2522.956871]  [<ffffffff8151183e>] bch_btree_node_get+0x10e/0x280
>> [ 2522.956875]  [<ffffffff81511c02>] bch_btree_map_nodes_recurse+0xe2/0x170
>> [ 2522.956880]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>> [ 2522.956886]  [<ffffffff811b51ea>] ? dio_bio_end_io+0x5a/0x90
>> [ 2522.956890]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>> [ 2522.956895]  [<ffffffff81514dce>] __bch_btree_map_nodes+0x13e/0x1c0
>> [ 2522.956899]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>> [ 2522.956905]  [<ffffffff8151ac6f>] ? bch_journal+0x42f/0x4b0
>> [ 2522.956909]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>> [ 2522.956915]  [<ffffffff8151e1be>] bch_data_insert_keys+0x3e/0x160
>> [ 2522.956920]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>> [ 2522.956925]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>> [ 2522.956930]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>> [ 2522.956933]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>> [ 2522.956938]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2522.956943]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>> [ 2522.956947]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2522.956951] INFO: task kworker/3:2:2471 blocked for more than 120 seconds.
>> [ 2522.956955] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [ 2522.956959] kworker/3:2     D ffffffff81813a60     0  2471      2 0x00000000
>> [ 2522.956966] Workqueue: events update_writeback_rate
>> [ 2522.956968]  ffff882a5d125cd0 0000000000000046 0000000000000003
>> ffff882a5d125d68
>> [ 2522.956972]  ffff882f8ec94cb0 ffff882a5d125fd8 ffff882a5d125fd8
>> ffff882a5d125fd8
>> [ 2522.956977]  ffff882fa6aeb320 ffff882f8ec94cb0 0000000200000003
>> ffff882f8ec94cb0
>> [ 2522.956981] Call Trace:
>> [ 2522.956987]  [<ffffffff816be299>] schedule+0x29/0x70
>> [ 2522.956992]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>> [ 2522.956997]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
>> [ 2522.957002]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>> [ 2522.957007]  [<ffffffff8151b010>] update_writeback_rate+0x30/0x230
>> [ 2522.957011]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>> [ 2522.957017]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>> [ 2522.957021]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>> [ 2522.957025]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>> [ 2522.957030]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2522.957035]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>> [ 2522.957039]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2522.957048] INFO: task kworker/u64:2:2540 blocked for more than 120 seconds.
>> [ 2522.957052] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [ 2522.957056] kworker/u64:2   D ffffffff81813a60     0  2540      2 0x00000000
>> [ 2522.957062] Workqueue: bch_btree_io btree_node_write_work
>> [ 2522.957064]  ffff882a4fdc9778 0000000000000046 ffff882a4fdc9738
>> ffff882f86e3be58
>> [ 2522.957069]  ffff882fa4bc1990 ffff882a4fdc9fd8 ffff882a4fdc9fd8
>> ffff882a4fdc9fd8
>> [ 2522.957073]  ffff882fa6aeb320 ffff882fa4bc1990 ffff882f86e3be00
>> ffff8828dba00d98
>> [ 2522.957078] Call Trace:
>> [ 2522.957083]  [<ffffffff816be299>] schedule+0x29/0x70
>> [ 2522.957088]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
>> [ 2522.957092]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
>> [ 2522.957102]  [<ffffffff81484090>] ? ata_scsiop_mode_sense+0x380/0x380
>> [ 2522.957107]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>> [ 2522.957111]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
>> [ 2522.957116]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
>> [ 2522.957121]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
>> [ 2522.957126]  [<ffffffff81076828>] ? resched_task+0x68/0x70
>> [ 2522.957130]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
>> [ 2522.957135]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
>> [ 2522.957140]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
>> [ 2522.957146]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
>> [ 2522.957152]  [<ffffffff81126e9c>] __alloc_pages_nodemask+0x60c/0x9b0
>> [ 2522.957158]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
>> [ 2522.957163]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
>> [ 2522.957168]  [<ffffffff81517fc8>] __btree_sort+0x48/0x230
>> [ 2522.957173]  [<ffffffff8151765c>] ? __bch_btree_iter_init+0x7c/0xc0
>> [ 2522.957178]  [<ffffffff81518301>] bch_btree_sort_partial+0x101/0x120
>> [ 2522.957182]  [<ffffffff8150f000>] ? __btree_node_write_done+0x100/0x100
>> [ 2522.957187]  [<ffffffff81518468>] bch_btree_sort_lazy+0x68/0x90
>> [ 2522.957191]  [<ffffffff815109ba>] bch_btree_node_write+0x36a/0x4a0
>> [ 2522.957196]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
>> [ 2522.957201]  [<ffffffff8106212c>] ? pwq_activate_delayed_work+0x4c/0xb0
>> [ 2522.957205]  [<ffffffff81511477>] btree_node_write_work+0x57/0x80
>> [ 2522.957210]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>> [ 2522.957215]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>> [ 2522.957219]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>> [ 2522.957223]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>> [ 2522.957228]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2522.957233]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>> [ 2522.957237]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>
>>
>> I can easily reproduce this, please let me know if you require any
>> info/test any patch.
>>
>> Thanks,
>> Neo
>>
>> On Mon, Aug 26, 2013 at 12:21 PM, Stefan Priebe <s.priebe@profihost.ag> wrote:
>> > Hi Kent,
>> >
>> > a new one under 3.10:
>> >
>> > 2013-08-26 21:05:30     INFO: task ceph-osd:8939 blocked for more than 120
>> > seconds.
>> > 2013-08-26 21:05:30     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> > disables this message.
>> > 2013-08-26 21:05:30     ceph-osd D ffffffff8160f760 0 8939 1 0x00000000
>> > 2013-08-26 21:05:30     ffff880bd9a61820 0000000000000086 ffff880bd9a61fd8
>> > 0000000000012c80
>> > 2013-08-26 21:05:30     ffff880bd9a60010 0000000000012c80 0000000000012c80
>> > 0000000000012c80
>> > 2013-08-26 21:05:30     ffff880bd9a61fd8 0000000000012c80 ffff880bd7abb1c0
>> > ffff880c48dee380
>> > 2013-08-26 21:05:29     [<ffffffff811e8128>] xfs_vm_readpages+0x18/0x20
>> > 2013-08-26 21:05:29     [<ffffffff810f1da3>] read_pages+0x43/0x100
>> > 2013-08-26 21:05:29     [<ffffffff810e64df>] ? __page_cache_alloc+0x9f/0xc0
>> > 2013-08-26 21:05:29     [<ffffffff810f1fab>]
>> > __do_page_cache_readahead+0x14b/0x160
>> > 2013-08-26 21:05:29     [<ffffffff810f1fdc>] ra_submit+0x1c/0x20
>> > 2013-08-26 21:05:29     [<ffffffff810f22a5>] ondemand_readahead+0x115/0x240
>> > 2013-08-26 21:05:29     [<ffffffff810f249e>]
>> > page_cache_sync_readahead+0x2e/0x40
>> > 2013-08-26 21:05:29     [<ffffffff810e7a5e>] T.1003+0x33e/0x430
>> > 2013-08-26 21:05:29     [<ffffffff810e7c30>]
>> > generic_file_aio_read+0xe0/0x220
>> > 2013-08-26 21:05:29     [<ffffffff811efd0a>] xfs_file_aio_read+0x15a/0x2a0
>> > 2013-08-26 21:05:29     [<ffffffff8114095a>] do_sync_read+0x7a/0xb0
>> > 2013-08-26 21:05:29     [<ffffffff81140df1>] vfs_read+0xb1/0x130
>> > 2013-08-26 21:05:29     [<ffffffff81141757>] SyS_pread64+0x97/0xa0
>> > 2013-08-26 21:05:29     [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b
>> > 2013-08-26 21:05:29     INFO: task ceph-osd:8896 blocked for more than 120
>> > seconds.
>> > 2013-08-26 21:05:29     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> > disables this message.
>> > 2013-08-26 21:05:29     ceph-osd D ffffffff8160f760 0 8896 1 0x00000000
>> > 2013-08-26 21:05:29     ffff880bd9a9dc28 0000000000000086 ffff880bd9a9dfd8
>> > 0000000000012c80
>> > 2013-08-26 21:05:29     ffff880bd9a9c010 0000000000012c80 0000000000012c80
>> > 0000000000012c80
>> > 2013-08-26 21:05:29     ffff880bd9a9dfd8 0000000000012c80 ffff880bd79298e0
>> > ffff880c48dd6380
>> > 2013-08-26 21:05:29     Call Trace:
>> > 2013-08-26 21:05:29     [<ffffffff810e60f0>] ? __lock_page+0x70/0x70
>> > 2013-08-26 21:05:28     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> > disables this message.
>> > 2013-08-26 21:05:28     ceph-osd D ffffffff8160f760 0 8798 1 0x00000000
>> > 2013-08-26 21:05:28     ffff880bd998b740 0000000000000086 ffff880bd998bfd8
>> > 0000000000012c80
>> > 2013-08-26 21:05:28     ffff880bd998a010 0000000000012c80 0000000000012c80
>> > 0000000000012c80
>> > 2013-08-26 21:05:28     ffff880bd998bfd8 0000000000012c80 ffff880bd99c0000
>> > ffff880c48dd6380
>> > 2013-08-26 21:05:28     Call Trace:
>> > 2013-08-26 21:05:28     [<ffffffff81524874>] schedule+0x24/0x70
>> > 2013-08-26 21:05:28     [<ffffffff81525d7d>]
>> > rwsem_down_read_failed+0x9d/0xe5
>> > 2013-08-26 21:05:28     [<ffffffff812aa0a4>]
>> > call_rwsem_down_read_failed+0x14/0x30
>> > 2013-08-26 21:05:28     [<ffffffff81523a42>] ? down_read+0x12/0x20
>> > 2013-08-26 21:05:28     [<ffffffffa01ead62>] btree_read_async+0xa2/0x1a0
>> > [bcache]
>> > 2013-08-26 21:05:28     [<ffffffffa01f7b73>] closure_queue+0x43/0x60
>> > [bcache]
>> > 2013-08-26 21:05:28     [<ffffffffa01eb903>] T.1050+0x63/0x70 [bcache]
>> > 2013-08-26 21:05:28     [<ffffffffa01ed6ea>]
>> > cached_dev_make_request+0x20a/0x350 [bcache]
>> > 2013-08-26 21:05:28     [<ffffffff81281952>] generic_make_request+0xc2/0x100
>> > 2013-08-26 21:05:28     [<ffffffff812819f7>] submit_bio+0x67/0x130
>> > 2013-08-26 21:05:28     [<ffffffff8117b240>] do_mpage_readpage+0x2a0/0x660
>> > 2013-08-26 21:05:28     [<ffffffff81101cee>] ?
>> > __inc_zone_page_state+0x2e/0x30
>> > 2013-08-26 21:05:28     [<ffffffff810e6581>] ?
>> > add_to_page_cache_locked+0x81/0x110
>> > 2013-08-26 21:05:28     [<ffffffff8117b76a>] mpage_readpages+0xfa/0x150
>> > 2013-08-26 21:05:28     [<ffffffff811e90c0>] ?
>> > xfs_get_blocks_direct+0x20/0x20
>> > 2013-08-26 21:05:28     [<ffffffff811e90c0>] ?
>> > xfs_get_blocks_direct+0x20/0x20
>> > 2013-08-26 21:05:28     [<ffffffff810eeda3>] ?
>> > __alloc_pages_nodemask+0x123/0x240
>> > 2013-08-26 21:05:27     [<ffffffff8115bf93>] ? __close_fd+0x43/0x80
>> > 2013-08-26 21:05:27     [<ffffffff8116dabe>] SyS_fdatasync+0xe/0x20
>> > 2013-08-26 21:05:27     [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b
>> > 2013-08-26 21:05:27     INFO: task ceph-osd:8746 blocked for more than 120
>> > seconds.
>> > 2013-08-26 21:05:27     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> > disables this message.
>> > 2013-08-26 21:05:27     ceph-osd D ffffffff8160f760 0 8746 1 0x00000000
>> > 2013-08-26 21:05:27     ffff880c2dde1d38 0000000000000086 ffff880c2dde1fd8
>> > 0000000000012c80
>> > 2013-08-26 21:05:27     ffff880c2dde0010 0000000000012c80 0000000000012c80
>> > 0000000000012c80
>> > 2013-08-26 21:05:27     ffff880c2dde1fd8 0000000000012c80 ffff880c239eb1c0
>> > ffff880c48dd6380
>> > 2013-08-26 21:05:27     Call Trace:
>> > 2013-08-26 21:05:27     [<ffffffff81524874>] schedule+0x24/0x70
>> > 2013-08-26 21:05:27     [<ffffffff81522bbd>] schedule_timeout+0x16d/0x200
>> > 2013-08-26 21:05:27     [<ffffffff81068eba>] ?
>> > __queue_delayed_work+0xaa/0x1a0
>> > 2013-08-26 21:05:27     [<ffffffff81068d19>] ?
>> > try_to_grab_pending+0x109/0x190
>> > 2013-08-26 21:05:27     [<ffffffff815251e5>] wait_for_completion+0x95/0x110
>> > 2013-08-26 21:05:27     [<ffffffff8107fd00>] ? try_to_wake_up+0x2a0/0x2a0
>> > 2013-08-26 21:05:27     [<ffffffff81167797>] ? bdi_queue_work+0x77/0xc0
>> > 2013-08-26 21:05:27     [<ffffffff81167863>]
>> > writeback_inodes_sb_nr+0x83/0xb0
>> > 2013-08-26 21:05:27     [<ffffffff811678ea>] writeback_inodes_sb+0x5a/0x70
>> > 2013-08-26 21:05:27     [<ffffffff8116dd0a>] __sync_filesystem+0x4a/0x50
>> > 2013-08-26 21:05:27     [<ffffffff8116dd42>] sync_filesystem+0x32/0x60
>> > 2013-08-26 21:05:27     [<ffffffff8116ddc0>] SyS_syncfs+0x50/0x90
>> > 2013-08-26 21:05:27     [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b
>> > 2013-08-26 21:05:27     INFO: task ceph-osd:8798 blocked for more than 120
>> > seconds.
>> >
>> > Stefan
>> >
>> > Am 22.08.2013 09:32, schrieb Stefan Priebe - Profihost AG:
>> >
>> >> great!
>> >>
>> >> Everything seems to work fine now! Except read_dirty always going to
>> >> negative values after a reboot.
>> >>
>> >> Stefan
>> >>
>> >> Am 22.08.2013 08:02, schrieb Kent Overstreet:
>> >>>
>> >>> On Thu, Aug 22, 2013 at 07:59:04AM +0200, Stefan Priebe wrote:
>> >>>>
>> >>>>
>> >>>>> schedule_timeout() is not the same as
>> >>>>> schedule_timeout_interruptible().
>> >>>>
>> >>>>
>> >>>> just search and replace? So i can try on my own.
>> >>>
>> >>>
>> >>> The one in read_dirty(), line ~330
>> >>> --
>> >>> To unsubscribe from this list: send the line "unsubscribe linux-bcache"
>> >>> in
>> >>> the body of a message to majordomo@vger.kernel.org
>> >>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >>>
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-28 18:12                               ` kernel neophyte
@ 2013-08-28 18:38                                 ` Stefan Priebe - Profihost AG
  2013-08-28 18:47                                   ` kernel neophyte
  0 siblings, 1 reply; 28+ messages in thread
From: Stefan Priebe - Profihost AG @ 2013-08-28 18:38 UTC (permalink / raw)
  To: kernel neophyte
  Cc: Kent Overstreet, Jens Axboe, linux-kernel@vger.kernel.org,
	linux-bcache@vger.kernel.org

I don't had one for a few days. Which kernel so you use?

Stefan

This mail was sent with my iPhone.

Am 28.08.2013 um 20:12 schrieb kernel neophyte <neophyte.hacker001@gmail.com>:

> On Tue, Aug 27, 2013 at 11:05 PM, Kent Overstreet <kmo@daterainc.com> wrote:
>> On Tue, Aug 27, 2013 at 08:37:54PM -0700, kernel neophyte wrote:
>>> Hi Kent,
>>> 
>>> I am still seeing deadlock:
>> 
>> Seeing blk_throtl_bio (what asshole misspells words in their function
>> names, so when you go to grep for them you grep for the wrong thing?) in
>> the backtrace - is that what it takes to hit it?
> 
> I don't think so,
> I had another deadlock:
> 
> [ 1179.193990] bio: create slab <bio-2> at 2
> [ 1179.194491] bcache: run_cache_set() invalidating existing data
> [ 1179.212308] bcache: register_cache() registered cache device rxd1
> [ 1203.846220] bcache: register_bdev() registered backing device sda3
> [ 1227.780088] bcache: bch_cached_dev_attach() Caching sda3 as bcache0
> on set a079dda3-27fd-4b1c-a616-d46110bfae6c
> [ 2162.497950] INFO: task kworker/0:0:4 blocked for more than 120 seconds.
> [ 2162.497961] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 2162.497967] kworker/0:0     D 0000000000000000     0     4      2 0x00000000
> [ 2162.497982] Workqueue: events write_dirty_finish
> [ 2162.497986]  ffff882fa6ac9b30 0000000000000046 ffffffff81511b8c
> ffffffff815137c0
> [ 2162.497992]  ffff882fa6a84cb0 ffff882fa6ac9fd8 ffff882fa6ac9fd8
> ffff882fa6ac9fd8
> [ 2162.497996]  ffff882f78139990 ffff882fa6a84cb0 ffffffff815117a1
> ffff882fa6a84cb0
> [ 2162.498001] Call Trace:
> [ 2162.498010]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
> [ 2162.498015]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
> [ 2162.498019]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
> [ 2162.498028]  [<ffffffff816be299>] schedule+0x29/0x70
> [ 2162.498034]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
> [ 2162.498043]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
> [ 2162.498048]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
> [ 2162.498052]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
> [ 2162.498057]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
> [ 2162.498066]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
> [ 2162.498072]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
> [ 2162.498076]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
> [ 2162.498082]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
> [ 2162.498087]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
> [ 2162.498093]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
> [ 2162.498100]  [<ffffffff810624d4>] process_one_work+0x174/0x490
> [ 2162.498105]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
> [ 2162.498110]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
> [ 2162.498115]  [<ffffffff81069f40>] kthread+0xc0/0xd0
> [ 2162.498120]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2162.498127]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
> [ 2162.498131]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2162.498145] INFO: task bcache_allocato:1766 blocked for more than
> 120 seconds.
> [ 2162.498149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 2162.498154] bcache_allocato D 0000000000000001     0  1766      2 0x00000000
> [ 2162.498158]  ffff8822630f3d88 0000000000000046 ffff8822630f3da8
> ffffffff810808ad
> [ 2162.498163]  ffff882f891de640 ffff8822630f3fd8 ffff8822630f3fd8
> ffff8822630f3fd8
> [ 2162.498167]  ffff882f81a91990 ffff882f891de640 ffff882f88b33320
> ffff8801fad20d98
> [ 2162.498172] Call Trace:
> [ 2162.498177]  [<ffffffff810808ad>] ? dequeue_task_fair+0x2cd/0x530
> [ 2162.498182]  [<ffffffff816be299>] schedule+0x29/0x70
> [ 2162.498187]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
> [ 2162.498192]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
> [ 2162.498197]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
> [ 2162.498202]  [<ffffffff8150cdbf>] bch_allocator_thread+0x10f/0xe20
> [ 2162.498207]  [<ffffffff8150ccb0>] ? bch_bucket_add_unused+0xe0/0xe0
> [ 2162.498211]  [<ffffffff81069f40>] kthread+0xc0/0xd0
> [ 2162.498215]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2162.498220]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
> [ 2162.498224]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2162.498228] INFO: task bcache_writebac:1771 blocked for more than
> 120 seconds.
> [ 2162.498233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 2162.498237] bcache_writebac D 0000000000000000     0  1771      2 0x00000000
> [ 2162.498241]  ffff88197d2adc58 0000000000000046 ffff88197d2add48
> ffffffff811269e2
> [ 2162.498245]  ffff882f891d8000 ffff88197d2adfd8 ffff88197d2adfd8
> ffff88197d2adfd8
> [ 2162.498250]  ffff882f81a91990 ffff882f891d8000 ffff88197d2adc68
> ffff882f89550ba0
> [ 2162.498254] Call Trace:
> [ 2162.498264]  [<ffffffff811269e2>] ? __alloc_pages_nodemask+0x152/0x9b0
> [ 2162.498269]  [<ffffffff816be299>] schedule+0x29/0x70
> [ 2162.498274]  [<ffffffff816bc015>] schedule_timeout+0x1e5/0x250
> [ 2162.498280]  [<ffffffff810799bf>] ? try_to_wake_up+0x24f/0x2b0
> [ 2162.498285]  [<ffffffff816bcd09>] __down_common+0x8e/0xe3
> [ 2162.498293]  [<ffffffff81160600>] ? alloc_pages_current+0x90/0x170
> [ 2162.498298]  [<ffffffff816bcdd1>] __down+0x1d/0x1f
> [ 2162.498304]  [<ffffffff8106f351>] down+0x41/0x50
> [ 2162.498310]  [<ffffffff8151ba7d>] bch_writeback_thread+0x35d/0x7f0
> [ 2162.498315]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
> [ 2162.498321]  [<ffffffff8151b720>] ? write_dirty+0xc0/0xc0
> [ 2162.498325]  [<ffffffff81069f40>] kthread+0xc0/0xd0
> [ 2162.498329]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2162.498334]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
> [ 2162.498338]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2162.498342] INFO: task kworker/0:2:1789 blocked for more than 120 seconds.
> [ 2162.498346] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 2162.498351] kworker/0:2     D ffffffff81813a60     0  1789      2 0x00000000
> [ 2162.498357] Workqueue: events write_dirty_finish
> [ 2162.498360]  ffff88277062fb30 0000000000000046 ffffffff81511b8c
> ffffffff815137c0
> [ 2162.498364]  ffff882fa6786640 ffff88277062ffd8 ffff88277062ffd8
> ffff88277062ffd8
> [ 2162.498368]  ffffffff81c10440 ffff882fa6786640 ffffffff815117a1
> ffff882fa6786640
> [ 2162.498373] Call Trace:
> [ 2162.498378]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
> [ 2162.498382]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
> [ 2162.498387]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
> [ 2162.498392]  [<ffffffff816be299>] schedule+0x29/0x70
> [ 2162.498397]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
> [ 2162.498402]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
> [ 2162.498407]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
> [ 2162.498411]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
> [ 2162.498416]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
> [ 2162.498420]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
> [ 2162.498425]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
> [ 2162.498430]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
> [ 2162.498435]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
> [ 2162.498439]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
> [ 2162.498444]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
> [ 2162.498449]  [<ffffffff810624d4>] process_one_work+0x174/0x490
> [ 2162.498454]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
> [ 2162.498459]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
> [ 2162.498463]  [<ffffffff81069f40>] kthread+0xc0/0xd0
> [ 2162.498467]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2162.498472]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
> [ 2162.498476]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2162.498480] INFO: task iozone:2052 blocked for more than 120 seconds.
> [ 2162.498484] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 2162.498488] iozone          D 0000000000000001     0  2052   1944 0x00000000
> [ 2162.498492]  ffff882f7e2d3978 0000000000000082 ffff882f7e2d3938
> ffffffff81301d7a
> [ 2162.498497]  ffff882f8a116640 ffff882f7e2d3fd8 ffff882f7e2d3fd8
> ffff882f7e2d3fd8
> [ 2162.498501]  ffff882f88b33320 ffff882f8a116640 ffff882f7e2d3978
> ffff882fbf2339f8
> [ 2162.498505] Call Trace:
> [ 2162.498511]  [<ffffffff81301d7a>] ? generic_make_request+0xca/0x100
> [ 2162.498517]  [<ffffffff816be299>] schedule+0x29/0x70
> [ 2162.498522]  [<ffffffff816be36f>] io_schedule+0x8f/0xd0
> [ 2162.498530]  [<ffffffff811b797c>] do_blockdev_direct_IO+0x1a7c/0x1fb0
> [ 2162.498539]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
> [ 2162.498546]  [<ffffffff811b7f05>] __blockdev_direct_IO+0x55/0x60
> [ 2162.498552]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
> [ 2162.498556]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
> [ 2162.498562]  [<ffffffffa0274349>] ext2_direct_IO+0x79/0xe0 [ext2]
> [ 2162.498567]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
> [ 2162.498572]  [<ffffffff8107e775>] ? set_next_entity+0xa5/0xc0
> [ 2162.498579]  [<ffffffff8104ade6>] ? current_fs_time+0x16/0x60
> [ 2162.498585]  [<ffffffff8111f126>] generic_file_direct_write+0xc6/0x180
> [ 2162.498590]  [<ffffffff8111f4bd>] __generic_file_aio_write+0x2dd/0x3b0
> [ 2162.498595]  [<ffffffff816bc364>] ? mutex_unlock+0x14/0x20
> [ 2162.498600]  [<ffffffff8111f5f9>] generic_file_aio_write+0x69/0xd0
> [ 2162.498607]  [<ffffffff8117b88a>] do_sync_write+0x7a/0xb0
> [ 2162.498612]  [<ffffffff8117c63e>] vfs_write+0xce/0x1e0
> [ 2162.498617]  [<ffffffff8117bba0>] ? generic_file_llseek_size+0x110/0x110
> [ 2162.498621]  [<ffffffff8117cb22>] SyS_write+0x52/0xa0
> [ 2162.498627]  [<ffffffff816c7a02>] system_call_fastpath+0x16/0x1b
> [ 2162.498630] INFO: task kworker/0:1:2293 blocked for more than 120 seconds.
> [ 2162.498635] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 2162.498639] kworker/0:1     D ffffffff81813a60     0  2293      2 0x00000000
> [ 2162.498645] Workqueue: events write_dirty_finish
> [ 2162.498647]  ffff882f8a05bb30 0000000000000046 ffffffff81511b8c
> ffffffff815137c0
> [ 2162.498652]  ffff882f8a163320 ffff882f8a05bfd8 ffff882f8a05bfd8
> ffff882f8a05bfd8
> [ 2162.498656]  ffffffff81c10440 ffff882f8a163320 ffffffff815117a1
> ffff882f8a163320
> [ 2162.498661] Call Trace:
> [ 2162.498665]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
> [ 2162.498669]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
> [ 2162.498674]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
> [ 2162.498679]  [<ffffffff816be299>] schedule+0x29/0x70
> [ 2162.498684]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
> [ 2162.498689]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
> [ 2162.498693]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
> [ 2162.498698]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
> [ 2162.498702]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
> [ 2162.498707]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
> [ 2162.498712]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
> [ 2162.498716]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
> [ 2162.498721]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
> [ 2162.498726]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
> [ 2162.498731]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
> [ 2162.498735]  [<ffffffff810624d4>] process_one_work+0x174/0x490
> [ 2162.498740]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
> [ 2162.498745]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
> [ 2162.498749]  [<ffffffff81069f40>] kthread+0xc0/0xd0
> [ 2162.498753]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2162.498758]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
> [ 2162.498762]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2162.498766] INFO: task kworker/0:3:2534 blocked for more than 120 seconds.
> [ 2162.498772] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 2162.498777] kworker/0:3     D ffffffff81813a60     0  2534      2 0x00000000
> [ 2162.498783] Workqueue: events write_dirty_finish
> [ 2162.498785]  ffff88237d0bdb30 0000000000000046 ffffffff81511b8c
> ffffffff815137c0
> [ 2162.498790]  ffff882fa66fe640 ffff88237d0bdfd8 ffff88237d0bdfd8
> ffff88237d0bdfd8
> [ 2162.498794]  ffffffff81c10440 ffff882fa66fe640 ffffffff815117a1
> ffff882fa66fe640
> [ 2162.498799] Call Trace:
> [ 2162.498803]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
> [ 2162.498807]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
> [ 2162.498812]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
> [ 2162.498817]  [<ffffffff816be299>] schedule+0x29/0x70
> [ 2162.498822]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
> [ 2162.498826]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
> [ 2162.498831]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
> [ 2162.498836]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
> [ 2162.498840]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
> [ 2162.498845]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
> [ 2162.498849]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
> [ 2162.498854]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
> [ 2162.498859]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
> [ 2162.498864]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
> [ 2162.498868]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
> [ 2162.498873]  [<ffffffff810624d4>] process_one_work+0x174/0x490
> [ 2162.498878]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
> [ 2162.498882]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
> [ 2162.498886]  [<ffffffff81069f40>] kthread+0xc0/0xd0
> [ 2162.498890]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2162.498896]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
> [ 2162.498900]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2162.498903] INFO: task kworker/0:4:2537 blocked for more than 120 seconds.
> [ 2162.498908] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 2162.498914] kworker/0:4     D 0000000000000000     0  2537      2 0x00000000
> [ 2162.498920] Workqueue: events write_dirty_finish
> [ 2162.498922]  ffff882f76d8fb30 0000000000000046 ffffffff81511b8c
> ffffffff815137c0
> [ 2162.498926]  ffff882f88b30000 ffff882f76d8ffd8 ffff882f76d8ffd8
> ffff882f76d8ffd8
> [ 2162.498931]  ffff882f78beb320 ffff882f88b30000 ffffffff815117a1
> ffff882f88b30000
> [ 2162.498935] Call Trace:
> [ 2162.498940]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
> [ 2162.498944]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
> [ 2162.498948]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
> [ 2162.498953]  [<ffffffff816be299>] schedule+0x29/0x70
> [ 2162.498958]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
> [ 2162.498963]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
> [ 2162.498968]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
> [ 2162.498972]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
> [ 2162.498977]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
> [ 2162.498981]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
> [ 2162.498986]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
> [ 2162.498990]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
> [ 2162.498995]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
> [ 2162.499000]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
> [ 2162.499004]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
> [ 2162.499009]  [<ffffffff810624d4>] process_one_work+0x174/0x490
> [ 2162.499014]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
> [ 2162.499019]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
> [ 2162.499023]  [<ffffffff81069f40>] kthread+0xc0/0xd0
> [ 2162.499027]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2162.499032]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
> [ 2162.499036]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2162.499039] INFO: task kworker/1:1:2544 blocked for more than 120 seconds.
> [ 2162.499044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 2162.499050] kworker/1:1     D ffffffff81813d40     0  2544      2 0x00000000
> [ 2162.499057] Workqueue: bcache bch_data_insert_keys
> [ 2162.499059]  ffff882fa677b3f8 0000000000000046 ffff882fa677b418
> ffff882f8a709000
> [ 2162.499063]  ffff882f88b33320 ffff882fa677bfd8 ffff882fa677bfd8
> ffff882fa677bfd8
> [ 2162.499068]  ffff882fa6471990 ffff882f88b33320 ffff882f88b33320
> ffff8801fad20d98
> [ 2162.499072] Call Trace:
> [ 2162.499077]  [<ffffffff816be299>] schedule+0x29/0x70
> [ 2162.499082]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
> [ 2162.499087]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
> [ 2162.499092]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
> [ 2162.499096]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
> [ 2162.499102]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
> [ 2162.499109]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
> [ 2162.499113]  [<ffffffff81076828>] ? resched_task+0x68/0x70
> [ 2162.499118]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
> [ 2162.499126]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
> [ 2162.499131]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
> [ 2162.499137]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
> [ 2162.499143]  [<ffffffff81126e9c>] __alloc_pages_nodemask+0x60c/0x9b0
> [ 2162.499149]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
> [ 2162.499155]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
> [ 2162.499159]  [<ffffffff8150ebb3>] mca_data_alloc+0x73/0x1d0
> [ 2162.499163]  [<ffffffff81510f37>] mca_alloc+0x277/0x470
> [ 2162.499168]  [<ffffffff81511d1c>] bch_btree_node_alloc+0x8c/0x1c0
> [ 2162.499173]  [<ffffffff81513020>] btree_split+0x110/0x5c0
> [ 2162.499179]  [<ffffffff81457d43>] ? scsi_pool_alloc_command+0x33/0x80
> [ 2162.499184]  [<ffffffff81515fc7>] ? bch_keylist_pop_front+0x47/0x50
> [ 2162.499188]  [<ffffffff8150fed6>] ? bch_btree_insert_keys+0x56/0x250
> [ 2162.499193]  [<ffffffff81082def>] ? update_group_power+0x13f/0x220
> [ 2162.499198]  [<ffffffff81513582>] bch_btree_insert_node+0xb2/0x2f0
> [ 2162.499202]  [<ffffffff815137e8>] btree_insert_fn+0x28/0x50
> [ 2162.499207]  [<ffffffff81511b8c>] bch_btree_map_nodes_recurse+0x6c/0x170
> [ 2162.499211]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
> [ 2162.499216]  [<ffffffff816bcc26>] ? down_write+0x16/0x40
> [ 2162.499221]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
> [ 2162.499225]  [<ffffffff81511c30>] bch_btree_map_nodes_recurse+0x110/0x170
> [ 2162.499230]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
> [ 2162.499235]  [<ffffffff811b51ea>] ? dio_bio_end_io+0x5a/0x90
> [ 2162.499239]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
> [ 2162.499244]  [<ffffffff81514dce>] __bch_btree_map_nodes+0x13e/0x1c0
> [ 2162.499248]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
> [ 2162.499254]  [<ffffffff8151ac6f>] ? bch_journal+0x42f/0x4b0
> [ 2162.499258]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
> [ 2162.499264]  [<ffffffff8151e1be>] bch_data_insert_keys+0x3e/0x160
> [ 2162.499268]  [<ffffffff810624d4>] process_one_work+0x174/0x490
> [ 2162.499273]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
> [ 2162.499278]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
> [ 2162.499282]  [<ffffffff81069f40>] kthread+0xc0/0xd0
> [ 2162.499286]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2162.499292]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
> [ 2162.499296]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2162.499299] INFO: task kworker/0:5:2552 blocked for more than 120 seconds.
> [ 2162.499304] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [ 2162.499310] kworker/0:5     D 0000000000000000     0  2552      2 0x00000000
> [ 2162.499411] Workqueue: events write_dirty_finish
> [ 2162.499413]  ffff88226314db30 0000000000000046 ffffffff81511b8c
> ffffffff815137c0
> [ 2162.499418]  ffff882fa6781990 ffff88226314dfd8 ffff88226314dfd8
> ffff88226314dfd8
> [ 2162.499422]  ffff882fa6a84cb0 ffff882fa6781990 ffffffff815117a1
> ffff882fa6781990
> [ 2162.499427] Call Trace:
> [ 2162.499431]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
> [ 2162.499435]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
> [ 2162.499440]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
> [ 2162.499445]  [<ffffffff816be299>] schedule+0x29/0x70
> [ 2162.499450]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
> [ 2162.499455]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
> [ 2162.499460]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
> [ 2162.499464]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
> [ 2162.499468]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
> [ 2162.499473]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
> [ 2162.499478]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
> [ 2162.499483]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
> [ 2162.499488]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
> [ 2162.499492]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
> [ 2162.499497]  [<ffffffff810624d4>] process_one_work+0x174/0x490
> [ 2162.499502]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
> [ 2162.499507]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
> [ 2162.499511]  [<ffffffff81069f40>] kthread+0xc0/0xd0
> [ 2162.499515]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> [ 2162.499520]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
> [ 2162.499524]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
> 
> 
> 
>>> [  930.169234] bcache: bch_cached_dev_attach() Caching sda3 as bcache0
>>> on set 06496904-10ca-489a-ae75-68c6a07d3db1
>>> [ 2522.956188] INFO: task bcache_writebac:2058 blocked for more than
>>> 120 seconds.
>>> [ 2522.956199] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>> disables this message.
>>> [ 2522.956205] bcache_writebac D ffffffff81813a60     0  2058      2 0x00000000
>>> [ 2522.956212]  ffff881c39c1fcd8 0000000000000046 0000000000000001
>>> 0000000000000001
>>> [ 2522.956218]  ffff882f8b56b320 ffff881c39c1ffd8 ffff881c39c1ffd8
>>> ffff881c39c1ffd8
>>> [ 2522.956222]  ffff882fa6aeb320 ffff882f8b56b320 ffff882f8b56b320
>>> ffff882f8b56b320
>>> [ 2522.956227] Call Trace:
>>> [ 2522.956241]  [<ffffffff816be299>] schedule+0x29/0x70
>>> [ 2522.956247]  [<ffffffff816befb5>] rwsem_down_write_failed+0xf5/0x1a0
>>> [ 2522.956255]  [<ffffffff81332c93>] call_rwsem_down_write_failed+0x13/0x20
>>> [ 2522.956261]  [<ffffffff816bcc41>] ? down_write+0x31/0x40
>>> [ 2522.956269]  [<ffffffff8151b782>] bch_writeback_thread+0x62/0x7f0
>>> [ 2522.956278]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
>>> [ 2522.956284]  [<ffffffff8151b720>] ? write_dirty+0xc0/0xc0
>>> [ 2522.956290]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>> [ 2522.956294]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2522.956302]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>> [ 2522.956306]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2522.956313] INFO: task kworker/u64:1:2433 blocked for more than 120 seconds.
>>> [ 2522.956317] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>> disables this message.
>>> [ 2522.956322] kworker/u64:1   D ffffffff81813a60     0  2433      2 0x00000000
>>> [ 2522.956332] Workqueue: writeback bdi_writeback_workfn (flush-252:0)
>>> [ 2522.956336]  ffff882f899d1560 0000000000000046 ffffffff81120635
>>> 0000000000000100
>>> [ 2522.956340]  ffff88219b488000 ffff882f899d1fd8 ffff882f899d1fd8
>>> ffff882f899d1fd8
>>> [ 2522.956345]  ffff882fa6aeb320 ffff88219b488000 ffffffff8131ad55
>>> ffff88219b488000
>>> [ 2522.956349] Call Trace:
>>> [ 2522.956359]  [<ffffffff81120635>] ? mempool_alloc_slab+0x15/0x20
>>> [ 2522.956367]  [<ffffffff8131ad55>] ? blk_throtl_bio+0x3b5/0x540
>>> [ 2522.956372]  [<ffffffff816be299>] schedule+0x29/0x70
>>> [ 2522.956378]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>> [ 2522.956383]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
>>> [ 2522.956388]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>> [ 2522.956394]  [<ffffffff8151fe32>] cached_dev_make_request+0x752/0xf00
>>> [ 2522.956400]  [<ffffffff81301af8>] ? generic_make_request_checks+0x1e8/0x3a0
>>> [ 2522.956405]  [<ffffffff81301d7a>] generic_make_request+0xca/0x100
>>> [ 2522.956409]  [<ffffffff81301e29>] submit_bio+0x79/0x160
>>> [ 2522.956417]  [<ffffffff811b1710>] ? bio_alloc_bioset+0xa0/0x1d0
>>> [ 2522.956421]  [<ffffffff811abd3f>] _submit_bh+0x13f/0x200
>>> [ 2522.956425]  [<ffffffff811abe10>] submit_bh+0x10/0x20
>>> [ 2522.956430]  [<ffffffff811af6b8>] __block_write_full_page+0x1d8/0x360
>>> [ 2522.956439]  [<ffffffff8145fc0b>] ? scsi_request_fn+0xbb/0x530
>>> [ 2522.956444]  [<ffffffff811ad3c0>] ? end_buffer_async_read+0x130/0x130
>>> [ 2522.956449]  [<ffffffff811b2c40>] ? I_BDEV+0x10/0x10
>>> [ 2522.956454]  [<ffffffff811b2c40>] ? I_BDEV+0x10/0x10
>>> [ 2522.956459]  [<ffffffff811af90a>] block_write_full_page_endio+0xca/0x100
>>> [ 2522.956464]  [<ffffffff811af955>] block_write_full_page+0x15/0x20
>>> [ 2522.956469]  [<ffffffff811b3608>] blkdev_writepage+0x18/0x20
>>> [ 2522.956473]  [<ffffffff81127d37>] __writepage+0x17/0x40
>>> [ 2522.956477]  [<ffffffff8112820e>] write_cache_pages+0x20e/0x460
>>> [ 2522.956481]  [<ffffffff81127d20>] ? set_page_dirty_lock+0x60/0x60
>>> [ 2522.956486]  [<ffffffff811ac251>] ? __set_page_dirty+0x71/0xc0
>>> [ 2522.956490]  [<ffffffff811284aa>] generic_writepages+0x4a/0x70
>>> [ 2522.956495]  [<ffffffff81129cd0>] do_writepages+0x20/0x40
>>> [ 2522.956501]  [<ffffffff811a36f5>] __writeback_single_inode+0x45/0x280
>>> [ 2522.956507]  [<ffffffff811269e2>] ? __alloc_pages_nodemask+0x152/0x9b0
>>> [ 2522.956513]  [<ffffffff811a495d>] writeback_sb_inodes+0x19d/0x3c0
>>> [ 2522.956518]  [<ffffffff811a4c1e>] __writeback_inodes_wb+0x9e/0xd0
>>> [ 2522.956523]  [<ffffffff811a4ecb>] wb_writeback+0x27b/0x320
>>> [ 2522.956529]  [<ffffffff81196228>] ? get_nr_dirty_inodes+0x58/0x80
>>> [ 2522.956534]  [<ffffffff811a500f>] wb_check_old_data_flush+0x9f/0xb0
>>> [ 2522.956539]  [<ffffffff811a67e1>] wb_do_writeback+0x151/0x1d0
>>> [ 2522.956547]  [<ffffffff8106539f>] ? set_worker_desc+0x6f/0x80
>>> [ 2522.956552]  [<ffffffff811a68da>] bdi_writeback_workfn+0x7a/0x200
>>> [ 2522.956557]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>> [ 2522.956562]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>> [ 2522.956567]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>>> [ 2522.956571]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>> [ 2522.956575]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2522.956581]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>> [ 2522.956585]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2522.956588] INFO: task iozone:2435 blocked for more than 120 seconds.
>>> [ 2522.956593] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>> disables this message.
>>> [ 2522.956597] iozone          D 0000000000000001     0  2435   1545 0x00000000
>>> [ 2522.956602]  ffff882a75671978 0000000000000082 ffff882a75671938
>>> ffffffff81301d7a
>>> [ 2522.956606]  ffff882f88dc1990 ffff882a75671fd8 ffff882a75671fd8
>>> ffff882a75671fd8
>>> [ 2522.956611]  ffff882f8ec91990 ffff882f88dc1990 ffff882a75671978
>>> ffff882fbf2539f8
>>> [ 2522.956615] Call Trace:
>>> [ 2522.956619]  [<ffffffff81301d7a>] ? generic_make_request+0xca/0x100
>>> [ 2522.956625]  [<ffffffff816be299>] schedule+0x29/0x70
>>> [ 2522.956630]  [<ffffffff816be36f>] io_schedule+0x8f/0xd0
>>> [ 2522.956636]  [<ffffffff811b797c>] do_blockdev_direct_IO+0x1a7c/0x1fb0
>>> [ 2522.956645]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
>>> [ 2522.956652]  [<ffffffff811b7f05>] __blockdev_direct_IO+0x55/0x60
>>> [ 2522.956658]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
>>> [ 2522.956662]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>> [ 2522.956669]  [<ffffffffa0263349>] ext2_direct_IO+0x79/0xe0 [ext2]
>>> [ 2522.956674]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
>>> [ 2522.956680]  [<ffffffff8104ade6>] ? current_fs_time+0x16/0x60
>>> [ 2522.956685]  [<ffffffff8111f126>] generic_file_direct_write+0xc6/0x180
>>> [ 2522.956690]  [<ffffffff8111f4bd>] __generic_file_aio_write+0x2dd/0x3b0
>>> [ 2522.956696]  [<ffffffff8111f5f9>] generic_file_aio_write+0x69/0xd0
>>> [ 2522.956702]  [<ffffffff8117b88a>] do_sync_write+0x7a/0xb0
>>> [ 2522.956706]  [<ffffffff811bb108>] ? fsnotify+0x1f8/0x2b0
>>> [ 2522.956712]  [<ffffffff8117c63e>] vfs_write+0xce/0x1e0
>>> [ 2522.956716]  [<ffffffff8117cb22>] SyS_write+0x52/0xa0
>>> [ 2522.956722]  [<ffffffff816c7a02>] system_call_fastpath+0x16/0x1b
>>> [ 2522.956726] INFO: task kworker/2:1:2470 blocked for more than 120 seconds.
>>> [ 2522.956730] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>> disables this message.
>>> [ 2522.956735] kworker/2:1     D ffffffff81813d40     0  2470      2 0x00000000
>>> [ 2522.956743] Workqueue: bcache bch_data_insert_keys
>>> [ 2522.956745]  ffff882a5b0c9628 0000000000000046 ffff882a5b0c9648
>>> 0000004000000000
>>> [ 2522.956749]  ffff882f8ec91990 ffff882a5b0c9fd8 ffff882a5b0c9fd8
>>> ffff882a5b0c9fd8
>>> [ 2522.956754]  ffff882fa6489990 ffff882f8ec91990 ffff882f8ec91990
>>> ffff8828dba00d98
>>> [ 2522.956758] Call Trace:
>>> [ 2522.956763]  [<ffffffff816be299>] schedule+0x29/0x70
>>> [ 2522.956769]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
>>> [ 2522.956774]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
>>> [ 2522.956779]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>> [ 2522.956783]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
>>> [ 2522.956791]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
>>> [ 2522.956799]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
>>> [ 2522.956805]  [<ffffffff81076828>] ? resched_task+0x68/0x70
>>> [ 2522.956810]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
>>> [ 2522.956816]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
>>> [ 2522.956822]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
>>> [ 2522.956827]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
>>> [ 2522.956833]  [<ffffffff81126e9c>] __alloc_pages_nodemask+0x60c/0x9b0
>>> [ 2522.956843]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
>>> [ 2522.956848]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
>>> [ 2522.956852]  [<ffffffff8150ebb3>] mca_data_alloc+0x73/0x1d0
>>> [ 2522.956857]  [<ffffffff8150ee5a>] mca_bucket_alloc+0x14a/0x1f0
>>> [ 2522.956861]  [<ffffffff81511020>] mca_alloc+0x360/0x470
>>> [ 2522.956866]  [<ffffffff81517031>] ? __bch_bset_search+0x1d1/0x480
>>> [ 2522.956871]  [<ffffffff8151183e>] bch_btree_node_get+0x10e/0x280
>>> [ 2522.956875]  [<ffffffff81511c02>] bch_btree_map_nodes_recurse+0xe2/0x170
>>> [ 2522.956880]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>> [ 2522.956886]  [<ffffffff811b51ea>] ? dio_bio_end_io+0x5a/0x90
>>> [ 2522.956890]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>> [ 2522.956895]  [<ffffffff81514dce>] __bch_btree_map_nodes+0x13e/0x1c0
>>> [ 2522.956899]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>> [ 2522.956905]  [<ffffffff8151ac6f>] ? bch_journal+0x42f/0x4b0
>>> [ 2522.956909]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>> [ 2522.956915]  [<ffffffff8151e1be>] bch_data_insert_keys+0x3e/0x160
>>> [ 2522.956920]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>> [ 2522.956925]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>> [ 2522.956930]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>>> [ 2522.956933]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>> [ 2522.956938]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2522.956943]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>> [ 2522.956947]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2522.956951] INFO: task kworker/3:2:2471 blocked for more than 120 seconds.
>>> [ 2522.956955] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>> disables this message.
>>> [ 2522.956959] kworker/3:2     D ffffffff81813a60     0  2471      2 0x00000000
>>> [ 2522.956966] Workqueue: events update_writeback_rate
>>> [ 2522.956968]  ffff882a5d125cd0 0000000000000046 0000000000000003
>>> ffff882a5d125d68
>>> [ 2522.956972]  ffff882f8ec94cb0 ffff882a5d125fd8 ffff882a5d125fd8
>>> ffff882a5d125fd8
>>> [ 2522.956977]  ffff882fa6aeb320 ffff882f8ec94cb0 0000000200000003
>>> ffff882f8ec94cb0
>>> [ 2522.956981] Call Trace:
>>> [ 2522.956987]  [<ffffffff816be299>] schedule+0x29/0x70
>>> [ 2522.956992]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>> [ 2522.956997]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
>>> [ 2522.957002]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>> [ 2522.957007]  [<ffffffff8151b010>] update_writeback_rate+0x30/0x230
>>> [ 2522.957011]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>> [ 2522.957017]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>> [ 2522.957021]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>>> [ 2522.957025]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>> [ 2522.957030]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2522.957035]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>> [ 2522.957039]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2522.957048] INFO: task kworker/u64:2:2540 blocked for more than 120 seconds.
>>> [ 2522.957052] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>> disables this message.
>>> [ 2522.957056] kworker/u64:2   D ffffffff81813a60     0  2540      2 0x00000000
>>> [ 2522.957062] Workqueue: bch_btree_io btree_node_write_work
>>> [ 2522.957064]  ffff882a4fdc9778 0000000000000046 ffff882a4fdc9738
>>> ffff882f86e3be58
>>> [ 2522.957069]  ffff882fa4bc1990 ffff882a4fdc9fd8 ffff882a4fdc9fd8
>>> ffff882a4fdc9fd8
>>> [ 2522.957073]  ffff882fa6aeb320 ffff882fa4bc1990 ffff882f86e3be00
>>> ffff8828dba00d98
>>> [ 2522.957078] Call Trace:
>>> [ 2522.957083]  [<ffffffff816be299>] schedule+0x29/0x70
>>> [ 2522.957088]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
>>> [ 2522.957092]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
>>> [ 2522.957102]  [<ffffffff81484090>] ? ata_scsiop_mode_sense+0x380/0x380
>>> [ 2522.957107]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>> [ 2522.957111]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
>>> [ 2522.957116]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
>>> [ 2522.957121]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
>>> [ 2522.957126]  [<ffffffff81076828>] ? resched_task+0x68/0x70
>>> [ 2522.957130]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
>>> [ 2522.957135]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
>>> [ 2522.957140]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
>>> [ 2522.957146]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
>>> [ 2522.957152]  [<ffffffff81126e9c>] __alloc_pages_nodemask+0x60c/0x9b0
>>> [ 2522.957158]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
>>> [ 2522.957163]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
>>> [ 2522.957168]  [<ffffffff81517fc8>] __btree_sort+0x48/0x230
>>> [ 2522.957173]  [<ffffffff8151765c>] ? __bch_btree_iter_init+0x7c/0xc0
>>> [ 2522.957178]  [<ffffffff81518301>] bch_btree_sort_partial+0x101/0x120
>>> [ 2522.957182]  [<ffffffff8150f000>] ? __btree_node_write_done+0x100/0x100
>>> [ 2522.957187]  [<ffffffff81518468>] bch_btree_sort_lazy+0x68/0x90
>>> [ 2522.957191]  [<ffffffff815109ba>] bch_btree_node_write+0x36a/0x4a0
>>> [ 2522.957196]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
>>> [ 2522.957201]  [<ffffffff8106212c>] ? pwq_activate_delayed_work+0x4c/0xb0
>>> [ 2522.957205]  [<ffffffff81511477>] btree_node_write_work+0x57/0x80
>>> [ 2522.957210]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>> [ 2522.957215]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>> [ 2522.957219]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>>> [ 2522.957223]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>> [ 2522.957228]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2522.957233]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>> [ 2522.957237]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> 
>>> 
>>> I can easily reproduce this, please let me know if you require any
>>> info/test any patch.
>>> 
>>> Thanks,
>>> Neo
>>> 
>>> On Mon, Aug 26, 2013 at 12:21 PM, Stefan Priebe <s.priebe@profihost.ag> wrote:
>>>> Hi Kent,
>>>> 
>>>> a new one under 3.10:
>>>> 
>>>> 2013-08-26 21:05:30     INFO: task ceph-osd:8939 blocked for more than 120
>>>> seconds.
>>>> 2013-08-26 21:05:30     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>> disables this message.
>>>> 2013-08-26 21:05:30     ceph-osd D ffffffff8160f760 0 8939 1 0x00000000
>>>> 2013-08-26 21:05:30     ffff880bd9a61820 0000000000000086 ffff880bd9a61fd8
>>>> 0000000000012c80
>>>> 2013-08-26 21:05:30     ffff880bd9a60010 0000000000012c80 0000000000012c80
>>>> 0000000000012c80
>>>> 2013-08-26 21:05:30     ffff880bd9a61fd8 0000000000012c80 ffff880bd7abb1c0
>>>> ffff880c48dee380
>>>> 2013-08-26 21:05:29     [<ffffffff811e8128>] xfs_vm_readpages+0x18/0x20
>>>> 2013-08-26 21:05:29     [<ffffffff810f1da3>] read_pages+0x43/0x100
>>>> 2013-08-26 21:05:29     [<ffffffff810e64df>] ? __page_cache_alloc+0x9f/0xc0
>>>> 2013-08-26 21:05:29     [<ffffffff810f1fab>]
>>>> __do_page_cache_readahead+0x14b/0x160
>>>> 2013-08-26 21:05:29     [<ffffffff810f1fdc>] ra_submit+0x1c/0x20
>>>> 2013-08-26 21:05:29     [<ffffffff810f22a5>] ondemand_readahead+0x115/0x240
>>>> 2013-08-26 21:05:29     [<ffffffff810f249e>]
>>>> page_cache_sync_readahead+0x2e/0x40
>>>> 2013-08-26 21:05:29     [<ffffffff810e7a5e>] T.1003+0x33e/0x430
>>>> 2013-08-26 21:05:29     [<ffffffff810e7c30>]
>>>> generic_file_aio_read+0xe0/0x220
>>>> 2013-08-26 21:05:29     [<ffffffff811efd0a>] xfs_file_aio_read+0x15a/0x2a0
>>>> 2013-08-26 21:05:29     [<ffffffff8114095a>] do_sync_read+0x7a/0xb0
>>>> 2013-08-26 21:05:29     [<ffffffff81140df1>] vfs_read+0xb1/0x130
>>>> 2013-08-26 21:05:29     [<ffffffff81141757>] SyS_pread64+0x97/0xa0
>>>> 2013-08-26 21:05:29     [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b
>>>> 2013-08-26 21:05:29     INFO: task ceph-osd:8896 blocked for more than 120
>>>> seconds.
>>>> 2013-08-26 21:05:29     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>> disables this message.
>>>> 2013-08-26 21:05:29     ceph-osd D ffffffff8160f760 0 8896 1 0x00000000
>>>> 2013-08-26 21:05:29     ffff880bd9a9dc28 0000000000000086 ffff880bd9a9dfd8
>>>> 0000000000012c80
>>>> 2013-08-26 21:05:29     ffff880bd9a9c010 0000000000012c80 0000000000012c80
>>>> 0000000000012c80
>>>> 2013-08-26 21:05:29     ffff880bd9a9dfd8 0000000000012c80 ffff880bd79298e0
>>>> ffff880c48dd6380
>>>> 2013-08-26 21:05:29     Call Trace:
>>>> 2013-08-26 21:05:29     [<ffffffff810e60f0>] ? __lock_page+0x70/0x70
>>>> 2013-08-26 21:05:28     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>> disables this message.
>>>> 2013-08-26 21:05:28     ceph-osd D ffffffff8160f760 0 8798 1 0x00000000
>>>> 2013-08-26 21:05:28     ffff880bd998b740 0000000000000086 ffff880bd998bfd8
>>>> 0000000000012c80
>>>> 2013-08-26 21:05:28     ffff880bd998a010 0000000000012c80 0000000000012c80
>>>> 0000000000012c80
>>>> 2013-08-26 21:05:28     ffff880bd998bfd8 0000000000012c80 ffff880bd99c0000
>>>> ffff880c48dd6380
>>>> 2013-08-26 21:05:28     Call Trace:
>>>> 2013-08-26 21:05:28     [<ffffffff81524874>] schedule+0x24/0x70
>>>> 2013-08-26 21:05:28     [<ffffffff81525d7d>]
>>>> rwsem_down_read_failed+0x9d/0xe5
>>>> 2013-08-26 21:05:28     [<ffffffff812aa0a4>]
>>>> call_rwsem_down_read_failed+0x14/0x30
>>>> 2013-08-26 21:05:28     [<ffffffff81523a42>] ? down_read+0x12/0x20
>>>> 2013-08-26 21:05:28     [<ffffffffa01ead62>] btree_read_async+0xa2/0x1a0
>>>> [bcache]
>>>> 2013-08-26 21:05:28     [<ffffffffa01f7b73>] closure_queue+0x43/0x60
>>>> [bcache]
>>>> 2013-08-26 21:05:28     [<ffffffffa01eb903>] T.1050+0x63/0x70 [bcache]
>>>> 2013-08-26 21:05:28     [<ffffffffa01ed6ea>]
>>>> cached_dev_make_request+0x20a/0x350 [bcache]
>>>> 2013-08-26 21:05:28     [<ffffffff81281952>] generic_make_request+0xc2/0x100
>>>> 2013-08-26 21:05:28     [<ffffffff812819f7>] submit_bio+0x67/0x130
>>>> 2013-08-26 21:05:28     [<ffffffff8117b240>] do_mpage_readpage+0x2a0/0x660
>>>> 2013-08-26 21:05:28     [<ffffffff81101cee>] ?
>>>> __inc_zone_page_state+0x2e/0x30
>>>> 2013-08-26 21:05:28     [<ffffffff810e6581>] ?
>>>> add_to_page_cache_locked+0x81/0x110
>>>> 2013-08-26 21:05:28     [<ffffffff8117b76a>] mpage_readpages+0xfa/0x150
>>>> 2013-08-26 21:05:28     [<ffffffff811e90c0>] ?
>>>> xfs_get_blocks_direct+0x20/0x20
>>>> 2013-08-26 21:05:28     [<ffffffff811e90c0>] ?
>>>> xfs_get_blocks_direct+0x20/0x20
>>>> 2013-08-26 21:05:28     [<ffffffff810eeda3>] ?
>>>> __alloc_pages_nodemask+0x123/0x240
>>>> 2013-08-26 21:05:27     [<ffffffff8115bf93>] ? __close_fd+0x43/0x80
>>>> 2013-08-26 21:05:27     [<ffffffff8116dabe>] SyS_fdatasync+0xe/0x20
>>>> 2013-08-26 21:05:27     [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b
>>>> 2013-08-26 21:05:27     INFO: task ceph-osd:8746 blocked for more than 120
>>>> seconds.
>>>> 2013-08-26 21:05:27     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>> disables this message.
>>>> 2013-08-26 21:05:27     ceph-osd D ffffffff8160f760 0 8746 1 0x00000000
>>>> 2013-08-26 21:05:27     ffff880c2dde1d38 0000000000000086 ffff880c2dde1fd8
>>>> 0000000000012c80
>>>> 2013-08-26 21:05:27     ffff880c2dde0010 0000000000012c80 0000000000012c80
>>>> 0000000000012c80
>>>> 2013-08-26 21:05:27     ffff880c2dde1fd8 0000000000012c80 ffff880c239eb1c0
>>>> ffff880c48dd6380
>>>> 2013-08-26 21:05:27     Call Trace:
>>>> 2013-08-26 21:05:27     [<ffffffff81524874>] schedule+0x24/0x70
>>>> 2013-08-26 21:05:27     [<ffffffff81522bbd>] schedule_timeout+0x16d/0x200
>>>> 2013-08-26 21:05:27     [<ffffffff81068eba>] ?
>>>> __queue_delayed_work+0xaa/0x1a0
>>>> 2013-08-26 21:05:27     [<ffffffff81068d19>] ?
>>>> try_to_grab_pending+0x109/0x190
>>>> 2013-08-26 21:05:27     [<ffffffff815251e5>] wait_for_completion+0x95/0x110
>>>> 2013-08-26 21:05:27     [<ffffffff8107fd00>] ? try_to_wake_up+0x2a0/0x2a0
>>>> 2013-08-26 21:05:27     [<ffffffff81167797>] ? bdi_queue_work+0x77/0xc0
>>>> 2013-08-26 21:05:27     [<ffffffff81167863>]
>>>> writeback_inodes_sb_nr+0x83/0xb0
>>>> 2013-08-26 21:05:27     [<ffffffff811678ea>] writeback_inodes_sb+0x5a/0x70
>>>> 2013-08-26 21:05:27     [<ffffffff8116dd0a>] __sync_filesystem+0x4a/0x50
>>>> 2013-08-26 21:05:27     [<ffffffff8116dd42>] sync_filesystem+0x32/0x60
>>>> 2013-08-26 21:05:27     [<ffffffff8116ddc0>] SyS_syncfs+0x50/0x90
>>>> 2013-08-26 21:05:27     [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b
>>>> 2013-08-26 21:05:27     INFO: task ceph-osd:8798 blocked for more than 120
>>>> seconds.
>>>> 
>>>> Stefan
>>>> 
>>>> Am 22.08.2013 09:32, schrieb Stefan Priebe - Profihost AG:
>>>> 
>>>>> great!
>>>>> 
>>>>> Everything seems to work fine now! Except read_dirty always going to
>>>>> negative values after a reboot.
>>>>> 
>>>>> Stefan
>>>>> 
>>>>> Am 22.08.2013 08:02, schrieb Kent Overstreet:
>>>>>> 
>>>>>> On Thu, Aug 22, 2013 at 07:59:04AM +0200, Stefan Priebe wrote:
>>>>>>> 
>>>>>>> 
>>>>>>>> schedule_timeout() is not the same as
>>>>>>>> schedule_timeout_interruptible().
>>>>>>> 
>>>>>>> 
>>>>>>> just search and replace? So i can try on my own.
>>>>>> 
>>>>>> 
>>>>>> The one in read_dirty(), line ~330
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-bcache"
>>>>>> in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-28 18:38                                 ` Stefan Priebe - Profihost AG
@ 2013-08-28 18:47                                   ` kernel neophyte
  2013-08-28 19:20                                     ` Stefan Priebe
  0 siblings, 1 reply; 28+ messages in thread
From: kernel neophyte @ 2013-08-28 18:47 UTC (permalink / raw)
  To: Stefan Priebe - Profihost AG
  Cc: Kent Overstreet, Jens Axboe, linux-kernel@vger.kernel.org,
	linux-bcache@vger.kernel.org

On Wed, Aug 28, 2013 at 11:38 AM, Stefan Priebe - Profihost AG
<s.priebe@profihost.ag> wrote:
> I don't had one for a few days. Which kernel so you use?

3.10 kernel with all of kent's stable patches and perf patches..

and all defaults except for:

echo writeback > /sys/block/bcache0/bcache/cache_mode
echo 0 > /sys/block/bcache0/bcache/sequential_cutoff
echo 0 > /sys/fs/bcache/1e26e5ca-5cb9-4c5a-ad7e-3c787524a9dc/congested_read_threshold_us
echo 0 > /sys/fs/bcache/1e26e5ca-5cb9-4c5a-ad7e-3c787524a9dc/congested_write_threshold_us

-Neo

>
> Stefan
>
> This mail was sent with my iPhone.
>
> Am 28.08.2013 um 20:12 schrieb kernel neophyte <neophyte.hacker001@gmail.com>:
>
>> On Tue, Aug 27, 2013 at 11:05 PM, Kent Overstreet <kmo@daterainc.com> wrote:
>>> On Tue, Aug 27, 2013 at 08:37:54PM -0700, kernel neophyte wrote:
>>>> Hi Kent,
>>>>
>>>> I am still seeing deadlock:
>>>
>>> Seeing blk_throtl_bio (what asshole misspells words in their function
>>> names, so when you go to grep for them you grep for the wrong thing?) in
>>> the backtrace - is that what it takes to hit it?
>>
>> I don't think so,
>> I had another deadlock:
>>
>> [ 1179.193990] bio: create slab <bio-2> at 2
>> [ 1179.194491] bcache: run_cache_set() invalidating existing data
>> [ 1179.212308] bcache: register_cache() registered cache device rxd1
>> [ 1203.846220] bcache: register_bdev() registered backing device sda3
>> [ 1227.780088] bcache: bch_cached_dev_attach() Caching sda3 as bcache0
>> on set a079dda3-27fd-4b1c-a616-d46110bfae6c
>> [ 2162.497950] INFO: task kworker/0:0:4 blocked for more than 120 seconds.
>> [ 2162.497961] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [ 2162.497967] kworker/0:0     D 0000000000000000     0     4      2 0x00000000
>> [ 2162.497982] Workqueue: events write_dirty_finish
>> [ 2162.497986]  ffff882fa6ac9b30 0000000000000046 ffffffff81511b8c
>> ffffffff815137c0
>> [ 2162.497992]  ffff882fa6a84cb0 ffff882fa6ac9fd8 ffff882fa6ac9fd8
>> ffff882fa6ac9fd8
>> [ 2162.497996]  ffff882f78139990 ffff882fa6a84cb0 ffffffff815117a1
>> ffff882fa6a84cb0
>> [ 2162.498001] Call Trace:
>> [ 2162.498010]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
>> [ 2162.498015]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>> [ 2162.498019]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>> [ 2162.498028]  [<ffffffff816be299>] schedule+0x29/0x70
>> [ 2162.498034]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>> [ 2162.498043]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
>> [ 2162.498048]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>> [ 2162.498052]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>> [ 2162.498057]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>> [ 2162.498066]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>> [ 2162.498072]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>> [ 2162.498076]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>> [ 2162.498082]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>> [ 2162.498087]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>> [ 2162.498093]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>> [ 2162.498100]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>> [ 2162.498105]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>> [ 2162.498110]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>> [ 2162.498115]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>> [ 2162.498120]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2162.498127]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>> [ 2162.498131]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2162.498145] INFO: task bcache_allocato:1766 blocked for more than
>> 120 seconds.
>> [ 2162.498149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [ 2162.498154] bcache_allocato D 0000000000000001     0  1766      2 0x00000000
>> [ 2162.498158]  ffff8822630f3d88 0000000000000046 ffff8822630f3da8
>> ffffffff810808ad
>> [ 2162.498163]  ffff882f891de640 ffff8822630f3fd8 ffff8822630f3fd8
>> ffff8822630f3fd8
>> [ 2162.498167]  ffff882f81a91990 ffff882f891de640 ffff882f88b33320
>> ffff8801fad20d98
>> [ 2162.498172] Call Trace:
>> [ 2162.498177]  [<ffffffff810808ad>] ? dequeue_task_fair+0x2cd/0x530
>> [ 2162.498182]  [<ffffffff816be299>] schedule+0x29/0x70
>> [ 2162.498187]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
>> [ 2162.498192]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
>> [ 2162.498197]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>> [ 2162.498202]  [<ffffffff8150cdbf>] bch_allocator_thread+0x10f/0xe20
>> [ 2162.498207]  [<ffffffff8150ccb0>] ? bch_bucket_add_unused+0xe0/0xe0
>> [ 2162.498211]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>> [ 2162.498215]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2162.498220]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>> [ 2162.498224]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2162.498228] INFO: task bcache_writebac:1771 blocked for more than
>> 120 seconds.
>> [ 2162.498233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [ 2162.498237] bcache_writebac D 0000000000000000     0  1771      2 0x00000000
>> [ 2162.498241]  ffff88197d2adc58 0000000000000046 ffff88197d2add48
>> ffffffff811269e2
>> [ 2162.498245]  ffff882f891d8000 ffff88197d2adfd8 ffff88197d2adfd8
>> ffff88197d2adfd8
>> [ 2162.498250]  ffff882f81a91990 ffff882f891d8000 ffff88197d2adc68
>> ffff882f89550ba0
>> [ 2162.498254] Call Trace:
>> [ 2162.498264]  [<ffffffff811269e2>] ? __alloc_pages_nodemask+0x152/0x9b0
>> [ 2162.498269]  [<ffffffff816be299>] schedule+0x29/0x70
>> [ 2162.498274]  [<ffffffff816bc015>] schedule_timeout+0x1e5/0x250
>> [ 2162.498280]  [<ffffffff810799bf>] ? try_to_wake_up+0x24f/0x2b0
>> [ 2162.498285]  [<ffffffff816bcd09>] __down_common+0x8e/0xe3
>> [ 2162.498293]  [<ffffffff81160600>] ? alloc_pages_current+0x90/0x170
>> [ 2162.498298]  [<ffffffff816bcdd1>] __down+0x1d/0x1f
>> [ 2162.498304]  [<ffffffff8106f351>] down+0x41/0x50
>> [ 2162.498310]  [<ffffffff8151ba7d>] bch_writeback_thread+0x35d/0x7f0
>> [ 2162.498315]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
>> [ 2162.498321]  [<ffffffff8151b720>] ? write_dirty+0xc0/0xc0
>> [ 2162.498325]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>> [ 2162.498329]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2162.498334]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>> [ 2162.498338]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2162.498342] INFO: task kworker/0:2:1789 blocked for more than 120 seconds.
>> [ 2162.498346] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [ 2162.498351] kworker/0:2     D ffffffff81813a60     0  1789      2 0x00000000
>> [ 2162.498357] Workqueue: events write_dirty_finish
>> [ 2162.498360]  ffff88277062fb30 0000000000000046 ffffffff81511b8c
>> ffffffff815137c0
>> [ 2162.498364]  ffff882fa6786640 ffff88277062ffd8 ffff88277062ffd8
>> ffff88277062ffd8
>> [ 2162.498368]  ffffffff81c10440 ffff882fa6786640 ffffffff815117a1
>> ffff882fa6786640
>> [ 2162.498373] Call Trace:
>> [ 2162.498378]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
>> [ 2162.498382]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>> [ 2162.498387]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>> [ 2162.498392]  [<ffffffff816be299>] schedule+0x29/0x70
>> [ 2162.498397]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>> [ 2162.498402]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
>> [ 2162.498407]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>> [ 2162.498411]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>> [ 2162.498416]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>> [ 2162.498420]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>> [ 2162.498425]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>> [ 2162.498430]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>> [ 2162.498435]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>> [ 2162.498439]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>> [ 2162.498444]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>> [ 2162.498449]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>> [ 2162.498454]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>> [ 2162.498459]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>> [ 2162.498463]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>> [ 2162.498467]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2162.498472]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>> [ 2162.498476]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2162.498480] INFO: task iozone:2052 blocked for more than 120 seconds.
>> [ 2162.498484] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [ 2162.498488] iozone          D 0000000000000001     0  2052   1944 0x00000000
>> [ 2162.498492]  ffff882f7e2d3978 0000000000000082 ffff882f7e2d3938
>> ffffffff81301d7a
>> [ 2162.498497]  ffff882f8a116640 ffff882f7e2d3fd8 ffff882f7e2d3fd8
>> ffff882f7e2d3fd8
>> [ 2162.498501]  ffff882f88b33320 ffff882f8a116640 ffff882f7e2d3978
>> ffff882fbf2339f8
>> [ 2162.498505] Call Trace:
>> [ 2162.498511]  [<ffffffff81301d7a>] ? generic_make_request+0xca/0x100
>> [ 2162.498517]  [<ffffffff816be299>] schedule+0x29/0x70
>> [ 2162.498522]  [<ffffffff816be36f>] io_schedule+0x8f/0xd0
>> [ 2162.498530]  [<ffffffff811b797c>] do_blockdev_direct_IO+0x1a7c/0x1fb0
>> [ 2162.498539]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
>> [ 2162.498546]  [<ffffffff811b7f05>] __blockdev_direct_IO+0x55/0x60
>> [ 2162.498552]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
>> [ 2162.498556]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>> [ 2162.498562]  [<ffffffffa0274349>] ext2_direct_IO+0x79/0xe0 [ext2]
>> [ 2162.498567]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
>> [ 2162.498572]  [<ffffffff8107e775>] ? set_next_entity+0xa5/0xc0
>> [ 2162.498579]  [<ffffffff8104ade6>] ? current_fs_time+0x16/0x60
>> [ 2162.498585]  [<ffffffff8111f126>] generic_file_direct_write+0xc6/0x180
>> [ 2162.498590]  [<ffffffff8111f4bd>] __generic_file_aio_write+0x2dd/0x3b0
>> [ 2162.498595]  [<ffffffff816bc364>] ? mutex_unlock+0x14/0x20
>> [ 2162.498600]  [<ffffffff8111f5f9>] generic_file_aio_write+0x69/0xd0
>> [ 2162.498607]  [<ffffffff8117b88a>] do_sync_write+0x7a/0xb0
>> [ 2162.498612]  [<ffffffff8117c63e>] vfs_write+0xce/0x1e0
>> [ 2162.498617]  [<ffffffff8117bba0>] ? generic_file_llseek_size+0x110/0x110
>> [ 2162.498621]  [<ffffffff8117cb22>] SyS_write+0x52/0xa0
>> [ 2162.498627]  [<ffffffff816c7a02>] system_call_fastpath+0x16/0x1b
>> [ 2162.498630] INFO: task kworker/0:1:2293 blocked for more than 120 seconds.
>> [ 2162.498635] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [ 2162.498639] kworker/0:1     D ffffffff81813a60     0  2293      2 0x00000000
>> [ 2162.498645] Workqueue: events write_dirty_finish
>> [ 2162.498647]  ffff882f8a05bb30 0000000000000046 ffffffff81511b8c
>> ffffffff815137c0
>> [ 2162.498652]  ffff882f8a163320 ffff882f8a05bfd8 ffff882f8a05bfd8
>> ffff882f8a05bfd8
>> [ 2162.498656]  ffffffff81c10440 ffff882f8a163320 ffffffff815117a1
>> ffff882f8a163320
>> [ 2162.498661] Call Trace:
>> [ 2162.498665]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
>> [ 2162.498669]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>> [ 2162.498674]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>> [ 2162.498679]  [<ffffffff816be299>] schedule+0x29/0x70
>> [ 2162.498684]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>> [ 2162.498689]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
>> [ 2162.498693]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>> [ 2162.498698]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>> [ 2162.498702]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>> [ 2162.498707]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>> [ 2162.498712]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>> [ 2162.498716]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>> [ 2162.498721]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>> [ 2162.498726]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>> [ 2162.498731]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>> [ 2162.498735]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>> [ 2162.498740]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>> [ 2162.498745]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>> [ 2162.498749]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>> [ 2162.498753]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2162.498758]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>> [ 2162.498762]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2162.498766] INFO: task kworker/0:3:2534 blocked for more than 120 seconds.
>> [ 2162.498772] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [ 2162.498777] kworker/0:3     D ffffffff81813a60     0  2534      2 0x00000000
>> [ 2162.498783] Workqueue: events write_dirty_finish
>> [ 2162.498785]  ffff88237d0bdb30 0000000000000046 ffffffff81511b8c
>> ffffffff815137c0
>> [ 2162.498790]  ffff882fa66fe640 ffff88237d0bdfd8 ffff88237d0bdfd8
>> ffff88237d0bdfd8
>> [ 2162.498794]  ffffffff81c10440 ffff882fa66fe640 ffffffff815117a1
>> ffff882fa66fe640
>> [ 2162.498799] Call Trace:
>> [ 2162.498803]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
>> [ 2162.498807]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>> [ 2162.498812]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>> [ 2162.498817]  [<ffffffff816be299>] schedule+0x29/0x70
>> [ 2162.498822]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>> [ 2162.498826]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
>> [ 2162.498831]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>> [ 2162.498836]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>> [ 2162.498840]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>> [ 2162.498845]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>> [ 2162.498849]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>> [ 2162.498854]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>> [ 2162.498859]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>> [ 2162.498864]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>> [ 2162.498868]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>> [ 2162.498873]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>> [ 2162.498878]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>> [ 2162.498882]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>> [ 2162.498886]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>> [ 2162.498890]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2162.498896]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>> [ 2162.498900]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2162.498903] INFO: task kworker/0:4:2537 blocked for more than 120 seconds.
>> [ 2162.498908] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [ 2162.498914] kworker/0:4     D 0000000000000000     0  2537      2 0x00000000
>> [ 2162.498920] Workqueue: events write_dirty_finish
>> [ 2162.498922]  ffff882f76d8fb30 0000000000000046 ffffffff81511b8c
>> ffffffff815137c0
>> [ 2162.498926]  ffff882f88b30000 ffff882f76d8ffd8 ffff882f76d8ffd8
>> ffff882f76d8ffd8
>> [ 2162.498931]  ffff882f78beb320 ffff882f88b30000 ffffffff815117a1
>> ffff882f88b30000
>> [ 2162.498935] Call Trace:
>> [ 2162.498940]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
>> [ 2162.498944]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>> [ 2162.498948]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>> [ 2162.498953]  [<ffffffff816be299>] schedule+0x29/0x70
>> [ 2162.498958]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>> [ 2162.498963]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
>> [ 2162.498968]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>> [ 2162.498972]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>> [ 2162.498977]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>> [ 2162.498981]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>> [ 2162.498986]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>> [ 2162.498990]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>> [ 2162.498995]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>> [ 2162.499000]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>> [ 2162.499004]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>> [ 2162.499009]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>> [ 2162.499014]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>> [ 2162.499019]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>> [ 2162.499023]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>> [ 2162.499027]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2162.499032]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>> [ 2162.499036]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2162.499039] INFO: task kworker/1:1:2544 blocked for more than 120 seconds.
>> [ 2162.499044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [ 2162.499050] kworker/1:1     D ffffffff81813d40     0  2544      2 0x00000000
>> [ 2162.499057] Workqueue: bcache bch_data_insert_keys
>> [ 2162.499059]  ffff882fa677b3f8 0000000000000046 ffff882fa677b418
>> ffff882f8a709000
>> [ 2162.499063]  ffff882f88b33320 ffff882fa677bfd8 ffff882fa677bfd8
>> ffff882fa677bfd8
>> [ 2162.499068]  ffff882fa6471990 ffff882f88b33320 ffff882f88b33320
>> ffff8801fad20d98
>> [ 2162.499072] Call Trace:
>> [ 2162.499077]  [<ffffffff816be299>] schedule+0x29/0x70
>> [ 2162.499082]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
>> [ 2162.499087]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
>> [ 2162.499092]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>> [ 2162.499096]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
>> [ 2162.499102]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
>> [ 2162.499109]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
>> [ 2162.499113]  [<ffffffff81076828>] ? resched_task+0x68/0x70
>> [ 2162.499118]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
>> [ 2162.499126]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
>> [ 2162.499131]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
>> [ 2162.499137]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
>> [ 2162.499143]  [<ffffffff81126e9c>] __alloc_pages_nodemask+0x60c/0x9b0
>> [ 2162.499149]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
>> [ 2162.499155]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
>> [ 2162.499159]  [<ffffffff8150ebb3>] mca_data_alloc+0x73/0x1d0
>> [ 2162.499163]  [<ffffffff81510f37>] mca_alloc+0x277/0x470
>> [ 2162.499168]  [<ffffffff81511d1c>] bch_btree_node_alloc+0x8c/0x1c0
>> [ 2162.499173]  [<ffffffff81513020>] btree_split+0x110/0x5c0
>> [ 2162.499179]  [<ffffffff81457d43>] ? scsi_pool_alloc_command+0x33/0x80
>> [ 2162.499184]  [<ffffffff81515fc7>] ? bch_keylist_pop_front+0x47/0x50
>> [ 2162.499188]  [<ffffffff8150fed6>] ? bch_btree_insert_keys+0x56/0x250
>> [ 2162.499193]  [<ffffffff81082def>] ? update_group_power+0x13f/0x220
>> [ 2162.499198]  [<ffffffff81513582>] bch_btree_insert_node+0xb2/0x2f0
>> [ 2162.499202]  [<ffffffff815137e8>] btree_insert_fn+0x28/0x50
>> [ 2162.499207]  [<ffffffff81511b8c>] bch_btree_map_nodes_recurse+0x6c/0x170
>> [ 2162.499211]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>> [ 2162.499216]  [<ffffffff816bcc26>] ? down_write+0x16/0x40
>> [ 2162.499221]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>> [ 2162.499225]  [<ffffffff81511c30>] bch_btree_map_nodes_recurse+0x110/0x170
>> [ 2162.499230]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>> [ 2162.499235]  [<ffffffff811b51ea>] ? dio_bio_end_io+0x5a/0x90
>> [ 2162.499239]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>> [ 2162.499244]  [<ffffffff81514dce>] __bch_btree_map_nodes+0x13e/0x1c0
>> [ 2162.499248]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>> [ 2162.499254]  [<ffffffff8151ac6f>] ? bch_journal+0x42f/0x4b0
>> [ 2162.499258]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>> [ 2162.499264]  [<ffffffff8151e1be>] bch_data_insert_keys+0x3e/0x160
>> [ 2162.499268]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>> [ 2162.499273]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>> [ 2162.499278]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>> [ 2162.499282]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>> [ 2162.499286]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2162.499292]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>> [ 2162.499296]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2162.499299] INFO: task kworker/0:5:2552 blocked for more than 120 seconds.
>> [ 2162.499304] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [ 2162.499310] kworker/0:5     D 0000000000000000     0  2552      2 0x00000000
>> [ 2162.499411] Workqueue: events write_dirty_finish
>> [ 2162.499413]  ffff88226314db30 0000000000000046 ffffffff81511b8c
>> ffffffff815137c0
>> [ 2162.499418]  ffff882fa6781990 ffff88226314dfd8 ffff88226314dfd8
>> ffff88226314dfd8
>> [ 2162.499422]  ffff882fa6a84cb0 ffff882fa6781990 ffffffff815117a1
>> ffff882fa6781990
>> [ 2162.499427] Call Trace:
>> [ 2162.499431]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
>> [ 2162.499435]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>> [ 2162.499440]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>> [ 2162.499445]  [<ffffffff816be299>] schedule+0x29/0x70
>> [ 2162.499450]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>> [ 2162.499455]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
>> [ 2162.499460]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>> [ 2162.499464]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>> [ 2162.499468]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>> [ 2162.499473]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>> [ 2162.499478]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>> [ 2162.499483]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>> [ 2162.499488]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>> [ 2162.499492]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>> [ 2162.499497]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>> [ 2162.499502]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>> [ 2162.499507]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>> [ 2162.499511]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>> [ 2162.499515]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>> [ 2162.499520]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>> [ 2162.499524]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>
>>
>>
>>>> [  930.169234] bcache: bch_cached_dev_attach() Caching sda3 as bcache0
>>>> on set 06496904-10ca-489a-ae75-68c6a07d3db1
>>>> [ 2522.956188] INFO: task bcache_writebac:2058 blocked for more than
>>>> 120 seconds.
>>>> [ 2522.956199] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>> disables this message.
>>>> [ 2522.956205] bcache_writebac D ffffffff81813a60     0  2058      2 0x00000000
>>>> [ 2522.956212]  ffff881c39c1fcd8 0000000000000046 0000000000000001
>>>> 0000000000000001
>>>> [ 2522.956218]  ffff882f8b56b320 ffff881c39c1ffd8 ffff881c39c1ffd8
>>>> ffff881c39c1ffd8
>>>> [ 2522.956222]  ffff882fa6aeb320 ffff882f8b56b320 ffff882f8b56b320
>>>> ffff882f8b56b320
>>>> [ 2522.956227] Call Trace:
>>>> [ 2522.956241]  [<ffffffff816be299>] schedule+0x29/0x70
>>>> [ 2522.956247]  [<ffffffff816befb5>] rwsem_down_write_failed+0xf5/0x1a0
>>>> [ 2522.956255]  [<ffffffff81332c93>] call_rwsem_down_write_failed+0x13/0x20
>>>> [ 2522.956261]  [<ffffffff816bcc41>] ? down_write+0x31/0x40
>>>> [ 2522.956269]  [<ffffffff8151b782>] bch_writeback_thread+0x62/0x7f0
>>>> [ 2522.956278]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
>>>> [ 2522.956284]  [<ffffffff8151b720>] ? write_dirty+0xc0/0xc0
>>>> [ 2522.956290]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>> [ 2522.956294]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2522.956302]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>> [ 2522.956306]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2522.956313] INFO: task kworker/u64:1:2433 blocked for more than 120 seconds.
>>>> [ 2522.956317] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>> disables this message.
>>>> [ 2522.956322] kworker/u64:1   D ffffffff81813a60     0  2433      2 0x00000000
>>>> [ 2522.956332] Workqueue: writeback bdi_writeback_workfn (flush-252:0)
>>>> [ 2522.956336]  ffff882f899d1560 0000000000000046 ffffffff81120635
>>>> 0000000000000100
>>>> [ 2522.956340]  ffff88219b488000 ffff882f899d1fd8 ffff882f899d1fd8
>>>> ffff882f899d1fd8
>>>> [ 2522.956345]  ffff882fa6aeb320 ffff88219b488000 ffffffff8131ad55
>>>> ffff88219b488000
>>>> [ 2522.956349] Call Trace:
>>>> [ 2522.956359]  [<ffffffff81120635>] ? mempool_alloc_slab+0x15/0x20
>>>> [ 2522.956367]  [<ffffffff8131ad55>] ? blk_throtl_bio+0x3b5/0x540
>>>> [ 2522.956372]  [<ffffffff816be299>] schedule+0x29/0x70
>>>> [ 2522.956378]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>> [ 2522.956383]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
>>>> [ 2522.956388]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>> [ 2522.956394]  [<ffffffff8151fe32>] cached_dev_make_request+0x752/0xf00
>>>> [ 2522.956400]  [<ffffffff81301af8>] ? generic_make_request_checks+0x1e8/0x3a0
>>>> [ 2522.956405]  [<ffffffff81301d7a>] generic_make_request+0xca/0x100
>>>> [ 2522.956409]  [<ffffffff81301e29>] submit_bio+0x79/0x160
>>>> [ 2522.956417]  [<ffffffff811b1710>] ? bio_alloc_bioset+0xa0/0x1d0
>>>> [ 2522.956421]  [<ffffffff811abd3f>] _submit_bh+0x13f/0x200
>>>> [ 2522.956425]  [<ffffffff811abe10>] submit_bh+0x10/0x20
>>>> [ 2522.956430]  [<ffffffff811af6b8>] __block_write_full_page+0x1d8/0x360
>>>> [ 2522.956439]  [<ffffffff8145fc0b>] ? scsi_request_fn+0xbb/0x530
>>>> [ 2522.956444]  [<ffffffff811ad3c0>] ? end_buffer_async_read+0x130/0x130
>>>> [ 2522.956449]  [<ffffffff811b2c40>] ? I_BDEV+0x10/0x10
>>>> [ 2522.956454]  [<ffffffff811b2c40>] ? I_BDEV+0x10/0x10
>>>> [ 2522.956459]  [<ffffffff811af90a>] block_write_full_page_endio+0xca/0x100
>>>> [ 2522.956464]  [<ffffffff811af955>] block_write_full_page+0x15/0x20
>>>> [ 2522.956469]  [<ffffffff811b3608>] blkdev_writepage+0x18/0x20
>>>> [ 2522.956473]  [<ffffffff81127d37>] __writepage+0x17/0x40
>>>> [ 2522.956477]  [<ffffffff8112820e>] write_cache_pages+0x20e/0x460
>>>> [ 2522.956481]  [<ffffffff81127d20>] ? set_page_dirty_lock+0x60/0x60
>>>> [ 2522.956486]  [<ffffffff811ac251>] ? __set_page_dirty+0x71/0xc0
>>>> [ 2522.956490]  [<ffffffff811284aa>] generic_writepages+0x4a/0x70
>>>> [ 2522.956495]  [<ffffffff81129cd0>] do_writepages+0x20/0x40
>>>> [ 2522.956501]  [<ffffffff811a36f5>] __writeback_single_inode+0x45/0x280
>>>> [ 2522.956507]  [<ffffffff811269e2>] ? __alloc_pages_nodemask+0x152/0x9b0
>>>> [ 2522.956513]  [<ffffffff811a495d>] writeback_sb_inodes+0x19d/0x3c0
>>>> [ 2522.956518]  [<ffffffff811a4c1e>] __writeback_inodes_wb+0x9e/0xd0
>>>> [ 2522.956523]  [<ffffffff811a4ecb>] wb_writeback+0x27b/0x320
>>>> [ 2522.956529]  [<ffffffff81196228>] ? get_nr_dirty_inodes+0x58/0x80
>>>> [ 2522.956534]  [<ffffffff811a500f>] wb_check_old_data_flush+0x9f/0xb0
>>>> [ 2522.956539]  [<ffffffff811a67e1>] wb_do_writeback+0x151/0x1d0
>>>> [ 2522.956547]  [<ffffffff8106539f>] ? set_worker_desc+0x6f/0x80
>>>> [ 2522.956552]  [<ffffffff811a68da>] bdi_writeback_workfn+0x7a/0x200
>>>> [ 2522.956557]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>> [ 2522.956562]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>> [ 2522.956567]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>>>> [ 2522.956571]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>> [ 2522.956575]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2522.956581]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>> [ 2522.956585]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2522.956588] INFO: task iozone:2435 blocked for more than 120 seconds.
>>>> [ 2522.956593] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>> disables this message.
>>>> [ 2522.956597] iozone          D 0000000000000001     0  2435   1545 0x00000000
>>>> [ 2522.956602]  ffff882a75671978 0000000000000082 ffff882a75671938
>>>> ffffffff81301d7a
>>>> [ 2522.956606]  ffff882f88dc1990 ffff882a75671fd8 ffff882a75671fd8
>>>> ffff882a75671fd8
>>>> [ 2522.956611]  ffff882f8ec91990 ffff882f88dc1990 ffff882a75671978
>>>> ffff882fbf2539f8
>>>> [ 2522.956615] Call Trace:
>>>> [ 2522.956619]  [<ffffffff81301d7a>] ? generic_make_request+0xca/0x100
>>>> [ 2522.956625]  [<ffffffff816be299>] schedule+0x29/0x70
>>>> [ 2522.956630]  [<ffffffff816be36f>] io_schedule+0x8f/0xd0
>>>> [ 2522.956636]  [<ffffffff811b797c>] do_blockdev_direct_IO+0x1a7c/0x1fb0
>>>> [ 2522.956645]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
>>>> [ 2522.956652]  [<ffffffff811b7f05>] __blockdev_direct_IO+0x55/0x60
>>>> [ 2522.956658]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
>>>> [ 2522.956662]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>> [ 2522.956669]  [<ffffffffa0263349>] ext2_direct_IO+0x79/0xe0 [ext2]
>>>> [ 2522.956674]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
>>>> [ 2522.956680]  [<ffffffff8104ade6>] ? current_fs_time+0x16/0x60
>>>> [ 2522.956685]  [<ffffffff8111f126>] generic_file_direct_write+0xc6/0x180
>>>> [ 2522.956690]  [<ffffffff8111f4bd>] __generic_file_aio_write+0x2dd/0x3b0
>>>> [ 2522.956696]  [<ffffffff8111f5f9>] generic_file_aio_write+0x69/0xd0
>>>> [ 2522.956702]  [<ffffffff8117b88a>] do_sync_write+0x7a/0xb0
>>>> [ 2522.956706]  [<ffffffff811bb108>] ? fsnotify+0x1f8/0x2b0
>>>> [ 2522.956712]  [<ffffffff8117c63e>] vfs_write+0xce/0x1e0
>>>> [ 2522.956716]  [<ffffffff8117cb22>] SyS_write+0x52/0xa0
>>>> [ 2522.956722]  [<ffffffff816c7a02>] system_call_fastpath+0x16/0x1b
>>>> [ 2522.956726] INFO: task kworker/2:1:2470 blocked for more than 120 seconds.
>>>> [ 2522.956730] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>> disables this message.
>>>> [ 2522.956735] kworker/2:1     D ffffffff81813d40     0  2470      2 0x00000000
>>>> [ 2522.956743] Workqueue: bcache bch_data_insert_keys
>>>> [ 2522.956745]  ffff882a5b0c9628 0000000000000046 ffff882a5b0c9648
>>>> 0000004000000000
>>>> [ 2522.956749]  ffff882f8ec91990 ffff882a5b0c9fd8 ffff882a5b0c9fd8
>>>> ffff882a5b0c9fd8
>>>> [ 2522.956754]  ffff882fa6489990 ffff882f8ec91990 ffff882f8ec91990
>>>> ffff8828dba00d98
>>>> [ 2522.956758] Call Trace:
>>>> [ 2522.956763]  [<ffffffff816be299>] schedule+0x29/0x70
>>>> [ 2522.956769]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
>>>> [ 2522.956774]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
>>>> [ 2522.956779]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>>> [ 2522.956783]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
>>>> [ 2522.956791]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
>>>> [ 2522.956799]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
>>>> [ 2522.956805]  [<ffffffff81076828>] ? resched_task+0x68/0x70
>>>> [ 2522.956810]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
>>>> [ 2522.956816]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
>>>> [ 2522.956822]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
>>>> [ 2522.956827]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
>>>> [ 2522.956833]  [<ffffffff81126e9c>] __alloc_pages_nodemask+0x60c/0x9b0
>>>> [ 2522.956843]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
>>>> [ 2522.956848]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
>>>> [ 2522.956852]  [<ffffffff8150ebb3>] mca_data_alloc+0x73/0x1d0
>>>> [ 2522.956857]  [<ffffffff8150ee5a>] mca_bucket_alloc+0x14a/0x1f0
>>>> [ 2522.956861]  [<ffffffff81511020>] mca_alloc+0x360/0x470
>>>> [ 2522.956866]  [<ffffffff81517031>] ? __bch_bset_search+0x1d1/0x480
>>>> [ 2522.956871]  [<ffffffff8151183e>] bch_btree_node_get+0x10e/0x280
>>>> [ 2522.956875]  [<ffffffff81511c02>] bch_btree_map_nodes_recurse+0xe2/0x170
>>>> [ 2522.956880]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>> [ 2522.956886]  [<ffffffff811b51ea>] ? dio_bio_end_io+0x5a/0x90
>>>> [ 2522.956890]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>> [ 2522.956895]  [<ffffffff81514dce>] __bch_btree_map_nodes+0x13e/0x1c0
>>>> [ 2522.956899]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>> [ 2522.956905]  [<ffffffff8151ac6f>] ? bch_journal+0x42f/0x4b0
>>>> [ 2522.956909]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>> [ 2522.956915]  [<ffffffff8151e1be>] bch_data_insert_keys+0x3e/0x160
>>>> [ 2522.956920]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>> [ 2522.956925]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>> [ 2522.956930]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>>>> [ 2522.956933]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>> [ 2522.956938]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2522.956943]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>> [ 2522.956947]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2522.956951] INFO: task kworker/3:2:2471 blocked for more than 120 seconds.
>>>> [ 2522.956955] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>> disables this message.
>>>> [ 2522.956959] kworker/3:2     D ffffffff81813a60     0  2471      2 0x00000000
>>>> [ 2522.956966] Workqueue: events update_writeback_rate
>>>> [ 2522.956968]  ffff882a5d125cd0 0000000000000046 0000000000000003
>>>> ffff882a5d125d68
>>>> [ 2522.956972]  ffff882f8ec94cb0 ffff882a5d125fd8 ffff882a5d125fd8
>>>> ffff882a5d125fd8
>>>> [ 2522.956977]  ffff882fa6aeb320 ffff882f8ec94cb0 0000000200000003
>>>> ffff882f8ec94cb0
>>>> [ 2522.956981] Call Trace:
>>>> [ 2522.956987]  [<ffffffff816be299>] schedule+0x29/0x70
>>>> [ 2522.956992]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>> [ 2522.956997]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
>>>> [ 2522.957002]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>> [ 2522.957007]  [<ffffffff8151b010>] update_writeback_rate+0x30/0x230
>>>> [ 2522.957011]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>> [ 2522.957017]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>> [ 2522.957021]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>>>> [ 2522.957025]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>> [ 2522.957030]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2522.957035]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>> [ 2522.957039]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2522.957048] INFO: task kworker/u64:2:2540 blocked for more than 120 seconds.
>>>> [ 2522.957052] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>> disables this message.
>>>> [ 2522.957056] kworker/u64:2   D ffffffff81813a60     0  2540      2 0x00000000
>>>> [ 2522.957062] Workqueue: bch_btree_io btree_node_write_work
>>>> [ 2522.957064]  ffff882a4fdc9778 0000000000000046 ffff882a4fdc9738
>>>> ffff882f86e3be58
>>>> [ 2522.957069]  ffff882fa4bc1990 ffff882a4fdc9fd8 ffff882a4fdc9fd8
>>>> ffff882a4fdc9fd8
>>>> [ 2522.957073]  ffff882fa6aeb320 ffff882fa4bc1990 ffff882f86e3be00
>>>> ffff8828dba00d98
>>>> [ 2522.957078] Call Trace:
>>>> [ 2522.957083]  [<ffffffff816be299>] schedule+0x29/0x70
>>>> [ 2522.957088]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
>>>> [ 2522.957092]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
>>>> [ 2522.957102]  [<ffffffff81484090>] ? ata_scsiop_mode_sense+0x380/0x380
>>>> [ 2522.957107]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>>> [ 2522.957111]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
>>>> [ 2522.957116]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
>>>> [ 2522.957121]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
>>>> [ 2522.957126]  [<ffffffff81076828>] ? resched_task+0x68/0x70
>>>> [ 2522.957130]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
>>>> [ 2522.957135]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
>>>> [ 2522.957140]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
>>>> [ 2522.957146]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
>>>> [ 2522.957152]  [<ffffffff81126e9c>] __alloc_pages_nodemask+0x60c/0x9b0
>>>> [ 2522.957158]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
>>>> [ 2522.957163]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
>>>> [ 2522.957168]  [<ffffffff81517fc8>] __btree_sort+0x48/0x230
>>>> [ 2522.957173]  [<ffffffff8151765c>] ? __bch_btree_iter_init+0x7c/0xc0
>>>> [ 2522.957178]  [<ffffffff81518301>] bch_btree_sort_partial+0x101/0x120
>>>> [ 2522.957182]  [<ffffffff8150f000>] ? __btree_node_write_done+0x100/0x100
>>>> [ 2522.957187]  [<ffffffff81518468>] bch_btree_sort_lazy+0x68/0x90
>>>> [ 2522.957191]  [<ffffffff815109ba>] bch_btree_node_write+0x36a/0x4a0
>>>> [ 2522.957196]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
>>>> [ 2522.957201]  [<ffffffff8106212c>] ? pwq_activate_delayed_work+0x4c/0xb0
>>>> [ 2522.957205]  [<ffffffff81511477>] btree_node_write_work+0x57/0x80
>>>> [ 2522.957210]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>> [ 2522.957215]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>> [ 2522.957219]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>>>> [ 2522.957223]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>> [ 2522.957228]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2522.957233]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>> [ 2522.957237]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>
>>>>
>>>> I can easily reproduce this, please let me know if you require any
>>>> info/test any patch.
>>>>
>>>> Thanks,
>>>> Neo
>>>>
>>>> On Mon, Aug 26, 2013 at 12:21 PM, Stefan Priebe <s.priebe@profihost.ag> wrote:
>>>>> Hi Kent,
>>>>>
>>>>> a new one under 3.10:
>>>>>
>>>>> 2013-08-26 21:05:30     INFO: task ceph-osd:8939 blocked for more than 120
>>>>> seconds.
>>>>> 2013-08-26 21:05:30     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> 2013-08-26 21:05:30     ceph-osd D ffffffff8160f760 0 8939 1 0x00000000
>>>>> 2013-08-26 21:05:30     ffff880bd9a61820 0000000000000086 ffff880bd9a61fd8
>>>>> 0000000000012c80
>>>>> 2013-08-26 21:05:30     ffff880bd9a60010 0000000000012c80 0000000000012c80
>>>>> 0000000000012c80
>>>>> 2013-08-26 21:05:30     ffff880bd9a61fd8 0000000000012c80 ffff880bd7abb1c0
>>>>> ffff880c48dee380
>>>>> 2013-08-26 21:05:29     [<ffffffff811e8128>] xfs_vm_readpages+0x18/0x20
>>>>> 2013-08-26 21:05:29     [<ffffffff810f1da3>] read_pages+0x43/0x100
>>>>> 2013-08-26 21:05:29     [<ffffffff810e64df>] ? __page_cache_alloc+0x9f/0xc0
>>>>> 2013-08-26 21:05:29     [<ffffffff810f1fab>]
>>>>> __do_page_cache_readahead+0x14b/0x160
>>>>> 2013-08-26 21:05:29     [<ffffffff810f1fdc>] ra_submit+0x1c/0x20
>>>>> 2013-08-26 21:05:29     [<ffffffff810f22a5>] ondemand_readahead+0x115/0x240
>>>>> 2013-08-26 21:05:29     [<ffffffff810f249e>]
>>>>> page_cache_sync_readahead+0x2e/0x40
>>>>> 2013-08-26 21:05:29     [<ffffffff810e7a5e>] T.1003+0x33e/0x430
>>>>> 2013-08-26 21:05:29     [<ffffffff810e7c30>]
>>>>> generic_file_aio_read+0xe0/0x220
>>>>> 2013-08-26 21:05:29     [<ffffffff811efd0a>] xfs_file_aio_read+0x15a/0x2a0
>>>>> 2013-08-26 21:05:29     [<ffffffff8114095a>] do_sync_read+0x7a/0xb0
>>>>> 2013-08-26 21:05:29     [<ffffffff81140df1>] vfs_read+0xb1/0x130
>>>>> 2013-08-26 21:05:29     [<ffffffff81141757>] SyS_pread64+0x97/0xa0
>>>>> 2013-08-26 21:05:29     [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b
>>>>> 2013-08-26 21:05:29     INFO: task ceph-osd:8896 blocked for more than 120
>>>>> seconds.
>>>>> 2013-08-26 21:05:29     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> 2013-08-26 21:05:29     ceph-osd D ffffffff8160f760 0 8896 1 0x00000000
>>>>> 2013-08-26 21:05:29     ffff880bd9a9dc28 0000000000000086 ffff880bd9a9dfd8
>>>>> 0000000000012c80
>>>>> 2013-08-26 21:05:29     ffff880bd9a9c010 0000000000012c80 0000000000012c80
>>>>> 0000000000012c80
>>>>> 2013-08-26 21:05:29     ffff880bd9a9dfd8 0000000000012c80 ffff880bd79298e0
>>>>> ffff880c48dd6380
>>>>> 2013-08-26 21:05:29     Call Trace:
>>>>> 2013-08-26 21:05:29     [<ffffffff810e60f0>] ? __lock_page+0x70/0x70
>>>>> 2013-08-26 21:05:28     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> 2013-08-26 21:05:28     ceph-osd D ffffffff8160f760 0 8798 1 0x00000000
>>>>> 2013-08-26 21:05:28     ffff880bd998b740 0000000000000086 ffff880bd998bfd8
>>>>> 0000000000012c80
>>>>> 2013-08-26 21:05:28     ffff880bd998a010 0000000000012c80 0000000000012c80
>>>>> 0000000000012c80
>>>>> 2013-08-26 21:05:28     ffff880bd998bfd8 0000000000012c80 ffff880bd99c0000
>>>>> ffff880c48dd6380
>>>>> 2013-08-26 21:05:28     Call Trace:
>>>>> 2013-08-26 21:05:28     [<ffffffff81524874>] schedule+0x24/0x70
>>>>> 2013-08-26 21:05:28     [<ffffffff81525d7d>]
>>>>> rwsem_down_read_failed+0x9d/0xe5
>>>>> 2013-08-26 21:05:28     [<ffffffff812aa0a4>]
>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>> 2013-08-26 21:05:28     [<ffffffff81523a42>] ? down_read+0x12/0x20
>>>>> 2013-08-26 21:05:28     [<ffffffffa01ead62>] btree_read_async+0xa2/0x1a0
>>>>> [bcache]
>>>>> 2013-08-26 21:05:28     [<ffffffffa01f7b73>] closure_queue+0x43/0x60
>>>>> [bcache]
>>>>> 2013-08-26 21:05:28     [<ffffffffa01eb903>] T.1050+0x63/0x70 [bcache]
>>>>> 2013-08-26 21:05:28     [<ffffffffa01ed6ea>]
>>>>> cached_dev_make_request+0x20a/0x350 [bcache]
>>>>> 2013-08-26 21:05:28     [<ffffffff81281952>] generic_make_request+0xc2/0x100
>>>>> 2013-08-26 21:05:28     [<ffffffff812819f7>] submit_bio+0x67/0x130
>>>>> 2013-08-26 21:05:28     [<ffffffff8117b240>] do_mpage_readpage+0x2a0/0x660
>>>>> 2013-08-26 21:05:28     [<ffffffff81101cee>] ?
>>>>> __inc_zone_page_state+0x2e/0x30
>>>>> 2013-08-26 21:05:28     [<ffffffff810e6581>] ?
>>>>> add_to_page_cache_locked+0x81/0x110
>>>>> 2013-08-26 21:05:28     [<ffffffff8117b76a>] mpage_readpages+0xfa/0x150
>>>>> 2013-08-26 21:05:28     [<ffffffff811e90c0>] ?
>>>>> xfs_get_blocks_direct+0x20/0x20
>>>>> 2013-08-26 21:05:28     [<ffffffff811e90c0>] ?
>>>>> xfs_get_blocks_direct+0x20/0x20
>>>>> 2013-08-26 21:05:28     [<ffffffff810eeda3>] ?
>>>>> __alloc_pages_nodemask+0x123/0x240
>>>>> 2013-08-26 21:05:27     [<ffffffff8115bf93>] ? __close_fd+0x43/0x80
>>>>> 2013-08-26 21:05:27     [<ffffffff8116dabe>] SyS_fdatasync+0xe/0x20
>>>>> 2013-08-26 21:05:27     [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b
>>>>> 2013-08-26 21:05:27     INFO: task ceph-osd:8746 blocked for more than 120
>>>>> seconds.
>>>>> 2013-08-26 21:05:27     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> 2013-08-26 21:05:27     ceph-osd D ffffffff8160f760 0 8746 1 0x00000000
>>>>> 2013-08-26 21:05:27     ffff880c2dde1d38 0000000000000086 ffff880c2dde1fd8
>>>>> 0000000000012c80
>>>>> 2013-08-26 21:05:27     ffff880c2dde0010 0000000000012c80 0000000000012c80
>>>>> 0000000000012c80
>>>>> 2013-08-26 21:05:27     ffff880c2dde1fd8 0000000000012c80 ffff880c239eb1c0
>>>>> ffff880c48dd6380
>>>>> 2013-08-26 21:05:27     Call Trace:
>>>>> 2013-08-26 21:05:27     [<ffffffff81524874>] schedule+0x24/0x70
>>>>> 2013-08-26 21:05:27     [<ffffffff81522bbd>] schedule_timeout+0x16d/0x200
>>>>> 2013-08-26 21:05:27     [<ffffffff81068eba>] ?
>>>>> __queue_delayed_work+0xaa/0x1a0
>>>>> 2013-08-26 21:05:27     [<ffffffff81068d19>] ?
>>>>> try_to_grab_pending+0x109/0x190
>>>>> 2013-08-26 21:05:27     [<ffffffff815251e5>] wait_for_completion+0x95/0x110
>>>>> 2013-08-26 21:05:27     [<ffffffff8107fd00>] ? try_to_wake_up+0x2a0/0x2a0
>>>>> 2013-08-26 21:05:27     [<ffffffff81167797>] ? bdi_queue_work+0x77/0xc0
>>>>> 2013-08-26 21:05:27     [<ffffffff81167863>]
>>>>> writeback_inodes_sb_nr+0x83/0xb0
>>>>> 2013-08-26 21:05:27     [<ffffffff811678ea>] writeback_inodes_sb+0x5a/0x70
>>>>> 2013-08-26 21:05:27     [<ffffffff8116dd0a>] __sync_filesystem+0x4a/0x50
>>>>> 2013-08-26 21:05:27     [<ffffffff8116dd42>] sync_filesystem+0x32/0x60
>>>>> 2013-08-26 21:05:27     [<ffffffff8116ddc0>] SyS_syncfs+0x50/0x90
>>>>> 2013-08-26 21:05:27     [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b
>>>>> 2013-08-26 21:05:27     INFO: task ceph-osd:8798 blocked for more than 120
>>>>> seconds.
>>>>>
>>>>> Stefan
>>>>>
>>>>> Am 22.08.2013 09:32, schrieb Stefan Priebe - Profihost AG:
>>>>>
>>>>>> great!
>>>>>>
>>>>>> Everything seems to work fine now! Except read_dirty always going to
>>>>>> negative values after a reboot.
>>>>>>
>>>>>> Stefan
>>>>>>
>>>>>> Am 22.08.2013 08:02, schrieb Kent Overstreet:
>>>>>>>
>>>>>>> On Thu, Aug 22, 2013 at 07:59:04AM +0200, Stefan Priebe wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>> schedule_timeout() is not the same as
>>>>>>>>> schedule_timeout_interruptible().
>>>>>>>>
>>>>>>>>
>>>>>>>> just search and replace? So i can try on my own.
>>>>>>>
>>>>>>>
>>>>>>> The one in read_dirty(), line ~330
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-bcache"
>>>>>>> in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-28 18:47                                   ` kernel neophyte
@ 2013-08-28 19:20                                     ` Stefan Priebe
  2013-08-28 20:12                                       ` kernel neophyte
  0 siblings, 1 reply; 28+ messages in thread
From: Stefan Priebe @ 2013-08-28 19:20 UTC (permalink / raw)
  To: kernel neophyte
  Cc: Kent Overstreet, Jens Axboe, linux-kernel@vger.kernel.org,
	linux-bcache@vger.kernel.org

Am 28.08.2013 20:47, schrieb kernel neophyte:
> On Wed, Aug 28, 2013 at 11:38 AM, Stefan Priebe - Profihost AG
> <s.priebe@profihost.ag> wrote:
>> I don't had one for a few days. Which kernel so you use?
>
> 3.10 kernel with all of kent's stable patches and perf patches..

Which exact 3.10 version? Which patches on top?

> and all defaults except for:
>
> echo writeback > /sys/block/bcache0/bcache/cache_mode
> echo 0 > /sys/block/bcache0/bcache/sequential_cutoff
> echo 0 > /sys/fs/bcache/1e26e5ca-5cb9-4c5a-ad7e-3c787524a9dc/congested_read_threshold_us
> echo 0 > /sys/fs/bcache/1e26e5ca-5cb9-4c5a-ad7e-3c787524a9dc/congested_write_threshold_us
>
> -Neo
>
>>
>> Stefan
>>
>> This mail was sent with my iPhone.
>>
>> Am 28.08.2013 um 20:12 schrieb kernel neophyte <neophyte.hacker001@gmail.com>:
>>
>>> On Tue, Aug 27, 2013 at 11:05 PM, Kent Overstreet <kmo@daterainc.com> wrote:
>>>> On Tue, Aug 27, 2013 at 08:37:54PM -0700, kernel neophyte wrote:
>>>>> Hi Kent,
>>>>>
>>>>> I am still seeing deadlock:
>>>>
>>>> Seeing blk_throtl_bio (what asshole misspells words in their function
>>>> names, so when you go to grep for them you grep for the wrong thing?) in
>>>> the backtrace - is that what it takes to hit it?
>>>
>>> I don't think so,
>>> I had another deadlock:
>>>
>>> [ 1179.193990] bio: create slab <bio-2> at 2
>>> [ 1179.194491] bcache: run_cache_set() invalidating existing data
>>> [ 1179.212308] bcache: register_cache() registered cache device rxd1
>>> [ 1203.846220] bcache: register_bdev() registered backing device sda3
>>> [ 1227.780088] bcache: bch_cached_dev_attach() Caching sda3 as bcache0
>>> on set a079dda3-27fd-4b1c-a616-d46110bfae6c
>>> [ 2162.497950] INFO: task kworker/0:0:4 blocked for more than 120 seconds.
>>> [ 2162.497961] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>> disables this message.
>>> [ 2162.497967] kworker/0:0     D 0000000000000000     0     4      2 0x00000000
>>> [ 2162.497982] Workqueue: events write_dirty_finish
>>> [ 2162.497986]  ffff882fa6ac9b30 0000000000000046 ffffffff81511b8c
>>> ffffffff815137c0
>>> [ 2162.497992]  ffff882fa6a84cb0 ffff882fa6ac9fd8 ffff882fa6ac9fd8
>>> ffff882fa6ac9fd8
>>> [ 2162.497996]  ffff882f78139990 ffff882fa6a84cb0 ffffffff815117a1
>>> ffff882fa6a84cb0
>>> [ 2162.498001] Call Trace:
>>> [ 2162.498010]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
>>> [ 2162.498015]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>> [ 2162.498019]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>> [ 2162.498028]  [<ffffffff816be299>] schedule+0x29/0x70
>>> [ 2162.498034]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>> [ 2162.498043]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
>>> [ 2162.498048]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>> [ 2162.498052]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>> [ 2162.498057]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>> [ 2162.498066]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>> [ 2162.498072]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>> [ 2162.498076]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>> [ 2162.498082]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>> [ 2162.498087]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>> [ 2162.498093]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>> [ 2162.498100]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>> [ 2162.498105]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>> [ 2162.498110]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>>> [ 2162.498115]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>> [ 2162.498120]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2162.498127]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>> [ 2162.498131]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2162.498145] INFO: task bcache_allocato:1766 blocked for more than
>>> 120 seconds.
>>> [ 2162.498149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>> disables this message.
>>> [ 2162.498154] bcache_allocato D 0000000000000001     0  1766      2 0x00000000
>>> [ 2162.498158]  ffff8822630f3d88 0000000000000046 ffff8822630f3da8
>>> ffffffff810808ad
>>> [ 2162.498163]  ffff882f891de640 ffff8822630f3fd8 ffff8822630f3fd8
>>> ffff8822630f3fd8
>>> [ 2162.498167]  ffff882f81a91990 ffff882f891de640 ffff882f88b33320
>>> ffff8801fad20d98
>>> [ 2162.498172] Call Trace:
>>> [ 2162.498177]  [<ffffffff810808ad>] ? dequeue_task_fair+0x2cd/0x530
>>> [ 2162.498182]  [<ffffffff816be299>] schedule+0x29/0x70
>>> [ 2162.498187]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
>>> [ 2162.498192]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
>>> [ 2162.498197]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>> [ 2162.498202]  [<ffffffff8150cdbf>] bch_allocator_thread+0x10f/0xe20
>>> [ 2162.498207]  [<ffffffff8150ccb0>] ? bch_bucket_add_unused+0xe0/0xe0
>>> [ 2162.498211]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>> [ 2162.498215]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2162.498220]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>> [ 2162.498224]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2162.498228] INFO: task bcache_writebac:1771 blocked for more than
>>> 120 seconds.
>>> [ 2162.498233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>> disables this message.
>>> [ 2162.498237] bcache_writebac D 0000000000000000     0  1771      2 0x00000000
>>> [ 2162.498241]  ffff88197d2adc58 0000000000000046 ffff88197d2add48
>>> ffffffff811269e2
>>> [ 2162.498245]  ffff882f891d8000 ffff88197d2adfd8 ffff88197d2adfd8
>>> ffff88197d2adfd8
>>> [ 2162.498250]  ffff882f81a91990 ffff882f891d8000 ffff88197d2adc68
>>> ffff882f89550ba0
>>> [ 2162.498254] Call Trace:
>>> [ 2162.498264]  [<ffffffff811269e2>] ? __alloc_pages_nodemask+0x152/0x9b0
>>> [ 2162.498269]  [<ffffffff816be299>] schedule+0x29/0x70
>>> [ 2162.498274]  [<ffffffff816bc015>] schedule_timeout+0x1e5/0x250
>>> [ 2162.498280]  [<ffffffff810799bf>] ? try_to_wake_up+0x24f/0x2b0
>>> [ 2162.498285]  [<ffffffff816bcd09>] __down_common+0x8e/0xe3
>>> [ 2162.498293]  [<ffffffff81160600>] ? alloc_pages_current+0x90/0x170
>>> [ 2162.498298]  [<ffffffff816bcdd1>] __down+0x1d/0x1f
>>> [ 2162.498304]  [<ffffffff8106f351>] down+0x41/0x50
>>> [ 2162.498310]  [<ffffffff8151ba7d>] bch_writeback_thread+0x35d/0x7f0
>>> [ 2162.498315]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
>>> [ 2162.498321]  [<ffffffff8151b720>] ? write_dirty+0xc0/0xc0
>>> [ 2162.498325]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>> [ 2162.498329]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2162.498334]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>> [ 2162.498338]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2162.498342] INFO: task kworker/0:2:1789 blocked for more than 120 seconds.
>>> [ 2162.498346] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>> disables this message.
>>> [ 2162.498351] kworker/0:2     D ffffffff81813a60     0  1789      2 0x00000000
>>> [ 2162.498357] Workqueue: events write_dirty_finish
>>> [ 2162.498360]  ffff88277062fb30 0000000000000046 ffffffff81511b8c
>>> ffffffff815137c0
>>> [ 2162.498364]  ffff882fa6786640 ffff88277062ffd8 ffff88277062ffd8
>>> ffff88277062ffd8
>>> [ 2162.498368]  ffffffff81c10440 ffff882fa6786640 ffffffff815117a1
>>> ffff882fa6786640
>>> [ 2162.498373] Call Trace:
>>> [ 2162.498378]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
>>> [ 2162.498382]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>> [ 2162.498387]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>> [ 2162.498392]  [<ffffffff816be299>] schedule+0x29/0x70
>>> [ 2162.498397]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>> [ 2162.498402]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
>>> [ 2162.498407]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>> [ 2162.498411]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>> [ 2162.498416]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>> [ 2162.498420]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>> [ 2162.498425]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>> [ 2162.498430]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>> [ 2162.498435]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>> [ 2162.498439]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>> [ 2162.498444]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>> [ 2162.498449]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>> [ 2162.498454]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>> [ 2162.498459]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>>> [ 2162.498463]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>> [ 2162.498467]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2162.498472]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>> [ 2162.498476]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2162.498480] INFO: task iozone:2052 blocked for more than 120 seconds.
>>> [ 2162.498484] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>> disables this message.
>>> [ 2162.498488] iozone          D 0000000000000001     0  2052   1944 0x00000000
>>> [ 2162.498492]  ffff882f7e2d3978 0000000000000082 ffff882f7e2d3938
>>> ffffffff81301d7a
>>> [ 2162.498497]  ffff882f8a116640 ffff882f7e2d3fd8 ffff882f7e2d3fd8
>>> ffff882f7e2d3fd8
>>> [ 2162.498501]  ffff882f88b33320 ffff882f8a116640 ffff882f7e2d3978
>>> ffff882fbf2339f8
>>> [ 2162.498505] Call Trace:
>>> [ 2162.498511]  [<ffffffff81301d7a>] ? generic_make_request+0xca/0x100
>>> [ 2162.498517]  [<ffffffff816be299>] schedule+0x29/0x70
>>> [ 2162.498522]  [<ffffffff816be36f>] io_schedule+0x8f/0xd0
>>> [ 2162.498530]  [<ffffffff811b797c>] do_blockdev_direct_IO+0x1a7c/0x1fb0
>>> [ 2162.498539]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
>>> [ 2162.498546]  [<ffffffff811b7f05>] __blockdev_direct_IO+0x55/0x60
>>> [ 2162.498552]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
>>> [ 2162.498556]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>> [ 2162.498562]  [<ffffffffa0274349>] ext2_direct_IO+0x79/0xe0 [ext2]
>>> [ 2162.498567]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
>>> [ 2162.498572]  [<ffffffff8107e775>] ? set_next_entity+0xa5/0xc0
>>> [ 2162.498579]  [<ffffffff8104ade6>] ? current_fs_time+0x16/0x60
>>> [ 2162.498585]  [<ffffffff8111f126>] generic_file_direct_write+0xc6/0x180
>>> [ 2162.498590]  [<ffffffff8111f4bd>] __generic_file_aio_write+0x2dd/0x3b0
>>> [ 2162.498595]  [<ffffffff816bc364>] ? mutex_unlock+0x14/0x20
>>> [ 2162.498600]  [<ffffffff8111f5f9>] generic_file_aio_write+0x69/0xd0
>>> [ 2162.498607]  [<ffffffff8117b88a>] do_sync_write+0x7a/0xb0
>>> [ 2162.498612]  [<ffffffff8117c63e>] vfs_write+0xce/0x1e0
>>> [ 2162.498617]  [<ffffffff8117bba0>] ? generic_file_llseek_size+0x110/0x110
>>> [ 2162.498621]  [<ffffffff8117cb22>] SyS_write+0x52/0xa0
>>> [ 2162.498627]  [<ffffffff816c7a02>] system_call_fastpath+0x16/0x1b
>>> [ 2162.498630] INFO: task kworker/0:1:2293 blocked for more than 120 seconds.
>>> [ 2162.498635] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>> disables this message.
>>> [ 2162.498639] kworker/0:1     D ffffffff81813a60     0  2293      2 0x00000000
>>> [ 2162.498645] Workqueue: events write_dirty_finish
>>> [ 2162.498647]  ffff882f8a05bb30 0000000000000046 ffffffff81511b8c
>>> ffffffff815137c0
>>> [ 2162.498652]  ffff882f8a163320 ffff882f8a05bfd8 ffff882f8a05bfd8
>>> ffff882f8a05bfd8
>>> [ 2162.498656]  ffffffff81c10440 ffff882f8a163320 ffffffff815117a1
>>> ffff882f8a163320
>>> [ 2162.498661] Call Trace:
>>> [ 2162.498665]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
>>> [ 2162.498669]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>> [ 2162.498674]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>> [ 2162.498679]  [<ffffffff816be299>] schedule+0x29/0x70
>>> [ 2162.498684]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>> [ 2162.498689]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
>>> [ 2162.498693]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>> [ 2162.498698]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>> [ 2162.498702]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>> [ 2162.498707]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>> [ 2162.498712]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>> [ 2162.498716]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>> [ 2162.498721]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>> [ 2162.498726]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>> [ 2162.498731]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>> [ 2162.498735]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>> [ 2162.498740]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>> [ 2162.498745]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>>> [ 2162.498749]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>> [ 2162.498753]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2162.498758]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>> [ 2162.498762]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2162.498766] INFO: task kworker/0:3:2534 blocked for more than 120 seconds.
>>> [ 2162.498772] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>> disables this message.
>>> [ 2162.498777] kworker/0:3     D ffffffff81813a60     0  2534      2 0x00000000
>>> [ 2162.498783] Workqueue: events write_dirty_finish
>>> [ 2162.498785]  ffff88237d0bdb30 0000000000000046 ffffffff81511b8c
>>> ffffffff815137c0
>>> [ 2162.498790]  ffff882fa66fe640 ffff88237d0bdfd8 ffff88237d0bdfd8
>>> ffff88237d0bdfd8
>>> [ 2162.498794]  ffffffff81c10440 ffff882fa66fe640 ffffffff815117a1
>>> ffff882fa66fe640
>>> [ 2162.498799] Call Trace:
>>> [ 2162.498803]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
>>> [ 2162.498807]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>> [ 2162.498812]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>> [ 2162.498817]  [<ffffffff816be299>] schedule+0x29/0x70
>>> [ 2162.498822]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>> [ 2162.498826]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
>>> [ 2162.498831]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>> [ 2162.498836]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>> [ 2162.498840]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>> [ 2162.498845]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>> [ 2162.498849]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>> [ 2162.498854]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>> [ 2162.498859]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>> [ 2162.498864]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>> [ 2162.498868]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>> [ 2162.498873]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>> [ 2162.498878]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>> [ 2162.498882]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>>> [ 2162.498886]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>> [ 2162.498890]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2162.498896]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>> [ 2162.498900]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2162.498903] INFO: task kworker/0:4:2537 blocked for more than 120 seconds.
>>> [ 2162.498908] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>> disables this message.
>>> [ 2162.498914] kworker/0:4     D 0000000000000000     0  2537      2 0x00000000
>>> [ 2162.498920] Workqueue: events write_dirty_finish
>>> [ 2162.498922]  ffff882f76d8fb30 0000000000000046 ffffffff81511b8c
>>> ffffffff815137c0
>>> [ 2162.498926]  ffff882f88b30000 ffff882f76d8ffd8 ffff882f76d8ffd8
>>> ffff882f76d8ffd8
>>> [ 2162.498931]  ffff882f78beb320 ffff882f88b30000 ffffffff815117a1
>>> ffff882f88b30000
>>> [ 2162.498935] Call Trace:
>>> [ 2162.498940]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
>>> [ 2162.498944]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>> [ 2162.498948]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>> [ 2162.498953]  [<ffffffff816be299>] schedule+0x29/0x70
>>> [ 2162.498958]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>> [ 2162.498963]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
>>> [ 2162.498968]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>> [ 2162.498972]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>> [ 2162.498977]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>> [ 2162.498981]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>> [ 2162.498986]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>> [ 2162.498990]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>> [ 2162.498995]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>> [ 2162.499000]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>> [ 2162.499004]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>> [ 2162.499009]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>> [ 2162.499014]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>> [ 2162.499019]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>>> [ 2162.499023]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>> [ 2162.499027]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2162.499032]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>> [ 2162.499036]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2162.499039] INFO: task kworker/1:1:2544 blocked for more than 120 seconds.
>>> [ 2162.499044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>> disables this message.
>>> [ 2162.499050] kworker/1:1     D ffffffff81813d40     0  2544      2 0x00000000
>>> [ 2162.499057] Workqueue: bcache bch_data_insert_keys
>>> [ 2162.499059]  ffff882fa677b3f8 0000000000000046 ffff882fa677b418
>>> ffff882f8a709000
>>> [ 2162.499063]  ffff882f88b33320 ffff882fa677bfd8 ffff882fa677bfd8
>>> ffff882fa677bfd8
>>> [ 2162.499068]  ffff882fa6471990 ffff882f88b33320 ffff882f88b33320
>>> ffff8801fad20d98
>>> [ 2162.499072] Call Trace:
>>> [ 2162.499077]  [<ffffffff816be299>] schedule+0x29/0x70
>>> [ 2162.499082]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
>>> [ 2162.499087]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
>>> [ 2162.499092]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>> [ 2162.499096]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
>>> [ 2162.499102]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
>>> [ 2162.499109]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
>>> [ 2162.499113]  [<ffffffff81076828>] ? resched_task+0x68/0x70
>>> [ 2162.499118]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
>>> [ 2162.499126]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
>>> [ 2162.499131]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
>>> [ 2162.499137]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
>>> [ 2162.499143]  [<ffffffff81126e9c>] __alloc_pages_nodemask+0x60c/0x9b0
>>> [ 2162.499149]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
>>> [ 2162.499155]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
>>> [ 2162.499159]  [<ffffffff8150ebb3>] mca_data_alloc+0x73/0x1d0
>>> [ 2162.499163]  [<ffffffff81510f37>] mca_alloc+0x277/0x470
>>> [ 2162.499168]  [<ffffffff81511d1c>] bch_btree_node_alloc+0x8c/0x1c0
>>> [ 2162.499173]  [<ffffffff81513020>] btree_split+0x110/0x5c0
>>> [ 2162.499179]  [<ffffffff81457d43>] ? scsi_pool_alloc_command+0x33/0x80
>>> [ 2162.499184]  [<ffffffff81515fc7>] ? bch_keylist_pop_front+0x47/0x50
>>> [ 2162.499188]  [<ffffffff8150fed6>] ? bch_btree_insert_keys+0x56/0x250
>>> [ 2162.499193]  [<ffffffff81082def>] ? update_group_power+0x13f/0x220
>>> [ 2162.499198]  [<ffffffff81513582>] bch_btree_insert_node+0xb2/0x2f0
>>> [ 2162.499202]  [<ffffffff815137e8>] btree_insert_fn+0x28/0x50
>>> [ 2162.499207]  [<ffffffff81511b8c>] bch_btree_map_nodes_recurse+0x6c/0x170
>>> [ 2162.499211]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>> [ 2162.499216]  [<ffffffff816bcc26>] ? down_write+0x16/0x40
>>> [ 2162.499221]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>> [ 2162.499225]  [<ffffffff81511c30>] bch_btree_map_nodes_recurse+0x110/0x170
>>> [ 2162.499230]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>> [ 2162.499235]  [<ffffffff811b51ea>] ? dio_bio_end_io+0x5a/0x90
>>> [ 2162.499239]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>> [ 2162.499244]  [<ffffffff81514dce>] __bch_btree_map_nodes+0x13e/0x1c0
>>> [ 2162.499248]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>> [ 2162.499254]  [<ffffffff8151ac6f>] ? bch_journal+0x42f/0x4b0
>>> [ 2162.499258]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>> [ 2162.499264]  [<ffffffff8151e1be>] bch_data_insert_keys+0x3e/0x160
>>> [ 2162.499268]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>> [ 2162.499273]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>> [ 2162.499278]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>>> [ 2162.499282]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>> [ 2162.499286]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2162.499292]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>> [ 2162.499296]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2162.499299] INFO: task kworker/0:5:2552 blocked for more than 120 seconds.
>>> [ 2162.499304] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>> disables this message.
>>> [ 2162.499310] kworker/0:5     D 0000000000000000     0  2552      2 0x00000000
>>> [ 2162.499411] Workqueue: events write_dirty_finish
>>> [ 2162.499413]  ffff88226314db30 0000000000000046 ffffffff81511b8c
>>> ffffffff815137c0
>>> [ 2162.499418]  ffff882fa6781990 ffff88226314dfd8 ffff88226314dfd8
>>> ffff88226314dfd8
>>> [ 2162.499422]  ffff882fa6a84cb0 ffff882fa6781990 ffffffff815117a1
>>> ffff882fa6781990
>>> [ 2162.499427] Call Trace:
>>> [ 2162.499431]  [<ffffffff81511b8c>] ? bch_btree_map_nodes_recurse+0x6c/0x170
>>> [ 2162.499435]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>> [ 2162.499440]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>> [ 2162.499445]  [<ffffffff816be299>] schedule+0x29/0x70
>>> [ 2162.499450]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>> [ 2162.499455]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
>>> [ 2162.499460]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>> [ 2162.499464]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>> [ 2162.499468]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>> [ 2162.499473]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>> [ 2162.499478]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>> [ 2162.499483]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>> [ 2162.499488]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>> [ 2162.499492]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>> [ 2162.499497]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>> [ 2162.499502]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>> [ 2162.499507]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>>> [ 2162.499511]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>> [ 2162.499515]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>> [ 2162.499520]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>> [ 2162.499524]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>
>>>
>>>
>>>>> [  930.169234] bcache: bch_cached_dev_attach() Caching sda3 as bcache0
>>>>> on set 06496904-10ca-489a-ae75-68c6a07d3db1
>>>>> [ 2522.956188] INFO: task bcache_writebac:2058 blocked for more than
>>>>> 120 seconds.
>>>>> [ 2522.956199] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [ 2522.956205] bcache_writebac D ffffffff81813a60     0  2058      2 0x00000000
>>>>> [ 2522.956212]  ffff881c39c1fcd8 0000000000000046 0000000000000001
>>>>> 0000000000000001
>>>>> [ 2522.956218]  ffff882f8b56b320 ffff881c39c1ffd8 ffff881c39c1ffd8
>>>>> ffff881c39c1ffd8
>>>>> [ 2522.956222]  ffff882fa6aeb320 ffff882f8b56b320 ffff882f8b56b320
>>>>> ffff882f8b56b320
>>>>> [ 2522.956227] Call Trace:
>>>>> [ 2522.956241]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>> [ 2522.956247]  [<ffffffff816befb5>] rwsem_down_write_failed+0xf5/0x1a0
>>>>> [ 2522.956255]  [<ffffffff81332c93>] call_rwsem_down_write_failed+0x13/0x20
>>>>> [ 2522.956261]  [<ffffffff816bcc41>] ? down_write+0x31/0x40
>>>>> [ 2522.956269]  [<ffffffff8151b782>] bch_writeback_thread+0x62/0x7f0
>>>>> [ 2522.956278]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
>>>>> [ 2522.956284]  [<ffffffff8151b720>] ? write_dirty+0xc0/0xc0
>>>>> [ 2522.956290]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>> [ 2522.956294]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2522.956302]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>> [ 2522.956306]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2522.956313] INFO: task kworker/u64:1:2433 blocked for more than 120 seconds.
>>>>> [ 2522.956317] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [ 2522.956322] kworker/u64:1   D ffffffff81813a60     0  2433      2 0x00000000
>>>>> [ 2522.956332] Workqueue: writeback bdi_writeback_workfn (flush-252:0)
>>>>> [ 2522.956336]  ffff882f899d1560 0000000000000046 ffffffff81120635
>>>>> 0000000000000100
>>>>> [ 2522.956340]  ffff88219b488000 ffff882f899d1fd8 ffff882f899d1fd8
>>>>> ffff882f899d1fd8
>>>>> [ 2522.956345]  ffff882fa6aeb320 ffff88219b488000 ffffffff8131ad55
>>>>> ffff88219b488000
>>>>> [ 2522.956349] Call Trace:
>>>>> [ 2522.956359]  [<ffffffff81120635>] ? mempool_alloc_slab+0x15/0x20
>>>>> [ 2522.956367]  [<ffffffff8131ad55>] ? blk_throtl_bio+0x3b5/0x540
>>>>> [ 2522.956372]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>> [ 2522.956378]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>> [ 2522.956383]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
>>>>> [ 2522.956388]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>> [ 2522.956394]  [<ffffffff8151fe32>] cached_dev_make_request+0x752/0xf00
>>>>> [ 2522.956400]  [<ffffffff81301af8>] ? generic_make_request_checks+0x1e8/0x3a0
>>>>> [ 2522.956405]  [<ffffffff81301d7a>] generic_make_request+0xca/0x100
>>>>> [ 2522.956409]  [<ffffffff81301e29>] submit_bio+0x79/0x160
>>>>> [ 2522.956417]  [<ffffffff811b1710>] ? bio_alloc_bioset+0xa0/0x1d0
>>>>> [ 2522.956421]  [<ffffffff811abd3f>] _submit_bh+0x13f/0x200
>>>>> [ 2522.956425]  [<ffffffff811abe10>] submit_bh+0x10/0x20
>>>>> [ 2522.956430]  [<ffffffff811af6b8>] __block_write_full_page+0x1d8/0x360
>>>>> [ 2522.956439]  [<ffffffff8145fc0b>] ? scsi_request_fn+0xbb/0x530
>>>>> [ 2522.956444]  [<ffffffff811ad3c0>] ? end_buffer_async_read+0x130/0x130
>>>>> [ 2522.956449]  [<ffffffff811b2c40>] ? I_BDEV+0x10/0x10
>>>>> [ 2522.956454]  [<ffffffff811b2c40>] ? I_BDEV+0x10/0x10
>>>>> [ 2522.956459]  [<ffffffff811af90a>] block_write_full_page_endio+0xca/0x100
>>>>> [ 2522.956464]  [<ffffffff811af955>] block_write_full_page+0x15/0x20
>>>>> [ 2522.956469]  [<ffffffff811b3608>] blkdev_writepage+0x18/0x20
>>>>> [ 2522.956473]  [<ffffffff81127d37>] __writepage+0x17/0x40
>>>>> [ 2522.956477]  [<ffffffff8112820e>] write_cache_pages+0x20e/0x460
>>>>> [ 2522.956481]  [<ffffffff81127d20>] ? set_page_dirty_lock+0x60/0x60
>>>>> [ 2522.956486]  [<ffffffff811ac251>] ? __set_page_dirty+0x71/0xc0
>>>>> [ 2522.956490]  [<ffffffff811284aa>] generic_writepages+0x4a/0x70
>>>>> [ 2522.956495]  [<ffffffff81129cd0>] do_writepages+0x20/0x40
>>>>> [ 2522.956501]  [<ffffffff811a36f5>] __writeback_single_inode+0x45/0x280
>>>>> [ 2522.956507]  [<ffffffff811269e2>] ? __alloc_pages_nodemask+0x152/0x9b0
>>>>> [ 2522.956513]  [<ffffffff811a495d>] writeback_sb_inodes+0x19d/0x3c0
>>>>> [ 2522.956518]  [<ffffffff811a4c1e>] __writeback_inodes_wb+0x9e/0xd0
>>>>> [ 2522.956523]  [<ffffffff811a4ecb>] wb_writeback+0x27b/0x320
>>>>> [ 2522.956529]  [<ffffffff81196228>] ? get_nr_dirty_inodes+0x58/0x80
>>>>> [ 2522.956534]  [<ffffffff811a500f>] wb_check_old_data_flush+0x9f/0xb0
>>>>> [ 2522.956539]  [<ffffffff811a67e1>] wb_do_writeback+0x151/0x1d0
>>>>> [ 2522.956547]  [<ffffffff8106539f>] ? set_worker_desc+0x6f/0x80
>>>>> [ 2522.956552]  [<ffffffff811a68da>] bdi_writeback_workfn+0x7a/0x200
>>>>> [ 2522.956557]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>> [ 2522.956562]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>> [ 2522.956567]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>>>>> [ 2522.956571]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>> [ 2522.956575]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2522.956581]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>> [ 2522.956585]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2522.956588] INFO: task iozone:2435 blocked for more than 120 seconds.
>>>>> [ 2522.956593] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [ 2522.956597] iozone          D 0000000000000001     0  2435   1545 0x00000000
>>>>> [ 2522.956602]  ffff882a75671978 0000000000000082 ffff882a75671938
>>>>> ffffffff81301d7a
>>>>> [ 2522.956606]  ffff882f88dc1990 ffff882a75671fd8 ffff882a75671fd8
>>>>> ffff882a75671fd8
>>>>> [ 2522.956611]  ffff882f8ec91990 ffff882f88dc1990 ffff882a75671978
>>>>> ffff882fbf2539f8
>>>>> [ 2522.956615] Call Trace:
>>>>> [ 2522.956619]  [<ffffffff81301d7a>] ? generic_make_request+0xca/0x100
>>>>> [ 2522.956625]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>> [ 2522.956630]  [<ffffffff816be36f>] io_schedule+0x8f/0xd0
>>>>> [ 2522.956636]  [<ffffffff811b797c>] do_blockdev_direct_IO+0x1a7c/0x1fb0
>>>>> [ 2522.956645]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
>>>>> [ 2522.956652]  [<ffffffff811b7f05>] __blockdev_direct_IO+0x55/0x60
>>>>> [ 2522.956658]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
>>>>> [ 2522.956662]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>> [ 2522.956669]  [<ffffffffa0263349>] ext2_direct_IO+0x79/0xe0 [ext2]
>>>>> [ 2522.956674]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
>>>>> [ 2522.956680]  [<ffffffff8104ade6>] ? current_fs_time+0x16/0x60
>>>>> [ 2522.956685]  [<ffffffff8111f126>] generic_file_direct_write+0xc6/0x180
>>>>> [ 2522.956690]  [<ffffffff8111f4bd>] __generic_file_aio_write+0x2dd/0x3b0
>>>>> [ 2522.956696]  [<ffffffff8111f5f9>] generic_file_aio_write+0x69/0xd0
>>>>> [ 2522.956702]  [<ffffffff8117b88a>] do_sync_write+0x7a/0xb0
>>>>> [ 2522.956706]  [<ffffffff811bb108>] ? fsnotify+0x1f8/0x2b0
>>>>> [ 2522.956712]  [<ffffffff8117c63e>] vfs_write+0xce/0x1e0
>>>>> [ 2522.956716]  [<ffffffff8117cb22>] SyS_write+0x52/0xa0
>>>>> [ 2522.956722]  [<ffffffff816c7a02>] system_call_fastpath+0x16/0x1b
>>>>> [ 2522.956726] INFO: task kworker/2:1:2470 blocked for more than 120 seconds.
>>>>> [ 2522.956730] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [ 2522.956735] kworker/2:1     D ffffffff81813d40     0  2470      2 0x00000000
>>>>> [ 2522.956743] Workqueue: bcache bch_data_insert_keys
>>>>> [ 2522.956745]  ffff882a5b0c9628 0000000000000046 ffff882a5b0c9648
>>>>> 0000004000000000
>>>>> [ 2522.956749]  ffff882f8ec91990 ffff882a5b0c9fd8 ffff882a5b0c9fd8
>>>>> ffff882a5b0c9fd8
>>>>> [ 2522.956754]  ffff882fa6489990 ffff882f8ec91990 ffff882f8ec91990
>>>>> ffff8828dba00d98
>>>>> [ 2522.956758] Call Trace:
>>>>> [ 2522.956763]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>> [ 2522.956769]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
>>>>> [ 2522.956774]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
>>>>> [ 2522.956779]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>>>> [ 2522.956783]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
>>>>> [ 2522.956791]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
>>>>> [ 2522.956799]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
>>>>> [ 2522.956805]  [<ffffffff81076828>] ? resched_task+0x68/0x70
>>>>> [ 2522.956810]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
>>>>> [ 2522.956816]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
>>>>> [ 2522.956822]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
>>>>> [ 2522.956827]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
>>>>> [ 2522.956833]  [<ffffffff81126e9c>] __alloc_pages_nodemask+0x60c/0x9b0
>>>>> [ 2522.956843]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
>>>>> [ 2522.956848]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
>>>>> [ 2522.956852]  [<ffffffff8150ebb3>] mca_data_alloc+0x73/0x1d0
>>>>> [ 2522.956857]  [<ffffffff8150ee5a>] mca_bucket_alloc+0x14a/0x1f0
>>>>> [ 2522.956861]  [<ffffffff81511020>] mca_alloc+0x360/0x470
>>>>> [ 2522.956866]  [<ffffffff81517031>] ? __bch_bset_search+0x1d1/0x480
>>>>> [ 2522.956871]  [<ffffffff8151183e>] bch_btree_node_get+0x10e/0x280
>>>>> [ 2522.956875]  [<ffffffff81511c02>] bch_btree_map_nodes_recurse+0xe2/0x170
>>>>> [ 2522.956880]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>>> [ 2522.956886]  [<ffffffff811b51ea>] ? dio_bio_end_io+0x5a/0x90
>>>>> [ 2522.956890]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>> [ 2522.956895]  [<ffffffff81514dce>] __bch_btree_map_nodes+0x13e/0x1c0
>>>>> [ 2522.956899]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>>> [ 2522.956905]  [<ffffffff8151ac6f>] ? bch_journal+0x42f/0x4b0
>>>>> [ 2522.956909]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>> [ 2522.956915]  [<ffffffff8151e1be>] bch_data_insert_keys+0x3e/0x160
>>>>> [ 2522.956920]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>> [ 2522.956925]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>> [ 2522.956930]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>>>>> [ 2522.956933]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>> [ 2522.956938]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2522.956943]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>> [ 2522.956947]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2522.956951] INFO: task kworker/3:2:2471 blocked for more than 120 seconds.
>>>>> [ 2522.956955] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [ 2522.956959] kworker/3:2     D ffffffff81813a60     0  2471      2 0x00000000
>>>>> [ 2522.956966] Workqueue: events update_writeback_rate
>>>>> [ 2522.956968]  ffff882a5d125cd0 0000000000000046 0000000000000003
>>>>> ffff882a5d125d68
>>>>> [ 2522.956972]  ffff882f8ec94cb0 ffff882a5d125fd8 ffff882a5d125fd8
>>>>> ffff882a5d125fd8
>>>>> [ 2522.956977]  ffff882fa6aeb320 ffff882f8ec94cb0 0000000200000003
>>>>> ffff882f8ec94cb0
>>>>> [ 2522.956981] Call Trace:
>>>>> [ 2522.956987]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>> [ 2522.956992]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>> [ 2522.956997]  [<ffffffff81332c64>] call_rwsem_down_read_failed+0x14/0x30
>>>>> [ 2522.957002]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>> [ 2522.957007]  [<ffffffff8151b010>] update_writeback_rate+0x30/0x230
>>>>> [ 2522.957011]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>> [ 2522.957017]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>> [ 2522.957021]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>>>>> [ 2522.957025]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>> [ 2522.957030]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2522.957035]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>> [ 2522.957039]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2522.957048] INFO: task kworker/u64:2:2540 blocked for more than 120 seconds.
>>>>> [ 2522.957052] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [ 2522.957056] kworker/u64:2   D ffffffff81813a60     0  2540      2 0x00000000
>>>>> [ 2522.957062] Workqueue: bch_btree_io btree_node_write_work
>>>>> [ 2522.957064]  ffff882a4fdc9778 0000000000000046 ffff882a4fdc9738
>>>>> ffff882f86e3be58
>>>>> [ 2522.957069]  ffff882fa4bc1990 ffff882a4fdc9fd8 ffff882a4fdc9fd8
>>>>> ffff882a4fdc9fd8
>>>>> [ 2522.957073]  ffff882fa6aeb320 ffff882fa4bc1990 ffff882f86e3be00
>>>>> ffff8828dba00d98
>>>>> [ 2522.957078] Call Trace:
>>>>> [ 2522.957083]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>> [ 2522.957088]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
>>>>> [ 2522.957092]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
>>>>> [ 2522.957102]  [<ffffffff81484090>] ? ata_scsiop_mode_sense+0x380/0x380
>>>>> [ 2522.957107]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>>>> [ 2522.957111]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
>>>>> [ 2522.957116]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
>>>>> [ 2522.957121]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
>>>>> [ 2522.957126]  [<ffffffff81076828>] ? resched_task+0x68/0x70
>>>>> [ 2522.957130]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
>>>>> [ 2522.957135]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
>>>>> [ 2522.957140]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
>>>>> [ 2522.957146]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
>>>>> [ 2522.957152]  [<ffffffff81126e9c>] __alloc_pages_nodemask+0x60c/0x9b0
>>>>> [ 2522.957158]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
>>>>> [ 2522.957163]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
>>>>> [ 2522.957168]  [<ffffffff81517fc8>] __btree_sort+0x48/0x230
>>>>> [ 2522.957173]  [<ffffffff8151765c>] ? __bch_btree_iter_init+0x7c/0xc0
>>>>> [ 2522.957178]  [<ffffffff81518301>] bch_btree_sort_partial+0x101/0x120
>>>>> [ 2522.957182]  [<ffffffff8150f000>] ? __btree_node_write_done+0x100/0x100
>>>>> [ 2522.957187]  [<ffffffff81518468>] bch_btree_sort_lazy+0x68/0x90
>>>>> [ 2522.957191]  [<ffffffff815109ba>] bch_btree_node_write+0x36a/0x4a0
>>>>> [ 2522.957196]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
>>>>> [ 2522.957201]  [<ffffffff8106212c>] ? pwq_activate_delayed_work+0x4c/0xb0
>>>>> [ 2522.957205]  [<ffffffff81511477>] btree_node_write_work+0x57/0x80
>>>>> [ 2522.957210]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>> [ 2522.957215]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>> [ 2522.957219]  [<ffffffff81063570>] ? manage_workers.isra.23+0x2d0/0x2d0
>>>>> [ 2522.957223]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>> [ 2522.957228]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2522.957233]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>> [ 2522.957237]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>
>>>>>
>>>>> I can easily reproduce this, please let me know if you require any
>>>>> info/test any patch.
>>>>>
>>>>> Thanks,
>>>>> Neo
>>>>>
>>>>> On Mon, Aug 26, 2013 at 12:21 PM, Stefan Priebe <s.priebe@profihost.ag> wrote:
>>>>>> Hi Kent,
>>>>>>
>>>>>> a new one under 3.10:
>>>>>>
>>>>>> 2013-08-26 21:05:30     INFO: task ceph-osd:8939 blocked for more than 120
>>>>>> seconds.
>>>>>> 2013-08-26 21:05:30     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> 2013-08-26 21:05:30     ceph-osd D ffffffff8160f760 0 8939 1 0x00000000
>>>>>> 2013-08-26 21:05:30     ffff880bd9a61820 0000000000000086 ffff880bd9a61fd8
>>>>>> 0000000000012c80
>>>>>> 2013-08-26 21:05:30     ffff880bd9a60010 0000000000012c80 0000000000012c80
>>>>>> 0000000000012c80
>>>>>> 2013-08-26 21:05:30     ffff880bd9a61fd8 0000000000012c80 ffff880bd7abb1c0
>>>>>> ffff880c48dee380
>>>>>> 2013-08-26 21:05:29     [<ffffffff811e8128>] xfs_vm_readpages+0x18/0x20
>>>>>> 2013-08-26 21:05:29     [<ffffffff810f1da3>] read_pages+0x43/0x100
>>>>>> 2013-08-26 21:05:29     [<ffffffff810e64df>] ? __page_cache_alloc+0x9f/0xc0
>>>>>> 2013-08-26 21:05:29     [<ffffffff810f1fab>]
>>>>>> __do_page_cache_readahead+0x14b/0x160
>>>>>> 2013-08-26 21:05:29     [<ffffffff810f1fdc>] ra_submit+0x1c/0x20
>>>>>> 2013-08-26 21:05:29     [<ffffffff810f22a5>] ondemand_readahead+0x115/0x240
>>>>>> 2013-08-26 21:05:29     [<ffffffff810f249e>]
>>>>>> page_cache_sync_readahead+0x2e/0x40
>>>>>> 2013-08-26 21:05:29     [<ffffffff810e7a5e>] T.1003+0x33e/0x430
>>>>>> 2013-08-26 21:05:29     [<ffffffff810e7c30>]
>>>>>> generic_file_aio_read+0xe0/0x220
>>>>>> 2013-08-26 21:05:29     [<ffffffff811efd0a>] xfs_file_aio_read+0x15a/0x2a0
>>>>>> 2013-08-26 21:05:29     [<ffffffff8114095a>] do_sync_read+0x7a/0xb0
>>>>>> 2013-08-26 21:05:29     [<ffffffff81140df1>] vfs_read+0xb1/0x130
>>>>>> 2013-08-26 21:05:29     [<ffffffff81141757>] SyS_pread64+0x97/0xa0
>>>>>> 2013-08-26 21:05:29     [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b
>>>>>> 2013-08-26 21:05:29     INFO: task ceph-osd:8896 blocked for more than 120
>>>>>> seconds.
>>>>>> 2013-08-26 21:05:29     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> 2013-08-26 21:05:29     ceph-osd D ffffffff8160f760 0 8896 1 0x00000000
>>>>>> 2013-08-26 21:05:29     ffff880bd9a9dc28 0000000000000086 ffff880bd9a9dfd8
>>>>>> 0000000000012c80
>>>>>> 2013-08-26 21:05:29     ffff880bd9a9c010 0000000000012c80 0000000000012c80
>>>>>> 0000000000012c80
>>>>>> 2013-08-26 21:05:29     ffff880bd9a9dfd8 0000000000012c80 ffff880bd79298e0
>>>>>> ffff880c48dd6380
>>>>>> 2013-08-26 21:05:29     Call Trace:
>>>>>> 2013-08-26 21:05:29     [<ffffffff810e60f0>] ? __lock_page+0x70/0x70
>>>>>> 2013-08-26 21:05:28     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> 2013-08-26 21:05:28     ceph-osd D ffffffff8160f760 0 8798 1 0x00000000
>>>>>> 2013-08-26 21:05:28     ffff880bd998b740 0000000000000086 ffff880bd998bfd8
>>>>>> 0000000000012c80
>>>>>> 2013-08-26 21:05:28     ffff880bd998a010 0000000000012c80 0000000000012c80
>>>>>> 0000000000012c80
>>>>>> 2013-08-26 21:05:28     ffff880bd998bfd8 0000000000012c80 ffff880bd99c0000
>>>>>> ffff880c48dd6380
>>>>>> 2013-08-26 21:05:28     Call Trace:
>>>>>> 2013-08-26 21:05:28     [<ffffffff81524874>] schedule+0x24/0x70
>>>>>> 2013-08-26 21:05:28     [<ffffffff81525d7d>]
>>>>>> rwsem_down_read_failed+0x9d/0xe5
>>>>>> 2013-08-26 21:05:28     [<ffffffff812aa0a4>]
>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>> 2013-08-26 21:05:28     [<ffffffff81523a42>] ? down_read+0x12/0x20
>>>>>> 2013-08-26 21:05:28     [<ffffffffa01ead62>] btree_read_async+0xa2/0x1a0
>>>>>> [bcache]
>>>>>> 2013-08-26 21:05:28     [<ffffffffa01f7b73>] closure_queue+0x43/0x60
>>>>>> [bcache]
>>>>>> 2013-08-26 21:05:28     [<ffffffffa01eb903>] T.1050+0x63/0x70 [bcache]
>>>>>> 2013-08-26 21:05:28     [<ffffffffa01ed6ea>]
>>>>>> cached_dev_make_request+0x20a/0x350 [bcache]
>>>>>> 2013-08-26 21:05:28     [<ffffffff81281952>] generic_make_request+0xc2/0x100
>>>>>> 2013-08-26 21:05:28     [<ffffffff812819f7>] submit_bio+0x67/0x130
>>>>>> 2013-08-26 21:05:28     [<ffffffff8117b240>] do_mpage_readpage+0x2a0/0x660
>>>>>> 2013-08-26 21:05:28     [<ffffffff81101cee>] ?
>>>>>> __inc_zone_page_state+0x2e/0x30
>>>>>> 2013-08-26 21:05:28     [<ffffffff810e6581>] ?
>>>>>> add_to_page_cache_locked+0x81/0x110
>>>>>> 2013-08-26 21:05:28     [<ffffffff8117b76a>] mpage_readpages+0xfa/0x150
>>>>>> 2013-08-26 21:05:28     [<ffffffff811e90c0>] ?
>>>>>> xfs_get_blocks_direct+0x20/0x20
>>>>>> 2013-08-26 21:05:28     [<ffffffff811e90c0>] ?
>>>>>> xfs_get_blocks_direct+0x20/0x20
>>>>>> 2013-08-26 21:05:28     [<ffffffff810eeda3>] ?
>>>>>> __alloc_pages_nodemask+0x123/0x240
>>>>>> 2013-08-26 21:05:27     [<ffffffff8115bf93>] ? __close_fd+0x43/0x80
>>>>>> 2013-08-26 21:05:27     [<ffffffff8116dabe>] SyS_fdatasync+0xe/0x20
>>>>>> 2013-08-26 21:05:27     [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b
>>>>>> 2013-08-26 21:05:27     INFO: task ceph-osd:8746 blocked for more than 120
>>>>>> seconds.
>>>>>> 2013-08-26 21:05:27     "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> 2013-08-26 21:05:27     ceph-osd D ffffffff8160f760 0 8746 1 0x00000000
>>>>>> 2013-08-26 21:05:27     ffff880c2dde1d38 0000000000000086 ffff880c2dde1fd8
>>>>>> 0000000000012c80
>>>>>> 2013-08-26 21:05:27     ffff880c2dde0010 0000000000012c80 0000000000012c80
>>>>>> 0000000000012c80
>>>>>> 2013-08-26 21:05:27     ffff880c2dde1fd8 0000000000012c80 ffff880c239eb1c0
>>>>>> ffff880c48dd6380
>>>>>> 2013-08-26 21:05:27     Call Trace:
>>>>>> 2013-08-26 21:05:27     [<ffffffff81524874>] schedule+0x24/0x70
>>>>>> 2013-08-26 21:05:27     [<ffffffff81522bbd>] schedule_timeout+0x16d/0x200
>>>>>> 2013-08-26 21:05:27     [<ffffffff81068eba>] ?
>>>>>> __queue_delayed_work+0xaa/0x1a0
>>>>>> 2013-08-26 21:05:27     [<ffffffff81068d19>] ?
>>>>>> try_to_grab_pending+0x109/0x190
>>>>>> 2013-08-26 21:05:27     [<ffffffff815251e5>] wait_for_completion+0x95/0x110
>>>>>> 2013-08-26 21:05:27     [<ffffffff8107fd00>] ? try_to_wake_up+0x2a0/0x2a0
>>>>>> 2013-08-26 21:05:27     [<ffffffff81167797>] ? bdi_queue_work+0x77/0xc0
>>>>>> 2013-08-26 21:05:27     [<ffffffff81167863>]
>>>>>> writeback_inodes_sb_nr+0x83/0xb0
>>>>>> 2013-08-26 21:05:27     [<ffffffff811678ea>] writeback_inodes_sb+0x5a/0x70
>>>>>> 2013-08-26 21:05:27     [<ffffffff8116dd0a>] __sync_filesystem+0x4a/0x50
>>>>>> 2013-08-26 21:05:27     [<ffffffff8116dd42>] sync_filesystem+0x32/0x60
>>>>>> 2013-08-26 21:05:27     [<ffffffff8116ddc0>] SyS_syncfs+0x50/0x90
>>>>>> 2013-08-26 21:05:27     [<ffffffff81526b12>] system_call_fastpath+0x16/0x1b
>>>>>> 2013-08-26 21:05:27     INFO: task ceph-osd:8798 blocked for more than 120
>>>>>> seconds.
>>>>>>
>>>>>> Stefan
>>>>>>
>>>>>> Am 22.08.2013 09:32, schrieb Stefan Priebe - Profihost AG:
>>>>>>
>>>>>>> great!
>>>>>>>
>>>>>>> Everything seems to work fine now! Except read_dirty always going to
>>>>>>> negative values after a reboot.
>>>>>>>
>>>>>>> Stefan
>>>>>>>
>>>>>>> Am 22.08.2013 08:02, schrieb Kent Overstreet:
>>>>>>>>
>>>>>>>> On Thu, Aug 22, 2013 at 07:59:04AM +0200, Stefan Priebe wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> schedule_timeout() is not the same as
>>>>>>>>>> schedule_timeout_interruptible().
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> just search and replace? So i can try on my own.
>>>>>>>>
>>>>>>>>
>>>>>>>> The one in read_dirty(), line ~330
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-bcache"
>>>>>>>> in
>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-28 19:20                                     ` Stefan Priebe
@ 2013-08-28 20:12                                       ` kernel neophyte
  2013-08-28 20:15                                         ` Stefan Priebe
  0 siblings, 1 reply; 28+ messages in thread
From: kernel neophyte @ 2013-08-28 20:12 UTC (permalink / raw)
  To: Stefan Priebe
  Cc: Kent Overstreet, Jens Axboe, linux-kernel@vger.kernel.org,
	linux-bcache@vger.kernel.org

On Wed, Aug 28, 2013 at 12:20 PM, Stefan Priebe <s.priebe@profihost.ag> wrote:
> Am 28.08.2013 20:47, schrieb kernel neophyte:
>
>> On Wed, Aug 28, 2013 at 11:38 AM, Stefan Priebe - Profihost AG
>> <s.priebe@profihost.ag> wrote:
>>>
>>> I don't had one for a few days. Which kernel so you use?
>>
>>
>> 3.10 kernel with all of kent's stable patches and perf patches..
>
>
> Which exact 3.10 version? Which patches on top?

3.10.0-rc7
with following patches:
bcache: Fix a dumb CPU spinning bug in writeback
bcache: Fix a flush/fua performance bug
bcache: Fix a writeback performance regression
bcache: FUA fixes

I have also tried bcache perf branch i.e.
(http://evilpiepirate.org/git/linux-bcache.git/log/?h=bcache-perf)

>
>
>> and all defaults except for:
>>
>> echo writeback > /sys/block/bcache0/bcache/cache_mode
>> echo 0 > /sys/block/bcache0/bcache/sequential_cutoff
>> echo 0 >
>> /sys/fs/bcache/1e26e5ca-5cb9-4c5a-ad7e-3c787524a9dc/congested_read_threshold_us
>> echo 0 >
>> /sys/fs/bcache/1e26e5ca-5cb9-4c5a-ad7e-3c787524a9dc/congested_write_threshold_us
>>
>> -Neo
>>
>>>
>>> Stefan
>>>
>>> This mail was sent with my iPhone.
>>>
>>> Am 28.08.2013 um 20:12 schrieb kernel neophyte
>>> <neophyte.hacker001@gmail.com>:
>>>
>>>> On Tue, Aug 27, 2013 at 11:05 PM, Kent Overstreet <kmo@daterainc.com>
>>>> wrote:
>>>>>
>>>>> On Tue, Aug 27, 2013 at 08:37:54PM -0700, kernel neophyte wrote:
>>>>>>
>>>>>> Hi Kent,
>>>>>>
>>>>>> I am still seeing deadlock:
>>>>>
>>>>>
>>>>> Seeing blk_throtl_bio (what asshole misspells words in their function
>>>>> names, so when you go to grep for them you grep for the wrong thing?)
>>>>> in
>>>>> the backtrace - is that what it takes to hit it?
>>>>
>>>>
>>>> I don't think so,
>>>> I had another deadlock:
>>>>
>>>> [ 1179.193990] bio: create slab <bio-2> at 2
>>>> [ 1179.194491] bcache: run_cache_set() invalidating existing data
>>>> [ 1179.212308] bcache: register_cache() registered cache device rxd1
>>>> [ 1203.846220] bcache: register_bdev() registered backing device sda3
>>>> [ 1227.780088] bcache: bch_cached_dev_attach() Caching sda3 as bcache0
>>>> on set a079dda3-27fd-4b1c-a616-d46110bfae6c
>>>> [ 2162.497950] INFO: task kworker/0:0:4 blocked for more than 120
>>>> seconds.
>>>> [ 2162.497961] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>> disables this message.
>>>> [ 2162.497967] kworker/0:0     D 0000000000000000     0     4      2
>>>> 0x00000000
>>>> [ 2162.497982] Workqueue: events write_dirty_finish
>>>> [ 2162.497986]  ffff882fa6ac9b30 0000000000000046 ffffffff81511b8c
>>>> ffffffff815137c0
>>>> [ 2162.497992]  ffff882fa6a84cb0 ffff882fa6ac9fd8 ffff882fa6ac9fd8
>>>> ffff882fa6ac9fd8
>>>> [ 2162.497996]  ffff882f78139990 ffff882fa6a84cb0 ffffffff815117a1
>>>> ffff882fa6a84cb0
>>>> [ 2162.498001] Call Trace:
>>>> [ 2162.498010]  [<ffffffff81511b8c>] ?
>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>> [ 2162.498015]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>> [ 2162.498019]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>> [ 2162.498028]  [<ffffffff816be299>] schedule+0x29/0x70
>>>> [ 2162.498034]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>> [ 2162.498043]  [<ffffffff81332c64>]
>>>> call_rwsem_down_read_failed+0x14/0x30
>>>> [ 2162.498048]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>> [ 2162.498052]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>> [ 2162.498057]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>> [ 2162.498066]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>> [ 2162.498072]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>> [ 2162.498076]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>> [ 2162.498082]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>> [ 2162.498087]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>> [ 2162.498093]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>> [ 2162.498100]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>> [ 2162.498105]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>> [ 2162.498110]  [<ffffffff81063570>] ?
>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>> [ 2162.498115]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>> [ 2162.498120]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2162.498127]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>> [ 2162.498131]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2162.498145] INFO: task bcache_allocato:1766 blocked for more than
>>>> 120 seconds.
>>>> [ 2162.498149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>> disables this message.
>>>> [ 2162.498154] bcache_allocato D 0000000000000001     0  1766      2
>>>> 0x00000000
>>>> [ 2162.498158]  ffff8822630f3d88 0000000000000046 ffff8822630f3da8
>>>> ffffffff810808ad
>>>> [ 2162.498163]  ffff882f891de640 ffff8822630f3fd8 ffff8822630f3fd8
>>>> ffff8822630f3fd8
>>>> [ 2162.498167]  ffff882f81a91990 ffff882f891de640 ffff882f88b33320
>>>> ffff8801fad20d98
>>>> [ 2162.498172] Call Trace:
>>>> [ 2162.498177]  [<ffffffff810808ad>] ? dequeue_task_fair+0x2cd/0x530
>>>> [ 2162.498182]  [<ffffffff816be299>] schedule+0x29/0x70
>>>> [ 2162.498187]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
>>>> [ 2162.498192]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
>>>> [ 2162.498197]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>>> [ 2162.498202]  [<ffffffff8150cdbf>] bch_allocator_thread+0x10f/0xe20
>>>> [ 2162.498207]  [<ffffffff8150ccb0>] ? bch_bucket_add_unused+0xe0/0xe0
>>>> [ 2162.498211]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>> [ 2162.498215]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2162.498220]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>> [ 2162.498224]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2162.498228] INFO: task bcache_writebac:1771 blocked for more than
>>>> 120 seconds.
>>>> [ 2162.498233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>> disables this message.
>>>> [ 2162.498237] bcache_writebac D 0000000000000000     0  1771      2
>>>> 0x00000000
>>>> [ 2162.498241]  ffff88197d2adc58 0000000000000046 ffff88197d2add48
>>>> ffffffff811269e2
>>>> [ 2162.498245]  ffff882f891d8000 ffff88197d2adfd8 ffff88197d2adfd8
>>>> ffff88197d2adfd8
>>>> [ 2162.498250]  ffff882f81a91990 ffff882f891d8000 ffff88197d2adc68
>>>> ffff882f89550ba0
>>>> [ 2162.498254] Call Trace:
>>>> [ 2162.498264]  [<ffffffff811269e2>] ?
>>>> __alloc_pages_nodemask+0x152/0x9b0
>>>> [ 2162.498269]  [<ffffffff816be299>] schedule+0x29/0x70
>>>> [ 2162.498274]  [<ffffffff816bc015>] schedule_timeout+0x1e5/0x250
>>>> [ 2162.498280]  [<ffffffff810799bf>] ? try_to_wake_up+0x24f/0x2b0
>>>> [ 2162.498285]  [<ffffffff816bcd09>] __down_common+0x8e/0xe3
>>>> [ 2162.498293]  [<ffffffff81160600>] ? alloc_pages_current+0x90/0x170
>>>> [ 2162.498298]  [<ffffffff816bcdd1>] __down+0x1d/0x1f
>>>> [ 2162.498304]  [<ffffffff8106f351>] down+0x41/0x50
>>>> [ 2162.498310]  [<ffffffff8151ba7d>] bch_writeback_thread+0x35d/0x7f0
>>>> [ 2162.498315]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
>>>> [ 2162.498321]  [<ffffffff8151b720>] ? write_dirty+0xc0/0xc0
>>>> [ 2162.498325]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>> [ 2162.498329]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2162.498334]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>> [ 2162.498338]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2162.498342] INFO: task kworker/0:2:1789 blocked for more than 120
>>>> seconds.
>>>> [ 2162.498346] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>> disables this message.
>>>> [ 2162.498351] kworker/0:2     D ffffffff81813a60     0  1789      2
>>>> 0x00000000
>>>> [ 2162.498357] Workqueue: events write_dirty_finish
>>>> [ 2162.498360]  ffff88277062fb30 0000000000000046 ffffffff81511b8c
>>>> ffffffff815137c0
>>>> [ 2162.498364]  ffff882fa6786640 ffff88277062ffd8 ffff88277062ffd8
>>>> ffff88277062ffd8
>>>> [ 2162.498368]  ffffffff81c10440 ffff882fa6786640 ffffffff815117a1
>>>> ffff882fa6786640
>>>> [ 2162.498373] Call Trace:
>>>> [ 2162.498378]  [<ffffffff81511b8c>] ?
>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>> [ 2162.498382]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>> [ 2162.498387]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>> [ 2162.498392]  [<ffffffff816be299>] schedule+0x29/0x70
>>>> [ 2162.498397]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>> [ 2162.498402]  [<ffffffff81332c64>]
>>>> call_rwsem_down_read_failed+0x14/0x30
>>>> [ 2162.498407]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>> [ 2162.498411]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>> [ 2162.498416]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>> [ 2162.498420]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>> [ 2162.498425]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>> [ 2162.498430]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>> [ 2162.498435]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>> [ 2162.498439]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>> [ 2162.498444]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>> [ 2162.498449]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>> [ 2162.498454]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>> [ 2162.498459]  [<ffffffff81063570>] ?
>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>> [ 2162.498463]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>> [ 2162.498467]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2162.498472]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>> [ 2162.498476]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2162.498480] INFO: task iozone:2052 blocked for more than 120 seconds.
>>>> [ 2162.498484] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>> disables this message.
>>>> [ 2162.498488] iozone          D 0000000000000001     0  2052   1944
>>>> 0x00000000
>>>> [ 2162.498492]  ffff882f7e2d3978 0000000000000082 ffff882f7e2d3938
>>>> ffffffff81301d7a
>>>> [ 2162.498497]  ffff882f8a116640 ffff882f7e2d3fd8 ffff882f7e2d3fd8
>>>> ffff882f7e2d3fd8
>>>> [ 2162.498501]  ffff882f88b33320 ffff882f8a116640 ffff882f7e2d3978
>>>> ffff882fbf2339f8
>>>> [ 2162.498505] Call Trace:
>>>> [ 2162.498511]  [<ffffffff81301d7a>] ? generic_make_request+0xca/0x100
>>>> [ 2162.498517]  [<ffffffff816be299>] schedule+0x29/0x70
>>>> [ 2162.498522]  [<ffffffff816be36f>] io_schedule+0x8f/0xd0
>>>> [ 2162.498530]  [<ffffffff811b797c>] do_blockdev_direct_IO+0x1a7c/0x1fb0
>>>> [ 2162.498539]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60
>>>> [ext2]
>>>> [ 2162.498546]  [<ffffffff811b7f05>] __blockdev_direct_IO+0x55/0x60
>>>> [ 2162.498552]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60
>>>> [ext2]
>>>> [ 2162.498556]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>> [ 2162.498562]  [<ffffffffa0274349>] ext2_direct_IO+0x79/0xe0 [ext2]
>>>> [ 2162.498567]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60
>>>> [ext2]
>>>> [ 2162.498572]  [<ffffffff8107e775>] ? set_next_entity+0xa5/0xc0
>>>> [ 2162.498579]  [<ffffffff8104ade6>] ? current_fs_time+0x16/0x60
>>>> [ 2162.498585]  [<ffffffff8111f126>]
>>>> generic_file_direct_write+0xc6/0x180
>>>> [ 2162.498590]  [<ffffffff8111f4bd>]
>>>> __generic_file_aio_write+0x2dd/0x3b0
>>>> [ 2162.498595]  [<ffffffff816bc364>] ? mutex_unlock+0x14/0x20
>>>> [ 2162.498600]  [<ffffffff8111f5f9>] generic_file_aio_write+0x69/0xd0
>>>> [ 2162.498607]  [<ffffffff8117b88a>] do_sync_write+0x7a/0xb0
>>>> [ 2162.498612]  [<ffffffff8117c63e>] vfs_write+0xce/0x1e0
>>>> [ 2162.498617]  [<ffffffff8117bba0>] ?
>>>> generic_file_llseek_size+0x110/0x110
>>>> [ 2162.498621]  [<ffffffff8117cb22>] SyS_write+0x52/0xa0
>>>> [ 2162.498627]  [<ffffffff816c7a02>] system_call_fastpath+0x16/0x1b
>>>> [ 2162.498630] INFO: task kworker/0:1:2293 blocked for more than 120
>>>> seconds.
>>>> [ 2162.498635] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>> disables this message.
>>>> [ 2162.498639] kworker/0:1     D ffffffff81813a60     0  2293      2
>>>> 0x00000000
>>>> [ 2162.498645] Workqueue: events write_dirty_finish
>>>> [ 2162.498647]  ffff882f8a05bb30 0000000000000046 ffffffff81511b8c
>>>> ffffffff815137c0
>>>> [ 2162.498652]  ffff882f8a163320 ffff882f8a05bfd8 ffff882f8a05bfd8
>>>> ffff882f8a05bfd8
>>>> [ 2162.498656]  ffffffff81c10440 ffff882f8a163320 ffffffff815117a1
>>>> ffff882f8a163320
>>>> [ 2162.498661] Call Trace:
>>>> [ 2162.498665]  [<ffffffff81511b8c>] ?
>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>> [ 2162.498669]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>> [ 2162.498674]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>> [ 2162.498679]  [<ffffffff816be299>] schedule+0x29/0x70
>>>> [ 2162.498684]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>> [ 2162.498689]  [<ffffffff81332c64>]
>>>> call_rwsem_down_read_failed+0x14/0x30
>>>> [ 2162.498693]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>> [ 2162.498698]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>> [ 2162.498702]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>> [ 2162.498707]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>> [ 2162.498712]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>> [ 2162.498716]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>> [ 2162.498721]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>> [ 2162.498726]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>> [ 2162.498731]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>> [ 2162.498735]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>> [ 2162.498740]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>> [ 2162.498745]  [<ffffffff81063570>] ?
>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>> [ 2162.498749]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>> [ 2162.498753]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2162.498758]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>> [ 2162.498762]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2162.498766] INFO: task kworker/0:3:2534 blocked for more than 120
>>>> seconds.
>>>> [ 2162.498772] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>> disables this message.
>>>> [ 2162.498777] kworker/0:3     D ffffffff81813a60     0  2534      2
>>>> 0x00000000
>>>> [ 2162.498783] Workqueue: events write_dirty_finish
>>>> [ 2162.498785]  ffff88237d0bdb30 0000000000000046 ffffffff81511b8c
>>>> ffffffff815137c0
>>>> [ 2162.498790]  ffff882fa66fe640 ffff88237d0bdfd8 ffff88237d0bdfd8
>>>> ffff88237d0bdfd8
>>>> [ 2162.498794]  ffffffff81c10440 ffff882fa66fe640 ffffffff815117a1
>>>> ffff882fa66fe640
>>>> [ 2162.498799] Call Trace:
>>>> [ 2162.498803]  [<ffffffff81511b8c>] ?
>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>> [ 2162.498807]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>> [ 2162.498812]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>> [ 2162.498817]  [<ffffffff816be299>] schedule+0x29/0x70
>>>> [ 2162.498822]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>> [ 2162.498826]  [<ffffffff81332c64>]
>>>> call_rwsem_down_read_failed+0x14/0x30
>>>> [ 2162.498831]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>> [ 2162.498836]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>> [ 2162.498840]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>> [ 2162.498845]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>> [ 2162.498849]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>> [ 2162.498854]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>> [ 2162.498859]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>> [ 2162.498864]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>> [ 2162.498868]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>> [ 2162.498873]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>> [ 2162.498878]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>> [ 2162.498882]  [<ffffffff81063570>] ?
>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>> [ 2162.498886]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>> [ 2162.498890]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2162.498896]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>> [ 2162.498900]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2162.498903] INFO: task kworker/0:4:2537 blocked for more than 120
>>>> seconds.
>>>> [ 2162.498908] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>> disables this message.
>>>> [ 2162.498914] kworker/0:4     D 0000000000000000     0  2537      2
>>>> 0x00000000
>>>> [ 2162.498920] Workqueue: events write_dirty_finish
>>>> [ 2162.498922]  ffff882f76d8fb30 0000000000000046 ffffffff81511b8c
>>>> ffffffff815137c0
>>>> [ 2162.498926]  ffff882f88b30000 ffff882f76d8ffd8 ffff882f76d8ffd8
>>>> ffff882f76d8ffd8
>>>> [ 2162.498931]  ffff882f78beb320 ffff882f88b30000 ffffffff815117a1
>>>> ffff882f88b30000
>>>> [ 2162.498935] Call Trace:
>>>> [ 2162.498940]  [<ffffffff81511b8c>] ?
>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>> [ 2162.498944]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>> [ 2162.498948]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>> [ 2162.498953]  [<ffffffff816be299>] schedule+0x29/0x70
>>>> [ 2162.498958]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>> [ 2162.498963]  [<ffffffff81332c64>]
>>>> call_rwsem_down_read_failed+0x14/0x30
>>>> [ 2162.498968]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>> [ 2162.498972]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>> [ 2162.498977]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>> [ 2162.498981]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>> [ 2162.498986]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>> [ 2162.498990]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>> [ 2162.498995]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>> [ 2162.499000]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>> [ 2162.499004]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>> [ 2162.499009]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>> [ 2162.499014]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>> [ 2162.499019]  [<ffffffff81063570>] ?
>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>> [ 2162.499023]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>> [ 2162.499027]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2162.499032]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>> [ 2162.499036]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2162.499039] INFO: task kworker/1:1:2544 blocked for more than 120
>>>> seconds.
>>>> [ 2162.499044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>> disables this message.
>>>> [ 2162.499050] kworker/1:1     D ffffffff81813d40     0  2544      2
>>>> 0x00000000
>>>> [ 2162.499057] Workqueue: bcache bch_data_insert_keys
>>>> [ 2162.499059]  ffff882fa677b3f8 0000000000000046 ffff882fa677b418
>>>> ffff882f8a709000
>>>> [ 2162.499063]  ffff882f88b33320 ffff882fa677bfd8 ffff882fa677bfd8
>>>> ffff882fa677bfd8
>>>> [ 2162.499068]  ffff882fa6471990 ffff882f88b33320 ffff882f88b33320
>>>> ffff8801fad20d98
>>>> [ 2162.499072] Call Trace:
>>>> [ 2162.499077]  [<ffffffff816be299>] schedule+0x29/0x70
>>>> [ 2162.499082]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
>>>> [ 2162.499087]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
>>>> [ 2162.499092]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>>> [ 2162.499096]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
>>>> [ 2162.499102]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
>>>> [ 2162.499109]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
>>>> [ 2162.499113]  [<ffffffff81076828>] ? resched_task+0x68/0x70
>>>> [ 2162.499118]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
>>>> [ 2162.499126]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
>>>> [ 2162.499131]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
>>>> [ 2162.499137]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
>>>> [ 2162.499143]  [<ffffffff81126e9c>] __alloc_pages_nodemask+0x60c/0x9b0
>>>> [ 2162.499149]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
>>>> [ 2162.499155]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
>>>> [ 2162.499159]  [<ffffffff8150ebb3>] mca_data_alloc+0x73/0x1d0
>>>> [ 2162.499163]  [<ffffffff81510f37>] mca_alloc+0x277/0x470
>>>> [ 2162.499168]  [<ffffffff81511d1c>] bch_btree_node_alloc+0x8c/0x1c0
>>>> [ 2162.499173]  [<ffffffff81513020>] btree_split+0x110/0x5c0
>>>> [ 2162.499179]  [<ffffffff81457d43>] ? scsi_pool_alloc_command+0x33/0x80
>>>> [ 2162.499184]  [<ffffffff81515fc7>] ? bch_keylist_pop_front+0x47/0x50
>>>> [ 2162.499188]  [<ffffffff8150fed6>] ? bch_btree_insert_keys+0x56/0x250
>>>> [ 2162.499193]  [<ffffffff81082def>] ? update_group_power+0x13f/0x220
>>>> [ 2162.499198]  [<ffffffff81513582>] bch_btree_insert_node+0xb2/0x2f0
>>>> [ 2162.499202]  [<ffffffff815137e8>] btree_insert_fn+0x28/0x50
>>>> [ 2162.499207]  [<ffffffff81511b8c>]
>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>> [ 2162.499211]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>> [ 2162.499216]  [<ffffffff816bcc26>] ? down_write+0x16/0x40
>>>> [ 2162.499221]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>> [ 2162.499225]  [<ffffffff81511c30>]
>>>> bch_btree_map_nodes_recurse+0x110/0x170
>>>> [ 2162.499230]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>> [ 2162.499235]  [<ffffffff811b51ea>] ? dio_bio_end_io+0x5a/0x90
>>>> [ 2162.499239]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>> [ 2162.499244]  [<ffffffff81514dce>] __bch_btree_map_nodes+0x13e/0x1c0
>>>> [ 2162.499248]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>> [ 2162.499254]  [<ffffffff8151ac6f>] ? bch_journal+0x42f/0x4b0
>>>> [ 2162.499258]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>> [ 2162.499264]  [<ffffffff8151e1be>] bch_data_insert_keys+0x3e/0x160
>>>> [ 2162.499268]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>> [ 2162.499273]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>> [ 2162.499278]  [<ffffffff81063570>] ?
>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>> [ 2162.499282]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>> [ 2162.499286]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2162.499292]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>> [ 2162.499296]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2162.499299] INFO: task kworker/0:5:2552 blocked for more than 120
>>>> seconds.
>>>> [ 2162.499304] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>> disables this message.
>>>> [ 2162.499310] kworker/0:5     D 0000000000000000     0  2552      2
>>>> 0x00000000
>>>> [ 2162.499411] Workqueue: events write_dirty_finish
>>>> [ 2162.499413]  ffff88226314db30 0000000000000046 ffffffff81511b8c
>>>> ffffffff815137c0
>>>> [ 2162.499418]  ffff882fa6781990 ffff88226314dfd8 ffff88226314dfd8
>>>> ffff88226314dfd8
>>>> [ 2162.499422]  ffff882fa6a84cb0 ffff882fa6781990 ffffffff815117a1
>>>> ffff882fa6781990
>>>> [ 2162.499427] Call Trace:
>>>> [ 2162.499431]  [<ffffffff81511b8c>] ?
>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>> [ 2162.499435]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>> [ 2162.499440]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>> [ 2162.499445]  [<ffffffff816be299>] schedule+0x29/0x70
>>>> [ 2162.499450]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>> [ 2162.499455]  [<ffffffff81332c64>]
>>>> call_rwsem_down_read_failed+0x14/0x30
>>>> [ 2162.499460]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>> [ 2162.499464]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>> [ 2162.499468]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>> [ 2162.499473]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>> [ 2162.499478]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>> [ 2162.499483]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>> [ 2162.499488]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>> [ 2162.499492]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>> [ 2162.499497]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>> [ 2162.499502]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>> [ 2162.499507]  [<ffffffff81063570>] ?
>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>> [ 2162.499511]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>> [ 2162.499515]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>> [ 2162.499520]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>> [ 2162.499524]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>
>>>>
>>>>
>>>>>> [  930.169234] bcache: bch_cached_dev_attach() Caching sda3 as bcache0
>>>>>> on set 06496904-10ca-489a-ae75-68c6a07d3db1
>>>>>> [ 2522.956188] INFO: task bcache_writebac:2058 blocked for more than
>>>>>> 120 seconds.
>>>>>> [ 2522.956199] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2522.956205] bcache_writebac D ffffffff81813a60     0  2058      2
>>>>>> 0x00000000
>>>>>> [ 2522.956212]  ffff881c39c1fcd8 0000000000000046 0000000000000001
>>>>>> 0000000000000001
>>>>>> [ 2522.956218]  ffff882f8b56b320 ffff881c39c1ffd8 ffff881c39c1ffd8
>>>>>> ffff881c39c1ffd8
>>>>>> [ 2522.956222]  ffff882fa6aeb320 ffff882f8b56b320 ffff882f8b56b320
>>>>>> ffff882f8b56b320
>>>>>> [ 2522.956227] Call Trace:
>>>>>> [ 2522.956241]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2522.956247]  [<ffffffff816befb5>]
>>>>>> rwsem_down_write_failed+0xf5/0x1a0
>>>>>> [ 2522.956255]  [<ffffffff81332c93>]
>>>>>> call_rwsem_down_write_failed+0x13/0x20
>>>>>> [ 2522.956261]  [<ffffffff816bcc41>] ? down_write+0x31/0x40
>>>>>> [ 2522.956269]  [<ffffffff8151b782>] bch_writeback_thread+0x62/0x7f0
>>>>>> [ 2522.956278]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
>>>>>> [ 2522.956284]  [<ffffffff8151b720>] ? write_dirty+0xc0/0xc0
>>>>>> [ 2522.956290]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>> [ 2522.956294]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2522.956302]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>> [ 2522.956306]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2522.956313] INFO: task kworker/u64:1:2433 blocked for more than 120
>>>>>> seconds.
>>>>>> [ 2522.956317] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2522.956322] kworker/u64:1   D ffffffff81813a60     0  2433      2
>>>>>> 0x00000000
>>>>>> [ 2522.956332] Workqueue: writeback bdi_writeback_workfn (flush-252:0)
>>>>>> [ 2522.956336]  ffff882f899d1560 0000000000000046 ffffffff81120635
>>>>>> 0000000000000100
>>>>>> [ 2522.956340]  ffff88219b488000 ffff882f899d1fd8 ffff882f899d1fd8
>>>>>> ffff882f899d1fd8
>>>>>> [ 2522.956345]  ffff882fa6aeb320 ffff88219b488000 ffffffff8131ad55
>>>>>> ffff88219b488000
>>>>>> [ 2522.956349] Call Trace:
>>>>>> [ 2522.956359]  [<ffffffff81120635>] ? mempool_alloc_slab+0x15/0x20
>>>>>> [ 2522.956367]  [<ffffffff8131ad55>] ? blk_throtl_bio+0x3b5/0x540
>>>>>> [ 2522.956372]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2522.956378]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>>> [ 2522.956383]  [<ffffffff81332c64>]
>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>> [ 2522.956388]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>>> [ 2522.956394]  [<ffffffff8151fe32>]
>>>>>> cached_dev_make_request+0x752/0xf00
>>>>>> [ 2522.956400]  [<ffffffff81301af8>] ?
>>>>>> generic_make_request_checks+0x1e8/0x3a0
>>>>>> [ 2522.956405]  [<ffffffff81301d7a>] generic_make_request+0xca/0x100
>>>>>> [ 2522.956409]  [<ffffffff81301e29>] submit_bio+0x79/0x160
>>>>>> [ 2522.956417]  [<ffffffff811b1710>] ? bio_alloc_bioset+0xa0/0x1d0
>>>>>> [ 2522.956421]  [<ffffffff811abd3f>] _submit_bh+0x13f/0x200
>>>>>> [ 2522.956425]  [<ffffffff811abe10>] submit_bh+0x10/0x20
>>>>>> [ 2522.956430]  [<ffffffff811af6b8>]
>>>>>> __block_write_full_page+0x1d8/0x360
>>>>>> [ 2522.956439]  [<ffffffff8145fc0b>] ? scsi_request_fn+0xbb/0x530
>>>>>> [ 2522.956444]  [<ffffffff811ad3c0>] ?
>>>>>> end_buffer_async_read+0x130/0x130
>>>>>> [ 2522.956449]  [<ffffffff811b2c40>] ? I_BDEV+0x10/0x10
>>>>>> [ 2522.956454]  [<ffffffff811b2c40>] ? I_BDEV+0x10/0x10
>>>>>> [ 2522.956459]  [<ffffffff811af90a>]
>>>>>> block_write_full_page_endio+0xca/0x100
>>>>>> [ 2522.956464]  [<ffffffff811af955>] block_write_full_page+0x15/0x20
>>>>>> [ 2522.956469]  [<ffffffff811b3608>] blkdev_writepage+0x18/0x20
>>>>>> [ 2522.956473]  [<ffffffff81127d37>] __writepage+0x17/0x40
>>>>>> [ 2522.956477]  [<ffffffff8112820e>] write_cache_pages+0x20e/0x460
>>>>>> [ 2522.956481]  [<ffffffff81127d20>] ? set_page_dirty_lock+0x60/0x60
>>>>>> [ 2522.956486]  [<ffffffff811ac251>] ? __set_page_dirty+0x71/0xc0
>>>>>> [ 2522.956490]  [<ffffffff811284aa>] generic_writepages+0x4a/0x70
>>>>>> [ 2522.956495]  [<ffffffff81129cd0>] do_writepages+0x20/0x40
>>>>>> [ 2522.956501]  [<ffffffff811a36f5>]
>>>>>> __writeback_single_inode+0x45/0x280
>>>>>> [ 2522.956507]  [<ffffffff811269e2>] ?
>>>>>> __alloc_pages_nodemask+0x152/0x9b0
>>>>>> [ 2522.956513]  [<ffffffff811a495d>] writeback_sb_inodes+0x19d/0x3c0
>>>>>> [ 2522.956518]  [<ffffffff811a4c1e>] __writeback_inodes_wb+0x9e/0xd0
>>>>>> [ 2522.956523]  [<ffffffff811a4ecb>] wb_writeback+0x27b/0x320
>>>>>> [ 2522.956529]  [<ffffffff81196228>] ? get_nr_dirty_inodes+0x58/0x80
>>>>>> [ 2522.956534]  [<ffffffff811a500f>] wb_check_old_data_flush+0x9f/0xb0
>>>>>> [ 2522.956539]  [<ffffffff811a67e1>] wb_do_writeback+0x151/0x1d0
>>>>>> [ 2522.956547]  [<ffffffff8106539f>] ? set_worker_desc+0x6f/0x80
>>>>>> [ 2522.956552]  [<ffffffff811a68da>] bdi_writeback_workfn+0x7a/0x200
>>>>>> [ 2522.956557]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>> [ 2522.956562]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>> [ 2522.956567]  [<ffffffff81063570>] ?
>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>> [ 2522.956571]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>> [ 2522.956575]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2522.956581]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>> [ 2522.956585]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2522.956588] INFO: task iozone:2435 blocked for more than 120
>>>>>> seconds.
>>>>>> [ 2522.956593] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2522.956597] iozone          D 0000000000000001     0  2435   1545
>>>>>> 0x00000000
>>>>>> [ 2522.956602]  ffff882a75671978 0000000000000082 ffff882a75671938
>>>>>> ffffffff81301d7a
>>>>>> [ 2522.956606]  ffff882f88dc1990 ffff882a75671fd8 ffff882a75671fd8
>>>>>> ffff882a75671fd8
>>>>>> [ 2522.956611]  ffff882f8ec91990 ffff882f88dc1990 ffff882a75671978
>>>>>> ffff882fbf2539f8
>>>>>> [ 2522.956615] Call Trace:
>>>>>> [ 2522.956619]  [<ffffffff81301d7a>] ? generic_make_request+0xca/0x100
>>>>>> [ 2522.956625]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2522.956630]  [<ffffffff816be36f>] io_schedule+0x8f/0xd0
>>>>>> [ 2522.956636]  [<ffffffff811b797c>]
>>>>>> do_blockdev_direct_IO+0x1a7c/0x1fb0
>>>>>> [ 2522.956645]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60
>>>>>> [ext2]
>>>>>> [ 2522.956652]  [<ffffffff811b7f05>] __blockdev_direct_IO+0x55/0x60
>>>>>> [ 2522.956658]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60
>>>>>> [ext2]
>>>>>> [ 2522.956662]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>>> [ 2522.956669]  [<ffffffffa0263349>] ext2_direct_IO+0x79/0xe0 [ext2]
>>>>>> [ 2522.956674]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60
>>>>>> [ext2]
>>>>>> [ 2522.956680]  [<ffffffff8104ade6>] ? current_fs_time+0x16/0x60
>>>>>> [ 2522.956685]  [<ffffffff8111f126>]
>>>>>> generic_file_direct_write+0xc6/0x180
>>>>>> [ 2522.956690]  [<ffffffff8111f4bd>]
>>>>>> __generic_file_aio_write+0x2dd/0x3b0
>>>>>> [ 2522.956696]  [<ffffffff8111f5f9>] generic_file_aio_write+0x69/0xd0
>>>>>> [ 2522.956702]  [<ffffffff8117b88a>] do_sync_write+0x7a/0xb0
>>>>>> [ 2522.956706]  [<ffffffff811bb108>] ? fsnotify+0x1f8/0x2b0
>>>>>> [ 2522.956712]  [<ffffffff8117c63e>] vfs_write+0xce/0x1e0
>>>>>> [ 2522.956716]  [<ffffffff8117cb22>] SyS_write+0x52/0xa0
>>>>>> [ 2522.956722]  [<ffffffff816c7a02>] system_call_fastpath+0x16/0x1b
>>>>>> [ 2522.956726] INFO: task kworker/2:1:2470 blocked for more than 120
>>>>>> seconds.
>>>>>> [ 2522.956730] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2522.956735] kworker/2:1     D ffffffff81813d40     0  2470      2
>>>>>> 0x00000000
>>>>>> [ 2522.956743] Workqueue: bcache bch_data_insert_keys
>>>>>> [ 2522.956745]  ffff882a5b0c9628 0000000000000046 ffff882a5b0c9648
>>>>>> 0000004000000000
>>>>>> [ 2522.956749]  ffff882f8ec91990 ffff882a5b0c9fd8 ffff882a5b0c9fd8
>>>>>> ffff882a5b0c9fd8
>>>>>> [ 2522.956754]  ffff882fa6489990 ffff882f8ec91990 ffff882f8ec91990
>>>>>> ffff8828dba00d98
>>>>>> [ 2522.956758] Call Trace:
>>>>>> [ 2522.956763]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2522.956769]  [<ffffffff816be57e>]
>>>>>> schedule_preempt_disabled+0xe/0x10
>>>>>> [ 2522.956774]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
>>>>>> [ 2522.956779]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>>>>> [ 2522.956783]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
>>>>>> [ 2522.956791]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
>>>>>> [ 2522.956799]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
>>>>>> [ 2522.956805]  [<ffffffff81076828>] ? resched_task+0x68/0x70
>>>>>> [ 2522.956810]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
>>>>>> [ 2522.956816]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
>>>>>> [ 2522.956822]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
>>>>>> [ 2522.956827]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
>>>>>> [ 2522.956833]  [<ffffffff81126e9c>]
>>>>>> __alloc_pages_nodemask+0x60c/0x9b0
>>>>>> [ 2522.956843]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
>>>>>> [ 2522.956848]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
>>>>>> [ 2522.956852]  [<ffffffff8150ebb3>] mca_data_alloc+0x73/0x1d0
>>>>>> [ 2522.956857]  [<ffffffff8150ee5a>] mca_bucket_alloc+0x14a/0x1f0
>>>>>> [ 2522.956861]  [<ffffffff81511020>] mca_alloc+0x360/0x470
>>>>>> [ 2522.956866]  [<ffffffff81517031>] ? __bch_bset_search+0x1d1/0x480
>>>>>> [ 2522.956871]  [<ffffffff8151183e>] bch_btree_node_get+0x10e/0x280
>>>>>> [ 2522.956875]  [<ffffffff81511c02>]
>>>>>> bch_btree_map_nodes_recurse+0xe2/0x170
>>>>>> [ 2522.956880]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2522.956886]  [<ffffffff811b51ea>] ? dio_bio_end_io+0x5a/0x90
>>>>>> [ 2522.956890]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>>> [ 2522.956895]  [<ffffffff81514dce>] __bch_btree_map_nodes+0x13e/0x1c0
>>>>>> [ 2522.956899]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2522.956905]  [<ffffffff8151ac6f>] ? bch_journal+0x42f/0x4b0
>>>>>> [ 2522.956909]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>>> [ 2522.956915]  [<ffffffff8151e1be>] bch_data_insert_keys+0x3e/0x160
>>>>>> [ 2522.956920]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>> [ 2522.956925]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>> [ 2522.956930]  [<ffffffff81063570>] ?
>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>> [ 2522.956933]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>> [ 2522.956938]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2522.956943]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>> [ 2522.956947]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2522.956951] INFO: task kworker/3:2:2471 blocked for more than 120
>>>>>> seconds.
>>>>>> [ 2522.956955] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2522.956959] kworker/3:2     D ffffffff81813a60     0  2471      2
>>>>>> 0x00000000
>>>>>> [ 2522.956966] Workqueue: events update_writeback_rate
>>>>>> [ 2522.956968]  ffff882a5d125cd0 0000000000000046 0000000000000003
>>>>>> ffff882a5d125d68
>>>>>> [ 2522.956972]  ffff882f8ec94cb0 ffff882a5d125fd8 ffff882a5d125fd8
>>>>>> ffff882a5d125fd8
>>>>>> [ 2522.956977]  ffff882fa6aeb320 ffff882f8ec94cb0 0000000200000003
>>>>>> ffff882f8ec94cb0
>>>>>> [ 2522.956981] Call Trace:
>>>>>> [ 2522.956987]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2522.956992]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>>> [ 2522.956997]  [<ffffffff81332c64>]
>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>> [ 2522.957002]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>>> [ 2522.957007]  [<ffffffff8151b010>] update_writeback_rate+0x30/0x230
>>>>>> [ 2522.957011]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>> [ 2522.957017]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>> [ 2522.957021]  [<ffffffff81063570>] ?
>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>> [ 2522.957025]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>> [ 2522.957030]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2522.957035]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>> [ 2522.957039]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2522.957048] INFO: task kworker/u64:2:2540 blocked for more than 120
>>>>>> seconds.
>>>>>> [ 2522.957052] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2522.957056] kworker/u64:2   D ffffffff81813a60     0  2540      2
>>>>>> 0x00000000
>>>>>> [ 2522.957062] Workqueue: bch_btree_io btree_node_write_work
>>>>>> [ 2522.957064]  ffff882a4fdc9778 0000000000000046 ffff882a4fdc9738
>>>>>> ffff882f86e3be58
>>>>>> [ 2522.957069]  ffff882fa4bc1990 ffff882a4fdc9fd8 ffff882a4fdc9fd8
>>>>>> ffff882a4fdc9fd8
>>>>>> [ 2522.957073]  ffff882fa6aeb320 ffff882fa4bc1990 ffff882f86e3be00
>>>>>> ffff8828dba00d98
>>>>>> [ 2522.957078] Call Trace:
>>>>>> [ 2522.957083]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2522.957088]  [<ffffffff816be57e>]
>>>>>> schedule_preempt_disabled+0xe/0x10
>>>>>> [ 2522.957092]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
>>>>>> [ 2522.957102]  [<ffffffff81484090>] ?
>>>>>> ata_scsiop_mode_sense+0x380/0x380
>>>>>> [ 2522.957107]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>>>>> [ 2522.957111]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
>>>>>> [ 2522.957116]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
>>>>>> [ 2522.957121]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
>>>>>> [ 2522.957126]  [<ffffffff81076828>] ? resched_task+0x68/0x70
>>>>>> [ 2522.957130]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
>>>>>> [ 2522.957135]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
>>>>>> [ 2522.957140]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
>>>>>> [ 2522.957146]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
>>>>>> [ 2522.957152]  [<ffffffff81126e9c>]
>>>>>> __alloc_pages_nodemask+0x60c/0x9b0
>>>>>> [ 2522.957158]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
>>>>>> [ 2522.957163]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
>>>>>> [ 2522.957168]  [<ffffffff81517fc8>] __btree_sort+0x48/0x230
>>>>>> [ 2522.957173]  [<ffffffff8151765c>] ? __bch_btree_iter_init+0x7c/0xc0
>>>>>> [ 2522.957178]  [<ffffffff81518301>]
>>>>>> bch_btree_sort_partial+0x101/0x120
>>>>>> [ 2522.957182]  [<ffffffff8150f000>] ?
>>>>>> __btree_node_write_done+0x100/0x100
>>>>>> [ 2522.957187]  [<ffffffff81518468>] bch_btree_sort_lazy+0x68/0x90
>>>>>> [ 2522.957191]  [<ffffffff815109ba>] bch_btree_node_write+0x36a/0x4a0
>>>>>> [ 2522.957196]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
>>>>>> [ 2522.957201]  [<ffffffff8106212c>] ?
>>>>>> pwq_activate_delayed_work+0x4c/0xb0
>>>>>> [ 2522.957205]  [<ffffffff81511477>] btree_node_write_work+0x57/0x80
>>>>>> [ 2522.957210]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>> [ 2522.957215]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>> [ 2522.957219]  [<ffffffff81063570>] ?
>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>> [ 2522.957223]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>> [ 2522.957228]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2522.957233]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>> [ 2522.957237]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>>
>>>>>>
>>>>>> I can easily reproduce this, please let me know if you require any
>>>>>> info/test any patch.
>>>>>>
>>>>>> Thanks,
>>>>>> Neo
>>>>>>
>>>>>> On Mon, Aug 26, 2013 at 12:21 PM, Stefan Priebe
>>>>>> <s.priebe@profihost.ag> wrote:
>>>>>>>
>>>>>>> Hi Kent,
>>>>>>>
>>>>>>> a new one under 3.10:
>>>>>>>
>>>>>>> 2013-08-26 21:05:30     INFO: task ceph-osd:8939 blocked for more
>>>>>>> than 120
>>>>>>> seconds.
>>>>>>> 2013-08-26 21:05:30     "echo 0 >
>>>>>>> /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>> disables this message.
>>>>>>> 2013-08-26 21:05:30     ceph-osd D ffffffff8160f760 0 8939 1
>>>>>>> 0x00000000
>>>>>>> 2013-08-26 21:05:30     ffff880bd9a61820 0000000000000086
>>>>>>> ffff880bd9a61fd8
>>>>>>> 0000000000012c80
>>>>>>> 2013-08-26 21:05:30     ffff880bd9a60010 0000000000012c80
>>>>>>> 0000000000012c80
>>>>>>> 0000000000012c80
>>>>>>> 2013-08-26 21:05:30     ffff880bd9a61fd8 0000000000012c80
>>>>>>> ffff880bd7abb1c0
>>>>>>> ffff880c48dee380
>>>>>>> 2013-08-26 21:05:29     [<ffffffff811e8128>]
>>>>>>> xfs_vm_readpages+0x18/0x20
>>>>>>> 2013-08-26 21:05:29     [<ffffffff810f1da3>] read_pages+0x43/0x100
>>>>>>> 2013-08-26 21:05:29     [<ffffffff810e64df>] ?
>>>>>>> __page_cache_alloc+0x9f/0xc0
>>>>>>> 2013-08-26 21:05:29     [<ffffffff810f1fab>]
>>>>>>> __do_page_cache_readahead+0x14b/0x160
>>>>>>> 2013-08-26 21:05:29     [<ffffffff810f1fdc>] ra_submit+0x1c/0x20
>>>>>>> 2013-08-26 21:05:29     [<ffffffff810f22a5>]
>>>>>>> ondemand_readahead+0x115/0x240
>>>>>>> 2013-08-26 21:05:29     [<ffffffff810f249e>]
>>>>>>> page_cache_sync_readahead+0x2e/0x40
>>>>>>> 2013-08-26 21:05:29     [<ffffffff810e7a5e>] T.1003+0x33e/0x430
>>>>>>> 2013-08-26 21:05:29     [<ffffffff810e7c30>]
>>>>>>> generic_file_aio_read+0xe0/0x220
>>>>>>> 2013-08-26 21:05:29     [<ffffffff811efd0a>]
>>>>>>> xfs_file_aio_read+0x15a/0x2a0
>>>>>>> 2013-08-26 21:05:29     [<ffffffff8114095a>] do_sync_read+0x7a/0xb0
>>>>>>> 2013-08-26 21:05:29     [<ffffffff81140df1>] vfs_read+0xb1/0x130
>>>>>>> 2013-08-26 21:05:29     [<ffffffff81141757>] SyS_pread64+0x97/0xa0
>>>>>>> 2013-08-26 21:05:29     [<ffffffff81526b12>]
>>>>>>> system_call_fastpath+0x16/0x1b
>>>>>>> 2013-08-26 21:05:29     INFO: task ceph-osd:8896 blocked for more
>>>>>>> than 120
>>>>>>> seconds.
>>>>>>> 2013-08-26 21:05:29     "echo 0 >
>>>>>>> /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>> disables this message.
>>>>>>> 2013-08-26 21:05:29     ceph-osd D ffffffff8160f760 0 8896 1
>>>>>>> 0x00000000
>>>>>>> 2013-08-26 21:05:29     ffff880bd9a9dc28 0000000000000086
>>>>>>> ffff880bd9a9dfd8
>>>>>>> 0000000000012c80
>>>>>>> 2013-08-26 21:05:29     ffff880bd9a9c010 0000000000012c80
>>>>>>> 0000000000012c80
>>>>>>> 0000000000012c80
>>>>>>> 2013-08-26 21:05:29     ffff880bd9a9dfd8 0000000000012c80
>>>>>>> ffff880bd79298e0
>>>>>>> ffff880c48dd6380
>>>>>>> 2013-08-26 21:05:29     Call Trace:
>>>>>>> 2013-08-26 21:05:29     [<ffffffff810e60f0>] ? __lock_page+0x70/0x70
>>>>>>> 2013-08-26 21:05:28     "echo 0 >
>>>>>>> /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>> disables this message.
>>>>>>> 2013-08-26 21:05:28     ceph-osd D ffffffff8160f760 0 8798 1
>>>>>>> 0x00000000
>>>>>>> 2013-08-26 21:05:28     ffff880bd998b740 0000000000000086
>>>>>>> ffff880bd998bfd8
>>>>>>> 0000000000012c80
>>>>>>> 2013-08-26 21:05:28     ffff880bd998a010 0000000000012c80
>>>>>>> 0000000000012c80
>>>>>>> 0000000000012c80
>>>>>>> 2013-08-26 21:05:28     ffff880bd998bfd8 0000000000012c80
>>>>>>> ffff880bd99c0000
>>>>>>> ffff880c48dd6380
>>>>>>> 2013-08-26 21:05:28     Call Trace:
>>>>>>> 2013-08-26 21:05:28     [<ffffffff81524874>] schedule+0x24/0x70
>>>>>>> 2013-08-26 21:05:28     [<ffffffff81525d7d>]
>>>>>>> rwsem_down_read_failed+0x9d/0xe5
>>>>>>> 2013-08-26 21:05:28     [<ffffffff812aa0a4>]
>>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>>> 2013-08-26 21:05:28     [<ffffffff81523a42>] ? down_read+0x12/0x20
>>>>>>> 2013-08-26 21:05:28     [<ffffffffa01ead62>]
>>>>>>> btree_read_async+0xa2/0x1a0
>>>>>>> [bcache]
>>>>>>> 2013-08-26 21:05:28     [<ffffffffa01f7b73>] closure_queue+0x43/0x60
>>>>>>> [bcache]
>>>>>>> 2013-08-26 21:05:28     [<ffffffffa01eb903>] T.1050+0x63/0x70
>>>>>>> [bcache]
>>>>>>> 2013-08-26 21:05:28     [<ffffffffa01ed6ea>]
>>>>>>> cached_dev_make_request+0x20a/0x350 [bcache]
>>>>>>> 2013-08-26 21:05:28     [<ffffffff81281952>]
>>>>>>> generic_make_request+0xc2/0x100
>>>>>>> 2013-08-26 21:05:28     [<ffffffff812819f7>] submit_bio+0x67/0x130
>>>>>>> 2013-08-26 21:05:28     [<ffffffff8117b240>]
>>>>>>> do_mpage_readpage+0x2a0/0x660
>>>>>>> 2013-08-26 21:05:28     [<ffffffff81101cee>] ?
>>>>>>> __inc_zone_page_state+0x2e/0x30
>>>>>>> 2013-08-26 21:05:28     [<ffffffff810e6581>] ?
>>>>>>> add_to_page_cache_locked+0x81/0x110
>>>>>>> 2013-08-26 21:05:28     [<ffffffff8117b76a>]
>>>>>>> mpage_readpages+0xfa/0x150
>>>>>>> 2013-08-26 21:05:28     [<ffffffff811e90c0>] ?
>>>>>>> xfs_get_blocks_direct+0x20/0x20
>>>>>>> 2013-08-26 21:05:28     [<ffffffff811e90c0>] ?
>>>>>>> xfs_get_blocks_direct+0x20/0x20
>>>>>>> 2013-08-26 21:05:28     [<ffffffff810eeda3>] ?
>>>>>>> __alloc_pages_nodemask+0x123/0x240
>>>>>>> 2013-08-26 21:05:27     [<ffffffff8115bf93>] ? __close_fd+0x43/0x80
>>>>>>> 2013-08-26 21:05:27     [<ffffffff8116dabe>] SyS_fdatasync+0xe/0x20
>>>>>>> 2013-08-26 21:05:27     [<ffffffff81526b12>]
>>>>>>> system_call_fastpath+0x16/0x1b
>>>>>>> 2013-08-26 21:05:27     INFO: task ceph-osd:8746 blocked for more
>>>>>>> than 120
>>>>>>> seconds.
>>>>>>> 2013-08-26 21:05:27     "echo 0 >
>>>>>>> /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>> disables this message.
>>>>>>> 2013-08-26 21:05:27     ceph-osd D ffffffff8160f760 0 8746 1
>>>>>>> 0x00000000
>>>>>>> 2013-08-26 21:05:27     ffff880c2dde1d38 0000000000000086
>>>>>>> ffff880c2dde1fd8
>>>>>>> 0000000000012c80
>>>>>>> 2013-08-26 21:05:27     ffff880c2dde0010 0000000000012c80
>>>>>>> 0000000000012c80
>>>>>>> 0000000000012c80
>>>>>>> 2013-08-26 21:05:27     ffff880c2dde1fd8 0000000000012c80
>>>>>>> ffff880c239eb1c0
>>>>>>> ffff880c48dd6380
>>>>>>> 2013-08-26 21:05:27     Call Trace:
>>>>>>> 2013-08-26 21:05:27     [<ffffffff81524874>] schedule+0x24/0x70
>>>>>>> 2013-08-26 21:05:27     [<ffffffff81522bbd>]
>>>>>>> schedule_timeout+0x16d/0x200
>>>>>>> 2013-08-26 21:05:27     [<ffffffff81068eba>] ?
>>>>>>> __queue_delayed_work+0xaa/0x1a0
>>>>>>> 2013-08-26 21:05:27     [<ffffffff81068d19>] ?
>>>>>>> try_to_grab_pending+0x109/0x190
>>>>>>> 2013-08-26 21:05:27     [<ffffffff815251e5>]
>>>>>>> wait_for_completion+0x95/0x110
>>>>>>> 2013-08-26 21:05:27     [<ffffffff8107fd00>] ?
>>>>>>> try_to_wake_up+0x2a0/0x2a0
>>>>>>> 2013-08-26 21:05:27     [<ffffffff81167797>] ?
>>>>>>> bdi_queue_work+0x77/0xc0
>>>>>>> 2013-08-26 21:05:27     [<ffffffff81167863>]
>>>>>>> writeback_inodes_sb_nr+0x83/0xb0
>>>>>>> 2013-08-26 21:05:27     [<ffffffff811678ea>]
>>>>>>> writeback_inodes_sb+0x5a/0x70
>>>>>>> 2013-08-26 21:05:27     [<ffffffff8116dd0a>]
>>>>>>> __sync_filesystem+0x4a/0x50
>>>>>>> 2013-08-26 21:05:27     [<ffffffff8116dd42>]
>>>>>>> sync_filesystem+0x32/0x60
>>>>>>> 2013-08-26 21:05:27     [<ffffffff8116ddc0>] SyS_syncfs+0x50/0x90
>>>>>>> 2013-08-26 21:05:27     [<ffffffff81526b12>]
>>>>>>> system_call_fastpath+0x16/0x1b
>>>>>>> 2013-08-26 21:05:27     INFO: task ceph-osd:8798 blocked for more
>>>>>>> than 120
>>>>>>> seconds.
>>>>>>>
>>>>>>> Stefan
>>>>>>>
>>>>>>> Am 22.08.2013 09:32, schrieb Stefan Priebe - Profihost AG:
>>>>>>>
>>>>>>>> great!
>>>>>>>>
>>>>>>>> Everything seems to work fine now! Except read_dirty always going to
>>>>>>>> negative values after a reboot.
>>>>>>>>
>>>>>>>> Stefan
>>>>>>>>
>>>>>>>> Am 22.08.2013 08:02, schrieb Kent Overstreet:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Aug 22, 2013 at 07:59:04AM +0200, Stefan Priebe wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> schedule_timeout() is not the same as
>>>>>>>>>>> schedule_timeout_interruptible().
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> just search and replace? So i can try on my own.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The one in read_dirty(), line ~330
>>>>>>>>> --
>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>> linux-bcache"
>>>>>>>>> in
>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>>
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>> linux-bcache" in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-bcache"
>>>> in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-28 20:12                                       ` kernel neophyte
@ 2013-08-28 20:15                                         ` Stefan Priebe
  2013-08-28 22:45                                           ` kernel neophyte
  2013-08-29 18:52                                           ` Stefan Priebe
  0 siblings, 2 replies; 28+ messages in thread
From: Stefan Priebe @ 2013-08-28 20:15 UTC (permalink / raw)
  To: kernel neophyte
  Cc: Kent Overstreet, Jens Axboe, linux-kernel@vger.kernel.org,
	linux-bcache@vger.kernel.org

sorry but that's completely wrong.

please use branch bcache-for-3.10

http://evilpiepirate.org/git/linux-bcache.git/log/?h=bcache-for-3.10

Stefan

Am 28.08.2013 22:12, schrieb kernel neophyte:
> On Wed, Aug 28, 2013 at 12:20 PM, Stefan Priebe <s.priebe@profihost.ag> wrote:
>> Am 28.08.2013 20:47, schrieb kernel neophyte:
>>
>>> On Wed, Aug 28, 2013 at 11:38 AM, Stefan Priebe - Profihost AG
>>> <s.priebe@profihost.ag> wrote:
>>>>
>>>> I don't had one for a few days. Which kernel so you use?
>>>
>>>
>>> 3.10 kernel with all of kent's stable patches and perf patches..
>>
>>
>> Which exact 3.10 version? Which patches on top?
>
> 3.10.0-rc7
> with following patches:
> bcache: Fix a dumb CPU spinning bug in writeback
> bcache: Fix a flush/fua performance bug
> bcache: Fix a writeback performance regression
> bcache: FUA fixes
>
> I have also tried bcache perf branch i.e.
> (http://evilpiepirate.org/git/linux-bcache.git/log/?h=bcache-perf)
>
>>
>>
>>> and all defaults except for:
>>>
>>> echo writeback > /sys/block/bcache0/bcache/cache_mode
>>> echo 0 > /sys/block/bcache0/bcache/sequential_cutoff
>>> echo 0 >
>>> /sys/fs/bcache/1e26e5ca-5cb9-4c5a-ad7e-3c787524a9dc/congested_read_threshold_us
>>> echo 0 >
>>> /sys/fs/bcache/1e26e5ca-5cb9-4c5a-ad7e-3c787524a9dc/congested_write_threshold_us
>>>
>>> -Neo
>>>
>>>>
>>>> Stefan
>>>>
>>>> This mail was sent with my iPhone.
>>>>
>>>> Am 28.08.2013 um 20:12 schrieb kernel neophyte
>>>> <neophyte.hacker001@gmail.com>:
>>>>
>>>>> On Tue, Aug 27, 2013 at 11:05 PM, Kent Overstreet <kmo@daterainc.com>
>>>>> wrote:
>>>>>>
>>>>>> On Tue, Aug 27, 2013 at 08:37:54PM -0700, kernel neophyte wrote:
>>>>>>>
>>>>>>> Hi Kent,
>>>>>>>
>>>>>>> I am still seeing deadlock:
>>>>>>
>>>>>>
>>>>>> Seeing blk_throtl_bio (what asshole misspells words in their function
>>>>>> names, so when you go to grep for them you grep for the wrong thing?)
>>>>>> in
>>>>>> the backtrace - is that what it takes to hit it?
>>>>>
>>>>>
>>>>> I don't think so,
>>>>> I had another deadlock:
>>>>>
>>>>> [ 1179.193990] bio: create slab <bio-2> at 2
>>>>> [ 1179.194491] bcache: run_cache_set() invalidating existing data
>>>>> [ 1179.212308] bcache: register_cache() registered cache device rxd1
>>>>> [ 1203.846220] bcache: register_bdev() registered backing device sda3
>>>>> [ 1227.780088] bcache: bch_cached_dev_attach() Caching sda3 as bcache0
>>>>> on set a079dda3-27fd-4b1c-a616-d46110bfae6c
>>>>> [ 2162.497950] INFO: task kworker/0:0:4 blocked for more than 120
>>>>> seconds.
>>>>> [ 2162.497961] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [ 2162.497967] kworker/0:0     D 0000000000000000     0     4      2
>>>>> 0x00000000
>>>>> [ 2162.497982] Workqueue: events write_dirty_finish
>>>>> [ 2162.497986]  ffff882fa6ac9b30 0000000000000046 ffffffff81511b8c
>>>>> ffffffff815137c0
>>>>> [ 2162.497992]  ffff882fa6a84cb0 ffff882fa6ac9fd8 ffff882fa6ac9fd8
>>>>> ffff882fa6ac9fd8
>>>>> [ 2162.497996]  ffff882f78139990 ffff882fa6a84cb0 ffffffff815117a1
>>>>> ffff882fa6a84cb0
>>>>> [ 2162.498001] Call Trace:
>>>>> [ 2162.498010]  [<ffffffff81511b8c>] ?
>>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>>> [ 2162.498015]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>>> [ 2162.498019]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>>> [ 2162.498028]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>> [ 2162.498034]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>> [ 2162.498043]  [<ffffffff81332c64>]
>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>> [ 2162.498048]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>> [ 2162.498052]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>>> [ 2162.498057]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>>> [ 2162.498066]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>> [ 2162.498072]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>>> [ 2162.498076]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>> [ 2162.498082]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>>> [ 2162.498087]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>>> [ 2162.498093]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>>> [ 2162.498100]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>> [ 2162.498105]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>> [ 2162.498110]  [<ffffffff81063570>] ?
>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>> [ 2162.498115]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>> [ 2162.498120]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2162.498127]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>> [ 2162.498131]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2162.498145] INFO: task bcache_allocato:1766 blocked for more than
>>>>> 120 seconds.
>>>>> [ 2162.498149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [ 2162.498154] bcache_allocato D 0000000000000001     0  1766      2
>>>>> 0x00000000
>>>>> [ 2162.498158]  ffff8822630f3d88 0000000000000046 ffff8822630f3da8
>>>>> ffffffff810808ad
>>>>> [ 2162.498163]  ffff882f891de640 ffff8822630f3fd8 ffff8822630f3fd8
>>>>> ffff8822630f3fd8
>>>>> [ 2162.498167]  ffff882f81a91990 ffff882f891de640 ffff882f88b33320
>>>>> ffff8801fad20d98
>>>>> [ 2162.498172] Call Trace:
>>>>> [ 2162.498177]  [<ffffffff810808ad>] ? dequeue_task_fair+0x2cd/0x530
>>>>> [ 2162.498182]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>> [ 2162.498187]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
>>>>> [ 2162.498192]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
>>>>> [ 2162.498197]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>>>> [ 2162.498202]  [<ffffffff8150cdbf>] bch_allocator_thread+0x10f/0xe20
>>>>> [ 2162.498207]  [<ffffffff8150ccb0>] ? bch_bucket_add_unused+0xe0/0xe0
>>>>> [ 2162.498211]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>> [ 2162.498215]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2162.498220]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>> [ 2162.498224]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2162.498228] INFO: task bcache_writebac:1771 blocked for more than
>>>>> 120 seconds.
>>>>> [ 2162.498233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [ 2162.498237] bcache_writebac D 0000000000000000     0  1771      2
>>>>> 0x00000000
>>>>> [ 2162.498241]  ffff88197d2adc58 0000000000000046 ffff88197d2add48
>>>>> ffffffff811269e2
>>>>> [ 2162.498245]  ffff882f891d8000 ffff88197d2adfd8 ffff88197d2adfd8
>>>>> ffff88197d2adfd8
>>>>> [ 2162.498250]  ffff882f81a91990 ffff882f891d8000 ffff88197d2adc68
>>>>> ffff882f89550ba0
>>>>> [ 2162.498254] Call Trace:
>>>>> [ 2162.498264]  [<ffffffff811269e2>] ?
>>>>> __alloc_pages_nodemask+0x152/0x9b0
>>>>> [ 2162.498269]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>> [ 2162.498274]  [<ffffffff816bc015>] schedule_timeout+0x1e5/0x250
>>>>> [ 2162.498280]  [<ffffffff810799bf>] ? try_to_wake_up+0x24f/0x2b0
>>>>> [ 2162.498285]  [<ffffffff816bcd09>] __down_common+0x8e/0xe3
>>>>> [ 2162.498293]  [<ffffffff81160600>] ? alloc_pages_current+0x90/0x170
>>>>> [ 2162.498298]  [<ffffffff816bcdd1>] __down+0x1d/0x1f
>>>>> [ 2162.498304]  [<ffffffff8106f351>] down+0x41/0x50
>>>>> [ 2162.498310]  [<ffffffff8151ba7d>] bch_writeback_thread+0x35d/0x7f0
>>>>> [ 2162.498315]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
>>>>> [ 2162.498321]  [<ffffffff8151b720>] ? write_dirty+0xc0/0xc0
>>>>> [ 2162.498325]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>> [ 2162.498329]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2162.498334]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>> [ 2162.498338]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2162.498342] INFO: task kworker/0:2:1789 blocked for more than 120
>>>>> seconds.
>>>>> [ 2162.498346] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [ 2162.498351] kworker/0:2     D ffffffff81813a60     0  1789      2
>>>>> 0x00000000
>>>>> [ 2162.498357] Workqueue: events write_dirty_finish
>>>>> [ 2162.498360]  ffff88277062fb30 0000000000000046 ffffffff81511b8c
>>>>> ffffffff815137c0
>>>>> [ 2162.498364]  ffff882fa6786640 ffff88277062ffd8 ffff88277062ffd8
>>>>> ffff88277062ffd8
>>>>> [ 2162.498368]  ffffffff81c10440 ffff882fa6786640 ffffffff815117a1
>>>>> ffff882fa6786640
>>>>> [ 2162.498373] Call Trace:
>>>>> [ 2162.498378]  [<ffffffff81511b8c>] ?
>>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>>> [ 2162.498382]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>>> [ 2162.498387]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>>> [ 2162.498392]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>> [ 2162.498397]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>> [ 2162.498402]  [<ffffffff81332c64>]
>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>> [ 2162.498407]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>> [ 2162.498411]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>>> [ 2162.498416]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>>> [ 2162.498420]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>> [ 2162.498425]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>>> [ 2162.498430]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>> [ 2162.498435]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>>> [ 2162.498439]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>>> [ 2162.498444]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>>> [ 2162.498449]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>> [ 2162.498454]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>> [ 2162.498459]  [<ffffffff81063570>] ?
>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>> [ 2162.498463]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>> [ 2162.498467]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2162.498472]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>> [ 2162.498476]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2162.498480] INFO: task iozone:2052 blocked for more than 120 seconds.
>>>>> [ 2162.498484] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [ 2162.498488] iozone          D 0000000000000001     0  2052   1944
>>>>> 0x00000000
>>>>> [ 2162.498492]  ffff882f7e2d3978 0000000000000082 ffff882f7e2d3938
>>>>> ffffffff81301d7a
>>>>> [ 2162.498497]  ffff882f8a116640 ffff882f7e2d3fd8 ffff882f7e2d3fd8
>>>>> ffff882f7e2d3fd8
>>>>> [ 2162.498501]  ffff882f88b33320 ffff882f8a116640 ffff882f7e2d3978
>>>>> ffff882fbf2339f8
>>>>> [ 2162.498505] Call Trace:
>>>>> [ 2162.498511]  [<ffffffff81301d7a>] ? generic_make_request+0xca/0x100
>>>>> [ 2162.498517]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>> [ 2162.498522]  [<ffffffff816be36f>] io_schedule+0x8f/0xd0
>>>>> [ 2162.498530]  [<ffffffff811b797c>] do_blockdev_direct_IO+0x1a7c/0x1fb0
>>>>> [ 2162.498539]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60
>>>>> [ext2]
>>>>> [ 2162.498546]  [<ffffffff811b7f05>] __blockdev_direct_IO+0x55/0x60
>>>>> [ 2162.498552]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60
>>>>> [ext2]
>>>>> [ 2162.498556]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>> [ 2162.498562]  [<ffffffffa0274349>] ext2_direct_IO+0x79/0xe0 [ext2]
>>>>> [ 2162.498567]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60
>>>>> [ext2]
>>>>> [ 2162.498572]  [<ffffffff8107e775>] ? set_next_entity+0xa5/0xc0
>>>>> [ 2162.498579]  [<ffffffff8104ade6>] ? current_fs_time+0x16/0x60
>>>>> [ 2162.498585]  [<ffffffff8111f126>]
>>>>> generic_file_direct_write+0xc6/0x180
>>>>> [ 2162.498590]  [<ffffffff8111f4bd>]
>>>>> __generic_file_aio_write+0x2dd/0x3b0
>>>>> [ 2162.498595]  [<ffffffff816bc364>] ? mutex_unlock+0x14/0x20
>>>>> [ 2162.498600]  [<ffffffff8111f5f9>] generic_file_aio_write+0x69/0xd0
>>>>> [ 2162.498607]  [<ffffffff8117b88a>] do_sync_write+0x7a/0xb0
>>>>> [ 2162.498612]  [<ffffffff8117c63e>] vfs_write+0xce/0x1e0
>>>>> [ 2162.498617]  [<ffffffff8117bba0>] ?
>>>>> generic_file_llseek_size+0x110/0x110
>>>>> [ 2162.498621]  [<ffffffff8117cb22>] SyS_write+0x52/0xa0
>>>>> [ 2162.498627]  [<ffffffff816c7a02>] system_call_fastpath+0x16/0x1b
>>>>> [ 2162.498630] INFO: task kworker/0:1:2293 blocked for more than 120
>>>>> seconds.
>>>>> [ 2162.498635] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [ 2162.498639] kworker/0:1     D ffffffff81813a60     0  2293      2
>>>>> 0x00000000
>>>>> [ 2162.498645] Workqueue: events write_dirty_finish
>>>>> [ 2162.498647]  ffff882f8a05bb30 0000000000000046 ffffffff81511b8c
>>>>> ffffffff815137c0
>>>>> [ 2162.498652]  ffff882f8a163320 ffff882f8a05bfd8 ffff882f8a05bfd8
>>>>> ffff882f8a05bfd8
>>>>> [ 2162.498656]  ffffffff81c10440 ffff882f8a163320 ffffffff815117a1
>>>>> ffff882f8a163320
>>>>> [ 2162.498661] Call Trace:
>>>>> [ 2162.498665]  [<ffffffff81511b8c>] ?
>>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>>> [ 2162.498669]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>>> [ 2162.498674]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>>> [ 2162.498679]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>> [ 2162.498684]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>> [ 2162.498689]  [<ffffffff81332c64>]
>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>> [ 2162.498693]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>> [ 2162.498698]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>>> [ 2162.498702]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>>> [ 2162.498707]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>> [ 2162.498712]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>>> [ 2162.498716]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>> [ 2162.498721]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>>> [ 2162.498726]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>>> [ 2162.498731]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>>> [ 2162.498735]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>> [ 2162.498740]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>> [ 2162.498745]  [<ffffffff81063570>] ?
>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>> [ 2162.498749]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>> [ 2162.498753]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2162.498758]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>> [ 2162.498762]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2162.498766] INFO: task kworker/0:3:2534 blocked for more than 120
>>>>> seconds.
>>>>> [ 2162.498772] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [ 2162.498777] kworker/0:3     D ffffffff81813a60     0  2534      2
>>>>> 0x00000000
>>>>> [ 2162.498783] Workqueue: events write_dirty_finish
>>>>> [ 2162.498785]  ffff88237d0bdb30 0000000000000046 ffffffff81511b8c
>>>>> ffffffff815137c0
>>>>> [ 2162.498790]  ffff882fa66fe640 ffff88237d0bdfd8 ffff88237d0bdfd8
>>>>> ffff88237d0bdfd8
>>>>> [ 2162.498794]  ffffffff81c10440 ffff882fa66fe640 ffffffff815117a1
>>>>> ffff882fa66fe640
>>>>> [ 2162.498799] Call Trace:
>>>>> [ 2162.498803]  [<ffffffff81511b8c>] ?
>>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>>> [ 2162.498807]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>>> [ 2162.498812]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>>> [ 2162.498817]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>> [ 2162.498822]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>> [ 2162.498826]  [<ffffffff81332c64>]
>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>> [ 2162.498831]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>> [ 2162.498836]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>>> [ 2162.498840]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>>> [ 2162.498845]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>> [ 2162.498849]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>>> [ 2162.498854]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>> [ 2162.498859]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>>> [ 2162.498864]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>>> [ 2162.498868]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>>> [ 2162.498873]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>> [ 2162.498878]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>> [ 2162.498882]  [<ffffffff81063570>] ?
>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>> [ 2162.498886]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>> [ 2162.498890]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2162.498896]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>> [ 2162.498900]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2162.498903] INFO: task kworker/0:4:2537 blocked for more than 120
>>>>> seconds.
>>>>> [ 2162.498908] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [ 2162.498914] kworker/0:4     D 0000000000000000     0  2537      2
>>>>> 0x00000000
>>>>> [ 2162.498920] Workqueue: events write_dirty_finish
>>>>> [ 2162.498922]  ffff882f76d8fb30 0000000000000046 ffffffff81511b8c
>>>>> ffffffff815137c0
>>>>> [ 2162.498926]  ffff882f88b30000 ffff882f76d8ffd8 ffff882f76d8ffd8
>>>>> ffff882f76d8ffd8
>>>>> [ 2162.498931]  ffff882f78beb320 ffff882f88b30000 ffffffff815117a1
>>>>> ffff882f88b30000
>>>>> [ 2162.498935] Call Trace:
>>>>> [ 2162.498940]  [<ffffffff81511b8c>] ?
>>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>>> [ 2162.498944]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>>> [ 2162.498948]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>>> [ 2162.498953]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>> [ 2162.498958]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>> [ 2162.498963]  [<ffffffff81332c64>]
>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>> [ 2162.498968]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>> [ 2162.498972]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>>> [ 2162.498977]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>>> [ 2162.498981]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>> [ 2162.498986]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>>> [ 2162.498990]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>> [ 2162.498995]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>>> [ 2162.499000]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>>> [ 2162.499004]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>>> [ 2162.499009]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>> [ 2162.499014]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>> [ 2162.499019]  [<ffffffff81063570>] ?
>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>> [ 2162.499023]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>> [ 2162.499027]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2162.499032]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>> [ 2162.499036]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2162.499039] INFO: task kworker/1:1:2544 blocked for more than 120
>>>>> seconds.
>>>>> [ 2162.499044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [ 2162.499050] kworker/1:1     D ffffffff81813d40     0  2544      2
>>>>> 0x00000000
>>>>> [ 2162.499057] Workqueue: bcache bch_data_insert_keys
>>>>> [ 2162.499059]  ffff882fa677b3f8 0000000000000046 ffff882fa677b418
>>>>> ffff882f8a709000
>>>>> [ 2162.499063]  ffff882f88b33320 ffff882fa677bfd8 ffff882fa677bfd8
>>>>> ffff882fa677bfd8
>>>>> [ 2162.499068]  ffff882fa6471990 ffff882f88b33320 ffff882f88b33320
>>>>> ffff8801fad20d98
>>>>> [ 2162.499072] Call Trace:
>>>>> [ 2162.499077]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>> [ 2162.499082]  [<ffffffff816be57e>] schedule_preempt_disabled+0xe/0x10
>>>>> [ 2162.499087]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
>>>>> [ 2162.499092]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>>>> [ 2162.499096]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
>>>>> [ 2162.499102]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
>>>>> [ 2162.499109]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
>>>>> [ 2162.499113]  [<ffffffff81076828>] ? resched_task+0x68/0x70
>>>>> [ 2162.499118]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
>>>>> [ 2162.499126]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
>>>>> [ 2162.499131]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
>>>>> [ 2162.499137]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
>>>>> [ 2162.499143]  [<ffffffff81126e9c>] __alloc_pages_nodemask+0x60c/0x9b0
>>>>> [ 2162.499149]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
>>>>> [ 2162.499155]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
>>>>> [ 2162.499159]  [<ffffffff8150ebb3>] mca_data_alloc+0x73/0x1d0
>>>>> [ 2162.499163]  [<ffffffff81510f37>] mca_alloc+0x277/0x470
>>>>> [ 2162.499168]  [<ffffffff81511d1c>] bch_btree_node_alloc+0x8c/0x1c0
>>>>> [ 2162.499173]  [<ffffffff81513020>] btree_split+0x110/0x5c0
>>>>> [ 2162.499179]  [<ffffffff81457d43>] ? scsi_pool_alloc_command+0x33/0x80
>>>>> [ 2162.499184]  [<ffffffff81515fc7>] ? bch_keylist_pop_front+0x47/0x50
>>>>> [ 2162.499188]  [<ffffffff8150fed6>] ? bch_btree_insert_keys+0x56/0x250
>>>>> [ 2162.499193]  [<ffffffff81082def>] ? update_group_power+0x13f/0x220
>>>>> [ 2162.499198]  [<ffffffff81513582>] bch_btree_insert_node+0xb2/0x2f0
>>>>> [ 2162.499202]  [<ffffffff815137e8>] btree_insert_fn+0x28/0x50
>>>>> [ 2162.499207]  [<ffffffff81511b8c>]
>>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>>> [ 2162.499211]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>>> [ 2162.499216]  [<ffffffff816bcc26>] ? down_write+0x16/0x40
>>>>> [ 2162.499221]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>>> [ 2162.499225]  [<ffffffff81511c30>]
>>>>> bch_btree_map_nodes_recurse+0x110/0x170
>>>>> [ 2162.499230]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>>> [ 2162.499235]  [<ffffffff811b51ea>] ? dio_bio_end_io+0x5a/0x90
>>>>> [ 2162.499239]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>> [ 2162.499244]  [<ffffffff81514dce>] __bch_btree_map_nodes+0x13e/0x1c0
>>>>> [ 2162.499248]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>>> [ 2162.499254]  [<ffffffff8151ac6f>] ? bch_journal+0x42f/0x4b0
>>>>> [ 2162.499258]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>> [ 2162.499264]  [<ffffffff8151e1be>] bch_data_insert_keys+0x3e/0x160
>>>>> [ 2162.499268]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>> [ 2162.499273]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>> [ 2162.499278]  [<ffffffff81063570>] ?
>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>> [ 2162.499282]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>> [ 2162.499286]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2162.499292]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>> [ 2162.499296]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2162.499299] INFO: task kworker/0:5:2552 blocked for more than 120
>>>>> seconds.
>>>>> [ 2162.499304] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>> disables this message.
>>>>> [ 2162.499310] kworker/0:5     D 0000000000000000     0  2552      2
>>>>> 0x00000000
>>>>> [ 2162.499411] Workqueue: events write_dirty_finish
>>>>> [ 2162.499413]  ffff88226314db30 0000000000000046 ffffffff81511b8c
>>>>> ffffffff815137c0
>>>>> [ 2162.499418]  ffff882fa6781990 ffff88226314dfd8 ffff88226314dfd8
>>>>> ffff88226314dfd8
>>>>> [ 2162.499422]  ffff882fa6a84cb0 ffff882fa6781990 ffffffff815117a1
>>>>> ffff882fa6781990
>>>>> [ 2162.499427] Call Trace:
>>>>> [ 2162.499431]  [<ffffffff81511b8c>] ?
>>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>>> [ 2162.499435]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>>> [ 2162.499440]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>>> [ 2162.499445]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>> [ 2162.499450]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>> [ 2162.499455]  [<ffffffff81332c64>]
>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>> [ 2162.499460]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>> [ 2162.499464]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>>> [ 2162.499468]  [<ffffffff815137c0>] ? bch_btree_insert_node+0x2f0/0x2f0
>>>>> [ 2162.499473]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>>> [ 2162.499478]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>> [ 2162.499483]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>>> [ 2162.499488]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>>> [ 2162.499492]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>>> [ 2162.499497]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>> [ 2162.499502]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>> [ 2162.499507]  [<ffffffff81063570>] ?
>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>> [ 2162.499511]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>> [ 2162.499515]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>> [ 2162.499520]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>> [ 2162.499524]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>
>>>>>
>>>>>
>>>>>>> [  930.169234] bcache: bch_cached_dev_attach() Caching sda3 as bcache0
>>>>>>> on set 06496904-10ca-489a-ae75-68c6a07d3db1
>>>>>>> [ 2522.956188] INFO: task bcache_writebac:2058 blocked for more than
>>>>>>> 120 seconds.
>>>>>>> [ 2522.956199] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>> disables this message.
>>>>>>> [ 2522.956205] bcache_writebac D ffffffff81813a60     0  2058      2
>>>>>>> 0x00000000
>>>>>>> [ 2522.956212]  ffff881c39c1fcd8 0000000000000046 0000000000000001
>>>>>>> 0000000000000001
>>>>>>> [ 2522.956218]  ffff882f8b56b320 ffff881c39c1ffd8 ffff881c39c1ffd8
>>>>>>> ffff881c39c1ffd8
>>>>>>> [ 2522.956222]  ffff882fa6aeb320 ffff882f8b56b320 ffff882f8b56b320
>>>>>>> ffff882f8b56b320
>>>>>>> [ 2522.956227] Call Trace:
>>>>>>> [ 2522.956241]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>>> [ 2522.956247]  [<ffffffff816befb5>]
>>>>>>> rwsem_down_write_failed+0xf5/0x1a0
>>>>>>> [ 2522.956255]  [<ffffffff81332c93>]
>>>>>>> call_rwsem_down_write_failed+0x13/0x20
>>>>>>> [ 2522.956261]  [<ffffffff816bcc41>] ? down_write+0x31/0x40
>>>>>>> [ 2522.956269]  [<ffffffff8151b782>] bch_writeback_thread+0x62/0x7f0
>>>>>>> [ 2522.956278]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
>>>>>>> [ 2522.956284]  [<ffffffff8151b720>] ? write_dirty+0xc0/0xc0
>>>>>>> [ 2522.956290]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>>> [ 2522.956294]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>>> [ 2522.956302]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>>> [ 2522.956306]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>>> [ 2522.956313] INFO: task kworker/u64:1:2433 blocked for more than 120
>>>>>>> seconds.
>>>>>>> [ 2522.956317] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>> disables this message.
>>>>>>> [ 2522.956322] kworker/u64:1   D ffffffff81813a60     0  2433      2
>>>>>>> 0x00000000
>>>>>>> [ 2522.956332] Workqueue: writeback bdi_writeback_workfn (flush-252:0)
>>>>>>> [ 2522.956336]  ffff882f899d1560 0000000000000046 ffffffff81120635
>>>>>>> 0000000000000100
>>>>>>> [ 2522.956340]  ffff88219b488000 ffff882f899d1fd8 ffff882f899d1fd8
>>>>>>> ffff882f899d1fd8
>>>>>>> [ 2522.956345]  ffff882fa6aeb320 ffff88219b488000 ffffffff8131ad55
>>>>>>> ffff88219b488000
>>>>>>> [ 2522.956349] Call Trace:
>>>>>>> [ 2522.956359]  [<ffffffff81120635>] ? mempool_alloc_slab+0x15/0x20
>>>>>>> [ 2522.956367]  [<ffffffff8131ad55>] ? blk_throtl_bio+0x3b5/0x540
>>>>>>> [ 2522.956372]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>>> [ 2522.956378]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>>>> [ 2522.956383]  [<ffffffff81332c64>]
>>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>>> [ 2522.956388]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>>>> [ 2522.956394]  [<ffffffff8151fe32>]
>>>>>>> cached_dev_make_request+0x752/0xf00
>>>>>>> [ 2522.956400]  [<ffffffff81301af8>] ?
>>>>>>> generic_make_request_checks+0x1e8/0x3a0
>>>>>>> [ 2522.956405]  [<ffffffff81301d7a>] generic_make_request+0xca/0x100
>>>>>>> [ 2522.956409]  [<ffffffff81301e29>] submit_bio+0x79/0x160
>>>>>>> [ 2522.956417]  [<ffffffff811b1710>] ? bio_alloc_bioset+0xa0/0x1d0
>>>>>>> [ 2522.956421]  [<ffffffff811abd3f>] _submit_bh+0x13f/0x200
>>>>>>> [ 2522.956425]  [<ffffffff811abe10>] submit_bh+0x10/0x20
>>>>>>> [ 2522.956430]  [<ffffffff811af6b8>]
>>>>>>> __block_write_full_page+0x1d8/0x360
>>>>>>> [ 2522.956439]  [<ffffffff8145fc0b>] ? scsi_request_fn+0xbb/0x530
>>>>>>> [ 2522.956444]  [<ffffffff811ad3c0>] ?
>>>>>>> end_buffer_async_read+0x130/0x130
>>>>>>> [ 2522.956449]  [<ffffffff811b2c40>] ? I_BDEV+0x10/0x10
>>>>>>> [ 2522.956454]  [<ffffffff811b2c40>] ? I_BDEV+0x10/0x10
>>>>>>> [ 2522.956459]  [<ffffffff811af90a>]
>>>>>>> block_write_full_page_endio+0xca/0x100
>>>>>>> [ 2522.956464]  [<ffffffff811af955>] block_write_full_page+0x15/0x20
>>>>>>> [ 2522.956469]  [<ffffffff811b3608>] blkdev_writepage+0x18/0x20
>>>>>>> [ 2522.956473]  [<ffffffff81127d37>] __writepage+0x17/0x40
>>>>>>> [ 2522.956477]  [<ffffffff8112820e>] write_cache_pages+0x20e/0x460
>>>>>>> [ 2522.956481]  [<ffffffff81127d20>] ? set_page_dirty_lock+0x60/0x60
>>>>>>> [ 2522.956486]  [<ffffffff811ac251>] ? __set_page_dirty+0x71/0xc0
>>>>>>> [ 2522.956490]  [<ffffffff811284aa>] generic_writepages+0x4a/0x70
>>>>>>> [ 2522.956495]  [<ffffffff81129cd0>] do_writepages+0x20/0x40
>>>>>>> [ 2522.956501]  [<ffffffff811a36f5>]
>>>>>>> __writeback_single_inode+0x45/0x280
>>>>>>> [ 2522.956507]  [<ffffffff811269e2>] ?
>>>>>>> __alloc_pages_nodemask+0x152/0x9b0
>>>>>>> [ 2522.956513]  [<ffffffff811a495d>] writeback_sb_inodes+0x19d/0x3c0
>>>>>>> [ 2522.956518]  [<ffffffff811a4c1e>] __writeback_inodes_wb+0x9e/0xd0
>>>>>>> [ 2522.956523]  [<ffffffff811a4ecb>] wb_writeback+0x27b/0x320
>>>>>>> [ 2522.956529]  [<ffffffff81196228>] ? get_nr_dirty_inodes+0x58/0x80
>>>>>>> [ 2522.956534]  [<ffffffff811a500f>] wb_check_old_data_flush+0x9f/0xb0
>>>>>>> [ 2522.956539]  [<ffffffff811a67e1>] wb_do_writeback+0x151/0x1d0
>>>>>>> [ 2522.956547]  [<ffffffff8106539f>] ? set_worker_desc+0x6f/0x80
>>>>>>> [ 2522.956552]  [<ffffffff811a68da>] bdi_writeback_workfn+0x7a/0x200
>>>>>>> [ 2522.956557]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>>> [ 2522.956562]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>>> [ 2522.956567]  [<ffffffff81063570>] ?
>>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>>> [ 2522.956571]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>>> [ 2522.956575]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>>> [ 2522.956581]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>>> [ 2522.956585]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>>> [ 2522.956588] INFO: task iozone:2435 blocked for more than 120
>>>>>>> seconds.
>>>>>>> [ 2522.956593] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>> disables this message.
>>>>>>> [ 2522.956597] iozone          D 0000000000000001     0  2435   1545
>>>>>>> 0x00000000
>>>>>>> [ 2522.956602]  ffff882a75671978 0000000000000082 ffff882a75671938
>>>>>>> ffffffff81301d7a
>>>>>>> [ 2522.956606]  ffff882f88dc1990 ffff882a75671fd8 ffff882a75671fd8
>>>>>>> ffff882a75671fd8
>>>>>>> [ 2522.956611]  ffff882f8ec91990 ffff882f88dc1990 ffff882a75671978
>>>>>>> ffff882fbf2539f8
>>>>>>> [ 2522.956615] Call Trace:
>>>>>>> [ 2522.956619]  [<ffffffff81301d7a>] ? generic_make_request+0xca/0x100
>>>>>>> [ 2522.956625]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>>> [ 2522.956630]  [<ffffffff816be36f>] io_schedule+0x8f/0xd0
>>>>>>> [ 2522.956636]  [<ffffffff811b797c>]
>>>>>>> do_blockdev_direct_IO+0x1a7c/0x1fb0
>>>>>>> [ 2522.956645]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60
>>>>>>> [ext2]
>>>>>>> [ 2522.956652]  [<ffffffff811b7f05>] __blockdev_direct_IO+0x55/0x60
>>>>>>> [ 2522.956658]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60
>>>>>>> [ext2]
>>>>>>> [ 2522.956662]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>>>> [ 2522.956669]  [<ffffffffa0263349>] ext2_direct_IO+0x79/0xe0 [ext2]
>>>>>>> [ 2522.956674]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60
>>>>>>> [ext2]
>>>>>>> [ 2522.956680]  [<ffffffff8104ade6>] ? current_fs_time+0x16/0x60
>>>>>>> [ 2522.956685]  [<ffffffff8111f126>]
>>>>>>> generic_file_direct_write+0xc6/0x180
>>>>>>> [ 2522.956690]  [<ffffffff8111f4bd>]
>>>>>>> __generic_file_aio_write+0x2dd/0x3b0
>>>>>>> [ 2522.956696]  [<ffffffff8111f5f9>] generic_file_aio_write+0x69/0xd0
>>>>>>> [ 2522.956702]  [<ffffffff8117b88a>] do_sync_write+0x7a/0xb0
>>>>>>> [ 2522.956706]  [<ffffffff811bb108>] ? fsnotify+0x1f8/0x2b0
>>>>>>> [ 2522.956712]  [<ffffffff8117c63e>] vfs_write+0xce/0x1e0
>>>>>>> [ 2522.956716]  [<ffffffff8117cb22>] SyS_write+0x52/0xa0
>>>>>>> [ 2522.956722]  [<ffffffff816c7a02>] system_call_fastpath+0x16/0x1b
>>>>>>> [ 2522.956726] INFO: task kworker/2:1:2470 blocked for more than 120
>>>>>>> seconds.
>>>>>>> [ 2522.956730] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>> disables this message.
>>>>>>> [ 2522.956735] kworker/2:1     D ffffffff81813d40     0  2470      2
>>>>>>> 0x00000000
>>>>>>> [ 2522.956743] Workqueue: bcache bch_data_insert_keys
>>>>>>> [ 2522.956745]  ffff882a5b0c9628 0000000000000046 ffff882a5b0c9648
>>>>>>> 0000004000000000
>>>>>>> [ 2522.956749]  ffff882f8ec91990 ffff882a5b0c9fd8 ffff882a5b0c9fd8
>>>>>>> ffff882a5b0c9fd8
>>>>>>> [ 2522.956754]  ffff882fa6489990 ffff882f8ec91990 ffff882f8ec91990
>>>>>>> ffff8828dba00d98
>>>>>>> [ 2522.956758] Call Trace:
>>>>>>> [ 2522.956763]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>>> [ 2522.956769]  [<ffffffff816be57e>]
>>>>>>> schedule_preempt_disabled+0xe/0x10
>>>>>>> [ 2522.956774]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
>>>>>>> [ 2522.956779]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>>>>>> [ 2522.956783]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
>>>>>>> [ 2522.956791]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
>>>>>>> [ 2522.956799]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
>>>>>>> [ 2522.956805]  [<ffffffff81076828>] ? resched_task+0x68/0x70
>>>>>>> [ 2522.956810]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
>>>>>>> [ 2522.956816]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
>>>>>>> [ 2522.956822]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
>>>>>>> [ 2522.956827]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
>>>>>>> [ 2522.956833]  [<ffffffff81126e9c>]
>>>>>>> __alloc_pages_nodemask+0x60c/0x9b0
>>>>>>> [ 2522.956843]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
>>>>>>> [ 2522.956848]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
>>>>>>> [ 2522.956852]  [<ffffffff8150ebb3>] mca_data_alloc+0x73/0x1d0
>>>>>>> [ 2522.956857]  [<ffffffff8150ee5a>] mca_bucket_alloc+0x14a/0x1f0
>>>>>>> [ 2522.956861]  [<ffffffff81511020>] mca_alloc+0x360/0x470
>>>>>>> [ 2522.956866]  [<ffffffff81517031>] ? __bch_bset_search+0x1d1/0x480
>>>>>>> [ 2522.956871]  [<ffffffff8151183e>] bch_btree_node_get+0x10e/0x280
>>>>>>> [ 2522.956875]  [<ffffffff81511c02>]
>>>>>>> bch_btree_map_nodes_recurse+0xe2/0x170
>>>>>>> [ 2522.956880]  [<ffffffff815137c0>] ?
>>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>>> [ 2522.956886]  [<ffffffff811b51ea>] ? dio_bio_end_io+0x5a/0x90
>>>>>>> [ 2522.956890]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>>>> [ 2522.956895]  [<ffffffff81514dce>] __bch_btree_map_nodes+0x13e/0x1c0
>>>>>>> [ 2522.956899]  [<ffffffff815137c0>] ?
>>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>>> [ 2522.956905]  [<ffffffff8151ac6f>] ? bch_journal+0x42f/0x4b0
>>>>>>> [ 2522.956909]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>>>> [ 2522.956915]  [<ffffffff8151e1be>] bch_data_insert_keys+0x3e/0x160
>>>>>>> [ 2522.956920]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>>> [ 2522.956925]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>>> [ 2522.956930]  [<ffffffff81063570>] ?
>>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>>> [ 2522.956933]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>>> [ 2522.956938]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>>> [ 2522.956943]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>>> [ 2522.956947]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>>> [ 2522.956951] INFO: task kworker/3:2:2471 blocked for more than 120
>>>>>>> seconds.
>>>>>>> [ 2522.956955] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>> disables this message.
>>>>>>> [ 2522.956959] kworker/3:2     D ffffffff81813a60     0  2471      2
>>>>>>> 0x00000000
>>>>>>> [ 2522.956966] Workqueue: events update_writeback_rate
>>>>>>> [ 2522.956968]  ffff882a5d125cd0 0000000000000046 0000000000000003
>>>>>>> ffff882a5d125d68
>>>>>>> [ 2522.956972]  ffff882f8ec94cb0 ffff882a5d125fd8 ffff882a5d125fd8
>>>>>>> ffff882a5d125fd8
>>>>>>> [ 2522.956977]  ffff882fa6aeb320 ffff882f8ec94cb0 0000000200000003
>>>>>>> ffff882f8ec94cb0
>>>>>>> [ 2522.956981] Call Trace:
>>>>>>> [ 2522.956987]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>>> [ 2522.956992]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>>>> [ 2522.956997]  [<ffffffff81332c64>]
>>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>>> [ 2522.957002]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>>>> [ 2522.957007]  [<ffffffff8151b010>] update_writeback_rate+0x30/0x230
>>>>>>> [ 2522.957011]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>>> [ 2522.957017]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>>> [ 2522.957021]  [<ffffffff81063570>] ?
>>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>>> [ 2522.957025]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>>> [ 2522.957030]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>>> [ 2522.957035]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>>> [ 2522.957039]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>>> [ 2522.957048] INFO: task kworker/u64:2:2540 blocked for more than 120
>>>>>>> seconds.
>>>>>>> [ 2522.957052] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>> disables this message.
>>>>>>> [ 2522.957056] kworker/u64:2   D ffffffff81813a60     0  2540      2
>>>>>>> 0x00000000
>>>>>>> [ 2522.957062] Workqueue: bch_btree_io btree_node_write_work
>>>>>>> [ 2522.957064]  ffff882a4fdc9778 0000000000000046 ffff882a4fdc9738
>>>>>>> ffff882f86e3be58
>>>>>>> [ 2522.957069]  ffff882fa4bc1990 ffff882a4fdc9fd8 ffff882a4fdc9fd8
>>>>>>> ffff882a4fdc9fd8
>>>>>>> [ 2522.957073]  ffff882fa6aeb320 ffff882fa4bc1990 ffff882f86e3be00
>>>>>>> ffff8828dba00d98
>>>>>>> [ 2522.957078] Call Trace:
>>>>>>> [ 2522.957083]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>>> [ 2522.957088]  [<ffffffff816be57e>]
>>>>>>> schedule_preempt_disabled+0xe/0x10
>>>>>>> [ 2522.957092]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
>>>>>>> [ 2522.957102]  [<ffffffff81484090>] ?
>>>>>>> ata_scsiop_mode_sense+0x380/0x380
>>>>>>> [ 2522.957107]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>>>>>> [ 2522.957111]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
>>>>>>> [ 2522.957116]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
>>>>>>> [ 2522.957121]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
>>>>>>> [ 2522.957126]  [<ffffffff81076828>] ? resched_task+0x68/0x70
>>>>>>> [ 2522.957130]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
>>>>>>> [ 2522.957135]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
>>>>>>> [ 2522.957140]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
>>>>>>> [ 2522.957146]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
>>>>>>> [ 2522.957152]  [<ffffffff81126e9c>]
>>>>>>> __alloc_pages_nodemask+0x60c/0x9b0
>>>>>>> [ 2522.957158]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
>>>>>>> [ 2522.957163]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
>>>>>>> [ 2522.957168]  [<ffffffff81517fc8>] __btree_sort+0x48/0x230
>>>>>>> [ 2522.957173]  [<ffffffff8151765c>] ? __bch_btree_iter_init+0x7c/0xc0
>>>>>>> [ 2522.957178]  [<ffffffff81518301>]
>>>>>>> bch_btree_sort_partial+0x101/0x120
>>>>>>> [ 2522.957182]  [<ffffffff8150f000>] ?
>>>>>>> __btree_node_write_done+0x100/0x100
>>>>>>> [ 2522.957187]  [<ffffffff81518468>] bch_btree_sort_lazy+0x68/0x90
>>>>>>> [ 2522.957191]  [<ffffffff815109ba>] bch_btree_node_write+0x36a/0x4a0
>>>>>>> [ 2522.957196]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
>>>>>>> [ 2522.957201]  [<ffffffff8106212c>] ?
>>>>>>> pwq_activate_delayed_work+0x4c/0xb0
>>>>>>> [ 2522.957205]  [<ffffffff81511477>] btree_node_write_work+0x57/0x80
>>>>>>> [ 2522.957210]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>>> [ 2522.957215]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>>> [ 2522.957219]  [<ffffffff81063570>] ?
>>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>>> [ 2522.957223]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>>> [ 2522.957228]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>>> [ 2522.957233]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>>> [ 2522.957237]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>>>
>>>>>>>
>>>>>>> I can easily reproduce this, please let me know if you require any
>>>>>>> info/test any patch.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Neo
>>>>>>>
>>>>>>> On Mon, Aug 26, 2013 at 12:21 PM, Stefan Priebe
>>>>>>> <s.priebe@profihost.ag> wrote:
>>>>>>>>
>>>>>>>> Hi Kent,
>>>>>>>>
>>>>>>>> a new one under 3.10:
>>>>>>>>
>>>>>>>> 2013-08-26 21:05:30     INFO: task ceph-osd:8939 blocked for more
>>>>>>>> than 120
>>>>>>>> seconds.
>>>>>>>> 2013-08-26 21:05:30     "echo 0 >
>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>> disables this message.
>>>>>>>> 2013-08-26 21:05:30     ceph-osd D ffffffff8160f760 0 8939 1
>>>>>>>> 0x00000000
>>>>>>>> 2013-08-26 21:05:30     ffff880bd9a61820 0000000000000086
>>>>>>>> ffff880bd9a61fd8
>>>>>>>> 0000000000012c80
>>>>>>>> 2013-08-26 21:05:30     ffff880bd9a60010 0000000000012c80
>>>>>>>> 0000000000012c80
>>>>>>>> 0000000000012c80
>>>>>>>> 2013-08-26 21:05:30     ffff880bd9a61fd8 0000000000012c80
>>>>>>>> ffff880bd7abb1c0
>>>>>>>> ffff880c48dee380
>>>>>>>> 2013-08-26 21:05:29     [<ffffffff811e8128>]
>>>>>>>> xfs_vm_readpages+0x18/0x20
>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810f1da3>] read_pages+0x43/0x100
>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810e64df>] ?
>>>>>>>> __page_cache_alloc+0x9f/0xc0
>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810f1fab>]
>>>>>>>> __do_page_cache_readahead+0x14b/0x160
>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810f1fdc>] ra_submit+0x1c/0x20
>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810f22a5>]
>>>>>>>> ondemand_readahead+0x115/0x240
>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810f249e>]
>>>>>>>> page_cache_sync_readahead+0x2e/0x40
>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810e7a5e>] T.1003+0x33e/0x430
>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810e7c30>]
>>>>>>>> generic_file_aio_read+0xe0/0x220
>>>>>>>> 2013-08-26 21:05:29     [<ffffffff811efd0a>]
>>>>>>>> xfs_file_aio_read+0x15a/0x2a0
>>>>>>>> 2013-08-26 21:05:29     [<ffffffff8114095a>] do_sync_read+0x7a/0xb0
>>>>>>>> 2013-08-26 21:05:29     [<ffffffff81140df1>] vfs_read+0xb1/0x130
>>>>>>>> 2013-08-26 21:05:29     [<ffffffff81141757>] SyS_pread64+0x97/0xa0
>>>>>>>> 2013-08-26 21:05:29     [<ffffffff81526b12>]
>>>>>>>> system_call_fastpath+0x16/0x1b
>>>>>>>> 2013-08-26 21:05:29     INFO: task ceph-osd:8896 blocked for more
>>>>>>>> than 120
>>>>>>>> seconds.
>>>>>>>> 2013-08-26 21:05:29     "echo 0 >
>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>> disables this message.
>>>>>>>> 2013-08-26 21:05:29     ceph-osd D ffffffff8160f760 0 8896 1
>>>>>>>> 0x00000000
>>>>>>>> 2013-08-26 21:05:29     ffff880bd9a9dc28 0000000000000086
>>>>>>>> ffff880bd9a9dfd8
>>>>>>>> 0000000000012c80
>>>>>>>> 2013-08-26 21:05:29     ffff880bd9a9c010 0000000000012c80
>>>>>>>> 0000000000012c80
>>>>>>>> 0000000000012c80
>>>>>>>> 2013-08-26 21:05:29     ffff880bd9a9dfd8 0000000000012c80
>>>>>>>> ffff880bd79298e0
>>>>>>>> ffff880c48dd6380
>>>>>>>> 2013-08-26 21:05:29     Call Trace:
>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810e60f0>] ? __lock_page+0x70/0x70
>>>>>>>> 2013-08-26 21:05:28     "echo 0 >
>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>> disables this message.
>>>>>>>> 2013-08-26 21:05:28     ceph-osd D ffffffff8160f760 0 8798 1
>>>>>>>> 0x00000000
>>>>>>>> 2013-08-26 21:05:28     ffff880bd998b740 0000000000000086
>>>>>>>> ffff880bd998bfd8
>>>>>>>> 0000000000012c80
>>>>>>>> 2013-08-26 21:05:28     ffff880bd998a010 0000000000012c80
>>>>>>>> 0000000000012c80
>>>>>>>> 0000000000012c80
>>>>>>>> 2013-08-26 21:05:28     ffff880bd998bfd8 0000000000012c80
>>>>>>>> ffff880bd99c0000
>>>>>>>> ffff880c48dd6380
>>>>>>>> 2013-08-26 21:05:28     Call Trace:
>>>>>>>> 2013-08-26 21:05:28     [<ffffffff81524874>] schedule+0x24/0x70
>>>>>>>> 2013-08-26 21:05:28     [<ffffffff81525d7d>]
>>>>>>>> rwsem_down_read_failed+0x9d/0xe5
>>>>>>>> 2013-08-26 21:05:28     [<ffffffff812aa0a4>]
>>>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>>>> 2013-08-26 21:05:28     [<ffffffff81523a42>] ? down_read+0x12/0x20
>>>>>>>> 2013-08-26 21:05:28     [<ffffffffa01ead62>]
>>>>>>>> btree_read_async+0xa2/0x1a0
>>>>>>>> [bcache]
>>>>>>>> 2013-08-26 21:05:28     [<ffffffffa01f7b73>] closure_queue+0x43/0x60
>>>>>>>> [bcache]
>>>>>>>> 2013-08-26 21:05:28     [<ffffffffa01eb903>] T.1050+0x63/0x70
>>>>>>>> [bcache]
>>>>>>>> 2013-08-26 21:05:28     [<ffffffffa01ed6ea>]
>>>>>>>> cached_dev_make_request+0x20a/0x350 [bcache]
>>>>>>>> 2013-08-26 21:05:28     [<ffffffff81281952>]
>>>>>>>> generic_make_request+0xc2/0x100
>>>>>>>> 2013-08-26 21:05:28     [<ffffffff812819f7>] submit_bio+0x67/0x130
>>>>>>>> 2013-08-26 21:05:28     [<ffffffff8117b240>]
>>>>>>>> do_mpage_readpage+0x2a0/0x660
>>>>>>>> 2013-08-26 21:05:28     [<ffffffff81101cee>] ?
>>>>>>>> __inc_zone_page_state+0x2e/0x30
>>>>>>>> 2013-08-26 21:05:28     [<ffffffff810e6581>] ?
>>>>>>>> add_to_page_cache_locked+0x81/0x110
>>>>>>>> 2013-08-26 21:05:28     [<ffffffff8117b76a>]
>>>>>>>> mpage_readpages+0xfa/0x150
>>>>>>>> 2013-08-26 21:05:28     [<ffffffff811e90c0>] ?
>>>>>>>> xfs_get_blocks_direct+0x20/0x20
>>>>>>>> 2013-08-26 21:05:28     [<ffffffff811e90c0>] ?
>>>>>>>> xfs_get_blocks_direct+0x20/0x20
>>>>>>>> 2013-08-26 21:05:28     [<ffffffff810eeda3>] ?
>>>>>>>> __alloc_pages_nodemask+0x123/0x240
>>>>>>>> 2013-08-26 21:05:27     [<ffffffff8115bf93>] ? __close_fd+0x43/0x80
>>>>>>>> 2013-08-26 21:05:27     [<ffffffff8116dabe>] SyS_fdatasync+0xe/0x20
>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81526b12>]
>>>>>>>> system_call_fastpath+0x16/0x1b
>>>>>>>> 2013-08-26 21:05:27     INFO: task ceph-osd:8746 blocked for more
>>>>>>>> than 120
>>>>>>>> seconds.
>>>>>>>> 2013-08-26 21:05:27     "echo 0 >
>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>> disables this message.
>>>>>>>> 2013-08-26 21:05:27     ceph-osd D ffffffff8160f760 0 8746 1
>>>>>>>> 0x00000000
>>>>>>>> 2013-08-26 21:05:27     ffff880c2dde1d38 0000000000000086
>>>>>>>> ffff880c2dde1fd8
>>>>>>>> 0000000000012c80
>>>>>>>> 2013-08-26 21:05:27     ffff880c2dde0010 0000000000012c80
>>>>>>>> 0000000000012c80
>>>>>>>> 0000000000012c80
>>>>>>>> 2013-08-26 21:05:27     ffff880c2dde1fd8 0000000000012c80
>>>>>>>> ffff880c239eb1c0
>>>>>>>> ffff880c48dd6380
>>>>>>>> 2013-08-26 21:05:27     Call Trace:
>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81524874>] schedule+0x24/0x70
>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81522bbd>]
>>>>>>>> schedule_timeout+0x16d/0x200
>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81068eba>] ?
>>>>>>>> __queue_delayed_work+0xaa/0x1a0
>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81068d19>] ?
>>>>>>>> try_to_grab_pending+0x109/0x190
>>>>>>>> 2013-08-26 21:05:27     [<ffffffff815251e5>]
>>>>>>>> wait_for_completion+0x95/0x110
>>>>>>>> 2013-08-26 21:05:27     [<ffffffff8107fd00>] ?
>>>>>>>> try_to_wake_up+0x2a0/0x2a0
>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81167797>] ?
>>>>>>>> bdi_queue_work+0x77/0xc0
>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81167863>]
>>>>>>>> writeback_inodes_sb_nr+0x83/0xb0
>>>>>>>> 2013-08-26 21:05:27     [<ffffffff811678ea>]
>>>>>>>> writeback_inodes_sb+0x5a/0x70
>>>>>>>> 2013-08-26 21:05:27     [<ffffffff8116dd0a>]
>>>>>>>> __sync_filesystem+0x4a/0x50
>>>>>>>> 2013-08-26 21:05:27     [<ffffffff8116dd42>]
>>>>>>>> sync_filesystem+0x32/0x60
>>>>>>>> 2013-08-26 21:05:27     [<ffffffff8116ddc0>] SyS_syncfs+0x50/0x90
>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81526b12>]
>>>>>>>> system_call_fastpath+0x16/0x1b
>>>>>>>> 2013-08-26 21:05:27     INFO: task ceph-osd:8798 blocked for more
>>>>>>>> than 120
>>>>>>>> seconds.
>>>>>>>>
>>>>>>>> Stefan
>>>>>>>>
>>>>>>>> Am 22.08.2013 09:32, schrieb Stefan Priebe - Profihost AG:
>>>>>>>>
>>>>>>>>> great!
>>>>>>>>>
>>>>>>>>> Everything seems to work fine now! Except read_dirty always going to
>>>>>>>>> negative values after a reboot.
>>>>>>>>>
>>>>>>>>> Stefan
>>>>>>>>>
>>>>>>>>> Am 22.08.2013 08:02, schrieb Kent Overstreet:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 22, 2013 at 07:59:04AM +0200, Stefan Priebe wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> schedule_timeout() is not the same as
>>>>>>>>>>>> schedule_timeout_interruptible().
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> just search and replace? So i can try on my own.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The one in read_dirty(), line ~330
>>>>>>>>>> --
>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>>> linux-bcache"
>>>>>>>>>> in
>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>>>
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>> linux-bcache" in
>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe linux-bcache"
>>>>> in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-28 20:15                                         ` Stefan Priebe
@ 2013-08-28 22:45                                           ` kernel neophyte
  2013-08-29 18:52                                           ` Stefan Priebe
  1 sibling, 0 replies; 28+ messages in thread
From: kernel neophyte @ 2013-08-28 22:45 UTC (permalink / raw)
  To: Stefan Priebe
  Cc: Kent Overstreet, Jens Axboe, linux-kernel@vger.kernel.org,
	linux-bcache@vger.kernel.org

On Wed, Aug 28, 2013 at 1:15 PM, Stefan Priebe <s.priebe@profihost.ag> wrote:
> sorry but that's completely wrong.
>
> please use branch bcache-for-3.10
>
> http://evilpiepirate.org/git/linux-bcache.git/log/?h=bcache-for-3.10

nope :-( doesn't help I still see deadlocks:

[  437.123184] bcache: bch_cached_dev_attach() Caching sda3 as bcache0
on set a9ad97be-0918-427a-8d2a-db6e0a2c0e7a
[ 1442.355992] INFO: task kworker/0:0:4 blocked for more than 120 seconds.
[ 1442.356003] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1442.356009] kworker/0:0     D 0000000000000000     0     4      2 0x00000000
[ 1442.356023] Workqueue: bch_allocator bch_allocator_thread
[ 1442.356027]  ffff882fa6a85c98 0000000000000046 ffff882fa6a85cb8
ffffffff8107fd3d
[ 1442.356033]  ffff882fa6a64560 ffff882fa6a85fd8 ffff882fa6a85fd8
ffff882fa6a85fd8
[ 1442.356037]  ffff882fa6a62e40 ffff882fa6a64560 ffff882fa5f85c80
ffff88211ce60db0
[ 1442.356042] Call Trace:
[ 1442.356053]  [<ffffffff8107fd3d>] ? dequeue_task_fair+0x2cd/0x530
[ 1442.356063]  [<ffffffff816ba149>] schedule+0x29/0x70
[ 1442.356068]  [<ffffffff816ba42e>] schedule_preempt_disabled+0xe/0x10
[ 1442.356073]  [<ffffffff816b8722>] __mutex_lock_slowpath+0x112/0x1b0
[ 1442.356078]  [<ffffffff816b829a>] mutex_lock+0x2a/0x50
[ 1442.356082]  [<ffffffff81509432>] bch_allocator_thread+0x112/0xf30
[ 1442.356089]  [<ffffffff8106a420>] ? add_wait_queue+0x60/0x60
[ 1442.356096]  [<ffffffff81062454>] process_one_work+0x174/0x490
[ 1442.356101]  [<ffffffff8106351b>] worker_thread+0x11b/0x370
[ 1442.356106]  [<ffffffff81063400>] ? manage_workers.isra.21+0x2d0/0x2d0
[ 1442.356110]  [<ffffffff81069c60>] kthread+0xc0/0xd0
[ 1442.356115]  [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0
[ 1442.356122]  [<ffffffff816c381c>] ret_from_fork+0x7c/0xb0
[ 1442.356126]  [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0
[ 1442.356132] INFO: task kworker/1:1:47 blocked for more than 120 seconds.
[ 1442.356136] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1442.356140] kworker/1:1     D ffffffff8180c0c0     0    47      2 0x00000000
[ 1442.356150] Workqueue: bcache bch_journal
[ 1442.356153]  ffff882fa542d4f8 0000000000000046 ffff882fa542d518
0000000800000000
[ 1442.356157]  ffff882fa5f85c80 ffff882fa542dfd8 ffff882fa542dfd8
ffff882fa542dfd8
[ 1442.356162]  ffff882fa6439720 ffff882fa5f85c80 ffff882fa5f85c80
ffff88211ce60db0
[ 1442.356167] Call Trace:
[ 1442.356172]  [<ffffffff816ba149>] schedule+0x29/0x70
[ 1442.356177]  [<ffffffff816ba42e>] schedule_preempt_disabled+0xe/0x10
[ 1442.356182]  [<ffffffff816b8722>] __mutex_lock_slowpath+0x112/0x1b0
[ 1442.356187]  [<ffffffff816b829a>] mutex_lock+0x2a/0x50
[ 1442.356191]  [<ffffffff8150d6b5>] bch_mca_shrink+0x1c5/0x310
[ 1442.356198]  [<ffffffff8117ba02>] ? prune_super+0x162/0x1b0
[ 1442.356206]  [<ffffffff8112d054>] shrink_slab+0x154/0x300
[ 1442.356213]  [<ffffffff81076118>] ? resched_task+0x68/0x70
[ 1442.356218]  [<ffffffff81076a45>] ? check_preempt_curr+0x75/0xa0
[ 1442.356225]  [<ffffffff811387c9>] ? fragmentation_index+0x19/0x70
[ 1442.356231]  [<ffffffff8112f8af>] do_try_to_free_pages+0x20f/0x4b0
[ 1442.356236]  [<ffffffff8112fd04>] try_to_free_pages+0xe4/0x1a0
[ 1442.356245]  [<ffffffff8112536c>] __alloc_pages_nodemask+0x60c/0x990
[ 1442.356249]  [<ffffffff81076118>] ? resched_task+0x68/0x70
[ 1442.356257]  [<ffffffff8115e94a>] alloc_pages_current+0xba/0x170
[ 1442.356263]  [<ffffffff81120a5e>] __get_free_pages+0xe/0x40
[ 1442.356268]  [<ffffffff8150ac73>] mca_data_alloc+0x73/0x1d0
[ 1442.356272]  [<ffffffff8150b2ba>] mca_bucket_alloc+0x14a/0x1f0
[ 1442.356276]  [<ffffffff8150d3a0>] mca_alloc+0x320/0x470
[ 1442.356281]  [<ffffffff8150e37b>] bch_btree_node_alloc+0x9b/0x200
[ 1442.356285]  [<ffffffff81080d95>] ? enqueue_entity+0x1d5/0xa90
[ 1442.356290]  [<ffffffff81510099>] btree_split+0x119/0x770
[ 1442.356298]  [<ffffffff81025018>] ? native_smp_send_reschedule+0x48/0x60
[ 1442.356303]  [<ffffffff81513cac>] ? __bch_bset_search+0x3fc/0x4a0
[ 1442.356307]  [<ffffffff81080d95>] ? enqueue_entity+0x1d5/0xa90
[ 1442.356312]  [<ffffffff81510adb>] bch_btree_insert_recurse+0x3eb/0x500
[ 1442.356316]  [<ffffffff81510a78>] bch_btree_insert_recurse+0x388/0x500
[ 1442.356321]  [<ffffffff81510d46>] bch_btree_insert+0x156/0x330
[ 1442.356327]  [<ffffffff8151ce9a>] bch_btree_insert_async+0x1a/0xe0
[ 1442.356332]  [<ffffffff81517ba5>] bch_journal+0x45/0x260
[ 1442.356337]  [<ffffffff81062454>] process_one_work+0x174/0x490
[ 1442.356342]  [<ffffffff8106351b>] worker_thread+0x11b/0x370
[ 1442.356347]  [<ffffffff81063400>] ? manage_workers.isra.21+0x2d0/0x2d0
[ 1442.356350]  [<ffffffff81069c60>] kthread+0xc0/0xd0
[ 1442.356355]  [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0
[ 1442.356360]  [<ffffffff816c381c>] ret_from_fork+0x7c/0xb0
[ 1442.356364]  [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0
[ 1442.356373] INFO: task kworker/u64:7:349 blocked for more than 120 seconds.
[ 1442.356377] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1442.356382] kworker/u64:7   D ffffffff8180bde0     0   349      2 0x00000000
[ 1442.356390] Workqueue: bcache_writeback write_dirty_finish
[ 1442.356392]  ffff882f88e8ba90 0000000000000046 ffff882f88e8ba48
ffffffff81138789
[ 1442.356397]  ffff882f88c8dc80 ffff882f88e8bfd8 ffff882f88e8bfd8
ffff882f88e8bfd8
[ 1442.356401]  ffffffff81c10440 ffff882f88c8dc80 ffff882f88e8bbd3
ffff882f88c8dc80
[ 1442.356406] Call Trace:
[ 1442.356410]  [<ffffffff81138789>] ? zone_statistics+0x99/0xc0
[ 1442.356415]  [<ffffffff816ba149>] schedule+0x29/0x70
[ 1442.356421]  [<ffffffff816baf9d>] rwsem_down_read_failed+0x9d/0xe5
[ 1442.356429]  [<ffffffff8132eae4>] call_rwsem_down_read_failed+0x14/0x30
[ 1442.356434]  [<ffffffff816b8b34>] ? down_read+0x24/0x2b
[ 1442.356438]  [<ffffffff81510d00>] bch_btree_insert+0x110/0x330
[ 1442.356444]  [<ffffffff8151891a>] write_dirty_finish+0x27a/0x2c0
[ 1442.356450]  [<ffffffff81062454>] process_one_work+0x174/0x490
[ 1442.356454]  [<ffffffff8106351b>] worker_thread+0x11b/0x370
[ 1442.356459]  [<ffffffff81063400>] ? manage_workers.isra.21+0x2d0/0x2d0
[ 1442.356463]  [<ffffffff81069c60>] kthread+0xc0/0xd0
[ 1442.356467]  [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0
[ 1442.356472]  [<ffffffff816c381c>] ret_from_fork+0x7c/0xb0
[ 1442.356476]  [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0
[ 1442.356485] INFO: task kworker/0:2:1967 blocked for more than 120 seconds.
[ 1442.356489] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1442.356493] kworker/0:2     D 0000000000000000     0  1967      2 0x00000000
[ 1442.356500] Workqueue: bcache bch_journal
[ 1442.356502]  ffff882f890b7bf0 0000000000000046 0000000100000000
ffff882fbf273140
[ 1442.356507]  ffff882f88d8ae40 ffff882f890b7fd8 ffff882f890b7fd8
ffff882f890b7fd8
[ 1442.356511]  ffff882f924d4560 ffff882f88d8ae40 ffffffff81516b08
ffff882f88d8ae40
[ 1442.356516] Call Trace:
[ 1442.356521]  [<ffffffff81516b08>] ? journal_write_endio+0x38/0x40
[ 1442.356526]  [<ffffffff816ba149>] schedule+0x29/0x70
[ 1442.356530]  [<ffffffff816baf9d>] rwsem_down_read_failed+0x9d/0xe5
[ 1442.356536]  [<ffffffff81053a90>] ? internal_add_timer+0x20/0x50
[ 1442.356541]  [<ffffffff8132eae4>] call_rwsem_down_read_failed+0x14/0x30
[ 1442.356546]  [<ffffffff816b8b34>] ? down_read+0x24/0x2b
[ 1442.356550]  [<ffffffff81510d00>] bch_btree_insert+0x110/0x330
[ 1442.356556]  [<ffffffff8151ce9a>] bch_btree_insert_async+0x1a/0xe0
[ 1442.356561]  [<ffffffff81517ba5>] bch_journal+0x45/0x260
[ 1442.356566]  [<ffffffff81062454>] process_one_work+0x174/0x490
[ 1442.356570]  [<ffffffff8106351b>] worker_thread+0x11b/0x370
[ 1442.356575]  [<ffffffff81063400>] ? manage_workers.isra.21+0x2d0/0x2d0
[ 1442.356579]  [<ffffffff81069c60>] kthread+0xc0/0xd0
[ 1442.356583]  [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0
[ 1442.356588]  [<ffffffff816c381c>] ret_from_fork+0x7c/0xb0
[ 1442.356592]  [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0
[ 1442.356595] INFO: task kworker/u64:0:1980 blocked for more than 120 seconds.
[ 1442.356600] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1442.356604] kworker/u64:0   D ffffffff8180bde0     0  1980      2 0x00000000
[ 1442.356610] Workqueue: bch_btree_io btree_write_work
[ 1442.356612]  ffff8822b4447cc8 0000000000000046 ffff8822b4447ca8
ffffffff81514c30
[ 1442.356616]  ffff882f924d4560 ffff8822b4447fd8 ffff8822b4447fd8
ffff8822b4447fd8
[ 1442.356621]  ffff882fa6ab0000 ffff882f924d4560 0000000000000000
ffff882f924d4560
[ 1442.356625] Call Trace:
[ 1442.356630]  [<ffffffff81514c30>] ? __btree_sort+0x160/0x250
[ 1442.356635]  [<ffffffff816ba149>] schedule+0x29/0x70
[ 1442.356640]  [<ffffffff816bae55>] rwsem_down_write_failed+0xf5/0x1a0
[ 1442.356645]  [<ffffffff8132eb13>] call_rwsem_down_write_failed+0x13/0x20
[ 1442.356649]  [<ffffffff816b8b01>] ? down_write+0x31/0x40
[ 1442.356654]  [<ffffffff8150bfbf>] btree_write_work+0x2f/0x60
[ 1442.356658]  [<ffffffff81062454>] process_one_work+0x174/0x490
[ 1442.356663]  [<ffffffff8106351b>] worker_thread+0x11b/0x370
[ 1442.356668]  [<ffffffff81063400>] ? manage_workers.isra.21+0x2d0/0x2d0
[ 1442.356671]  [<ffffffff81069c60>] kthread+0xc0/0xd0
[ 1442.356676]  [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0
[ 1442.356680]  [<ffffffff816c381c>] ret_from_fork+0x7c/0xb0
[ 1442.356684]  [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0
[ 1442.356688] INFO: task iozone:1993 blocked for more than 120 seconds.
[ 1442.356692] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1442.356696] iozone          D 0000000000000000     0  1993   1470 0x00000000
[ 1442.356700]  ffff881a3c981978 0000000000000082 ffff881a3c981938
ffffffff812fdc7a
[ 1442.356705]  ffff882f88d89720 ffff881a3c981fd8 ffff881a3c981fd8
ffff881a3c981fd8
[ 1442.356710]  ffff882fa5f85c80 ffff882f88d89720 ffff881a3c981978
ffff882fbf2339f8
[ 1442.356714] Call Trace:
[ 1442.356721]  [<ffffffff812fdc7a>] ? generic_make_request+0xca/0x100
[ 1442.356726]  [<ffffffff816ba149>] schedule+0x29/0x70
[ 1442.356731]  [<ffffffff816ba21f>] io_schedule+0x8f/0xd0
[ 1442.356740]  [<ffffffff811b375c>] do_blockdev_direct_IO+0x1a7c/0x1fb0
[ 1442.356751]  [<ffffffffa02e3a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
[ 1442.356757]  [<ffffffff811b3ce5>] __blockdev_direct_IO+0x55/0x60
[ 1442.356763]  [<ffffffffa02e3a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
[ 1442.356769]  [<ffffffffa02e4349>] ext2_direct_IO+0x79/0xe0 [ext2]
[ 1442.356775]  [<ffffffffa02e3a80>] ? ext2_get_blocks+0xa60/0xa60 [ext2]
[ 1442.356781]  [<ffffffff8104ad96>] ? current_fs_time+0x16/0x60
[ 1442.356787]  [<ffffffff8111d7b6>] generic_file_direct_write+0xc6/0x180
[ 1442.356793]  [<ffffffff8111db4d>] __generic_file_aio_write+0x2dd/0x3b0
[ 1442.356798]  [<ffffffff8111dc89>] generic_file_aio_write+0x69/0xd0
[ 1442.356803]  [<ffffffff8117767a>] do_sync_write+0x7a/0xb0
[ 1442.356808]  [<ffffffff8117842e>] vfs_write+0xce/0x1e0
[ 1442.356812]  [<ffffffff81178912>] SyS_write+0x52/0xa0
[ 1442.356818]  [<ffffffff816c38c2>] system_call_fastpath+0x16/0x1b
[ 1562.355919] INFO: task kworker/0:0:4 blocked for more than 120 seconds.
[ 1562.355927] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1562.355935] kworker/0:0     D 0000000000000000     0     4      2 0x00000000
[ 1562.355945] Workqueue: bch_allocator bch_allocator_thread
[ 1562.355949]  ffff882fa6a85c98 0000000000000046 ffff882fa6a85cb8
ffffffff8107fd3d
[ 1562.355954]  ffff882fa6a64560 ffff882fa6a85fd8 ffff882fa6a85fd8
ffff882fa6a85fd8
[ 1562.355959]  ffff882fa6a62e40 ffff882fa6a64560 ffff882fa5f85c80
ffff88211ce60db0
[ 1562.355963] Call Trace:
[ 1562.355971]  [<ffffffff8107fd3d>] ? dequeue_task_fair+0x2cd/0x530
[ 1562.355977]  [<ffffffff816ba149>] schedule+0x29/0x70
[ 1562.355982]  [<ffffffff816ba42e>] schedule_preempt_disabled+0xe/0x10
[ 1562.355987]  [<ffffffff816b8722>] __mutex_lock_slowpath+0x112/0x1b0
[ 1562.355992]  [<ffffffff816b829a>] mutex_lock+0x2a/0x50
[ 1562.355996]  [<ffffffff81509432>] bch_allocator_thread+0x112/0xf30
[ 1562.356001]  [<ffffffff8106a420>] ? add_wait_queue+0x60/0x60
[ 1562.356006]  [<ffffffff81062454>] process_one_work+0x174/0x490
[ 1562.356011]  [<ffffffff8106351b>] worker_thread+0x11b/0x370
[ 1562.356016]  [<ffffffff81063400>] ? manage_workers.isra.21+0x2d0/0x2d0
[ 1562.356020]  [<ffffffff81069c60>] kthread+0xc0/0xd0
[ 1562.356025]  [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0
[ 1562.356030]  [<ffffffff816c381c>] ret_from_fork+0x7c/0xb0
[ 1562.356035]  [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0
[ 1562.356040] INFO: task kworker/1:1:47 blocked for more than 120 seconds.
[ 1562.356045] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1562.356051] kworker/1:1     D ffffffff8180c0c0     0    47      2 0x00000000
[ 1562.356059] Workqueue: bcache bch_journal
[ 1562.356061]  ffff882fa542d4f8 0000000000000046 ffff882fa542d518
0000000800000000
[ 1562.356066]  ffff882fa5f85c80 ffff882fa542dfd8 ffff882fa542dfd8
ffff882fa542dfd8
[ 1562.356070]  ffff882fa6439720 ffff882fa5f85c80 ffff882fa5f85c80
ffff88211ce60db0
[ 1562.356075] Call Trace:
[ 1562.356080]  [<ffffffff816ba149>] schedule+0x29/0x70
[ 1562.356085]  [<ffffffff816ba42e>] schedule_preempt_disabled+0xe/0x10
[ 1562.356089]  [<ffffffff816b8722>] __mutex_lock_slowpath+0x112/0x1b0
[ 1562.356094]  [<ffffffff816b829a>] mutex_lock+0x2a/0x50
[ 1562.356098]  [<ffffffff8150d6b5>] bch_mca_shrink+0x1c5/0x310
[ 1562.356104]  [<ffffffff8117ba02>] ? prune_super+0x162/0x1b0
[ 1562.356109]  [<ffffffff8112d054>] shrink_slab+0x154/0x300
[ 1562.356115]  [<ffffffff81076118>] ? resched_task+0x68/0x70
[ 1562.356120]  [<ffffffff81076a45>] ? check_preempt_curr+0x75/0xa0
[ 1562.356125]  [<ffffffff811387c9>] ? fragmentation_index+0x19/0x70
[ 1562.356130]  [<ffffffff8112f8af>] do_try_to_free_pages+0x20f/0x4b0
[ 1562.356135]  [<ffffffff8112fd04>] try_to_free_pages+0xe4/0x1a0
[ 1562.356142]  [<ffffffff8112536c>] __alloc_pages_nodemask+0x60c/0x990
[ 1562.356147]  [<ffffffff81076118>] ? resched_task+0x68/0x70
[ 1562.356153]  [<ffffffff8115e94a>] alloc_pages_current+0xba/0x170
[ 1562.356159]  [<ffffffff81120a5e>] __get_free_pages+0xe/0x40
[ 1562.356163]  [<ffffffff8150ac73>] mca_data_alloc+0x73/0x1d0
[ 1562.356168]  [<ffffffff8150b2ba>] mca_bucket_alloc+0x14a/0x1f0
[ 1562.356172]  [<ffffffff8150d3a0>] mca_alloc+0x320/0x470
[ 1562.356177]  [<ffffffff8150e37b>] bch_btree_node_alloc+0x9b/0x200
[ 1562.356181]  [<ffffffff81080d95>] ? enqueue_entity+0x1d5/0xa90
[ 1562.356186]  [<ffffffff81510099>] btree_split+0x119/0x770
[ 1562.356191]  [<ffffffff81025018>] ? native_smp_send_reschedule+0x48/0x60
[ 1562.356196]  [<ffffffff81513cac>] ? __bch_bset_search+0x3fc/0x4a0
[ 1562.356200]  [<ffffffff81080d95>] ? enqueue_entity+0x1d5/0xa90
[ 1562.356205]  [<ffffffff81510adb>] bch_btree_insert_recurse+0x3eb/0x500
[ 1562.356209]  [<ffffffff81510a78>] bch_btree_insert_recurse+0x388/0x500
[ 1562.356214]  [<ffffffff81510d46>] bch_btree_insert+0x156/0x330
[ 1562.356220]  [<ffffffff8151ce9a>] bch_btree_insert_async+0x1a/0xe0
[ 1562.356225]  [<ffffffff81517ba5>] bch_journal+0x45/0x260
[ 1562.356229]  [<ffffffff81062454>] process_one_work+0x174/0x490
[ 1562.356234]  [<ffffffff8106351b>] worker_thread+0x11b/0x370
[ 1562.356239]  [<ffffffff81063400>] ? manage_workers.isra.21+0x2d0/0x2d0
[ 1562.356243]  [<ffffffff81069c60>] kthread+0xc0/0xd0
[ 1562.356247]  [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0
[ 1562.356252]  [<ffffffff816c381c>] ret_from_fork+0x7c/0xb0
[ 1562.356256]  [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0
[ 1562.356262] INFO: task kworker/u64:7:349 blocked for more than 120 seconds.
[ 1562.356267] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1562.356273] kworker/u64:7   D ffffffff8180bde0     0   349      2 0x00000000
[ 1562.356280] Workqueue: bcache_writeback write_dirty_finish
[ 1562.356282]  ffff882f88e8ba90 0000000000000046 ffff882f88e8ba48
ffffffff81138789
[ 1562.356286]  ffff882f88c8dc80 ffff882f88e8bfd8 ffff882f88e8bfd8
ffff882f88e8bfd8
[ 1562.356291]  ffffffff81c10440 ffff882f88c8dc80 ffff882f88e8bbd3
ffff882f88c8dc80
[ 1562.356295] Call Trace:
[ 1562.356300]  [<ffffffff81138789>] ? zone_statistics+0x99/0xc0
[ 1562.356305]  [<ffffffff816ba149>] schedule+0x29/0x70
[ 1562.356310]  [<ffffffff816baf9d>] rwsem_down_read_failed+0x9d/0xe5
[ 1562.356316]  [<ffffffff8132eae4>] call_rwsem_down_read_failed+0x14/0x30
[ 1562.356320]  [<ffffffff816b8b34>] ? down_read+0x24/0x2b
[ 1562.356325]  [<ffffffff81510d00>] bch_btree_insert+0x110/0x330
[ 1562.356330]  [<ffffffff8151891a>] write_dirty_finish+0x27a/0x2c0
[ 1562.356336]  [<ffffffff81062454>] process_one_work+0x174/0x490
[ 1562.356341]  [<ffffffff8106351b>] worker_thread+0x11b/0x370
[ 1562.356346]  [<ffffffff81063400>] ? manage_workers.isra.21+0x2d0/0x2d0
[ 1562.356350]  [<ffffffff81069c60>] kthread+0xc0/0xd0
[ 1562.356354]  [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0
[ 1562.356359]  [<ffffffff816c381c>] ret_from_fork+0x7c/0xb0
[ 1562.356363]  [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0
[ 1562.356373] INFO: task kworker/0:2:1967 blocked for more than 120 seconds.
[ 1562.356377] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 1562.356383] kworker/0:2     D 0000000000000000     0  1967      2 0x00000000
[ 1562.356389] Workqueue: bcache bch_journal
[ 1562.356391]  ffff882f890b7bf0 0000000000000046 0000000100000000
ffff882fbf273140
[ 1562.356396]  ffff882f88d8ae40 ffff882f890b7fd8 ffff882f890b7fd8
ffff882f890b7fd8
[ 1562.356400]  ffff882f924d4560 ffff882f88d8ae40 ffffffff81516b08
ffff882f88d8ae40
[ 1562.356405] Call Trace:
[ 1562.356410]  [<ffffffff81516b08>] ? journal_write_endio+0x38/0x40
[ 1562.356414]  [<ffffffff816ba149>] schedule+0x29/0x70
[ 1562.356419]  [<ffffffff816baf9d>] rwsem_down_read_failed+0x9d/0xe5
[ 1562.356424]  [<ffffffff81053a90>] ? internal_add_timer+0x20/0x50
[ 1562.356429]  [<ffffffff8132eae4>] call_rwsem_down_read_failed+0x14/0x30
[ 1562.356433]  [<ffffffff816b8b34>] ? down_read+0x24/0x2b
[ 1562.356438]  [<ffffffff81510d00>] bch_btree_insert+0x110/0x330
[ 1562.356444]  [<ffffffff8151ce9a>] bch_btree_insert_async+0x1a/0xe0
[ 1562.356449]  [<ffffffff81517ba5>] bch_journal+0x45/0x260
[ 1562.356453]  [<ffffffff81062454>] process_one_work+0x174/0x490
[ 1562.356458]  [<ffffffff8106351b>] worker_thread+0x11b/0x370
[ 1562.356463]  [<ffffffff81063400>] ? manage_workers.isra.21+0x2d0/0x2d0
[ 1562.356466]  [<ffffffff81069c60>] kthread+0xc0/0xd0
[ 1562.356471]  [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0
[ 1562.356476]  [<ffffffff816c381c>] ret_from_fork+0x7c/0xb0
[ 1562.356480]  [<ffffffff81069ba0>] ? flush_kthread_worker+0xb0/0xb0


-Neo

> Stefan
>
> Am 28.08.2013 22:12, schrieb kernel neophyte:
>
>> On Wed, Aug 28, 2013 at 12:20 PM, Stefan Priebe <s.priebe@profihost.ag>
>> wrote:
>>>
>>> Am 28.08.2013 20:47, schrieb kernel neophyte:
>>>
>>>> On Wed, Aug 28, 2013 at 11:38 AM, Stefan Priebe - Profihost AG
>>>> <s.priebe@profihost.ag> wrote:
>>>>>
>>>>>
>>>>> I don't had one for a few days. Which kernel so you use?
>>>>
>>>>
>>>>
>>>> 3.10 kernel with all of kent's stable patches and perf patches..
>>>
>>>
>>>
>>> Which exact 3.10 version? Which patches on top?
>>
>>
>> 3.10.0-rc7
>> with following patches:
>> bcache: Fix a dumb CPU spinning bug in writeback
>> bcache: Fix a flush/fua performance bug
>> bcache: Fix a writeback performance regression
>> bcache: FUA fixes
>>
>> I have also tried bcache perf branch i.e.
>> (http://evilpiepirate.org/git/linux-bcache.git/log/?h=bcache-perf)
>>
>>>
>>>
>>>> and all defaults except for:
>>>>
>>>> echo writeback > /sys/block/bcache0/bcache/cache_mode
>>>> echo 0 > /sys/block/bcache0/bcache/sequential_cutoff
>>>> echo 0 >
>>>>
>>>> /sys/fs/bcache/1e26e5ca-5cb9-4c5a-ad7e-3c787524a9dc/congested_read_threshold_us
>>>> echo 0 >
>>>>
>>>> /sys/fs/bcache/1e26e5ca-5cb9-4c5a-ad7e-3c787524a9dc/congested_write_threshold_us
>>>>
>>>> -Neo
>>>>
>>>>>
>>>>> Stefan
>>>>>
>>>>> This mail was sent with my iPhone.
>>>>>
>>>>> Am 28.08.2013 um 20:12 schrieb kernel neophyte
>>>>> <neophyte.hacker001@gmail.com>:
>>>>>
>>>>>> On Tue, Aug 27, 2013 at 11:05 PM, Kent Overstreet <kmo@daterainc.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Aug 27, 2013 at 08:37:54PM -0700, kernel neophyte wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Kent,
>>>>>>>>
>>>>>>>> I am still seeing deadlock:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Seeing blk_throtl_bio (what asshole misspells words in their function
>>>>>>> names, so when you go to grep for them you grep for the wrong thing?)
>>>>>>> in
>>>>>>> the backtrace - is that what it takes to hit it?
>>>>>>
>>>>>>
>>>>>>
>>>>>> I don't think so,
>>>>>> I had another deadlock:
>>>>>>
>>>>>> [ 1179.193990] bio: create slab <bio-2> at 2
>>>>>> [ 1179.194491] bcache: run_cache_set() invalidating existing data
>>>>>> [ 1179.212308] bcache: register_cache() registered cache device rxd1
>>>>>> [ 1203.846220] bcache: register_bdev() registered backing device sda3
>>>>>> [ 1227.780088] bcache: bch_cached_dev_attach() Caching sda3 as bcache0
>>>>>> on set a079dda3-27fd-4b1c-a616-d46110bfae6c
>>>>>> [ 2162.497950] INFO: task kworker/0:0:4 blocked for more than 120
>>>>>> seconds.
>>>>>> [ 2162.497961] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2162.497967] kworker/0:0     D 0000000000000000     0     4      2
>>>>>> 0x00000000
>>>>>> [ 2162.497982] Workqueue: events write_dirty_finish
>>>>>> [ 2162.497986]  ffff882fa6ac9b30 0000000000000046 ffffffff81511b8c
>>>>>> ffffffff815137c0
>>>>>> [ 2162.497992]  ffff882fa6a84cb0 ffff882fa6ac9fd8 ffff882fa6ac9fd8
>>>>>> ffff882fa6ac9fd8
>>>>>> [ 2162.497996]  ffff882f78139990 ffff882fa6a84cb0 ffffffff815117a1
>>>>>> ffff882fa6a84cb0
>>>>>> [ 2162.498001] Call Trace:
>>>>>> [ 2162.498010]  [<ffffffff81511b8c>] ?
>>>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>>>> [ 2162.498015]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.498019]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>>>> [ 2162.498028]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2162.498034]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>>> [ 2162.498043]  [<ffffffff81332c64>]
>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>> [ 2162.498048]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>>> [ 2162.498052]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>>>> [ 2162.498057]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.498066]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>>> [ 2162.498072]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>>>> [ 2162.498076]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>>> [ 2162.498082]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>>>> [ 2162.498087]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>>>> [ 2162.498093]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>>>> [ 2162.498100]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>> [ 2162.498105]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>> [ 2162.498110]  [<ffffffff81063570>] ?
>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>> [ 2162.498115]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>> [ 2162.498120]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498127]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>> [ 2162.498131]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498145] INFO: task bcache_allocato:1766 blocked for more than
>>>>>> 120 seconds.
>>>>>> [ 2162.498149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2162.498154] bcache_allocato D 0000000000000001     0  1766      2
>>>>>> 0x00000000
>>>>>> [ 2162.498158]  ffff8822630f3d88 0000000000000046 ffff8822630f3da8
>>>>>> ffffffff810808ad
>>>>>> [ 2162.498163]  ffff882f891de640 ffff8822630f3fd8 ffff8822630f3fd8
>>>>>> ffff8822630f3fd8
>>>>>> [ 2162.498167]  ffff882f81a91990 ffff882f891de640 ffff882f88b33320
>>>>>> ffff8801fad20d98
>>>>>> [ 2162.498172] Call Trace:
>>>>>> [ 2162.498177]  [<ffffffff810808ad>] ? dequeue_task_fair+0x2cd/0x530
>>>>>> [ 2162.498182]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2162.498187]  [<ffffffff816be57e>]
>>>>>> schedule_preempt_disabled+0xe/0x10
>>>>>> [ 2162.498192]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
>>>>>> [ 2162.498197]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>>>>> [ 2162.498202]  [<ffffffff8150cdbf>] bch_allocator_thread+0x10f/0xe20
>>>>>> [ 2162.498207]  [<ffffffff8150ccb0>] ? bch_bucket_add_unused+0xe0/0xe0
>>>>>> [ 2162.498211]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>> [ 2162.498215]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498220]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>> [ 2162.498224]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498228] INFO: task bcache_writebac:1771 blocked for more than
>>>>>> 120 seconds.
>>>>>> [ 2162.498233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2162.498237] bcache_writebac D 0000000000000000     0  1771      2
>>>>>> 0x00000000
>>>>>> [ 2162.498241]  ffff88197d2adc58 0000000000000046 ffff88197d2add48
>>>>>> ffffffff811269e2
>>>>>> [ 2162.498245]  ffff882f891d8000 ffff88197d2adfd8 ffff88197d2adfd8
>>>>>> ffff88197d2adfd8
>>>>>> [ 2162.498250]  ffff882f81a91990 ffff882f891d8000 ffff88197d2adc68
>>>>>> ffff882f89550ba0
>>>>>> [ 2162.498254] Call Trace:
>>>>>> [ 2162.498264]  [<ffffffff811269e2>] ?
>>>>>> __alloc_pages_nodemask+0x152/0x9b0
>>>>>> [ 2162.498269]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2162.498274]  [<ffffffff816bc015>] schedule_timeout+0x1e5/0x250
>>>>>> [ 2162.498280]  [<ffffffff810799bf>] ? try_to_wake_up+0x24f/0x2b0
>>>>>> [ 2162.498285]  [<ffffffff816bcd09>] __down_common+0x8e/0xe3
>>>>>> [ 2162.498293]  [<ffffffff81160600>] ? alloc_pages_current+0x90/0x170
>>>>>> [ 2162.498298]  [<ffffffff816bcdd1>] __down+0x1d/0x1f
>>>>>> [ 2162.498304]  [<ffffffff8106f351>] down+0x41/0x50
>>>>>> [ 2162.498310]  [<ffffffff8151ba7d>] bch_writeback_thread+0x35d/0x7f0
>>>>>> [ 2162.498315]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
>>>>>> [ 2162.498321]  [<ffffffff8151b720>] ? write_dirty+0xc0/0xc0
>>>>>> [ 2162.498325]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>> [ 2162.498329]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498334]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>> [ 2162.498338]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498342] INFO: task kworker/0:2:1789 blocked for more than 120
>>>>>> seconds.
>>>>>> [ 2162.498346] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2162.498351] kworker/0:2     D ffffffff81813a60     0  1789      2
>>>>>> 0x00000000
>>>>>> [ 2162.498357] Workqueue: events write_dirty_finish
>>>>>> [ 2162.498360]  ffff88277062fb30 0000000000000046 ffffffff81511b8c
>>>>>> ffffffff815137c0
>>>>>> [ 2162.498364]  ffff882fa6786640 ffff88277062ffd8 ffff88277062ffd8
>>>>>> ffff88277062ffd8
>>>>>> [ 2162.498368]  ffffffff81c10440 ffff882fa6786640 ffffffff815117a1
>>>>>> ffff882fa6786640
>>>>>> [ 2162.498373] Call Trace:
>>>>>> [ 2162.498378]  [<ffffffff81511b8c>] ?
>>>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>>>> [ 2162.498382]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.498387]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>>>> [ 2162.498392]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2162.498397]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>>> [ 2162.498402]  [<ffffffff81332c64>]
>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>> [ 2162.498407]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>>> [ 2162.498411]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>>>> [ 2162.498416]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.498420]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>>> [ 2162.498425]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>>>> [ 2162.498430]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>>> [ 2162.498435]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>>>> [ 2162.498439]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>>>> [ 2162.498444]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>>>> [ 2162.498449]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>> [ 2162.498454]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>> [ 2162.498459]  [<ffffffff81063570>] ?
>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>> [ 2162.498463]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>> [ 2162.498467]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498472]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>> [ 2162.498476]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498480] INFO: task iozone:2052 blocked for more than 120
>>>>>> seconds.
>>>>>> [ 2162.498484] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2162.498488] iozone          D 0000000000000001     0  2052   1944
>>>>>> 0x00000000
>>>>>> [ 2162.498492]  ffff882f7e2d3978 0000000000000082 ffff882f7e2d3938
>>>>>> ffffffff81301d7a
>>>>>> [ 2162.498497]  ffff882f8a116640 ffff882f7e2d3fd8 ffff882f7e2d3fd8
>>>>>> ffff882f7e2d3fd8
>>>>>> [ 2162.498501]  ffff882f88b33320 ffff882f8a116640 ffff882f7e2d3978
>>>>>> ffff882fbf2339f8
>>>>>> [ 2162.498505] Call Trace:
>>>>>> [ 2162.498511]  [<ffffffff81301d7a>] ? generic_make_request+0xca/0x100
>>>>>> [ 2162.498517]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2162.498522]  [<ffffffff816be36f>] io_schedule+0x8f/0xd0
>>>>>> [ 2162.498530]  [<ffffffff811b797c>]
>>>>>> do_blockdev_direct_IO+0x1a7c/0x1fb0
>>>>>> [ 2162.498539]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60
>>>>>> [ext2]
>>>>>> [ 2162.498546]  [<ffffffff811b7f05>] __blockdev_direct_IO+0x55/0x60
>>>>>> [ 2162.498552]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60
>>>>>> [ext2]
>>>>>> [ 2162.498556]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>>> [ 2162.498562]  [<ffffffffa0274349>] ext2_direct_IO+0x79/0xe0 [ext2]
>>>>>> [ 2162.498567]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60
>>>>>> [ext2]
>>>>>> [ 2162.498572]  [<ffffffff8107e775>] ? set_next_entity+0xa5/0xc0
>>>>>> [ 2162.498579]  [<ffffffff8104ade6>] ? current_fs_time+0x16/0x60
>>>>>> [ 2162.498585]  [<ffffffff8111f126>]
>>>>>> generic_file_direct_write+0xc6/0x180
>>>>>> [ 2162.498590]  [<ffffffff8111f4bd>]
>>>>>> __generic_file_aio_write+0x2dd/0x3b0
>>>>>> [ 2162.498595]  [<ffffffff816bc364>] ? mutex_unlock+0x14/0x20
>>>>>> [ 2162.498600]  [<ffffffff8111f5f9>] generic_file_aio_write+0x69/0xd0
>>>>>> [ 2162.498607]  [<ffffffff8117b88a>] do_sync_write+0x7a/0xb0
>>>>>> [ 2162.498612]  [<ffffffff8117c63e>] vfs_write+0xce/0x1e0
>>>>>> [ 2162.498617]  [<ffffffff8117bba0>] ?
>>>>>> generic_file_llseek_size+0x110/0x110
>>>>>> [ 2162.498621]  [<ffffffff8117cb22>] SyS_write+0x52/0xa0
>>>>>> [ 2162.498627]  [<ffffffff816c7a02>] system_call_fastpath+0x16/0x1b
>>>>>> [ 2162.498630] INFO: task kworker/0:1:2293 blocked for more than 120
>>>>>> seconds.
>>>>>> [ 2162.498635] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2162.498639] kworker/0:1     D ffffffff81813a60     0  2293      2
>>>>>> 0x00000000
>>>>>> [ 2162.498645] Workqueue: events write_dirty_finish
>>>>>> [ 2162.498647]  ffff882f8a05bb30 0000000000000046 ffffffff81511b8c
>>>>>> ffffffff815137c0
>>>>>> [ 2162.498652]  ffff882f8a163320 ffff882f8a05bfd8 ffff882f8a05bfd8
>>>>>> ffff882f8a05bfd8
>>>>>> [ 2162.498656]  ffffffff81c10440 ffff882f8a163320 ffffffff815117a1
>>>>>> ffff882f8a163320
>>>>>> [ 2162.498661] Call Trace:
>>>>>> [ 2162.498665]  [<ffffffff81511b8c>] ?
>>>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>>>> [ 2162.498669]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.498674]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>>>> [ 2162.498679]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2162.498684]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>>> [ 2162.498689]  [<ffffffff81332c64>]
>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>> [ 2162.498693]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>>> [ 2162.498698]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>>>> [ 2162.498702]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.498707]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>>> [ 2162.498712]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>>>> [ 2162.498716]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>>> [ 2162.498721]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>>>> [ 2162.498726]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>>>> [ 2162.498731]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>>>> [ 2162.498735]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>> [ 2162.498740]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>> [ 2162.498745]  [<ffffffff81063570>] ?
>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>> [ 2162.498749]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>> [ 2162.498753]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498758]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>> [ 2162.498762]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498766] INFO: task kworker/0:3:2534 blocked for more than 120
>>>>>> seconds.
>>>>>> [ 2162.498772] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2162.498777] kworker/0:3     D ffffffff81813a60     0  2534      2
>>>>>> 0x00000000
>>>>>> [ 2162.498783] Workqueue: events write_dirty_finish
>>>>>> [ 2162.498785]  ffff88237d0bdb30 0000000000000046 ffffffff81511b8c
>>>>>> ffffffff815137c0
>>>>>> [ 2162.498790]  ffff882fa66fe640 ffff88237d0bdfd8 ffff88237d0bdfd8
>>>>>> ffff88237d0bdfd8
>>>>>> [ 2162.498794]  ffffffff81c10440 ffff882fa66fe640 ffffffff815117a1
>>>>>> ffff882fa66fe640
>>>>>> [ 2162.498799] Call Trace:
>>>>>> [ 2162.498803]  [<ffffffff81511b8c>] ?
>>>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>>>> [ 2162.498807]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.498812]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>>>> [ 2162.498817]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2162.498822]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>>> [ 2162.498826]  [<ffffffff81332c64>]
>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>> [ 2162.498831]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>>> [ 2162.498836]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>>>> [ 2162.498840]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.498845]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>>> [ 2162.498849]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>>>> [ 2162.498854]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>>> [ 2162.498859]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>>>> [ 2162.498864]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>>>> [ 2162.498868]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>>>> [ 2162.498873]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>> [ 2162.498878]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>> [ 2162.498882]  [<ffffffff81063570>] ?
>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>> [ 2162.498886]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>> [ 2162.498890]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498896]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>> [ 2162.498900]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498903] INFO: task kworker/0:4:2537 blocked for more than 120
>>>>>> seconds.
>>>>>> [ 2162.498908] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2162.498914] kworker/0:4     D 0000000000000000     0  2537      2
>>>>>> 0x00000000
>>>>>> [ 2162.498920] Workqueue: events write_dirty_finish
>>>>>> [ 2162.498922]  ffff882f76d8fb30 0000000000000046 ffffffff81511b8c
>>>>>> ffffffff815137c0
>>>>>> [ 2162.498926]  ffff882f88b30000 ffff882f76d8ffd8 ffff882f76d8ffd8
>>>>>> ffff882f76d8ffd8
>>>>>> [ 2162.498931]  ffff882f78beb320 ffff882f88b30000 ffffffff815117a1
>>>>>> ffff882f88b30000
>>>>>> [ 2162.498935] Call Trace:
>>>>>> [ 2162.498940]  [<ffffffff81511b8c>] ?
>>>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>>>> [ 2162.498944]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.498948]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>>>> [ 2162.498953]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2162.498958]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>>> [ 2162.498963]  [<ffffffff81332c64>]
>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>> [ 2162.498968]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>>> [ 2162.498972]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>>>> [ 2162.498977]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.498981]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>>> [ 2162.498986]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>>>> [ 2162.498990]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>>> [ 2162.498995]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>>>> [ 2162.499000]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>>>> [ 2162.499004]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>>>> [ 2162.499009]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>> [ 2162.499014]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>> [ 2162.499019]  [<ffffffff81063570>] ?
>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>> [ 2162.499023]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>> [ 2162.499027]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.499032]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>> [ 2162.499036]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.499039] INFO: task kworker/1:1:2544 blocked for more than 120
>>>>>> seconds.
>>>>>> [ 2162.499044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2162.499050] kworker/1:1     D ffffffff81813d40     0  2544      2
>>>>>> 0x00000000
>>>>>> [ 2162.499057] Workqueue: bcache bch_data_insert_keys
>>>>>> [ 2162.499059]  ffff882fa677b3f8 0000000000000046 ffff882fa677b418
>>>>>> ffff882f8a709000
>>>>>> [ 2162.499063]  ffff882f88b33320 ffff882fa677bfd8 ffff882fa677bfd8
>>>>>> ffff882fa677bfd8
>>>>>> [ 2162.499068]  ffff882fa6471990 ffff882f88b33320 ffff882f88b33320
>>>>>> ffff8801fad20d98
>>>>>> [ 2162.499072] Call Trace:
>>>>>> [ 2162.499077]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2162.499082]  [<ffffffff816be57e>]
>>>>>> schedule_preempt_disabled+0xe/0x10
>>>>>> [ 2162.499087]  [<ffffffff816bc862>] __mutex_lock_slowpath+0x112/0x1b0
>>>>>> [ 2162.499092]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>>>>> [ 2162.499096]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
>>>>>> [ 2162.499102]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
>>>>>> [ 2162.499109]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
>>>>>> [ 2162.499113]  [<ffffffff81076828>] ? resched_task+0x68/0x70
>>>>>> [ 2162.499118]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
>>>>>> [ 2162.499126]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
>>>>>> [ 2162.499131]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
>>>>>> [ 2162.499137]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
>>>>>> [ 2162.499143]  [<ffffffff81126e9c>]
>>>>>> __alloc_pages_nodemask+0x60c/0x9b0
>>>>>> [ 2162.499149]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
>>>>>> [ 2162.499155]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
>>>>>> [ 2162.499159]  [<ffffffff8150ebb3>] mca_data_alloc+0x73/0x1d0
>>>>>> [ 2162.499163]  [<ffffffff81510f37>] mca_alloc+0x277/0x470
>>>>>> [ 2162.499168]  [<ffffffff81511d1c>] bch_btree_node_alloc+0x8c/0x1c0
>>>>>> [ 2162.499173]  [<ffffffff81513020>] btree_split+0x110/0x5c0
>>>>>> [ 2162.499179]  [<ffffffff81457d43>] ?
>>>>>> scsi_pool_alloc_command+0x33/0x80
>>>>>> [ 2162.499184]  [<ffffffff81515fc7>] ? bch_keylist_pop_front+0x47/0x50
>>>>>> [ 2162.499188]  [<ffffffff8150fed6>] ?
>>>>>> bch_btree_insert_keys+0x56/0x250
>>>>>> [ 2162.499193]  [<ffffffff81082def>] ? update_group_power+0x13f/0x220
>>>>>> [ 2162.499198]  [<ffffffff81513582>] bch_btree_insert_node+0xb2/0x2f0
>>>>>> [ 2162.499202]  [<ffffffff815137e8>] btree_insert_fn+0x28/0x50
>>>>>> [ 2162.499207]  [<ffffffff81511b8c>]
>>>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>>>> [ 2162.499211]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.499216]  [<ffffffff816bcc26>] ? down_write+0x16/0x40
>>>>>> [ 2162.499221]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>>>> [ 2162.499225]  [<ffffffff81511c30>]
>>>>>> bch_btree_map_nodes_recurse+0x110/0x170
>>>>>> [ 2162.499230]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.499235]  [<ffffffff811b51ea>] ? dio_bio_end_io+0x5a/0x90
>>>>>> [ 2162.499239]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>>> [ 2162.499244]  [<ffffffff81514dce>] __bch_btree_map_nodes+0x13e/0x1c0
>>>>>> [ 2162.499248]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.499254]  [<ffffffff8151ac6f>] ? bch_journal+0x42f/0x4b0
>>>>>> [ 2162.499258]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>>> [ 2162.499264]  [<ffffffff8151e1be>] bch_data_insert_keys+0x3e/0x160
>>>>>> [ 2162.499268]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>> [ 2162.499273]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>> [ 2162.499278]  [<ffffffff81063570>] ?
>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>> [ 2162.499282]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>> [ 2162.499286]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.499292]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>> [ 2162.499296]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.499299] INFO: task kworker/0:5:2552 blocked for more than 120
>>>>>> seconds.
>>>>>> [ 2162.499304] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2162.499310] kworker/0:5     D 0000000000000000     0  2552      2
>>>>>> 0x00000000
>>>>>> [ 2162.499411] Workqueue: events write_dirty_finish
>>>>>> [ 2162.499413]  ffff88226314db30 0000000000000046 ffffffff81511b8c
>>>>>> ffffffff815137c0
>>>>>> [ 2162.499418]  ffff882fa6781990 ffff88226314dfd8 ffff88226314dfd8
>>>>>> ffff88226314dfd8
>>>>>> [ 2162.499422]  ffff882fa6a84cb0 ffff882fa6781990 ffffffff815117a1
>>>>>> ffff882fa6781990
>>>>>> [ 2162.499427] Call Trace:
>>>>>> [ 2162.499431]  [<ffffffff81511b8c>] ?
>>>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>>>> [ 2162.499435]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.499440]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>>>> [ 2162.499445]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2162.499450]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>>> [ 2162.499455]  [<ffffffff81332c64>]
>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>> [ 2162.499460]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>>> [ 2162.499464]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>>>> [ 2162.499468]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.499473]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>>>> [ 2162.499478]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>>> [ 2162.499483]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>>>> [ 2162.499488]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>>>> [ 2162.499492]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>>>> [ 2162.499497]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>> [ 2162.499502]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>> [ 2162.499507]  [<ffffffff81063570>] ?
>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>> [ 2162.499511]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>> [ 2162.499515]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.499520]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>> [ 2162.499524]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>>
>>>>>>
>>>>>>
>>>>>>>> [  930.169234] bcache: bch_cached_dev_attach() Caching sda3 as
>>>>>>>> bcache0
>>>>>>>> on set 06496904-10ca-489a-ae75-68c6a07d3db1
>>>>>>>> [ 2522.956188] INFO: task bcache_writebac:2058 blocked for more than
>>>>>>>> 120 seconds.
>>>>>>>> [ 2522.956199] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>> disables this message.
>>>>>>>> [ 2522.956205] bcache_writebac D ffffffff81813a60     0  2058      2
>>>>>>>> 0x00000000
>>>>>>>> [ 2522.956212]  ffff881c39c1fcd8 0000000000000046 0000000000000001
>>>>>>>> 0000000000000001
>>>>>>>> [ 2522.956218]  ffff882f8b56b320 ffff881c39c1ffd8 ffff881c39c1ffd8
>>>>>>>> ffff881c39c1ffd8
>>>>>>>> [ 2522.956222]  ffff882fa6aeb320 ffff882f8b56b320 ffff882f8b56b320
>>>>>>>> ffff882f8b56b320
>>>>>>>> [ 2522.956227] Call Trace:
>>>>>>>> [ 2522.956241]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>>>> [ 2522.956247]  [<ffffffff816befb5>]
>>>>>>>> rwsem_down_write_failed+0xf5/0x1a0
>>>>>>>> [ 2522.956255]  [<ffffffff81332c93>]
>>>>>>>> call_rwsem_down_write_failed+0x13/0x20
>>>>>>>> [ 2522.956261]  [<ffffffff816bcc41>] ? down_write+0x31/0x40
>>>>>>>> [ 2522.956269]  [<ffffffff8151b782>] bch_writeback_thread+0x62/0x7f0
>>>>>>>> [ 2522.956278]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
>>>>>>>> [ 2522.956284]  [<ffffffff8151b720>] ? write_dirty+0xc0/0xc0
>>>>>>>> [ 2522.956290]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>>>> [ 2522.956294]  [<ffffffff81069e80>] ?
>>>>>>>> flush_kthread_worker+0xb0/0xb0
>>>>>>>> [ 2522.956302]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>>>> [ 2522.956306]  [<ffffffff81069e80>] ?
>>>>>>>> flush_kthread_worker+0xb0/0xb0
>>>>>>>> [ 2522.956313] INFO: task kworker/u64:1:2433 blocked for more than
>>>>>>>> 120
>>>>>>>> seconds.
>>>>>>>> [ 2522.956317] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>> disables this message.
>>>>>>>> [ 2522.956322] kworker/u64:1   D ffffffff81813a60     0  2433      2
>>>>>>>> 0x00000000
>>>>>>>> [ 2522.956332] Workqueue: writeback bdi_writeback_workfn
>>>>>>>> (flush-252:0)
>>>>>>>> [ 2522.956336]  ffff882f899d1560 0000000000000046 ffffffff81120635
>>>>>>>> 0000000000000100
>>>>>>>> [ 2522.956340]  ffff88219b488000 ffff882f899d1fd8 ffff882f899d1fd8
>>>>>>>> ffff882f899d1fd8
>>>>>>>> [ 2522.956345]  ffff882fa6aeb320 ffff88219b488000 ffffffff8131ad55
>>>>>>>> ffff88219b488000
>>>>>>>> [ 2522.956349] Call Trace:
>>>>>>>> [ 2522.956359]  [<ffffffff81120635>] ? mempool_alloc_slab+0x15/0x20
>>>>>>>> [ 2522.956367]  [<ffffffff8131ad55>] ? blk_throtl_bio+0x3b5/0x540
>>>>>>>> [ 2522.956372]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>>>> [ 2522.956378]  [<ffffffff816bf0fd>]
>>>>>>>> rwsem_down_read_failed+0x9d/0xe5
>>>>>>>> [ 2522.956383]  [<ffffffff81332c64>]
>>>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>>>> [ 2522.956388]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>>>>> [ 2522.956394]  [<ffffffff8151fe32>]
>>>>>>>> cached_dev_make_request+0x752/0xf00
>>>>>>>> [ 2522.956400]  [<ffffffff81301af8>] ?
>>>>>>>> generic_make_request_checks+0x1e8/0x3a0
>>>>>>>> [ 2522.956405]  [<ffffffff81301d7a>] generic_make_request+0xca/0x100
>>>>>>>> [ 2522.956409]  [<ffffffff81301e29>] submit_bio+0x79/0x160
>>>>>>>> [ 2522.956417]  [<ffffffff811b1710>] ? bio_alloc_bioset+0xa0/0x1d0
>>>>>>>> [ 2522.956421]  [<ffffffff811abd3f>] _submit_bh+0x13f/0x200
>>>>>>>> [ 2522.956425]  [<ffffffff811abe10>] submit_bh+0x10/0x20
>>>>>>>> [ 2522.956430]  [<ffffffff811af6b8>]
>>>>>>>> __block_write_full_page+0x1d8/0x360
>>>>>>>> [ 2522.956439]  [<ffffffff8145fc0b>] ? scsi_request_fn+0xbb/0x530
>>>>>>>> [ 2522.956444]  [<ffffffff811ad3c0>] ?
>>>>>>>> end_buffer_async_read+0x130/0x130
>>>>>>>> [ 2522.956449]  [<ffffffff811b2c40>] ? I_BDEV+0x10/0x10
>>>>>>>> [ 2522.956454]  [<ffffffff811b2c40>] ? I_BDEV+0x10/0x10
>>>>>>>> [ 2522.956459]  [<ffffffff811af90a>]
>>>>>>>> block_write_full_page_endio+0xca/0x100
>>>>>>>> [ 2522.956464]  [<ffffffff811af955>] block_write_full_page+0x15/0x20
>>>>>>>> [ 2522.956469]  [<ffffffff811b3608>] blkdev_writepage+0x18/0x20
>>>>>>>> [ 2522.956473]  [<ffffffff81127d37>] __writepage+0x17/0x40
>>>>>>>> [ 2522.956477]  [<ffffffff8112820e>] write_cache_pages+0x20e/0x460
>>>>>>>> [ 2522.956481]  [<ffffffff81127d20>] ? set_page_dirty_lock+0x60/0x60
>>>>>>>> [ 2522.956486]  [<ffffffff811ac251>] ? __set_page_dirty+0x71/0xc0
>>>>>>>> [ 2522.956490]  [<ffffffff811284aa>] generic_writepages+0x4a/0x70
>>>>>>>> [ 2522.956495]  [<ffffffff81129cd0>] do_writepages+0x20/0x40
>>>>>>>> [ 2522.956501]  [<ffffffff811a36f5>]
>>>>>>>> __writeback_single_inode+0x45/0x280
>>>>>>>> [ 2522.956507]  [<ffffffff811269e2>] ?
>>>>>>>> __alloc_pages_nodemask+0x152/0x9b0
>>>>>>>> [ 2522.956513]  [<ffffffff811a495d>] writeback_sb_inodes+0x19d/0x3c0
>>>>>>>> [ 2522.956518]  [<ffffffff811a4c1e>] __writeback_inodes_wb+0x9e/0xd0
>>>>>>>> [ 2522.956523]  [<ffffffff811a4ecb>] wb_writeback+0x27b/0x320
>>>>>>>> [ 2522.956529]  [<ffffffff81196228>] ? get_nr_dirty_inodes+0x58/0x80
>>>>>>>> [ 2522.956534]  [<ffffffff811a500f>]
>>>>>>>> wb_check_old_data_flush+0x9f/0xb0
>>>>>>>> [ 2522.956539]  [<ffffffff811a67e1>] wb_do_writeback+0x151/0x1d0
>>>>>>>> [ 2522.956547]  [<ffffffff8106539f>] ? set_worker_desc+0x6f/0x80
>>>>>>>> [ 2522.956552]  [<ffffffff811a68da>] bdi_writeback_workfn+0x7a/0x200
>>>>>>>> [ 2522.956557]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>>>> [ 2522.956562]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>>>> [ 2522.956567]  [<ffffffff81063570>] ?
>>>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>>>> [ 2522.956571]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>>>> [ 2522.956575]  [<ffffffff81069e80>] ?
>>>>>>>> flush_kthread_worker+0xb0/0xb0
>>>>>>>> [ 2522.956581]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>>>> [ 2522.956585]  [<ffffffff81069e80>] ?
>>>>>>>> flush_kthread_worker+0xb0/0xb0
>>>>>>>> [ 2522.956588] INFO: task iozone:2435 blocked for more than 120
>>>>>>>> seconds.
>>>>>>>> [ 2522.956593] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>> disables this message.
>>>>>>>> [ 2522.956597] iozone          D 0000000000000001     0  2435   1545
>>>>>>>> 0x00000000
>>>>>>>> [ 2522.956602]  ffff882a75671978 0000000000000082 ffff882a75671938
>>>>>>>> ffffffff81301d7a
>>>>>>>> [ 2522.956606]  ffff882f88dc1990 ffff882a75671fd8 ffff882a75671fd8
>>>>>>>> ffff882a75671fd8
>>>>>>>> [ 2522.956611]  ffff882f8ec91990 ffff882f88dc1990 ffff882a75671978
>>>>>>>> ffff882fbf2539f8
>>>>>>>> [ 2522.956615] Call Trace:
>>>>>>>> [ 2522.956619]  [<ffffffff81301d7a>] ?
>>>>>>>> generic_make_request+0xca/0x100
>>>>>>>> [ 2522.956625]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>>>> [ 2522.956630]  [<ffffffff816be36f>] io_schedule+0x8f/0xd0
>>>>>>>> [ 2522.956636]  [<ffffffff811b797c>]
>>>>>>>> do_blockdev_direct_IO+0x1a7c/0x1fb0
>>>>>>>> [ 2522.956645]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60
>>>>>>>> [ext2]
>>>>>>>> [ 2522.956652]  [<ffffffff811b7f05>] __blockdev_direct_IO+0x55/0x60
>>>>>>>> [ 2522.956658]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60
>>>>>>>> [ext2]
>>>>>>>> [ 2522.956662]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>>>>> [ 2522.956669]  [<ffffffffa0263349>] ext2_direct_IO+0x79/0xe0 [ext2]
>>>>>>>> [ 2522.956674]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60
>>>>>>>> [ext2]
>>>>>>>> [ 2522.956680]  [<ffffffff8104ade6>] ? current_fs_time+0x16/0x60
>>>>>>>> [ 2522.956685]  [<ffffffff8111f126>]
>>>>>>>> generic_file_direct_write+0xc6/0x180
>>>>>>>> [ 2522.956690]  [<ffffffff8111f4bd>]
>>>>>>>> __generic_file_aio_write+0x2dd/0x3b0
>>>>>>>> [ 2522.956696]  [<ffffffff8111f5f9>]
>>>>>>>> generic_file_aio_write+0x69/0xd0
>>>>>>>> [ 2522.956702]  [<ffffffff8117b88a>] do_sync_write+0x7a/0xb0
>>>>>>>> [ 2522.956706]  [<ffffffff811bb108>] ? fsnotify+0x1f8/0x2b0
>>>>>>>> [ 2522.956712]  [<ffffffff8117c63e>] vfs_write+0xce/0x1e0
>>>>>>>> [ 2522.956716]  [<ffffffff8117cb22>] SyS_write+0x52/0xa0
>>>>>>>> [ 2522.956722]  [<ffffffff816c7a02>] system_call_fastpath+0x16/0x1b
>>>>>>>> [ 2522.956726] INFO: task kworker/2:1:2470 blocked for more than 120
>>>>>>>> seconds.
>>>>>>>> [ 2522.956730] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>> disables this message.
>>>>>>>> [ 2522.956735] kworker/2:1     D ffffffff81813d40     0  2470      2
>>>>>>>> 0x00000000
>>>>>>>> [ 2522.956743] Workqueue: bcache bch_data_insert_keys
>>>>>>>> [ 2522.956745]  ffff882a5b0c9628 0000000000000046 ffff882a5b0c9648
>>>>>>>> 0000004000000000
>>>>>>>> [ 2522.956749]  ffff882f8ec91990 ffff882a5b0c9fd8 ffff882a5b0c9fd8
>>>>>>>> ffff882a5b0c9fd8
>>>>>>>> [ 2522.956754]  ffff882fa6489990 ffff882f8ec91990 ffff882f8ec91990
>>>>>>>> ffff8828dba00d98
>>>>>>>> [ 2522.956758] Call Trace:
>>>>>>>> [ 2522.956763]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>>>> [ 2522.956769]  [<ffffffff816be57e>]
>>>>>>>> schedule_preempt_disabled+0xe/0x10
>>>>>>>> [ 2522.956774]  [<ffffffff816bc862>]
>>>>>>>> __mutex_lock_slowpath+0x112/0x1b0
>>>>>>>> [ 2522.956779]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>>>>>>> [ 2522.956783]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
>>>>>>>> [ 2522.956791]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
>>>>>>>> [ 2522.956799]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
>>>>>>>> [ 2522.956805]  [<ffffffff81076828>] ? resched_task+0x68/0x70
>>>>>>>> [ 2522.956810]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
>>>>>>>> [ 2522.956816]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
>>>>>>>> [ 2522.956822]  [<ffffffff8113140f>]
>>>>>>>> do_try_to_free_pages+0x20f/0x4b0
>>>>>>>> [ 2522.956827]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
>>>>>>>> [ 2522.956833]  [<ffffffff81126e9c>]
>>>>>>>> __alloc_pages_nodemask+0x60c/0x9b0
>>>>>>>> [ 2522.956843]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
>>>>>>>> [ 2522.956848]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
>>>>>>>> [ 2522.956852]  [<ffffffff8150ebb3>] mca_data_alloc+0x73/0x1d0
>>>>>>>> [ 2522.956857]  [<ffffffff8150ee5a>] mca_bucket_alloc+0x14a/0x1f0
>>>>>>>> [ 2522.956861]  [<ffffffff81511020>] mca_alloc+0x360/0x470
>>>>>>>> [ 2522.956866]  [<ffffffff81517031>] ? __bch_bset_search+0x1d1/0x480
>>>>>>>> [ 2522.956871]  [<ffffffff8151183e>] bch_btree_node_get+0x10e/0x280
>>>>>>>> [ 2522.956875]  [<ffffffff81511c02>]
>>>>>>>> bch_btree_map_nodes_recurse+0xe2/0x170
>>>>>>>> [ 2522.956880]  [<ffffffff815137c0>] ?
>>>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>>>> [ 2522.956886]  [<ffffffff811b51ea>] ? dio_bio_end_io+0x5a/0x90
>>>>>>>> [ 2522.956890]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>>>>> [ 2522.956895]  [<ffffffff81514dce>]
>>>>>>>> __bch_btree_map_nodes+0x13e/0x1c0
>>>>>>>> [ 2522.956899]  [<ffffffff815137c0>] ?
>>>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>>>> [ 2522.956905]  [<ffffffff8151ac6f>] ? bch_journal+0x42f/0x4b0
>>>>>>>> [ 2522.956909]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>>>>> [ 2522.956915]  [<ffffffff8151e1be>] bch_data_insert_keys+0x3e/0x160
>>>>>>>> [ 2522.956920]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>>>> [ 2522.956925]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>>>> [ 2522.956930]  [<ffffffff81063570>] ?
>>>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>>>> [ 2522.956933]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>>>> [ 2522.956938]  [<ffffffff81069e80>] ?
>>>>>>>> flush_kthread_worker+0xb0/0xb0
>>>>>>>> [ 2522.956943]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>>>> [ 2522.956947]  [<ffffffff81069e80>] ?
>>>>>>>> flush_kthread_worker+0xb0/0xb0
>>>>>>>> [ 2522.956951] INFO: task kworker/3:2:2471 blocked for more than 120
>>>>>>>> seconds.
>>>>>>>> [ 2522.956955] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>> disables this message.
>>>>>>>> [ 2522.956959] kworker/3:2     D ffffffff81813a60     0  2471      2
>>>>>>>> 0x00000000
>>>>>>>> [ 2522.956966] Workqueue: events update_writeback_rate
>>>>>>>> [ 2522.956968]  ffff882a5d125cd0 0000000000000046 0000000000000003
>>>>>>>> ffff882a5d125d68
>>>>>>>> [ 2522.956972]  ffff882f8ec94cb0 ffff882a5d125fd8 ffff882a5d125fd8
>>>>>>>> ffff882a5d125fd8
>>>>>>>> [ 2522.956977]  ffff882fa6aeb320 ffff882f8ec94cb0 0000000200000003
>>>>>>>> ffff882f8ec94cb0
>>>>>>>> [ 2522.956981] Call Trace:
>>>>>>>> [ 2522.956987]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>>>> [ 2522.956992]  [<ffffffff816bf0fd>]
>>>>>>>> rwsem_down_read_failed+0x9d/0xe5
>>>>>>>> [ 2522.956997]  [<ffffffff81332c64>]
>>>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>>>> [ 2522.957002]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>>>>> [ 2522.957007]  [<ffffffff8151b010>]
>>>>>>>> update_writeback_rate+0x30/0x230
>>>>>>>> [ 2522.957011]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>>>> [ 2522.957017]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>>>> [ 2522.957021]  [<ffffffff81063570>] ?
>>>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>>>> [ 2522.957025]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>>>> [ 2522.957030]  [<ffffffff81069e80>] ?
>>>>>>>> flush_kthread_worker+0xb0/0xb0
>>>>>>>> [ 2522.957035]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>>>> [ 2522.957039]  [<ffffffff81069e80>] ?
>>>>>>>> flush_kthread_worker+0xb0/0xb0
>>>>>>>> [ 2522.957048] INFO: task kworker/u64:2:2540 blocked for more than
>>>>>>>> 120
>>>>>>>> seconds.
>>>>>>>> [ 2522.957052] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>> disables this message.
>>>>>>>> [ 2522.957056] kworker/u64:2   D ffffffff81813a60     0  2540      2
>>>>>>>> 0x00000000
>>>>>>>> [ 2522.957062] Workqueue: bch_btree_io btree_node_write_work
>>>>>>>> [ 2522.957064]  ffff882a4fdc9778 0000000000000046 ffff882a4fdc9738
>>>>>>>> ffff882f86e3be58
>>>>>>>> [ 2522.957069]  ffff882fa4bc1990 ffff882a4fdc9fd8 ffff882a4fdc9fd8
>>>>>>>> ffff882a4fdc9fd8
>>>>>>>> [ 2522.957073]  ffff882fa6aeb320 ffff882fa4bc1990 ffff882f86e3be00
>>>>>>>> ffff8828dba00d98
>>>>>>>> [ 2522.957078] Call Trace:
>>>>>>>> [ 2522.957083]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>>>> [ 2522.957088]  [<ffffffff816be57e>]
>>>>>>>> schedule_preempt_disabled+0xe/0x10
>>>>>>>> [ 2522.957092]  [<ffffffff816bc862>]
>>>>>>>> __mutex_lock_slowpath+0x112/0x1b0
>>>>>>>> [ 2522.957102]  [<ffffffff81484090>] ?
>>>>>>>> ata_scsiop_mode_sense+0x380/0x380
>>>>>>>> [ 2522.957107]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>>>>>>> [ 2522.957111]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
>>>>>>>> [ 2522.957116]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
>>>>>>>> [ 2522.957121]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
>>>>>>>> [ 2522.957126]  [<ffffffff81076828>] ? resched_task+0x68/0x70
>>>>>>>> [ 2522.957130]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
>>>>>>>> [ 2522.957135]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
>>>>>>>> [ 2522.957140]  [<ffffffff8113140f>]
>>>>>>>> do_try_to_free_pages+0x20f/0x4b0
>>>>>>>> [ 2522.957146]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
>>>>>>>> [ 2522.957152]  [<ffffffff81126e9c>]
>>>>>>>> __alloc_pages_nodemask+0x60c/0x9b0
>>>>>>>> [ 2522.957158]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
>>>>>>>> [ 2522.957163]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
>>>>>>>> [ 2522.957168]  [<ffffffff81517fc8>] __btree_sort+0x48/0x230
>>>>>>>> [ 2522.957173]  [<ffffffff8151765c>] ?
>>>>>>>> __bch_btree_iter_init+0x7c/0xc0
>>>>>>>> [ 2522.957178]  [<ffffffff81518301>]
>>>>>>>> bch_btree_sort_partial+0x101/0x120
>>>>>>>> [ 2522.957182]  [<ffffffff8150f000>] ?
>>>>>>>> __btree_node_write_done+0x100/0x100
>>>>>>>> [ 2522.957187]  [<ffffffff81518468>] bch_btree_sort_lazy+0x68/0x90
>>>>>>>> [ 2522.957191]  [<ffffffff815109ba>]
>>>>>>>> bch_btree_node_write+0x36a/0x4a0
>>>>>>>> [ 2522.957196]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
>>>>>>>> [ 2522.957201]  [<ffffffff8106212c>] ?
>>>>>>>> pwq_activate_delayed_work+0x4c/0xb0
>>>>>>>> [ 2522.957205]  [<ffffffff81511477>] btree_node_write_work+0x57/0x80
>>>>>>>> [ 2522.957210]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>>>> [ 2522.957215]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>>>> [ 2522.957219]  [<ffffffff81063570>] ?
>>>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>>>> [ 2522.957223]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>>>> [ 2522.957228]  [<ffffffff81069e80>] ?
>>>>>>>> flush_kthread_worker+0xb0/0xb0
>>>>>>>> [ 2522.957233]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>>>> [ 2522.957237]  [<ffffffff81069e80>] ?
>>>>>>>> flush_kthread_worker+0xb0/0xb0
>>>>>>>>
>>>>>>>>
>>>>>>>> I can easily reproduce this, please let me know if you require any
>>>>>>>> info/test any patch.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Neo
>>>>>>>>
>>>>>>>> On Mon, Aug 26, 2013 at 12:21 PM, Stefan Priebe
>>>>>>>> <s.priebe@profihost.ag> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Kent,
>>>>>>>>>
>>>>>>>>> a new one under 3.10:
>>>>>>>>>
>>>>>>>>> 2013-08-26 21:05:30     INFO: task ceph-osd:8939 blocked for more
>>>>>>>>> than 120
>>>>>>>>> seconds.
>>>>>>>>> 2013-08-26 21:05:30     "echo 0 >
>>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>>> disables this message.
>>>>>>>>> 2013-08-26 21:05:30     ceph-osd D ffffffff8160f760 0 8939 1
>>>>>>>>> 0x00000000
>>>>>>>>> 2013-08-26 21:05:30     ffff880bd9a61820 0000000000000086
>>>>>>>>> ffff880bd9a61fd8
>>>>>>>>> 0000000000012c80
>>>>>>>>> 2013-08-26 21:05:30     ffff880bd9a60010 0000000000012c80
>>>>>>>>> 0000000000012c80
>>>>>>>>> 0000000000012c80
>>>>>>>>> 2013-08-26 21:05:30     ffff880bd9a61fd8 0000000000012c80
>>>>>>>>> ffff880bd7abb1c0
>>>>>>>>> ffff880c48dee380
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff811e8128>]
>>>>>>>>> xfs_vm_readpages+0x18/0x20
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810f1da3>] read_pages+0x43/0x100
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810e64df>] ?
>>>>>>>>> __page_cache_alloc+0x9f/0xc0
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810f1fab>]
>>>>>>>>> __do_page_cache_readahead+0x14b/0x160
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810f1fdc>] ra_submit+0x1c/0x20
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810f22a5>]
>>>>>>>>> ondemand_readahead+0x115/0x240
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810f249e>]
>>>>>>>>> page_cache_sync_readahead+0x2e/0x40
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810e7a5e>] T.1003+0x33e/0x430
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810e7c30>]
>>>>>>>>> generic_file_aio_read+0xe0/0x220
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff811efd0a>]
>>>>>>>>> xfs_file_aio_read+0x15a/0x2a0
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff8114095a>] do_sync_read+0x7a/0xb0
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff81140df1>] vfs_read+0xb1/0x130
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff81141757>] SyS_pread64+0x97/0xa0
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff81526b12>]
>>>>>>>>> system_call_fastpath+0x16/0x1b
>>>>>>>>> 2013-08-26 21:05:29     INFO: task ceph-osd:8896 blocked for more
>>>>>>>>> than 120
>>>>>>>>> seconds.
>>>>>>>>> 2013-08-26 21:05:29     "echo 0 >
>>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>>> disables this message.
>>>>>>>>> 2013-08-26 21:05:29     ceph-osd D ffffffff8160f760 0 8896 1
>>>>>>>>> 0x00000000
>>>>>>>>> 2013-08-26 21:05:29     ffff880bd9a9dc28 0000000000000086
>>>>>>>>> ffff880bd9a9dfd8
>>>>>>>>> 0000000000012c80
>>>>>>>>> 2013-08-26 21:05:29     ffff880bd9a9c010 0000000000012c80
>>>>>>>>> 0000000000012c80
>>>>>>>>> 0000000000012c80
>>>>>>>>> 2013-08-26 21:05:29     ffff880bd9a9dfd8 0000000000012c80
>>>>>>>>> ffff880bd79298e0
>>>>>>>>> ffff880c48dd6380
>>>>>>>>> 2013-08-26 21:05:29     Call Trace:
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810e60f0>] ?
>>>>>>>>> __lock_page+0x70/0x70
>>>>>>>>> 2013-08-26 21:05:28     "echo 0 >
>>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>>> disables this message.
>>>>>>>>> 2013-08-26 21:05:28     ceph-osd D ffffffff8160f760 0 8798 1
>>>>>>>>> 0x00000000
>>>>>>>>> 2013-08-26 21:05:28     ffff880bd998b740 0000000000000086
>>>>>>>>> ffff880bd998bfd8
>>>>>>>>> 0000000000012c80
>>>>>>>>> 2013-08-26 21:05:28     ffff880bd998a010 0000000000012c80
>>>>>>>>> 0000000000012c80
>>>>>>>>> 0000000000012c80
>>>>>>>>> 2013-08-26 21:05:28     ffff880bd998bfd8 0000000000012c80
>>>>>>>>> ffff880bd99c0000
>>>>>>>>> ffff880c48dd6380
>>>>>>>>> 2013-08-26 21:05:28     Call Trace:
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff81524874>] schedule+0x24/0x70
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff81525d7d>]
>>>>>>>>> rwsem_down_read_failed+0x9d/0xe5
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff812aa0a4>]
>>>>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff81523a42>] ? down_read+0x12/0x20
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffffa01ead62>]
>>>>>>>>> btree_read_async+0xa2/0x1a0
>>>>>>>>> [bcache]
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffffa01f7b73>]
>>>>>>>>> closure_queue+0x43/0x60
>>>>>>>>> [bcache]
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffffa01eb903>] T.1050+0x63/0x70
>>>>>>>>> [bcache]
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffffa01ed6ea>]
>>>>>>>>> cached_dev_make_request+0x20a/0x350 [bcache]
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff81281952>]
>>>>>>>>> generic_make_request+0xc2/0x100
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff812819f7>] submit_bio+0x67/0x130
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff8117b240>]
>>>>>>>>> do_mpage_readpage+0x2a0/0x660
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff81101cee>] ?
>>>>>>>>> __inc_zone_page_state+0x2e/0x30
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff810e6581>] ?
>>>>>>>>> add_to_page_cache_locked+0x81/0x110
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff8117b76a>]
>>>>>>>>> mpage_readpages+0xfa/0x150
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff811e90c0>] ?
>>>>>>>>> xfs_get_blocks_direct+0x20/0x20
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff811e90c0>] ?
>>>>>>>>> xfs_get_blocks_direct+0x20/0x20
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff810eeda3>] ?
>>>>>>>>> __alloc_pages_nodemask+0x123/0x240
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff8115bf93>] ? __close_fd+0x43/0x80
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff8116dabe>] SyS_fdatasync+0xe/0x20
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81526b12>]
>>>>>>>>> system_call_fastpath+0x16/0x1b
>>>>>>>>> 2013-08-26 21:05:27     INFO: task ceph-osd:8746 blocked for more
>>>>>>>>> than 120
>>>>>>>>> seconds.
>>>>>>>>> 2013-08-26 21:05:27     "echo 0 >
>>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>>> disables this message.
>>>>>>>>> 2013-08-26 21:05:27     ceph-osd D ffffffff8160f760 0 8746 1
>>>>>>>>> 0x00000000
>>>>>>>>> 2013-08-26 21:05:27     ffff880c2dde1d38 0000000000000086
>>>>>>>>> ffff880c2dde1fd8
>>>>>>>>> 0000000000012c80
>>>>>>>>> 2013-08-26 21:05:27     ffff880c2dde0010 0000000000012c80
>>>>>>>>> 0000000000012c80
>>>>>>>>> 0000000000012c80
>>>>>>>>> 2013-08-26 21:05:27     ffff880c2dde1fd8 0000000000012c80
>>>>>>>>> ffff880c239eb1c0
>>>>>>>>> ffff880c48dd6380
>>>>>>>>> 2013-08-26 21:05:27     Call Trace:
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81524874>] schedule+0x24/0x70
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81522bbd>]
>>>>>>>>> schedule_timeout+0x16d/0x200
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81068eba>] ?
>>>>>>>>> __queue_delayed_work+0xaa/0x1a0
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81068d19>] ?
>>>>>>>>> try_to_grab_pending+0x109/0x190
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff815251e5>]
>>>>>>>>> wait_for_completion+0x95/0x110
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff8107fd00>] ?
>>>>>>>>> try_to_wake_up+0x2a0/0x2a0
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81167797>] ?
>>>>>>>>> bdi_queue_work+0x77/0xc0
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81167863>]
>>>>>>>>> writeback_inodes_sb_nr+0x83/0xb0
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff811678ea>]
>>>>>>>>> writeback_inodes_sb+0x5a/0x70
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff8116dd0a>]
>>>>>>>>> __sync_filesystem+0x4a/0x50
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff8116dd42>]
>>>>>>>>> sync_filesystem+0x32/0x60
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff8116ddc0>] SyS_syncfs+0x50/0x90
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81526b12>]
>>>>>>>>> system_call_fastpath+0x16/0x1b
>>>>>>>>> 2013-08-26 21:05:27     INFO: task ceph-osd:8798 blocked for more
>>>>>>>>> than 120
>>>>>>>>> seconds.
>>>>>>>>>
>>>>>>>>> Stefan
>>>>>>>>>
>>>>>>>>> Am 22.08.2013 09:32, schrieb Stefan Priebe - Profihost AG:
>>>>>>>>>
>>>>>>>>>> great!
>>>>>>>>>>
>>>>>>>>>> Everything seems to work fine now! Except read_dirty always going
>>>>>>>>>> to
>>>>>>>>>> negative values after a reboot.
>>>>>>>>>>
>>>>>>>>>> Stefan
>>>>>>>>>>
>>>>>>>>>> Am 22.08.2013 08:02, schrieb Kent Overstreet:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Aug 22, 2013 at 07:59:04AM +0200, Stefan Priebe wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> schedule_timeout() is not the same as
>>>>>>>>>>>>> schedule_timeout_interruptible().
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> just search and replace? So i can try on my own.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> The one in read_dirty(), line ~330
>>>>>>>>>>> --
>>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>>>> linux-bcache"
>>>>>>>>>>> in
>>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>>> More majordomo info at
>>>>>>>>>>> http://vger.kernel.org/majordomo-info.html
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>> linux-bcache" in
>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>> linux-bcache"
>>>>>> in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 28+ messages in thread

* Re: bcache: Fix a writeback performance regression
  2013-08-28 20:15                                         ` Stefan Priebe
  2013-08-28 22:45                                           ` kernel neophyte
@ 2013-08-29 18:52                                           ` Stefan Priebe
  1 sibling, 0 replies; 28+ messages in thread
From: Stefan Priebe @ 2013-08-29 18:52 UTC (permalink / raw)
  To: kernel neophyte
  Cc: Kent Overstreet, Jens Axboe, linux-kernel@vger.kernel.org,
	linux-bcache@vger.kernel.org

sorry seems i see something similiar:
http://pastebin.com/raw.php?i=ZqgLf9gp

Stefan

Am 28.08.2013 22:15, schrieb Stefan Priebe:
> sorry but that's completely wrong.
>
> please use branch bcache-for-3.10
>
> http://evilpiepirate.org/git/linux-bcache.git/log/?h=bcache-for-3.10
>
> Stefan
>
> Am 28.08.2013 22:12, schrieb kernel neophyte:
>> On Wed, Aug 28, 2013 at 12:20 PM, Stefan Priebe
>> <s.priebe@profihost.ag> wrote:
>>> Am 28.08.2013 20:47, schrieb kernel neophyte:
>>>
>>>> On Wed, Aug 28, 2013 at 11:38 AM, Stefan Priebe - Profihost AG
>>>> <s.priebe@profihost.ag> wrote:
>>>>>
>>>>> I don't had one for a few days. Which kernel so you use?
>>>>
>>>>
>>>> 3.10 kernel with all of kent's stable patches and perf patches..
>>>
>>>
>>> Which exact 3.10 version? Which patches on top?
>>
>> 3.10.0-rc7
>> with following patches:
>> bcache: Fix a dumb CPU spinning bug in writeback
>> bcache: Fix a flush/fua performance bug
>> bcache: Fix a writeback performance regression
>> bcache: FUA fixes
>>
>> I have also tried bcache perf branch i.e.
>> (http://evilpiepirate.org/git/linux-bcache.git/log/?h=bcache-perf)
>>
>>>
>>>
>>>> and all defaults except for:
>>>>
>>>> echo writeback > /sys/block/bcache0/bcache/cache_mode
>>>> echo 0 > /sys/block/bcache0/bcache/sequential_cutoff
>>>> echo 0 >
>>>> /sys/fs/bcache/1e26e5ca-5cb9-4c5a-ad7e-3c787524a9dc/congested_read_threshold_us
>>>>
>>>> echo 0 >
>>>> /sys/fs/bcache/1e26e5ca-5cb9-4c5a-ad7e-3c787524a9dc/congested_write_threshold_us
>>>>
>>>>
>>>> -Neo
>>>>
>>>>>
>>>>> Stefan
>>>>>
>>>>> This mail was sent with my iPhone.
>>>>>
>>>>> Am 28.08.2013 um 20:12 schrieb kernel neophyte
>>>>> <neophyte.hacker001@gmail.com>:
>>>>>
>>>>>> On Tue, Aug 27, 2013 at 11:05 PM, Kent Overstreet <kmo@daterainc.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> On Tue, Aug 27, 2013 at 08:37:54PM -0700, kernel neophyte wrote:
>>>>>>>>
>>>>>>>> Hi Kent,
>>>>>>>>
>>>>>>>> I am still seeing deadlock:
>>>>>>>
>>>>>>>
>>>>>>> Seeing blk_throtl_bio (what asshole misspells words in their
>>>>>>> function
>>>>>>> names, so when you go to grep for them you grep for the wrong
>>>>>>> thing?)
>>>>>>> in
>>>>>>> the backtrace - is that what it takes to hit it?
>>>>>>
>>>>>>
>>>>>> I don't think so,
>>>>>> I had another deadlock:
>>>>>>
>>>>>> [ 1179.193990] bio: create slab <bio-2> at 2
>>>>>> [ 1179.194491] bcache: run_cache_set() invalidating existing data
>>>>>> [ 1179.212308] bcache: register_cache() registered cache device rxd1
>>>>>> [ 1203.846220] bcache: register_bdev() registered backing device sda3
>>>>>> [ 1227.780088] bcache: bch_cached_dev_attach() Caching sda3 as
>>>>>> bcache0
>>>>>> on set a079dda3-27fd-4b1c-a616-d46110bfae6c
>>>>>> [ 2162.497950] INFO: task kworker/0:0:4 blocked for more than 120
>>>>>> seconds.
>>>>>> [ 2162.497961] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2162.497967] kworker/0:0     D 0000000000000000     0     4      2
>>>>>> 0x00000000
>>>>>> [ 2162.497982] Workqueue: events write_dirty_finish
>>>>>> [ 2162.497986]  ffff882fa6ac9b30 0000000000000046 ffffffff81511b8c
>>>>>> ffffffff815137c0
>>>>>> [ 2162.497992]  ffff882fa6a84cb0 ffff882fa6ac9fd8 ffff882fa6ac9fd8
>>>>>> ffff882fa6ac9fd8
>>>>>> [ 2162.497996]  ffff882f78139990 ffff882fa6a84cb0 ffffffff815117a1
>>>>>> ffff882fa6a84cb0
>>>>>> [ 2162.498001] Call Trace:
>>>>>> [ 2162.498010]  [<ffffffff81511b8c>] ?
>>>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>>>> [ 2162.498015]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.498019]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>>>> [ 2162.498028]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2162.498034]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>>> [ 2162.498043]  [<ffffffff81332c64>]
>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>> [ 2162.498048]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>>> [ 2162.498052]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>>>> [ 2162.498057]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.498066]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>>> [ 2162.498072]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>>>> [ 2162.498076]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>>> [ 2162.498082]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>>>> [ 2162.498087]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>>>> [ 2162.498093]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>>>> [ 2162.498100]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>> [ 2162.498105]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>> [ 2162.498110]  [<ffffffff81063570>] ?
>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>> [ 2162.498115]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>> [ 2162.498120]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498127]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>> [ 2162.498131]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498145] INFO: task bcache_allocato:1766 blocked for more than
>>>>>> 120 seconds.
>>>>>> [ 2162.498149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2162.498154] bcache_allocato D 0000000000000001     0  1766      2
>>>>>> 0x00000000
>>>>>> [ 2162.498158]  ffff8822630f3d88 0000000000000046 ffff8822630f3da8
>>>>>> ffffffff810808ad
>>>>>> [ 2162.498163]  ffff882f891de640 ffff8822630f3fd8 ffff8822630f3fd8
>>>>>> ffff8822630f3fd8
>>>>>> [ 2162.498167]  ffff882f81a91990 ffff882f891de640 ffff882f88b33320
>>>>>> ffff8801fad20d98
>>>>>> [ 2162.498172] Call Trace:
>>>>>> [ 2162.498177]  [<ffffffff810808ad>] ? dequeue_task_fair+0x2cd/0x530
>>>>>> [ 2162.498182]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2162.498187]  [<ffffffff816be57e>]
>>>>>> schedule_preempt_disabled+0xe/0x10
>>>>>> [ 2162.498192]  [<ffffffff816bc862>]
>>>>>> __mutex_lock_slowpath+0x112/0x1b0
>>>>>> [ 2162.498197]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>>>>> [ 2162.498202]  [<ffffffff8150cdbf>] bch_allocator_thread+0x10f/0xe20
>>>>>> [ 2162.498207]  [<ffffffff8150ccb0>] ?
>>>>>> bch_bucket_add_unused+0xe0/0xe0
>>>>>> [ 2162.498211]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>> [ 2162.498215]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498220]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>> [ 2162.498224]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498228] INFO: task bcache_writebac:1771 blocked for more than
>>>>>> 120 seconds.
>>>>>> [ 2162.498233] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2162.498237] bcache_writebac D 0000000000000000     0  1771      2
>>>>>> 0x00000000
>>>>>> [ 2162.498241]  ffff88197d2adc58 0000000000000046 ffff88197d2add48
>>>>>> ffffffff811269e2
>>>>>> [ 2162.498245]  ffff882f891d8000 ffff88197d2adfd8 ffff88197d2adfd8
>>>>>> ffff88197d2adfd8
>>>>>> [ 2162.498250]  ffff882f81a91990 ffff882f891d8000 ffff88197d2adc68
>>>>>> ffff882f89550ba0
>>>>>> [ 2162.498254] Call Trace:
>>>>>> [ 2162.498264]  [<ffffffff811269e2>] ?
>>>>>> __alloc_pages_nodemask+0x152/0x9b0
>>>>>> [ 2162.498269]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2162.498274]  [<ffffffff816bc015>] schedule_timeout+0x1e5/0x250
>>>>>> [ 2162.498280]  [<ffffffff810799bf>] ? try_to_wake_up+0x24f/0x2b0
>>>>>> [ 2162.498285]  [<ffffffff816bcd09>] __down_common+0x8e/0xe3
>>>>>> [ 2162.498293]  [<ffffffff81160600>] ? alloc_pages_current+0x90/0x170
>>>>>> [ 2162.498298]  [<ffffffff816bcdd1>] __down+0x1d/0x1f
>>>>>> [ 2162.498304]  [<ffffffff8106f351>] down+0x41/0x50
>>>>>> [ 2162.498310]  [<ffffffff8151ba7d>] bch_writeback_thread+0x35d/0x7f0
>>>>>> [ 2162.498315]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
>>>>>> [ 2162.498321]  [<ffffffff8151b720>] ? write_dirty+0xc0/0xc0
>>>>>> [ 2162.498325]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>> [ 2162.498329]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498334]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>> [ 2162.498338]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498342] INFO: task kworker/0:2:1789 blocked for more than 120
>>>>>> seconds.
>>>>>> [ 2162.498346] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2162.498351] kworker/0:2     D ffffffff81813a60     0  1789      2
>>>>>> 0x00000000
>>>>>> [ 2162.498357] Workqueue: events write_dirty_finish
>>>>>> [ 2162.498360]  ffff88277062fb30 0000000000000046 ffffffff81511b8c
>>>>>> ffffffff815137c0
>>>>>> [ 2162.498364]  ffff882fa6786640 ffff88277062ffd8 ffff88277062ffd8
>>>>>> ffff88277062ffd8
>>>>>> [ 2162.498368]  ffffffff81c10440 ffff882fa6786640 ffffffff815117a1
>>>>>> ffff882fa6786640
>>>>>> [ 2162.498373] Call Trace:
>>>>>> [ 2162.498378]  [<ffffffff81511b8c>] ?
>>>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>>>> [ 2162.498382]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.498387]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>>>> [ 2162.498392]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2162.498397]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>>> [ 2162.498402]  [<ffffffff81332c64>]
>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>> [ 2162.498407]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>>> [ 2162.498411]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>>>> [ 2162.498416]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.498420]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>>> [ 2162.498425]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>>>> [ 2162.498430]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>>> [ 2162.498435]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>>>> [ 2162.498439]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>>>> [ 2162.498444]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>>>> [ 2162.498449]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>> [ 2162.498454]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>> [ 2162.498459]  [<ffffffff81063570>] ?
>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>> [ 2162.498463]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>> [ 2162.498467]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498472]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>> [ 2162.498476]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498480] INFO: task iozone:2052 blocked for more than 120
>>>>>> seconds.
>>>>>> [ 2162.498484] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2162.498488] iozone          D 0000000000000001     0  2052   1944
>>>>>> 0x00000000
>>>>>> [ 2162.498492]  ffff882f7e2d3978 0000000000000082 ffff882f7e2d3938
>>>>>> ffffffff81301d7a
>>>>>> [ 2162.498497]  ffff882f8a116640 ffff882f7e2d3fd8 ffff882f7e2d3fd8
>>>>>> ffff882f7e2d3fd8
>>>>>> [ 2162.498501]  ffff882f88b33320 ffff882f8a116640 ffff882f7e2d3978
>>>>>> ffff882fbf2339f8
>>>>>> [ 2162.498505] Call Trace:
>>>>>> [ 2162.498511]  [<ffffffff81301d7a>] ?
>>>>>> generic_make_request+0xca/0x100
>>>>>> [ 2162.498517]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2162.498522]  [<ffffffff816be36f>] io_schedule+0x8f/0xd0
>>>>>> [ 2162.498530]  [<ffffffff811b797c>]
>>>>>> do_blockdev_direct_IO+0x1a7c/0x1fb0
>>>>>> [ 2162.498539]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60
>>>>>> [ext2]
>>>>>> [ 2162.498546]  [<ffffffff811b7f05>] __blockdev_direct_IO+0x55/0x60
>>>>>> [ 2162.498552]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60
>>>>>> [ext2]
>>>>>> [ 2162.498556]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>>> [ 2162.498562]  [<ffffffffa0274349>] ext2_direct_IO+0x79/0xe0 [ext2]
>>>>>> [ 2162.498567]  [<ffffffffa0273a80>] ? ext2_get_blocks+0xa60/0xa60
>>>>>> [ext2]
>>>>>> [ 2162.498572]  [<ffffffff8107e775>] ? set_next_entity+0xa5/0xc0
>>>>>> [ 2162.498579]  [<ffffffff8104ade6>] ? current_fs_time+0x16/0x60
>>>>>> [ 2162.498585]  [<ffffffff8111f126>]
>>>>>> generic_file_direct_write+0xc6/0x180
>>>>>> [ 2162.498590]  [<ffffffff8111f4bd>]
>>>>>> __generic_file_aio_write+0x2dd/0x3b0
>>>>>> [ 2162.498595]  [<ffffffff816bc364>] ? mutex_unlock+0x14/0x20
>>>>>> [ 2162.498600]  [<ffffffff8111f5f9>] generic_file_aio_write+0x69/0xd0
>>>>>> [ 2162.498607]  [<ffffffff8117b88a>] do_sync_write+0x7a/0xb0
>>>>>> [ 2162.498612]  [<ffffffff8117c63e>] vfs_write+0xce/0x1e0
>>>>>> [ 2162.498617]  [<ffffffff8117bba0>] ?
>>>>>> generic_file_llseek_size+0x110/0x110
>>>>>> [ 2162.498621]  [<ffffffff8117cb22>] SyS_write+0x52/0xa0
>>>>>> [ 2162.498627]  [<ffffffff816c7a02>] system_call_fastpath+0x16/0x1b
>>>>>> [ 2162.498630] INFO: task kworker/0:1:2293 blocked for more than 120
>>>>>> seconds.
>>>>>> [ 2162.498635] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2162.498639] kworker/0:1     D ffffffff81813a60     0  2293      2
>>>>>> 0x00000000
>>>>>> [ 2162.498645] Workqueue: events write_dirty_finish
>>>>>> [ 2162.498647]  ffff882f8a05bb30 0000000000000046 ffffffff81511b8c
>>>>>> ffffffff815137c0
>>>>>> [ 2162.498652]  ffff882f8a163320 ffff882f8a05bfd8 ffff882f8a05bfd8
>>>>>> ffff882f8a05bfd8
>>>>>> [ 2162.498656]  ffffffff81c10440 ffff882f8a163320 ffffffff815117a1
>>>>>> ffff882f8a163320
>>>>>> [ 2162.498661] Call Trace:
>>>>>> [ 2162.498665]  [<ffffffff81511b8c>] ?
>>>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>>>> [ 2162.498669]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.498674]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>>>> [ 2162.498679]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2162.498684]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>>> [ 2162.498689]  [<ffffffff81332c64>]
>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>> [ 2162.498693]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>>> [ 2162.498698]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>>>> [ 2162.498702]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.498707]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>>> [ 2162.498712]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>>>> [ 2162.498716]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>>> [ 2162.498721]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>>>> [ 2162.498726]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>>>> [ 2162.498731]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>>>> [ 2162.498735]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>> [ 2162.498740]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>> [ 2162.498745]  [<ffffffff81063570>] ?
>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>> [ 2162.498749]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>> [ 2162.498753]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498758]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>> [ 2162.498762]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498766] INFO: task kworker/0:3:2534 blocked for more than 120
>>>>>> seconds.
>>>>>> [ 2162.498772] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2162.498777] kworker/0:3     D ffffffff81813a60     0  2534      2
>>>>>> 0x00000000
>>>>>> [ 2162.498783] Workqueue: events write_dirty_finish
>>>>>> [ 2162.498785]  ffff88237d0bdb30 0000000000000046 ffffffff81511b8c
>>>>>> ffffffff815137c0
>>>>>> [ 2162.498790]  ffff882fa66fe640 ffff88237d0bdfd8 ffff88237d0bdfd8
>>>>>> ffff88237d0bdfd8
>>>>>> [ 2162.498794]  ffffffff81c10440 ffff882fa66fe640 ffffffff815117a1
>>>>>> ffff882fa66fe640
>>>>>> [ 2162.498799] Call Trace:
>>>>>> [ 2162.498803]  [<ffffffff81511b8c>] ?
>>>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>>>> [ 2162.498807]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.498812]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>>>> [ 2162.498817]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2162.498822]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>>> [ 2162.498826]  [<ffffffff81332c64>]
>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>> [ 2162.498831]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>>> [ 2162.498836]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>>>> [ 2162.498840]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.498845]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>>> [ 2162.498849]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>>>> [ 2162.498854]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>>> [ 2162.498859]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>>>> [ 2162.498864]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>>>> [ 2162.498868]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>>>> [ 2162.498873]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>> [ 2162.498878]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>> [ 2162.498882]  [<ffffffff81063570>] ?
>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>> [ 2162.498886]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>> [ 2162.498890]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498896]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>> [ 2162.498900]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.498903] INFO: task kworker/0:4:2537 blocked for more than 120
>>>>>> seconds.
>>>>>> [ 2162.498908] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2162.498914] kworker/0:4     D 0000000000000000     0  2537      2
>>>>>> 0x00000000
>>>>>> [ 2162.498920] Workqueue: events write_dirty_finish
>>>>>> [ 2162.498922]  ffff882f76d8fb30 0000000000000046 ffffffff81511b8c
>>>>>> ffffffff815137c0
>>>>>> [ 2162.498926]  ffff882f88b30000 ffff882f76d8ffd8 ffff882f76d8ffd8
>>>>>> ffff882f76d8ffd8
>>>>>> [ 2162.498931]  ffff882f78beb320 ffff882f88b30000 ffffffff815117a1
>>>>>> ffff882f88b30000
>>>>>> [ 2162.498935] Call Trace:
>>>>>> [ 2162.498940]  [<ffffffff81511b8c>] ?
>>>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>>>> [ 2162.498944]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.498948]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>>>> [ 2162.498953]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2162.498958]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>>> [ 2162.498963]  [<ffffffff81332c64>]
>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>> [ 2162.498968]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>>> [ 2162.498972]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>>>> [ 2162.498977]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.498981]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>>> [ 2162.498986]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>>>> [ 2162.498990]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>>> [ 2162.498995]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>>>> [ 2162.499000]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>>>> [ 2162.499004]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>>>> [ 2162.499009]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>> [ 2162.499014]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>> [ 2162.499019]  [<ffffffff81063570>] ?
>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>> [ 2162.499023]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>> [ 2162.499027]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.499032]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>> [ 2162.499036]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.499039] INFO: task kworker/1:1:2544 blocked for more than 120
>>>>>> seconds.
>>>>>> [ 2162.499044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2162.499050] kworker/1:1     D ffffffff81813d40     0  2544      2
>>>>>> 0x00000000
>>>>>> [ 2162.499057] Workqueue: bcache bch_data_insert_keys
>>>>>> [ 2162.499059]  ffff882fa677b3f8 0000000000000046 ffff882fa677b418
>>>>>> ffff882f8a709000
>>>>>> [ 2162.499063]  ffff882f88b33320 ffff882fa677bfd8 ffff882fa677bfd8
>>>>>> ffff882fa677bfd8
>>>>>> [ 2162.499068]  ffff882fa6471990 ffff882f88b33320 ffff882f88b33320
>>>>>> ffff8801fad20d98
>>>>>> [ 2162.499072] Call Trace:
>>>>>> [ 2162.499077]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2162.499082]  [<ffffffff816be57e>]
>>>>>> schedule_preempt_disabled+0xe/0x10
>>>>>> [ 2162.499087]  [<ffffffff816bc862>]
>>>>>> __mutex_lock_slowpath+0x112/0x1b0
>>>>>> [ 2162.499092]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>>>>> [ 2162.499096]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
>>>>>> [ 2162.499102]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
>>>>>> [ 2162.499109]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
>>>>>> [ 2162.499113]  [<ffffffff81076828>] ? resched_task+0x68/0x70
>>>>>> [ 2162.499118]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
>>>>>> [ 2162.499126]  [<ffffffff8113a379>] ? fragmentation_index+0x19/0x70
>>>>>> [ 2162.499131]  [<ffffffff8113140f>] do_try_to_free_pages+0x20f/0x4b0
>>>>>> [ 2162.499137]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
>>>>>> [ 2162.499143]  [<ffffffff81126e9c>]
>>>>>> __alloc_pages_nodemask+0x60c/0x9b0
>>>>>> [ 2162.499149]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
>>>>>> [ 2162.499155]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
>>>>>> [ 2162.499159]  [<ffffffff8150ebb3>] mca_data_alloc+0x73/0x1d0
>>>>>> [ 2162.499163]  [<ffffffff81510f37>] mca_alloc+0x277/0x470
>>>>>> [ 2162.499168]  [<ffffffff81511d1c>] bch_btree_node_alloc+0x8c/0x1c0
>>>>>> [ 2162.499173]  [<ffffffff81513020>] btree_split+0x110/0x5c0
>>>>>> [ 2162.499179]  [<ffffffff81457d43>] ?
>>>>>> scsi_pool_alloc_command+0x33/0x80
>>>>>> [ 2162.499184]  [<ffffffff81515fc7>] ?
>>>>>> bch_keylist_pop_front+0x47/0x50
>>>>>> [ 2162.499188]  [<ffffffff8150fed6>] ?
>>>>>> bch_btree_insert_keys+0x56/0x250
>>>>>> [ 2162.499193]  [<ffffffff81082def>] ? update_group_power+0x13f/0x220
>>>>>> [ 2162.499198]  [<ffffffff81513582>] bch_btree_insert_node+0xb2/0x2f0
>>>>>> [ 2162.499202]  [<ffffffff815137e8>] btree_insert_fn+0x28/0x50
>>>>>> [ 2162.499207]  [<ffffffff81511b8c>]
>>>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>>>> [ 2162.499211]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.499216]  [<ffffffff816bcc26>] ? down_write+0x16/0x40
>>>>>> [ 2162.499221]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>>>> [ 2162.499225]  [<ffffffff81511c30>]
>>>>>> bch_btree_map_nodes_recurse+0x110/0x170
>>>>>> [ 2162.499230]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.499235]  [<ffffffff811b51ea>] ? dio_bio_end_io+0x5a/0x90
>>>>>> [ 2162.499239]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>>> [ 2162.499244]  [<ffffffff81514dce>]
>>>>>> __bch_btree_map_nodes+0x13e/0x1c0
>>>>>> [ 2162.499248]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.499254]  [<ffffffff8151ac6f>] ? bch_journal+0x42f/0x4b0
>>>>>> [ 2162.499258]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>>> [ 2162.499264]  [<ffffffff8151e1be>] bch_data_insert_keys+0x3e/0x160
>>>>>> [ 2162.499268]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>> [ 2162.499273]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>> [ 2162.499278]  [<ffffffff81063570>] ?
>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>> [ 2162.499282]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>> [ 2162.499286]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.499292]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>> [ 2162.499296]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.499299] INFO: task kworker/0:5:2552 blocked for more than 120
>>>>>> seconds.
>>>>>> [ 2162.499304] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>> disables this message.
>>>>>> [ 2162.499310] kworker/0:5     D 0000000000000000     0  2552      2
>>>>>> 0x00000000
>>>>>> [ 2162.499411] Workqueue: events write_dirty_finish
>>>>>> [ 2162.499413]  ffff88226314db30 0000000000000046 ffffffff81511b8c
>>>>>> ffffffff815137c0
>>>>>> [ 2162.499418]  ffff882fa6781990 ffff88226314dfd8 ffff88226314dfd8
>>>>>> ffff88226314dfd8
>>>>>> [ 2162.499422]  ffff882fa6a84cb0 ffff882fa6781990 ffffffff815117a1
>>>>>> ffff882fa6781990
>>>>>> [ 2162.499427] Call Trace:
>>>>>> [ 2162.499431]  [<ffffffff81511b8c>] ?
>>>>>> bch_btree_map_nodes_recurse+0x6c/0x170
>>>>>> [ 2162.499435]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.499440]  [<ffffffff815117a1>] ? bch_btree_node_get+0x71/0x280
>>>>>> [ 2162.499445]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>> [ 2162.499450]  [<ffffffff816bf0fd>] rwsem_down_read_failed+0x9d/0xe5
>>>>>> [ 2162.499455]  [<ffffffff81332c64>]
>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>> [ 2162.499460]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>>> [ 2162.499464]  [<ffffffff81514d75>] __bch_btree_map_nodes+0xe5/0x1c0
>>>>>> [ 2162.499468]  [<ffffffff815137c0>] ?
>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>> [ 2162.499473]  [<ffffffff8151b21e>] ? dirty_io_destructor+0xe/0x10
>>>>>> [ 2162.499478]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>>> [ 2162.499483]  [<ffffffff8151b553>] write_dirty_finish+0x1e3/0x270
>>>>>> [ 2162.499488]  [<ffffffff8107e7fb>] ? pick_next_task_fair+0x6b/0x150
>>>>>> [ 2162.499492]  [<ffffffff810765f3>] ? dequeue_task+0x93/0xb0
>>>>>> [ 2162.499497]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>> [ 2162.499502]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>> [ 2162.499507]  [<ffffffff81063570>] ?
>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>> [ 2162.499511]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>> [ 2162.499515]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>> [ 2162.499520]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>> [ 2162.499524]  [<ffffffff81069e80>] ? flush_kthread_worker+0xb0/0xb0
>>>>>>
>>>>>>
>>>>>>
>>>>>>>> [  930.169234] bcache: bch_cached_dev_attach() Caching sda3 as
>>>>>>>> bcache0
>>>>>>>> on set 06496904-10ca-489a-ae75-68c6a07d3db1
>>>>>>>> [ 2522.956188] INFO: task bcache_writebac:2058 blocked for more
>>>>>>>> than
>>>>>>>> 120 seconds.
>>>>>>>> [ 2522.956199] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>> disables this message.
>>>>>>>> [ 2522.956205] bcache_writebac D ffffffff81813a60     0
>>>>>>>> 2058      2
>>>>>>>> 0x00000000
>>>>>>>> [ 2522.956212]  ffff881c39c1fcd8 0000000000000046 0000000000000001
>>>>>>>> 0000000000000001
>>>>>>>> [ 2522.956218]  ffff882f8b56b320 ffff881c39c1ffd8 ffff881c39c1ffd8
>>>>>>>> ffff881c39c1ffd8
>>>>>>>> [ 2522.956222]  ffff882fa6aeb320 ffff882f8b56b320 ffff882f8b56b320
>>>>>>>> ffff882f8b56b320
>>>>>>>> [ 2522.956227] Call Trace:
>>>>>>>> [ 2522.956241]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>>>> [ 2522.956247]  [<ffffffff816befb5>]
>>>>>>>> rwsem_down_write_failed+0xf5/0x1a0
>>>>>>>> [ 2522.956255]  [<ffffffff81332c93>]
>>>>>>>> call_rwsem_down_write_failed+0x13/0x20
>>>>>>>> [ 2522.956261]  [<ffffffff816bcc41>] ? down_write+0x31/0x40
>>>>>>>> [ 2522.956269]  [<ffffffff8151b782>]
>>>>>>>> bch_writeback_thread+0x62/0x7f0
>>>>>>>> [ 2522.956278]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
>>>>>>>> [ 2522.956284]  [<ffffffff8151b720>] ? write_dirty+0xc0/0xc0
>>>>>>>> [ 2522.956290]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>>>> [ 2522.956294]  [<ffffffff81069e80>] ?
>>>>>>>> flush_kthread_worker+0xb0/0xb0
>>>>>>>> [ 2522.956302]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>>>> [ 2522.956306]  [<ffffffff81069e80>] ?
>>>>>>>> flush_kthread_worker+0xb0/0xb0
>>>>>>>> [ 2522.956313] INFO: task kworker/u64:1:2433 blocked for more
>>>>>>>> than 120
>>>>>>>> seconds.
>>>>>>>> [ 2522.956317] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>> disables this message.
>>>>>>>> [ 2522.956322] kworker/u64:1   D ffffffff81813a60     0
>>>>>>>> 2433      2
>>>>>>>> 0x00000000
>>>>>>>> [ 2522.956332] Workqueue: writeback bdi_writeback_workfn
>>>>>>>> (flush-252:0)
>>>>>>>> [ 2522.956336]  ffff882f899d1560 0000000000000046 ffffffff81120635
>>>>>>>> 0000000000000100
>>>>>>>> [ 2522.956340]  ffff88219b488000 ffff882f899d1fd8 ffff882f899d1fd8
>>>>>>>> ffff882f899d1fd8
>>>>>>>> [ 2522.956345]  ffff882fa6aeb320 ffff88219b488000 ffffffff8131ad55
>>>>>>>> ffff88219b488000
>>>>>>>> [ 2522.956349] Call Trace:
>>>>>>>> [ 2522.956359]  [<ffffffff81120635>] ? mempool_alloc_slab+0x15/0x20
>>>>>>>> [ 2522.956367]  [<ffffffff8131ad55>] ? blk_throtl_bio+0x3b5/0x540
>>>>>>>> [ 2522.956372]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>>>> [ 2522.956378]  [<ffffffff816bf0fd>]
>>>>>>>> rwsem_down_read_failed+0x9d/0xe5
>>>>>>>> [ 2522.956383]  [<ffffffff81332c64>]
>>>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>>>> [ 2522.956388]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>>>>> [ 2522.956394]  [<ffffffff8151fe32>]
>>>>>>>> cached_dev_make_request+0x752/0xf00
>>>>>>>> [ 2522.956400]  [<ffffffff81301af8>] ?
>>>>>>>> generic_make_request_checks+0x1e8/0x3a0
>>>>>>>> [ 2522.956405]  [<ffffffff81301d7a>]
>>>>>>>> generic_make_request+0xca/0x100
>>>>>>>> [ 2522.956409]  [<ffffffff81301e29>] submit_bio+0x79/0x160
>>>>>>>> [ 2522.956417]  [<ffffffff811b1710>] ? bio_alloc_bioset+0xa0/0x1d0
>>>>>>>> [ 2522.956421]  [<ffffffff811abd3f>] _submit_bh+0x13f/0x200
>>>>>>>> [ 2522.956425]  [<ffffffff811abe10>] submit_bh+0x10/0x20
>>>>>>>> [ 2522.956430]  [<ffffffff811af6b8>]
>>>>>>>> __block_write_full_page+0x1d8/0x360
>>>>>>>> [ 2522.956439]  [<ffffffff8145fc0b>] ? scsi_request_fn+0xbb/0x530
>>>>>>>> [ 2522.956444]  [<ffffffff811ad3c0>] ?
>>>>>>>> end_buffer_async_read+0x130/0x130
>>>>>>>> [ 2522.956449]  [<ffffffff811b2c40>] ? I_BDEV+0x10/0x10
>>>>>>>> [ 2522.956454]  [<ffffffff811b2c40>] ? I_BDEV+0x10/0x10
>>>>>>>> [ 2522.956459]  [<ffffffff811af90a>]
>>>>>>>> block_write_full_page_endio+0xca/0x100
>>>>>>>> [ 2522.956464]  [<ffffffff811af955>]
>>>>>>>> block_write_full_page+0x15/0x20
>>>>>>>> [ 2522.956469]  [<ffffffff811b3608>] blkdev_writepage+0x18/0x20
>>>>>>>> [ 2522.956473]  [<ffffffff81127d37>] __writepage+0x17/0x40
>>>>>>>> [ 2522.956477]  [<ffffffff8112820e>] write_cache_pages+0x20e/0x460
>>>>>>>> [ 2522.956481]  [<ffffffff81127d20>] ?
>>>>>>>> set_page_dirty_lock+0x60/0x60
>>>>>>>> [ 2522.956486]  [<ffffffff811ac251>] ? __set_page_dirty+0x71/0xc0
>>>>>>>> [ 2522.956490]  [<ffffffff811284aa>] generic_writepages+0x4a/0x70
>>>>>>>> [ 2522.956495]  [<ffffffff81129cd0>] do_writepages+0x20/0x40
>>>>>>>> [ 2522.956501]  [<ffffffff811a36f5>]
>>>>>>>> __writeback_single_inode+0x45/0x280
>>>>>>>> [ 2522.956507]  [<ffffffff811269e2>] ?
>>>>>>>> __alloc_pages_nodemask+0x152/0x9b0
>>>>>>>> [ 2522.956513]  [<ffffffff811a495d>]
>>>>>>>> writeback_sb_inodes+0x19d/0x3c0
>>>>>>>> [ 2522.956518]  [<ffffffff811a4c1e>]
>>>>>>>> __writeback_inodes_wb+0x9e/0xd0
>>>>>>>> [ 2522.956523]  [<ffffffff811a4ecb>] wb_writeback+0x27b/0x320
>>>>>>>> [ 2522.956529]  [<ffffffff81196228>] ?
>>>>>>>> get_nr_dirty_inodes+0x58/0x80
>>>>>>>> [ 2522.956534]  [<ffffffff811a500f>]
>>>>>>>> wb_check_old_data_flush+0x9f/0xb0
>>>>>>>> [ 2522.956539]  [<ffffffff811a67e1>] wb_do_writeback+0x151/0x1d0
>>>>>>>> [ 2522.956547]  [<ffffffff8106539f>] ? set_worker_desc+0x6f/0x80
>>>>>>>> [ 2522.956552]  [<ffffffff811a68da>]
>>>>>>>> bdi_writeback_workfn+0x7a/0x200
>>>>>>>> [ 2522.956557]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>>>> [ 2522.956562]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>>>> [ 2522.956567]  [<ffffffff81063570>] ?
>>>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>>>> [ 2522.956571]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>>>> [ 2522.956575]  [<ffffffff81069e80>] ?
>>>>>>>> flush_kthread_worker+0xb0/0xb0
>>>>>>>> [ 2522.956581]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>>>> [ 2522.956585]  [<ffffffff81069e80>] ?
>>>>>>>> flush_kthread_worker+0xb0/0xb0
>>>>>>>> [ 2522.956588] INFO: task iozone:2435 blocked for more than 120
>>>>>>>> seconds.
>>>>>>>> [ 2522.956593] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>> disables this message.
>>>>>>>> [ 2522.956597] iozone          D 0000000000000001     0  2435
>>>>>>>> 1545
>>>>>>>> 0x00000000
>>>>>>>> [ 2522.956602]  ffff882a75671978 0000000000000082 ffff882a75671938
>>>>>>>> ffffffff81301d7a
>>>>>>>> [ 2522.956606]  ffff882f88dc1990 ffff882a75671fd8 ffff882a75671fd8
>>>>>>>> ffff882a75671fd8
>>>>>>>> [ 2522.956611]  ffff882f8ec91990 ffff882f88dc1990 ffff882a75671978
>>>>>>>> ffff882fbf2539f8
>>>>>>>> [ 2522.956615] Call Trace:
>>>>>>>> [ 2522.956619]  [<ffffffff81301d7a>] ?
>>>>>>>> generic_make_request+0xca/0x100
>>>>>>>> [ 2522.956625]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>>>> [ 2522.956630]  [<ffffffff816be36f>] io_schedule+0x8f/0xd0
>>>>>>>> [ 2522.956636]  [<ffffffff811b797c>]
>>>>>>>> do_blockdev_direct_IO+0x1a7c/0x1fb0
>>>>>>>> [ 2522.956645]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60
>>>>>>>> [ext2]
>>>>>>>> [ 2522.956652]  [<ffffffff811b7f05>] __blockdev_direct_IO+0x55/0x60
>>>>>>>> [ 2522.956658]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60
>>>>>>>> [ext2]
>>>>>>>> [ 2522.956662]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>>>>> [ 2522.956669]  [<ffffffffa0263349>] ext2_direct_IO+0x79/0xe0
>>>>>>>> [ext2]
>>>>>>>> [ 2522.956674]  [<ffffffffa0262a80>] ? ext2_get_blocks+0xa60/0xa60
>>>>>>>> [ext2]
>>>>>>>> [ 2522.956680]  [<ffffffff8104ade6>] ? current_fs_time+0x16/0x60
>>>>>>>> [ 2522.956685]  [<ffffffff8111f126>]
>>>>>>>> generic_file_direct_write+0xc6/0x180
>>>>>>>> [ 2522.956690]  [<ffffffff8111f4bd>]
>>>>>>>> __generic_file_aio_write+0x2dd/0x3b0
>>>>>>>> [ 2522.956696]  [<ffffffff8111f5f9>]
>>>>>>>> generic_file_aio_write+0x69/0xd0
>>>>>>>> [ 2522.956702]  [<ffffffff8117b88a>] do_sync_write+0x7a/0xb0
>>>>>>>> [ 2522.956706]  [<ffffffff811bb108>] ? fsnotify+0x1f8/0x2b0
>>>>>>>> [ 2522.956712]  [<ffffffff8117c63e>] vfs_write+0xce/0x1e0
>>>>>>>> [ 2522.956716]  [<ffffffff8117cb22>] SyS_write+0x52/0xa0
>>>>>>>> [ 2522.956722]  [<ffffffff816c7a02>] system_call_fastpath+0x16/0x1b
>>>>>>>> [ 2522.956726] INFO: task kworker/2:1:2470 blocked for more than
>>>>>>>> 120
>>>>>>>> seconds.
>>>>>>>> [ 2522.956730] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>> disables this message.
>>>>>>>> [ 2522.956735] kworker/2:1     D ffffffff81813d40     0
>>>>>>>> 2470      2
>>>>>>>> 0x00000000
>>>>>>>> [ 2522.956743] Workqueue: bcache bch_data_insert_keys
>>>>>>>> [ 2522.956745]  ffff882a5b0c9628 0000000000000046 ffff882a5b0c9648
>>>>>>>> 0000004000000000
>>>>>>>> [ 2522.956749]  ffff882f8ec91990 ffff882a5b0c9fd8 ffff882a5b0c9fd8
>>>>>>>> ffff882a5b0c9fd8
>>>>>>>> [ 2522.956754]  ffff882fa6489990 ffff882f8ec91990 ffff882f8ec91990
>>>>>>>> ffff8828dba00d98
>>>>>>>> [ 2522.956758] Call Trace:
>>>>>>>> [ 2522.956763]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>>>> [ 2522.956769]  [<ffffffff816be57e>]
>>>>>>>> schedule_preempt_disabled+0xe/0x10
>>>>>>>> [ 2522.956774]  [<ffffffff816bc862>]
>>>>>>>> __mutex_lock_slowpath+0x112/0x1b0
>>>>>>>> [ 2522.956779]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>>>>>>> [ 2522.956783]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
>>>>>>>> [ 2522.956791]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
>>>>>>>> [ 2522.956799]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
>>>>>>>> [ 2522.956805]  [<ffffffff81076828>] ? resched_task+0x68/0x70
>>>>>>>> [ 2522.956810]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
>>>>>>>> [ 2522.956816]  [<ffffffff8113a379>] ?
>>>>>>>> fragmentation_index+0x19/0x70
>>>>>>>> [ 2522.956822]  [<ffffffff8113140f>]
>>>>>>>> do_try_to_free_pages+0x20f/0x4b0
>>>>>>>> [ 2522.956827]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
>>>>>>>> [ 2522.956833]  [<ffffffff81126e9c>]
>>>>>>>> __alloc_pages_nodemask+0x60c/0x9b0
>>>>>>>> [ 2522.956843]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
>>>>>>>> [ 2522.956848]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
>>>>>>>> [ 2522.956852]  [<ffffffff8150ebb3>] mca_data_alloc+0x73/0x1d0
>>>>>>>> [ 2522.956857]  [<ffffffff8150ee5a>] mca_bucket_alloc+0x14a/0x1f0
>>>>>>>> [ 2522.956861]  [<ffffffff81511020>] mca_alloc+0x360/0x470
>>>>>>>> [ 2522.956866]  [<ffffffff81517031>] ?
>>>>>>>> __bch_bset_search+0x1d1/0x480
>>>>>>>> [ 2522.956871]  [<ffffffff8151183e>] bch_btree_node_get+0x10e/0x280
>>>>>>>> [ 2522.956875]  [<ffffffff81511c02>]
>>>>>>>> bch_btree_map_nodes_recurse+0xe2/0x170
>>>>>>>> [ 2522.956880]  [<ffffffff815137c0>] ?
>>>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>>>> [ 2522.956886]  [<ffffffff811b51ea>] ? dio_bio_end_io+0x5a/0x90
>>>>>>>> [ 2522.956890]  [<ffffffff8107f991>] ? update_curr+0x141/0x1f0
>>>>>>>> [ 2522.956895]  [<ffffffff81514dce>]
>>>>>>>> __bch_btree_map_nodes+0x13e/0x1c0
>>>>>>>> [ 2522.956899]  [<ffffffff815137c0>] ?
>>>>>>>> bch_btree_insert_node+0x2f0/0x2f0
>>>>>>>> [ 2522.956905]  [<ffffffff8151ac6f>] ? bch_journal+0x42f/0x4b0
>>>>>>>> [ 2522.956909]  [<ffffffff81514f04>] bch_btree_insert+0xb4/0x120
>>>>>>>> [ 2522.956915]  [<ffffffff8151e1be>]
>>>>>>>> bch_data_insert_keys+0x3e/0x160
>>>>>>>> [ 2522.956920]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>>>> [ 2522.956925]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>>>> [ 2522.956930]  [<ffffffff81063570>] ?
>>>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>>>> [ 2522.956933]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>>>> [ 2522.956938]  [<ffffffff81069e80>] ?
>>>>>>>> flush_kthread_worker+0xb0/0xb0
>>>>>>>> [ 2522.956943]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>>>> [ 2522.956947]  [<ffffffff81069e80>] ?
>>>>>>>> flush_kthread_worker+0xb0/0xb0
>>>>>>>> [ 2522.956951] INFO: task kworker/3:2:2471 blocked for more than
>>>>>>>> 120
>>>>>>>> seconds.
>>>>>>>> [ 2522.956955] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>> disables this message.
>>>>>>>> [ 2522.956959] kworker/3:2     D ffffffff81813a60     0
>>>>>>>> 2471      2
>>>>>>>> 0x00000000
>>>>>>>> [ 2522.956966] Workqueue: events update_writeback_rate
>>>>>>>> [ 2522.956968]  ffff882a5d125cd0 0000000000000046 0000000000000003
>>>>>>>> ffff882a5d125d68
>>>>>>>> [ 2522.956972]  ffff882f8ec94cb0 ffff882a5d125fd8 ffff882a5d125fd8
>>>>>>>> ffff882a5d125fd8
>>>>>>>> [ 2522.956977]  ffff882fa6aeb320 ffff882f8ec94cb0 0000000200000003
>>>>>>>> ffff882f8ec94cb0
>>>>>>>> [ 2522.956981] Call Trace:
>>>>>>>> [ 2522.956987]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>>>> [ 2522.956992]  [<ffffffff816bf0fd>]
>>>>>>>> rwsem_down_read_failed+0x9d/0xe5
>>>>>>>> [ 2522.956997]  [<ffffffff81332c64>]
>>>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>>>> [ 2522.957002]  [<ffffffff816bcc74>] ? down_read+0x24/0x2b
>>>>>>>> [ 2522.957007]  [<ffffffff8151b010>]
>>>>>>>> update_writeback_rate+0x30/0x230
>>>>>>>> [ 2522.957011]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>>>> [ 2522.957017]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>>>> [ 2522.957021]  [<ffffffff81063570>] ?
>>>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>>>> [ 2522.957025]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>>>> [ 2522.957030]  [<ffffffff81069e80>] ?
>>>>>>>> flush_kthread_worker+0xb0/0xb0
>>>>>>>> [ 2522.957035]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>>>> [ 2522.957039]  [<ffffffff81069e80>] ?
>>>>>>>> flush_kthread_worker+0xb0/0xb0
>>>>>>>> [ 2522.957048] INFO: task kworker/u64:2:2540 blocked for more
>>>>>>>> than 120
>>>>>>>> seconds.
>>>>>>>> [ 2522.957052] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>> disables this message.
>>>>>>>> [ 2522.957056] kworker/u64:2   D ffffffff81813a60     0
>>>>>>>> 2540      2
>>>>>>>> 0x00000000
>>>>>>>> [ 2522.957062] Workqueue: bch_btree_io btree_node_write_work
>>>>>>>> [ 2522.957064]  ffff882a4fdc9778 0000000000000046 ffff882a4fdc9738
>>>>>>>> ffff882f86e3be58
>>>>>>>> [ 2522.957069]  ffff882fa4bc1990 ffff882a4fdc9fd8 ffff882a4fdc9fd8
>>>>>>>> ffff882a4fdc9fd8
>>>>>>>> [ 2522.957073]  ffff882fa6aeb320 ffff882fa4bc1990 ffff882f86e3be00
>>>>>>>> ffff8828dba00d98
>>>>>>>> [ 2522.957078] Call Trace:
>>>>>>>> [ 2522.957083]  [<ffffffff816be299>] schedule+0x29/0x70
>>>>>>>> [ 2522.957088]  [<ffffffff816be57e>]
>>>>>>>> schedule_preempt_disabled+0xe/0x10
>>>>>>>> [ 2522.957092]  [<ffffffff816bc862>]
>>>>>>>> __mutex_lock_slowpath+0x112/0x1b0
>>>>>>>> [ 2522.957102]  [<ffffffff81484090>] ?
>>>>>>>> ata_scsiop_mode_sense+0x380/0x380
>>>>>>>> [ 2522.957107]  [<ffffffff816bc3da>] mutex_lock+0x2a/0x50
>>>>>>>> [ 2522.957111]  [<ffffffff815112e5>] bch_mca_shrink+0x1b5/0x2f0
>>>>>>>> [ 2522.957116]  [<ffffffff8117fc32>] ? prune_super+0x162/0x1b0
>>>>>>>> [ 2522.957121]  [<ffffffff8112ebb4>] shrink_slab+0x154/0x300
>>>>>>>> [ 2522.957126]  [<ffffffff81076828>] ? resched_task+0x68/0x70
>>>>>>>> [ 2522.957130]  [<ffffffff81077165>] ? check_preempt_curr+0x75/0xa0
>>>>>>>> [ 2522.957135]  [<ffffffff8113a379>] ?
>>>>>>>> fragmentation_index+0x19/0x70
>>>>>>>> [ 2522.957140]  [<ffffffff8113140f>]
>>>>>>>> do_try_to_free_pages+0x20f/0x4b0
>>>>>>>> [ 2522.957146]  [<ffffffff81131864>] try_to_free_pages+0xe4/0x1a0
>>>>>>>> [ 2522.957152]  [<ffffffff81126e9c>]
>>>>>>>> __alloc_pages_nodemask+0x60c/0x9b0
>>>>>>>> [ 2522.957158]  [<ffffffff8116062a>] alloc_pages_current+0xba/0x170
>>>>>>>> [ 2522.957163]  [<ffffffff8112240e>] __get_free_pages+0xe/0x40
>>>>>>>> [ 2522.957168]  [<ffffffff81517fc8>] __btree_sort+0x48/0x230
>>>>>>>> [ 2522.957173]  [<ffffffff8151765c>] ?
>>>>>>>> __bch_btree_iter_init+0x7c/0xc0
>>>>>>>> [ 2522.957178]  [<ffffffff81518301>]
>>>>>>>> bch_btree_sort_partial+0x101/0x120
>>>>>>>> [ 2522.957182]  [<ffffffff8150f000>] ?
>>>>>>>> __btree_node_write_done+0x100/0x100
>>>>>>>> [ 2522.957187]  [<ffffffff81518468>] bch_btree_sort_lazy+0x68/0x90
>>>>>>>> [ 2522.957191]  [<ffffffff815109ba>]
>>>>>>>> bch_btree_node_write+0x36a/0x4a0
>>>>>>>> [ 2522.957196]  [<ffffffff8108458b>] ? idle_balance+0xeb/0x150
>>>>>>>> [ 2522.957201]  [<ffffffff8106212c>] ?
>>>>>>>> pwq_activate_delayed_work+0x4c/0xb0
>>>>>>>> [ 2522.957205]  [<ffffffff81511477>]
>>>>>>>> btree_node_write_work+0x57/0x80
>>>>>>>> [ 2522.957210]  [<ffffffff810624d4>] process_one_work+0x174/0x490
>>>>>>>> [ 2522.957215]  [<ffffffff8106368b>] worker_thread+0x11b/0x370
>>>>>>>> [ 2522.957219]  [<ffffffff81063570>] ?
>>>>>>>> manage_workers.isra.23+0x2d0/0x2d0
>>>>>>>> [ 2522.957223]  [<ffffffff81069f40>] kthread+0xc0/0xd0
>>>>>>>> [ 2522.957228]  [<ffffffff81069e80>] ?
>>>>>>>> flush_kthread_worker+0xb0/0xb0
>>>>>>>> [ 2522.957233]  [<ffffffff816c795c>] ret_from_fork+0x7c/0xb0
>>>>>>>> [ 2522.957237]  [<ffffffff81069e80>] ?
>>>>>>>> flush_kthread_worker+0xb0/0xb0
>>>>>>>>
>>>>>>>>
>>>>>>>> I can easily reproduce this, please let me know if you require any
>>>>>>>> info/test any patch.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Neo
>>>>>>>>
>>>>>>>> On Mon, Aug 26, 2013 at 12:21 PM, Stefan Priebe
>>>>>>>> <s.priebe@profihost.ag> wrote:
>>>>>>>>>
>>>>>>>>> Hi Kent,
>>>>>>>>>
>>>>>>>>> a new one under 3.10:
>>>>>>>>>
>>>>>>>>> 2013-08-26 21:05:30     INFO: task ceph-osd:8939 blocked for more
>>>>>>>>> than 120
>>>>>>>>> seconds.
>>>>>>>>> 2013-08-26 21:05:30     "echo 0 >
>>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>>> disables this message.
>>>>>>>>> 2013-08-26 21:05:30     ceph-osd D ffffffff8160f760 0 8939 1
>>>>>>>>> 0x00000000
>>>>>>>>> 2013-08-26 21:05:30     ffff880bd9a61820 0000000000000086
>>>>>>>>> ffff880bd9a61fd8
>>>>>>>>> 0000000000012c80
>>>>>>>>> 2013-08-26 21:05:30     ffff880bd9a60010 0000000000012c80
>>>>>>>>> 0000000000012c80
>>>>>>>>> 0000000000012c80
>>>>>>>>> 2013-08-26 21:05:30     ffff880bd9a61fd8 0000000000012c80
>>>>>>>>> ffff880bd7abb1c0
>>>>>>>>> ffff880c48dee380
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff811e8128>]
>>>>>>>>> xfs_vm_readpages+0x18/0x20
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810f1da3>] read_pages+0x43/0x100
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810e64df>] ?
>>>>>>>>> __page_cache_alloc+0x9f/0xc0
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810f1fab>]
>>>>>>>>> __do_page_cache_readahead+0x14b/0x160
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810f1fdc>] ra_submit+0x1c/0x20
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810f22a5>]
>>>>>>>>> ondemand_readahead+0x115/0x240
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810f249e>]
>>>>>>>>> page_cache_sync_readahead+0x2e/0x40
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810e7a5e>] T.1003+0x33e/0x430
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810e7c30>]
>>>>>>>>> generic_file_aio_read+0xe0/0x220
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff811efd0a>]
>>>>>>>>> xfs_file_aio_read+0x15a/0x2a0
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff8114095a>]
>>>>>>>>> do_sync_read+0x7a/0xb0
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff81140df1>] vfs_read+0xb1/0x130
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff81141757>] SyS_pread64+0x97/0xa0
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff81526b12>]
>>>>>>>>> system_call_fastpath+0x16/0x1b
>>>>>>>>> 2013-08-26 21:05:29     INFO: task ceph-osd:8896 blocked for more
>>>>>>>>> than 120
>>>>>>>>> seconds.
>>>>>>>>> 2013-08-26 21:05:29     "echo 0 >
>>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>>> disables this message.
>>>>>>>>> 2013-08-26 21:05:29     ceph-osd D ffffffff8160f760 0 8896 1
>>>>>>>>> 0x00000000
>>>>>>>>> 2013-08-26 21:05:29     ffff880bd9a9dc28 0000000000000086
>>>>>>>>> ffff880bd9a9dfd8
>>>>>>>>> 0000000000012c80
>>>>>>>>> 2013-08-26 21:05:29     ffff880bd9a9c010 0000000000012c80
>>>>>>>>> 0000000000012c80
>>>>>>>>> 0000000000012c80
>>>>>>>>> 2013-08-26 21:05:29     ffff880bd9a9dfd8 0000000000012c80
>>>>>>>>> ffff880bd79298e0
>>>>>>>>> ffff880c48dd6380
>>>>>>>>> 2013-08-26 21:05:29     Call Trace:
>>>>>>>>> 2013-08-26 21:05:29     [<ffffffff810e60f0>] ?
>>>>>>>>> __lock_page+0x70/0x70
>>>>>>>>> 2013-08-26 21:05:28     "echo 0 >
>>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>>> disables this message.
>>>>>>>>> 2013-08-26 21:05:28     ceph-osd D ffffffff8160f760 0 8798 1
>>>>>>>>> 0x00000000
>>>>>>>>> 2013-08-26 21:05:28     ffff880bd998b740 0000000000000086
>>>>>>>>> ffff880bd998bfd8
>>>>>>>>> 0000000000012c80
>>>>>>>>> 2013-08-26 21:05:28     ffff880bd998a010 0000000000012c80
>>>>>>>>> 0000000000012c80
>>>>>>>>> 0000000000012c80
>>>>>>>>> 2013-08-26 21:05:28     ffff880bd998bfd8 0000000000012c80
>>>>>>>>> ffff880bd99c0000
>>>>>>>>> ffff880c48dd6380
>>>>>>>>> 2013-08-26 21:05:28     Call Trace:
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff81524874>] schedule+0x24/0x70
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff81525d7d>]
>>>>>>>>> rwsem_down_read_failed+0x9d/0xe5
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff812aa0a4>]
>>>>>>>>> call_rwsem_down_read_failed+0x14/0x30
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff81523a42>] ? down_read+0x12/0x20
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffffa01ead62>]
>>>>>>>>> btree_read_async+0xa2/0x1a0
>>>>>>>>> [bcache]
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffffa01f7b73>]
>>>>>>>>> closure_queue+0x43/0x60
>>>>>>>>> [bcache]
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffffa01eb903>] T.1050+0x63/0x70
>>>>>>>>> [bcache]
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffffa01ed6ea>]
>>>>>>>>> cached_dev_make_request+0x20a/0x350 [bcache]
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff81281952>]
>>>>>>>>> generic_make_request+0xc2/0x100
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff812819f7>] submit_bio+0x67/0x130
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff8117b240>]
>>>>>>>>> do_mpage_readpage+0x2a0/0x660
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff81101cee>] ?
>>>>>>>>> __inc_zone_page_state+0x2e/0x30
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff810e6581>] ?
>>>>>>>>> add_to_page_cache_locked+0x81/0x110
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff8117b76a>]
>>>>>>>>> mpage_readpages+0xfa/0x150
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff811e90c0>] ?
>>>>>>>>> xfs_get_blocks_direct+0x20/0x20
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff811e90c0>] ?
>>>>>>>>> xfs_get_blocks_direct+0x20/0x20
>>>>>>>>> 2013-08-26 21:05:28     [<ffffffff810eeda3>] ?
>>>>>>>>> __alloc_pages_nodemask+0x123/0x240
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff8115bf93>] ?
>>>>>>>>> __close_fd+0x43/0x80
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff8116dabe>]
>>>>>>>>> SyS_fdatasync+0xe/0x20
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81526b12>]
>>>>>>>>> system_call_fastpath+0x16/0x1b
>>>>>>>>> 2013-08-26 21:05:27     INFO: task ceph-osd:8746 blocked for more
>>>>>>>>> than 120
>>>>>>>>> seconds.
>>>>>>>>> 2013-08-26 21:05:27     "echo 0 >
>>>>>>>>> /proc/sys/kernel/hung_task_timeout_secs"
>>>>>>>>> disables this message.
>>>>>>>>> 2013-08-26 21:05:27     ceph-osd D ffffffff8160f760 0 8746 1
>>>>>>>>> 0x00000000
>>>>>>>>> 2013-08-26 21:05:27     ffff880c2dde1d38 0000000000000086
>>>>>>>>> ffff880c2dde1fd8
>>>>>>>>> 0000000000012c80
>>>>>>>>> 2013-08-26 21:05:27     ffff880c2dde0010 0000000000012c80
>>>>>>>>> 0000000000012c80
>>>>>>>>> 0000000000012c80
>>>>>>>>> 2013-08-26 21:05:27     ffff880c2dde1fd8 0000000000012c80
>>>>>>>>> ffff880c239eb1c0
>>>>>>>>> ffff880c48dd6380
>>>>>>>>> 2013-08-26 21:05:27     Call Trace:
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81524874>] schedule+0x24/0x70
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81522bbd>]
>>>>>>>>> schedule_timeout+0x16d/0x200
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81068eba>] ?
>>>>>>>>> __queue_delayed_work+0xaa/0x1a0
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81068d19>] ?
>>>>>>>>> try_to_grab_pending+0x109/0x190
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff815251e5>]
>>>>>>>>> wait_for_completion+0x95/0x110
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff8107fd00>] ?
>>>>>>>>> try_to_wake_up+0x2a0/0x2a0
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81167797>] ?
>>>>>>>>> bdi_queue_work+0x77/0xc0
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81167863>]
>>>>>>>>> writeback_inodes_sb_nr+0x83/0xb0
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff811678ea>]
>>>>>>>>> writeback_inodes_sb+0x5a/0x70
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff8116dd0a>]
>>>>>>>>> __sync_filesystem+0x4a/0x50
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff8116dd42>]
>>>>>>>>> sync_filesystem+0x32/0x60
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff8116ddc0>] SyS_syncfs+0x50/0x90
>>>>>>>>> 2013-08-26 21:05:27     [<ffffffff81526b12>]
>>>>>>>>> system_call_fastpath+0x16/0x1b
>>>>>>>>> 2013-08-26 21:05:27     INFO: task ceph-osd:8798 blocked for more
>>>>>>>>> than 120
>>>>>>>>> seconds.
>>>>>>>>>
>>>>>>>>> Stefan
>>>>>>>>>
>>>>>>>>> Am 22.08.2013 09:32, schrieb Stefan Priebe - Profihost AG:
>>>>>>>>>
>>>>>>>>>> great!
>>>>>>>>>>
>>>>>>>>>> Everything seems to work fine now! Except read_dirty always
>>>>>>>>>> going to
>>>>>>>>>> negative values after a reboot.
>>>>>>>>>>
>>>>>>>>>> Stefan
>>>>>>>>>>
>>>>>>>>>> Am 22.08.2013 08:02, schrieb Kent Overstreet:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Aug 22, 2013 at 07:59:04AM +0200, Stefan Priebe wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> schedule_timeout() is not the same as
>>>>>>>>>>>>> schedule_timeout_interruptible().
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> just search and replace? So i can try on my own.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> The one in read_dirty(), line ~330
>>>>>>>>>>> --
>>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>>>> linux-bcache"
>>>>>>>>>>> in
>>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>>> More majordomo info at
>>>>>>>>>>> http://vger.kernel.org/majordomo-info.html
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>> linux-bcache" in
>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>> linux-bcache"
>>>>>> in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 28+ messages in thread

end of thread, other threads:[~2013-08-29 18:52 UTC | newest]

Thread overview: 28+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-08-14 22:59 [GIT PULL] bcache fixes for 3.11 Kent Overstreet
2013-08-15  6:43 ` Stefan Priebe - Profihost AG
2013-08-15  7:07   ` Kent Overstreet
2013-08-16 10:11   ` Stefan Priebe - Profihost AG
2013-08-18 22:09     ` bcache: Fix a writeback performance regression Stefan Priebe
2013-08-19 22:27       ` Kent Overstreet
2013-08-20  8:01         ` Stefan Priebe - Profihost AG
2013-08-20  8:07           ` Stefan Priebe - Profihost AG
2013-08-21 21:10             ` Kent Overstreet
2013-08-21 23:47             ` Kent Overstreet
2013-08-22  5:25               ` Stefan Priebe
2013-08-22  5:27               ` Stefan Priebe
2013-08-22  5:43                 ` Kent Overstreet
2013-08-22  5:59                   ` Stefan Priebe
2013-08-22  6:02                     ` Kent Overstreet
2013-08-22  7:32                       ` Stefan Priebe - Profihost AG
2013-08-22  7:42                         ` Kent Overstreet
2013-08-26 19:21                         ` Stefan Priebe
2013-08-28  3:37                           ` kernel neophyte
2013-08-28  6:05                             ` Kent Overstreet
2013-08-28 18:12                               ` kernel neophyte
2013-08-28 18:38                                 ` Stefan Priebe - Profihost AG
2013-08-28 18:47                                   ` kernel neophyte
2013-08-28 19:20                                     ` Stefan Priebe
2013-08-28 20:12                                       ` kernel neophyte
2013-08-28 20:15                                         ` Stefan Priebe
2013-08-28 22:45                                           ` kernel neophyte
2013-08-29 18:52                                           ` Stefan Priebe

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).