Issue (as expected) when upgrading from 3.12 to 3.13.7

Issue (as expected) when upgrading from 3.12 to 3.13.7

[Francis Moreau]

Hello,

As I was expecting I'm getting new issues with bcache after upgrading my
kernel:

[Mar30 12:24] INFO: task bcache_writebac:155 blocked for more than 120
seconds.
[  +0.000007]       Not tainted 3.13.7-1-ARCH #1
[  +0.000002] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[  +0.000003] bcache_writebac D 0000000000000000     0   155      2
0x00000000
[  +0.000006]  ffff8804049c5eb8 0000000000000046 ffff880405816c00
ffff8804049c5fd8
[  +0.000005]  0000000000014440 0000000000014440 ffff880405816c00
ffff8804049c5e38
[  +0.000004]  ffffffff810925fa 24448b48e0d83524 fa83ffffee31e918
0000000000000046
[  +0.000004] Call Trace:
[  +0.000010]  [<ffffffff810925fa>] ? try_to_wake_up+0x1fa/0x2c0
[  +0.000006]  [<ffffffff81092712>] ? default_wake_function+0x12/0x20
[  +0.000005]  [<ffffffff810a2538>] ? __wake_up_common+0x58/0x90
[  +0.000023]  [<ffffffffa03c7470>] ? read_dirty_endio+0x60/0x60 [bcache]
[  +0.000005]  [<ffffffff81516239>] schedule+0x29/0x70
[  +0.000006]  [<ffffffff8108358d>] kthread+0xad/0xf0
[  +0.000005]  [<ffffffff810834e0>] ? kthread_create_on_node+0x180/0x180
[  +0.000005]  [<ffffffff81521abc>] ret_from_fork+0x7c/0xb0
[  +0.000005]  [<ffffffff810834e0>] ? kthread_create_on_node+0x180/0x180

That's just sad to see how unstable was/is bcache sub system and I still
don't understand why it was not marked as experimental.

3.12 was/is buggy when used with writeback mode, and I just had to live
with cache-through. I tried to get some helps (and others too) but no
answers.

I read there were issue with 3.13, so I delayed my kernel upgrade as far
I could due to this. I was hoping that issues were fixed in 3.13.x
stable tree but apparently not.

Hopefully this one will not be ignored.

Thanks

Re: Issue (as expected) when upgrading from 3.12 to 3.13.7

[Sitsofe Wheeler]

Hi,

On Sun, Mar 30, 2014 at 12:53:12PM +0200, Francis Moreau wrote:
> 
> As I was expecting I'm getting new issues with bcache after upgrading my
> kernel:
> 
> [Mar30 12:24] INFO: task bcache_writebac:155 blocked for more than 120
> seconds.
> [  +0.000007]       Not tainted 3.13.7-1-ARCH #1
> [  +0.000002] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [  +0.000003] bcache_writebac D 0000000000000000     0   155      2
> 0x00000000
> [  +0.000006]  ffff8804049c5eb8 0000000000000046 ffff880405816c00
> ffff8804049c5fd8
> [  +0.000005]  0000000000014440 0000000000014440 ffff880405816c00
> ffff8804049c5e38
> [  +0.000004]  ffffffff810925fa 24448b48e0d83524 fa83ffffee31e918
> 0000000000000046
> [  +0.000004] Call Trace:
> [  +0.000010]  [<ffffffff810925fa>] ? try_to_wake_up+0x1fa/0x2c0
> [  +0.000006]  [<ffffffff81092712>] ? default_wake_function+0x12/0x20
> [  +0.000005]  [<ffffffff810a2538>] ? __wake_up_common+0x58/0x90
> [  +0.000023]  [<ffffffffa03c7470>] ? read_dirty_endio+0x60/0x60 [bcache]
> [  +0.000005]  [<ffffffff81516239>] schedule+0x29/0x70
> [  +0.000006]  [<ffffffff8108358d>] kthread+0xad/0xf0
> [  +0.000005]  [<ffffffff810834e0>] ? kthread_create_on_node+0x180/0x180
> [  +0.000005]  [<ffffffff81521abc>] ret_from_fork+0x7c/0xb0
> [  +0.000005]  [<ffffffff810834e0>] ? kthread_create_on_node+0x180/0x180
> 
> That's just sad to see how unstable was/is bcache sub system and I still
> don't understand why it was not marked as experimental.
> 
> 3.12 was/is buggy when used with writeback mode, and I just had to live
> with cache-through. I tried to get some helps (and others too) but no
> answers.
> 
> I read there were issue with 3.13, so I delayed my kernel upgrade as far
> I could due to this. I was hoping that issues were fixed in 3.13.x
> stable tree but apparently not.
> 
> Hopefully this one will not be ignored.

Francis - I've found it helps if you add all the relevant addresses
mentioned in https://www.kernel.org/doc/linux/MAINTAINERS . For this
mail I'm going to explicitly CC Kent to help this get noticed.
Additionally do you also see this problem with 3.14?

-- 
Sitsofe | http://sucs.org/~sits/

Re: Issue (as expected) when upgrading from 3.12 to 3.13.7

[Francis Moreau]

On 03/30/2014 01:47 PM, Sitsofe Wheeler wrote:
> Hi,
> 
> On Sun, Mar 30, 2014 at 12:53:12PM +0200, Francis Moreau wrote:
>>
>> As I was expecting I'm getting new issues with bcache after upgrading my
>> kernel:
>>
>> [Mar30 12:24] INFO: task bcache_writebac:155 blocked for more than 120
>> seconds.
>> [  +0.000007]       Not tainted 3.13.7-1-ARCH #1
>> [  +0.000002] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [  +0.000003] bcache_writebac D 0000000000000000     0   155      2
>> 0x00000000
>> [  +0.000006]  ffff8804049c5eb8 0000000000000046 ffff880405816c00
>> ffff8804049c5fd8
>> [  +0.000005]  0000000000014440 0000000000014440 ffff880405816c00
>> ffff8804049c5e38
>> [  +0.000004]  ffffffff810925fa 24448b48e0d83524 fa83ffffee31e918
>> 0000000000000046
>> [  +0.000004] Call Trace:
>> [  +0.000010]  [<ffffffff810925fa>] ? try_to_wake_up+0x1fa/0x2c0
>> [  +0.000006]  [<ffffffff81092712>] ? default_wake_function+0x12/0x20
>> [  +0.000005]  [<ffffffff810a2538>] ? __wake_up_common+0x58/0x90
>> [  +0.000023]  [<ffffffffa03c7470>] ? read_dirty_endio+0x60/0x60 [bcache]
>> [  +0.000005]  [<ffffffff81516239>] schedule+0x29/0x70
>> [  +0.000006]  [<ffffffff8108358d>] kthread+0xad/0xf0
>> [  +0.000005]  [<ffffffff810834e0>] ? kthread_create_on_node+0x180/0x180
>> [  +0.000005]  [<ffffffff81521abc>] ret_from_fork+0x7c/0xb0
>> [  +0.000005]  [<ffffffff810834e0>] ? kthread_create_on_node+0x180/0x180
>>
>> That's just sad to see how unstable was/is bcache sub system and I still
>> don't understand why it was not marked as experimental.
>>
>> 3.12 was/is buggy when used with writeback mode, and I just had to live
>> with cache-through. I tried to get some helps (and others too) but no
>> answers.
>>
>> I read there were issue with 3.13, so I delayed my kernel upgrade as far
>> I could due to this. I was hoping that issues were fixed in 3.13.x
>> stable tree but apparently not.
>>
>> Hopefully this one will not be ignored.
> 
> Francis - I've found it helps if you add all the relevant addresses
> mentioned in https://www.kernel.org/doc/linux/MAINTAINERS . For this
> mail I'm going to explicitly CC Kent to help this get noticed.
> Additionally do you also see this problem with 3.14?
> 

Sorry 3.14 is not available yet on my distro.

Thanks

Re: Issue (as expected) when upgrading from 3.12 to 3.13.7

[Sitsofe Wheeler]

On Sun, Mar 30, 2014 at 02:08:57PM +0200, Francis Moreau wrote:
> On 03/30/2014 01:47 PM, Sitsofe Wheeler wrote:
> > 
> > Francis - I've found it helps if you add all the relevant addresses
> > mentioned in https://www.kernel.org/doc/linux/MAINTAINERS . For this
> > mail I'm going to explicitly CC Kent to help this get noticed.
> > Additionally do you also see this problem with 3.14?
> 
> Sorry 3.14 is not available yet on my distro.

But 3.14.rc7 is via linux-git right?

-- 
Sitsofe | http://sucs.org/~sits/

Re: Issue (as expected) when upgrading from 3.12 to 3.13.7

[Francis Moreau]

On 03/30/2014 03:04 PM, Sitsofe Wheeler wrote:
> On Sun, Mar 30, 2014 at 02:08:57PM +0200, Francis Moreau wrote:
>> On 03/30/2014 01:47 PM, Sitsofe Wheeler wrote:
>>>
>>> Francis - I've found it helps if you add all the relevant addresses
>>> mentioned in https://www.kernel.org/doc/linux/MAINTAINERS . For this
>>> mail I'm going to explicitly CC Kent to help this get noticed.
>>> Additionally do you also see this problem with 3.14?
>>
>> Sorry 3.14 is not available yet on my distro.
> 
> But 3.14.rc7 is via linux-git right?
> 

I was initialy reporting for 3.13.7.

Re: Issue (as expected) when upgrading from 3.12 to 3.13.7

[Francis Moreau]

still having the same trace with 3.13.8.

On 03/30/2014 01:47 PM, Sitsofe Wheeler wrote:
> Hi,
> 
> On Sun, Mar 30, 2014 at 12:53:12PM +0200, Francis Moreau wrote:
>>
>> As I was expecting I'm getting new issues with bcache after upgrading my
>> kernel:
>>
>> [Mar30 12:24] INFO: task bcache_writebac:155 blocked for more than 120
>> seconds.
>> [  +0.000007]       Not tainted 3.13.7-1-ARCH #1
>> [  +0.000002] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [  +0.000003] bcache_writebac D 0000000000000000     0   155      2
>> 0x00000000
>> [  +0.000006]  ffff8804049c5eb8 0000000000000046 ffff880405816c00
>> ffff8804049c5fd8
>> [  +0.000005]  0000000000014440 0000000000014440 ffff880405816c00
>> ffff8804049c5e38
>> [  +0.000004]  ffffffff810925fa 24448b48e0d83524 fa83ffffee31e918
>> 0000000000000046
>> [  +0.000004] Call Trace:
>> [  +0.000010]  [<ffffffff810925fa>] ? try_to_wake_up+0x1fa/0x2c0
>> [  +0.000006]  [<ffffffff81092712>] ? default_wake_function+0x12/0x20
>> [  +0.000005]  [<ffffffff810a2538>] ? __wake_up_common+0x58/0x90
>> [  +0.000023]  [<ffffffffa03c7470>] ? read_dirty_endio+0x60/0x60 [bcache]
>> [  +0.000005]  [<ffffffff81516239>] schedule+0x29/0x70
>> [  +0.000006]  [<ffffffff8108358d>] kthread+0xad/0xf0
>> [  +0.000005]  [<ffffffff810834e0>] ? kthread_create_on_node+0x180/0x180
>> [  +0.000005]  [<ffffffff81521abc>] ret_from_fork+0x7c/0xb0
>> [  +0.000005]  [<ffffffff810834e0>] ? kthread_create_on_node+0x180/0x180
>>
>> That's just sad to see how unstable was/is bcache sub system and I still
>> don't understand why it was not marked as experimental.
>>
>> 3.12 was/is buggy when used with writeback mode, and I just had to live
>> with cache-through. I tried to get some helps (and others too) but no
>> answers.
>>
>> I read there were issue with 3.13, so I delayed my kernel upgrade as far
>> I could due to this. I was hoping that issues were fixed in 3.13.x
>> stable tree but apparently not.
>>
>> Hopefully this one will not be ignored.
> 
> Francis - I've found it helps if you add all the relevant addresses
> mentioned in https://www.kernel.org/doc/linux/MAINTAINERS . For this
> mail I'm going to explicitly CC Kent to help this get noticed.
> Additionally do you also see this problem with 3.14?
> 

Re: Issue (as expected) when upgrading from 3.12 to 3.13.7

[Peter Kieser]

[-- Attachment #1: Type: text/plain, Size: 439 bytes --]


On 2014-03-30 3:53 AM, Francis Moreau wrote:
> That's just sad to see how unstable was/is bcache sub system and I still
> don't understand why it was not marked as experimental.

IMO, It should still be marked as experimental. Possibly, even removed 
from mainline as the maintainer is not particularly responsive to major 
bugs. Certainly not production ready and it's been through many releases 
of the kernel now.

-Peter


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4504 bytes --]

Re: Issue (as expected) when upgrading from 3.12 to 3.13.7

[Francis Moreau]

On 03/31/2014 05:20 AM, Peter Kieser wrote:
> 
> On 2014-03-30 3:53 AM, Francis Moreau wrote:
>> That's just sad to see how unstable was/is bcache sub system and I still
>> don't understand why it was not marked as experimental.
> 
> IMO, It should still be marked as experimental. Possibly, even removed 
> from mainline as the maintainer is not particularly responsive to major 
> bugs. Certainly not production ready and it's been through many releases 
> of the kernel now.

Well, removing from mainline is quite extreme but moving it to the
staging stuff would be a possible alternative.

And it should definitely be marked experimental in stable trees such as
3.12 (LTS kernel) IMHO. Specially since those trees don't get any fixes.

Re: Issue (as expected) when upgrading from 3.12 to 3.13.7

[Vivek Dasmohapatra]

On Sun, 30 Mar 2014, Francis Moreau wrote:

Hi Francis: A quick question for you if you have the time:
How is the bcache constructed? ie which devices are in it?

I have had similar lockup problems when using partitions
as backing, but not whole devices. That is to say:

    { ram0 : sdb1 }  often deadlocks for me (depending on the offset
                     of the partition)

    { ram0 : sdb  }  always seems to work fine

Although in neither case am I able to tear down the device and
flush it all to backing after heavy use: That always locks up
for me. (Fortunately I'm making throwaway filesystems for a build
daemon so that part I can live with).

Re: Issue (as expected) when upgrading from 3.12 to 3.13.7

[Francis Moreau]

On 04/03/2014 12:02 PM, Vivek Dasmohapatra wrote:
> On Sun, 30 Mar 2014, Francis Moreau wrote:
> 
> Hi Francis: A quick question for you if you have the time:
> How is the bcache constructed? ie which devices are in it?
> 

sure: I'm using bcache on a very basic setup: no MD or LVM involved.
/dev/sda4 (900Mo) is the backing device while /dev/sdb (120G) is the
cache device. On top of bcache0 I'm using ext4 and I'm using it as my
root device.

> I have had similar lockup problems when using partitions
> as backing, but not whole devices. That is to say:
> 
>     { ram0 : sdb1 }  often deadlocks for me (depending on the offset
>                      of the partition)
> 
>     { ram0 : sdb  }  always seems to work fine
> 
> Although in neither case am I able to tear down the device and
> flush it all to backing after heavy use: That always locks up
> for me. (Fortunately I'm making throwaway filesystems for a build
> daemon so that part I can live with).
> 

For now it doesn't endup to lock up (cross fingers). It only fills the
kernel logs, well I hope.

Thanks.

Re: Issue (as expected) when upgrading from 3.12 to 3.13.7

[Vivek Dasmohapatra]

On Thu, 3 Apr 2014, Francis Moreau wrote:

> On 04/03/2014 12:02 PM, Vivek Dasmohapatra wrote:
>> On Sun, 30 Mar 2014, Francis Moreau wrote:
>>
>> Hi Francis: A quick question for you if you have the time:
>> How is the bcache constructed? ie which devices are in it?
>>
>
> sure: I'm using bcache on a very basic setup: no MD or LVM involved.
> /dev/sda4 (900Mo) is the backing device while /dev/sdb (120G) is the
> cache device. On top of bcache0 I'm using ext4 and I'm using it as my
> root device.

Ok, interesting. I've had some luck tweaking the offset of the backing
device partition: Some offsets trigger the bug consistently and others
never seem to, so it seems to be some sort of weird alignment bug.

Re: Issue (as expected) when upgrading from 3.12 to 3.13.7

[Sam Fulcomer]

Vivek Dasmohapatra <vivek <at> collabora.co.uk> writes:

> 
> On Thu, 3 Apr 2014, Francis Moreau wrote:
> 
> > On 04/03/2014 12:02 PM, Vivek Dasmohapatra wrote:
> >> On Sun, 30 Mar 2014, Francis Moreau wrote:
> >>
> >> Hi Francis: A quick question for you if you have the time:
> >> How is the bcache constructed? ie which devices are in it?
> >>
> >
> > sure: I'm using bcache on a very basic setup: no MD or LVM involved.
> > /dev/sda4 (900Mo) is the backing device while /dev/sdb (120G) is the
> > cache device. On top of bcache0 I'm using ext4 and I'm using it as my
> > root device.
> 
> Ok, interesting. I've had some luck tweaking the offset of the backing
> device partition: Some offsets trigger the bug consistently and others
> never seem to, so it seems to be some sort of weird alignment bug.
> 
> 


I'd started a new thread before I realized how to reply to this.

The problem appears to be that bcache_writeback is trying to do
 something it shouldn't when in writethrough  mode 
(bcache_writeback is on by default even for writethrough).

echo 0 > /sys/block/bcache0/bcache/writeback_running

...appears to fix it.