Kernel Oops (bCache hangs on registering 3rd device)

Kernel Oops (bCache hangs on registering 3rd device)

[James Sefton]

Hi,

Seems I broke something.

I have run into this problem about 3 times now where bCache (or something!) 
seems to hang and I only just thought to check kern.log before trying to reboot 
so was previously assuming something else had just got stuck!

I only had 3 servers available to me and once this problem occurs, the servers 
seems to get stuck rebooting so I cannot do another test to confirm tonight.  (I 
have put a reboot request in for them to be force rebooted but that will not get 
actioned until tomorrow)

Here is what I know about the most recent time this happened:

I had already set-up and been using /dev/bcache0 and /dev/bcache1.  They are 
attached to the cache set and set to writeback.

The problem occurred when I ran "echo /dev/rbd2 >/sys/fs/bcache/register".   
This device had been newly formatted with "make-bcache -B /dev/rbd2"   (rbd2 is 
a SAN based block device)

If it makes any difference - these commands are in a script so the register 
command would have been run *immediately* after the make-bcache command 
returned.  (I am going to try sticking a sleep 3 or something in there tomorrow 
- just in case its related to trying to register the device so quickly after it 
was prepared)

At this point my console just stops.  I cannot CTRL+C to abort the echo command.
top shows increasing load average - presumably it thinks a process is stuck 
waiting for something.  Rebooting server kicks me out of console and then never 
comes back.

/dev/rbd0 and /dev/rbd1 are already registered at this point and were working 
fine.  If I open another console (did this before rebooting) and create 
/dev/rbd3, and then try and register it - the same happens again.

Here is what came up in kern.log:

http://pastebin.com/xNBuv3sj
(Using Gmane to post and it complained about long lines)

Any ideas?

Cheers,

James

Re: Kernel Oops (bCache hangs on registering 3rd device)

[James Sefton]

James Sefton <james@...> writes:

> 
> Hi,
> 
> Seems I broke something.
> 
> I have run into this problem about 3 times now where bCache (or something!) 
> seems to hang and I only just thought to check kern.log before trying to 
reboot 
> so was previously assuming something else had just got stuck!
> 
> I only had 3 servers available to me and once this problem occurs, the servers 
> seems to get stuck rebooting so I cannot do another test to confirm tonight.  
(I 
> have put a reboot request in for them to be force rebooted but that will not 
get 
> actioned until tomorrow)
> 
> Here is what I know about the most recent time this happened:
> 
> I had already set-up and been using /dev/bcache0 and /dev/bcache1.  They are 
> attached to the cache set and set to writeback.
> 
> The problem occurred when I ran "echo /dev/rbd2 >/sys/fs/bcache/register".   
> This device had been newly formatted with "make-bcache -B /dev/rbd2"   (rbd2 
is 
> a SAN based block device)
> 
> If it makes any difference - these commands are in a script so the register 
> command would have been run *immediately* after the make-bcache command 
> returned.  (I am going to try sticking a sleep 3 or something in there 
tomorrow 
> - just in case its related to trying to register the device so quickly after 
it 
> was prepared)
> 
> At this point my console just stops.  I cannot CTRL+C to abort the echo 
command.
> top shows increasing load average - presumably it thinks a process is stuck 
> waiting for something.  Rebooting server kicks me out of console and then 
never 
> comes back.
> 
> /dev/rbd0 and /dev/rbd1 are already registered at this point and were working 
> fine.  If I open another console (did this before rebooting) and create 
> /dev/rbd3, and then try and register it - the same happens again.
> 
> Here is what came up in kern.log:
> 
> http://pastebin.com/xNBuv3sj
> (Using Gmane to post and it complained about long lines)
> 
> Any ideas?
> 
> Cheers,
> 
> James
> 
> 


Yes, reproduced it after a clean boot.
It gets stuck creating the third cache device. (/dev/bcache2)

Here is the latest info from kern.log, but it looks very similar to what I 
posted before.

http://pastebin.com/r8YRSNGR

Any idea if I am doing something wrong or is this a bug?
I need to create up to 64 cache devices.  Possibly slightly over that on rare 
occasions.  (The majority of servers will have 16-32)

Many Thanks,

James

Re: Kernel Oops (bCache hangs on registering 3rd device)

[James Sefton]

James Sefton <james@...> writes:

> 
> James Sefton <james@...> writes:
> 
> > 
> > Hi,
> > 
> > Seems I broke something.
> > <.. pruned previous post ..>
> 
> Yes, reproduced it after a clean boot.
> It gets stuck creating the third cache device. (/dev/bcache2)
> 
> Here is the latest info from kern.log, but it looks very similar to what I 
> posted before.
> 
> http://pastebin.com/r8YRSNGR
> 
> Any idea if I am doing something wrong or is this a bug?
> I need to create up to 64 cache devices.  Possibly slightly over that on rare 
> occasions.  (The majority of servers will have 16-32)
> 
> Many Thanks,
> 
> James
> 
> 


Aha,  I have more information.

ls /dev/block -l

<filtered results to relevant>
lrwxrwxrwx 1 root root 10 Nov  5 12:51 252:0 -> ../bcache0
lrwxrwxrwx 1 root root 10 Nov  5 12:51 252:1 -> ../bcache1
lrwxrwxrwx 1 root root 12 Nov  5 12:51 252:2 -> ../bcache1p1


Our script does things in the following order:
1. Registers bcache0 - OK
 - links bcache0 in /dev/block/252:0
2. Registers bcache1 - OK
 - links bcache1 in /dev/block/252:1
3. Partitions bcache1 - OK  (single partition at the moment)
 - links bcache1p1 in /dev/block/252:2
4. Partitions bcache0 - Partition table writes okay, but bcache0p1 does not get 
created.
 - presumably attempts to link /dev/block/252:1 or something, and fails.
5. Registers bcache2 - FAIL, prevents any further bcache devices being 
registered and prevents system shut-down.
 - attempts to link bcache2 in /dev/block/252:2, which is already used for 
bcache1p1.

Probably explains why this has not been seen before since we only just added 
support for partitions with the patch details you recently gave me.

Any chance of a patch to fix the numbering?
(I wish my C++ was up to scratch so that I could do it myself as it looks like 
it could be relatively simple!)

Many Thanks,

James

Re: Kernel Oops (bCache hangs on registering 3rd device)

[Joseph Glanville]

On 6 November 2012 00:15, James Sefton <james-3k2nYdb70uTQXOPxS62xeg@public.gmane.org> wrote:
> James Sefton <james@...> writes:
>
>>
>> James Sefton <james@...> writes:
>>
>> >
>> > Hi,
>> >
>> > Seems I broke something.
>> > <.. pruned previous post ..>
>>
>> Yes, reproduced it after a clean boot.
>> It gets stuck creating the third cache device. (/dev/bcache2)
>>
>> Here is the latest info from kern.log, but it looks very similar to what I
>> posted before.
>>
>> http://pastebin.com/r8YRSNGR
>>
>> Any idea if I am doing something wrong or is this a bug?
>> I need to create up to 64 cache devices.  Possibly slightly over that on rare
>> occasions.  (The majority of servers will have 16-32)
>>
>> Many Thanks,
>>
>> James
>>
>>
>
>
> Aha,  I have more information.
>
> ls /dev/block -l
>
> <filtered results to relevant>
> lrwxrwxrwx 1 root root 10 Nov  5 12:51 252:0 -> ../bcache0
> lrwxrwxrwx 1 root root 10 Nov  5 12:51 252:1 -> ../bcache1
> lrwxrwxrwx 1 root root 12 Nov  5 12:51 252:2 -> ../bcache1p1
>
>
> Our script does things in the following order:
> 1. Registers bcache0 - OK
>  - links bcache0 in /dev/block/252:0
> 2. Registers bcache1 - OK
>  - links bcache1 in /dev/block/252:1
> 3. Partitions bcache1 - OK  (single partition at the moment)
>  - links bcache1p1 in /dev/block/252:2
> 4. Partitions bcache0 - Partition table writes okay, but bcache0p1 does not get
> created.
>  - presumably attempts to link /dev/block/252:1 or something, and fails.
> 5. Registers bcache2 - FAIL, prevents any further bcache devices being
> registered and prevents system shut-down.
>  - attempts to link bcache2 in /dev/block/252:2, which is already used for
> bcache1p1.
>
> Probably explains why this has not been seen before since we only just added
> support for partitions with the patch details you recently gave me.
>
> Any chance of a patch to fix the numbering?
> (I wish my C++ was up to scratch so that I could do it myself as it looks like
> it could be relatively simple!)
>
> Many Thanks,
>
> James
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Though not a fix, a workaround could be to use LVM rather than partitions.

LVM ontop of bcache works quite nicely you just need to add the
following to /etc/lvm.conf

types = [ "bcache", 16 ]

Also check your filter line to ensure that /dev/bcacheX will be scanned as a PV.

Once you have done that you can create a volume group as usual:

vgcreate vg0 /dev/bcache0

And then any number of LVs:

lvcreate -n root -L50G vg0

In my opinion this is considerably better than partitions.

Joseph.

-- 
CTO | Orion Virtualisation Solutions | www.orionvm.com.au
Phone: 1300 56 99 52 | Mobile: 0428 754 846

Re: Kernel Oops (bCache hangs on registering 3rd device)

[James Sefton]

Joseph Glanville <joseph.glanville@...> writes:

> 
> On 6 November 2012 00:15, James Sefton <james@...> wrote:
> > James Sefton <james@...> writes:
> >
> > Aha,  I have more information.
> >
> > ls /dev/block -l
> >
> > <filtered results to relevant>
> > lrwxrwxrwx 1 root root 10 Nov  5 12:51 252:0 -> ../bcache0
> > lrwxrwxrwx 1 root root 10 Nov  5 12:51 252:1 -> ../bcache1
> > lrwxrwxrwx 1 root root 12 Nov  5 12:51 252:2 -> ../bcache1p1
> >
> >
> > Our script does things in the following order:
> > 1. Registers bcache0 - OK
> >  - links bcache0 in /dev/block/252:0
> > 2. Registers bcache1 - OK
> >  - links bcache1 in /dev/block/252:1
> > 3. Partitions bcache1 - OK  (single partition at the moment)
> >  - links bcache1p1 in /dev/block/252:2
> > 4. Partitions bcache0 - Partition table writes okay, but bcache0p1 does not 
get
> > created.
> >  - presumably attempts to link /dev/block/252:1 or something, and fails.
> > 5. Registers bcache2 - FAIL, prevents any further bcache devices being
> > registered and prevents system shut-down.
> >  - attempts to link bcache2 in /dev/block/252:2, which is already used for
> > bcache1p1.
> >
> > Probably explains why this has not been seen before since we only just added
> > support for partitions with the patch details you recently gave me.
> >
> > Any chance of a patch to fix the numbering?
> > (I wish my C++ was up to scratch so that I could do it myself as it looks 
like
> > it could be relatively simple!)
> >
> > Many Thanks,
> >
> > James
> >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> > the body of a message to majordomo@...
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> Though not a fix, a workaround could be to use LVM rather than partitions.
> 
> LVM ontop of bcache works quite nicely you just need to add the
> following to /etc/lvm.conf
> 
> types = [ "bcache", 16 ]
> 
> Also check your filter line to ensure that /dev/bcacheX will be scanned as a 
PV.
> 
> Once you have done that you can create a volume group as usual:
> 
> vgcreate vg0 /dev/bcache0
> 
> And then any number of LVs:
> 
> lvcreate -n root -L50G vg0
> 
> In my opinion this is considerably better than partitions.
> 
> Joseph.
> 


Hi Joseph,

Many thanks for the suggestion.

I wish I could do that but the cache devices are being exposed to virtual 
machines as virtual disks and so the clients could easily repartition them back 
which would then reproduce the conflicting block device numbers.

I am also not sure how well LVM might cope if the PV's (etc..) were detected 
both in the host operating system and within a guest.

Even if LVM could work - I would still need to be able to accommodate a client 
the does not want LVM and chooses to repartition the device, and currently that 
would break the host server.

I do really appreciate the suggestion though.

Regards,

James

Re: Kernel Oops (bCache hangs on registering 3rd device)

[Joseph Glanville]

On 6 November 2012 01:32, James Sefton <james-3k2nYdb70uTQXOPxS62xeg@public.gmane.org> wrote:
> Joseph Glanville <joseph.glanville@...> writes:
>
>>
>> On 6 November 2012 00:15, James Sefton <james@...> wrote:
>> > James Sefton <james@...> writes:
>> >
>> > Aha,  I have more information.
>> >
>> > ls /dev/block -l
>> >
>> > <filtered results to relevant>
>> > lrwxrwxrwx 1 root root 10 Nov  5 12:51 252:0 -> ../bcache0
>> > lrwxrwxrwx 1 root root 10 Nov  5 12:51 252:1 -> ../bcache1
>> > lrwxrwxrwx 1 root root 12 Nov  5 12:51 252:2 -> ../bcache1p1
>> >
>> >
>> > Our script does things in the following order:
>> > 1. Registers bcache0 - OK
>> >  - links bcache0 in /dev/block/252:0
>> > 2. Registers bcache1 - OK
>> >  - links bcache1 in /dev/block/252:1
>> > 3. Partitions bcache1 - OK  (single partition at the moment)
>> >  - links bcache1p1 in /dev/block/252:2
>> > 4. Partitions bcache0 - Partition table writes okay, but bcache0p1 does not
> get
>> > created.
>> >  - presumably attempts to link /dev/block/252:1 or something, and fails.
>> > 5. Registers bcache2 - FAIL, prevents any further bcache devices being
>> > registered and prevents system shut-down.
>> >  - attempts to link bcache2 in /dev/block/252:2, which is already used for
>> > bcache1p1.
>> >
>> > Probably explains why this has not been seen before since we only just added
>> > support for partitions with the patch details you recently gave me.
>> >
>> > Any chance of a patch to fix the numbering?
>> > (I wish my C++ was up to scratch so that I could do it myself as it looks
> like
>> > it could be relatively simple!)
>> >
>> > Many Thanks,
>> >
>> > James
>> >
>> >
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
>> > the body of a message to majordomo@...
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>> Though not a fix, a workaround could be to use LVM rather than partitions.
>>
>> LVM ontop of bcache works quite nicely you just need to add the
>> following to /etc/lvm.conf
>>
>> types = [ "bcache", 16 ]
>>
>> Also check your filter line to ensure that /dev/bcacheX will be scanned as a
> PV.
>>
>> Once you have done that you can create a volume group as usual:
>>
>> vgcreate vg0 /dev/bcache0
>>
>> And then any number of LVs:
>>
>> lvcreate -n root -L50G vg0
>>
>> In my opinion this is considerably better than partitions.
>>
>> Joseph.
>>
>
>
> Hi Joseph,
>
> Many thanks for the suggestion.
>
> I wish I could do that but the cache devices are being exposed to virtual
> machines as virtual disks and so the clients could easily repartition them back
> which would then reproduce the conflicting block device numbers.
>
> I am also not sure how well LVM might cope if the PV's (etc..) were detected
> both in the host operating system and within a guest.
>
> Even if LVM could work - I would still need to be able to accommodate a client
> the does not want LVM and chooses to repartition the device, and currently that
> would break the host server.

What kind of guests are these?

If I am correct this is some sort of VPS environment, probably Xen or KVM?

In that case if you pass through a logical volume and you properly
adjust the filter in the host OS to only scan the bcache devices and
not traverse recursively this works perfectly.
The guest OS will be able to partition the logical volume and the
guest will not be able the see the underlying volume group.

>
> I do really appreciate the suggestion though.
>
> Regards,
>
> James
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Joseph.

-- 
CTO | Orion Virtualisation Solutions | www.orionvm.com.au
Phone: 1300 56 99 52 | Mobile: 0428 754 846

Re: Kernel Oops (bCache hangs on registering 3rd device)

[James Sefton]

Joseph Glanville <joseph.glanville@...> writes:

> >>
> >> Though not a fix, a workaround could be to use LVM rather than partitions.
> >>
> >> LVM ontop of bcache works quite nicely you just need to add the
> >> following to /etc/lvm.conf
> >>
> >> types = [ "bcache", 16 ]
> >>
> >> Also check your filter line to ensure that /dev/bcacheX will be scanned as 
a
> > PV.
> >>
> >> Once you have done that you can create a volume group as usual:
> >>
> >> vgcreate vg0 /dev/bcache0
> >>
> >> And then any number of LVs:
> >>
> >> lvcreate -n root -L50G vg0
> >>
> >> In my opinion this is considerably better than partitions.
> >>
> >> Joseph.
> >>
> >
> > I wish I could do that but the cache devices are being exposed to virtual
> > machines as virtual disks and so the clients could easily repartition them 
back
> > which would then reproduce the conflicting block device numbers.
> >
> > I am also not sure how well LVM might cope if the PV's (etc..) were detected
> > both in the host operating system and within a guest.
> >
> 
> What kind of guests are these?
> 
> If I am correct this is some sort of VPS environment, probably Xen or KVM?
> 
> In that case if you pass through a logical volume and you properly
> adjust the filter in the host OS to only scan the bcache devices and
> not traverse recursively this works perfectly.
> The guest OS will be able to partition the logical volume and the
> guest will not be able the see the underlying volume group.
> 
> 
> Joseph.
> 


Hi Joseph,

Again, thank you for putting in the time to help me.

Yes, it is KVM.

I don't think I had thought about it the way you describe. (Passing an LV to KVM 
instead of the bcache device)   I will certainly give it a try.  Sounds like it 
could solve all my problems.

I will drop a reply here if it works.

Thanks again,

James

Re: Kernel Oops (bCache hangs on registering 3rd device)

[Joseph Glanville]

On 6 November 2012 03:02, James Sefton <james-3k2nYdb70uTQXOPxS62xeg@public.gmane.org> wrote:
> Joseph Glanville <joseph.glanville@...> writes:
>
>> >>
>> >> Though not a fix, a workaround could be to use LVM rather than partitions.
>> >>
>> >> LVM ontop of bcache works quite nicely you just need to add the
>> >> following to /etc/lvm.conf
>> >>
>> >> types = [ "bcache", 16 ]
>> >>
>> >> Also check your filter line to ensure that /dev/bcacheX will be scanned as
> a
>> > PV.
>> >>
>> >> Once you have done that you can create a volume group as usual:
>> >>
>> >> vgcreate vg0 /dev/bcache0
>> >>
>> >> And then any number of LVs:
>> >>
>> >> lvcreate -n root -L50G vg0
>> >>
>> >> In my opinion this is considerably better than partitions.
>> >>
>> >> Joseph.
>> >>
>> >
>> > I wish I could do that but the cache devices are being exposed to virtual
>> > machines as virtual disks and so the clients could easily repartition them
> back
>> > which would then reproduce the conflicting block device numbers.
>> >
>> > I am also not sure how well LVM might cope if the PV's (etc..) were detected
>> > both in the host operating system and within a guest.
>> >
>>
>> What kind of guests are these?
>>
>> If I am correct this is some sort of VPS environment, probably Xen or KVM?
>>
>> In that case if you pass through a logical volume and you properly
>> adjust the filter in the host OS to only scan the bcache devices and
>> not traverse recursively this works perfectly.
>> The guest OS will be able to partition the logical volume and the
>> guest will not be able the see the underlying volume group.
>>
>>
>> Joseph.
>>
>
>
> Hi Joseph,
>
> Again, thank you for putting in the time to help me.
>
> Yes, it is KVM.
>
> I don't think I had thought about it the way you describe. (Passing an LV to KVM
> instead of the bcache device)   I will certainly give it a try.  Sounds like it
> could solve all my problems.
>
> I will drop a reply here if it works.
>
> Thanks again,

No problem.

A word of warning though, if you intend to use LVM inside the guests
(or if they are untrusted guests) then you should certainly adjust the
filter line in lvm.conf to prevent recursive scanning.
Otherwise you will see the guest PVs (and thus VGs and LVs) inside the
host. Which is usually -bad-.

Something like:

filter = ["a/bcache.*/", "r/.*/"]

You obviously will need to also have an appropriate white list filter
for other LVM volumes you want to use. ( "a/sdb/" for example)

>
> James
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
CTO | Orion Virtualisation Solutions | www.orionvm.com.au
Phone: 1300 56 99 52 | Mobile: 0428 754 846