blkback and bcache

blkback and bcache

[James Harper]

I'm having trouble using blkback under gplpv when the disk is on top of a bcache device. My devices are layered as follows:

/dev/sd[ab]
md0 (RAID1)
bcache
lvm

It seems that bcache presents a 4K sector size to Linux, which is then reflected by lvm and in turn blkback.

Obviously GPLPV isn't handling 4K sectors correctly... any suggestions as to what I might need to do to make this work properly? As a last resort I should be able to fake 512 byte sectors to Windows but would prefer that Windows knew it was dealing with a device with 4K sectors underneath.

Thanks

James

Re: [Xen-devel] blkback and bcache

[Joseph Glanville]

On 13 August 2012 17:16, James Harper <james.harper-NMzNsA1hOHcW+bLBXbPJGg@public.gmane.org> wrote:
> I'm having trouble using blkback under gplpv when the disk is on top of a bcache device. My devices are layered as follows:
>
> /dev/sd[ab]
> md0 (RAID1)
> bcache
> lvm
>
> It seems that bcache presents a 4K sector size to Linux, which is then reflected by lvm and in turn blkback.
>
> Obviously GPLPV isn't handling 4K sectors correctly... any suggestions as to what I might need to do to make this work properly? As a last resort I should be able to fake 512 byte sectors to Windows but would prefer that Windows knew it was dealing with a device with 4K sectors underneath.
>
> Thanks
>
> James
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel-GuqFBffKawuEi8DpZVb4nw@public.gmane.org
> http://lists.xen.org/xen-devel

This could very well be the issue I was having, I haven't been able to
pull the latest bcache code for a few days (repo down?) but if I can
help debug let me know.

-- 
CTO | Orion Virtualisation Solutions | www.orionvm.com.au
Phone: 1300 56 99 52 | Mobile: 0428 754 846

Re: blkback and bcache

[Joseph Glanville]

On 13 August 2012 17:16, James Harper <james.harper@bendigoit.com.au> wrote:
> I'm having trouble using blkback under gplpv when the disk is on top of a bcache device. My devices are layered as follows:
>
> /dev/sd[ab]
> md0 (RAID1)
> bcache
> lvm
>
> It seems that bcache presents a 4K sector size to Linux, which is then reflected by lvm and in turn blkback.
>
> Obviously GPLPV isn't handling 4K sectors correctly... any suggestions as to what I might need to do to make this work properly? As a last resort I should be able to fake 512 byte sectors to Windows but would prefer that Windows knew it was dealing with a device with 4K sectors underneath.
>
> Thanks
>
> James
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

This could very well be the issue I was having, I haven't been able to
pull the latest bcache code for a few days (repo down?) but if I can
help debug let me know.

-- 
CTO | Orion Virtualisation Solutions | www.orionvm.com.au
Phone: 1300 56 99 52 | Mobile: 0428 754 846

RE: [Xen-devel] blkback and bcache

[James Harper]

> 
> This could very well be the issue I was having, I haven't been able to pull the
> latest bcache code for a few days (repo down?) but if I can help debug let me
> know.
> 

Is it Windows or Linux giving you problems? I've only tested Windows so far.

James

Re: blkback and bcache

[James Harper]

> 
> This could very well be the issue I was having, I haven't been able to pull the
> latest bcache code for a few days (repo down?) but if I can help debug let me
> know.
> 

Is it Windows or Linux giving you problems? I've only tested Windows so far.

James

Re: [Xen-devel] blkback and bcache

[Joseph Glanville]

On 13 August 2012 18:17, James Harper <james.harper-NMzNsA1hOHcW+bLBXbPJGg@public.gmane.org> wrote:
>>
>> This could very well be the issue I was having, I haven't been able to pull the
>> latest bcache code for a few days (repo down?) but if I can help debug let me
>> know.
>>
>
> Is it Windows or Linux giving you problems? I've only tested Windows so far.
>
> James
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Both, I wasn't able to get either to work correctly with blkback.

-- 
CTO | Orion Virtualisation Solutions | www.orionvm.com.au
Phone: 1300 56 99 52 | Mobile: 0428 754 846

Re: blkback and bcache

[Joseph Glanville]

On 13 August 2012 18:17, James Harper <james.harper@bendigoit.com.au> wrote:
>>
>> This could very well be the issue I was having, I haven't been able to pull the
>> latest bcache code for a few days (repo down?) but if I can help debug let me
>> know.
>>
>
> Is it Windows or Linux giving you problems? I've only tested Windows so far.
>
> James
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Both, I wasn't able to get either to work correctly with blkback.

-- 
CTO | Orion Virtualisation Solutions | www.orionvm.com.au
Phone: 1300 56 99 52 | Mobile: 0428 754 846

Re: [Xen-devel] blkback and bcache

[Kent Overstreet]

On Mon, Aug 13, 2012 at 08:53:44PM +1000, Joseph Glanville wrote:
> On 13 August 2012 18:17, James Harper <james.harper-NMzNsA1hOHcW+bLBXbPJGg@public.gmane.org> wrote:
> >>
> >> This could very well be the issue I was having, I haven't been able to pull the
> >> latest bcache code for a few days (repo down?) but if I can help debug let me
> >> know.
> >>
> >
> > Is it Windows or Linux giving you problems? I've only tested Windows so far.
> >
> > James
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> Both, I wasn't able to get either to work correctly with blkback.

Just mentioned this in the other thread, but if this is due to the 4k
blocksize - that's easy to fix: just format with 512 byte blocksize

make-bcache --block 512

Maybe I should change the default.

Re: blkback and bcache

[Kent Overstreet]

On Mon, Aug 13, 2012 at 08:53:44PM +1000, Joseph Glanville wrote:
> On 13 August 2012 18:17, James Harper <james.harper@bendigoit.com.au> wrote:
> >>
> >> This could very well be the issue I was having, I haven't been able to pull the
> >> latest bcache code for a few days (repo down?) but if I can help debug let me
> >> know.
> >>
> >
> > Is it Windows or Linux giving you problems? I've only tested Windows so far.
> >
> > James
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> Both, I wasn't able to get either to work correctly with blkback.

Just mentioned this in the other thread, but if this is due to the 4k
blocksize - that's easy to fix: just format with 512 byte blocksize

make-bcache --block 512

Maybe I should change the default.

RE: [Xen-devel] blkback and bcache

[James Harper]

> 
> Just mentioned this in the other thread, but if this is due to the 4k blocksize -
> that's easy to fix: just format with 512 byte blocksize
> 
> make-bcache --block 512
> 
> Maybe I should change the default.

I suggest making the default 512, but also print a warning if the user didn't explicitly set it,eg "Block size not set - defaulting to 512 bytes"

James

Re: blkback and bcache

[James Harper]

> 
> Just mentioned this in the other thread, but if this is due to the 4k blocksize -
> that's easy to fix: just format with 512 byte blocksize
> 
> make-bcache --block 512
> 
> Maybe I should change the default.

I suggest making the default 512, but also print a warning if the user didn't explicitly set it,eg "Block size not set - defaulting to 512 bytes"

James

Re: [Xen-devel] blkback and bcache

[Joseph Glanville]

Hi James, Kent.

I can confirm this is the issue I was seeing, thanks for locating the
blkback issue!
Is there anything xen-devel should be doing about this? I wouldn't
expect blkback to care about block size...

Joseph.

On 14 August 2012 09:30, James Harper <james.harper-NMzNsA1hOHcW+bLBXbPJGg@public.gmane.org> wrote:
>>
>> Just mentioned this in the other thread, but if this is due to the 4k blocksize -
>> that's easy to fix: just format with 512 byte blocksize
>>
>> make-bcache --block 512
>>
>> Maybe I should change the default.
>
> I suggest making the default 512, but also print a warning if the user didn't explicitly set it,eg "Block size not set - defaulting to 512 bytes"
>
> James
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
CTO | Orion Virtualisation Solutions | www.orionvm.com.au
Phone: 1300 56 99 52 | Mobile: 0428 754 846

Re: blkback and bcache

[Joseph Glanville]

Hi James, Kent.

I can confirm this is the issue I was seeing, thanks for locating the
blkback issue!
Is there anything xen-devel should be doing about this? I wouldn't
expect blkback to care about block size...

Joseph.

On 14 August 2012 09:30, James Harper <james.harper@bendigoit.com.au> wrote:
>>
>> Just mentioned this in the other thread, but if this is due to the 4k blocksize -
>> that's easy to fix: just format with 512 byte blocksize
>>
>> make-bcache --block 512
>>
>> Maybe I should change the default.
>
> I suggest making the default 512, but also print a warning if the user didn't explicitly set it,eg "Block size not set - defaulting to 512 bytes"
>
> James
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
CTO | Orion Virtualisation Solutions | www.orionvm.com.au
Phone: 1300 56 99 52 | Mobile: 0428 754 846

Re: [Xen-devel] blkback and bcache

[Kent Overstreet]

On Wed, Aug 15, 2012 at 10:40:08PM +1000, Joseph Glanville wrote:
> Hi James, Kent.
> 
> I can confirm this is the issue I was seeing, thanks for locating the
> blkback issue!

Cool!

> Is there anything xen-devel should be doing about this? I wouldn't
> expect blkback to care about block size...

Well, I wouldn't be surprised if Windows doesn't work on a device with
block size > 512 bytes. But Linux (ext4 at least) certaintly does work
with 4k blocks - unless maybe it was breaking on something in the boot
process?

So it sounds like this might be indicative of a bug in blkback.

> Joseph.
> 
> On 14 August 2012 09:30, James Harper <james.harper-NMzNsA1hOHcW+bLBXbPJGg@public.gmane.org> wrote:
> >>
> >> Just mentioned this in the other thread, but if this is due to the 4k blocksize -
> >> that's easy to fix: just format with 512 byte blocksize
> >>
> >> make-bcache --block 512
> >>
> >> Maybe I should change the default.
> >
> > I suggest making the default 512, but also print a warning if the user didn't explicitly set it,eg "Block size not set - defaulting to 512 bytes"
> >
> > James
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> 
> -- 
> CTO | Orion Virtualisation Solutions | www.orionvm.com.au
> Phone: 1300 56 99 52 | Mobile: 0428 754 846

Re: blkback and bcache

[Kent Overstreet]

On Wed, Aug 15, 2012 at 10:40:08PM +1000, Joseph Glanville wrote:
> Hi James, Kent.
> 
> I can confirm this is the issue I was seeing, thanks for locating the
> blkback issue!

Cool!

> Is there anything xen-devel should be doing about this? I wouldn't
> expect blkback to care about block size...

Well, I wouldn't be surprised if Windows doesn't work on a device with
block size > 512 bytes. But Linux (ext4 at least) certaintly does work
with 4k blocks - unless maybe it was breaking on something in the boot
process?

So it sounds like this might be indicative of a bug in blkback.

> Joseph.
> 
> On 14 August 2012 09:30, James Harper <james.harper@bendigoit.com.au> wrote:
> >>
> >> Just mentioned this in the other thread, but if this is due to the 4k blocksize -
> >> that's easy to fix: just format with 512 byte blocksize
> >>
> >> make-bcache --block 512
> >>
> >> Maybe I should change the default.
> >
> > I suggest making the default 512, but also print a warning if the user didn't explicitly set it,eg "Block size not set - defaulting to 512 bytes"
> >
> > James
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> 
> -- 
> CTO | Orion Virtualisation Solutions | www.orionvm.com.au
> Phone: 1300 56 99 52 | Mobile: 0428 754 846

RE: [Xen-devel] blkback and bcache

[James Harper]

> > Is there anything xen-devel should be doing about this? I wouldn't
> > expect blkback to care about block size...
> 
> Well, I wouldn't be surprised if Windows doesn't work on a device with block
> size > 512 bytes. But Linux (ext4 at least) certaintly does work with 4k blocks -
> unless maybe it was breaking on something in the boot process?
> 
> So it sounds like this might be indicative of a bug in blkback.
> 

As far as I can tell, blkback works internally in 512 byte sectors, but does a bounds check of the 512 byte sector offset against the native sector count, so if you have 4K sectors you get an error trying to access anything past 1/8th of the disk. This is a problem, but qemu can't boot off anything but 512 byte sectors so that is the more limiting factor. I suspect a more thorough audit of blkback's behaviour wrt 4K sectors would be in order before simply fixing the bounds check bug, although it seems to work fine.

Curiously, despite literature to the contrary, Windows itself doesn't seem to care if the sector size is 4K. I added a second disk to an already running Windows DomU, partitioned it, and formatted it, and tested it (copying files on, off, rebooting, etc). As long as the size of my partition was <1/8th the size of the disk it worked fine.

James

Re: blkback and bcache

[James Harper]

> > Is there anything xen-devel should be doing about this? I wouldn't
> > expect blkback to care about block size...
> 
> Well, I wouldn't be surprised if Windows doesn't work on a device with block
> size > 512 bytes. But Linux (ext4 at least) certaintly does work with 4k blocks -
> unless maybe it was breaking on something in the boot process?
> 
> So it sounds like this might be indicative of a bug in blkback.
> 

As far as I can tell, blkback works internally in 512 byte sectors, but does a bounds check of the 512 byte sector offset against the native sector count, so if you have 4K sectors you get an error trying to access anything past 1/8th of the disk. This is a problem, but qemu can't boot off anything but 512 byte sectors so that is the more limiting factor. I suspect a more thorough audit of blkback's behaviour wrt 4K sectors would be in order before simply fixing the bounds check bug, although it seems to work fine.

Curiously, despite literature to the contrary, Windows itself doesn't seem to care if the sector size is 4K. I added a second disk to an already running Windows DomU, partitioned it, and formatted it, and tested it (copying files on, off, rebooting, etc). As long as the size of my partition was <1/8th the size of the disk it worked fine.

James