bcache on NBD block devices

bcache on NBD block devices

[Juan Antonio Martinez]

Hi all

(This is my first post on this list. I'm still a novice on bcache and
proper nettiquete on this list, so apologize for my mistakes)

Scenario: several NBD LTSP Fat Clients on (a bit) obsolete hardware
kernel 3.11.0-15 on Ubuntu 13.10

/dev/sda1 as local cache device
/dev/nbd1 as remote NBD backing bcache device

- I've created and registered /dev/sda1 as cache device without problems
- To create nbd file to be exported I've typed following sequence:

# create an empty file
root# dd if=/dev/zero of=bcache_test.img bs=1M count=64
# use it as loop block device
root# losetup /dev/loop0
# make it a backing bcache device
root# make-bcache -B /dev/loop0
# create a tiny fs on it
root# mksquashfs /usr/local /dev/bcache0 -noappend

Notice the squashfs: I only need to export fs in read-only mode

At this point I can mount file throught bcache w/o problems,
so unregister, de-loopize and copy raw bcache_test.img file to nbd
server host to be exported.

nbd-server is properly configure: I can list, connect and use file:

root# nbd-client server -N bcache_test /dev/nbd1 -persist -timeout 60 
Negotiation: ..size = 64MB
bs=1024, sz=67108864 bytes
timeout=60

Using bcache-super-show tells me that file is ok for bcache

root#  bcache-super-show /dev/nbd1
sb.magic		ok
sb.first_sector		8 [match]
sb.csum			AC160BF8DFFB808D [match]
sb.version		1 [backing device]

dev.label		(empty)
dev.uuid		936f8be4-7a90-4d2b-b118-5a45a1e37603
dev.sectors_per_block	1
dev.sectors_per_bucket	1024
dev.data.first_sector	16
dev.data.cache_mode	0 [writethrough]
dev.data.cache_state	1 [clean]

cset.uuid		479b0df0-56d3-4f31-951f-ffb69d8d233b

But.... At this point /dev/bcache0 is not created so cannot register it
manually with "echo <CSET-UUID> > /sys/block/bcache0/bcache/attach"

Item worse: NBD doesn't provide bcache directory under /sys/block/nbd1,
so can't execute "echo 1 > /sys/block/nbd1/bcache/running"

Notice that If I "dd" /dev/nbd1 to a local file and "losetup" on
it, /dev/bcache0 appears automagically attached to /dev/loop0

¿Any ideas?
¿Does NBD supports bcache in kernel 3.11?
¿If doesn't, any idea on the best way to provide local disk cache for
NBD remote devices?

Thanks in advance
Juan Antonio



------------------------------------------------------------------------------
Flow-based real-time traffic analytics software. Cisco certified tool.
Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer
Customize your own dashboards, set traffic alerts and generate reports.
Network behavioral analysis & security monitoring. All-in-one tool.
http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk
_______________________________________________
Nbd-general mailing list
Nbd-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nbd-general

Re: bcache on NBD block devices

[Paul Clements]

[-- Attachment #1.1: Type: text/plain, Size: 958 bytes --]

On Tue, Feb 25, 2014 at 9:40 AM, Juan Antonio Martinez
<jantonio-oytJ0Z/vEwre5aOfsHch1g@public.gmane.org>wrote:

> Hi all
>
> (This is my first post on this list. I'm still a novice on bcache and
> proper nettiquete on this list, so apologize for my mistakes)
>
> Scenario: several NBD LTSP Fat Clients on (a bit) obsolete hardware
> kernel 3.11.0-15 on Ubuntu 13.10
>
> /dev/sda1 as local cache device
> /dev/nbd1 as remote NBD backing bcache device
>
> - I've created and registered /dev/sda1 as cache device without problems
> - To create nbd file to be exported I've typed following sequence:
>
> # create an empty file
> root# dd if=/dev/zero of=bcache_test.img bs=1M count=64
> # use it as loop block device
> root# losetup /dev/loop0
> # make it a backing bcache device
> root# make-bcache -B /dev/loop0
>

have you tried doing this step from the client machine instead, i.e.:

make-bache -B /dev/nbd1

after the nbd connection is up? does that work?

[-- Attachment #1.2: Type: text/html, Size: 1438 bytes --]

[-- Attachment #2: Type: text/plain, Size: 436 bytes --]

------------------------------------------------------------------------------
Flow-based real-time traffic analytics software. Cisco certified tool.
Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer
Customize your own dashboards, set traffic alerts and generate reports.
Network behavioral analysis & security monitoring. All-in-one tool.
http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk

[-- Attachment #3: Type: text/plain, Size: 192 bytes --]

_______________________________________________
Nbd-general mailing list
Nbd-general-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/nbd-general

Re: bcache on NBD block devices

[Juan Antonio Martinez]

El mar, 25-02-2014 a las 11:37 -0500, Paul Clements escribió:
> On Tue, Feb 25, 2014 at 9:40 AM, Juan Antonio Martinez
> <jantonio@dit.upm.es> wrote:
> > Hi all
> > (This is my first post on this list. I'm still a novice on bcache and
> > proper nettiquete on this list, so apologize for my mistakes)
> > Scenario: several NBD LTSP Fat Clients on (a bit) obsolete hardware
> > kernel 3.11.0-15 on Ubuntu 13.10
> > /dev/sda1 as local cache device
> > /dev/nbd1 as remote NBD backing bcache device
> > - I've created and registered /dev/sda1 as cache device without problems
> > - To create nbd file to be exported I've typed following sequence:
> > # create an empty file
> > root# dd if=/dev/zero of=bcache_test.img bs=1M count=64
> > # use it as loop block device
> > root# losetup /dev/loop0
> > # make it a backing bcache device
> > root# make-bcache -B /dev/loop0

> have you tried doing this step from the client machine instead, i.e.:
> make-bache -B /dev/nbd1
> after the nbd connection is up? does that work?

Just tested: works fine.... but only in the LTSP client where I did the
changes, and had to tell nbd-server to export imagefile in read-write
mode

Remember that my configuration consist in several (>250!) LTSP-NBD
clients sharing _the_same_ mounted squashfs image file. I cannot for
obious reasons make the image file to be shared read-write on every
clients at the same time... and seems that bcache needs write access to
the backing block device to work

¿Is this correct? If so, bcache is not for me :-( ¿Any alternatives? 

Perhaps I coult try to create bcache'd image file, and then modify NBD
server permissions to export in copy-on-write-mode... but this would
work if bcache only writes in bcache superblock ( my image files are
typically 8Gb size each) as this will create one cow file per client on
server

Cheers
Juan Antonio


------------------------------------------------------------------------------
Flow-based real-time traffic analytics software. Cisco certified tool.
Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer
Customize your own dashboards, set traffic alerts and generate reports.
Network behavioral analysis & security monitoring. All-in-one tool.
http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk
_______________________________________________
Nbd-general mailing list
Nbd-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nbd-general

Re: bcache on NBD block devices

[Juan Antonio Martinez]

I answer mySelf: exporting NBD image file in copy-on-write mode, and
starting several clients at the same time I can see that diff file is
only 4096 bytes long ( the size of backing device bcache superblock) and
seems not to increase size. So I can continue working with bcache :-)

Thanks for your time. Cheers
Juan Antonio

El mié, 26-02-2014 a las 09:31 +0100, Juan Antonio Martinez escribió:
> El mar, 25-02-2014 a las 11:37 -0500, Paul Clements escribió:
> > On Tue, Feb 25, 2014 at 9:40 AM, Juan Antonio Martinez
> > <jantonio@dit.upm.es> wrote:
> > > Hi all
> > > (This is my first post on this list. I'm still a novice on bcache and
> > > proper nettiquete on this list, so apologize for my mistakes)
> > > Scenario: several NBD LTSP Fat Clients on (a bit) obsolete hardware
> > > kernel 3.11.0-15 on Ubuntu 13.10
> > > /dev/sda1 as local cache device
> > > /dev/nbd1 as remote NBD backing bcache device
> > > - I've created and registered /dev/sda1 as cache device without problems
> > > - To create nbd file to be exported I've typed following sequence:
> > > # create an empty file
> > > root# dd if=/dev/zero of=bcache_test.img bs=1M count=64
> > > # use it as loop block device
> > > root# losetup /dev/loop0
> > > # make it a backing bcache device
> > > root# make-bcache -B /dev/loop0
> 
> > have you tried doing this step from the client machine instead, i.e.:
> > make-bache -B /dev/nbd1
> > after the nbd connection is up? does that work?
> 
> Just tested: works fine.... but only in the LTSP client where I did the
> changes, and had to tell nbd-server to export imagefile in read-write
> mode
> 
> Remember that my configuration consist in several (>250!) LTSP-NBD
> clients sharing _the_same_ mounted squashfs image file. I cannot for
> obious reasons make the image file to be shared read-write on every
> clients at the same time... and seems that bcache needs write access to
> the backing block device to work
> 
> ¿Is this correct? If so, bcache is not for me :-( ¿Any alternatives? 
> 
> Perhaps I coult try to create bcache'd image file, and then modify NBD
> server permissions to export in copy-on-write-mode... but this would
> work if bcache only writes in bcache superblock ( my image files are
> typically 8Gb size each) as this will create one cow file per client on
> server
> 
> Cheers
> Juan Antonio