bcachefs with cache device and backing device

bcachefs with cache device and backing device

[Martin McClure]

I'm trying out bcachefs for the first time. I can follow the
instructions (format a cache device, mount it type bcache) and it works.

What I'd like to do, however, is to have an SSD cache device, and a
rotating-disk backing device, as a bcachefs filesystem. Is this
something that bcachefs can do at present? If so, what's the basic
procedure?

Thanks,

-Martin

Re: bcachefs with cache device and backing device

[Kent Overstreet]

On Thu, May 12, 2016 at 06:50:26PM -0700, Martin McClure wrote:
> I'm trying out bcachefs for the first time. I can follow the
> instructions (format a cache device, mount it type bcache) and it works.
> 
> What I'd like to do, however, is to have an SSD cache device, and a
> rotating-disk backing device, as a bcachefs filesystem. Is this
> something that bcachefs can do at present? If so, what's the basic
> procedure?

Yeah - tiering replaces cache/backing devices

IIRC,

bcache format --tier 0 -C <SSD> --tier 1 -C <spinning rust>

(the -C is going to go away at some point)

Re: bcachefs with cache device and backing device

[Martin McClure]

On 05/12/2016 09:36 PM, Kent Overstreet wrote:
>> What I'd like to do, however, is to have an SSD cache device, and a
>> rotating-disk backing device, as a bcachefs filesystem. Is this
>> something that bcachefs can do at present? If so, what's the basic
>> procedure?
>
> Yeah - tiering replaces cache/backing devices
>
> IIRC,
>
> bcache format --tier 0 -C <SSD> --tier 1 -C <spinning rust>
>
> (the -C is going to go away at some point)

Thanks! I'll try that out tomorrow.

-Martin

Re: bcachefs with cache device and backing device

[Martin McClure]

On 05/12/2016 09:36 PM, Kent Overstreet wrote:
> 
> Yeah - tiering replaces cache/backing devices
> 
> IIRC,
> 
> bcache format --tier 0 -C <SSD> --tier 1 -C <spinning rust>
> 
> (the -C is going to go away at some point)
> 

Had a chance to play with this some more, but still not getting it to
work...

Formatting seems to work, and once I do this:

  echo /dev/sdd1 > /sys/fs/bcache/register
  echo /dev/nvme0n1 > /sys/fs/bcache/register
  echo 1 > /sys/fs/bcache/<set-uuid>/blockdev_volume_create

a /dev/bcache0 has been created. However, if I try to mount it:

  mount -t bcache /dev/bcache0 /mnt

it says:

  mount: No such file or directory

with a return code of 32, which is documented as "mount failure".

At this point I reach the limit of my current understanding, but would
like to understand more.

Thanks,

-Martin

Re: bcachefs with cache device and backing device

[Kent Overstreet]

On Tue, May 17, 2016 at 07:46:33PM -0700, Martin McClure wrote:
> On 05/12/2016 09:36 PM, Kent Overstreet wrote:
> > 
> > Yeah - tiering replaces cache/backing devices
> > 
> > IIRC,
> > 
> > bcache format --tier 0 -C <SSD> --tier 1 -C <spinning rust>
> > 
> > (the -C is going to go away at some point)
> > 
> 
> Had a chance to play with this some more, but still not getting it to
> work...
> 
> Formatting seems to work, and once I do this:
> 
>   echo /dev/sdd1 > /sys/fs/bcache/register
>   echo /dev/nvme0n1 > /sys/fs/bcache/register
>   echo 1 > /sys/fs/bcache/<set-uuid>/blockdev_volume_create
> 
> a /dev/bcache0 has been created. However, if I try to mount it:
> 
>   mount -t bcache /dev/bcache0 /mnt
> 
> it says:
> 
>   mount: No such file or directory
> 
> with a return code of 32, which is documented as "mount failure".
> 
> At this point I reach the limit of my current understanding, but would
> like to understand more.

The intended mount path for multi device filesystems is currently broken...
Chris got it working (to my surprise) by - I belive - registering all the
devices via /sys/fs/bcache/register, and then mounting just one of the block
devices - Chris, is that correct?

FYI though: tiering mode is _not_ as well tested currently, and Chris managed to
hit a bug when he completely filled up tier 0 and it then wouldn't mount,
because the btree updates journal replay was doing wouldn't fit on tier 0 - and
I haven't merged the fix I sent him because it was kind of a hack.

But I just got back from vacation so I should be able to start working through
these issues...

Re: bcachefs with cache device and backing device

[Christopher James Halse Rogers]

On Wed, May 18, 2016 at 2:01 PM, Kent Overstreet 
<kent.overstreet@gmail.com> wrote:
> On Tue, May 17, 2016 at 07:46:33PM -0700, Martin McClure wrote:
>>  On 05/12/2016 09:36 PM, Kent Overstreet wrote:
>>  >
>>  > Yeah - tiering replaces cache/backing devices
>>  >
>>  > IIRC,
>>  >
>>  > bcache format --tier 0 -C <SSD> --tier 1 -C <spinning rust>
>>  >
>>  > (the -C is going to go away at some point)
>>  >
>> 
>>  Had a chance to play with this some more, but still not getting it 
>> to
>>  work...
>> 
>>  Formatting seems to work, and once I do this:
>> 
>>    echo /dev/sdd1 > /sys/fs/bcache/register
>>    echo /dev/nvme0n1 > /sys/fs/bcache/register
>>    echo 1 > /sys/fs/bcache/<set-uuid>/blockdev_volume_create
>> 
>>  a /dev/bcache0 has been created. However, if I try to mount it:
>> 
>>    mount -t bcache /dev/bcache0 /mnt
>> 
>>  it says:
>> 
>>    mount: No such file or directory
>> 
>>  with a return code of 32, which is documented as "mount failure".
>> 
>>  At this point I reach the limit of my current understanding, but 
>> would
>>  like to understand more.
> 
> The intended mount path for multi device filesystems is currently 
> broken...
> Chris got it working (to my surprise) by - I belive - registering all 
> the
> devices via /sys/fs/bcache/register, and then mounting just one of 
> the block
> devices - Chris, is that correct?

That is indeed correct. Once the volume has all its components 
registered, it can be mounted by any of the block devices.

Re: bcachefs with cache device and backing device

[Kent Overstreet]

On Wed, May 18, 2016 at 03:06:39PM +1000, Christopher James Halse Rogers wrote:
> On Wed, May 18, 2016 at 2:01 PM, Kent Overstreet <kent.overstreet@gmail.com>
> wrote:
> > On Tue, May 17, 2016 at 07:46:33PM -0700, Martin McClure wrote:
> > >  On 05/12/2016 09:36 PM, Kent Overstreet wrote:
> > >  >
> > >  > Yeah - tiering replaces cache/backing devices
> > >  >
> > >  > IIRC,
> > >  >
> > >  > bcache format --tier 0 -C <SSD> --tier 1 -C <spinning rust>
> > >  >
> > >  > (the -C is going to go away at some point)
> > >  >
> > > 
> > >  Had a chance to play with this some more, but still not getting it
> > > to
> > >  work...
> > > 
> > >  Formatting seems to work, and once I do this:
> > > 
> > >    echo /dev/sdd1 > /sys/fs/bcache/register
> > >    echo /dev/nvme0n1 > /sys/fs/bcache/register
> > >    echo 1 > /sys/fs/bcache/<set-uuid>/blockdev_volume_create
> > > 
> > >  a /dev/bcache0 has been created. However, if I try to mount it:
> > > 
> > >    mount -t bcache /dev/bcache0 /mnt
> > > 
> > >  it says:
> > > 
> > >    mount: No such file or directory
> > > 
> > >  with a return code of 32, which is documented as "mount failure".
> > > 
> > >  At this point I reach the limit of my current understanding, but
> > > would
> > >  like to understand more.
> > 
> > The intended mount path for multi device filesystems is currently
> > broken...
> > Chris got it working (to my surprise) by - I belive - registering all
> > the
> > devices via /sys/fs/bcache/register, and then mounting just one of the
> > block
> > devices - Chris, is that correct?
> 
> That is indeed correct. Once the volume has all its components registered,
> it can be mounted by any of the block devices.

Actually, I just tested and mounting multile devices directly _does_ work - you
just pass a colon separated list of devices to mount:

mount -t bcache /dev/sdb:/dev/sdc /mnt

I could have sworn this was broken, but worked just now...

Re: bcachefs with cache device and backing device

[Martin McClure]

On 05/17/2016 10:41 PM, Kent Overstreet wrote:
> Actually, I just tested and mounting multile devices directly _does_ work - you
> just pass a colon separated list of devices to mount:
> 
> mount -t bcache /dev/sdb:/dev/sdc /mnt
> 
> I could have sworn this was broken, but worked just now...

This works for me. Thanks for the feedback!

For any others wanting to walk this path in the near future, here's the
entire sequence that is working for me (as of kernel and tools pulled on
2016-05-13):

  bcache format --tier 0 --bucket=512k --cache_mode=writeback \
      -C /dev/nvme0n1 \
      --tier 1 -C /dev/sdd1

  echo /dev/sdd1 > /sys/fs/bcache/register
  echo /dev/nvme0n1 > /sys/fs/bcache/register
  mount -t bcache /dev/nvme0n1:/dev/sdd1 /mnt/whatever

Regards,

-Martin

Re: bcachefs with cache device and backing device

[Christopher James Halse Rogers]

On Fri, May 20, 2016 at 10:41 AM, Martin McClure 
<martin.mcclure@gemtalksystems.com> wrote:
> On 05/17/2016 10:41 PM, Kent Overstreet wrote:
>>  Actually, I just tested and mounting multile devices directly 
>> _does_ work - you
>>  just pass a colon separated list of devices to mount:
>> 
>>  mount -t bcache /dev/sdb:/dev/sdc /mnt
>> 
>>  I could have sworn this was broken, but worked just now...
> 
> This works for me. Thanks for the feedback!
> 
> For any others wanting to walk this path in the near future, here's 
> the
> entire sequence that is working for me (as of kernel and tools pulled 
> on
> 2016-05-13):
> 
>   bcache format --tier 0 --bucket=512k --cache_mode=writeback \
>       -C /dev/nvme0n1 \
>       --tier 1 -C /dev/sdd1
> 
>   echo /dev/sdd1 > /sys/fs/bcache/register
>   echo /dev/nvme0n1 > /sys/fs/bcache/register

You should be able to remove these steps, I think?

> 
>   mount -t bcache /dev/nvme0n1:/dev/sdd1 /mnt/whatever

If you've pre-registered, you should be able to just
mount -t bcache /dev/nvme0n1 /mnt/whatever
or, entirely equivalently:
mount -t bcache /dev/sdd1 /mnt/whatever

Re: bcachefs with cache device and backing device

[Martin McClure]

On 05/19/2016 05:44 PM, Christopher James Halse Rogers wrote:
>>
>>   bcache format --tier 0 --bucket=512k --cache_mode=writeback \
>>       -C /dev/nvme0n1 \
>>       --tier 1 -C /dev/sdd1
>>
>>   echo /dev/sdd1 > /sys/fs/bcache/register
>>   echo /dev/nvme0n1 > /sys/fs/bcache/register
> 
> You should be able to remove these steps, I think?

Yes, I don't seem to need those.

> 
>>
>>   mount -t bcache /dev/nvme0n1:/dev/sdd1 /mnt/whatever
> 
> If you've pre-registered, you should be able to just
> mount -t bcache /dev/nvme0n1 /mnt/whatever
> or, entirely equivalently:
> mount -t bcache /dev/sdd1 /mnt/whatever
> 

Yes. Looks like I need to *either* explicitly pre-register the devices
*or* specify them both in the mount command.

Thanks for the clarification.

The option that I'm going with for now is:

  bcache format --tier 0 --bucket=512k --cache_mode=writeback \
      -C /dev/nvme0n1 \
      --tier 1 -C /dev/sdd1

  mount -t bcache /dev/nvme0n1:/dev/sdd1 /mnt/whatever

since that shows me both devices in the output of "mount", reminding me
and anyone else trying to figure out what's going on with this machine
that both devices are involved in this filesystem.

Regards,

-Martin