bcachefs: can bcachefs export block devices?

bcachefs: can bcachefs export block devices?

[Eric Wheeler]

I have a few bcachefs questions that pertain to its use with block 
devices.  

Does bcachefs's implementation reuse and update the existing 
bcache code such that the block device driver inherits the bcachefs 
improvements?  I understand the cache superblock changed, maybe the cached 
dev super too.

Can bcachefs provide /dev/bcacheN devices without loop.ko?  

If so, are these simply filesystem objects (files)?


--
Eric Wheeler

Re: bcachefs: can bcachefs export block devices?

[Kent Overstreet]

On Wed, May 25, 2016 at 02:47:29PM -0700, Eric Wheeler wrote:
> 
> I have a few bcachefs questions that pertain to its use with block 
> devices.  
> 
> Does bcachefs's implementation reuse and update the existing 
> bcache code such that the block device driver inherits the bcachefs 
> improvements?  I understand the cache superblock changed, maybe the cached 
> dev super too.

Yes, all of the existing functionality is still there (though some of it's
broken at the moment because I haven't been running those tests; if you're
interested in using bcache-dev for the old style caching (there are performance
and robustness improvements) it wouldn't take me long to get it working again).

> Can bcachefs provide /dev/bcacheN devices without loop.ko?  
> 
> If so, are these simply filesystem objects (files)?

No, at least not currently - the "export a block device" code and the filesystem
code are effectively thin wrappers around the core bcache IO path (bch_read()
and bch_write()) - but the two different interfaces don't have anything to do
with each other.

The way it works is the first 4096 inode numbers are owned by the block device
interface - inodes in that range are for either cached devices or thin
provisioned volumes. The filesystem code owns inode numbers >= 4096.

So while blockdev volumes/cached data do have inodes, they're not reachable via
the filesystem because there will never be dirents that point to them (also,
they use a different inode type with extra fields for the UUID/label).

However - there isn't anything preventing us from writing a bit of new code and
hooking it up to an ioctl or sysfs interface to say "look up this path and
create a block device for that file". The only remotely tricky bit would be
pagecache cache coherency, but I think you get the new block device to just use
the same address_space (pagecache cache for an inode) as the filesystem inode.

So, probably doable in ~100 lines of code or so.

Re: bcachefs: can bcachefs export block devices?

[Eric Wheeler]

> On Wed, May 25, 2016 at 02:47:29PM -0700, Eric Wheeler wrote:
> > Does bcachefs's implementation reuse and update the existing 
> > bcache code such that the block device driver inherits the bcachefs 
> > improvements?  I understand the cache superblock changed, maybe the cached 
> > dev super too.
> 
> Yes, all of the existing functionality is still there (though some of it's
> broken at the moment because I haven't been running those tests; if you're
> interested in using bcache-dev for the old style caching (there are performance
> and robustness improvements) it wouldn't take me long to get it working again).

I can test that once its working.  Would it use the same bcachefs tools 
for formatting superblocks?

Relatedly, can you point out the best place to abstract cachemeta-v1 vs. 
cachemeta-v2 for simultaneous use?  Could it be just a bunch of function 
pointers in the cachedev struct and assignment during initialization for 
v1/v2?  Have the call arguments changed? What functions would need 
abstractions (the smallest v1/v2 intersection)?

> > Can bcachefs provide /dev/bcacheN devices without loop.ko?  
> > 
> > If so, are these simply filesystem objects (files)?
> 
> The way it works is the first 4096 inode numbers are owned by the block device
> interface - inodes in that range are for either cached devices or thin
> provisioned volumes. The filesystem code owns inode numbers >= 4096.
> 
> So while blockdev volumes/cached data do have inodes, they're not reachable via
> the filesystem because there will never be dirents that point to them (also,
> they use a different inode type with extra fields for the UUID/label).

Thats a neat implementation.  Would creating a dirent for such an inode 
expose the block device with the same size and content (and ordering) if 
if the inode were compatable?  Would the blockdev be block-size aligned 
versus the file or might the file have an alignment requirement?

I'm particularly excited about this as a precursor to snapshot support, 
especially if udev could help produce something like this:

  /dev/disk/by-path/bcache-mydiskfile -> /dev/bcacheN
  /dev/disk/by-path/bcache-mydisksnap -> /dev/bcacheN+1


--
Eric Wheeler

On Wed, 25 May 2016, Kent Overstreet wrote:

> 
> However - there isn't anything preventing us from writing a bit of new code and
> hooking it up to an ioctl or sysfs interface to say "look up this path and
> create a block device for that file". The only remotely tricky bit would be
> pagecache cache coherency, but I think you get the new block device to just use
> the same address_space (pagecache cache for an inode) as the filesystem inode.
> 
> So, probably doable in ~100 lines of code or so.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: bcachefs: can bcachefs export block devices?

[Kent Overstreet]

On Fri, May 27, 2016 at 07:45:32PM -0700, Eric Wheeler wrote:
> > On Wed, May 25, 2016 at 02:47:29PM -0700, Eric Wheeler wrote:
> > > Does bcachefs's implementation reuse and update the existing 
> > > bcache code such that the block device driver inherits the bcachefs 
> > > improvements?  I understand the cache superblock changed, maybe the cached 
> > > dev super too.
> > 
> > Yes, all of the existing functionality is still there (though some of it's
> > broken at the moment because I haven't been running those tests; if you're
> > interested in using bcache-dev for the old style caching (there are performance
> > and robustness improvements) it wouldn't take me long to get it working again).
> 
> I can test that once its working.  Would it use the same bcachefs tools 
> for formatting superblocks?
> 
> Relatedly, can you point out the best place to abstract cachemeta-v1 vs. 
> cachemeta-v2 for simultaneous use?  Could it be just a bunch of function 
> pointers in the cachedev struct and assignment during initialization for 
> v1/v2?  Have the call arguments changed? What functions would need 
> abstractions (the smallest v1/v2 intersection)?

You mean compile a kernel that supports both old and new on disk format?

Realistically the only way that's going to happen is to completely fork the
source code, ext2/3/4 style.

Although that's going to have to happen eventually.

> > > Can bcachefs provide /dev/bcacheN devices without loop.ko?  
> > > 
> > > If so, are these simply filesystem objects (files)?
> > 
> > The way it works is the first 4096 inode numbers are owned by the block device
> > interface - inodes in that range are for either cached devices or thin
> > provisioned volumes. The filesystem code owns inode numbers >= 4096.
> > 
> > So while blockdev volumes/cached data do have inodes, they're not reachable via
> > the filesystem because there will never be dirents that point to them (also,
> > they use a different inode type with extra fields for the UUID/label).
> 
> Thats a neat implementation.  Would creating a dirent for such an inode 
> expose the block device with the same size and content (and ordering) if 
> if the inode were compatable?  Would the blockdev be block-size aligned 
> versus the file or might the file have an alignment requirement?

What we'd want to do is add an ioctl or something to take a fs inode (a normal
file, that already has a dirent) and create at runtime a block device for it.

> I'm particularly excited about this as a precursor to snapshot support, 
> especially if udev could help produce something like this:
> 
>   /dev/disk/by-path/bcache-mydiskfile -> /dev/bcacheN
>   /dev/disk/by-path/bcache-mydisksnap -> /dev/bcacheN+1

Not sure what you mean by precursor - that would still require essentially the
entire snapshots implementation. But yes, once we have snapshots we could do
that too.

Re: bcachefs: can bcachefs export block devices?

[Eric Wheeler]

On Wed, 3 Aug 2016, Kent Overstreet wrote:

> On Fri, May 27, 2016 at 07:45:32PM -0700, Eric Wheeler wrote:
> > > On Wed, May 25, 2016 at 02:47:29PM -0700, Eric Wheeler wrote:
> > > > Does bcachefs's implementation reuse and update the existing 
> > > > bcache code such that the block device driver inherits the bcachefs 
> > > > improvements?  I understand the cache superblock changed, maybe the cached 
> > > > dev super too.
> > > 
> > > Yes, all of the existing functionality is still there (though some of it's
> > > broken at the moment because I haven't been running those tests; if you're
> > > interested in using bcache-dev for the old style caching (there are performance
> > > and robustness improvements) it wouldn't take me long to get it working again).
> > 
> > I can test that once its working.  Would it use the same bcachefs tools 
> > for formatting superblocks?
> > 
> > Relatedly, can you point out the best place to abstract cachemeta-v1 vs. 
> > cachemeta-v2 for simultaneous use?  Could it be just a bunch of function 
> > pointers in the cachedev struct and assignment during initialization for 
> > v1/v2?  Have the call arguments changed? What functions would need 
> > abstractions (the smallest v1/v2 intersection)?
> 
> You mean compile a kernel that supports both old and new on disk format?
> 
> Realistically the only way that's going to happen is to completely fork the
> source code, ext2/3/4 style.
> Although that's going to have to happen eventually.

Sure, that makes sense.  At what point would you want to do that rename so 
bcache-dev can be pulled into the kernel tree?
	
> > > > Can bcachefs provide /dev/bcacheN devices without loop.ko?  
> > > > 
> > > > If so, are these simply filesystem objects (files)?
> > > 
> > > The way it works is the first 4096 inode numbers are owned by the block device
> > > interface - inodes in that range are for either cached devices or thin
> > > provisioned volumes. The filesystem code owns inode numbers >= 4096.
> > > 
> > > So while blockdev volumes/cached data do have inodes, they're not reachable via
> > > the filesystem because there will never be dirents that point to them (also,
> > > they use a different inode type with extra fields for the UUID/label).
> > 
> > Thats a neat implementation.  Would creating a dirent for such an inode 
> > expose the block device with the same size and content (and ordering) if 
> > if the inode were compatable?  Would the blockdev be block-size aligned 
> > versus the file or might the file have an alignment requirement?
> 
> What we'd want to do is add an ioctl or something to take a fs inode (a normal
> file, that already has a dirent) and create at runtime a block device for it.

You had mentioned changing on-disk format related to this and NFS support.  
Is that coming along too?

> > I'm particularly excited about this as a precursor to snapshot support, 
> > especially if udev could help produce something like this:
> > 
> >   /dev/disk/by-path/bcache-mydiskfile -> /dev/bcacheN
> >   /dev/disk/by-path/bcache-mydisksnap -> /dev/bcacheN+1
> 
> Not sure what you mean by precursor - that would still require essentially the
> entire snapshots implementation. But yes, once we have snapshots we could do
> that too.

Precursor, as in, export an arbitrary file as a blockdev even if snapshots 
aren't ready yet.  I can start testing in our testbed once files can be 
exported as blocks, whether or not they support snapshots.

Other questions:

Is FIEMAP supported so uncached fils can be read in disk-linear order?  
Hmm, I wonder, what does FIEMAP even mean when the file spreads across 
multiple disks?  Maybe it doesn't apply here.  Really what I'm looking for 
is a way to list which blocks have changed between two snapshots for easy 
incremental backups (eg, `btrfs send`).


I'm excited about checksum support.  If an SSD bitflips, will it fail the 
whole disk, or just report an error and attempt to re-read from another 
volume?  

Right now btrfs/zfs is the only viable checksum filesystem with recovery, 
and there aren't any viable blockdevice checksumming implementations 
(dm-csum didn't take off and the PoC academic example splicing into md 
raid isn't really ready either).


--
Eric Wheeler

Re: bcachefs: can bcachefs export block devices?

[Kent Overstreet]

On Thu, Aug 04, 2016 at 04:46:58PM -0700, Eric Wheeler wrote:
> On Wed, 3 Aug 2016, Kent Overstreet wrote:
> > On Fri, May 27, 2016 at 07:45:32PM -0700, Eric Wheeler wrote:
> >
> > Realistically the only way that's going to happen is to completely fork the
> > source code, ext2/3/4 style.
> > Although that's going to have to happen eventually.
> 
> Sure, that makes sense.  At what point would you want to do that rename so 
> bcache-dev can be pulled into the kernel tree?

Probably not until after the on disk format is stabilized for good, which isn't
going to be until after snapshots are at least minimally working.

> > > > The way it works is the first 4096 inode numbers are owned by the block device
> > > > interface - inodes in that range are for either cached devices or thin
> > > > provisioned volumes. The filesystem code owns inode numbers >= 4096.
> > > > 
> > > > So while blockdev volumes/cached data do have inodes, they're not reachable via
> > > > the filesystem because there will never be dirents that point to them (also,
> > > > they use a different inode type with extra fields for the UUID/label).
> > > 
> > > Thats a neat implementation.  Would creating a dirent for such an inode 
> > > expose the block device with the same size and content (and ordering) if 
> > > if the inode were compatable?  Would the blockdev be block-size aligned 
> > > versus the file or might the file have an alignment requirement?
> > 
> > What we'd want to do is add an ioctl or something to take a fs inode (a normal
> > file, that already has a dirent) and create at runtime a block device for it.
> 
> You had mentioned changing on-disk format related to this and NFS support.  
> Is that coming along too?

Yeah, the transactions stuff I wrote about on Patreon is for NFS support. I
think I'll be able to do NFS support without an on disk format change, but I
will need an on disk format change at some point.

> 
> > > I'm particularly excited about this as a precursor to snapshot support, 
> > > especially if udev could help produce something like this:
> > > 
> > >   /dev/disk/by-path/bcache-mydiskfile -> /dev/bcacheN
> > >   /dev/disk/by-path/bcache-mydisksnap -> /dev/bcacheN+1
> > 
> > Not sure what you mean by precursor - that would still require essentially the
> > entire snapshots implementation. But yes, once we have snapshots we could do
> > that too.
> 
> Precursor, as in, export an arbitrary file as a blockdev even if snapshots 
> aren't ready yet.  I can start testing in our testbed once files can be 
> exported as blocks, whether or not they support snapshots.

I may have asked this before, but is loopback really not good enough? I thought
performance of the loopback driver had improved recently (I know Christoph was
working on this).

> Other questions:
> 
> Is FIEMAP supported so uncached fils can be read in disk-linear order?  
> Hmm, I wonder, what does FIEMAP even mean when the file spreads across 
> multiple disks?  Maybe it doesn't apply here.  Really what I'm looking for 
> is a way to list which blocks have changed between two snapshots for easy 
> incremental backups (eg, `btrfs send`).

Yeah, fiemap works. I don't remember if there's any provisions in fiemap for
multiple devices.

We'll have real send/receive at some point though - we've got a version number
field in struct bkey, so we'll have a "send all keys greater than version number
foo".

> I'm excited about checksum support.  If an SSD bitflips, will it fail the 
> whole disk, or just report an error and attempt to re-read from another 
> volume?  

It'll fail that individual IO, and whether or not it fails the entire device
should be configurable (don't believe it is yet).

It should reread from another replica if available but I'm not sure if that's
done yet - I haven't looked at what's left with replication in quite awhile.

> Right now btrfs/zfs is the only viable checksum filesystem with recovery, 
> and there aren't any viable blockdevice checksumming implementations 
> (dm-csum didn't take off and the PoC academic example splicing into md 
> raid isn't really ready either).

Maybe I'll start working more on the replication stuff, it would be nice to get
that stuff finished off... Are you chipping in on Patreon? :)

Re: bcachefs: can bcachefs export block devices?

[Eric Wheeler]

On Fri, 5 Aug 2016, Kent Overstreet wrote:

> On Thu, Aug 04, 2016 at 04:46:58PM -0700, Eric Wheeler wrote:
> > On Wed, 3 Aug 2016, Kent Overstreet wrote:
> > > On Fri, May 27, 2016 at 07:45:32PM -0700, Eric Wheeler wrote:
> > >
> > > Realistically the only way that's going to happen is to completely fork the
> > > source code, ext2/3/4 style.
> > > Although that's going to have to happen eventually.
> > 
> > Sure, that makes sense.  At what point would you want to do that rename so 
> > bcache-dev can be pulled into the kernel tree?
> 
> Probably not until after the on disk format is stabilized for good, which isn't
> going to be until after snapshots are at least minimally working.

Hi Kent, 

How is the on-disk format stability these days related to NFS and 
snapshots and blockdev export?

more below
 
> > > > > The way it works is the first 4096 inode numbers are owned by the block device
> > > > > interface - inodes in that range are for either cached devices or thin
> > > > > provisioned volumes. The filesystem code owns inode numbers >= 4096.
> > > > > 
> > > > > So while blockdev volumes/cached data do have inodes, they're not reachable via
> > > > > the filesystem because there will never be dirents that point to them (also,
> > > > > they use a different inode type with extra fields for the UUID/label).
> > > > 
> > > > Thats a neat implementation.  Would creating a dirent for such an inode 
> > > > expose the block device with the same size and content (and ordering) if 
> > > > if the inode were compatable?  Would the blockdev be block-size aligned 
> > > > versus the file or might the file have an alignment requirement?
> > > 
> > > What we'd want to do is add an ioctl or something to take a fs inode (a normal
> > > file, that already has a dirent) and create at runtime a block device for it.
> > 
> > You had mentioned changing on-disk format related to this and NFS support.  
> > Is that coming along too?
> 
> Yeah, the transactions stuff I wrote about on Patreon is for NFS support. I
> think I'll be able to do NFS support without an on disk format change, but I
> will need an on disk format change at some point.
> 
> > 
> > > > I'm particularly excited about this as a precursor to snapshot support, 
> > > > especially if udev could help produce something like this:
> > > > 
> > > >   /dev/disk/by-path/bcache-mydiskfile -> /dev/bcacheN
> > > >   /dev/disk/by-path/bcache-mydisksnap -> /dev/bcacheN+1
> > > 
> > > Not sure what you mean by precursor - that would still require essentially the
> > > entire snapshots implementation. But yes, once we have snapshots we could do
> > > that too.
> > 
> > Precursor, as in, export an arbitrary file as a blockdev even if snapshots 
> > aren't ready yet.  I can start testing in our testbed once files can be 
> > exported as blocks, whether or not they support snapshots.
> 
> I may have asked this before, but is loopback really not good enough? I thought
> performance of the loopback driver had improved recently (I know Christoph was
> working on this).

Well, I don't know about "good enough" but I can say that the current 
bcache v1 asynchronous block IO performance is wonderful.  It seems to me 
that loopN's are confined to a single thread and incur pagecache 
duplication (maybe better with DIO loops, but that was always very slow in 
my testing).  

So, if its easy enough to get bcachefs to export blockdevs then it would 
be neat to benchmark bcachefs snapshots vs dm-thin volumes.  Compression 
is definitely a win here feature-wise.  This would make bcachefs the first 
compressed block device without loop in Linux when it gets merged!

> > Other questions:
> > 
> > Is FIEMAP supported so uncached fils can be read in disk-linear order?  
> > Hmm, I wonder, what does FIEMAP even mean when the file spreads across 
> > multiple disks?  Maybe it doesn't apply here.  Really what I'm looking for 
> > is a way to list which blocks have changed between two snapshots for easy 
> > incremental backups (eg, `btrfs send`).
> 
> Yeah, fiemap works. I don't remember if there's any provisions in fiemap for
> multiple devices.
> 
> We'll have real send/receive at some point though - we've got a version number
> field in struct bkey, so we'll have a "send all keys greater than version number
> foo".
> 
> > I'm excited about checksum support.  If an SSD bitflips, will it fail the 
> > whole disk, or just report an error and attempt to re-read from another 
> > volume?  
> 
> It'll fail that individual IO, and whether or not it fails the entire device
> should be configurable (don't believe it is yet).
> 
> It should reread from another replica if available but I'm not sure if that's
> done yet - I haven't looked at what's left with replication in quite awhile.

Any news on multi-disk replication?
 
> > Right now btrfs/zfs is the only viable checksum filesystem with recovery, 
> > and there aren't any viable blockdevice checksumming implementations 
> > (dm-csum didn't take off and the PoC academic example splicing into md 
> > raid isn't really ready either).
> 
> Maybe I'll start working more on the replication stuff, it would be nice to get
> that stuff finished off... 

Indeed :)

> Are you chipping in on Patreon? :)

Indeed!  I'm looking forward to your next post.

--
Eric Wheeler

> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: bcachefs: can bcachefs export block devices?

[Martin Raiber]

[-- Attachment #1: Type: text/plain, Size: 917 bytes --]

Hi,

On 09.02.2017 23:17 Eric Wheeler wrote:
> Well, I don't know about "good enough" but I can say that the current 
> bcache v1 asynchronous block IO performance is wonderful.  It seems to me 
> that loopN's are confined to a single thread and incur pagecache 
> duplication (maybe better with DIO loops, but that was always very slow in 
> my testing).  
>
> So, if its easy enough to get bcachefs to export blockdevs then it would 
> be neat to benchmark bcachefs snapshots vs dm-thin volumes.  Compression 
> is definitely a win here feature-wise.  This would make bcachefs the first 
> compressed block device without loop in Linux when it gets merged!
Did you test this with kernel >=4.4 and losetup --direct-io=on ? This
makes loop use direct, async IO which is much faster and doesn't double
cache. So it might indeed be a distraction to pull this into bcachefs.

Regards,
Martin Raiber


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3826 bytes --]

Re: bcachefs: can bcachefs export block devices?

[Eric Wheeler]

On Wed, 25 May 2016, Kent Overstreet wrote:

> On Wed, May 25, 2016 at 02:47:29PM -0700, Eric Wheeler wrote:
> > 
> > I have a few bcachefs questions that pertain to its use with block 
> > devices.  
> > 
> > Does bcachefs's implementation reuse and update the existing 
> > bcache code such that the block device driver inherits the bcachefs 
> > improvements?  I understand the cache superblock changed, maybe the cached 
> > dev super too.
> 
> Yes, all of the existing functionality is still there (though some of it's
> broken at the moment because I haven't been running those tests; if you're
> interested in using bcache-dev for the old style caching (there are performance
> and robustness improvements) it wouldn't take me long to get it working again).
> 
> > Can bcachefs provide /dev/bcacheN devices without loop.ko?  
> > 
> > If so, are these simply filesystem objects (files)?
> 
> No, at least not currently - the "export a block device" code and the filesystem
> code are effectively thin wrappers around the core bcache IO path (bch_read()
> and bch_write()) - but the two different interfaces don't have anything to do
> with each other.
> 
> The way it works is the first 4096 inode numbers are owned by the block device
> interface - inodes in that range are for either cached devices or thin
> provisioned volumes. The filesystem code owns inode numbers >= 4096.
> 
> So while blockdev volumes/cached data do have inodes, they're not reachable via
> the filesystem because there will never be dirents that point to them (also,
> they use a different inode type with extra fields for the UUID/label).
> 
> However - there isn't anything preventing us from writing a bit of new code and
> hooking it up to an ioctl or sysfs interface to say "look up this path and
> create a block device for that file". The only remotely tricky bit would be
> pagecache cache coherency, but I think you get the new block device to just use
> the same address_space (pagecache cache for an inode) as the filesystem inode.
> 
> So, probably doable in ~100 lines of code or so.

Hey Kent,

I noticed that blockdev.c got an overhaul in bcache-dev recently.  Getting 
closer to blockdev export from bcachefs?

Actually it looks like you've been pretty busy on that branch in general.  

What's new?  


--
Eric Wheeler



> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: bcachefs: can bcachefs export block devices?

[Kent Overstreet]

On Wed, Aug 03, 2016 at 05:30:02PM -0700, Eric Wheeler wrote:
> Hey Kent,
> 
> I noticed that blockdev.c got an overhaul in bcache-dev recently.  Getting 
> closer to blockdev export from bcachefs?

No, not really. I've actually hardly touched that stuff - to the extent that I
have I've just been fixing a few random bugs here and there to get the old style
bcache functionality closer to working again.

> Actually it looks like you've been pretty busy on that branch in general.  
> 
> What's new?  

Oh, random stuff. I've been posting some on patreon lately:
https://www.patreon.com/bcachefs

(And yes, I do need money if anyone would be willing to chip in).

The main user visible thing is compression is finally useful - I finally
finished the disk accounting stuff so that enabling compression actually lets
you store more data on a fs.

Besides that, it's mainly been bug fixing and internal refactoring working
towards snapshots. I'm gonna write something up about the plan for snapshots
soon.