Re: [ANNOUNCE] bcachefs!

[Kent Overstreet <kent.overstreet@gmail.com>]

On Fri, Jul 17, 2015 at 04:58:17PM -0700, Ming Lin wrote:
> On Fri, 2015-07-17 at 16:51 -0700, Kent Overstreet wrote:
> > On Fri, Jul 17, 2015 at 04:48:31PM -0700, Ming Lin wrote:
> > > 
> > > On Fri, 2015-07-17 at 16:40 -0700, Kent Overstreet wrote:
> > > > On Fri, Jul 17, 2015 at 04:35:55PM -0700, Ming Lin wrote:
> > > > > 
> > > > > On Fri, 2015-07-17 at 16:17 -0700, Kent Overstreet wrote:
> > > > > > On Wed, Jul 15, 2015 at 12:39:36AM -0700, Ming Lin wrote:
> > > > > > > On Wed, Jul 15, 2015 at 12:15 AM, Ming Lin <mlin@kernel.org> wrote:
> > > > > > > > On Tue, 2015-07-14 at 23:58 -0700, Kent Overstreet wrote:
> > > > > > > >> Can you strace it?
> > > > > > > >
> > > > > > > > Strange. Now error message changed.
> > > > > > > 
> > > > > > > I mean sometimes it showed:
> > > > > > > 
> > > > > > > mount: /dev/sdt already mounted or /mnt/ busy
> > > > > > 
> > > > > > I have no idea what's going on, it works for me - is there anything unusual
> > > > > > about your setup? what kind of block device is /dev/sdt? is there any chance
> > > > > > there's another process that has it open? maybe try rebooting?
> > > > > 
> > > > > It's a regular HDD. I tried rebooting several times.
> > > > > 
> > > > > Now I try in qemu-kvm. Only the first time it can be mounted.
> > > > > 
> > > > > On host: qemu-img create hdd1.img 20G
> > > > > On guest: it's /dev/vda
> > > > > 
> > > > > root@block:~# bcacheadm format -C /dev/vda 
> > > > > UUID:			4730ed95-4c57-42db-856c-dbce36085625
> > > > > Set UUID:		e69ef0e0-0344-40d7-a6b1-c23d14745a32
> > > > > version:		6
> > > > > nbuckets:		40960
> > > > > block_size:		1
> > > > > bucket_size:		1024
> > > > > nr_in_set:		1
> > > > > nr_this_dev:		0
> > > > > first_bucket:		3
> > > > > 
> > > > > root@block:~# mount -t bcache /dev/vda /mnt/
> > > > > 
> > > > > root@block:~# mount |grep bcache
> > > > > /dev/vda on /mnt type bcache (rw,relatime)
> > > > > 
> > > > > root@block:~# reboot
> > > > > 
> > > > > root@block:~# dmesg |grep -i bcache
> > > > > [    2.548754] bcache: bch_journal_replay() journal replay done, 1 keys in 1 entries, seq 3
> > > > > [    2.636217] bcache: register_cache() registered cache device vda
> > > > > 
> > > > > 
> > > > > root@block:~# mount -t bcache /dev/vda /mnt/
> > > > > mount: No such file or directory
> > > > > 
> > > > > Now dmesg shows:
> > > > > 
> > > > > bcache: bch_open_as_blockdevs() register_cache_set err device already registered
> > > > 
> > > > Ohhhh.
> > > > 
> > > > The cache set is getting registered by the udev hooks. We should be able to
> > > > mount it anyways - same as you can mount any other fs in multiple locations.
> > > > 
> > > > I won't be able to fix this for at least a couple days, but for now - just
> > > > shut it down it via sysfs (echo 1 > /sys/fs/bcache/<uuid>/stop), then mount it.
> > > 
> > > It works!
> > > Any hint how to fix it? On udev or bcache-tool or kernel?
> > > I'd like to fix it.
> > 
> > The relevant code is in drivers/md/bcache/fs.c, bch_mount() ->
> > bch_open_as_blockdevs().
> > 
> > Part of the problem is that bcachefs isn't able to use much of the normal
> > generic mount path for block devices, partly because a fs can span multiple
> > block devices (same as btrfs).
> > 
> > I'm not sure the right way to fix it - it's going to take some thought, but
> > we want to do something like "is it already open? just take a ref on the
> > existing cache set".
> 
> I'll look into it.

Thanks

> And, echo 1 > /sys/fs/bcache/<uuid>/stop, got below.
> I'll also try to fix it.
> 
> [   25.826280] ======================================================
> [   25.828038] [ INFO: possible circular locking dependency detected ]
> [   25.828587] 4.1.0-00943-g3683e624 #7 Not tainted

I think the correct fix is to change cache_set_flush() to not hold register_lock
while it's calling into the sysfs code. Want to do that, and add a comment so it
doesn't get screwed up again? Also try and make sure we don't actually need
register_lock for what you take out from under it.

BTW - probably the most valuable thing you could help out with is the
documentation, in particular the guide:
http://bcache.evilpiepirate.org/BcacheGuide/

Can you read through (at least some of) that, and tell me what's useful and what
needs clarifying? And tell me what you'd like to see added to the guide next -
I'll try and work on documentation over the next two weeks, since I probably
won't be able to do much real coding with my test machines offline.