[bcachefs] problem with `bcache fsck`

[bcachefs] problem with `bcache fsck`

[Marcin]

Hi!
It's good to hear that silence on mailinglist doesn't mean that there is 
no changes in developing bcachefs!

I found some problems with bcache tools.
bcache fsck errors:

a) if I creat4e filesystem and run fsck before first mount I'm getting:
# bcache fsck /dev/sdd1
initializing new filesystem
error opening /dev/sdd1: cannot allocate new btree root

b) After first mount%umount fsck still doesn't work:
# bcache fsck /dev/sdd1
no journal entries found
filesystem contains errors, but repair impossible
error opening /dev/sdd1: fsck error

bcachefs-tools at commit d4e6736e59c96

Is bcache fsck able to do fsck on tiered devices?
I know that there is some stability problem with tiered devices because 
when I mounted tiered fs and started to copy files on it the host 
rebooted after some time. I have to run netconsole to catch what kernel 
says, then I post it on ML.

And I'm still missing compression saves free space on disk:)

Marcin

Re: [bcachefs] problem with `bcache fsck`

[Kent Overstreet]

On Thu, Mar 16, 2017 at 04:16:29PM +0100, Marcin wrote:
> Hi!
> It's good to hear that silence on mailinglist doesn't mean that there is no
> changes in developing bcachefs!

Join the irc channel! It's pretty active.

> I found some problems with bcache tools.
> bcache fsck errors:
> 
> a) if I creat4e filesystem and run fsck before first mount I'm getting:
> # bcache fsck /dev/sdd1
> initializing new filesystem
> error opening /dev/sdd1: cannot allocate new btree root
>
> b) After first mount%umount fsck still doesn't work:
> # bcache fsck /dev/sdd1
> no journal entries found
> filesystem contains errors, but repair impossible
> error opening /dev/sdd1: fsck error

Man, you have a talent for finding bugs :) I've reproduced it, working on it
now.

> Is bcache fsck able to do fsck on tiered devices?

Yes - just pass it all the devices, e.g.
bcache fsck /dev/sdb /dev/sdc

Also, keep in mind fsck is still run by default at mount time, there's normally
no need to use userspace fsck (and running fsck in userspace and then mounting
will be slower than just mounting and running fsck then, due to mount still
having to scan all the metadata which will be cache cold).

> I know that there is some stability problem with tiered devices because when
> I mounted tiered fs and started to copy files on it the host rebooted after
> some time. I have to run netconsole to catch what kernel says, then I post
> it on ML.

You might have been hitting a bug in a recent patch to improve readpages
performance - I backed it out last night after I discovered it.

> And I'm still missing compression saves free space on disk:)

Yeah, that's something I've been wanting to get to, but people have been
badgering me for replication more :)

Re: [bcachefs] problem with `bcache fsck`

[Marcin]

W dniu 2017-03-16 21:34, Kent Overstreet napisał(a):
> On Thu, Mar 16, 2017 at 04:16:29PM +0100, Marcin wrote:
>> Hi!
>> It's good to hear that silence on mailinglist doesn't mean that there 
>> is no
>> changes in developing bcachefs!
> 
> Join the irc channel! It's pretty active.

Done.


>> Is bcache fsck able to do fsck on tiered devices?
> 
> Yes - just pass it all the devices, e.g.
> bcache fsck /dev/sdb /dev/sdc
> 
> Also, keep in mind fsck is still run by default at mount time, there's 
> normally
> no need to use userspace fsck (and running fsck in userspace and then 
> mounting
> will be slower than just mounting and running fsck then, due to mount 
> still
> having to scan all the metadata which will be cache cold).

If I add bcachefs to fstab openrc expects userland fsck to works this is 
why I tried it to use.

>> I know that there is some stability problem with tiered devices 
>> because when
>> I mounted tiered fs and started to copy files on it the host rebooted 
>> after
>> some time. I have to run netconsole to catch what kernel says, then I 
>> post
>> it on ML.
> 
> You might have been hitting a bug in a recent patch to improve 
> readpages
> performance - I backed it out last night after I discovered it.

There is no new commit, I'm at
commit 7274700edcf99ab67d4f72eca4f07d0220b9abae
Author: Kent Overstreet <kent.overstreet@gmail.com>
Date:   Tue Mar 14 21:07:39 2017 -0800

     bcache: fix device hot add


I suspect that problem is due to using  --metadata_replicas=2. If you 
have a some spare time please try fileystsem formated with this 
parameters:
# bcache format --compression_type=gzip --metadata_replicas=2 
--data_checksum_type=crc64 --metadata_checksum_type=crc64 --tier=2 
/dev/sde1 --tier=3 /dev/sdd1



>> And I'm still missing compression saves free space on disk:)
> 
> Yeah, that's something I've been wanting to get to, but people have 
> been
> badgering me for replication more :)

You have implemented too many features, everyone find something usefull 
for self and want to have it. You should implement only writing and 
reading to/from files;)

Btw:
# curl https://evilpiepirate.org/git/bcache-tools.git/
curl: (51) SSL: no alternative certificate subject name matches target 
host name 'evilpiepirate.org'


Marcin

Re: [bcachefs] problem with `bcache fsck`

[Kent Overstreet]

On Thu, Mar 16, 2017 at 11:07:20PM +0100, Marcin wrote:
> If I add bcachefs to fstab openrc expects userland fsck to works this is why
> I tried it to use.

Fsck should be fixed now

> There is no new commit, I'm at
> commit 7274700edcf99ab67d4f72eca4f07d0220b9abae
> Author: Kent Overstreet <kent.overstreet@gmail.com>
> Date:   Tue Mar 14 21:07:39 2017 -0800
> 
>     bcache: fix device hot add

Ok, you're good then (at least re: the buffered reads bug)

> I suspect that problem is due to using  --metadata_replicas=2. If you have a
> some spare time please try fileystsem formated with this parameters:
> # bcache format --compression_type=gzip --metadata_replicas=2
> --data_checksum_type=crc64 --metadata_checksum_type=crc64 --tier=2 /dev/sde1
> --tier=3 /dev/sdd1

Alright, will do

> You have implemented too many features, everyone find something usefull for
> self and want to have it. You should implement only writing and reading
> to/from files;)

Heh :)

> Btw:
> # curl https://evilpiepirate.org/git/bcache-tools.git/
> curl: (51) SSL: no alternative certificate subject name matches target host
> name 'evilpiepirate.org'

Works for me now - I was updating SSL certs and some other stuff earlier
though, I might have broken it temporarily then

Re: [bcachefs] problem with `bcache fsck`

[Marcin]

W dniu 2017-03-17 00:57, Kent Overstreet napisał(a):
> On Thu, Mar 16, 2017 at 11:07:20PM +0100, Marcin wrote:
>> If I add bcachefs to fstab openrc expects userland fsck to works this 
>> is why
>> I tried it to use.
> 
> Fsck should be fixed now

Thanks, but when I pass only one device from two used in tiered fs 
`bcache fsck` becomes upset:
# bcache fsck /dev/sdd1
bcache: libbcache/buckets.h:87: gen_after: Assertion `!(r > 128U)' 
failed.

Also it happened I got segfault while using fsck. I don't have coredump, 
I'll try to catch it.

Marcin

Re: [bcachefs] problem with `bcache fsck`

[Marcin]

W dniu 2017-03-17 09:57, Marcin napisał(a):
> W dniu 2017-03-17 00:57, Kent Overstreet napisał(a):
>> On Thu, Mar 16, 2017 at 11:07:20PM +0100, Marcin wrote:
>>> If I add bcachefs to fstab openrc expects userland fsck to works this 
>>> is why
>>> I tried it to use.
>> 
>> Fsck should be fixed now
> 
> Thanks, but when I pass only one device from two used in tiered fs
> `bcache fsck` becomes upset:
> # bcache fsck /dev/sdd1
> bcache: libbcache/buckets.h:87: gen_after: Assertion `!(r > 128U)' 
> failed.
> 
> Also it happened I got segfault while using fsck. I don't have
> coredump, I'll try to catch it.

Here it is (about assertion):

# bcache format --compression_type=gzip --metadata_replicas=2 
--data_checksum_type=crc64 --metadata_checksum_type=crc64 --tier=2 
/dev/sde1 --tier=3 /dev/sdd1 >/dev/null &&  mount -t bcache 
/dev/sdd1:/dev/sde1 /mnt/test/ &&  umount /mnt/test &&  bcache fsck 
/dev/sde1
bcache: libbcache/buckets.h:87: gen_after: Assertion `!(r > 128U)' 
failed.
Przerwane (core dumped)


  # echo bt full | gdb -q /sbin/bcache 
/tmp/'core-1489788240-0-6-!sbin!bcache-3330'                             
                            [40/996]
Reading symbols from /sbin/bcache...Reading symbols from 
/usr/lib64/debug//sbin/bcache.debug...done.
done.
[New LWP 3330]
[New LWP 3338]
[New LWP 3339]
[New LWP 3332]
[New LWP 3333]
[New LWP 3335]
[New LWP 3336]
[New LWP 3337]
[New LWP 3331]
[New LWP 3334]

warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `bcache fsck /dev/sde1'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at 
../sysdeps/unix/sysv/linux/raise.c:54
54      ../sysdeps/unix/sysv/linux/raise.c: Nie ma takiego pliku ani 
katalogu.
[Current thread is 1 (Thread 0x7f0e547fb900 (LWP 3330))]
(gdb) #0  __GI_raise (sig=sig@entry=6) at 
../sysdeps/unix/sysv/linux/raise.c:54                                    
                                   [17/996]
         resultvar = 0
         pid = 3330
         selftid = 3330
#1  0x00007f0e52ca365a in __GI_abort () at abort.c:89
         save_stage = 2
         act = {__sigaction_handler = {sa_handler = 0x4, sa_sigaction = 
0x4}, sa_mask = {__val = {139699521867039, 255, 140733776075696, 
623191333, 0, 0, 0,
               21474836480, 139699496478336, 140733776075848, 
140733776075648, 139699496492088, 139699496505728, 0, 139699524009984, 
4587397}},
           sa_flags = 87, sa_restorer = 0x467e18 
<__PRETTY_FUNCTION__.13545>}
         sigs = {__val = {32, 0 <repeats 15 times>}}
#2  0x00007f0e52c9b197 in __assert_fail_base (fmt=<optimized out>, 
assertion=assertion@entry=0x45ff85 "!(r > 128U)",
     file=file@entry=0x45ff71 "libbcache/buckets.h", line=line@entry=87, 
function=function@entry=0x467e18 <__PRETTY_FUNCTION__.13545> 
"gen_after")
     at assert.c:92
         str = 0xe95170 ""
         total = 4096
#3  0x00007f0e52c9b242 in __GI___assert_fail 
(assertion=assertion@entry=0x45ff85 "!(r > 128U)", 
file=file@entry=0x45ff71 "libbcache/buckets.h",
     line=line@entry=87, function=function@entry=0x467e18 
<__PRETTY_FUNCTION__.13545> "gen_after") at assert.c:101
No locals.
#4  0x0000000000403164 in gen_after (b=<optimized out>, a=<optimized 
out>) at libbcache/buckets.h:87
No locals.
#5  0x00000000004299cd in gen_after (b=<optimized out>, a=<optimized 
out>) at libbcache/extents.c:629
         r = <optimized out>
#6  ptr_stale (ptr=0xe61e40, ca=<optimized out>) at 
libbcache/buckets.h:99
No locals.
#7  bch_btree_pick_ptr (c=c@entry=0xe22010, b=b@entry=0xe61e10) at 
libbcache/extents.c:606
         _ret = <optimized out>
         ca = <optimized out>
         root = <optimized out>
         e = {{{k = 0xe61e18, v = <optimized out>}, s_c = {k = 0xe61e18, 
v = <optimized out>}}}
         crc = <optimized out>
         ptr = 0xe61e40
         pick = {crc = {type = 0, _compressed_size = 0, 
_uncompressed_size = 0, offset = 0, nonce = 0, csum_type = 0, 
compression_type = 0, csum = {lo = 0,
               hi = 0}}, ptr = {type = 0, cached = 0, erasure_coded = 0, 
reservation = 0, offset = 0, dev = 0, gen = 0}, ca = 0x0}
#8  0x000000000043124c in bch_btree_node_read (c=c@entry=0xe22010, 
b=b@entry=0xe61e10) at libbcache/btree_io.c:1196
         cl = {{{wq = 0x0, s = 0x0, list = {next = 0x0}, fn = 0x0}, work 
= {data = {counter = 0}, entry = {next = 0x0, prev = 0x0}, func = 0x0}},
           parent = 0x0, remaining = {counter = -2147483647}}
         bio = <optimized out>
         pick = {crc = {type = 7, _compressed_size = 7110, 
_uncompressed_size = 2570, offset = 435, nonce = 0, csum_type = 0, 
compression_type = 0, csum = {
               lo = 4386366, hi = 467}}, ptr = {type = 0, cached = 0, 
erasure_coded = 1, reservation = 1, offset = 25916528, dev = 0, gen = 
0},
           ca = 0x7f0e4dcfc000}
#9  0x00000000004363bc in bch_btree_root_read (c=c@entry=0xe22010, 
id=id@entry=BTREE_ID_EXTENTS, k=<optimized out>, level=0) at 
libbcache/btree_io.c:1254
         cl = {{{wq = 0x0, s = 0x0, list = {next = 0x0}, fn = 0x0}, work 
= {data = {counter = 0}, entry = {next = 0x0, prev = 0x0}, func = 0x0}},
           parent = 0x0, remaining = {counter = -2147483647}}
         b = 0xe61e10
         ret = 0
         __PRETTY_FUNCTION__ = "bch_btree_root_read"
#10 0x00000000004498b7 in __bch_fs_start (c=c@entry=0xe22010) at 
libbcache/super.c:807
         level = <optimized out>
         err = 0x46114b "error reading btree root"
         i = <optimized out>
         id = 0
         now = <optimized out>
         journal = {next = 0x7f0e40000990, prev = 0x7f0e40000990}
         j = 0x7f0e400009a0
---Type <return> to continue, or q <return> to quit---        ret = 0
         __PRETTY_FUNCTION__ = "__bch_fs_start"
#11 0x000000000044aa92 in bch_fs_open (devices=<optimized out>, 
nr_devices=nr_devices@entry=1, opts=..., ret=ret@entry=0x7fff22bb2170)
     at libbcache/super.c:1797
         err = 0x461249 "insufficient devices"
         c = 0xe22010
         sb = 0xe20850
         i = 1
         best_sb = <optimized out>
#12 0x000000000045246e in cmd_fsck (argc=<optimized out>, 
argv=argv@entry=0x7fff22bb22d8) at cmd_fsck.c:55
         opts = {read_only = -1 '\377', nostart = -1 '\377', errors = -1 
'\377', metadata_replicas = -1 '\377', data_replicas = -1 '\377',
           metadata_replicas_required = -1 '\377', data_replicas_required 
= -1 '\377', metadata_checksum = -1 '\377', data_checksum = -1 '\377',
           compression = -1 '\377', str_hash = -1 '\377', inodes_32bit = 
-1 '\377', gc_reserve_percent = -1 '\377', root_reserve_percent = -1 
'\377',
           wide_macs = -1 '\377', verbose_recovery = -1 '\377', posix_acl 
= -1 '\377', journal_flush_disabled = -1 '\377', nofsck = -1 '\377',
           fix_errors = -1 '\377', nochanges = -1 '\377', noreplay = -1 
'\377', norecovery = -1 '\377', noexcl = -1 '\377', sb = -1}
         c = 0x0
         err = <optimized out>
         opt = <optimized out>
#13 0x0000000000403554 in main (argc=2, argv=0x7fff22bb22d8) at 
bcache.c:134
         cmd = 0x7fff22bb451d "fsck"
(gdb) quit

Re: [bcachefs] problem with `bcache fsck`

[Marcin]

W dniu 2017-03-17 00:57, Kent Overstreet napisał(a):
> On Thu, Mar 16, 2017 at 11:07:20PM +0100, Marcin wrote:
>> If I add bcachefs to fstab openrc expects userland fsck to works this 
>> is why
>> I tried it to use.
> 
> Fsck should be fixed now
> 
>> There is no new commit, I'm at
>> commit 7274700edcf99ab67d4f72eca4f07d0220b9abae
>> Author: Kent Overstreet <kent.overstreet@gmail.com>
>> Date:   Tue Mar 14 21:07:39 2017 -0800
>> 
>>     bcache: fix device hot add
> 
> Ok, you're good then (at least re: the buffered reads bug)
> 
>> I suspect that problem is due to using  --metadata_replicas=2. If you 
>> have a
>> some spare time please try fileystsem formated with this parameters:
>> # bcache format --compression_type=gzip --metadata_replicas=2
>> --data_checksum_type=crc64 --metadata_checksum_type=crc64 --tier=2 
>> /dev/sde1
>> --tier=3 /dev/sdd1
> 
> Alright, will do

I configured logging using netconsole, I formated fs using favorite 
killer-combo[1], when I copied ~480000 rather small files the host 
rebooted. I only cought this:

Mar 17 23:17:00 [ 1247.409119] kernel BUG at 
drivers/md/bcache/buckets.c:505!
Mar 17 23:17:00 [ 1247.409149] invalid opcode: 0000 [#1] SMP
Mar 17 23:17:00 [ 1247.409176] Modules linked in:
Mar 17 23:17:00 netconsole
Mar 17 23:17:00 tun
[snip list of modules]

Marcin

[1] bcache format --compression_type=gzip --metadata_replicas=2 
--data_checksum_type=crc64 --metadata_checksum_type=crc64 --tier=2 
/dev/sde1 --tier=3 /dev/sdd1