Re: runtime btrfsck

[Hugo Mills <hugo@carfax.org.uk>]

[-- Attachment #1: Type: text/plain, Size: 7051 bytes --]

On Wed, May 10, 2017 at 11:20:58AM +0200, Stefan Priebe - Profihost AG wrote:
> Hello,
> 
> here's the output:
> # for block in 163316514816 163322413056 163325722624; do echo $block;
> btrfs-debug-tree -b $block /dev/mapper/crypt_md0|sed -re 's/(\t| )name:
> .*/\1name: HIDDEN/'; done
> 
> 163316514816
> btrfs-progs v4.8.5
> leaf 163316514816 items 188 free space 1387 generation 86739 owner 3892
> fs uuid 37b15dd8-b2e1-4585-98d0-cc0fa2a5a7c9
> chunk uuid b86efe94-ab40-4344-ac6b-46ec59c41b8f
[...]
>         item 37 key (23760 DIR_INDEX 36) itemoff 14278 itemsize 58
>                 location key (28124232 INODE_ITEM 0) type FILE
>                 transid 86739 data_len 0 name_len 28
>                 name: HIDDEN
>         item 38 key (23760 DIR_INDEX 37) itemoff 14220 itemsize 58
>                 location key (28124233 INODE_ITEM 0) type FILE
>                 transid 86739 data_len 0 name_len 28
>                 name: HIDDEN
>         item 39 key (23760 DIR_INDEX 38) itemoff 14165 itemsize 55
>                 location key (28124234 INODE_ITEM 0) type FILE
>                 transid 86739 data_len 0 name_len 25
>                 name: HIDDEN
>         item 40 key (23760 DIR_INDEX 22) itemoff 14115 itemsize 50
>                 location key (26923383 INODE_ITEM 0) type FILE
>                 transid 74009 data_len 0 name_len 20
>                 name: HIDDEN
>         item 41 key (23760 DIR_INDEX 23) itemoff 14067 itemsize 48
>                 location key (26923384 INODE_ITEM 0) type FILE
>                 transid 74009 data_len 0 name_len 18
>                 name: HIDDEN
[...]
> 163322413056
> btrfs-progs v4.8.5
> leaf 163322413056 items 113 free space 934 generation 86739 owner 3892
> fs uuid 37b15dd8-b2e1-4585-98d0-cc0fa2a5a7c9
> chunk uuid b86efe94-ab40-4344-ac6b-46ec59c41b8f
[...]
>         item 73 key (24016 DIR_INDEX 19) itemoff 9651 itemsize 62
>                 location key (28124251 INODE_ITEM 0) type FILE
>                 transid 86739 data_len 0 name_len 32
>                 name: HIDDEN
>         item 74 key (24016 DIR_INDEX 20) itemoff 9592 itemsize 59
>                 location key (28124252 INODE_ITEM 0) type FILE
>                 transid 86739 data_len 0 name_len 29
>                 name: HIDDEN
>         item 75 key (24016 DIR_INDEX 4) itemoff 9538 itemsize 54
>                 location key (26923401 INODE_ITEM 0) type FILE
>                 transid 74009 data_len 0 name_len 24
>                 name: HIDDEN
>         item 76 key (24016 DIR_INDEX 5) itemoff 9486 itemsize 52
>                 location key (26923402 INODE_ITEM 0) type FILE
>                 transid 74009 data_len 0 name_len 22
>                 name: HIDDEN
[...]
> 163325722624
> btrfs-progs v4.8.5
> leaf 163325722624 items 78 free space 6563 generation 86739 owner 3892
> fs uuid 37b15dd8-b2e1-4585-98d0-cc0fa2a5a7c9
> chunk uuid b86efe94-ab40-4344-ac6b-46ec59c41b8f
[...]
>         item 62 key (24189 DIR_INDEX 16) itemoff 9409 itemsize 64
>                 location key (28124267 INODE_ITEM 0) type FILE
>                 transid 86739 data_len 0 name_len 34
>                 name: HIDDEN
>         item 63 key (24189 DIR_INDEX 17) itemoff 9349 itemsize 60
>                 location key (28124268 INODE_ITEM 0) type FILE
>                 transid 86739 data_len 0 name_len 30
>                 name: HIDDEN
>         item 64 key (24189 DIR_INDEX 4) itemoff 9296 itemsize 53
>                 location key (26923421 INODE_ITEM 0) type FILE
>                 transid 74010 data_len 0 name_len 23
>                 name: HIDDEN
>         item 65 key (24189 DIR_INDEX 5) itemoff 9236 itemsize 60
>                 location key (26923422 INODE_ITEM 0) type FILE
>                 transid 74010 data_len 0 name_len 30
>                 name: HIDDEN
[...]

   In each case, the tree node keys have simply been sorted
incorrectly -- the ordering is otherwise correct, but jumps backwards
at some point in the sequence. At least in the first instance, some of
the keys appear to have been duplicated: there are two (23760
DIR_INDEX 22) keys in the list. (I didn't check in detail with the
other two whether there are duplicates, but I wouldn't be surprised).

   I note that the jump in the keys in the first two cases is 16, and
the jump in the third case is 13. That _might_ suggest some kind of
bit-level error, but it's probably not in the RAM that was used to
store the blocks, as the error appears in a different place in each
block.

   I'm tentatively going to point the finger at your hardware for
this. It's probably going to need something really long and/or
stressful to pick it up, though (one of the CPU stress tests, for
example, and also a good long run with a RAM tester -- 24 hours or
longer).

   Hugo.

> Stefan
> Am 10.05.2017 um 11:08 schrieb Hugo Mills:
> > On Wed, May 10, 2017 at 10:40:31AM +0200, Stefan Priebe - Profihost AG wrote:
> >> Am 10.05.2017 um 09:40 schrieb Hugo Mills:
> >>> On Wed, May 10, 2017 at 09:36:30AM +0200, Stefan Priebe - Profihost AG wrote:
> >>>> Hello Roman,
> >>>>
> >>>> the FS is mountable. It just goes readonly when trying to write some data.
> >>>>
> >>>> The kernel msgs are:
> >>>> BTRFS critical (device dm-2): corrupt leaf, bad key order:
> >>>> block=163316514816,root=1, slot=39
> >>>> BTRFS critical (device dm-2): corrupt leaf, bad key order:
> >>>> block=163322413056,root=1, slot=74
> >>>> BTRFS critical (device dm-2): corrupt leaf, bad key order:
> >>>> block=163325722624,root=1, slot=63
> >>>> BTRFS critical (device dm-2): corrupt leaf, bad key order:
> >>>> block=163316514816,root=1, slot=39
> >>>> BTRFS: error (device dm-2) in btrfs_drop_snapshot:8839: errno=-5 IO failure
> >>>> BTRFS info (device dm-2): forced readonly
> >>>> BTRFS info (device dm-2): delayed_refs has NO entry
> >>>
> >>>    Can you show the output of btrfs-debug-tree -b <blocknum>, where
> >>> <blocknum> is each of the three "block=" values above?
> >>
> >> Can do that. But the lists are very long - should i upload them to
> >> pastebin? Is it ok to remove the name atribute - which provides filenames?
> > 
> >    On the mailing list would be preferable, as then the conversation
> > is completely preserved in archives. They shouldn't be so long that
> > vger rejects them. And yes, if you want to filter out the filenames,
> > do so (but _just_ the filename -- don't remove the whole line).
> > 
> >    Hugo.
> > 
> >> Stefan
> >>
> >>
> >>>    Hugo.
> >>>
> >>>> Greets,
> >>>> Stefan
> >>>> Am 10.05.2017 um 09:18 schrieb Roman Mamedov:
> >>>>> On Wed, 10 May 2017 09:02:46 +0200
> >>>>> Stefan Priebe - Profihost AG <s.priebe@profihost.ag> wrote:
> >>>>>
> >>>>>> how to fix bad key ordering?
> >>>>>
> >>>>> You should clarify does the FS in question mount (read-write? read-only?)
> >>>>> and what are the kernel messages if it does not.
> >>>>>
> >>>
> > 

-- 
Hugo Mills             | Alert status mauve ocelot: Slight chance of
hugo@... carfax.org.uk | brimstone. Be prepared to make a nice cup of tea.
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]