Re: Mount issue, mount /dev/sdc2: can't read superblock

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

[-- Attachment #1.1: Type: text/plain, Size: 12994 bytes --]



On 2018/12/24 下午8:48, Tomáš Metelka wrote:
> Hi Qu,
> 
> just 1 curious question (maybe 2) about your statement "log_root is 0":
> 
> What does it mean when log_root is non-zero?

This means there are some dirty log, namely caused by fsync().

You could consider log tree as some kind of journal used in ext/xfs.

Btrfs doesn't rely on log tree to keep its metadata consistent, but uses
it as a faster way to implement fsync().

For fs with dirty log, btrfs itself should be consistent no matter if we
replay the log or not.


The only certain thing a non-zero log tree shows is, there is definitely
an unexpected powerloss happened.
(But not vice verse, it's completely possible to hit a unexpected
powerloss without a dirty log, either it's lucky that no fsync() called
during that trans, or notreelog mount option is used)

> Because I have similar
> problem (unmountable FS ... I don't know how much but I know there's
> corrupted 2 subsequent items in chunk tree node)

Then the problem is not related to log root.
But either chunk tree get corrupted or metadata cow get exploited.

> and when I have made
> "btrfs inspect-internal dump-super":
> 
> superblock: bytenr=65536, device=/dev/sda4
> ...
> generation        2488742
> root            232408301568
> sys_array_size        97
> chunk_root_generation    2487902
> root_level        1
> chunk_root        242098421760
> chunk_root_level    1
> log_root        232433811456

log_root is only recorded in the primary super block.
So it's fine that your backup super block doesn't contain log root.

It's the designed behavior.

> log_root_transid    0
> log_root_level        0
> 
> superblock: bytenr=67108864, device=/dev/sda4
> ...
> generation        2488742
> root            232408301568
> sys_array_size        97
> chunk_root_generation    2487902
> root_level        1
> chunk_root        242098421760
> chunk_root_level    1
> log_root        0
> log_root_transid    0
> log_root_level        0
> 
> Unfortunately when I try to do "btrfs rescue chunk-recover" I get
> (beside others):
> 
> "...
> 
> Unrecoverable Chunks:
>   Chunk: start = 0, len = 4194304, type = 2, num_stripes = 1
>       Stripes list:
>       [ 0] Stripe: devid = 1, offset = 0
>       No block group.
>       No device extent.
> 
> Total Chunks:        184
>   Recoverable:        183
>   Unrecoverable:    1
> 
> Orphan Block Groups:
> 
> Orphan Device Extents:
> 
> Chunk tree recovery failed
> "
> 
> And when I try "btrfs restore -m -S -v -i -D <dev>" I get only:
> Could not open root, trying backup super
> Could not open root, trying backup super
> ERROR: superblock bytenr 274877906944 is larger than device size
> 212000047104
> Could not open root, trying backup super
> 
> Is it possible to recover data (at least some of them)? And is it worth
> to upgrade to newest btrfs-progs?

btrfs check --readonly output please.

btrfs check --readonly is always the most reliable and detailed output
for any possible recovery.

Also kernel message for the mount failure could help.

btrfs ins dump-tree/super is only useful when we have some ideas to verify.

Thanks,
Qu

> 
> uname -a:
> Linux tisc5 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018
> x86_64 x86_64 x86_64 GNU/Linux
> 
> btrfs-progs v4.15.1
> 
> Thanks
> Metaliza
> 
> 
> On 24. 12. 18 13:02, Qu Wenruo wrote:
>>
>>
>> On 2018/12/24 下午7:31, Peter Chant wrote:
>>> On 12/24/18 12:58 AM, Chris Murphy wrote:
>>>> On Sat, Dec 22, 2018 at 10:22 AM Peter Chant <pete@petezilla.co.uk>
>>>> wrote:
>>>>
>>>>> btrfs rescue super -v /dev/sdb2
>>>> ...
>>>>> All supers are valid, no need to recover
>>>>>
>>>>>
>>>>> btrfs insp dump-s -f <dev>
>>>> ...
>>>>> generation              7937947
>>>> ...
>>>>>          backup 0:
>>>>>                  backup_tree_root:       1113909100544   gen:
>>>>> 7937935    level: 1
>>>> ...
>>>>>          backup 1:
>>>>>                  backup_tree_root:       1113907347456   gen:
>>>>> 7937936    level: 1
>>>> ...
>>>>>          backup 2:
>>>>>                  backup_tree_root:       1113911951360   gen:
>>>>> 7937937    level: 1
>>>> ...
>>>>>          backup 3:
>>>>>                  backup_tree_root:       1113907494912   gen:
>>>>> 7937934    level: 1
>>>> ...
>>>>
>>>>
>>>> The kernel wrote out three valid checksummed supers, with what seems
>>>> to be a rather significant sanity violation. The super generation and
>>>> tree root address do not match any of the backup tree roots. The
>>>> *current* tree root is supposed to be in one of the backups as well.
>>>>
>>>
>>> I wonder if this is a result of my trying to fix things?  E.g. btrfs
>>> rescue super-recover or my attempts using the tools (and kernel) in Mint
>>> 18.1 at one point?
>>
>> At least super-recover is not responsible for this.
>> While btrfs check --repair could indeed cause problems.
>>
>> So it may be the case.
>>
>>>
>>> I must admit, early on I had assumed that either this file system was a
>>> simple fix or was completely trashed, so I thought I'd have a quick go
>>> at fixing it, or wipe it and start again.  But then I seemed to get
>>> close with only the one error, but unmountable.
>>>
>>>
>>>> Qu, any idea how this is even theoretically possible? Bit flip right
>>>> before the super is computed and checksummed? Seems like some kind of
>>>> corruption before checksum is computed.
>>>>
>>>>
>>>>> I'm getting suspicious of the drive as when I was trying the various
>>>>> btrfs rescue * tools I saw a 'bad block', or similar, error displayed.
>>>>> I also have a separate basic install on ext4 on the same disk.  Though
>>>>> e2fsck shows no errors and mounts fine I cannot log into that install.
>>>>> Maybe a coincidence, but too many bad things thrown up make me
>>>>> suspicious.  Whatever is happening this seems to be really fighting
>>>>> me.
>>>>
>>>> I'm not sure how even a bad device accounts for the super generation
>>>> and backup mismatches. That's damn strange.
>>>
>>> I'm less suspicious of the drive now.  I've been using an ext4 partition
>>> on the same drive for a few days now, having reinstalled on that and
>>> everything _seems_ fine.  Mind you, apart from usb sticks, I've not
>>> experienced a ssd failure.  Perhaps my hdd failure experience is not
>>> relevent, i.e. they work until they start throwing errors and then
>>> rapidly fail?
>>
>> I don't really believe a drive can be so easily corrupted to certain
>> bits while all other bits are OK.
>>
>>>
>>>
>>>>
>>>> If you get bored with the back and forth and just want to give up,
>>>> that's fine. I suggest that if you have the time and space, to take a
>>>> btrfs-image in case Qu or some other developer wants to look at this
>>>> file system at some point. The btrfs-image is a read only process, can
>>>> be set to scrub filenames, and only contains metadata. Size of the
>>>> resulting file is around 1/2 of the size of metadata, when doing
>>>> 'btrfs filesystem usage' or 'btrfs filesystem df'. So you'll need that
>>>> much free space to direct the command to.
>>>>
>>>> btrfs-image -ss -c9 -t4 <devicetoimage> pathtofile
>>>
>>> Just done that:
>>> bash-4.3# btrfs-image -ss -c9 -t4 /dev/sdd2
>>> /mnt/backup/btrfs_issue_dec_2018/btrfs_root_image_error_20181224.img
>>> WARNING: cannot find a hash collision for '..', generating garbage, it
>>> won't match indexes
>>>
>>>
>>>
>>>>
>>>> It might fail, if so you can try adding -w and see if that helps.
>>>
>>>
>>> OK, try with -w:
>>>
>>> OK, many many complaints about hash collisions:
>>> ...
>>> ARNING: cannot find a hash collision for 'ifup', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'catv', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'FDPC', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'LIBS', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'INTC', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'SPI', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'PDCA', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'EBI', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'SMC', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'WIFI', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'LWIP', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'HID', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'yun', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'avr4', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'avr6', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'WiFi', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'TFT', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'Knob', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'FP.h', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'SD.h', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'Beep', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'FORK', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'CHM', generating garbage, it
>>> won't match indexes
>>> WARNING: cannot find a hash collision for 'HandS', generating garbage,
>>> it won't match indexes
>>> WARNING: cannot find a hash collision for 'dm-0', generating garbage, it
>>> won't match indexes
>>>
>>>
>>> Now seems to stopped producing output.  Can't see if it is doing
>>> something useful.  (note, started again, more such messages)
>>
>> I don't know about other developers, normally I don't like btrfs-image
>> -ss at all.
>>
>> Even plain btrfs-image isn't so helpful, especially considering its size.
>>
>> Anyway, from all the data you collected, I suspect it's a corruption in
>> tree blocks allocation, maybe a btrfs bug in older kernels, which buried
>> a dangerous seed into the fs, breaking the metadata CoW.
>>
>> And one day, an unexpected powerloss makes the seed grow and screw up
>> the fs.
>>
>> Just a personal recommendation, for btrfs especially used with older
>> kernels, after a powerloss, it's highly recommended to run btrfs check
>> --readonly before mounting it.
>>
>> Thanks,
>> Qu
>>
>>>
>>>
>>>>
>>>> There is no log listed in the super so zero-log isn't indicated, and
>>>> also tells me there were no fsync's still flushing at the time of the
>>>> crash. The loss should be at most a minute of data, not an
>>>> inconsistent file system that can't be mounted anymore. Pretty weird.
>>>>
>>>
>>> I think I ran zero-log to see if that helped.  Given that there was no
>>> important data and I'd assume I'd either easily fix it, or wipe it and
>>> start over I may have taken the 'monkey radomly pounding the buttons'
>>> approach, short of 'btrfs check --repair'.  I only posted here as I
>>> though I'd fixed it apart from the one error!  If it were a simple fix
>>> then it was worth asking.
>>>
>>>
>>>> What were your mount options? Defaults? Anything custom like discard,
>>>> commit=, notreelog? Any non-default mount options themselves would not
>>>> be the cause of the problem, but might suggest partial ideas for what
>>>> might have happened.
>>>>
>>> fstab states:
>>> autodefrag,ssd,discard,noatime,defaults,subvol=_r_sl14.
>>> 2,compress=lzo
>>>
>>> However, I used an initrd, so I'm not sure if that is correct?
>>>
>>> Ok, digging into init within my initrd, the line where the root partion
>>> is mounted:
>>>    mount -o ro -t $ROOTFS $ROOTDEV /mnt
>>>
>>> Where $ROOTFS is:
>>> btrfs -o subvol=_r_sl14.2
>>>
>>> and $ROOTDEV is:
>>> /dev/disk/by-uuid/6496aabd-d6aa-49e0-96ca-e49c316edd8e
>>>
>>>
>>>
>>> Pete
>>>
>>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]