Force recalculation of a data block checksum

Force recalculation of a data block checksum

[Tomasz Melcer]

Hi,

I've got a USB-connected HDD with a btrfs partition. The partition 
contains a 1TB file, a disk image. The first `btrfs scrub` after writing 
that file found 3 logical bad blocks that developed somewhere in the 
middle of that file (logs below).

The full area of the btrfs partition can be read without I/O error, so I 
think there are two possible cases: either the data block was written 
incorrectly or an incorrect checksum is stored. The first case is 
obviously unrecoverable, but if it's the second case, fixing the problem 
should be as simple as recomputing the checksum for what is already stored.

How can I ask btrfs to recompute the checksum of a data block as it is 
stored on the drive? I don't see any command doing an operation like 
that, and I couldn't find anything on the topic on the internet.

Thanks,


Logs:

#v+
[ 7702.964265] BTRFS warning (device sdd1): checksum error at logical
5473719291904 on dev /dev/sdd1, sector 222940168, root 5, inode 1245769,
offset 97110921216, length 4096, links 1 (path: dysk/dysk.bin)
[ 7702.964274] BTRFS error (device sdd1): bdev /dev/sdd1 errs: wr 0,
rd 0, flush 0, corrupt 17, gen 0
[ 7702.964278] BTRFS error (device sdd1): unable to fixup (regular) 
error at logical 5473719291904 on dev /dev/sdd1
[…]
[ 9588.625906] BTRFS warning (device sdd1): checksum error at logical
5241172611072 on dev /dev/sdd1, sector 691494312, root 5, inode 1245769,
offset 310632271872, length 4096, links 1 (path: dysk/dysk.bin)
[ 9588.625916] BTRFS error (device sdd1): bdev /dev/sdd1 errs: wr 0,
rd 0, flush 0, corrupt 18, gen 0
[ 9588.625920] BTRFS error (device sdd1): unable to fixup (regular) 
error at logical 5241172611072 on dev /dev/sdd1
[…]
[20545.302898] BTRFS warning (device sdd1): checksum error at logical
3991747321856 on dev /dev/sdd1, sector 3185809200, root 5, inode 
1245769, offset 907925676032, length 4096, links 1 (path: dysk/dysk.bin)
[20545.302908] BTRFS error (device sdd1): bdev /dev/sdd1 errs: wr 0,
rd 0, flush 0, corrupt 19, gen 0
[20545.302912] BTRFS error (device sdd1): unable to fixup (regular) 
error at logical 3991747321856 on dev /dev/sdd1
#v-


-- 
Tomasz Melcer

Re: Force recalculation of a data block checksum

[Chris Murphy]

On Sat, Jul 23, 2016 at 8:03 PM, Tomasz Melcer <liori@exroot.org> wrote:
> Hi,
>
> I've got a USB-connected HDD with a btrfs partition. The partition contains
> a 1TB file, a disk image. The first `btrfs scrub` after writing that file
> found 3 logical bad blocks that developed somewhere in the middle of that
> file (logs below).
>
> The full area of the btrfs partition can be read without I/O error, so I
> think there are two possible cases: either the data block was written
> incorrectly or an incorrect checksum is stored. The first case is obviously
> unrecoverable, but if it's the second case, fixing the problem should be as
> simple as recomputing the checksum for what is already stored.
>
> How can I ask btrfs to recompute the checksum of a data block as it is
> stored on the drive? I don't see any command doing an operation like that,
> and I couldn't find anything on the topic on the internet.

Since btrfs-progs 3.17 'btrfs check --init-csum-tree' will create a
whole new csum tree.



-- 
Chris Murphy

Re: Force recalculation of a data block checksum

[Duncan]

Tomasz Melcer posted on Sun, 24 Jul 2016 04:03:08 +0200 as excerpted:

> How can I ask btrfs to recompute the checksum of a data block as it is
> stored on the drive? I don't see any command doing an operation like
> that, and I couldn't find anything on the topic on the internet.

There's no "easy" btrfs method to recompute the checksum of an individual 
block (you'd have to do some deep debugging and hex editing to manually 
rewrite the correct sum), but there /is/ one that reinitiates and 
recalculates checksums for the entire filesystem.

You don't mention what version of btrfs-progs you have, but you'll want a 
reasonably new one, as AFAIK the command is fairly new, and early 
versions of the command simply wiped the existing checksums without 
recalculating them.

btrfs check --init-csum-tree

... as found in the btrfs-check manpage.


Another alternative is using btrfs restore, again, see the btrfs-restore 
manpage.  Normally this command is used to get what you can off of a 
filesystem damaged badly enough it won't mount, but you can also use it 
to scrape specific files off an unmounted filesystem, if the checksum's 
bad, for instance.  In this case, you'd use the --path-regex option to 
restore only files matching that regex.  Of course, for a TB-sized file, 
you'll need a separate filesystem (need not be btrfs, but will need to be 
a filesystem that can handle a file that size) with at least a TB free to 
write it to.

I'd normally prefer the btrfs restore method, myself, particularly if the 
btrfs in question has other files that you don't want to risk in the 
process.  If that was the only file on the filesystem, I'd still try the 
restore method first, at least if I had some place to write it to, 
because btrfs restore is read-only against the btrfs you're restoring 
from, and thus has no chance of causing further damage, while of course 
btrfs check --init-csum-tree writes to the filesystem in question and 
thus is a higher risk.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Force recalculation of a data block checksum

[Chris Murphy]

The negative of --init-csum-tree is that it's going to work on all
data extents. It's going to take a while. It's probably faster to
extract the suspect file with restore onto another file system and
check its integrity by other means (compare to backup, database
integrity check, etc).


-- 
Chris Murphy

Re: Force recalculation of a data block checksum

[Tomasz Melcer]

On 24.07.2016 04:16, Chris Murphy wrote:
> On Sat, Jul 23, 2016 at 8:03 PM, Tomasz Melcer <liori@exroot.org> wrote:
>> How can I ask btrfs to recompute the checksum of a data block as it is
>> stored on the drive?
>
> Since btrfs-progs 3.17 'btrfs check --init-csum-tree' will create a
> whole new csum tree.

I can't make this command recalculate only checksum of that single file?


-- 
Tomasz Melcer

Re: Force recalculation of a data block checksum

[Andrei Borzenkov]

24.07.2016 07:32, Tomasz Melcer пишет:
> On 24.07.2016 04:16, Chris Murphy wrote:
>> On Sat, Jul 23, 2016 at 8:03 PM, Tomasz Melcer <liori@exroot.org> wrote:
>>> How can I ask btrfs to recompute the checksum of a data block as it is
>>> stored on the drive?
>>
>> Since btrfs-progs 3.17 'btrfs check --init-csum-tree' will create a
>> whole new csum tree.
> 
> I can't make this command recalculate only checksum of that single file?
> 
> 

Rewriting single block in question should do it. As you can read from
raw device and have both physical and logical block address something like

dd if=/dev/sdd1 skip=222940168 count=8 | dd of=/path/to/dysk/dysk.bin
seek=189669768 conv=notrunc count=8

If you have possibility to verify file content(s), this technique can be
used to read over unreadable parts of file by combining multiple dd's
from file and device.

It probably makes sense to create (read-only) snapshot to make sure you
have old content available.

Re: Force recalculation of a data block checksum

[Chris Murphy]

On Sat, Jul 23, 2016 at 10:32 PM, Tomasz Melcer <liori@exroot.org> wrote:
> On 24.07.2016 04:16, Chris Murphy wrote:
>>
>> On Sat, Jul 23, 2016 at 8:03 PM, Tomasz Melcer <liori@exroot.org> wrote:
>>>
>>> How can I ask btrfs to recompute the checksum of a data block as it is
>>> stored on the drive?
>>
>>
>> Since btrfs-progs 3.17 'btrfs check --init-csum-tree' will create a
>> whole new csum tree.
>
>
> I can't make this command recalculate only checksum of that single file?

The likelihood of the data block being wrong is very high. If the csum
itself were wrong, the csum of the leaf it's stored in would catch it,
except in the narrow edge case where the csum was computed wrong in
memory, and then put into the leaf which is then checksummed
correctly, i.e. the leaf csum says the leaf is good but it contains a
bad csum for just one block. Possible but, pretty rare if there isn't
a problem causing in-memory corruption. You'd see a lot more errors. A
few bad sectors are exactly what this most likely is. The drive ECC is
probably failing to correct the problem, but thinks it's right which
why there's no read error from the drive.


-- 
Chris Murphy

Re: Force recalculation of a data block checksum

[Tomasz Melcer]

On 24.07.2016 07:07, Andrei Borzenkov wrote:
> Rewriting single block in question should do it. As you can read from
> raw device and have both physical and logical block address something like
>
> dd if=/dev/sdd1 skip=222940168 count=8 | dd of=/path/to/dysk/dysk.bin
> seek=189669768 conv=notrunc count=8

Thank you for the one-liner, I thought of doing it myself before, but 
when I tried to figure out which numbers mean what in scrub messages, I 
got a bit lost.

However, this one-liner doesn't seem to work here; `dd` prints:

#v+
8+0 records in
8+0 records out
4096 bytes (4.1 kB) copied, 5.2857e-05 s, 77.5 MB/s
dd: writing to ‘/media/liori/USB/dysk/dysk.bin’: Input/output error
1+0 records in
0+0 records out
0 bytes (0 B) copied, 0.00364222 s, 0.0 kB/s
#v-


I'm not sure what would this message mean in this case. Is btrfs reading 
the block before writing? Maybe some block blacklist stored somewhere (I 
see that the corruption counter in scrub messages go up even if it 
reports the same errors in subsequent runs)?


> If you have possibility to verify file content(s), this technique can be
> used to read over unreadable parts of file by combining multiple dd's
> from file and device.

Some of the files stored in this `dysk/dysk.bin` disk image have some 
internal checksums (zip archives, etc.). That includes the ones hit by 
the checksum error. I'll try verifying these later.



-- 
Tomasz Melcer