What about storing more crc32 in the unused csum size for metadata?

What about storing more crc32 in the unused csum size for metadata?

[Qu Wenruo]

[Old layout]
Btrfs csum size for tree block is 32 bytes, and currently, only the
first 4 bytes(sector 0) is used, storing the crc32 for the whole
leaf/node.

Takes 4K as example for leafsize.
Sectors:
|---0---|---1---|---2---|---3---|---4---|---5---|---6---|---7---|
Sector 0: Csum of the leaf/node (32 ~ 4K)
Sector 1~7: Not used

So, what about restoring more crc32 in other sectors in the following
backward-compatible way?

[New layout]
Take leafsize as 4K for example.

Sectors:
|---0---|---1---|---2---|---3---|---4---|---5---|---6---|---7---|
Sector 0: Csum of the leaf/node (32~4K)(not changed)
Sector 1: Csum of the first eighths of the leaf/node (32~512)
Sector 2: Csum of the second eighths of the leaf/node (512~1024)
....
Sector 7: Csum of the last eighths of the leaf/node (3584 ~ 4096)

And due to the fact
crc(sector 0 ~ sector 7) = crc(crc(sector 0), crc(sector 1), ...)

We won't waste too much CPU time, and since we keep the behavior of
sector 0, so it is completely backward compatible.

[Advantage]
The advantage usage of the new layout is more accuracy in scrub and
btrfsck.
For scrub, even all duplication is broken, there is still chance for
scrub to rebuild the block if corruptions occurs in different sector.

And for btrfsck --repair, if node is corrupted, we don't need to drop
all the nodes/leaves belong to the nodes, but only drops nodes/leaves
belong to the sector.

Re: What about storing more crc32 in the unused csum size for metadata?

[Duncan]

Qu Wenruo posted on Mon, 12 Jan 2015 10:32:30 +0800 as excerpted:

> [New layout]
> Take leafsize as 4K for example.
> 
> Sectors:
> |---0---|---1---|---2---|---3---|---4---|---5---|---6---|---7---|
> Sector 0: Csum of the leaf/node (32~4K)(not changed)
> Sector 1: Csum of the first eighths of the leaf/node (32~512)
> Sector 2: Csum of the second eighths of the leaf/node (512~1024)
> ....
> Sector 7: Csum of the last eighths of the leaf/node (3584 ~ 4096)

OK, I've been up too long and should be sleeping, and it may be that's 
affecting my logic, but AFAICT either you're missing something obvious, 
or I am.

There are eight eights, and eight "sectors", but the first one is already 
used, thus leaving seven available.  You have the first eighth (ending at 
byte 512*1) in sector one (zero-based so the second sector), the second 
(ending at byte 512*2=1024) in sector two... the last (eighth) eighth 
(ending at byte 512*8=4096) in sector seven.

Of eighths 3-7 in that ..., which eighth do you skip csumming, or is one 
of them a full quarter instead of an eighth?

Of course if we could make them sevenths...  But making it four quarters 
instead of 7/8 is perhaps more reasonable, with the remaining three 
sectors remaining unused, and if only one fails csum we still save 75% of 
the leaf/node, instead of none of it, currently.

But maybe it makes complete sense as-is and I should be sleeping instead 
of trying to make sense of what is clearly not making sense to me at this 
point...

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: What about storing more crc32 in the unused csum size for metadata?

[Qu Wenruo]

-------- Original Message --------
Subject: Re: What about storing more crc32 in the unused csum size for 
metadata?
From: Duncan <1i5t5.duncan@cox.net>
To: <linux-btrfs@vger.kernel.org>
Date: 2015年01月12日 12:52
> Qu Wenruo posted on Mon, 12 Jan 2015 10:32:30 +0800 as excerpted:
>
>> [New layout]
>> Take leafsize as 4K for example.
>>
>> Sectors:
>> |---0---|---1---|---2---|---3---|---4---|---5---|---6---|---7---|
>> Sector 0: Csum of the leaf/node (32~4K)(not changed)
>> Sector 1: Csum of the first eighths of the leaf/node (32~512)
>> Sector 2: Csum of the second eighths of the leaf/node (512~1024)
>> ....
>> Sector 7: Csum of the last eighths of the leaf/node (3584 ~ 4096)
> OK, I've been up too long and should be sleeping, and it may be that's
> affecting my logic, but AFAICT either you're missing something obvious,
> or I am.
>
> There are eight eights, and eight "sectors", but the first one is already
> used, thus leaving seven available.  You have the first eighth (ending at
> byte 512*1) in sector one (zero-based so the second sector), the second
> (ending at byte 512*2=1024) in sector two... the last (eighth) eighth
> (ending at byte 512*8=4096) in sector seven.
>
> Of eighths 3-7 in that ..., which eighth do you skip csumming, or is one
> of them a full quarter instead of an eighth?
Oh, my fault, sector 1 should contains csum for the second eighth of the 
leaf/node...
Since the first eighth csum can be calculated  vice verse.

Thanks,
Qu
>
> Of course if we could make them sevenths...  But making it four quarters
> instead of 7/8 is perhaps more reasonable, with the remaining three
> sectors remaining unused, and if only one fails csum we still save 75% of
> the leaf/node, instead of none of it, currently.
>
> But maybe it makes complete sense as-is and I should be sleeping instead
> of trying to make sense of what is clearly not making sense to me at this
> point...
>

Re: What about storing more crc32 in the unused csum size for metadata?

[Duncan]

Qu Wenruo posted on Mon, 12 Jan 2015 12:56:14 +0800 as excerpted:

> Oh, my fault, sector 1 should contains csum for the second eighth of the
> leaf/node...
> Since the first eighth csum can be calculated  vice verse.
> 
> Thanks,

Thanks indeed.

/Now/ it makes sense.  Skip the one and reverse-calculate it.  (Which you 
had actually implied and I guess it doesn't matter which one as long as 
it's /only/ one, but given my general unfamiliarity with crc at a 
practical level and impaired logic state, and your hiding it in the ... 
gap, I needed a bit more help to make the leap. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman