Re: [PATCH v4 00/13] Add support for other checksums

[David Sterba <dsterba@suse.cz>]

On Mon, Jun 03, 2019 at 04:58:46PM +0200, Johannes Thumshirn wrote:
> This patchset add support for adding new checksum types in BTRFS.

V4 looks good to me, with a few minor fixups added to topic branch,
including the sha256 patch.  As noted this may not be merged and now
servers for the testing purposes.

> Currently BTRFS only supports CRC32C as data and metadata checksum, which is
> good if you only want to detect errors due to data corruption in hardware.
> 
> But CRC32C isn't able cover other use-cases like de-duplication or
> cryptographically save data integrity guarantees.
> 
> The following properties made SHA-256 interesting for these use-cases:
> - Still considered cryptographically sound
> - Reasonably well understood by the security industry
> - Result fits into the 32Byte/256Bit we have for the checksum in the on-disk
>   format
> - Small enough collision space to make it feasible for data de-duplication
> - Fast enough to calculate and offloadable to crypto hardware via the kernel's
>   crypto_shash framework.

Regarding hw offload, David pointed out that the ahash API would need to
be used and that turned out to be infeasible with current btrfs code. I
think the only hw-based improvements left are based on CPU instructions
(crc32c, SSE, AVX) but that's sufficient.

I also think software implementations of the checksum(s) are going to be
used in most cases, which kind of makes SHA-3 less appealing to us as
it's main point was 'excellent efficiency in hardware implementations'
(quoting NIST announcement [1]).

As has been suggested, BLAKE2 is for consideration, we only need the
kernel module which I'll provide for testing purposes. And the more I
know about it, the more I like it so we might have a winner, but the
selection is still open.

> The patchset also provides mechanisms for plumbing in different hash
> algorithms relatively easy.
> 
> This is an intermediate submission, as a) mkfs.btrfs support is still missing
> and

We'll need that one, briefly checking the progs souces, the same
cleanups will be needed there too.

> b) David requested to have three hash algorithms, where 1 is crc32c, one
> cryptographically secure and one in between.

Let me summarize the current satus:

for strong hash we have SHA256 and BLAKE2. For the fast hash xxhash and
murmur3 have been suggested. Let me add XXH3 and xxh128 for now (they're
not finalized yet).