[PATCH v2 00/11] btrfs-progs: support xxhash64 checksums

[PATCH v2 00/11] btrfs-progs: support xxhash64 checksums

[Johannes Thumshirn]

Now that Nikolay's XXHASH64 support for the Crypto API has landed and BTRFS is
prepared for an easy addition of new checksums, this patchset implements
XXHASH64 as a second, fast but not cryptographically secure checksum hash.

For changes since v1, please see the individual patches. Additionally a unit
test was added for regression testing this series.

David Sterba (3):
  btrfs-progs: update checksumming api
  btrfs-progs: add xxhash sources
  btrfs-progs: add xxhash64 as checksum algorithm

Johannes Thumshirn (8):
  btrfs-progs: don't blindly assume crc32c in csum_tree_block_size()
  btrfs-progs: cache csum_type in recover_control
  btrfs-progs: add checksum type to checksumming functions
  btrfs-progs: don't assume checksums are always 4 bytes
  btrfs-progs: pass checksum type to
    btrfs_csum_data()/btrfs_csum_final()
  btrfs-progs: simplify update_block_csum() in btrfs-sb-mod.c
  btrfs-progs: add option for checksum type to mkfs
  btrfs-progs: add test-case for mkfs with xxhash64

 Makefile                                    |    3 +-
 btrfs-corrupt-block.c                       |    3 +-
 btrfs-sb-mod.c                              |   30 +-
 check/main.c                                |   20 +-
 cmds/inspect-dump-super.c                   |   37 +-
 cmds/rescue-chunk-recover.c                 |   23 +-
 convert/common.c                            |   14 +-
 convert/main.c                              |    3 +-
 crypto/hash.c                               |   16 +
 crypto/hash.h                               |   10 +
 crypto/xxhash.c                             | 1024 +++++++++++++++++++++++++++
 crypto/xxhash.h                             |  445 ++++++++++++
 ctree.h                                     |   18 +-
 disk-io.c                                   |   80 ++-
 disk-io.h                                   |    8 +-
 file-item.c                                 |   11 +-
 free-space-cache.c                          |    2 +-
 image/main.c                                |    7 +-
 mkfs/common.c                               |   23 +-
 mkfs/common.h                               |    2 +
 mkfs/main.c                                 |   27 +-
 tests/mkfs-tests/001-basic-profiles/test.sh |    2 +
 22 files changed, 1686 insertions(+), 122 deletions(-)
 create mode 100644 crypto/hash.c
 create mode 100644 crypto/hash.h
 create mode 100644 crypto/xxhash.c
 create mode 100644 crypto/xxhash.h

-- 
2.16.4

[PATCH v2 01/11] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size()

[Johannes Thumshirn]

The callers of csum_tree_block_size() blindly assume we're only having
crc32c as a possible checksum and thus pass in
btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32] for the size argument of
csum_tree_block_size().

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
---
 mkfs/common.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/mkfs/common.c b/mkfs/common.c
index caca5e707233..b6e549b19272 100644
--- a/mkfs/common.c
+++ b/mkfs/common.c
@@ -101,7 +101,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
 	}
 
 	/* generate checksum */
-	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
 
 	/* write back root tree */
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_ROOT_TREE]);
@@ -292,7 +292,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, nritems);
-	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]);
 	if (ret != cfg->nodesize) {
 		ret = (ret < 0 ? -errno : -EIO);
@@ -380,7 +380,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, nritems);
-	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]);
 	if (ret != cfg->nodesize) {
 		ret = (ret < 0 ? -errno : -EIO);
@@ -420,7 +420,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, nritems);
-	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]);
 	if (ret != cfg->nodesize) {
 		ret = (ret < 0 ? -errno : -EIO);
@@ -433,7 +433,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, 0);
-	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]);
 	if (ret != cfg->nodesize) {
 		ret = (ret < 0 ? -errno : -EIO);
@@ -445,7 +445,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, 0);
-	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]);
 	if (ret != cfg->nodesize) {
 		ret = (ret < 0 ? -errno : -EIO);
@@ -456,7 +456,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE);
 	memcpy(buf->data, &super, sizeof(super));
 	buf->len = BTRFS_SUPER_INFO_SIZE;
-	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
 	ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
 			cfg->blocks[MKFS_SUPER_BLOCK]);
 	if (ret != BTRFS_SUPER_INFO_SIZE) {
-- 
2.16.4

Re: [PATCH v2 01/11] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size()

[David Sterba]

On Mon, Aug 26, 2019 at 01:48:43PM +0200, Johannes Thumshirn wrote:
> The callers of csum_tree_block_size() blindly assume we're only having
> crc32c as a possible checksum and thus pass in
> btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32] for the size argument of
> csum_tree_block_size().
> 
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> Reviewed-by: Nikolay Borisov <nborisov@suse.com>
> ---
>  mkfs/common.c | 14 +++++++-------
>  1 file changed, 7 insertions(+), 7 deletions(-)
> 
> diff --git a/mkfs/common.c b/mkfs/common.c
> index caca5e707233..b6e549b19272 100644
> --- a/mkfs/common.c
> +++ b/mkfs/common.c
> @@ -101,7 +101,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
>  	}
>  
>  	/* generate checksum */
> -	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
> +	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);

Where is btrfs_mkfs_config::csum_type defined?

Re: [PATCH v2 01/11] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size()

[David Sterba]

On Tue, Aug 27, 2019 at 06:33:02PM +0200, David Sterba wrote:
> On Mon, Aug 26, 2019 at 01:48:43PM +0200, Johannes Thumshirn wrote:
> > The callers of csum_tree_block_size() blindly assume we're only having
> > crc32c as a possible checksum and thus pass in
> > btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32] for the size argument of
> > csum_tree_block_size().
> > 
> > Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> > Reviewed-by: Nikolay Borisov <nborisov@suse.com>
> > ---
> >  mkfs/common.c | 14 +++++++-------
> >  1 file changed, 7 insertions(+), 7 deletions(-)
> > 
> > diff --git a/mkfs/common.c b/mkfs/common.c
> > index caca5e707233..b6e549b19272 100644
> > --- a/mkfs/common.c
> > +++ b/mkfs/common.c
> > @@ -101,7 +101,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
> >  	}
> >  
> >  	/* generate checksum */
> > -	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
> > +	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
> 
> Where is btrfs_mkfs_config::csum_type defined?

Aha it's in 8/11.

Re: [PATCH v2 01/11] btrfs-progs: don't blindly assume crc32c in csum_tree_block_size()

[Johannes Thumshirn]

On 27/08/2019 18:36, David Sterba wrote:
> On Tue, Aug 27, 2019 at 06:33:02PM +0200, David Sterba wrote:
>> On Mon, Aug 26, 2019 at 01:48:43PM +0200, Johannes Thumshirn wrote:
>>> The callers of csum_tree_block_size() blindly assume we're only having
>>> crc32c as a possible checksum and thus pass in
>>> btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32] for the size argument of
>>> csum_tree_block_size().
>>>
>>> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
>>> Reviewed-by: Nikolay Borisov <nborisov@suse.com>
>>> ---
>>>  mkfs/common.c | 14 +++++++-------
>>>  1 file changed, 7 insertions(+), 7 deletions(-)
>>>
>>> diff --git a/mkfs/common.c b/mkfs/common.c
>>> index caca5e707233..b6e549b19272 100644
>>> --- a/mkfs/common.c
>>> +++ b/mkfs/common.c
>>> @@ -101,7 +101,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
>>>  	}
>>>  
>>>  	/* generate checksum */
>>> -	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
>>> +	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
>>
>> Where is btrfs_mkfs_config::csum_type defined?
> 
> Aha it's in 8/11.

Grr, more artifacts of rebasing/reorganizing the series.

Sorry for that.

-- 
Johannes Thumshirn                            SUSE Labs Filesystems
jthumshirn@suse.de                                +49 911 74053 689
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5
90409 Nürnberg
Germany
(HRB 247165, AG München)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

[PATCH v2 02/11] btrfs-progs: cache csum_type in recover_control

[Johannes Thumshirn]

Cache the super-block's checksum type field in 'struct recover_control'.
This will be needed for further refactoring the checksum handling.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
---
 cmds/rescue-chunk-recover.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index 608af9d49407..308731ea5ea6 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -47,6 +47,7 @@ struct recover_control {
 	int yes;
 
 	u16 csum_size;
+	u16 csum_type;
 	u32 sectorsize;
 	u32 nodesize;
 	u64 generation;
@@ -1530,6 +1531,7 @@ static int recover_prepare(struct recover_control *rc, const char *path)
 	rc->generation = btrfs_super_generation(sb);
 	rc->chunk_root_generation = btrfs_super_chunk_root_generation(sb);
 	rc->csum_size = btrfs_super_csum_size(sb);
+	rc->csum_type = btrfs_super_csum_type(sb);
 
 	/* if seed, the result of scanning below will be partial */
 	if (btrfs_super_flags(sb) & BTRFS_SUPER_FLAG_SEEDING) {
-- 
2.16.4

[PATCH v2 03/11] btrfs-progs: add checksum type to checksumming functions

[Johannes Thumshirn]

Add the checksum type to csum_tree_block_size(), __csum_tree_block_size()
and verify_tree_block_csum_silent().

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 btrfs-corrupt-block.c       |  3 ++-
 cmds/rescue-chunk-recover.c |  3 ++-
 convert/common.c            |  3 ++-
 convert/main.c              |  3 ++-
 disk-io.c                   | 21 ++++++++++++---------
 disk-io.h                   |  5 +++--
 mkfs/common.c               | 21 ++++++++++++++-------
 7 files changed, 37 insertions(+), 22 deletions(-)

diff --git a/btrfs-corrupt-block.c b/btrfs-corrupt-block.c
index bbef0c02e5d1..1dde9594bdcc 100644
--- a/btrfs-corrupt-block.c
+++ b/btrfs-corrupt-block.c
@@ -158,7 +158,8 @@ static void corrupt_keys(struct btrfs_trans_handle *trans,
 	if (!trans) {
 		u16 csum_size =
 			btrfs_super_csum_size(fs_info->super_copy);
-		csum_tree_block_size(eb, csum_size, 0);
+		u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
+		csum_tree_block_size(eb, csum_size, 0, csum_type);
 		write_extent_to_disk(eb);
 	}
 }
diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index 308731ea5ea6..1a368310d895 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -768,7 +768,8 @@ static int scan_one_device(void *dev_scan_struct)
 			continue;
 		}
 
-		if (verify_tree_block_csum_silent(buf, rc->csum_size)) {
+		if (verify_tree_block_csum_silent(buf, rc->csum_size,
+						  rc->csum_type)) {
 			bytenr += rc->sectorsize;
 			continue;
 		}
diff --git a/convert/common.c b/convert/common.c
index af4f8d372299..ba69e132677e 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -222,7 +222,8 @@ static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf,
 {
 	int ret;
 
-	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+			     cfg->csum_type);
 
 	/* Temporary extent buffer is always mapped 1:1 on disk */
 	ret = pwrite(fd, buf->data, buf->len, bytenr);
diff --git a/convert/main.c b/convert/main.c
index 9711874bd137..5e6b12431f59 100644
--- a/convert/main.c
+++ b/convert/main.c
@@ -1058,7 +1058,8 @@ static int migrate_super_block(int fd, u64 old_bytenr)
 	BUG_ON(btrfs_super_bytenr(super) != old_bytenr);
 	btrfs_set_super_bytenr(super, BTRFS_SUPER_INFO_OFFSET);
 
-	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0,
+			     btrfs_super_csum_type(super));
 	ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
 		BTRFS_SUPER_INFO_OFFSET);
 	if (ret != BTRFS_SUPER_INFO_SIZE)
diff --git a/disk-io.c b/disk-io.c
index be44eead5cef..9cb475bc6262 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -149,7 +149,7 @@ void btrfs_csum_final(u32 crc, u8 *result)
 }
 
 static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
-				  int verify, int silent)
+				  int verify, int silent, u16 csum_type)
 {
 	u8 result[BTRFS_CSUM_SIZE];
 	u32 len;
@@ -174,24 +174,27 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
 	return 0;
 }
 
-int csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, int verify)
+int csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, int verify,
+			 u16 csum_type)
 {
-	return __csum_tree_block_size(buf, csum_size, verify, 0);
+	return __csum_tree_block_size(buf, csum_size, verify, 0, csum_type);
 }
 
-int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size)
+int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size,
+				  u16 csum_type)
 {
-	return __csum_tree_block_size(buf, csum_size, 1, 1);
+	return __csum_tree_block_size(buf, csum_size, 1, 1, csum_type);
 }
 
 int csum_tree_block(struct btrfs_fs_info *fs_info,
 		    struct extent_buffer *buf, int verify)
 {
-	u16 csum_size =
-		btrfs_super_csum_size(fs_info->super_copy);
+	u16 csum_size = btrfs_super_csum_size(fs_info->super_copy);
+	u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
+
 	if (verify && fs_info->suppress_check_block_errors)
-		return verify_tree_block_csum_silent(buf, csum_size);
-	return csum_tree_block_size(buf, csum_size, verify);
+		return verify_tree_block_csum_silent(buf, csum_size, csum_type);
+	return csum_tree_block_size(buf, csum_size, verify, csum_type);
 }
 
 struct extent_buffer *btrfs_find_tree_block(struct btrfs_fs_info *fs_info,
diff --git a/disk-io.h b/disk-io.h
index 7b5c3806ba98..394997ad72cb 100644
--- a/disk-io.h
+++ b/disk-io.h
@@ -191,8 +191,9 @@ void btrfs_csum_final(u32 crc, u8 *result);
 
 int btrfs_open_device(struct btrfs_device *dev);
 int csum_tree_block_size(struct extent_buffer *buf, u16 csum_sectorsize,
-			 int verify);
-int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size);
+			 int verify, u16 csum_type);
+int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size,
+				  u16 csum_type);
 int btrfs_read_buffer(struct extent_buffer *buf, u64 parent_transid);
 int write_tree_block(struct btrfs_trans_handle *trans,
 		     struct btrfs_fs_info *fs_info,
diff --git a/mkfs/common.c b/mkfs/common.c
index b6e549b19272..9762391a8d2b 100644
--- a/mkfs/common.c
+++ b/mkfs/common.c
@@ -101,7 +101,8 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
 	}
 
 	/* generate checksum */
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+			     cfg->csum_type);
 
 	/* write back root tree */
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_ROOT_TREE]);
@@ -292,7 +293,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, nritems);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]);
 	if (ret != cfg->nodesize) {
 		ret = (ret < 0 ? -errno : -EIO);
@@ -380,7 +382,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, nritems);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]);
 	if (ret != cfg->nodesize) {
 		ret = (ret < 0 ? -errno : -EIO);
@@ -420,7 +423,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, nritems);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]);
 	if (ret != cfg->nodesize) {
 		ret = (ret < 0 ? -errno : -EIO);
@@ -433,7 +437,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, 0);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]);
 	if (ret != cfg->nodesize) {
 		ret = (ret < 0 ? -errno : -EIO);
@@ -445,7 +450,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, 0);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]);
 	if (ret != cfg->nodesize) {
 		ret = (ret < 0 ? -errno : -EIO);
@@ -456,7 +462,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE);
 	memcpy(buf->data, &super, sizeof(super));
 	buf->len = BTRFS_SUPER_INFO_SIZE;
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
+	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
 			cfg->blocks[MKFS_SUPER_BLOCK]);
 	if (ret != BTRFS_SUPER_INFO_SIZE) {
-- 
2.16.4

Re: [PATCH v2 03/11] btrfs-progs: add checksum type to checksumming functions

[Nikolay Borisov]

On 26.08.19 г. 14:48 ч., Johannes Thumshirn wrote:
> Add the checksum type to csum_tree_block_size(), __csum_tree_block_size()
> and verify_tree_block_csum_silent().
> 
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>

This just propagates args with no functional changes so:

Reviewed-by: Nikolay Borisov <nborisov@suse.com>

> ---
>  btrfs-corrupt-block.c       |  3 ++-
>  cmds/rescue-chunk-recover.c |  3 ++-
>  convert/common.c            |  3 ++-
>  convert/main.c              |  3 ++-
>  disk-io.c                   | 21 ++++++++++++---------
>  disk-io.h                   |  5 +++--
>  mkfs/common.c               | 21 ++++++++++++++-------
>  7 files changed, 37 insertions(+), 22 deletions(-)
> 
> diff --git a/btrfs-corrupt-block.c b/btrfs-corrupt-block.c
> index bbef0c02e5d1..1dde9594bdcc 100644
> --- a/btrfs-corrupt-block.c
> +++ b/btrfs-corrupt-block.c
> @@ -158,7 +158,8 @@ static void corrupt_keys(struct btrfs_trans_handle *trans,
>  	if (!trans) {
>  		u16 csum_size =
>  			btrfs_super_csum_size(fs_info->super_copy);
> -		csum_tree_block_size(eb, csum_size, 0);
> +		u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
> +		csum_tree_block_size(eb, csum_size, 0, csum_type);
>  		write_extent_to_disk(eb);
>  	}
>  }
> diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
> index 308731ea5ea6..1a368310d895 100644
> --- a/cmds/rescue-chunk-recover.c
> +++ b/cmds/rescue-chunk-recover.c
> @@ -768,7 +768,8 @@ static int scan_one_device(void *dev_scan_struct)
>  			continue;
>  		}
>  
> -		if (verify_tree_block_csum_silent(buf, rc->csum_size)) {
> +		if (verify_tree_block_csum_silent(buf, rc->csum_size,
> +						  rc->csum_type)) {
>  			bytenr += rc->sectorsize;
>  			continue;
>  		}
> diff --git a/convert/common.c b/convert/common.c
> index af4f8d372299..ba69e132677e 100644
> --- a/convert/common.c
> +++ b/convert/common.c
> @@ -222,7 +222,8 @@ static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf,
>  {
>  	int ret;
>  
> -	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
> +	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> +			     cfg->csum_type);
>  
>  	/* Temporary extent buffer is always mapped 1:1 on disk */
>  	ret = pwrite(fd, buf->data, buf->len, bytenr);
> diff --git a/convert/main.c b/convert/main.c
> index 9711874bd137..5e6b12431f59 100644
> --- a/convert/main.c
> +++ b/convert/main.c
> @@ -1058,7 +1058,8 @@ static int migrate_super_block(int fd, u64 old_bytenr)
>  	BUG_ON(btrfs_super_bytenr(super) != old_bytenr);
>  	btrfs_set_super_bytenr(super, BTRFS_SUPER_INFO_OFFSET);
>  
> -	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0);
> +	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0,
> +			     btrfs_super_csum_type(super));
>  	ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
>  		BTRFS_SUPER_INFO_OFFSET);
>  	if (ret != BTRFS_SUPER_INFO_SIZE)
> diff --git a/disk-io.c b/disk-io.c
> index be44eead5cef..9cb475bc6262 100644
> --- a/disk-io.c
> +++ b/disk-io.c
> @@ -149,7 +149,7 @@ void btrfs_csum_final(u32 crc, u8 *result)
>  }
>  
>  static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
> -				  int verify, int silent)
> +				  int verify, int silent, u16 csum_type)
>  {
>  	u8 result[BTRFS_CSUM_SIZE];
>  	u32 len;
> @@ -174,24 +174,27 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
>  	return 0;
>  }
>  
> -int csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, int verify)
> +int csum_tree_block_size(struct extent_buffer *buf, u16 csum_size, int verify,
> +			 u16 csum_type)
>  {
> -	return __csum_tree_block_size(buf, csum_size, verify, 0);
> +	return __csum_tree_block_size(buf, csum_size, verify, 0, csum_type);
>  }
>  
> -int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size)
> +int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size,
> +				  u16 csum_type)
>  {
> -	return __csum_tree_block_size(buf, csum_size, 1, 1);
> +	return __csum_tree_block_size(buf, csum_size, 1, 1, csum_type);
>  }
>  
>  int csum_tree_block(struct btrfs_fs_info *fs_info,
>  		    struct extent_buffer *buf, int verify)
>  {
> -	u16 csum_size =
> -		btrfs_super_csum_size(fs_info->super_copy);
> +	u16 csum_size = btrfs_super_csum_size(fs_info->super_copy);
> +	u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
> +
>  	if (verify && fs_info->suppress_check_block_errors)
> -		return verify_tree_block_csum_silent(buf, csum_size);
> -	return csum_tree_block_size(buf, csum_size, verify);
> +		return verify_tree_block_csum_silent(buf, csum_size, csum_type);
> +	return csum_tree_block_size(buf, csum_size, verify, csum_type);
>  }
>  
>  struct extent_buffer *btrfs_find_tree_block(struct btrfs_fs_info *fs_info,
> diff --git a/disk-io.h b/disk-io.h
> index 7b5c3806ba98..394997ad72cb 100644
> --- a/disk-io.h
> +++ b/disk-io.h
> @@ -191,8 +191,9 @@ void btrfs_csum_final(u32 crc, u8 *result);
>  
>  int btrfs_open_device(struct btrfs_device *dev);
>  int csum_tree_block_size(struct extent_buffer *buf, u16 csum_sectorsize,
> -			 int verify);
> -int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size);
> +			 int verify, u16 csum_type);
> +int verify_tree_block_csum_silent(struct extent_buffer *buf, u16 csum_size,
> +				  u16 csum_type);
>  int btrfs_read_buffer(struct extent_buffer *buf, u64 parent_transid);
>  int write_tree_block(struct btrfs_trans_handle *trans,
>  		     struct btrfs_fs_info *fs_info,
> diff --git a/mkfs/common.c b/mkfs/common.c
> index b6e549b19272..9762391a8d2b 100644
> --- a/mkfs/common.c
> +++ b/mkfs/common.c
> @@ -101,7 +101,8 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
>  	}
>  
>  	/* generate checksum */
> -	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
> +	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> +			     cfg->csum_type);
>  
>  	/* write back root tree */
>  	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_ROOT_TREE]);
> @@ -292,7 +293,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
>  	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]);
>  	btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID);
>  	btrfs_set_header_nritems(buf, nritems);
> -	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
> +	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> +			     cfg->csum_type);
>  	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]);
>  	if (ret != cfg->nodesize) {
>  		ret = (ret < 0 ? -errno : -EIO);
> @@ -380,7 +382,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
>  	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]);
>  	btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID);
>  	btrfs_set_header_nritems(buf, nritems);
> -	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
> +	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> +			     cfg->csum_type);
>  	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]);
>  	if (ret != cfg->nodesize) {
>  		ret = (ret < 0 ? -errno : -EIO);
> @@ -420,7 +423,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
>  	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]);
>  	btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID);
>  	btrfs_set_header_nritems(buf, nritems);
> -	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
> +	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> +			     cfg->csum_type);
>  	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]);
>  	if (ret != cfg->nodesize) {
>  		ret = (ret < 0 ? -errno : -EIO);
> @@ -433,7 +437,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
>  	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]);
>  	btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID);
>  	btrfs_set_header_nritems(buf, 0);
> -	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
> +	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> +			     cfg->csum_type);
>  	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]);
>  	if (ret != cfg->nodesize) {
>  		ret = (ret < 0 ? -errno : -EIO);
> @@ -445,7 +450,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
>  	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]);
>  	btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID);
>  	btrfs_set_header_nritems(buf, 0);
> -	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
> +	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> +			     cfg->csum_type);
>  	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]);
>  	if (ret != cfg->nodesize) {
>  		ret = (ret < 0 ? -errno : -EIO);
> @@ -456,7 +462,8 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
>  	memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE);
>  	memcpy(buf->data, &super, sizeof(super));
>  	buf->len = BTRFS_SUPER_INFO_SIZE;
> -	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0);
> +	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> +			     cfg->csum_type);
>  	ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
>  			cfg->blocks[MKFS_SUPER_BLOCK]);
>  	if (ret != BTRFS_SUPER_INFO_SIZE) {
> 

[PATCH v2 04/11] btrfs-progs: don't assume checksums are always 4 bytes

[Johannes Thumshirn]

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 btrfs-sb-mod.c              |  9 ++++++---
 check/main.c                |  2 +-
 cmds/inspect-dump-super.c   |  3 ++-
 cmds/rescue-chunk-recover.c |  2 +-
 convert/common.c            |  3 ++-
 disk-io.c                   | 12 ++++++------
 disk-io.h                   |  2 +-
 file-item.c                 |  2 +-
 8 files changed, 20 insertions(+), 15 deletions(-)

diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
index 16a26f772494..932c2a0432ef 100644
--- a/btrfs-sb-mod.c
+++ b/btrfs-sb-mod.c
@@ -37,7 +37,8 @@ static int check_csum_superblock(void *sb)
 	u32 crc = ~(u32)0;
 
 	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
-				crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+				(u8 *)&crc,
+				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
 	btrfs_csum_final(crc, result);
 
 	return !memcmp(sb, &result, csum_size);
@@ -50,10 +51,12 @@ static void update_block_csum(void *block, int is_sb)
 	u32 crc = ~(u32)0;
 
 	if (is_sb) {
-		crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE, crc,
+		crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE,
+				      (u8 *)&crc,
 				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
 	} else {
-		crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE, crc,
+		crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE,
+				      (u8 *)&crc,
 				BLOCKSIZE - BTRFS_CSUM_SIZE);
 	}
 	btrfs_csum_final(crc, result);
diff --git a/check/main.c b/check/main.c
index 8b340e5e7894..50498ad86dca 100644
--- a/check/main.c
+++ b/check/main.c
@@ -5622,7 +5622,7 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
 				tmp = offset + data_checked;
 
 				csum = btrfs_csum_data((char *)data + tmp,
-						csum, fs_info->sectorsize);
+						(u8 *)&csum, fs_info->sectorsize);
 				btrfs_csum_final(csum, (u8 *)&csum);
 
 				csum_offset = leaf_offset +
diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
index 65fb3506eac6..96ad3deca3d8 100644
--- a/cmds/inspect-dump-super.c
+++ b/cmds/inspect-dump-super.c
@@ -41,7 +41,8 @@ static int check_csum_sblock(void *sb, int csum_size)
 	u32 crc = ~(u32)0;
 
 	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
-				crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+				(u8 *)&crc,
+				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
 	btrfs_csum_final(crc, result);
 
 	return !memcmp(sb, &result, csum_size);
diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index 1a368310d895..0fddb5dd8ead 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -1902,7 +1902,7 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum)
 		goto out;
 	}
 	ret = 0;
-	csum_result = btrfs_csum_data(data, csum_result, len);
+	csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
 	btrfs_csum_final(csum_result, (u8 *)&csum_result);
 	if (csum_result != tree_csum)
 		ret = 1;
diff --git a/convert/common.c b/convert/common.c
index ba69e132677e..c3702dc403ed 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -65,7 +65,8 @@ static inline int write_temp_super(int fd, struct btrfs_super_block *sb,
        u32 crc = ~(u32)0;
        int ret;
 
-       crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc,
+       crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
+			     (u8 *)&crc,
                              BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
        btrfs_csum_final(crc, &sb->csum[0]);
        ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr);
diff --git a/disk-io.c b/disk-io.c
index 9cb475bc6262..a0c37c569d58 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -138,9 +138,9 @@ static void print_tree_block_error(struct btrfs_fs_info *fs_info,
 	}
 }
 
-u32 btrfs_csum_data(char *data, u32 seed, size_t len)
+u32 btrfs_csum_data(char *data, u8 *seed, size_t len)
 {
-	return crc32c(seed, data, len);
+	return crc32c(*(u32*)seed, data, len);
 }
 
 void btrfs_csum_final(u32 crc, u8 *result)
@@ -156,7 +156,7 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
 	u32 crc = ~(u32)0;
 
 	len = buf->len - BTRFS_CSUM_SIZE;
-	crc = crc32c(crc, buf->data + BTRFS_CSUM_SIZE, len);
+	crc = btrfs_csum_data(buf->data + BTRFS_CSUM_SIZE, (u8 *)&crc, len);
 	btrfs_csum_final(crc, result);
 
 	if (verify) {
@@ -1376,7 +1376,7 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
 	csum_size = btrfs_csum_sizes[csum_type];
 
 	crc = ~(u32)0;
-	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc,
+	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
 			      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
 	btrfs_csum_final(crc, result);
 
@@ -1631,7 +1631,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
 	if (fs_info->super_bytenr != BTRFS_SUPER_INFO_OFFSET) {
 		btrfs_set_super_bytenr(sb, fs_info->super_bytenr);
 		crc = ~(u32)0;
-		crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc,
+		crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
 				      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
 		btrfs_csum_final(crc, &sb->csum[0]);
 
@@ -1667,7 +1667,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
 		btrfs_set_super_bytenr(sb, bytenr);
 
 		crc = ~(u32)0;
-		crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc,
+		crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
 				      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
 		btrfs_csum_final(crc, &sb->csum[0]);
 
diff --git a/disk-io.h b/disk-io.h
index 394997ad72cb..92c87f28f8b2 100644
--- a/disk-io.h
+++ b/disk-io.h
@@ -186,7 +186,7 @@ int btrfs_free_fs_root(struct btrfs_root *root);
 void btrfs_mark_buffer_dirty(struct extent_buffer *buf);
 int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid);
 int btrfs_set_buffer_uptodate(struct extent_buffer *buf);
-u32 btrfs_csum_data(char *data, u32 seed, size_t len);
+u32 btrfs_csum_data(char *data, u8 *seed, size_t len);
 void btrfs_csum_final(u32 crc, u8 *result);
 
 int btrfs_open_device(struct btrfs_device *dev);
diff --git a/file-item.c b/file-item.c
index 3bf48c68913d..5f6548e9a74f 100644
--- a/file-item.c
+++ b/file-item.c
@@ -312,7 +312,7 @@ csum:
 	item = (struct btrfs_csum_item *)((unsigned char *)item +
 					  csum_offset * csum_size);
 found:
-	csum_result = btrfs_csum_data(data, csum_result, len);
+	csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
 	btrfs_csum_final(csum_result, (u8 *)&csum_result);
 	if (csum_result == 0) {
 		printk("csum result is 0 for block %llu\n",
-- 
2.16.4

Re: [PATCH v2 04/11] btrfs-progs: don't assume checksums are always 4 bytes

[Nikolay Borisov]

On 26.08.19 г. 14:48 ч., Johannes Thumshirn wrote:
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>

Reviewed-by: Nikolay Borisov <nborisov@suse.com>

> ---
>  btrfs-sb-mod.c              |  9 ++++++---
>  check/main.c                |  2 +-
>  cmds/inspect-dump-super.c   |  3 ++-
>  cmds/rescue-chunk-recover.c |  2 +-
>  convert/common.c            |  3 ++-
>  disk-io.c                   | 12 ++++++------
>  disk-io.h                   |  2 +-
>  file-item.c                 |  2 +-
>  8 files changed, 20 insertions(+), 15 deletions(-)
> 
> diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
> index 16a26f772494..932c2a0432ef 100644
> --- a/btrfs-sb-mod.c
> +++ b/btrfs-sb-mod.c
> @@ -37,7 +37,8 @@ static int check_csum_superblock(void *sb)
>  	u32 crc = ~(u32)0;
>  
>  	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
> -				crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> +				(u8 *)&crc,
> +				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
>  	btrfs_csum_final(crc, result);
>  
>  	return !memcmp(sb, &result, csum_size);
> @@ -50,10 +51,12 @@ static void update_block_csum(void *block, int is_sb)
>  	u32 crc = ~(u32)0;
>  
>  	if (is_sb) {
> -		crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE, crc,
> +		crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE,
> +				      (u8 *)&crc,
>  				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
>  	} else {
> -		crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE, crc,
> +		crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE,
> +				      (u8 *)&crc,
>  				BLOCKSIZE - BTRFS_CSUM_SIZE);
>  	}
>  	btrfs_csum_final(crc, result);
> diff --git a/check/main.c b/check/main.c
> index 8b340e5e7894..50498ad86dca 100644
> --- a/check/main.c
> +++ b/check/main.c
> @@ -5622,7 +5622,7 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
>  				tmp = offset + data_checked;
>  
>  				csum = btrfs_csum_data((char *)data + tmp,
> -						csum, fs_info->sectorsize);
> +						(u8 *)&csum, fs_info->sectorsize);
>  				btrfs_csum_final(csum, (u8 *)&csum);
>  
>  				csum_offset = leaf_offset +
> diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
> index 65fb3506eac6..96ad3deca3d8 100644
> --- a/cmds/inspect-dump-super.c
> +++ b/cmds/inspect-dump-super.c
> @@ -41,7 +41,8 @@ static int check_csum_sblock(void *sb, int csum_size)
>  	u32 crc = ~(u32)0;
>  
>  	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
> -				crc, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> +				(u8 *)&crc,
> +				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
>  	btrfs_csum_final(crc, result);
>  
>  	return !memcmp(sb, &result, csum_size);
> diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
> index 1a368310d895..0fddb5dd8ead 100644
> --- a/cmds/rescue-chunk-recover.c
> +++ b/cmds/rescue-chunk-recover.c
> @@ -1902,7 +1902,7 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum)
>  		goto out;
>  	}
>  	ret = 0;
> -	csum_result = btrfs_csum_data(data, csum_result, len);
> +	csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
>  	btrfs_csum_final(csum_result, (u8 *)&csum_result);
>  	if (csum_result != tree_csum)
>  		ret = 1;
> diff --git a/convert/common.c b/convert/common.c
> index ba69e132677e..c3702dc403ed 100644
> --- a/convert/common.c
> +++ b/convert/common.c
> @@ -65,7 +65,8 @@ static inline int write_temp_super(int fd, struct btrfs_super_block *sb,
>         u32 crc = ~(u32)0;
>         int ret;
>  
> -       crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc,
> +       crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
> +			     (u8 *)&crc,
>                               BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
>         btrfs_csum_final(crc, &sb->csum[0]);
>         ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr);
> diff --git a/disk-io.c b/disk-io.c
> index 9cb475bc6262..a0c37c569d58 100644
> --- a/disk-io.c
> +++ b/disk-io.c
> @@ -138,9 +138,9 @@ static void print_tree_block_error(struct btrfs_fs_info *fs_info,
>  	}
>  }
>  
> -u32 btrfs_csum_data(char *data, u32 seed, size_t len)
> +u32 btrfs_csum_data(char *data, u8 *seed, size_t len)
>  {
> -	return crc32c(seed, data, len);
> +	return crc32c(*(u32*)seed, data, len);
>  }
>  
>  void btrfs_csum_final(u32 crc, u8 *result)
> @@ -156,7 +156,7 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
>  	u32 crc = ~(u32)0;
>  
>  	len = buf->len - BTRFS_CSUM_SIZE;
> -	crc = crc32c(crc, buf->data + BTRFS_CSUM_SIZE, len);
> +	crc = btrfs_csum_data(buf->data + BTRFS_CSUM_SIZE, (u8 *)&crc, len);
>  	btrfs_csum_final(crc, result);
>  
>  	if (verify) {
> @@ -1376,7 +1376,7 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
>  	csum_size = btrfs_csum_sizes[csum_type];
>  
>  	crc = ~(u32)0;
> -	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc,
> +	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
>  			      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
>  	btrfs_csum_final(crc, result);
>  
> @@ -1631,7 +1631,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
>  	if (fs_info->super_bytenr != BTRFS_SUPER_INFO_OFFSET) {
>  		btrfs_set_super_bytenr(sb, fs_info->super_bytenr);
>  		crc = ~(u32)0;
> -		crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc,
> +		crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
>  				      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
>  		btrfs_csum_final(crc, &sb->csum[0]);
>  
> @@ -1667,7 +1667,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
>  		btrfs_set_super_bytenr(sb, bytenr);
>  
>  		crc = ~(u32)0;
> -		crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, crc,
> +		crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
>  				      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
>  		btrfs_csum_final(crc, &sb->csum[0]);
>  
> diff --git a/disk-io.h b/disk-io.h
> index 394997ad72cb..92c87f28f8b2 100644
> --- a/disk-io.h
> +++ b/disk-io.h
> @@ -186,7 +186,7 @@ int btrfs_free_fs_root(struct btrfs_root *root);
>  void btrfs_mark_buffer_dirty(struct extent_buffer *buf);
>  int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid);
>  int btrfs_set_buffer_uptodate(struct extent_buffer *buf);
> -u32 btrfs_csum_data(char *data, u32 seed, size_t len);
> +u32 btrfs_csum_data(char *data, u8 *seed, size_t len);
>  void btrfs_csum_final(u32 crc, u8 *result);
>  
>  int btrfs_open_device(struct btrfs_device *dev);
> diff --git a/file-item.c b/file-item.c
> index 3bf48c68913d..5f6548e9a74f 100644
> --- a/file-item.c
> +++ b/file-item.c
> @@ -312,7 +312,7 @@ csum:
>  	item = (struct btrfs_csum_item *)((unsigned char *)item +
>  					  csum_offset * csum_size);
>  found:
> -	csum_result = btrfs_csum_data(data, csum_result, len);
> +	csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
>  	btrfs_csum_final(csum_result, (u8 *)&csum_result);
>  	if (csum_result == 0) {
>  		printk("csum result is 0 for block %llu\n",
> 

[PATCH v2 05/11] btrfs-progs: pass checksum type to btrfs_csum_data()/btrfs_csum_final()

[Johannes Thumshirn]

In preparation to supporting new checksum algorithm pass the checksum type
to btrfs_csum_data/btrfs_csum_final, this allows us to encapsulate any
differences in processing into the respective functions

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>

---
Changes to v1:
- Update changelog and squashed the other identical patch
---
 btrfs-sb-mod.c              | 15 ++++++++++-----
 check/main.c                |  6 ++++--
 cmds/inspect-dump-super.c   |  8 ++++----
 cmds/rescue-chunk-recover.c | 10 ++++++----
 convert/common.c            |  5 +++--
 disk-io.c                   | 39 +++++++++++++++++++++++++++------------
 disk-io.h                   |  4 ++--
 file-item.c                 |  7 +++++--
 free-space-cache.c          |  2 +-
 image/main.c                |  2 +-
 10 files changed, 63 insertions(+), 35 deletions(-)

diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
index 932c2a0432ef..d9630f187d0f 100644
--- a/btrfs-sb-mod.c
+++ b/btrfs-sb-mod.c
@@ -35,11 +35,13 @@ static int check_csum_superblock(void *sb)
 {
 	u8 result[csum_size];
 	u32 crc = ~(u32)0;
+	u16 csum_type = btrfs_super_csum_type(sb);
 
-	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
+	crc = btrfs_csum_data(csum_type,
+			      (char *)sb + BTRFS_CSUM_SIZE,
 				(u8 *)&crc,
 				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-	btrfs_csum_final(crc, result);
+	btrfs_csum_final(csum_type, crc, result);
 
 	return !memcmp(sb, &result, csum_size);
 }
@@ -49,17 +51,20 @@ static void update_block_csum(void *block, int is_sb)
 	u8 result[csum_size];
 	struct btrfs_header *hdr;
 	u32 crc = ~(u32)0;
+	u16 csum_type = btrfs_super_csum_type(block);
 
 	if (is_sb) {
-		crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE,
+		crc = btrfs_csum_data(csum_type,
+				      (char *)block + BTRFS_CSUM_SIZE,
 				      (u8 *)&crc,
 				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
 	} else {
-		crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE,
+		crc = btrfs_csum_data(csum_type,
+				      (char *)block + BTRFS_CSUM_SIZE,
 				      (u8 *)&crc,
 				BLOCKSIZE - BTRFS_CSUM_SIZE);
 	}
-	btrfs_csum_final(crc, result);
+	btrfs_csum_final(csum_type, crc, result);
 	memset(block, 0, BTRFS_CSUM_SIZE);
 	hdr = (struct btrfs_header *)block;
 	memcpy(&hdr->csum, result, csum_size);
diff --git a/check/main.c b/check/main.c
index 50498ad86dca..d2fdf48b89c5 100644
--- a/check/main.c
+++ b/check/main.c
@@ -5581,6 +5581,7 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
 	struct btrfs_fs_info *fs_info = root->fs_info;
 	u64 offset = 0;
 	u16 csum_size = btrfs_super_csum_size(fs_info->super_copy);
+	u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
 	char *data;
 	unsigned long csum_offset;
 	u32 csum;
@@ -5621,9 +5622,10 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
 				csum = ~(u32)0;
 				tmp = offset + data_checked;
 
-				csum = btrfs_csum_data((char *)data + tmp,
+				csum = btrfs_csum_data(csum_type,
+						       (char *)data + tmp,
 						(u8 *)&csum, fs_info->sectorsize);
-				btrfs_csum_final(csum, (u8 *)&csum);
+				btrfs_csum_final(csum_type, csum, (u8 *)&csum);
 
 				csum_offset = leaf_offset +
 					 tmp / fs_info->sectorsize * csum_size;
diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
index 96ad3deca3d8..40019a6670ef 100644
--- a/cmds/inspect-dump-super.c
+++ b/cmds/inspect-dump-super.c
@@ -35,15 +35,15 @@
 #include "kernel-lib/crc32c.h"
 #include "common/help.h"
 
-static int check_csum_sblock(void *sb, int csum_size)
+static int check_csum_sblock(void *sb, int csum_size, u16 csum_type)
 {
 	u8 result[BTRFS_CSUM_SIZE];
 	u32 crc = ~(u32)0;
 
-	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
+	crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
 				(u8 *)&crc,
 				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-	btrfs_csum_final(crc, result);
+	btrfs_csum_final(csum_type, crc, result);
 
 	return !memcmp(sb, &result, csum_size);
 }
@@ -348,7 +348,7 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
 	if (csum_type != BTRFS_CSUM_TYPE_CRC32 ||
 	    csum_size != btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32])
 		printf(" [UNKNOWN CSUM TYPE OR SIZE]");
-	else if (check_csum_sblock(sb, csum_size))
+	else if (check_csum_sblock(sb, csum_size, csum_type))
 		printf(" [match]");
 	else
 		printf(" [DON'T MATCH]");
diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index 0fddb5dd8ead..cddae471f50c 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -1887,7 +1887,8 @@ static u64 calc_data_offset(struct btrfs_key *key,
 	return dev_offset + data_offset;
 }
 
-static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum)
+static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum,
+			  u16 csum_type)
 {
 	char *data;
 	int ret = 0;
@@ -1902,8 +1903,8 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum)
 		goto out;
 	}
 	ret = 0;
-	csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
-	btrfs_csum_final(csum_result, (u8 *)&csum_result);
+	csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len);
+	btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result);
 	if (csum_result != tree_csum)
 		ret = 1;
 out:
@@ -2102,7 +2103,8 @@ next_csum:
 						  devext->objectid, 1));
 
 		ret = check_one_csum(dev->fd, data_offset, blocksize,
-				     tree_csum);
+				     tree_csum,
+				     btrfs_super_csum_type(root->fs_info->super_copy));
 		if (ret < 0)
 			goto fail_out;
 		else if (ret > 0)
diff --git a/convert/common.c b/convert/common.c
index c3702dc403ed..5ad9a91b9ab6 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -63,12 +63,13 @@ static inline int write_temp_super(int fd, struct btrfs_super_block *sb,
                                   u64 sb_bytenr)
 {
        u32 crc = ~(u32)0;
+       u16 csum_type = btrfs_super_csum_type(sb);
        int ret;
 
-       crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
+       crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
 			     (u8 *)&crc,
                              BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-       btrfs_csum_final(crc, &sb->csum[0]);
+       btrfs_csum_final(csum_type, crc, &sb->csum[0]);
        ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr);
        if (ret < BTRFS_SUPER_INFO_SIZE)
                ret = (ret < 0 ? -errno : -EIO);
diff --git a/disk-io.c b/disk-io.c
index a0c37c569d58..7e538969c57a 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -138,14 +138,26 @@ static void print_tree_block_error(struct btrfs_fs_info *fs_info,
 	}
 }
 
-u32 btrfs_csum_data(char *data, u8 *seed, size_t len)
+u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len)
 {
-	return crc32c(*(u32*)seed, data, len);
+	switch (csum_type) {
+	case BTRFS_CSUM_TYPE_CRC32:
+		return crc32c(*(u32*)seed, data, len);
+	default: /* Not reached */
+		return ~(u32)0;
+	}
+
 }
 
-void btrfs_csum_final(u32 crc, u8 *result)
+void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result)
 {
-	put_unaligned_le32(~crc, result);
+	switch (csum_type) {
+	case BTRFS_CSUM_TYPE_CRC32:
+		put_unaligned_le32(~crc, result);
+		break;
+	default: /* Not reached */
+		break;
+	}
 }
 
 static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
@@ -156,8 +168,9 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
 	u32 crc = ~(u32)0;
 
 	len = buf->len - BTRFS_CSUM_SIZE;
-	crc = btrfs_csum_data(buf->data + BTRFS_CSUM_SIZE, (u8 *)&crc, len);
-	btrfs_csum_final(crc, result);
+	crc = btrfs_csum_data(csum_type, buf->data + BTRFS_CSUM_SIZE,
+			      (u8 *)&crc, len);
+	btrfs_csum_final(csum_type, crc, result);
 
 	if (verify) {
 		if (memcmp_extent_buffer(buf, result, 0, csum_size)) {
@@ -1376,9 +1389,10 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
 	csum_size = btrfs_csum_sizes[csum_type];
 
 	crc = ~(u32)0;
-	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
+	crc = btrfs_csum_data(csum_type,(char *)sb + BTRFS_CSUM_SIZE,
+			      (u8 *)&crc,
 			      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-	btrfs_csum_final(crc, result);
+	btrfs_csum_final(csum_type, crc, result);
 
 	if (memcmp(result, sb->csum, csum_size)) {
 		error("superblock checksum mismatch");
@@ -1616,6 +1630,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
 	u64 bytenr;
 	u32 crc;
 	int i, ret;
+	u16 csum_type = btrfs_super_csum_type(sb);
 
 	/*
 	 * We need to write super block after all metadata written.
@@ -1631,9 +1646,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
 	if (fs_info->super_bytenr != BTRFS_SUPER_INFO_OFFSET) {
 		btrfs_set_super_bytenr(sb, fs_info->super_bytenr);
 		crc = ~(u32)0;
-		crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
+		crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
 				      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-		btrfs_csum_final(crc, &sb->csum[0]);
+		btrfs_csum_final(csum_type, crc, &sb->csum[0]);
 
 		/*
 		 * super_copy is BTRFS_SUPER_INFO_SIZE bytes and is
@@ -1667,9 +1682,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
 		btrfs_set_super_bytenr(sb, bytenr);
 
 		crc = ~(u32)0;
-		crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
+		crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
 				      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-		btrfs_csum_final(crc, &sb->csum[0]);
+		btrfs_csum_final(csum_type, crc, &sb->csum[0]);
 
 		/*
 		 * super_copy is BTRFS_SUPER_INFO_SIZE bytes and is
diff --git a/disk-io.h b/disk-io.h
index 92c87f28f8b2..4b5e9ea86385 100644
--- a/disk-io.h
+++ b/disk-io.h
@@ -186,8 +186,8 @@ int btrfs_free_fs_root(struct btrfs_root *root);
 void btrfs_mark_buffer_dirty(struct extent_buffer *buf);
 int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid);
 int btrfs_set_buffer_uptodate(struct extent_buffer *buf);
-u32 btrfs_csum_data(char *data, u8 *seed, size_t len);
-void btrfs_csum_final(u32 crc, u8 *result);
+u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len);
+void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result);
 
 int btrfs_open_device(struct btrfs_device *dev);
 int csum_tree_block_size(struct extent_buffer *buf, u16 csum_sectorsize,
diff --git a/file-item.c b/file-item.c
index 5f6548e9a74f..78fdcecd0bab 100644
--- a/file-item.c
+++ b/file-item.c
@@ -202,6 +202,9 @@ int btrfs_csum_file_block(struct btrfs_trans_handle *trans,
 	u16 csum_size =
 		btrfs_super_csum_size(root->fs_info->super_copy);
 
+	u16 csum_type =
+		btrfs_super_csum_type(root->fs_info->super_copy);
+
 	path = btrfs_alloc_path();
 	if (!path)
 		return -ENOMEM;
@@ -312,8 +315,8 @@ csum:
 	item = (struct btrfs_csum_item *)((unsigned char *)item +
 					  csum_offset * csum_size);
 found:
-	csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
-	btrfs_csum_final(csum_result, (u8 *)&csum_result);
+	csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len);
+	btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result);
 	if (csum_result == 0) {
 		printk("csum result is 0 for block %llu\n",
 		       (unsigned long long)bytenr);
diff --git a/free-space-cache.c b/free-space-cache.c
index e872eb6a00db..8a57f86dc650 100644
--- a/free-space-cache.c
+++ b/free-space-cache.c
@@ -213,7 +213,7 @@ static int io_ctl_check_crc(struct io_ctl *io_ctl, int index)
 	io_ctl_map_page(io_ctl, 0);
 	crc = crc32c(crc, io_ctl->orig + offset,
 			io_ctl->root->fs_info->sectorsize - offset);
-	btrfs_csum_final(crc, (u8 *)&crc);
+	put_unaligned_le32(~crc, (u8 *)&crc);
 	if (val != crc) {
 		printk("btrfs: csum mismatch on free space cache\n");
 		io_ctl_unmap_page(io_ctl);
diff --git a/image/main.c b/image/main.c
index 28ef1609b5ff..0c8ffede56f5 100644
--- a/image/main.c
+++ b/image/main.c
@@ -124,7 +124,7 @@ static void csum_block(u8 *buf, size_t len)
 	u8 result[btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]];
 	u32 crc = ~(u32)0;
 	crc = crc32c(crc, buf + BTRFS_CSUM_SIZE, len - BTRFS_CSUM_SIZE);
-	btrfs_csum_final(crc, result);
+	put_unaligned_le32(~crc, result);
 	memcpy(buf, result, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]);
 }
 
-- 
2.16.4

Re: [PATCH v2 05/11] btrfs-progs: pass checksum type to btrfs_csum_data()/btrfs_csum_final()

[Nikolay Borisov]

On 26.08.19 г. 14:48 ч., Johannes Thumshirn wrote:
> In preparation to supporting new checksum algorithm pass the checksum type
> to btrfs_csum_data/btrfs_csum_final, this allows us to encapsulate any
> differences in processing into the respective functions
> 
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>

Reviewed-by: Nikolay Borisov <nborisov@suse.com>


> 
> ---
> Changes to v1:
> - Update changelog and squashed the other identical patch
> ---
>  btrfs-sb-mod.c              | 15 ++++++++++-----
>  check/main.c                |  6 ++++--
>  cmds/inspect-dump-super.c   |  8 ++++----
>  cmds/rescue-chunk-recover.c | 10 ++++++----
>  convert/common.c            |  5 +++--
>  disk-io.c                   | 39 +++++++++++++++++++++++++++------------
>  disk-io.h                   |  4 ++--
>  file-item.c                 |  7 +++++--
>  free-space-cache.c          |  2 +-
>  image/main.c                |  2 +-
>  10 files changed, 63 insertions(+), 35 deletions(-)
> 
> diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
> index 932c2a0432ef..d9630f187d0f 100644
> --- a/btrfs-sb-mod.c
> +++ b/btrfs-sb-mod.c
> @@ -35,11 +35,13 @@ static int check_csum_superblock(void *sb)
>  {
>  	u8 result[csum_size];
>  	u32 crc = ~(u32)0;
> +	u16 csum_type = btrfs_super_csum_type(sb);
>  
> -	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
> +	crc = btrfs_csum_data(csum_type,
> +			      (char *)sb + BTRFS_CSUM_SIZE,
>  				(u8 *)&crc,
>  				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> -	btrfs_csum_final(crc, result);
> +	btrfs_csum_final(csum_type, crc, result);
>  
>  	return !memcmp(sb, &result, csum_size);
>  }
> @@ -49,17 +51,20 @@ static void update_block_csum(void *block, int is_sb)
>  	u8 result[csum_size];
>  	struct btrfs_header *hdr;
>  	u32 crc = ~(u32)0;
> +	u16 csum_type = btrfs_super_csum_type(block);
>  
>  	if (is_sb) {
> -		crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE,
> +		crc = btrfs_csum_data(csum_type,
> +				      (char *)block + BTRFS_CSUM_SIZE,
>  				      (u8 *)&crc,
>  				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
>  	} else {
> -		crc = btrfs_csum_data((char *)block + BTRFS_CSUM_SIZE,
> +		crc = btrfs_csum_data(csum_type,
> +				      (char *)block + BTRFS_CSUM_SIZE,
>  				      (u8 *)&crc,
>  				BLOCKSIZE - BTRFS_CSUM_SIZE);
>  	}
> -	btrfs_csum_final(crc, result);
> +	btrfs_csum_final(csum_type, crc, result);
>  	memset(block, 0, BTRFS_CSUM_SIZE);
>  	hdr = (struct btrfs_header *)block;
>  	memcpy(&hdr->csum, result, csum_size);
> diff --git a/check/main.c b/check/main.c
> index 50498ad86dca..d2fdf48b89c5 100644
> --- a/check/main.c
> +++ b/check/main.c
> @@ -5581,6 +5581,7 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
>  	struct btrfs_fs_info *fs_info = root->fs_info;
>  	u64 offset = 0;
>  	u16 csum_size = btrfs_super_csum_size(fs_info->super_copy);
> +	u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
>  	char *data;
>  	unsigned long csum_offset;
>  	u32 csum;
> @@ -5621,9 +5622,10 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
>  				csum = ~(u32)0;
>  				tmp = offset + data_checked;
>  
> -				csum = btrfs_csum_data((char *)data + tmp,
> +				csum = btrfs_csum_data(csum_type,
> +						       (char *)data + tmp,
>  						(u8 *)&csum, fs_info->sectorsize);
> -				btrfs_csum_final(csum, (u8 *)&csum);
> +				btrfs_csum_final(csum_type, csum, (u8 *)&csum);
>  
>  				csum_offset = leaf_offset +
>  					 tmp / fs_info->sectorsize * csum_size;
> diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
> index 96ad3deca3d8..40019a6670ef 100644
> --- a/cmds/inspect-dump-super.c
> +++ b/cmds/inspect-dump-super.c
> @@ -35,15 +35,15 @@
>  #include "kernel-lib/crc32c.h"
>  #include "common/help.h"
>  
> -static int check_csum_sblock(void *sb, int csum_size)
> +static int check_csum_sblock(void *sb, int csum_size, u16 csum_type)
>  {
>  	u8 result[BTRFS_CSUM_SIZE];
>  	u32 crc = ~(u32)0;
>  
> -	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
> +	crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
>  				(u8 *)&crc,
>  				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> -	btrfs_csum_final(crc, result);
> +	btrfs_csum_final(csum_type, crc, result);
>  
>  	return !memcmp(sb, &result, csum_size);
>  }
> @@ -348,7 +348,7 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
>  	if (csum_type != BTRFS_CSUM_TYPE_CRC32 ||
>  	    csum_size != btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32])
>  		printf(" [UNKNOWN CSUM TYPE OR SIZE]");
> -	else if (check_csum_sblock(sb, csum_size))
> +	else if (check_csum_sblock(sb, csum_size, csum_type))
>  		printf(" [match]");
>  	else
>  		printf(" [DON'T MATCH]");
> diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
> index 0fddb5dd8ead..cddae471f50c 100644
> --- a/cmds/rescue-chunk-recover.c
> +++ b/cmds/rescue-chunk-recover.c
> @@ -1887,7 +1887,8 @@ static u64 calc_data_offset(struct btrfs_key *key,
>  	return dev_offset + data_offset;
>  }
>  
> -static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum)
> +static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum,
> +			  u16 csum_type)
>  {
>  	char *data;
>  	int ret = 0;
> @@ -1902,8 +1903,8 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum)
>  		goto out;
>  	}
>  	ret = 0;
> -	csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
> -	btrfs_csum_final(csum_result, (u8 *)&csum_result);
> +	csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len);
> +	btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result);
>  	if (csum_result != tree_csum)
>  		ret = 1;
>  out:
> @@ -2102,7 +2103,8 @@ next_csum:
>  						  devext->objectid, 1));
>  
>  		ret = check_one_csum(dev->fd, data_offset, blocksize,
> -				     tree_csum);
> +				     tree_csum,
> +				     btrfs_super_csum_type(root->fs_info->super_copy));
>  		if (ret < 0)
>  			goto fail_out;
>  		else if (ret > 0)
> diff --git a/convert/common.c b/convert/common.c
> index c3702dc403ed..5ad9a91b9ab6 100644
> --- a/convert/common.c
> +++ b/convert/common.c
> @@ -63,12 +63,13 @@ static inline int write_temp_super(int fd, struct btrfs_super_block *sb,
>                                    u64 sb_bytenr)
>  {
>         u32 crc = ~(u32)0;
> +       u16 csum_type = btrfs_super_csum_type(sb);
>         int ret;
>  
> -       crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE,
> +       crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
>  			     (u8 *)&crc,
>                               BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> -       btrfs_csum_final(crc, &sb->csum[0]);
> +       btrfs_csum_final(csum_type, crc, &sb->csum[0]);
>         ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr);
>         if (ret < BTRFS_SUPER_INFO_SIZE)
>                 ret = (ret < 0 ? -errno : -EIO);
> diff --git a/disk-io.c b/disk-io.c
> index a0c37c569d58..7e538969c57a 100644
> --- a/disk-io.c
> +++ b/disk-io.c
> @@ -138,14 +138,26 @@ static void print_tree_block_error(struct btrfs_fs_info *fs_info,
>  	}
>  }
>  
> -u32 btrfs_csum_data(char *data, u8 *seed, size_t len)
> +u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len)
>  {
> -	return crc32c(*(u32*)seed, data, len);
> +	switch (csum_type) {
> +	case BTRFS_CSUM_TYPE_CRC32:
> +		return crc32c(*(u32*)seed, data, len);
> +	default: /* Not reached */
> +		return ~(u32)0;
> +	}
> +
>  }
>  
> -void btrfs_csum_final(u32 crc, u8 *result)
> +void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result)
>  {
> -	put_unaligned_le32(~crc, result);
> +	switch (csum_type) {
> +	case BTRFS_CSUM_TYPE_CRC32:
> +		put_unaligned_le32(~crc, result);
> +		break;
> +	default: /* Not reached */
> +		break;
> +	}
>  }
>  
>  static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
> @@ -156,8 +168,9 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
>  	u32 crc = ~(u32)0;
>  
>  	len = buf->len - BTRFS_CSUM_SIZE;
> -	crc = btrfs_csum_data(buf->data + BTRFS_CSUM_SIZE, (u8 *)&crc, len);
> -	btrfs_csum_final(crc, result);
> +	crc = btrfs_csum_data(csum_type, buf->data + BTRFS_CSUM_SIZE,
> +			      (u8 *)&crc, len);
> +	btrfs_csum_final(csum_type, crc, result);
>  
>  	if (verify) {
>  		if (memcmp_extent_buffer(buf, result, 0, csum_size)) {
> @@ -1376,9 +1389,10 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
>  	csum_size = btrfs_csum_sizes[csum_type];
>  
>  	crc = ~(u32)0;
> -	crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
> +	crc = btrfs_csum_data(csum_type,(char *)sb + BTRFS_CSUM_SIZE,
> +			      (u8 *)&crc,
>  			      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> -	btrfs_csum_final(crc, result);
> +	btrfs_csum_final(csum_type, crc, result);
>  
>  	if (memcmp(result, sb->csum, csum_size)) {
>  		error("superblock checksum mismatch");
> @@ -1616,6 +1630,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
>  	u64 bytenr;
>  	u32 crc;
>  	int i, ret;
> +	u16 csum_type = btrfs_super_csum_type(sb);
>  
>  	/*
>  	 * We need to write super block after all metadata written.
> @@ -1631,9 +1646,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
>  	if (fs_info->super_bytenr != BTRFS_SUPER_INFO_OFFSET) {
>  		btrfs_set_super_bytenr(sb, fs_info->super_bytenr);
>  		crc = ~(u32)0;
> -		crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
> +		crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
>  				      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> -		btrfs_csum_final(crc, &sb->csum[0]);
> +		btrfs_csum_final(csum_type, crc, &sb->csum[0]);
>  
>  		/*
>  		 * super_copy is BTRFS_SUPER_INFO_SIZE bytes and is
> @@ -1667,9 +1682,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
>  		btrfs_set_super_bytenr(sb, bytenr);
>  
>  		crc = ~(u32)0;
> -		crc = btrfs_csum_data((char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
> +		crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
>  				      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
> -		btrfs_csum_final(crc, &sb->csum[0]);
> +		btrfs_csum_final(csum_type, crc, &sb->csum[0]);
>  
>  		/*
>  		 * super_copy is BTRFS_SUPER_INFO_SIZE bytes and is
> diff --git a/disk-io.h b/disk-io.h
> index 92c87f28f8b2..4b5e9ea86385 100644
> --- a/disk-io.h
> +++ b/disk-io.h
> @@ -186,8 +186,8 @@ int btrfs_free_fs_root(struct btrfs_root *root);
>  void btrfs_mark_buffer_dirty(struct extent_buffer *buf);
>  int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid);
>  int btrfs_set_buffer_uptodate(struct extent_buffer *buf);
> -u32 btrfs_csum_data(char *data, u8 *seed, size_t len);
> -void btrfs_csum_final(u32 crc, u8 *result);
> +u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len);
> +void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result);
>  
>  int btrfs_open_device(struct btrfs_device *dev);
>  int csum_tree_block_size(struct extent_buffer *buf, u16 csum_sectorsize,
> diff --git a/file-item.c b/file-item.c
> index 5f6548e9a74f..78fdcecd0bab 100644
> --- a/file-item.c
> +++ b/file-item.c
> @@ -202,6 +202,9 @@ int btrfs_csum_file_block(struct btrfs_trans_handle *trans,
>  	u16 csum_size =
>  		btrfs_super_csum_size(root->fs_info->super_copy);
>  
> +	u16 csum_type =
> +		btrfs_super_csum_type(root->fs_info->super_copy);
> +
>  	path = btrfs_alloc_path();
>  	if (!path)
>  		return -ENOMEM;
> @@ -312,8 +315,8 @@ csum:
>  	item = (struct btrfs_csum_item *)((unsigned char *)item +
>  					  csum_offset * csum_size);
>  found:
> -	csum_result = btrfs_csum_data(data, (u8 *)&csum_result, len);
> -	btrfs_csum_final(csum_result, (u8 *)&csum_result);
> +	csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len);
> +	btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result);
>  	if (csum_result == 0) {
>  		printk("csum result is 0 for block %llu\n",
>  		       (unsigned long long)bytenr);
> diff --git a/free-space-cache.c b/free-space-cache.c
> index e872eb6a00db..8a57f86dc650 100644
> --- a/free-space-cache.c
> +++ b/free-space-cache.c
> @@ -213,7 +213,7 @@ static int io_ctl_check_crc(struct io_ctl *io_ctl, int index)
>  	io_ctl_map_page(io_ctl, 0);
>  	crc = crc32c(crc, io_ctl->orig + offset,
>  			io_ctl->root->fs_info->sectorsize - offset);
> -	btrfs_csum_final(crc, (u8 *)&crc);
> +	put_unaligned_le32(~crc, (u8 *)&crc);
>  	if (val != crc) {
>  		printk("btrfs: csum mismatch on free space cache\n");
>  		io_ctl_unmap_page(io_ctl);
> diff --git a/image/main.c b/image/main.c
> index 28ef1609b5ff..0c8ffede56f5 100644
> --- a/image/main.c
> +++ b/image/main.c
> @@ -124,7 +124,7 @@ static void csum_block(u8 *buf, size_t len)
>  	u8 result[btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]];
>  	u32 crc = ~(u32)0;
>  	crc = crc32c(crc, buf + BTRFS_CSUM_SIZE, len - BTRFS_CSUM_SIZE);
> -	btrfs_csum_final(crc, result);
> +	put_unaligned_le32(~crc, result);
>  	memcpy(buf, result, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]);
>  }
>  
> 

[PATCH v2 06/11] btrfs-progs: simplify update_block_csum() in btrfs-sb-mod.c

[Johannes Thumshirn]

update_block_csum() in btrfs-sb-mod.c is always called with the 'is_sb'
argument set to 1.

Get rid of the special case for is_sb == 0.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
---
 btrfs-sb-mod.c | 19 ++++++-------------
 1 file changed, 6 insertions(+), 13 deletions(-)

diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
index d9630f187d0f..9e64b34d2b8f 100644
--- a/btrfs-sb-mod.c
+++ b/btrfs-sb-mod.c
@@ -46,24 +46,17 @@ static int check_csum_superblock(void *sb)
 	return !memcmp(sb, &result, csum_size);
 }
 
-static void update_block_csum(void *block, int is_sb)
+static void update_block_csum(void *block)
 {
 	u8 result[csum_size];
 	struct btrfs_header *hdr;
 	u32 crc = ~(u32)0;
 	u16 csum_type = btrfs_super_csum_type(block);
 
-	if (is_sb) {
-		crc = btrfs_csum_data(csum_type,
-				      (char *)block + BTRFS_CSUM_SIZE,
-				      (u8 *)&crc,
-				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-	} else {
-		crc = btrfs_csum_data(csum_type,
-				      (char *)block + BTRFS_CSUM_SIZE,
-				      (u8 *)&crc,
-				BLOCKSIZE - BTRFS_CSUM_SIZE);
-	}
+	crc = btrfs_csum_data(csum_type, (char *)block + BTRFS_CSUM_SIZE,
+			      (u8 *)&crc,
+			      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+
 	btrfs_csum_final(csum_type, crc, result);
 	memset(block, 0, BTRFS_CSUM_SIZE);
 	hdr = (struct btrfs_header *)block;
@@ -354,7 +347,7 @@ int main(int argc, char **argv) {
 
 	if (changed) {
 		printf("Update csum\n");
-		update_block_csum(buf, 1);
+		update_block_csum(buf);
 		ret = pwrite(fd, buf, BLOCKSIZE, off);
 		if (ret <= 0) {
 			printf("pwrite error %d at offset %llu\n",
-- 
2.16.4

[PATCH v2 07/11] btrfs-progs: update checksumming api

[Johannes Thumshirn]

From: David Sterba <dsterba@suse.com>

[XXX Not done!!! ]

This is based on a patch by David Sterba.

Signed-off-by: David Sterba <dsterba@suse.com>
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 btrfs-sb-mod.c              | 21 ++++++----------
 check/main.c                | 20 +++++++--------
 cmds/inspect-dump-super.c   |  9 +++----
 cmds/rescue-chunk-recover.c | 12 ++++++---
 convert/common.c            |  9 +++----
 disk-io.c                   | 59 ++++++++++++++++++---------------------------
 disk-io.h                   |  3 +--
 file-item.c                 |  8 +++---
 8 files changed, 60 insertions(+), 81 deletions(-)

diff --git a/btrfs-sb-mod.c b/btrfs-sb-mod.c
index 9e64b34d2b8f..105b556b0cf1 100644
--- a/btrfs-sb-mod.c
+++ b/btrfs-sb-mod.c
@@ -33,31 +33,24 @@ static int csum_size;
 
 static int check_csum_superblock(void *sb)
 {
-	u8 result[csum_size];
-	u32 crc = ~(u32)0;
+	u8 result[BTRFS_CSUM_SIZE];
 	u16 csum_type = btrfs_super_csum_type(sb);
 
-	crc = btrfs_csum_data(csum_type,
-			      (char *)sb + BTRFS_CSUM_SIZE,
-				(u8 *)&crc,
-				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-	btrfs_csum_final(csum_type, crc, result);
+	btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
+			result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
 
-	return !memcmp(sb, &result, csum_size);
+	return !memcmp(sb, result, csum_size);
 }
 
 static void update_block_csum(void *block)
 {
-	u8 result[csum_size];
+	u8 result[BTRFS_CSUM_SIZE];
 	struct btrfs_header *hdr;
-	u32 crc = ~(u32)0;
 	u16 csum_type = btrfs_super_csum_type(block);
 
-	crc = btrfs_csum_data(csum_type, (char *)block + BTRFS_CSUM_SIZE,
-			      (u8 *)&crc,
-			      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+	btrfs_csum_data(csum_type, (char *)block + BTRFS_CSUM_SIZE,
+			result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
 
-	btrfs_csum_final(csum_type, crc, result);
 	memset(block, 0, BTRFS_CSUM_SIZE);
 	hdr = (struct btrfs_header *)block;
 	memcpy(&hdr->csum, result, csum_size);
diff --git a/check/main.c b/check/main.c
index d2fdf48b89c5..22045a96ebf1 100644
--- a/check/main.c
+++ b/check/main.c
@@ -5582,10 +5582,10 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
 	u64 offset = 0;
 	u16 csum_size = btrfs_super_csum_size(fs_info->super_copy);
 	u16 csum_type = btrfs_super_csum_type(fs_info->super_copy);
-	char *data;
+	u8 *data;
 	unsigned long csum_offset;
-	u32 csum;
-	u32 csum_expected;
+	u8 result[BTRFS_CSUM_SIZE];
+	u8 csum_expected[BTRFS_CSUM_SIZE];
 	u64 read_len;
 	u64 data_checked = 0;
 	u64 tmp;
@@ -5611,7 +5611,7 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
 		for (mirror = 1; mirror <= num_copies; mirror++) {
 			read_len = num_bytes - offset;
 			/* read as much space once a time */
-			ret = read_extent_data(fs_info, data + offset,
+			ret = read_extent_data(fs_info, (char *)data + offset,
 					bytenr + offset, &read_len, mirror);
 			if (ret)
 				goto out;
@@ -5619,24 +5619,22 @@ static int check_extent_csums(struct btrfs_root *root, u64 bytenr,
 			data_checked = 0;
 			/* verify every 4k data's checksum */
 			while (data_checked < read_len) {
-				csum = ~(u32)0;
 				tmp = offset + data_checked;
 
-				csum = btrfs_csum_data(csum_type,
-						       (char *)data + tmp,
-						(u8 *)&csum, fs_info->sectorsize);
-				btrfs_csum_final(csum_type, csum, (u8 *)&csum);
+				btrfs_csum_data(csum_type, data + tmp,
+						result, fs_info->sectorsize);
 
 				csum_offset = leaf_offset +
 					 tmp / fs_info->sectorsize * csum_size;
 				read_extent_buffer(eb, (char *)&csum_expected,
 						   csum_offset, csum_size);
-				if (csum != csum_expected) {
+				if (memcmp(result, csum_expected, csum_size) != 0) {
 					csum_mismatch = true;
+					/* FIXME: format */
 					fprintf(stderr,
 			"mirror %d bytenr %llu csum %u expected csum %u\n",
 						mirror, bytenr + tmp,
-						csum, csum_expected);
+						result[0], csum_expected[0]);
 				}
 				data_checked += fs_info->sectorsize;
 			}
diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
index 40019a6670ef..58bf82b0bbd3 100644
--- a/cmds/inspect-dump-super.c
+++ b/cmds/inspect-dump-super.c
@@ -38,14 +38,11 @@
 static int check_csum_sblock(void *sb, int csum_size, u16 csum_type)
 {
 	u8 result[BTRFS_CSUM_SIZE];
-	u32 crc = ~(u32)0;
 
-	crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
-				(u8 *)&crc,
-				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-	btrfs_csum_final(csum_type, crc, result);
+	btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE,
+			result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
 
-	return !memcmp(sb, &result, csum_size);
+	return !memcmp(sb, result, csum_size);
 }
 
 static void print_sys_chunk_array(struct btrfs_super_block *sb)
diff --git a/cmds/rescue-chunk-recover.c b/cmds/rescue-chunk-recover.c
index cddae471f50c..4b6426ae0346 100644
--- a/cmds/rescue-chunk-recover.c
+++ b/cmds/rescue-chunk-recover.c
@@ -1892,7 +1892,11 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum,
 {
 	char *data;
 	int ret = 0;
-	u32 csum_result = ~(u32)0;
+	u8 result[BTRFS_CSUM_SIZE];
+	int csum_size = 0;
+	u8 expected_csum[BTRFS_CSUM_SIZE];
+
+	ASSERT(0);
 
 	data = malloc(len);
 	if (!data)
@@ -1903,9 +1907,9 @@ static int check_one_csum(int fd, u64 start, u32 len, u32 tree_csum,
 		goto out;
 	}
 	ret = 0;
-	csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len);
-	btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result);
-	if (csum_result != tree_csum)
+	put_unaligned_le32(tree_csum, expected_csum);
+	btrfs_csum_data(csum_type, (u8 *)data, result, len);
+	if (memcmp(result, expected_csum, csum_size) != 0)
 		ret = 1;
 out:
 	free(data);
diff --git a/convert/common.c b/convert/common.c
index 5ad9a91b9ab6..7936f8f10b29 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -62,14 +62,13 @@ static int reserve_free_space(struct cache_tree *free_tree, u64 len,
 static inline int write_temp_super(int fd, struct btrfs_super_block *sb,
                                   u64 sb_bytenr)
 {
-       u32 crc = ~(u32)0;
+       u8 result[BTRFS_CSUM_SIZE];
        u16 csum_type = btrfs_super_csum_type(sb);
        int ret;
 
-       crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE,
-			     (u8 *)&crc,
-                             BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-       btrfs_csum_final(csum_type, crc, &sb->csum[0]);
+       btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE,
+		       result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+       memcpy(&sb->csum[0], result, BTRFS_CSUM_SIZE);
        ret = pwrite(fd, sb, BTRFS_SUPER_INFO_SIZE, sb_bytenr);
        if (ret < BTRFS_SUPER_INFO_SIZE)
                ret = (ret < 0 ? -errno : -EIO);
diff --git a/disk-io.c b/disk-io.c
index 7e538969c57a..810c2e14294a 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -138,26 +138,22 @@ static void print_tree_block_error(struct btrfs_fs_info *fs_info,
 	}
 }
 
-u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len)
+int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len)
 {
-	switch (csum_type) {
-	case BTRFS_CSUM_TYPE_CRC32:
-		return crc32c(*(u32*)seed, data, len);
-	default: /* Not reached */
-		return ~(u32)0;
-	}
-
-}
+	u32 crc = ~(u32)0;
+	memset(out, 0, BTRFS_CSUM_SIZE);
 
-void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result)
-{
 	switch (csum_type) {
 	case BTRFS_CSUM_TYPE_CRC32:
-		put_unaligned_le32(~crc, result);
-		break;
-	default: /* Not reached */
-		break;
+		crc = crc32c(crc, data, len);
+		put_unaligned_le32(~crc, out);
+		return 0;
+	default:
+		fprintf(stderr, "ERROR: unknown csum type: %d\n", csum_type);
+		ASSERT(0);
 	}
+
+	return -1;
 }
 
 static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
@@ -165,20 +161,19 @@ static int __csum_tree_block_size(struct extent_buffer *buf, u16 csum_size,
 {
 	u8 result[BTRFS_CSUM_SIZE];
 	u32 len;
-	u32 crc = ~(u32)0;
 
 	len = buf->len - BTRFS_CSUM_SIZE;
-	crc = btrfs_csum_data(csum_type, buf->data + BTRFS_CSUM_SIZE,
-			      (u8 *)&crc, len);
-	btrfs_csum_final(csum_type, crc, result);
+	btrfs_csum_data(csum_type, (u8 *)buf->data + BTRFS_CSUM_SIZE,
+			result, len);
 
 	if (verify) {
 		if (memcmp_extent_buffer(buf, result, 0, csum_size)) {
+			/* FIXME: format */
 			if (!silent)
 				printk("checksum verify failed on %llu found %08X wanted %08X\n",
 				       (unsigned long long)buf->start,
-				       *((u32 *)result),
-				       *((u32*)(char *)buf->data));
+				       result[0],
+				       buf->data[0]);
 			return 1;
 		}
 	} else {
@@ -1367,7 +1362,6 @@ struct btrfs_root *open_ctree_fd(int fp, const char *path, u64 sb_bytenr,
 int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
 {
 	u8 result[BTRFS_CSUM_SIZE];
-	u32 crc;
 	u16 csum_type;
 	int csum_size;
 	u8 *metadata_uuid;
@@ -1388,11 +1382,8 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
 	}
 	csum_size = btrfs_csum_sizes[csum_type];
 
-	crc = ~(u32)0;
-	crc = btrfs_csum_data(csum_type,(char *)sb + BTRFS_CSUM_SIZE,
-			      (u8 *)&crc,
-			      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-	btrfs_csum_final(csum_type, crc, result);
+	btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE,
+			result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
 
 	if (memcmp(result, sb->csum, csum_size)) {
 		error("superblock checksum mismatch");
@@ -1628,7 +1619,7 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
 			    struct btrfs_device *device)
 {
 	u64 bytenr;
-	u32 crc;
+	u8 result[BTRFS_CSUM_SIZE];
 	int i, ret;
 	u16 csum_type = btrfs_super_csum_type(sb);
 
@@ -1645,10 +1636,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
 	}
 	if (fs_info->super_bytenr != BTRFS_SUPER_INFO_OFFSET) {
 		btrfs_set_super_bytenr(sb, fs_info->super_bytenr);
-		crc = ~(u32)0;
-		crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
-				      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-		btrfs_csum_final(csum_type, crc, &sb->csum[0]);
+		btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE, result,
+				BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
+		memcpy(&sb->csum[0], result, BTRFS_CSUM_SIZE);
 
 		/*
 		 * super_copy is BTRFS_SUPER_INFO_SIZE bytes and is
@@ -1681,10 +1671,9 @@ static int write_dev_supers(struct btrfs_fs_info *fs_info,
 
 		btrfs_set_super_bytenr(sb, bytenr);
 
-		crc = ~(u32)0;
-		crc = btrfs_csum_data(csum_type, (char *)sb + BTRFS_CSUM_SIZE, (u8 *)&crc,
+		btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE, result,
 				      BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
-		btrfs_csum_final(csum_type, crc, &sb->csum[0]);
+		memcpy(&sb->csum[0], result, BTRFS_CSUM_SIZE);
 
 		/*
 		 * super_copy is BTRFS_SUPER_INFO_SIZE bytes and is
diff --git a/disk-io.h b/disk-io.h
index 4b5e9ea86385..545cacda9a79 100644
--- a/disk-io.h
+++ b/disk-io.h
@@ -186,8 +186,7 @@ int btrfs_free_fs_root(struct btrfs_root *root);
 void btrfs_mark_buffer_dirty(struct extent_buffer *buf);
 int btrfs_buffer_uptodate(struct extent_buffer *buf, u64 parent_transid);
 int btrfs_set_buffer_uptodate(struct extent_buffer *buf);
-u32 btrfs_csum_data(u16 csum_type, char *data, u8 *seed, size_t len);
-void btrfs_csum_final(u16 csum_type, u32 crc, u8 *result);
+int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len);
 
 int btrfs_open_device(struct btrfs_device *dev);
 int csum_tree_block_size(struct extent_buffer *buf, u16 csum_sectorsize,
diff --git a/file-item.c b/file-item.c
index 78fdcecd0bab..c6e9d212bcab 100644
--- a/file-item.c
+++ b/file-item.c
@@ -195,7 +195,7 @@ int btrfs_csum_file_block(struct btrfs_trans_handle *trans,
 	struct btrfs_csum_item *item;
 	struct extent_buffer *leaf = NULL;
 	u64 csum_offset;
-	u32 csum_result = ~(u32)0;
+	u8 csum_result[BTRFS_CSUM_SIZE];
 	u32 sectorsize = root->fs_info->sectorsize;
 	u32 nritems;
 	u32 ins_size;
@@ -315,14 +315,14 @@ csum:
 	item = (struct btrfs_csum_item *)((unsigned char *)item +
 					  csum_offset * csum_size);
 found:
-	csum_result = btrfs_csum_data(csum_type, data, (u8 *)&csum_result, len);
-	btrfs_csum_final(csum_type, csum_result, (u8 *)&csum_result);
+	btrfs_csum_data(csum_type, (u8 *)data, csum_result, len);
+	/* FIXME: does not make sense for non-crc32c */
 	if (csum_result == 0) {
 		printk("csum result is 0 for block %llu\n",
 		       (unsigned long long)bytenr);
 	}
 
-	write_extent_buffer(leaf, &csum_result, (unsigned long)item,
+	write_extent_buffer(leaf, csum_result, (unsigned long)item,
 			    csum_size);
 	btrfs_mark_buffer_dirty(path->nodes[0]);
 fail:
-- 
2.16.4

[PATCH v2 08/11] btrfs-progs: add option for checksum type to mkfs

[Johannes Thumshirn]

Add an option to mkfs to specify which checksum algorithm will be used for
the filesystem.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 convert/common.c |  2 +-
 mkfs/common.c    |  2 +-
 mkfs/common.h    |  2 ++
 mkfs/main.c      | 21 ++++++++++++++++++++-
 4 files changed, 24 insertions(+), 3 deletions(-)

diff --git a/convert/common.c b/convert/common.c
index 7936f8f10b29..8f5fdbf507a4 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -135,7 +135,7 @@ static int setup_temp_super(int fd, struct btrfs_mkfs_config *cfg,
 	super->__unused_leafsize = cpu_to_le32(cfg->nodesize);
 	btrfs_set_super_nodesize(super, cfg->nodesize);
 	btrfs_set_super_stripesize(super, cfg->stripesize);
-	btrfs_set_super_csum_type(super, BTRFS_CSUM_TYPE_CRC32);
+	btrfs_set_super_csum_type(super, cfg->csum_type);
 	btrfs_set_super_chunk_root(super, chunk_bytenr);
 	btrfs_set_super_cache_generation(super, -1);
 	btrfs_set_super_incompat_flags(super, cfg->features);
diff --git a/mkfs/common.c b/mkfs/common.c
index 9762391a8d2b..4a417bd7a306 100644
--- a/mkfs/common.c
+++ b/mkfs/common.c
@@ -202,7 +202,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	super.__unused_leafsize = cpu_to_le32(cfg->nodesize);
 	btrfs_set_super_nodesize(&super, cfg->nodesize);
 	btrfs_set_super_stripesize(&super, cfg->stripesize);
-	btrfs_set_super_csum_type(&super, BTRFS_CSUM_TYPE_CRC32);
+	btrfs_set_super_csum_type(&super, cfg->csum_type);
 	btrfs_set_super_chunk_root_generation(&super, 1);
 	btrfs_set_super_cache_generation(&super, -1);
 	btrfs_set_super_incompat_flags(&super, cfg->features);
diff --git a/mkfs/common.h b/mkfs/common.h
index 28912906d0a9..1ca71a4fcce5 100644
--- a/mkfs/common.h
+++ b/mkfs/common.h
@@ -53,6 +53,8 @@ struct btrfs_mkfs_config {
 	u64 features;
 	/* Size of the filesystem in bytes */
 	u64 num_bytes;
+	/* checksum algorithm to use */
+	enum btrfs_csum_type csum_type;
 
 	/* Output fields, set during creation */
 
diff --git a/mkfs/main.c b/mkfs/main.c
index 8dbec0717b89..075e7e331ab4 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -346,6 +346,7 @@ static void print_usage(int ret)
 	printf("\t--shrink                (with --rootdir) shrink the filled filesystem to minimal size\n");
 	printf("\t-K|--nodiscard          do not perform whole device TRIM\n");
 	printf("\t-f|--force              force overwrite of existing filesystem\n");
+	printf("\t-C|--checksum           checksum algorithm to use (default: crc32c)\n");
 	printf("  general:\n");
 	printf("\t-q|--quiet              no messages except errors\n");
 	printf("\t-V|--version            print the mkfs.btrfs version and exit\n");
@@ -380,6 +381,18 @@ static u64 parse_profile(const char *s)
 	return 0;
 }
 
+static enum btrfs_csum_type parse_csum_type(const char *s)
+{
+	if (strcasecmp(s, "crc32c") == 0) {
+		return BTRFS_CSUM_TYPE_CRC32;
+	} else {
+		error("unknown csum type %s", s);
+		exit(1);
+	}
+	/* not reached */
+	return 0;
+}
+
 static char *parse_label(const char *input)
 {
 	int len = strlen(input);
@@ -826,6 +839,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 	u64 features = BTRFS_MKFS_DEFAULT_FEATURES;
 	struct mkfs_allocation allocation = { 0 };
 	struct btrfs_mkfs_config mkfs_cfg;
+	enum btrfs_csum_type csum_type = BTRFS_CSUM_TYPE_CRC32;
 
 	crc32c_optimization_init();
 
@@ -835,6 +849,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 		static const struct option long_options[] = {
 			{ "alloc-start", required_argument, NULL, 'A'},
 			{ "byte-count", required_argument, NULL, 'b' },
+			{ "checksum", required_argument, NULL, 'C' },
 			{ "force", no_argument, NULL, 'f' },
 			{ "leafsize", required_argument, NULL, 'l' },
 			{ "label", required_argument, NULL, 'L'},
@@ -854,7 +869,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 			{ NULL, 0, NULL, 0}
 		};
 
-		c = getopt_long(argc, argv, "A:b:fl:n:s:m:d:L:O:r:U:VMKq",
+		c = getopt_long(argc, argv, "A:b:C:fl:n:s:m:d:L:O:r:U:VMKq",
 				long_options, NULL);
 		if (c < 0)
 			break;
@@ -932,6 +947,9 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 			case GETOPT_VAL_SHRINK:
 				shrink_rootdir = true;
 				break;
+			case 'C':
+				csum_type = parse_csum_type(optarg);
+				break;
 			case GETOPT_VAL_HELP:
 			default:
 				print_usage(c != GETOPT_VAL_HELP);
@@ -1170,6 +1188,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
 	mkfs_cfg.sectorsize = sectorsize;
 	mkfs_cfg.stripesize = stripesize;
 	mkfs_cfg.features = features;
+	mkfs_cfg.csum_type = csum_type;
 
 	ret = make_btrfs(fd, &mkfs_cfg);
 	if (ret) {
-- 
2.16.4

Re: [PATCH v2 08/11] btrfs-progs: add option for checksum type to mkfs

[Nikolay Borisov]

On 26.08.19 г. 14:48 ч., Johannes Thumshirn wrote:
> Add an option to mkfs to specify which checksum algorithm will be used for
> the filesystem.
> 
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>

Reviewed-by: Nikolay Borisov <nborisov@suse.com>

> ---
>  convert/common.c |  2 +-
>  mkfs/common.c    |  2 +-
>  mkfs/common.h    |  2 ++
>  mkfs/main.c      | 21 ++++++++++++++++++++-
>  4 files changed, 24 insertions(+), 3 deletions(-)
> 
> diff --git a/convert/common.c b/convert/common.c
> index 7936f8f10b29..8f5fdbf507a4 100644
> --- a/convert/common.c
> +++ b/convert/common.c
> @@ -135,7 +135,7 @@ static int setup_temp_super(int fd, struct btrfs_mkfs_config *cfg,
>  	super->__unused_leafsize = cpu_to_le32(cfg->nodesize);
>  	btrfs_set_super_nodesize(super, cfg->nodesize);
>  	btrfs_set_super_stripesize(super, cfg->stripesize);
> -	btrfs_set_super_csum_type(super, BTRFS_CSUM_TYPE_CRC32);
> +	btrfs_set_super_csum_type(super, cfg->csum_type);
>  	btrfs_set_super_chunk_root(super, chunk_bytenr);
>  	btrfs_set_super_cache_generation(super, -1);
>  	btrfs_set_super_incompat_flags(super, cfg->features);
> diff --git a/mkfs/common.c b/mkfs/common.c
> index 9762391a8d2b..4a417bd7a306 100644
> --- a/mkfs/common.c
> +++ b/mkfs/common.c
> @@ -202,7 +202,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
>  	super.__unused_leafsize = cpu_to_le32(cfg->nodesize);
>  	btrfs_set_super_nodesize(&super, cfg->nodesize);
>  	btrfs_set_super_stripesize(&super, cfg->stripesize);
> -	btrfs_set_super_csum_type(&super, BTRFS_CSUM_TYPE_CRC32);
> +	btrfs_set_super_csum_type(&super, cfg->csum_type);
>  	btrfs_set_super_chunk_root_generation(&super, 1);
>  	btrfs_set_super_cache_generation(&super, -1);
>  	btrfs_set_super_incompat_flags(&super, cfg->features);
> diff --git a/mkfs/common.h b/mkfs/common.h
> index 28912906d0a9..1ca71a4fcce5 100644
> --- a/mkfs/common.h
> +++ b/mkfs/common.h
> @@ -53,6 +53,8 @@ struct btrfs_mkfs_config {
>  	u64 features;
>  	/* Size of the filesystem in bytes */
>  	u64 num_bytes;
> +	/* checksum algorithm to use */
> +	enum btrfs_csum_type csum_type;
>  
>  	/* Output fields, set during creation */
>  
> diff --git a/mkfs/main.c b/mkfs/main.c
> index 8dbec0717b89..075e7e331ab4 100644
> --- a/mkfs/main.c
> +++ b/mkfs/main.c
> @@ -346,6 +346,7 @@ static void print_usage(int ret)
>  	printf("\t--shrink                (with --rootdir) shrink the filled filesystem to minimal size\n");
>  	printf("\t-K|--nodiscard          do not perform whole device TRIM\n");
>  	printf("\t-f|--force              force overwrite of existing filesystem\n");
> +	printf("\t-C|--checksum           checksum algorithm to use (default: crc32c)\n");
>  	printf("  general:\n");
>  	printf("\t-q|--quiet              no messages except errors\n");
>  	printf("\t-V|--version            print the mkfs.btrfs version and exit\n");
> @@ -380,6 +381,18 @@ static u64 parse_profile(const char *s)
>  	return 0;
>  }
>  
> +static enum btrfs_csum_type parse_csum_type(const char *s)
> +{
> +	if (strcasecmp(s, "crc32c") == 0) {
> +		return BTRFS_CSUM_TYPE_CRC32;
> +	} else {
> +		error("unknown csum type %s", s);
> +		exit(1);
> +	}
> +	/* not reached */
> +	return 0;
> +}
> +
>  static char *parse_label(const char *input)
>  {
>  	int len = strlen(input);
> @@ -826,6 +839,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
>  	u64 features = BTRFS_MKFS_DEFAULT_FEATURES;
>  	struct mkfs_allocation allocation = { 0 };
>  	struct btrfs_mkfs_config mkfs_cfg;
> +	enum btrfs_csum_type csum_type = BTRFS_CSUM_TYPE_CRC32;
>  
>  	crc32c_optimization_init();
>  
> @@ -835,6 +849,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
>  		static const struct option long_options[] = {
>  			{ "alloc-start", required_argument, NULL, 'A'},
>  			{ "byte-count", required_argument, NULL, 'b' },
> +			{ "checksum", required_argument, NULL, 'C' },
>  			{ "force", no_argument, NULL, 'f' },
>  			{ "leafsize", required_argument, NULL, 'l' },
>  			{ "label", required_argument, NULL, 'L'},
> @@ -854,7 +869,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
>  			{ NULL, 0, NULL, 0}
>  		};
>  
> -		c = getopt_long(argc, argv, "A:b:fl:n:s:m:d:L:O:r:U:VMKq",
> +		c = getopt_long(argc, argv, "A:b:C:fl:n:s:m:d:L:O:r:U:VMKq",
>  				long_options, NULL);
>  		if (c < 0)
>  			break;
> @@ -932,6 +947,9 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
>  			case GETOPT_VAL_SHRINK:
>  				shrink_rootdir = true;
>  				break;
> +			case 'C':
> +				csum_type = parse_csum_type(optarg);
> +				break;
>  			case GETOPT_VAL_HELP:
>  			default:
>  				print_usage(c != GETOPT_VAL_HELP);
> @@ -1170,6 +1188,7 @@ int BOX_MAIN(mkfs)(int argc, char **argv)
>  	mkfs_cfg.sectorsize = sectorsize;
>  	mkfs_cfg.stripesize = stripesize;
>  	mkfs_cfg.features = features;
> +	mkfs_cfg.csum_type = csum_type;
>  
>  	ret = make_btrfs(fd, &mkfs_cfg);
>  	if (ret) {
> 

[PATCH v2 09/11] btrfs-progs: add xxhash sources

[Johannes Thumshirn]

From: David Sterba <dsterba@suse.com>

git://github.com/Cyan4973/xxHash

Signed-off-by: David Sterba <dsterba@suse.com>
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 crypto/xxhash.c | 1024 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 crypto/xxhash.h |  445 ++++++++++++++++++++++++
 2 files changed, 1469 insertions(+)
 create mode 100644 crypto/xxhash.c
 create mode 100644 crypto/xxhash.h

diff --git a/crypto/xxhash.c b/crypto/xxhash.c
new file mode 100644
index 000000000000..af9d02795ac6
--- /dev/null
+++ b/crypto/xxhash.c
@@ -0,0 +1,1024 @@
+/*
+*  xxHash - Fast Hash algorithm
+*  Copyright (C) 2012-2016, Yann Collet
+*
+*  BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
+*
+*  Redistribution and use in source and binary forms, with or without
+*  modification, are permitted provided that the following conditions are
+*  met:
+*
+*  * Redistributions of source code must retain the above copyright
+*  notice, this list of conditions and the following disclaimer.
+*  * Redistributions in binary form must reproduce the above
+*  copyright notice, this list of conditions and the following disclaimer
+*  in the documentation and/or other materials provided with the
+*  distribution.
+*
+*  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+*  "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+*  LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+*  A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+*  OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+*  SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+*  LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+*  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+*  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+*  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+*  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+*
+*  You can contact the author at :
+*  - xxHash homepage: http://www.xxhash.com
+*  - xxHash source repository : https://github.com/Cyan4973/xxHash
+*/
+
+
+/* *************************************
+*  Tuning parameters
+***************************************/
+/*!XXH_FORCE_MEMORY_ACCESS :
+ * By default, access to unaligned memory is controlled by `memcpy()`, which is safe and portable.
+ * Unfortunately, on some target/compiler combinations, the generated assembly is sub-optimal.
+ * The below switch allow to select different access method for improved performance.
+ * Method 0 (default) : use `memcpy()`. Safe and portable.
+ * Method 1 : `__packed` statement. It depends on compiler extension (ie, not portable).
+ *            This method is safe if your compiler supports it, and *generally* as fast or faster than `memcpy`.
+ * Method 2 : direct access. This method doesn't depend on compiler but violate C standard.
+ *            It can generate buggy code on targets which do not support unaligned memory accesses.
+ *            But in some circumstances, it's the only known way to get the most performance (ie GCC + ARMv6)
+ * See http://stackoverflow.com/a/32095106/646947 for details.
+ * Prefer these methods in priority order (0 > 1 > 2)
+ */
+#ifndef XXH_FORCE_MEMORY_ACCESS   /* can be defined externally, on command line for example */
+#  if defined(__GNUC__) && ( defined(__ARM_ARCH_6__) || defined(__ARM_ARCH_6J__) \
+                        || defined(__ARM_ARCH_6K__) || defined(__ARM_ARCH_6Z__) \
+                        || defined(__ARM_ARCH_6ZK__) || defined(__ARM_ARCH_6T2__) )
+#    define XXH_FORCE_MEMORY_ACCESS 2
+#  elif (defined(__INTEL_COMPILER) && !defined(_WIN32)) || \
+  (defined(__GNUC__) && ( defined(__ARM_ARCH_7__) || defined(__ARM_ARCH_7A__) \
+                    || defined(__ARM_ARCH_7R__) || defined(__ARM_ARCH_7M__) \
+                    || defined(__ARM_ARCH_7S__) ))
+#    define XXH_FORCE_MEMORY_ACCESS 1
+#  endif
+#endif
+
+/*!XXH_ACCEPT_NULL_INPUT_POINTER :
+ * If input pointer is NULL, xxHash default behavior is to dereference it, triggering a segfault.
+ * When this macro is enabled, xxHash actively checks input for null pointer.
+ * It it is, result for null input pointers is the same as a null-length input.
+ */
+#ifndef XXH_ACCEPT_NULL_INPUT_POINTER   /* can be defined externally */
+#  define XXH_ACCEPT_NULL_INPUT_POINTER 0
+#endif
+
+/*!XXH_FORCE_ALIGN_CHECK :
+ * This is a minor performance trick, only useful with lots of very small keys.
+ * It means : check for aligned/unaligned input.
+ * The check costs one initial branch per hash;
+ * set it to 0 when the input is guaranteed to be aligned,
+ * or when alignment doesn't matter for performance.
+ */
+#ifndef XXH_FORCE_ALIGN_CHECK /* can be defined externally */
+#  if defined(__i386) || defined(_M_IX86) || defined(__x86_64__) || defined(_M_X64)
+#    define XXH_FORCE_ALIGN_CHECK 0
+#  else
+#    define XXH_FORCE_ALIGN_CHECK 1
+#  endif
+#endif
+
+
+/* *************************************
+*  Includes & Memory related functions
+***************************************/
+/*! Modify the local functions below should you wish to use some other memory routines
+*   for malloc(), free() */
+#include <stdlib.h>
+static void* XXH_malloc(size_t s) { return malloc(s); }
+static void  XXH_free  (void* p)  { free(p); }
+/*! and for memcpy() */
+#include <string.h>
+static void* XXH_memcpy(void* dest, const void* src, size_t size) { return memcpy(dest,src,size); }
+
+#include <assert.h>   /* assert */
+
+#define XXH_STATIC_LINKING_ONLY
+#include "xxhash.h"
+
+
+/* *************************************
+*  Compiler Specific Options
+***************************************/
+#ifdef _MSC_VER    /* Visual Studio */
+#  pragma warning(disable : 4127)      /* disable: C4127: conditional expression is constant */
+#  define XXH_FORCE_INLINE static __forceinline
+#  define XXH_NO_INLINE static __declspec(noinline)
+#else
+#  if defined (__cplusplus) || defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L   /* C99 */
+#    ifdef __GNUC__
+#      define XXH_FORCE_INLINE static inline __attribute__((always_inline))
+#      define XXH_NO_INLINE static __attribute__((noinline))
+#    else
+#      define XXH_FORCE_INLINE static inline
+#      define XXH_NO_INLINE static
+#    endif
+#  else
+#    define XXH_FORCE_INLINE static
+#    define XXH_NO_INLINE static
+#  endif /* __STDC_VERSION__ */
+#endif
+
+
+/* *************************************
+*  Basic Types
+***************************************/
+#ifndef MEM_MODULE
+# if !defined (__VMS) \
+  && (defined (__cplusplus) \
+  || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) )
+#   include <stdint.h>
+    typedef uint8_t  BYTE;
+    typedef uint16_t U16;
+    typedef uint32_t U32;
+# else
+    typedef unsigned char      BYTE;
+    typedef unsigned short     U16;
+    typedef unsigned int       U32;
+# endif
+#endif
+
+
+/* ===   Memory access   === */
+
+#if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==2))
+
+/* Force direct memory access. Only works on CPU which support unaligned memory access in hardware */
+static U32 XXH_read32(const void* memPtr) { return *(const U32*) memPtr; }
+
+#elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==1))
+
+/* __pack instructions are safer, but compiler specific, hence potentially problematic for some compilers */
+/* currently only defined for gcc and icc */
+typedef union { U32 u32; } __attribute__((packed)) unalign;
+static U32 XXH_read32(const void* ptr) { return ((const unalign*)ptr)->u32; }
+
+#else
+
+/* portable and safe solution. Generally efficient.
+ * see : http://stackoverflow.com/a/32095106/646947
+ */
+static U32 XXH_read32(const void* memPtr)
+{
+    U32 val;
+    memcpy(&val, memPtr, sizeof(val));
+    return val;
+}
+
+#endif   /* XXH_FORCE_DIRECT_MEMORY_ACCESS */
+
+
+/* ===   Endianess   === */
+typedef enum { XXH_bigEndian=0, XXH_littleEndian=1 } XXH_endianess;
+
+/* XXH_CPU_LITTLE_ENDIAN can be defined externally, for example on the compiler command line */
+#ifndef XXH_CPU_LITTLE_ENDIAN
+static int XXH_isLittleEndian(void)
+{
+    const union { U32 u; BYTE c[4]; } one = { 1 };   /* don't use static : performance detrimental  */
+    return one.c[0];
+}
+#   define XXH_CPU_LITTLE_ENDIAN   XXH_isLittleEndian()
+#endif
+
+
+
+
+/* ****************************************
+*  Compiler-specific Functions and Macros
+******************************************/
+#define XXH_GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__)
+
+/* Note : although _rotl exists for minGW (GCC under windows), performance seems poor */
+#if defined(_MSC_VER)
+#  define XXH_rotl32(x,r) _rotl(x,r)
+#  define XXH_rotl64(x,r) _rotl64(x,r)
+#else
+#  define XXH_rotl32(x,r) (((x) << (r)) | ((x) >> (32 - (r))))
+#  define XXH_rotl64(x,r) (((x) << (r)) | ((x) >> (64 - (r))))
+#endif
+
+#if defined(_MSC_VER)     /* Visual Studio */
+#  define XXH_swap32 _byteswap_ulong
+#elif XXH_GCC_VERSION >= 403
+#  define XXH_swap32 __builtin_bswap32
+#else
+static U32 XXH_swap32 (U32 x)
+{
+    return  ((x << 24) & 0xff000000 ) |
+            ((x <<  8) & 0x00ff0000 ) |
+            ((x >>  8) & 0x0000ff00 ) |
+            ((x >> 24) & 0x000000ff );
+}
+#endif
+
+
+/* ***************************
+*  Memory reads
+*****************************/
+typedef enum { XXH_aligned, XXH_unaligned } XXH_alignment;
+
+XXH_FORCE_INLINE U32 XXH_readLE32(const void* ptr)
+{
+    return XXH_CPU_LITTLE_ENDIAN ? XXH_read32(ptr) : XXH_swap32(XXH_read32(ptr));
+}
+
+static U32 XXH_readBE32(const void* ptr)
+{
+    return XXH_CPU_LITTLE_ENDIAN ? XXH_swap32(XXH_read32(ptr)) : XXH_read32(ptr);
+}
+
+XXH_FORCE_INLINE U32
+XXH_readLE32_align(const void* ptr, XXH_alignment align)
+{
+    if (align==XXH_unaligned) {
+        return XXH_readLE32(ptr);
+    } else {
+        return XXH_CPU_LITTLE_ENDIAN ? *(const U32*)ptr : XXH_swap32(*(const U32*)ptr);
+    }
+}
+
+
+/* *************************************
+*  Macros
+***************************************/
+#define XXH_STATIC_ASSERT(c)  { enum { XXH_sa = 1/(int)(!!(c)) }; }  /* use after variable declarations */
+XXH_PUBLIC_API unsigned XXH_versionNumber (void) { return XXH_VERSION_NUMBER; }
+
+
+/* *******************************************************************
+*  32-bit hash functions
+*********************************************************************/
+static const U32 PRIME32_1 = 2654435761U;   /* 0b10011110001101110111100110110001 */
+static const U32 PRIME32_2 = 2246822519U;   /* 0b10000101111010111100101001110111 */
+static const U32 PRIME32_3 = 3266489917U;   /* 0b11000010101100101010111000111101 */
+static const U32 PRIME32_4 =  668265263U;   /* 0b00100111110101001110101100101111 */
+static const U32 PRIME32_5 =  374761393U;   /* 0b00010110010101100110011110110001 */
+
+static U32 XXH32_round(U32 acc, U32 input)
+{
+    acc += input * PRIME32_2;
+    acc  = XXH_rotl32(acc, 13);
+    acc *= PRIME32_1;
+#if defined(__GNUC__) && defined(__SSE4_1__) && !defined(XXH_ENABLE_AUTOVECTORIZE)
+    /* UGLY HACK:
+     * This inline assembly hack forces acc into a normal register. This is the
+     * only thing that prevents GCC and Clang from autovectorizing the XXH32 loop
+     * (pragmas and attributes don't work for some resason) without globally
+     * disabling SSE4.1.
+     *
+     * The reason we want to avoid vectorization is because despite working on
+     * 4 integers at a time, there are multiple factors slowing XXH32 down on
+     * SSE4:
+     * - There's a ridiculous amount of lag from pmulld (10 cycles of latency on newer chips!)
+     *   making it slightly slower to multiply four integers at once compared to four
+     *   integers independently. Even when pmulld was fastest, Sandy/Ivy Bridge, it is
+     *   still not worth it to go into SSE just to multiply unless doing a long operation.
+     *
+     * - Four instructions are required to rotate,
+     *      movqda tmp,  v // not required with VEX encoding
+     *      pslld  tmp, 13 // tmp <<= 13
+     *      psrld  v,   19 // x >>= 19
+     *      por    v,  tmp // x |= tmp
+     *   compared to one for scalar:
+     *      roll   v, 13    // reliably fast across the board
+     *      shldl  v, v, 13 // Sandy Bridge and later prefer this for some reason
+     *
+     * - Instruction level parallelism is actually more beneficial here because the
+     *   SIMD actually serializes this operation: While v1 is rotating, v2 can load data,
+     *   while v3 can multiply. SSE forces them to operate together.
+     *
+     * How this hack works:
+     * __asm__(""       // Declare an assembly block but don't declare any instructions
+     *          :       // However, as an Input/Output Operand,
+     *          "+r"    // constrain a read/write operand (+) as a general purpose register (r).
+     *          (acc)   // and set acc as the operand
+     * );
+     *
+     * Because of the 'r', the compiler has promised that seed will be in a
+     * general purpose register and the '+' says that it will be 'read/write',
+     * so it has to assume it has changed. It is like volatile without all the
+     * loads and stores.
+     *
+     * Since the argument has to be in a normal register (not an SSE register),
+     * each time XXH32_round is called, it is impossible to vectorize. */
+    __asm__("" : "+r" (acc));
+#endif
+    return acc;
+}
+
+/* mix all bits */
+static U32 XXH32_avalanche(U32 h32)
+{
+    h32 ^= h32 >> 15;
+    h32 *= PRIME32_2;
+    h32 ^= h32 >> 13;
+    h32 *= PRIME32_3;
+    h32 ^= h32 >> 16;
+    return(h32);
+}
+
+#define XXH_get32bits(p) XXH_readLE32_align(p, align)
+
+static U32
+XXH32_finalize(U32 h32, const void* ptr, size_t len, XXH_alignment align)
+
+{
+    const BYTE* p = (const BYTE*)ptr;
+
+#define PROCESS1               \
+    h32 += (*p++) * PRIME32_5; \
+    h32 = XXH_rotl32(h32, 11) * PRIME32_1 ;
+
+#define PROCESS4                         \
+    h32 += XXH_get32bits(p) * PRIME32_3; \
+    p+=4;                                \
+    h32  = XXH_rotl32(h32, 17) * PRIME32_4 ;
+
+    switch(len&15)  /* or switch(bEnd - p) */
+    {
+      case 12:      PROCESS4;
+                    /* fallthrough */
+      case 8:       PROCESS4;
+                    /* fallthrough */
+      case 4:       PROCESS4;
+                    return XXH32_avalanche(h32);
+
+      case 13:      PROCESS4;
+                    /* fallthrough */
+      case 9:       PROCESS4;
+                    /* fallthrough */
+      case 5:       PROCESS4;
+                    PROCESS1;
+                    return XXH32_avalanche(h32);
+
+      case 14:      PROCESS4;
+                    /* fallthrough */
+      case 10:      PROCESS4;
+                    /* fallthrough */
+      case 6:       PROCESS4;
+                    PROCESS1;
+                    PROCESS1;
+                    return XXH32_avalanche(h32);
+
+      case 15:      PROCESS4;
+                    /* fallthrough */
+      case 11:      PROCESS4;
+                    /* fallthrough */
+      case 7:       PROCESS4;
+                    /* fallthrough */
+      case 3:       PROCESS1;
+                    /* fallthrough */
+      case 2:       PROCESS1;
+                    /* fallthrough */
+      case 1:       PROCESS1;
+                    /* fallthrough */
+      case 0:       return XXH32_avalanche(h32);
+    }
+    assert(0);
+    return h32;   /* reaching this point is deemed impossible */
+}
+
+XXH_FORCE_INLINE U32
+XXH32_endian_align(const void* input, size_t len, U32 seed, XXH_alignment align)
+{
+    const BYTE* p = (const BYTE*)input;
+    const BYTE* bEnd = p + len;
+    U32 h32;
+
+#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1)
+    if (p==NULL) {
+        len=0;
+        bEnd=p=(const BYTE*)(size_t)16;
+    }
+#endif
+
+    if (len>=16) {
+        const BYTE* const limit = bEnd - 15;
+        U32 v1 = seed + PRIME32_1 + PRIME32_2;
+        U32 v2 = seed + PRIME32_2;
+        U32 v3 = seed + 0;
+        U32 v4 = seed - PRIME32_1;
+
+        do {
+            v1 = XXH32_round(v1, XXH_get32bits(p)); p+=4;
+            v2 = XXH32_round(v2, XXH_get32bits(p)); p+=4;
+            v3 = XXH32_round(v3, XXH_get32bits(p)); p+=4;
+            v4 = XXH32_round(v4, XXH_get32bits(p)); p+=4;
+        } while (p < limit);
+
+        h32 = XXH_rotl32(v1, 1)  + XXH_rotl32(v2, 7)
+            + XXH_rotl32(v3, 12) + XXH_rotl32(v4, 18);
+    } else {
+        h32  = seed + PRIME32_5;
+    }
+
+    h32 += (U32)len;
+
+    return XXH32_finalize(h32, p, len&15, align);
+}
+
+
+XXH_PUBLIC_API unsigned int XXH32 (const void* input, size_t len, unsigned int seed)
+{
+#if 0
+    /* Simple version, good for code maintenance, but unfortunately slow for small inputs */
+    XXH32_state_t state;
+    XXH32_reset(&state, seed);
+    XXH32_update(&state, input, len);
+    return XXH32_digest(&state);
+
+#else
+
+    if (XXH_FORCE_ALIGN_CHECK) {
+        if ((((size_t)input) & 3) == 0) {   /* Input is 4-bytes aligned, leverage the speed benefit */
+            return XXH32_endian_align(input, len, seed, XXH_aligned);
+    }   }
+
+    return XXH32_endian_align(input, len, seed, XXH_unaligned);
+#endif
+}
+
+
+
+/*======   Hash streaming   ======*/
+
+XXH_PUBLIC_API XXH32_state_t* XXH32_createState(void)
+{
+    return (XXH32_state_t*)XXH_malloc(sizeof(XXH32_state_t));
+}
+XXH_PUBLIC_API XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr)
+{
+    XXH_free(statePtr);
+    return XXH_OK;
+}
+
+XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* dstState, const XXH32_state_t* srcState)
+{
+    memcpy(dstState, srcState, sizeof(*dstState));
+}
+
+XXH_PUBLIC_API XXH_errorcode XXH32_reset(XXH32_state_t* statePtr, unsigned int seed)
+{
+    XXH32_state_t state;   /* using a local state to memcpy() in order to avoid strict-aliasing warnings */
+    memset(&state, 0, sizeof(state));
+    state.v1 = seed + PRIME32_1 + PRIME32_2;
+    state.v2 = seed + PRIME32_2;
+    state.v3 = seed + 0;
+    state.v4 = seed - PRIME32_1;
+    /* do not write into reserved, planned to be removed in a future version */
+    memcpy(statePtr, &state, sizeof(state) - sizeof(state.reserved));
+    return XXH_OK;
+}
+
+
+XXH_PUBLIC_API XXH_errorcode
+XXH32_update(XXH32_state_t* state, const void* input, size_t len)
+{
+    if (input==NULL)
+#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1)
+        return XXH_OK;
+#else
+        return XXH_ERROR;
+#endif
+
+    {   const BYTE* p = (const BYTE*)input;
+        const BYTE* const bEnd = p + len;
+
+        state->total_len_32 += (XXH32_hash_t)len;
+        state->large_len |= (XXH32_hash_t)((len>=16) | (state->total_len_32>=16));
+
+        if (state->memsize + len < 16)  {   /* fill in tmp buffer */
+            XXH_memcpy((BYTE*)(state->mem32) + state->memsize, input, len);
+            state->memsize += (XXH32_hash_t)len;
+            return XXH_OK;
+        }
+
+        if (state->memsize) {   /* some data left from previous update */
+            XXH_memcpy((BYTE*)(state->mem32) + state->memsize, input, 16-state->memsize);
+            {   const U32* p32 = state->mem32;
+                state->v1 = XXH32_round(state->v1, XXH_readLE32(p32)); p32++;
+                state->v2 = XXH32_round(state->v2, XXH_readLE32(p32)); p32++;
+                state->v3 = XXH32_round(state->v3, XXH_readLE32(p32)); p32++;
+                state->v4 = XXH32_round(state->v4, XXH_readLE32(p32));
+            }
+            p += 16-state->memsize;
+            state->memsize = 0;
+        }
+
+        if (p <= bEnd-16) {
+            const BYTE* const limit = bEnd - 16;
+            U32 v1 = state->v1;
+            U32 v2 = state->v2;
+            U32 v3 = state->v3;
+            U32 v4 = state->v4;
+
+            do {
+                v1 = XXH32_round(v1, XXH_readLE32(p)); p+=4;
+                v2 = XXH32_round(v2, XXH_readLE32(p)); p+=4;
+                v3 = XXH32_round(v3, XXH_readLE32(p)); p+=4;
+                v4 = XXH32_round(v4, XXH_readLE32(p)); p+=4;
+            } while (p<=limit);
+
+            state->v1 = v1;
+            state->v2 = v2;
+            state->v3 = v3;
+            state->v4 = v4;
+        }
+
+        if (p < bEnd) {
+            XXH_memcpy(state->mem32, p, (size_t)(bEnd-p));
+            state->memsize = (unsigned)(bEnd-p);
+        }
+    }
+
+    return XXH_OK;
+}
+
+
+XXH_PUBLIC_API unsigned int XXH32_digest (const XXH32_state_t* state)
+{
+    U32 h32;
+
+    if (state->large_len) {
+        h32 = XXH_rotl32(state->v1, 1)
+            + XXH_rotl32(state->v2, 7)
+            + XXH_rotl32(state->v3, 12)
+            + XXH_rotl32(state->v4, 18);
+    } else {
+        h32 = state->v3 /* == seed */ + PRIME32_5;
+    }
+
+    h32 += state->total_len_32;
+
+    return XXH32_finalize(h32, state->mem32, state->memsize, XXH_aligned);
+}
+
+
+/*======   Canonical representation   ======*/
+
+/*! Default XXH result types are basic unsigned 32 and 64 bits.
+*   The canonical representation follows human-readable write convention, aka big-endian (large digits first).
+*   These functions allow transformation of hash result into and from its canonical format.
+*   This way, hash values can be written into a file or buffer, remaining comparable across different systems.
+*/
+
+XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash)
+{
+    XXH_STATIC_ASSERT(sizeof(XXH32_canonical_t) == sizeof(XXH32_hash_t));
+    if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap32(hash);
+    memcpy(dst, &hash, sizeof(*dst));
+}
+
+XXH_PUBLIC_API XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src)
+{
+    return XXH_readBE32(src);
+}
+
+
+#ifndef XXH_NO_LONG_LONG
+
+/* *******************************************************************
+*  64-bit hash functions
+*********************************************************************/
+
+/*======   Memory access   ======*/
+
+#ifndef MEM_MODULE
+# define MEM_MODULE
+# if !defined (__VMS) \
+  && (defined (__cplusplus) \
+  || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) )
+#   include <stdint.h>
+    typedef uint64_t U64;
+# else
+    /* if compiler doesn't support unsigned long long, replace by another 64-bit type */
+    typedef unsigned long long U64;
+# endif
+#endif
+
+
+#if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==2))
+
+/* Force direct memory access. Only works on CPU which support unaligned memory access in hardware */
+static U64 XXH_read64(const void* memPtr) { return *(const U64*) memPtr; }
+
+#elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==1))
+
+/* __pack instructions are safer, but compiler specific, hence potentially problematic for some compilers */
+/* currently only defined for gcc and icc */
+typedef union { U32 u32; U64 u64; } __attribute__((packed)) unalign64;
+static U64 XXH_read64(const void* ptr) { return ((const unalign64*)ptr)->u64; }
+
+#else
+
+/* portable and safe solution. Generally efficient.
+ * see : http://stackoverflow.com/a/32095106/646947
+ */
+
+static U64 XXH_read64(const void* memPtr)
+{
+    U64 val;
+    memcpy(&val, memPtr, sizeof(val));
+    return val;
+}
+
+#endif   /* XXH_FORCE_DIRECT_MEMORY_ACCESS */
+
+#if defined(_MSC_VER)     /* Visual Studio */
+#  define XXH_swap64 _byteswap_uint64
+#elif XXH_GCC_VERSION >= 403
+#  define XXH_swap64 __builtin_bswap64
+#else
+static U64 XXH_swap64 (U64 x)
+{
+    return  ((x << 56) & 0xff00000000000000ULL) |
+            ((x << 40) & 0x00ff000000000000ULL) |
+            ((x << 24) & 0x0000ff0000000000ULL) |
+            ((x << 8)  & 0x000000ff00000000ULL) |
+            ((x >> 8)  & 0x00000000ff000000ULL) |
+            ((x >> 24) & 0x0000000000ff0000ULL) |
+            ((x >> 40) & 0x000000000000ff00ULL) |
+            ((x >> 56) & 0x00000000000000ffULL);
+}
+#endif
+
+XXH_FORCE_INLINE U64 XXH_readLE64(const void* ptr)
+{
+    return XXH_CPU_LITTLE_ENDIAN ? XXH_read64(ptr) : XXH_swap64(XXH_read64(ptr));
+}
+
+static U64 XXH_readBE64(const void* ptr)
+{
+    return XXH_CPU_LITTLE_ENDIAN ? XXH_swap64(XXH_read64(ptr)) : XXH_read64(ptr);
+}
+
+XXH_FORCE_INLINE U64
+XXH_readLE64_align(const void* ptr, XXH_alignment align)
+{
+    if (align==XXH_unaligned)
+        return XXH_readLE64(ptr);
+    else
+        return XXH_CPU_LITTLE_ENDIAN ? *(const U64*)ptr : XXH_swap64(*(const U64*)ptr);
+}
+
+
+/*======   xxh64   ======*/
+
+static const U64 PRIME64_1 = 11400714785074694791ULL;   /* 0b1001111000110111011110011011000110000101111010111100101010000111 */
+static const U64 PRIME64_2 = 14029467366897019727ULL;   /* 0b1100001010110010101011100011110100100111110101001110101101001111 */
+static const U64 PRIME64_3 =  1609587929392839161ULL;   /* 0b0001011001010110011001111011000110011110001101110111100111111001 */
+static const U64 PRIME64_4 =  9650029242287828579ULL;   /* 0b1000010111101011110010100111011111000010101100101010111001100011 */
+static const U64 PRIME64_5 =  2870177450012600261ULL;   /* 0b0010011111010100111010110010111100010110010101100110011111000101 */
+
+static U64 XXH64_round(U64 acc, U64 input)
+{
+    acc += input * PRIME64_2;
+    acc  = XXH_rotl64(acc, 31);
+    acc *= PRIME64_1;
+    return acc;
+}
+
+static U64 XXH64_mergeRound(U64 acc, U64 val)
+{
+    val  = XXH64_round(0, val);
+    acc ^= val;
+    acc  = acc * PRIME64_1 + PRIME64_4;
+    return acc;
+}
+
+static U64 XXH64_avalanche(U64 h64)
+{
+    h64 ^= h64 >> 33;
+    h64 *= PRIME64_2;
+    h64 ^= h64 >> 29;
+    h64 *= PRIME64_3;
+    h64 ^= h64 >> 32;
+    return h64;
+}
+
+
+#define XXH_get64bits(p) XXH_readLE64_align(p, align)
+
+static U64
+XXH64_finalize(U64 h64, const void* ptr, size_t len, XXH_alignment align)
+{
+    const BYTE* p = (const BYTE*)ptr;
+
+#define PROCESS1_64            \
+    h64 ^= (*p++) * PRIME64_5; \
+    h64 = XXH_rotl64(h64, 11) * PRIME64_1;
+
+#define PROCESS4_64          \
+    h64 ^= (U64)(XXH_get32bits(p)) * PRIME64_1; \
+    p+=4;                    \
+    h64 = XXH_rotl64(h64, 23) * PRIME64_2 + PRIME64_3;
+
+#define PROCESS8_64 {        \
+    U64 const k1 = XXH64_round(0, XXH_get64bits(p)); \
+    p+=8;                    \
+    h64 ^= k1;               \
+    h64  = XXH_rotl64(h64,27) * PRIME64_1 + PRIME64_4; \
+}
+
+    switch(len&31) {
+      case 24: PROCESS8_64;
+                    /* fallthrough */
+      case 16: PROCESS8_64;
+                    /* fallthrough */
+      case  8: PROCESS8_64;
+               return XXH64_avalanche(h64);
+
+      case 28: PROCESS8_64;
+                    /* fallthrough */
+      case 20: PROCESS8_64;
+                    /* fallthrough */
+      case 12: PROCESS8_64;
+                    /* fallthrough */
+      case  4: PROCESS4_64;
+               return XXH64_avalanche(h64);
+
+      case 25: PROCESS8_64;
+                    /* fallthrough */
+      case 17: PROCESS8_64;
+                    /* fallthrough */
+      case  9: PROCESS8_64;
+               PROCESS1_64;
+               return XXH64_avalanche(h64);
+
+      case 29: PROCESS8_64;
+                    /* fallthrough */
+      case 21: PROCESS8_64;
+                    /* fallthrough */
+      case 13: PROCESS8_64;
+                    /* fallthrough */
+      case  5: PROCESS4_64;
+               PROCESS1_64;
+               return XXH64_avalanche(h64);
+
+      case 26: PROCESS8_64;
+                    /* fallthrough */
+      case 18: PROCESS8_64;
+                    /* fallthrough */
+      case 10: PROCESS8_64;
+               PROCESS1_64;
+               PROCESS1_64;
+               return XXH64_avalanche(h64);
+
+      case 30: PROCESS8_64;
+                    /* fallthrough */
+      case 22: PROCESS8_64;
+                    /* fallthrough */
+      case 14: PROCESS8_64;
+                    /* fallthrough */
+      case  6: PROCESS4_64;
+               PROCESS1_64;
+               PROCESS1_64;
+               return XXH64_avalanche(h64);
+
+      case 27: PROCESS8_64;
+                    /* fallthrough */
+      case 19: PROCESS8_64;
+                    /* fallthrough */
+      case 11: PROCESS8_64;
+               PROCESS1_64;
+               PROCESS1_64;
+               PROCESS1_64;
+               return XXH64_avalanche(h64);
+
+      case 31: PROCESS8_64;
+                    /* fallthrough */
+      case 23: PROCESS8_64;
+                    /* fallthrough */
+      case 15: PROCESS8_64;
+                    /* fallthrough */
+      case  7: PROCESS4_64;
+                    /* fallthrough */
+      case  3: PROCESS1_64;
+                    /* fallthrough */
+      case  2: PROCESS1_64;
+                    /* fallthrough */
+      case  1: PROCESS1_64;
+                    /* fallthrough */
+      case  0: return XXH64_avalanche(h64);
+    }
+
+    /* impossible to reach */
+    assert(0);
+    return 0;  /* unreachable, but some compilers complain without it */
+}
+
+XXH_FORCE_INLINE U64
+XXH64_endian_align(const void* input, size_t len, U64 seed, XXH_alignment align)
+{
+    const BYTE* p = (const BYTE*)input;
+    const BYTE* bEnd = p + len;
+    U64 h64;
+
+#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1)
+    if (p==NULL) {
+        len=0;
+        bEnd=p=(const BYTE*)(size_t)32;
+    }
+#endif
+
+    if (len>=32) {
+        const BYTE* const limit = bEnd - 32;
+        U64 v1 = seed + PRIME64_1 + PRIME64_2;
+        U64 v2 = seed + PRIME64_2;
+        U64 v3 = seed + 0;
+        U64 v4 = seed - PRIME64_1;
+
+        do {
+            v1 = XXH64_round(v1, XXH_get64bits(p)); p+=8;
+            v2 = XXH64_round(v2, XXH_get64bits(p)); p+=8;
+            v3 = XXH64_round(v3, XXH_get64bits(p)); p+=8;
+            v4 = XXH64_round(v4, XXH_get64bits(p)); p+=8;
+        } while (p<=limit);
+
+        h64 = XXH_rotl64(v1, 1) + XXH_rotl64(v2, 7) + XXH_rotl64(v3, 12) + XXH_rotl64(v4, 18);
+        h64 = XXH64_mergeRound(h64, v1);
+        h64 = XXH64_mergeRound(h64, v2);
+        h64 = XXH64_mergeRound(h64, v3);
+        h64 = XXH64_mergeRound(h64, v4);
+
+    } else {
+        h64  = seed + PRIME64_5;
+    }
+
+    h64 += (U64) len;
+
+    return XXH64_finalize(h64, p, len, align);
+}
+
+
+XXH_PUBLIC_API unsigned long long XXH64 (const void* input, size_t len, unsigned long long seed)
+{
+#if 0
+    /* Simple version, good for code maintenance, but unfortunately slow for small inputs */
+    XXH64_state_t state;
+    XXH64_reset(&state, seed);
+    XXH64_update(&state, input, len);
+    return XXH64_digest(&state);
+
+#else
+
+    if (XXH_FORCE_ALIGN_CHECK) {
+        if ((((size_t)input) & 7)==0) {  /* Input is aligned, let's leverage the speed advantage */
+            return XXH64_endian_align(input, len, seed, XXH_aligned);
+    }   }
+
+    return XXH64_endian_align(input, len, seed, XXH_unaligned);
+
+#endif
+}
+
+/*======   Hash Streaming   ======*/
+
+XXH_PUBLIC_API XXH64_state_t* XXH64_createState(void)
+{
+    return (XXH64_state_t*)XXH_malloc(sizeof(XXH64_state_t));
+}
+XXH_PUBLIC_API XXH_errorcode XXH64_freeState(XXH64_state_t* statePtr)
+{
+    XXH_free(statePtr);
+    return XXH_OK;
+}
+
+XXH_PUBLIC_API void XXH64_copyState(XXH64_state_t* dstState, const XXH64_state_t* srcState)
+{
+    memcpy(dstState, srcState, sizeof(*dstState));
+}
+
+XXH_PUBLIC_API XXH_errorcode XXH64_reset(XXH64_state_t* statePtr, unsigned long long seed)
+{
+    XXH64_state_t state;   /* using a local state to memcpy() in order to avoid strict-aliasing warnings */
+    memset(&state, 0, sizeof(state));
+    state.v1 = seed + PRIME64_1 + PRIME64_2;
+    state.v2 = seed + PRIME64_2;
+    state.v3 = seed + 0;
+    state.v4 = seed - PRIME64_1;
+     /* do not write into reserved, planned to be removed in a future version */
+    memcpy(statePtr, &state, sizeof(state) - sizeof(state.reserved));
+    return XXH_OK;
+}
+
+XXH_PUBLIC_API XXH_errorcode
+XXH64_update (XXH64_state_t* state, const void* input, size_t len)
+{
+    if (input==NULL)
+#if defined(XXH_ACCEPT_NULL_INPUT_POINTER) && (XXH_ACCEPT_NULL_INPUT_POINTER>=1)
+        return XXH_OK;
+#else
+        return XXH_ERROR;
+#endif
+
+    {   const BYTE* p = (const BYTE*)input;
+        const BYTE* const bEnd = p + len;
+
+        state->total_len += len;
+
+        if (state->memsize + len < 32) {  /* fill in tmp buffer */
+            XXH_memcpy(((BYTE*)state->mem64) + state->memsize, input, len);
+            state->memsize += (U32)len;
+            return XXH_OK;
+        }
+
+        if (state->memsize) {   /* tmp buffer is full */
+            XXH_memcpy(((BYTE*)state->mem64) + state->memsize, input, 32-state->memsize);
+            state->v1 = XXH64_round(state->v1, XXH_readLE64(state->mem64+0));
+            state->v2 = XXH64_round(state->v2, XXH_readLE64(state->mem64+1));
+            state->v3 = XXH64_round(state->v3, XXH_readLE64(state->mem64+2));
+            state->v4 = XXH64_round(state->v4, XXH_readLE64(state->mem64+3));
+            p += 32-state->memsize;
+            state->memsize = 0;
+        }
+
+        if (p+32 <= bEnd) {
+            const BYTE* const limit = bEnd - 32;
+            U64 v1 = state->v1;
+            U64 v2 = state->v2;
+            U64 v3 = state->v3;
+            U64 v4 = state->v4;
+
+            do {
+                v1 = XXH64_round(v1, XXH_readLE64(p)); p+=8;
+                v2 = XXH64_round(v2, XXH_readLE64(p)); p+=8;
+                v3 = XXH64_round(v3, XXH_readLE64(p)); p+=8;
+                v4 = XXH64_round(v4, XXH_readLE64(p)); p+=8;
+            } while (p<=limit);
+
+            state->v1 = v1;
+            state->v2 = v2;
+            state->v3 = v3;
+            state->v4 = v4;
+        }
+
+        if (p < bEnd) {
+            XXH_memcpy(state->mem64, p, (size_t)(bEnd-p));
+            state->memsize = (unsigned)(bEnd-p);
+        }
+    }
+
+    return XXH_OK;
+}
+
+
+XXH_PUBLIC_API unsigned long long XXH64_digest (const XXH64_state_t* state)
+{
+    U64 h64;
+
+    if (state->total_len >= 32) {
+        U64 const v1 = state->v1;
+        U64 const v2 = state->v2;
+        U64 const v3 = state->v3;
+        U64 const v4 = state->v4;
+
+        h64 = XXH_rotl64(v1, 1) + XXH_rotl64(v2, 7) + XXH_rotl64(v3, 12) + XXH_rotl64(v4, 18);
+        h64 = XXH64_mergeRound(h64, v1);
+        h64 = XXH64_mergeRound(h64, v2);
+        h64 = XXH64_mergeRound(h64, v3);
+        h64 = XXH64_mergeRound(h64, v4);
+    } else {
+        h64  = state->v3 /*seed*/ + PRIME64_5;
+    }
+
+    h64 += (U64) state->total_len;
+
+    return XXH64_finalize(h64, state->mem64, (size_t)state->total_len, XXH_aligned);
+}
+
+
+/*====== Canonical representation   ======*/
+
+XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH64_canonical_t* dst, XXH64_hash_t hash)
+{
+    XXH_STATIC_ASSERT(sizeof(XXH64_canonical_t) == sizeof(XXH64_hash_t));
+    if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap64(hash);
+    memcpy(dst, &hash, sizeof(*dst));
+}
+
+XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src)
+{
+    return XXH_readBE64(src);
+}
+
+
+
+/* *********************************************************************
+*  XXH3
+*  New generation hash designed for speed on small keys and vectorization
+************************************************************************ */
+
+#include "xxh3.h"
+
+
+#endif  /* XXH_NO_LONG_LONG */
diff --git a/crypto/xxhash.h b/crypto/xxhash.h
new file mode 100644
index 000000000000..9ee05e5dc146
--- /dev/null
+++ b/crypto/xxhash.h
@@ -0,0 +1,445 @@
+/*
+   xxHash - Extremely Fast Hash algorithm
+   Header File
+   Copyright (C) 2012-2016, Yann Collet.
+
+   BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
+
+   Redistribution and use in source and binary forms, with or without
+   modification, are permitted provided that the following conditions are
+   met:
+
+       * Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer.
+       * Redistributions in binary form must reproduce the above
+   copyright notice, this list of conditions and the following disclaimer
+   in the documentation and/or other materials provided with the
+   distribution.
+
+   THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+   "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+   LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+   A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+   OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+   THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+   (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+   OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+   You can contact the author at :
+   - xxHash source repository : https://github.com/Cyan4973/xxHash
+*/
+
+/* Notice extracted from xxHash homepage :
+
+xxHash is an extremely fast Hash algorithm, running at RAM speed limits.
+It also successfully passes all tests from the SMHasher suite.
+
+Comparison (single thread, Windows Seven 32 bits, using SMHasher on a Core 2 Duo @3GHz)
+
+Name            Speed       Q.Score   Author
+xxHash          5.4 GB/s     10
+CrapWow         3.2 GB/s      2       Andrew
+MumurHash 3a    2.7 GB/s     10       Austin Appleby
+SpookyHash      2.0 GB/s     10       Bob Jenkins
+SBox            1.4 GB/s      9       Bret Mulvey
+Lookup3         1.2 GB/s      9       Bob Jenkins
+SuperFastHash   1.2 GB/s      1       Paul Hsieh
+CityHash64      1.05 GB/s    10       Pike & Alakuijala
+FNV             0.55 GB/s     5       Fowler, Noll, Vo
+CRC32           0.43 GB/s     9
+MD5-32          0.33 GB/s    10       Ronald L. Rivest
+SHA1-32         0.28 GB/s    10
+
+Q.Score is a measure of quality of the hash function.
+It depends on successfully passing SMHasher test set.
+10 is a perfect score.
+
+A 64-bit version, named XXH64, is available since r35.
+It offers much better speed, but for 64-bit applications only.
+Name     Speed on 64 bits    Speed on 32 bits
+XXH64       13.8 GB/s            1.9 GB/s
+XXH32        6.8 GB/s            6.0 GB/s
+*/
+
+#ifndef XXHASH_H_5627135585666179
+#define XXHASH_H_5627135585666179 1
+
+#if defined (__cplusplus)
+extern "C" {
+#endif
+
+
+/* ****************************
+*  Definitions
+******************************/
+#include <stddef.h>   /* size_t */
+typedef enum { XXH_OK=0, XXH_ERROR } XXH_errorcode;
+
+
+/* ****************************
+ *  API modifier
+ ******************************/
+/** XXH_INLINE_ALL (and XXH_PRIVATE_API)
+ *  This is useful to include xxhash functions in `static` mode
+ *  in order to inline them, and remove their symbol from the public list.
+ *  Inlining can offer dramatic performance improvement on small keys.
+ *  Methodology :
+ *     #define XXH_INLINE_ALL
+ *     #include "xxhash.h"
+ * `xxhash.c` is automatically included.
+ *  It's not useful to compile and link it as a separate module.
+ */
+#if defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API)
+#  ifndef XXH_STATIC_LINKING_ONLY
+#    define XXH_STATIC_LINKING_ONLY
+#  endif
+#  if defined(__GNUC__)
+#    define XXH_PUBLIC_API static __inline __attribute__((unused))
+#  elif defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */)
+#    define XXH_PUBLIC_API static inline
+#  elif defined(_MSC_VER)
+#    define XXH_PUBLIC_API static __inline
+#  else
+     /* this version may generate warnings for unused static functions */
+#    define XXH_PUBLIC_API static
+#  endif
+#else
+#  if defined(WIN32) && defined(_MSC_VER) && (defined(XXH_IMPORT) || defined(XXH_EXPORT))
+#    ifdef XXH_EXPORT
+#      define XXH_PUBLIC_API __declspec(dllexport)
+#    elif XXH_IMPORT
+#      define XXH_PUBLIC_API __declspec(dllimport)
+#    endif
+#  else
+#    define XXH_PUBLIC_API   /* do nothing */
+#  endif
+#endif /* XXH_INLINE_ALL || XXH_PRIVATE_API */
+
+/*! XXH_NAMESPACE, aka Namespace Emulation :
+ *
+ * If you want to include _and expose_ xxHash functions from within your own library,
+ * but also want to avoid symbol collisions with other libraries which may also include xxHash,
+ *
+ * you can use XXH_NAMESPACE, to automatically prefix any public symbol from xxhash library
+ * with the value of XXH_NAMESPACE (therefore, avoid NULL and numeric values).
+ *
+ * Note that no change is required within the calling program as long as it includes `xxhash.h` :
+ * regular symbol name will be automatically translated by this header.
+ */
+#ifdef XXH_NAMESPACE
+#  define XXH_CAT(A,B) A##B
+#  define XXH_NAME2(A,B) XXH_CAT(A,B)
+#  define XXH_versionNumber XXH_NAME2(XXH_NAMESPACE, XXH_versionNumber)
+#  define XXH32 XXH_NAME2(XXH_NAMESPACE, XXH32)
+#  define XXH32_createState XXH_NAME2(XXH_NAMESPACE, XXH32_createState)
+#  define XXH32_freeState XXH_NAME2(XXH_NAMESPACE, XXH32_freeState)
+#  define XXH32_reset XXH_NAME2(XXH_NAMESPACE, XXH32_reset)
+#  define XXH32_update XXH_NAME2(XXH_NAMESPACE, XXH32_update)
+#  define XXH32_digest XXH_NAME2(XXH_NAMESPACE, XXH32_digest)
+#  define XXH32_copyState XXH_NAME2(XXH_NAMESPACE, XXH32_copyState)
+#  define XXH32_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH32_canonicalFromHash)
+#  define XXH32_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH32_hashFromCanonical)
+#  define XXH64 XXH_NAME2(XXH_NAMESPACE, XXH64)
+#  define XXH64_createState XXH_NAME2(XXH_NAMESPACE, XXH64_createState)
+#  define XXH64_freeState XXH_NAME2(XXH_NAMESPACE, XXH64_freeState)
+#  define XXH64_reset XXH_NAME2(XXH_NAMESPACE, XXH64_reset)
+#  define XXH64_update XXH_NAME2(XXH_NAMESPACE, XXH64_update)
+#  define XXH64_digest XXH_NAME2(XXH_NAMESPACE, XXH64_digest)
+#  define XXH64_copyState XXH_NAME2(XXH_NAMESPACE, XXH64_copyState)
+#  define XXH64_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH64_canonicalFromHash)
+#  define XXH64_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH64_hashFromCanonical)
+#endif
+
+
+/* *************************************
+*  Version
+***************************************/
+#define XXH_VERSION_MAJOR    0
+#define XXH_VERSION_MINOR    7
+#define XXH_VERSION_RELEASE  0
+#define XXH_VERSION_NUMBER  (XXH_VERSION_MAJOR *100*100 + XXH_VERSION_MINOR *100 + XXH_VERSION_RELEASE)
+XXH_PUBLIC_API unsigned XXH_versionNumber (void);
+
+
+/*-**********************************************************************
+*  32-bit hash
+************************************************************************/
+typedef unsigned int XXH32_hash_t;
+
+/*! XXH32() :
+    Calculate the 32-bit hash of sequence "length" bytes stored at memory address "input".
+    The memory between input & input+length must be valid (allocated and read-accessible).
+    "seed" can be used to alter the result predictably.
+    Speed on Core 2 Duo @ 3 GHz (single thread, SMHasher benchmark) : 5.4 GB/s */
+XXH_PUBLIC_API XXH32_hash_t XXH32 (const void* input, size_t length, unsigned int seed);
+
+/*======   Streaming   ======*/
+typedef struct XXH32_state_s XXH32_state_t;   /* incomplete type */
+XXH_PUBLIC_API XXH32_state_t* XXH32_createState(void);
+XXH_PUBLIC_API XXH_errorcode  XXH32_freeState(XXH32_state_t* statePtr);
+XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* dst_state, const XXH32_state_t* src_state);
+
+XXH_PUBLIC_API XXH_errorcode XXH32_reset  (XXH32_state_t* statePtr, unsigned int seed);
+XXH_PUBLIC_API XXH_errorcode XXH32_update (XXH32_state_t* statePtr, const void* input, size_t length);
+XXH_PUBLIC_API XXH32_hash_t  XXH32_digest (const XXH32_state_t* statePtr);
+
+/*
+ * Streaming functions generate the xxHash of an input provided in multiple segments.
+ * Note that, for small input, they are slower than single-call functions, due to state management.
+ * For small inputs, prefer `XXH32()` and `XXH64()`, which are better optimized.
+ *
+ * XXH state must first be allocated, using XXH*_createState() .
+ *
+ * Start a new hash by initializing state with a seed, using XXH*_reset().
+ *
+ * Then, feed the hash state by calling XXH*_update() as many times as necessary.
+ * The function returns an error code, with 0 meaning OK, and any other value meaning there is an error.
+ *
+ * Finally, a hash value can be produced anytime, by using XXH*_digest().
+ * This function returns the nn-bits hash as an int or long long.
+ *
+ * It's still possible to continue inserting input into the hash state after a digest,
+ * and generate some new hashes later on, by calling again XXH*_digest().
+ *
+ * When done, free XXH state space if it was allocated dynamically.
+ */
+
+/*======   Canonical representation   ======*/
+
+typedef struct { unsigned char digest[4]; } XXH32_canonical_t;
+XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash);
+XXH_PUBLIC_API XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src);
+
+/* Default result type for XXH functions are primitive unsigned 32 and 64 bits.
+ * The canonical representation uses human-readable write convention, aka big-endian (large digits first).
+ * These functions allow transformation of hash result into and from its canonical format.
+ * This way, hash values can be written into a file / memory, and remain comparable on different systems and programs.
+ */
+
+
+#ifndef XXH_NO_LONG_LONG
+/*-**********************************************************************
+*  64-bit hash
+************************************************************************/
+typedef unsigned long long XXH64_hash_t;
+
+/*! XXH64() :
+    Calculate the 64-bit hash of sequence of length "len" stored at memory address "input".
+    "seed" can be used to alter the result predictably.
+    This function runs faster on 64-bit systems, but slower on 32-bit systems (see benchmark).
+*/
+XXH_PUBLIC_API XXH64_hash_t XXH64 (const void* input, size_t length, unsigned long long seed);
+
+/*======   Streaming   ======*/
+typedef struct XXH64_state_s XXH64_state_t;   /* incomplete type */
+XXH_PUBLIC_API XXH64_state_t* XXH64_createState(void);
+XXH_PUBLIC_API XXH_errorcode  XXH64_freeState(XXH64_state_t* statePtr);
+XXH_PUBLIC_API void XXH64_copyState(XXH64_state_t* dst_state, const XXH64_state_t* src_state);
+
+XXH_PUBLIC_API XXH_errorcode XXH64_reset  (XXH64_state_t* statePtr, unsigned long long seed);
+XXH_PUBLIC_API XXH_errorcode XXH64_update (XXH64_state_t* statePtr, const void* input, size_t length);
+XXH_PUBLIC_API XXH64_hash_t  XXH64_digest (const XXH64_state_t* statePtr);
+
+/*======   Canonical representation   ======*/
+typedef struct { unsigned char digest[8]; } XXH64_canonical_t;
+XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH64_canonical_t* dst, XXH64_hash_t hash);
+XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src);
+
+
+#endif  /* XXH_NO_LONG_LONG */
+
+
+
+#ifdef XXH_STATIC_LINKING_ONLY
+
+/* ================================================================================================
+   This section contains declarations which are not guaranteed to remain stable.
+   They may change in future versions, becoming incompatible with a different version of the library.
+   These declarations should only be used with static linking.
+   Never use them in association with dynamic linking !
+=================================================================================================== */
+
+/* These definitions are only present to allow
+ * static allocation of XXH state, on stack or in a struct for example.
+ * Never **ever** use members directly. */
+
+#if !defined (__VMS) \
+  && (defined (__cplusplus) \
+  || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) )
+#   include <stdint.h>
+
+struct XXH32_state_s {
+   uint32_t total_len_32;
+   uint32_t large_len;
+   uint32_t v1;
+   uint32_t v2;
+   uint32_t v3;
+   uint32_t v4;
+   uint32_t mem32[4];
+   uint32_t memsize;
+   uint32_t reserved;   /* never read nor write, might be removed in a future version */
+};   /* typedef'd to XXH32_state_t */
+
+struct XXH64_state_s {
+   uint64_t total_len;
+   uint64_t v1;
+   uint64_t v2;
+   uint64_t v3;
+   uint64_t v4;
+   uint64_t mem64[4];
+   uint32_t memsize;
+   uint32_t reserved[2];   /* never read nor write, might be removed in a future version */
+};   /* typedef'd to XXH64_state_t */
+
+# else
+
+struct XXH32_state_s {
+   XXH32_hash_t total_len_32;
+   XXH32_hash_t large_len;
+   XXH32_hash_t v1;
+   XXH32_hash_t v2;
+   XXH32_hash_t v3;
+   XXH32_hash_t v4;
+   XXH32_hash_t mem32[4];
+   XXH32_hash_t memsize;
+   XXH32_hash_t reserved;   /* never read nor write, might be removed in a future version */
+};   /* typedef'd to XXH32_state_t */
+
+#   ifndef XXH_NO_LONG_LONG  /* remove 64-bit support */
+struct XXH64_state_s {
+   XXH64_hash_t total_len;
+   XXH64_hash_t v1;
+   XXH64_hash_t v2;
+   XXH64_hash_t v3;
+   XXH64_hash_t v4;
+   XXH64_hash_t mem64[4];
+   XXH32_hash_t memsize;
+   XXH32_hash_t reserved[2];     /* never read nor write, might be removed in a future version */
+};   /* typedef'd to XXH64_state_t */
+#    endif
+
+# endif
+
+
+/*-**********************************************************************
+*  XXH3
+*  New experimental hash
+************************************************************************/
+#ifndef XXH_NO_LONG_LONG
+
+
+/* ============================================
+ * XXH3 is a new hash algorithm,
+ * featuring vastly improved speed performance
+ * for both small and large inputs.
+ * A full speed analysis will be published,
+ * it requires a lot more space than this comment can handle.
+ * In general, expect XXH3 to run about ~2x faster on large inputs,
+ * and >3x faster on small ones, though exact difference depend on platform.
+ *
+ * The algorithm is portable, will generate the same hash on all platforms.
+ * It benefits greatly from vectorization units, but does not require it.
+ *
+ * XXH3 offers 2 variants, _64bits and _128bits.
+ * When only 64 bits are needed, prefer calling the _64bits variant :
+ * it reduces the amount of mixing, resulting in faster speed on small inputs.
+ * It's also generally simpler to manipulate a scalar type than a struct.
+ * Note : the low 64-bit field of the _128bits variant is the same as _64bits result.
+ *
+ * The XXH3 algorithm is still considered experimental.
+ * It's possible to use it for ephemeral data, but avoid storing long-term values for later re-use.
+ * While labelled experimental, the produced result can still change between versions.
+ *
+ * The API currently supports one-shot hashing only.
+ * The full version will include streaming capability, and canonical representation
+ * Long term optional feature may include custom secret keys, and secret key generation.
+ *
+ * There are still a number of opened questions that community can influence during the experimental period.
+ * I'm trying to list a few of them below, though don't consider this list as complete.
+ *
+ * - 128-bits output type : currently defined as a structure of 2 64-bits fields.
+ *                          That's because 128-bit values do not exist in C standard.
+ *                          Note that it means that, at byte level, result is not identical depending on endianess.
+ *                          However, at field level, they are identical on all platforms.
+ *                          The canonical representation will solve the issue of identical byte-level representation across platforms,
+ *                          which is necessary for serialization.
+ *                          Would there be a better representation for a 128-bit hash result ?
+ *                          Are the names of the inner 64-bit fields important ? Should they be changed ?
+ *
+ * - Canonical representation : for the 64-bit variant, canonical representation is the same as XXH64() (aka big-endian).
+ *                          What should it be for the 128-bit variant ?
+ *                          Since it's no longer a scalar value, big-endian representation is no longer an obvious choice.
+ *                          One possibility : represent it as the concatenation of two 64-bits canonical representation (aka 2x big-endian)
+ *                          Another one : represent it in the same order as natural order in the struct for little-endian platforms.
+ *                                        Less consistent with existing convention for XXH32/XXH64, but may be more natural for little-endian platforms.
+ *
+ * - Associated functions for 128-bit hash : simple things, such as checking if 2 hashes are equal, become more difficult with struct.
+ *                          Granted, it's not terribly difficult to create a comparator, but it's still a workload.
+ *                          Would it be beneficial to declare and define a comparator function for XXH128_hash_t ?
+ *                          Are there other operations on XXH128_hash_t which would be desirable ?
+ *
+ * - Variant compatibility : The low 64-bit field of the _128bits variant is the same as the result of _64bits.
+ *                          This is not a compulsory behavior. It just felt that it "wouldn't hurt", and might even help in some (unidentified) cases.
+ *                          But it might influence the design of XXH128_hash_t, in ways which may block other possibilities.
+ *                          Good idea, bad idea ?
+ *
+ * - Seed type for 128-bits variant : currently, it's a single 64-bit value, like the 64-bit variant.
+ *                          It could be argued that it's more logical to offer a 128-bit seed input parameter for a 128-bit hash.
+ *                          Although it's also more difficult to use, since it requires to declare and pass a structure instead of a value.
+ *                          It would either replace current choice, or add a new one.
+ *                          Farmhash, for example, offers both variants (the 128-bits seed variant is called `doubleSeed`).
+ *                          If both 64-bit and 128-bit seeds are possible, which variant should be called XXH128 ?
+ *
+ * - Result for len==0 : Currently, the result of hashing a zero-length input is the seed.
+ *                          This mimics the behavior of a crc : in which case, a seed is effectively an accumulator, so it's not updated if input is empty.
+ *                          Consequently, by default, when no seed specified, it returns zero. That part seems okay (it used to be a request for XXH32/XXH64).
+ *                          But is it still fine to return the seed when the seed is non-zero ?
+ *                          Are there use case which would depend on this behavior, or would prefer a mixing of the seed ?
+ */
+
+#ifdef XXH_NAMESPACE
+#  define XXH128 XXH_NAME2(XXH_NAMESPACE, XXH128)
+#  define XXH3_64bits XXH_NAME2(XXH_NAMESPACE, XXH3_64bits)
+#  define XXH3_64bits_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_withSeed)
+#  define XXH3_128bits XXH_NAME2(XXH_NAMESPACE, XXH3_128bits)
+#  define XXH3_128bits_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_withSeed)
+#endif
+
+
+typedef struct {
+    XXH64_hash_t low64;
+    XXH64_hash_t high64;
+} XXH128_hash_t;
+
+XXH_PUBLIC_API XXH128_hash_t XXH128(const void* data, size_t len, unsigned long long seed);
+
+/* note : variants without seed produce same result as variant with seed == 0 */
+XXH_PUBLIC_API XXH64_hash_t  XXH3_64bits(const void* data, size_t len);
+XXH_PUBLIC_API XXH64_hash_t  XXH3_64bits_withSeed(const void* data, size_t len, unsigned long long seed);
+XXH_PUBLIC_API XXH128_hash_t XXH3_128bits(const void* data, size_t len);
+XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_withSeed(const void* data, size_t len, unsigned long long seed);  /* == XXH128() */
+
+
+#endif  /* XXH_NO_LONG_LONG */
+
+
+/*-**********************************************************************
+*  XXH_INLINE_ALL
+************************************************************************/
+#if defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API)
+#  include "xxhash.c"   /* include xxhash function bodies as `static`, for inlining */
+#endif
+
+
+
+#endif /* XXH_STATIC_LINKING_ONLY */
+
+
+#if defined (__cplusplus)
+}
+#endif
+
+#endif /* XXHASH_H_5627135585666179 */
-- 
2.16.4

Re: [PATCH v2 09/11] btrfs-progs: add xxhash sources

[Nikolay Borisov]

On 26.08.19 г. 14:48 ч., Johannes Thumshirn wrote:
> From: David Sterba <dsterba@suse.com>
> 
> git://github.com/Cyan4973/xxHash
> 
> Signed-off-by: David Sterba <dsterba@suse.com>
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>> ---
>  crypto/xxhash.c | 1024 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  crypto/xxhash.h |  445 ++++++++++++++++++++++++
>  2 files changed, 1469 insertions(+)


Existing crc32c implementation is under kernel-lib/, whereas the xxhash
will be under crypto. I think we should have some consistency among the
various hashes we might be implementing in the future. I don't have
strong preferences either way. I guess crypto/ makes more sense.

<snip>

Re: [PATCH v2 09/11] btrfs-progs: add xxhash sources

[David Sterba]

On Tue, Aug 27, 2019 at 04:05:57PM +0300, Nikolay Borisov wrote:
> 
> 
> On 26.08.19 г. 14:48 ч., Johannes Thumshirn wrote:
> > From: David Sterba <dsterba@suse.com>
> > 
> > git://github.com/Cyan4973/xxHash
> > 
> > Signed-off-by: David Sterba <dsterba@suse.com>
> > Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>> ---
> >  crypto/xxhash.c | 1024 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  crypto/xxhash.h |  445 ++++++++++++++++++++++++
> >  2 files changed, 1469 insertions(+)
> 
> 
> Existing crc32c implementation is under kernel-lib/, whereas the xxhash
> will be under crypto. I think we should have some consistency among the
> various hashes we might be implementing in the future. I don't have
> strong preferences either way. I guess crypto/ makes more sense.

The kernel-lib contains 99%-copies of the kernel files and they're
mostly unrelated. The crc32 files could be moved to crypto/ as we'll
have more than that and xxhash there.

The idea with the local versions of hashes is to provide a fallback
implementations of everything we need but the preferred option will be
linking to some existing crypto library like openssl. So we'll need some
glue code etc. This isn't entirely suitable for kerne-lib.

[PATCH v2 10/11] btrfs-progs: add xxhash64 as checksum algorithm

[Johannes Thumshirn]

From: David Sterba <dsterba@suse.com>

Add xxhash64 as another checksumming algorithm.

Signed-off-by: David Sterba <dsterba@suse.com>
Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 Makefile                  |  3 ++-
 cmds/inspect-dump-super.c | 25 ++++++++++++++++---------
 convert/common.c          |  2 +-
 convert/main.c            |  2 +-
 crypto/hash.c             | 16 ++++++++++++++++
 crypto/hash.h             | 10 ++++++++++
 crypto/xxhash.c           |  2 +-
 ctree.h                   | 18 +++++++++++++-----
 disk-io.c                 |  7 +++++--
 image/main.c              |  5 +++--
 mkfs/common.c             | 14 +++++++-------
 mkfs/main.c               |  6 +++++-
 12 files changed, 80 insertions(+), 30 deletions(-)
 create mode 100644 crypto/hash.c
 create mode 100644 crypto/hash.h

diff --git a/Makefile b/Makefile
index 82417d19a9f8..1982e6f5d70e 100644
--- a/Makefile
+++ b/Makefile
@@ -151,7 +151,8 @@ cmds_objects = cmds/subvolume.o cmds/filesystem.o cmds/device.o cmds/scrub.o \
 	       mkfs/common.o check/mode-common.o check/mode-lowmem.o
 libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o \
 		   kernel-lib/crc32c.o common/messages.o \
-		   uuid-tree.o utils-lib.o common/rbtree-utils.o
+		   uuid-tree.o utils-lib.o common/rbtree-utils.o \
+		   crypto/hash.o crypto/xxhash.o
 libbtrfs_headers = send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
 	       kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
 	       kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
index 58bf82b0bbd3..1001c8aa5c85 100644
--- a/cmds/inspect-dump-super.c
+++ b/cmds/inspect-dump-super.c
@@ -311,6 +311,17 @@ static void print_readable_super_flag(u64 flag)
 				     super_flags_num, BTRFS_SUPER_FLAG_SUPP);
 }
 
+static bool is_valid_csum_type(u16 csum_type)
+{
+	switch (csum_type) {
+	case BTRFS_CSUM_TYPE_CRC32:
+	case BTRFS_CSUM_TYPE_XXHASH:
+		return true;
+	default:
+		return false;
+	}
+}
+
 static void dump_superblock(struct btrfs_super_block *sb, int full)
 {
 	int i;
@@ -326,15 +337,11 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
 	csum_type = btrfs_super_csum_type(sb);
 	csum_size = BTRFS_CSUM_SIZE;
 	printf("csum_type\t\t%hu (", csum_type);
-	if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) {
+	if (csum_type >= ARRAY_SIZE(btrfs_csums)) {
 		printf("INVALID");
 	} else {
-		if (csum_type == BTRFS_CSUM_TYPE_CRC32) {
-			printf("crc32c");
-			csum_size = btrfs_csum_sizes[csum_type];
-		} else {
-			printf("unknown");
-		}
+		printf("%s", btrfs_csums[csum_type].name);
+		csum_size = btrfs_csums[csum_type].size;
 	}
 	printf(")\n");
 	printf("csum_size\t\t%llu\n", (unsigned long long)csum_size);
@@ -342,8 +349,8 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
 	printf("csum\t\t\t0x");
 	for (i = 0, p = sb->csum; i < csum_size; i++)
 		printf("%02x", p[i]);
-	if (csum_type != BTRFS_CSUM_TYPE_CRC32 ||
-	    csum_size != btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32])
+	if (!is_valid_csum_type(csum_type) ||
+	    csum_size != btrfs_csums[csum_type].size)
 		printf(" [UNKNOWN CSUM TYPE OR SIZE]");
 	else if (check_csum_sblock(sb, csum_size, csum_type))
 		printf(" [match]");
diff --git a/convert/common.c b/convert/common.c
index 8f5fdbf507a4..e479d70f4e9e 100644
--- a/convert/common.c
+++ b/convert/common.c
@@ -223,7 +223,7 @@ static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf,
 {
 	int ret;
 
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
 			     cfg->csum_type);
 
 	/* Temporary extent buffer is always mapped 1:1 on disk */
diff --git a/convert/main.c b/convert/main.c
index 5e6b12431f59..5eb2a59fb68a 100644
--- a/convert/main.c
+++ b/convert/main.c
@@ -1058,7 +1058,7 @@ static int migrate_super_block(int fd, u64 old_bytenr)
 	BUG_ON(btrfs_super_bytenr(super) != old_bytenr);
 	btrfs_set_super_bytenr(super, BTRFS_SUPER_INFO_OFFSET);
 
-	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0,
+	csum_tree_block_size(buf, btrfs_csums[BTRFS_CSUM_TYPE_CRC32].size, 0,
 			     btrfs_super_csum_type(super));
 	ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
 		BTRFS_SUPER_INFO_OFFSET);
diff --git a/crypto/hash.c b/crypto/hash.c
new file mode 100644
index 000000000000..fda7fc4e9f23
--- /dev/null
+++ b/crypto/hash.c
@@ -0,0 +1,16 @@
+#include "crypto/hash.h"
+#include "crypto/xxhash.h"
+
+int hash_xxhash(const u8 *buf, size_t length, u8 *out)
+{
+	XXH64_hash_t hash;
+
+	hash = XXH64(buf, length, 0);
+	/* NOTE: we're not taking the canonical form here but the plain hash to
+	 * be compatible with the kernel implementation!
+	 */
+	memcpy(out, &hash, 8);
+
+	return 0;
+}
+
diff --git a/crypto/hash.h b/crypto/hash.h
new file mode 100644
index 000000000000..45c1ef17bc57
--- /dev/null
+++ b/crypto/hash.h
@@ -0,0 +1,10 @@
+#ifndef CRYPTO_HASH_H
+#define CRYPTO_HASH_H
+
+#include "../kerncompat.h"
+
+#define CRYPTO_HASH_SIZE_MAX	32
+
+int hash_xxhash(const u8 *buf, size_t length, u8 *out);
+
+#endif
diff --git a/crypto/xxhash.c b/crypto/xxhash.c
index af9d02795ac6..7f381c8b56a0 100644
--- a/crypto/xxhash.c
+++ b/crypto/xxhash.c
@@ -1018,7 +1018,7 @@ XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src
 *  New generation hash designed for speed on small keys and vectorization
 ************************************************************************ */
 
-#include "xxh3.h"
+/* #include "xxh3.h" */
 
 
 #endif  /* XXH_NO_LONG_LONG */
diff --git a/ctree.h b/ctree.h
index b73abe140b1c..b0671f41a8d3 100644
--- a/ctree.h
+++ b/ctree.h
@@ -165,10 +165,18 @@ struct btrfs_free_space_ctl;
 #define BTRFS_CSUM_SIZE 32
 
 /* csum types */
-#define BTRFS_CSUM_TYPE_CRC32	0
+enum btrfs_csum_type {
+	BTRFS_CSUM_TYPE_CRC32	= 0,
+	BTRFS_CSUM_TYPE_XXHASH	= 1,
+};
 
-/* four bytes for CRC32 */
-static int btrfs_csum_sizes[] = { 4 };
+static struct btrfs_csum {
+	u16 size;
+	const char *name;
+} btrfs_csums[] = {
+	[BTRFS_CSUM_TYPE_CRC32] = { 4, "crc32c" },
+	[BTRFS_CSUM_TYPE_XXHASH] = { 8, "xxhash64" },
+};
 
 #define BTRFS_EMPTY_DIR_SIZE 0
 
@@ -2264,8 +2272,8 @@ BTRFS_SETGET_STACK_FUNCS(super_magic, struct btrfs_super_block, magic, 64);
 static inline int btrfs_super_csum_size(struct btrfs_super_block *s)
 {
 	int t = btrfs_super_csum_type(s);
-	BUG_ON(t >= ARRAY_SIZE(btrfs_csum_sizes));
-	return btrfs_csum_sizes[t];
+	BUG_ON(t >= ARRAY_SIZE(btrfs_csums));
+	return btrfs_csums[t].size;
 }
 
 static inline unsigned long btrfs_leaf_data(struct extent_buffer *l)
diff --git a/disk-io.c b/disk-io.c
index 810c2e14294a..ce0b746f4db9 100644
--- a/disk-io.c
+++ b/disk-io.c
@@ -34,6 +34,7 @@
 #include "print-tree.h"
 #include "common/rbtree-utils.h"
 #include "common/device-scan.h"
+#include "crypto/hash.h"
 
 /* specified errno for check_tree_block */
 #define BTRFS_BAD_BYTENR		(-1)
@@ -148,6 +149,8 @@ int btrfs_csum_data(u16 csum_type, const u8 *data, u8 *out, size_t len)
 		crc = crc32c(crc, data, len);
 		put_unaligned_le32(~crc, out);
 		return 0;
+	case BTRFS_CSUM_TYPE_XXHASH:
+		return hash_xxhash(data, len, out);
 	default:
 		fprintf(stderr, "ERROR: unknown csum type: %d\n", csum_type);
 		ASSERT(0);
@@ -1376,11 +1379,11 @@ int btrfs_check_super(struct btrfs_super_block *sb, unsigned sbflags)
 	}
 
 	csum_type = btrfs_super_csum_type(sb);
-	if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) {
+	if (csum_type >= ARRAY_SIZE(btrfs_csums)) {
 		error("unsupported checksum algorithm %u", csum_type);
 		return -EIO;
 	}
-	csum_size = btrfs_csum_sizes[csum_type];
+	csum_size = btrfs_csums[csum_type].size;
 
 	btrfs_csum_data(csum_type, (u8 *)sb + BTRFS_CSUM_SIZE,
 			result, BTRFS_SUPER_INFO_SIZE - BTRFS_CSUM_SIZE);
diff --git a/image/main.c b/image/main.c
index 0c8ffede56f5..1265152cf524 100644
--- a/image/main.c
+++ b/image/main.c
@@ -121,11 +121,12 @@ static struct extent_buffer *alloc_dummy_eb(u64 bytenr, u32 size);
 
 static void csum_block(u8 *buf, size_t len)
 {
-	u8 result[btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]];
+	u16 csum_size = btrfs_csums[BTRFS_CSUM_TYPE_CRC32].size;
+	u8 result[csum_size];
 	u32 crc = ~(u32)0;
 	crc = crc32c(crc, buf + BTRFS_CSUM_SIZE, len - BTRFS_CSUM_SIZE);
 	put_unaligned_le32(~crc, result);
-	memcpy(buf, result, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32]);
+	memcpy(buf, result, csum_size);
 }
 
 static int has_name(struct btrfs_key *key)
diff --git a/mkfs/common.c b/mkfs/common.c
index 4a417bd7a306..939be5eb2dc2 100644
--- a/mkfs/common.c
+++ b/mkfs/common.c
@@ -101,7 +101,7 @@ static int btrfs_create_tree_root(int fd, struct btrfs_mkfs_config *cfg,
 	}
 
 	/* generate checksum */
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
 			     cfg->csum_type);
 
 	/* write back root tree */
@@ -293,7 +293,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_EXTENT_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_EXTENT_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, nritems);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
 			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_EXTENT_TREE]);
 	if (ret != cfg->nodesize) {
@@ -382,7 +382,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CHUNK_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_CHUNK_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, nritems);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
 			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CHUNK_TREE]);
 	if (ret != cfg->nodesize) {
@@ -423,7 +423,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_DEV_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_DEV_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, nritems);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
 			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_DEV_TREE]);
 	if (ret != cfg->nodesize) {
@@ -437,7 +437,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_FS_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_FS_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, 0);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
 			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_FS_TREE]);
 	if (ret != cfg->nodesize) {
@@ -450,7 +450,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	btrfs_set_header_bytenr(buf, cfg->blocks[MKFS_CSUM_TREE]);
 	btrfs_set_header_owner(buf, BTRFS_CSUM_TREE_OBJECTID);
 	btrfs_set_header_nritems(buf, 0);
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
 			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, cfg->nodesize, cfg->blocks[MKFS_CSUM_TREE]);
 	if (ret != cfg->nodesize) {
@@ -462,7 +462,7 @@ int make_btrfs(int fd, struct btrfs_mkfs_config *cfg)
 	memset(buf->data, 0, BTRFS_SUPER_INFO_SIZE);
 	memcpy(buf->data, &super, sizeof(super));
 	buf->len = BTRFS_SUPER_INFO_SIZE;
-	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
+	csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
 			     cfg->csum_type);
 	ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
 			cfg->blocks[MKFS_SUPER_BLOCK]);
diff --git a/mkfs/main.c b/mkfs/main.c
index 075e7e331ab4..c78481da50c2 100644
--- a/mkfs/main.c
+++ b/mkfs/main.c
@@ -385,6 +385,9 @@ static enum btrfs_csum_type parse_csum_type(const char *s)
 {
 	if (strcasecmp(s, "crc32c") == 0) {
 		return BTRFS_CSUM_TYPE_CRC32;
+	} else if (strcasecmp(s, "xxhash64") == 0 ||
+		   strcasecmp(s, "xxhash") == 0) {
+		return BTRFS_CSUM_TYPE_XXHASH;
 	} else {
 		error("unknown csum type %s", s);
 		exit(1);
@@ -1370,7 +1373,8 @@ raid_groups:
 			pretty_size(allocation.system));
 		printf("SSD detected:       %s\n", ssd ? "yes" : "no");
 		btrfs_parse_features_to_string(features_buf, features);
-		printf("Incompat features:  %s", features_buf);
+		printf("Incompat features:  %s\n", features_buf);
+		printf("Checksum:           %s", btrfs_csums[csum_type].name);
 		printf("\n");
 
 		list_all_devices(root);
-- 
2.16.4

Re: [PATCH v2 10/11] btrfs-progs: add xxhash64 as checksum algorithm

[Nikolay Borisov]

On 26.08.19 г. 14:48 ч., Johannes Thumshirn wrote:
> From: David Sterba <dsterba@suse.com>
> 
> Add xxhash64 as another checksumming algorithm.
> 
> Signed-off-by: David Sterba <dsterba@suse.com>
> Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
> ---
>  Makefile                  |  3 ++-
>  cmds/inspect-dump-super.c | 25 ++++++++++++++++---------
>  convert/common.c          |  2 +-
>  convert/main.c            |  2 +-
>  crypto/hash.c             | 16 ++++++++++++++++
>  crypto/hash.h             | 10 ++++++++++
>  crypto/xxhash.c           |  2 +-
>  ctree.h                   | 18 +++++++++++++-----
>  disk-io.c                 |  7 +++++--
>  image/main.c              |  5 +++--
>  mkfs/common.c             | 14 +++++++-------
>  mkfs/main.c               |  6 +++++-
>  12 files changed, 80 insertions(+), 30 deletions(-)
>  create mode 100644 crypto/hash.c
>  create mode 100644 crypto/hash.h
> 
> diff --git a/Makefile b/Makefile
> index 82417d19a9f8..1982e6f5d70e 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -151,7 +151,8 @@ cmds_objects = cmds/subvolume.o cmds/filesystem.o cmds/device.o cmds/scrub.o \
>  	       mkfs/common.o check/mode-common.o check/mode-lowmem.o
>  libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o \
>  		   kernel-lib/crc32c.o common/messages.o \
> -		   uuid-tree.o utils-lib.o common/rbtree-utils.o
> +		   uuid-tree.o utils-lib.o common/rbtree-utils.o \
> +		   crypto/hash.o crypto/xxhash.o
>  libbtrfs_headers = send-stream.h send-utils.h send.h kernel-lib/rbtree.h btrfs-list.h \
>  	       kernel-lib/crc32c.h kernel-lib/list.h kerncompat.h \
>  	       kernel-lib/radix-tree.h kernel-lib/sizes.h kernel-lib/raid56.h \
> diff --git a/cmds/inspect-dump-super.c b/cmds/inspect-dump-super.c
> index 58bf82b0bbd3..1001c8aa5c85 100644
> --- a/cmds/inspect-dump-super.c
> +++ b/cmds/inspect-dump-super.c
> @@ -311,6 +311,17 @@ static void print_readable_super_flag(u64 flag)
>  				     super_flags_num, BTRFS_SUPER_FLAG_SUPP);
>  }
>  
> +static bool is_valid_csum_type(u16 csum_type)
> +{
> +	switch (csum_type) {
> +	case BTRFS_CSUM_TYPE_CRC32:
> +	case BTRFS_CSUM_TYPE_XXHASH:
> +		return true;
> +	default:
> +		return false;
> +	}
> +}
> +
>  static void dump_superblock(struct btrfs_super_block *sb, int full)
>  {
>  	int i;
> @@ -326,15 +337,11 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
>  	csum_type = btrfs_super_csum_type(sb);
>  	csum_size = BTRFS_CSUM_SIZE;
>  	printf("csum_type\t\t%hu (", csum_type);
> -	if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) {
> +	if (csum_type >= ARRAY_SIZE(btrfs_csums)) {
Why not is_valid_csum_type ?

>  		printf("INVALID");
>  	} else {
> -		if (csum_type == BTRFS_CSUM_TYPE_CRC32) {
> -			printf("crc32c");
> -			csum_size = btrfs_csum_sizes[csum_type];
> -		} else {
> -			printf("unknown");
> -		}
> +		printf("%s", btrfs_csums[csum_type].name);
> +		csum_size = btrfs_csums[csum_type].size;
>  	}
>  	printf(")\n");
>  	printf("csum_size\t\t%llu\n", (unsigned long long)csum_size);
> @@ -342,8 +349,8 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
>  	printf("csum\t\t\t0x");
>  	for (i = 0, p = sb->csum; i < csum_size; i++)
>  		printf("%02x", p[i]);
> -	if (csum_type != BTRFS_CSUM_TYPE_CRC32 ||
> -	    csum_size != btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32])
> +	if (!is_valid_csum_type(csum_type) ||
> +	    csum_size != btrfs_csums[csum_type].size)

That second check - can it ever trigger? If the csum_type >= ARRAY_SIZE
goes into the else branch then csum_size == btrfs_csums[csum_type].size
so this check is guaranteed to never fail. OTOH, if we print invalid
above then csum_type is guaranteed to be above ARRAY_SIZE(btrfs_csums)
and I thin this guarantees that !is_valid_csum_type(csum_type) is going
to be true e.g. we will print UNKNOWN CSUM type. So I guess a simple

'if (!is_valid_csum_type(csum_type)' will suffice here?


>  		printf(" [UNKNOWN CSUM TYPE OR SIZE]");
>  	else if (check_csum_sblock(sb, csum_size, csum_type))
>  		printf(" [match]");
> diff --git a/convert/common.c b/convert/common.c
> index 8f5fdbf507a4..e479d70f4e9e 100644
> --- a/convert/common.c
> +++ b/convert/common.c
> @@ -223,7 +223,7 @@ static inline int write_temp_extent_buffer(int fd, struct extent_buffer *buf,
>  {
>  	int ret;
>  
> -	csum_tree_block_size(buf, btrfs_csum_sizes[cfg->csum_type], 0,
> +	csum_tree_block_size(buf, btrfs_csums[cfg->csum_type].size, 0,
>  			     cfg->csum_type);
>  
>  	/* Temporary extent buffer is always mapped 1:1 on disk */
> diff --git a/convert/main.c b/convert/main.c
> index 5e6b12431f59..5eb2a59fb68a 100644
> --- a/convert/main.c
> +++ b/convert/main.c
> @@ -1058,7 +1058,7 @@ static int migrate_super_block(int fd, u64 old_bytenr)
>  	BUG_ON(btrfs_super_bytenr(super) != old_bytenr);
>  	btrfs_set_super_bytenr(super, BTRFS_SUPER_INFO_OFFSET);
>  
> -	csum_tree_block_size(buf, btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32], 0,
> +	csum_tree_block_size(buf, btrfs_csums[BTRFS_CSUM_TYPE_CRC32].size, 0,
>  			     btrfs_super_csum_type(super));
>  	ret = pwrite(fd, buf->data, BTRFS_SUPER_INFO_SIZE,
>  		BTRFS_SUPER_INFO_OFFSET);
> diff --git a/crypto/hash.c b/crypto/hash.c
> new file mode 100644
> index 000000000000..fda7fc4e9f23
> --- /dev/null
> +++ b/crypto/hash.c
> @@ -0,0 +1,16 @@
> +#include "crypto/hash.h"
> +#include "crypto/xxhash.h"
> +
> +int hash_xxhash(const u8 *buf, size_t length, u8 *out)
> +{
> +	XXH64_hash_t hash;
> +
> +	hash = XXH64(buf, length, 0);
> +	/* NOTE: we're not taking the canonical form here but the plain hash to
> +	 * be compatible with the kernel implementation!
> +	 */
> +	memcpy(out, &hash, 8);
> +
> +	return 0;
> +}
> +
> diff --git a/crypto/hash.h b/crypto/hash.h
> new file mode 100644
> index 000000000000..45c1ef17bc57
> --- /dev/null
> +++ b/crypto/hash.h
> @@ -0,0 +1,10 @@
> +#ifndef CRYPTO_HASH_H
> +#define CRYPTO_HASH_H
> +
> +#include "../kerncompat.h"
> +
> +#define CRYPTO_HASH_SIZE_MAX	32
> +
> +int hash_xxhash(const u8 *buf, size_t length, u8 *out);
> +
> +#endif
> diff --git a/crypto/xxhash.c b/crypto/xxhash.c
> index af9d02795ac6..7f381c8b56a0 100644
> --- a/crypto/xxhash.c
> +++ b/crypto/xxhash.c
> @@ -1018,7 +1018,7 @@ XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(const XXH64_canonical_t* src
>  *  New generation hash designed for speed on small keys and vectorization
>  ************************************************************************ */
>  
> -#include "xxh3.h"
> +/* #include "xxh3.h" */

Does that mean progs compilation is broken by the previous patch since
it includes a file which cannot be found?


>  
>  
>  #endif  /* XXH_NO_LONG_LONG */


<snip>

> diff --git a/mkfs/main.c b/mkfs/main.c
> index 075e7e331ab4..c78481da50c2 100644
> --- a/mkfs/main.c
> +++ b/mkfs/main.c
> @@ -385,6 +385,9 @@ static enum btrfs_csum_type parse_csum_type(const char *s)
>  {
>  	if (strcasecmp(s, "crc32c") == 0) {
>  		return BTRFS_CSUM_TYPE_CRC32;
> +	} else if (strcasecmp(s, "xxhash64") == 0 ||
> +		   strcasecmp(s, "xxhash") == 0) {

Don't we want to be very explicit about only supporting xxhash64, and
not aliasing xxhash to mean xxhash64? I.e remove the xxhash comparison
and consider it invalid.

> +		return BTRFS_CSUM_TYPE_XXHASH;
>  	} else {
>  		error("unknown csum type %s", s);
>  		exit(1);
> @@ -1370,7 +1373,8 @@ raid_groups:
>  			pretty_size(allocation.system));
>  		printf("SSD detected:       %s\n", ssd ? "yes" : "no");
>  		btrfs_parse_features_to_string(features_buf, features);
> -		printf("Incompat features:  %s", features_buf);
> +		printf("Incompat features:  %s\n", features_buf);
> +		printf("Checksum:           %s", btrfs_csums[csum_type].name);
>  		printf("\n");
>  
>  		list_all_devices(root);
> 

Re: [PATCH v2 10/11] btrfs-progs: add xxhash64 as checksum algorithm

[David Sterba]

On Tue, Aug 27, 2019 at 05:16:31PM +0300, Nikolay Borisov wrote:
> > --- a/mkfs/main.c
> > +++ b/mkfs/main.c
> > @@ -385,6 +385,9 @@ static enum btrfs_csum_type parse_csum_type(const char *s)
> >  {
> >  	if (strcasecmp(s, "crc32c") == 0) {
> >  		return BTRFS_CSUM_TYPE_CRC32;
> > +	} else if (strcasecmp(s, "xxhash64") == 0 ||
> > +		   strcasecmp(s, "xxhash") == 0) {
> 
> Don't we want to be very explicit about only supporting xxhash64, and
> not aliasing xxhash to mean xxhash64? I.e remove the xxhash comparison
> and consider it invalid.

The aliases are for user convenience, I believe xxhash is widely
understood as xxhash64 so '64' is not strictly necessary. If we
ever introduce the 128bit version of xxhash, then it'll be probably
called XXH3 anyway.

Re: [PATCH v2 10/11] btrfs-progs: add xxhash64 as checksum algorithm

[Johannes Thumshirn]

On 27/08/2019 16:16, Nikolay Borisov wrote:
[...]

>>  static void dump_superblock(struct btrfs_super_block *sb, int full)
>>  {
>>  	int i;
>> @@ -326,15 +337,11 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
>>  	csum_type = btrfs_super_csum_type(sb);
>>  	csum_size = BTRFS_CSUM_SIZE;
>>  	printf("csum_type\t\t%hu (", csum_type);
>> -	if (csum_type >= ARRAY_SIZE(btrfs_csum_sizes)) {
>> +	if (csum_type >= ARRAY_SIZE(btrfs_csums)) {
> Why not is_valid_csum_type ?

Fixed.

>>  		printf("INVALID");
>>  	} else {
>> -		if (csum_type == BTRFS_CSUM_TYPE_CRC32) {
>> -			printf("crc32c");
>> -			csum_size = btrfs_csum_sizes[csum_type];
>> -		} else {
>> -			printf("unknown");
>> -		}
>> +		printf("%s", btrfs_csums[csum_type].name);
>> +		csum_size = btrfs_csums[csum_type].size;
>>  	}
>>  	printf(")\n");
>>  	printf("csum_size\t\t%llu\n", (unsigned long long)csum_size);
>> @@ -342,8 +349,8 @@ static void dump_superblock(struct btrfs_super_block *sb, int full)
>>  	printf("csum\t\t\t0x");
>>  	for (i = 0, p = sb->csum; i < csum_size; i++)
>>  		printf("%02x", p[i]);
>> -	if (csum_type != BTRFS_CSUM_TYPE_CRC32 ||
>> -	    csum_size != btrfs_csum_sizes[BTRFS_CSUM_TYPE_CRC32])
>> +	if (!is_valid_csum_type(csum_type) ||
>> +	    csum_size != btrfs_csums[csum_type].size)
> 
> That second check - can it ever trigger? If the csum_type >= ARRAY_SIZE
> goes into the else branch then csum_size == btrfs_csums[csum_type].size
> so this check is guaranteed to never fail. OTOH, if we print invalid
> above then csum_type is guaranteed to be above ARRAY_SIZE(btrfs_csums)
> and I thin this guarantees that !is_valid_csum_type(csum_type) is going
> to be true e.g. we will print UNKNOWN CSUM type. So I guess a simple
> 
> 'if (!is_valid_csum_type(csum_type)' will suffice here?

Right, fixed as well.

[...]

>> -#include "xxh3.h"
>> +/* #include "xxh3.h" */
> 
> Does that mean progs compilation is broken by the previous patch since
> it includes a file which cannot be found?

It broke bisectability, fix it now.


[...]
>>  	if (strcasecmp(s, "crc32c") == 0) {
>>  		return BTRFS_CSUM_TYPE_CRC32;
>> +	} else if (strcasecmp(s, "xxhash64") == 0 ||
>> +		   strcasecmp(s, "xxhash") == 0) {
> 
> Don't we want to be very explicit about only supporting xxhash64, and
> not aliasing xxhash to mean xxhash64? I.e remove the xxhash comparison
> and consider it invalid.

I'll keep that alias as per Dave's comment.

Thanks,
	Johannes
-- 
Johannes Thumshirn                            SUSE Labs Filesystems
jthumshirn@suse.de                                +49 911 74053 689
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5
90409 Nürnberg
Germany
(HRB 247165, AG München)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

[PATCH v2 11/11] btrfs-progs: add test-case for mkfs with xxhash64

[Johannes Thumshirn]

Add test-cases for creating a file-system xxhash64 as checksumming
algorithm.

Signed-off-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 tests/mkfs-tests/001-basic-profiles/test.sh | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tests/mkfs-tests/001-basic-profiles/test.sh b/tests/mkfs-tests/001-basic-profiles/test.sh
index 6e295274119d..3fa3c8ad42d1 100755
--- a/tests/mkfs-tests/001-basic-profiles/test.sh
+++ b/tests/mkfs-tests/001-basic-profiles/test.sh
@@ -46,6 +46,8 @@ test_mkfs_single  -d  single  -m  dup
 test_mkfs_single  -d  dup     -m  single
 test_mkfs_single  -d  dup     -m  dup
 test_mkfs_single  -d  dup     -m  dup     --mixed
+test_mkfs_single  -C xxhash64
+test_mkfs_single  -C xxhash
 
 test_mkfs_multi
 test_mkfs_multi   -d  single  -m  single
-- 
2.16.4