Btrfs progs release 6.2

Btrfs progs release 6.2

[David Sterba]

Hi,

btrfs-progs version 6.2 have been released.

Changelog:
   * receive: fix a corruption when decompressing zstd extents
   * subvol sync: print total number and deletion progress
   * accelerated hash algorithm implementations in fallback mode on x86_64
   * fi mkswapfile: new option --uuid
   * new global option --log=level to set the verbosity level directly
   * other:
      * experimental: update checksum conversion (not usable yet)
      * build actually requires -std=gnu11
      * refactor help option formatting, auto wrap long lines

Tarballs: https://www.kernel.org/pub/linux/kernel/people/kdave/btrfs-progs/
Git: git://git.kernel.org/pub/scm/linux/kernel/git/kdave/btrfs-progs.git

Re: Btrfs progs release 6.2

[Neal Gompa]

On Tue, Feb 28, 2023 at 3:07 PM David Sterba <dsterba@suse.com> wrote:
>
> Hi,
>
> btrfs-progs version 6.2 have been released.
>
> Changelog:
>    * receive: fix a corruption when decompressing zstd extents
>    * subvol sync: print total number and deletion progress
>    * accelerated hash algorithm implementations in fallback mode on x86_64
>    * fi mkswapfile: new option --uuid
>    * new global option --log=level to set the verbosity level directly
>    * other:
>       * experimental: update checksum conversion (not usable yet)
>       * build actually requires -std=gnu11
>       * refactor help option formatting, auto wrap long lines
>
> Tarballs: https://www.kernel.org/pub/linux/kernel/people/kdave/btrfs-progs/
> Git: git://git.kernel.org/pub/scm/linux/kernel/git/kdave/btrfs-progs.git

This release does not compile on Fedora Rawhide.

With the current btrfs-progs spec file in Fedora bumped to 6.2, I get
this error:

>     [LD]     btrfstune
> gcc -o btrfstune tune/main.o tune/seeding.o tune/change-uuid.o tune/change-metadata-uuid.o tune/convert-bgt.o tune/change-csum.o kernel-lib/list_sort.o kernel-lib/raid56.o kernel-lib/rbtree.o kernel-lib/tables.o kernel-shared/backref.o kernel-shared/ctree.o kernel-shared/delayed-ref.o kernel-shared/dir-item.o kernel-shared/disk-io.o kernel-shared/extent-tree.o kernel-shared/extent_io.o kernel-shared/file-item.o kernel-shared/file.o kernel-shared/free-space-cache.o kernel-shared/free-space-tree.o kernel-shared/inode-item.o kernel-shared/inode.o kernel-shared/print-tree.o kernel-shared/root-tree.o kernel-shared/transaction.o kernel-shared/ulist.o kernel-shared/uuid-tree.o kernel-shared/volumes.o kernel-shared/zoned.o common/cpu-utils.o common/device-scan.o common/device-utils.o common/extent-cache.o common/filesystem-utils.o common/format-output.o common/fsfeatures.o common/help.o common/messages.o common/open-utils.o common/parse-utils.o common/path-utils.o common/rbtree-utils.o common/send-stream.o common/send-utils.o common/string-table.o common/string-utils.o common/task-utils.o common/units.o common/utils.o check/qgroup-verify.o check/repair.o cmds/receive-dump.o crypto/crc32c.o crypto/hash.o crypto/xxhash.o libbtrfsutil/stubs.o libbtrfsutil/subvolume.o libbtrfsutil.a -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes  -rdynamic -L.   -luuid -lblkid -ludev -L. -pthread -lgcrypt
> /usr/bin/ld: /tmp/ccPRdR5s.ltrans5.ltrans.o: in function `hash_init_accel':
> /builddir/build/BUILD/btrfs-progs-v6.2/crypto/hash.c:26: undefined reference to `blake2_init_accel'
> /usr/bin/ld: /builddir/build/BUILD/btrfs-progs-v6.2/crypto/hash.c:27: undefined reference to `sha256_init_accel'
> /usr/bin/ld: /tmp/ccPRdR5s.ltrans5.ltrans.o: in function `hash_init_blake2':
> /builddir/build/BUILD/btrfs-progs-v6.2/crypto/hash.c:37: undefined reference to `blake2_init_accel'
> /usr/bin/ld: /tmp/ccPRdR5s.ltrans5.ltrans.o: in function `hash_init_sha256':
> /builddir/build/BUILD/btrfs-progs-v6.2/crypto/hash.c:42: undefined reference to `sha256_init_accel'
> collect2: error: ld returned 1 exit status
> make: *** [Makefile:673: btrfstune] Error 1


-- 
真実はいつも一つ！/ Always, there's only one truth!

Re: Btrfs progs release 6.2

[David Sterba]

On Tue, Feb 28, 2023 at 05:46:10PM -0500, Neal Gompa wrote:
> On Tue, Feb 28, 2023 at 3:07 PM David Sterba <dsterba@suse.com> wrote:
> >
> > Hi,
> >
> > btrfs-progs version 6.2 have been released.
> >
> > Changelog:
> >    * receive: fix a corruption when decompressing zstd extents
> >    * subvol sync: print total number and deletion progress
> >    * accelerated hash algorithm implementations in fallback mode on x86_64
> >    * fi mkswapfile: new option --uuid
> >    * new global option --log=level to set the verbosity level directly
> >    * other:
> >       * experimental: update checksum conversion (not usable yet)
> >       * build actually requires -std=gnu11
> >       * refactor help option formatting, auto wrap long lines
> >
> > Tarballs: https://www.kernel.org/pub/linux/kernel/people/kdave/btrfs-progs/
> > Git: git://git.kernel.org/pub/scm/linux/kernel/git/kdave/btrfs-progs.git
> 
> This release does not compile on Fedora Rawhide.
> 
> With the current btrfs-progs spec file in Fedora bumped to 6.2, I get
> this error:
> 
> >     [LD]     btrfstune
> > gcc -o btrfstune tune/main.o tune/seeding.o tune/change-uuid.o tune/change-metadata-uuid.o tune/convert-bgt.o tune/change-csum.o kernel-lib/list_sort.o kernel-lib/raid56.o kernel-lib/rbtree.o kernel-lib/tables.o kernel-shared/backref.o kernel-shared/ctree.o kernel-shared/delayed-ref.o kernel-shared/dir-item.o kernel-shared/disk-io.o kernel-shared/extent-tree.o kernel-shared/extent_io.o kernel-shared/file-item.o kernel-shared/file.o kernel-shared/free-space-cache.o kernel-shared/free-space-tree.o kernel-shared/inode-item.o kernel-shared/inode.o kernel-shared/print-tree.o kernel-shared/root-tree.o kernel-shared/transaction.o kernel-shared/ulist.o kernel-shared/uuid-tree.o kernel-shared/volumes.o kernel-shared/zoned.o common/cpu-utils.o common/device-scan.o common/device-utils.o common/extent-cache.o common/filesystem-utils.o common/format-output.o common/fsfeatures.o common/help.o common/messages.o common/open-utils.o common/parse-utils.o common/path-utils.o common/rbtree-utils.o common/send-stream.o common/send-utils.o common/string-table.o common/string-utils.o common/task-utils.o common/units.o common/utils.o check/qgroup-verify.o check/repair.o cmds/receive-dump.o crypto/crc32c.o crypto/hash.o crypto/xxhash.o libbtrfsutil/stubs.o libbtrfsutil/subvolume.o libbtrfsutil.a -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes  -rdynamic -L.   -luuid -lblkid -ludev -L. -pthread -lgcrypt
> > /usr/bin/ld: /tmp/ccPRdR5s.ltrans5.ltrans.o: in function `hash_init_accel':
> > /builddir/build/BUILD/btrfs-progs-v6.2/crypto/hash.c:26: undefined reference to `blake2_init_accel'
> > /usr/bin/ld: /builddir/build/BUILD/btrfs-progs-v6.2/crypto/hash.c:27: undefined reference to `sha256_init_accel'
> > /usr/bin/ld: /tmp/ccPRdR5s.ltrans5.ltrans.o: in function `hash_init_blake2':
> > /builddir/build/BUILD/btrfs-progs-v6.2/crypto/hash.c:37: undefined reference to `blake2_init_accel'
> > /usr/bin/ld: /tmp/ccPRdR5s.ltrans5.ltrans.o: in function `hash_init_sha256':
> > /builddir/build/BUILD/btrfs-progs-v6.2/crypto/hash.c:42: undefined reference to `sha256_init_accel'
> > collect2: error: ld returned 1 exit status
> > make: *** [Makefile:673: btrfstune] Error 1

I've been chasing this error on various build target and compiler
combinations. What's the gcc version and architecture?

Re: Btrfs progs release 6.2

[Tomasz Kłoczko]

On Tue, 28 Feb 2023 at 20:07, David Sterba <dsterba@suse.com> wrote:
>
> Hi,
>
> btrfs-progs version 6.2 have been released.
>
> Changelog:
>    * receive: fix a corruption when decompressing zstd extents
>    * subvol sync: print total number and deletion progress
>    * accelerated hash algorithm implementations in fallback mode on x86_64
>    * fi mkswapfile: new option --uuid
>    * new global option --log=level to set the verbosity level directly
>    * other:
>       * experimental: update checksum conversion (not usable yet)
>       * build actually requires -std=gnu11
>       * refactor help option formatting, auto wrap long lines

Just tested new dist tar ball and looks like something is wrong
because linking fails on:

    [LD]     btrfs
/usr/bin/gcc -o btrfs btrfs.o kernel-lib/list_sort.o
kernel-lib/raid56.o kernel-lib/rbtree.o kernel-lib/tables.o
kernel-shared/backref.o kernel-shared/ctree.o
kernel-shared/delayed-ref.o kernel-shared/dir-item.o
kernel-shared/disk-io.o kernel-shared/extent-tree.o
kernel-shared/extent_io.o kernel-shared/file-item.o
kernel-shared/file.o kernel-shared/free-space-cache.o
kernel-shared/free-space-tree.o kernel-shared/inode-item.o
kernel-shared/inode.o kernel-shared/print-tree.o
kernel-shared/root-tree.o kernel-shared/transaction.o
kernel-shared/ulist.o kernel-shared/uuid-tree.o
kernel-shared/volumes.o kernel-shared/zoned.o common/cpu-utils.o
common/device-scan.o common/device-utils.o common/extent-cache.o
common/filesystem-utils.o common/format-output.o common/fsfeatures.o
common/help.o common/messages.o common/open-utils.o
common/parse-utils.o common/path-utils.o common/rbtree-utils.o
common/send-stream.o common/send-utils.o common/string-table.o
common/string-utils.o common/task-utils.o common/units.o
common/utils.o check/qgroup-verify.o check/repair.o
cmds/receive-dump.o crypto/crc32c.o crypto/hash.o crypto/xxhash.o
libbtrfsutil/stubs.o libbtrfsutil/subvolume.o cmds/subvolume.o
cmds/subvolume-list.o cmds/filesystem.o cmds/device.o cmds/scrub.o
cmds/inspect.o cmds/balance.o cmds/send.o cmds/receive.o cmds/quota.o
cmds/qgroup.o cmds/replace.o check/main.o cmds/restore.o cmds/rescue.o
cmds/rescue-chunk-recover.o cmds/rescue-super-recover.o
cmds/property.o cmds/filesystem-usage.o cmds/inspect-dump-tree.o
cmds/inspect-dump-super.o cmds/inspect-tree-stats.o
cmds/filesystem-du.o cmds/reflink.o mkfs/common.o check/mode-common.o
check/mode-lowmem.o check/clear-cache.o libbtrfsutil.a -Wl,--as-needed
-Wl,--gc-sections -Wl,-z,now
-specs=/usr/lib/rpm/redhat/redhat-hardened-ld -flto=auto
-flto-partition=none -fuse-linker-plugin -Wl,--build-id=sha1 -rdynamic
-L.   -luuid -lblkid -ludev -L. -pthread -lkcapi -lz -llzo2 -lzstd
/usr/bin/ld: /tmp/ccTTIULh.lto.o: in function `hash_init_accel':
/home/tkloczko/rpmbuild/BUILD/btrfs-progs-v6.2/crypto/hash.c:26:
undefined reference to `blake2_init_accel'
/usr/bin/ld: /home/tkloczko/rpmbuild/BUILD/btrfs-progs-v6.2/crypto/hash.c:27:
undefined reference to `sha256_init_accel'
/usr/bin/ld: /tmp/ccTTIULh.lto.o: in function `hash_init_blake2':
/home/tkloczko/rpmbuild/BUILD/btrfs-progs-v6.2/crypto/hash.c:37:
undefined reference to `blake2_init_accel'
/usr/bin/ld: /tmp/ccTTIULh.lto.o: in function `hash_init_sha256':
/home/tkloczko/rpmbuild/BUILD/btrfs-progs-v6.2/crypto/hash.c:42:
undefined reference to `sha256_init_accel'
collect2: error: ld returned 1 exit status
make: *** [Makefile:629: btrfs] Error 1

kloczek
--
Tomasz Kłoczko | LinkedIn: http://lnkd.in/FXPWxH

Re: Btrfs progs release 6.2

[David Sterba]

On Wed, Mar 01, 2023 at 08:17:59AM +0800, Qu Wenruo wrote:
> On 2023/3/1 07:07, Tomasz Kłoczko wrote:
> > On Tue, 28 Feb 2023 at 20:07, David Sterba <dsterba@suse.com> wrote:
> cmds/property.o cmds/filesystem-usage.o cmds/inspect-dump-tree.o 
> cmds/inspect-dump-super.o cmds/inspect-tree-stats.o cmds/filesystem-du.o 
> cmds/reflink.o mkfs/common.o check/mode-common.o check/mode-lowmem.o 
> check/clear-cache.o libbtrfsutil.a  -rdynamic -L.   -luuid  -lblkid 
> -ludev  -L. -pthread  -lz  -llzo2 -lzstd
> 
> According to the Makefile, it looks like Fedora build is not using the 
> built-in crypto code.
> 
> If using libsodium, I got the same error, as libsodium goes a different 
> name for its blake2b_init (crypto_generichash_blake2b_init).

Oh right, thanks, I can reproduce it now.

Re: Btrfs progs release 6.2

[Qu Wenruo]

On 2023/3/1 07:07, Tomasz Kłoczko wrote:
> On Tue, 28 Feb 2023 at 20:07, David Sterba <dsterba@suse.com> wrote:
>>
>> Hi,
>>
>> btrfs-progs version 6.2 have been released.
>>
>> Changelog:
>>     * receive: fix a corruption when decompressing zstd extents
>>     * subvol sync: print total number and deletion progress
>>     * accelerated hash algorithm implementations in fallback mode on x86_64
>>     * fi mkswapfile: new option --uuid
>>     * new global option --log=level to set the verbosity level directly
>>     * other:
>>        * experimental: update checksum conversion (not usable yet)
>>        * build actually requires -std=gnu11
>>        * refactor help option formatting, auto wrap long lines
> 
> Just tested new dist tar ball and looks like something is wrong
> because linking fails on:
> 
>      [LD]     btrfs
> /usr/bin/gcc -o btrfs btrfs.o kernel-lib/list_sort.o
> kernel-lib/raid56.o kernel-lib/rbtree.o kernel-lib/tables.o
> kernel-shared/backref.o kernel-shared/ctree.o
> kernel-shared/delayed-ref.o kernel-shared/dir-item.o
> kernel-shared/disk-io.o kernel-shared/extent-tree.o
> kernel-shared/extent_io.o kernel-shared/file-item.o
> kernel-shared/file.o kernel-shared/free-space-cache.o
> kernel-shared/free-space-tree.o kernel-shared/inode-item.o
> kernel-shared/inode.o kernel-shared/print-tree.o
> kernel-shared/root-tree.o kernel-shared/transaction.o
> kernel-shared/ulist.o kernel-shared/uuid-tree.o
> kernel-shared/volumes.o kernel-shared/zoned.o common/cpu-utils.o
> common/device-scan.o common/device-utils.o common/extent-cache.o
> common/filesystem-utils.o common/format-output.o common/fsfeatures.o
> common/help.o common/messages.o common/open-utils.o
> common/parse-utils.o common/path-utils.o common/rbtree-utils.o
> common/send-stream.o common/send-utils.o common/string-table.o
> common/string-utils.o common/task-utils.o common/units.o
> common/utils.o check/qgroup-verify.o check/repair.o
> cmds/receive-dump.o crypto/crc32c.o crypto/hash.o crypto/xxhash.o
> libbtrfsutil/stubs.o libbtrfsutil/subvolume.o cmds/subvolume.o
> cmds/subvolume-list.o cmds/filesystem.o cmds/device.o cmds/scrub.o
> cmds/inspect.o cmds/balance.o cmds/send.o cmds/receive.o cmds/quota.o
> cmds/qgroup.o cmds/replace.o check/main.o cmds/restore.o cmds/rescue.o
> cmds/rescue-chunk-recover.o cmds/rescue-super-recover.o
> cmds/property.o cmds/filesystem-usage.o cmds/inspect-dump-tree.o
> cmds/inspect-dump-super.o cmds/inspect-tree-stats.o
> cmds/filesystem-du.o cmds/reflink.o mkfs/common.o check/mode-common.o
> check/mode-lowmem.o check/clear-cache.o libbtrfsutil.a -Wl,--as-needed
> -Wl,--gc-sections -Wl,-z,now
> -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -flto=auto
> -flto-partition=none -fuse-linker-plugin -Wl,--build-id=sha1 -rdynamic
> -L.   -luuid -lblkid -ludev -L. -pthread -lkcapi -lz -llzo2 -lzstd
> /usr/bin/ld: /tmp/ccTTIULh.lto.o: in function `hash_init_accel':
> /home/tkloczko/rpmbuild/BUILD/btrfs-progs-v6.2/crypto/hash.c:26:
> undefined reference to `blake2_init_accel'
> /usr/bin/ld: /home/tkloczko/rpmbuild/BUILD/btrfs-progs-v6.2/crypto/hash.c:27:
> undefined reference to `sha256_init_accel'
> /usr/bin/ld: /tmp/ccTTIULh.lto.o: in function `hash_init_blake2':
> /home/tkloczko/rpmbuild/BUILD/btrfs-progs-v6.2/crypto/hash.c:37:
> undefined reference to `blake2_init_accel'
> /usr/bin/ld: /tmp/ccTTIULh.lto.o: in function `hash_init_sha256':
> /home/tkloczko/rpmbuild/BUILD/btrfs-progs-v6.2/crypto/hash.c:42:
> undefined reference to `sha256_init_accel'
> collect2: error: ld returned 1 exit status
> make: *** [Makefile:629: btrfs] Error 1

Compared to my passing build (Archlinux, thus most of things should be 
pretty close to rawhide), it looks like Fedora is missing the blake2 and 
sha256 object files:

     [LD]     btrfs
gcc -o btrfs btrfs.o kernel-lib/list_sort.o kernel-lib/raid56.o 
kernel-lib/rbtree.o kernel-lib/tables.o kernel-shared/backref.o 
kernel-shared/ctree.o kernel-shared/delayed-ref.o 
kernel-shared/dir-item.o kernel-shared/disk-io.o 
kernel-shared/extent-tree.o kernel-shared/extent_io.o 
kernel-shared/file-item.o kernel-shared/file.o 
kernel-shared/free-space-cache.o kernel-shared/free-space-tree.o 
kernel-shared/inode-item.o kernel-shared/inode.o 
kernel-shared/print-tree.o kernel-shared/root-tree.o 
kernel-shared/transaction.o kernel-shared/ulist.o 
kernel-shared/uuid-tree.o kernel-shared/volumes.o kernel-shared/zoned.o 
common/cpu-utils.o common/device-scan.o common/device-utils.o 
common/extent-cache.o common/filesystem-utils.o common/format-output.o 
common/fsfeatures.o common/help.o common/messages.o common/open-utils.o 
common/parse-utils.o common/path-utils.o common/rbtree-utils.o 
common/send-stream.o common/send-utils.o common/string-table.o 
common/string-utils.o common/task-utils.o common/units.o common/utils.o 
check/qgroup-verify.o check/repair.o cmds/receive-dump.o crypto/crc32c.o 
crypto/hash.o crypto/xxhash.o crypto/sha224-256.o crypto/blake2b-ref.o 
crypto/blake2b-sse2.o crypto/blake2b-sse41.o crypto/blake2b-avx2.o 
crypto/sha256-x86.o libbtrfsutil/stubs.o libbtrfsutil/subvolume.o 
cmds/subvolume.o cmds/subvolume-list.o cmds/filesystem.o cmds/device.o 
cmds/scrub.o cmds/inspect.o cmds/balance.o cmds/send.o cmds/receive.o 
cmds/quota.o cmds/qgroup.o cmds/replace.o check/main.o cmds/restore.o 
cmds/rescue.o cmds/rescue-chunk-recover.o cmds/rescue-super-recover.o 
cmds/property.o cmds/filesystem-usage.o cmds/inspect-dump-tree.o 
cmds/inspect-dump-super.o cmds/inspect-tree-stats.o cmds/filesystem-du.o 
cmds/reflink.o mkfs/common.o check/mode-common.o check/mode-lowmem.o 
check/clear-cache.o libbtrfsutil.a  -rdynamic -L.   -luuid  -lblkid 
-ludev  -L. -pthread  -lz  -llzo2 -lzstd

According to the Makefile, it looks like Fedora build is not using the 
built-in crypto code.

If using libsodium, I got the same error, as libsodium goes a different 
name for its blake2b_init (crypto_generichash_blake2b_init).

We need more work to support external crypto libraries.

For now, please use builtin crypto to pass the build.

Thanks,
Qu

> 
> kloczek
> --
> Tomasz Kłoczko | LinkedIn: http://lnkd.in/FXPWxH

Re: Btrfs progs release 6.2

[Qu Wenruo]

On 2023/3/1 08:15, David Sterba wrote:
> On Wed, Mar 01, 2023 at 08:17:59AM +0800, Qu Wenruo wrote:
>> On 2023/3/1 07:07, Tomasz Kłoczko wrote:
>>> On Tue, 28 Feb 2023 at 20:07, David Sterba <dsterba@suse.com> wrote:
>> cmds/property.o cmds/filesystem-usage.o cmds/inspect-dump-tree.o
>> cmds/inspect-dump-super.o cmds/inspect-tree-stats.o cmds/filesystem-du.o
>> cmds/reflink.o mkfs/common.o check/mode-common.o check/mode-lowmem.o
>> check/clear-cache.o libbtrfsutil.a  -rdynamic -L.   -luuid  -lblkid
>> -ludev  -L. -pthread  -lz  -llzo2 -lzstd
>>
>> According to the Makefile, it looks like Fedora build is not using the
>> built-in crypto code.
>>
>> If using libsodium, I got the same error, as libsodium goes a different
>> name for its blake2b_init (crypto_generichash_blake2b_init).
> 
> Oh right, thanks, I can reproduce it now.

And bisection points to the following two patches:

bbf703bfd3f68958d33d139eb22057ab397e6c68 btrfs-progs: crypto: call 
sha256 implementations by pointer
d1c366ee42bd3d2abb4fd855ac4a496b720d8bb6 btrfs-progs: crypto: call 
blake2 implementations by pointer

Thanks,
Qu

Re: Btrfs progs release 6.2

[David Sterba]

On Wed, Mar 01, 2023 at 08:30:05AM +0800, Qu Wenruo wrote:
> 
> 
> On 2023/3/1 08:15, David Sterba wrote:
> > On Wed, Mar 01, 2023 at 08:17:59AM +0800, Qu Wenruo wrote:
> >> On 2023/3/1 07:07, Tomasz Kłoczko wrote:
> >>> On Tue, 28 Feb 2023 at 20:07, David Sterba <dsterba@suse.com> wrote:
> >> cmds/property.o cmds/filesystem-usage.o cmds/inspect-dump-tree.o
> >> cmds/inspect-dump-super.o cmds/inspect-tree-stats.o cmds/filesystem-du.o
> >> cmds/reflink.o mkfs/common.o check/mode-common.o check/mode-lowmem.o
> >> check/clear-cache.o libbtrfsutil.a  -rdynamic -L.   -luuid  -lblkid
> >> -ludev  -L. -pthread  -lz  -llzo2 -lzstd
> >>
> >> According to the Makefile, it looks like Fedora build is not using the
> >> built-in crypto code.
> >>
> >> If using libsodium, I got the same error, as libsodium goes a different
> >> name for its blake2b_init (crypto_generichash_blake2b_init).
> > 
> > Oh right, thanks, I can reproduce it now.
> 
> And bisection points to the following two patches:
> 
> bbf703bfd3f68958d33d139eb22057ab397e6c68 btrfs-progs: crypto: call 
> sha256 implementations by pointer
> d1c366ee42bd3d2abb4fd855ac4a496b720d8bb6 btrfs-progs: crypto: call 
> blake2 implementations by pointer

Yeah I know, I did some last minute changes to the commits because 32bit
builds on intel and arm failed due to the flag and feature detection.
The fix is not straightforward but now I have something that works.

Re: Btrfs progs release 6.2

[Neal Gompa]

On Tue, Feb 28, 2023 at 6:08 PM David Sterba <dsterba@suse.cz> wrote:
>
> On Tue, Feb 28, 2023 at 05:46:10PM -0500, Neal Gompa wrote:
> > On Tue, Feb 28, 2023 at 3:07 PM David Sterba <dsterba@suse.com> wrote:
> > >
> > > Hi,
> > >
> > > btrfs-progs version 6.2 have been released.
> > >
> > > Changelog:
> > >    * receive: fix a corruption when decompressing zstd extents
> > >    * subvol sync: print total number and deletion progress
> > >    * accelerated hash algorithm implementations in fallback mode on x86_64
> > >    * fi mkswapfile: new option --uuid
> > >    * new global option --log=level to set the verbosity level directly
> > >    * other:
> > >       * experimental: update checksum conversion (not usable yet)
> > >       * build actually requires -std=gnu11
> > >       * refactor help option formatting, auto wrap long lines
> > >
> > > Tarballs: https://www.kernel.org/pub/linux/kernel/people/kdave/btrfs-progs/
> > > Git: git://git.kernel.org/pub/scm/linux/kernel/git/kdave/btrfs-progs.git
> >
> > This release does not compile on Fedora Rawhide.
> >
> > With the current btrfs-progs spec file in Fedora bumped to 6.2, I get
> > this error:
> >
> > >     [LD]     btrfstune
> > > gcc -o btrfstune tune/main.o tune/seeding.o tune/change-uuid.o tune/change-metadata-uuid.o tune/convert-bgt.o tune/change-csum.o kernel-lib/list_sort.o kernel-lib/raid56.o kernel-lib/rbtree.o kernel-lib/tables.o kernel-shared/backref.o kernel-shared/ctree.o kernel-shared/delayed-ref.o kernel-shared/dir-item.o kernel-shared/disk-io.o kernel-shared/extent-tree.o kernel-shared/extent_io.o kernel-shared/file-item.o kernel-shared/file.o kernel-shared/free-space-cache.o kernel-shared/free-space-tree.o kernel-shared/inode-item.o kernel-shared/inode.o kernel-shared/print-tree.o kernel-shared/root-tree.o kernel-shared/transaction.o kernel-shared/ulist.o kernel-shared/uuid-tree.o kernel-shared/volumes.o kernel-shared/zoned.o common/cpu-utils.o common/device-scan.o common/device-utils.o common/extent-cache.o common/filesystem-utils.o common/format-output.o common/fsfeatures.o common/help.o common/messages.o common/open-utils.o common/parse-utils.o common/path-utils.o common/rbtree-utils.o common/send-stream.o common/send-utils.o common/string-table.o common/string-utils.o common/task-utils.o common/units.o common/utils.o check/qgroup-verify.o check/repair.o cmds/receive-dump.o crypto/crc32c.o crypto/hash.o crypto/xxhash.o libbtrfsutil/stubs.o libbtrfsutil/subvolume.o libbtrfsutil.a -Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes  -rdynamic -L.   -luuid -lblkid -ludev -L. -pthread -lgcrypt
> > > /usr/bin/ld: /tmp/ccPRdR5s.ltrans5.ltrans.o: in function `hash_init_accel':
> > > /builddir/build/BUILD/btrfs-progs-v6.2/crypto/hash.c:26: undefined reference to `blake2_init_accel'
> > > /usr/bin/ld: /builddir/build/BUILD/btrfs-progs-v6.2/crypto/hash.c:27: undefined reference to `sha256_init_accel'
> > > /usr/bin/ld: /tmp/ccPRdR5s.ltrans5.ltrans.o: in function `hash_init_blake2':
> > > /builddir/build/BUILD/btrfs-progs-v6.2/crypto/hash.c:37: undefined reference to `blake2_init_accel'
> > > /usr/bin/ld: /tmp/ccPRdR5s.ltrans5.ltrans.o: in function `hash_init_sha256':
> > > /builddir/build/BUILD/btrfs-progs-v6.2/crypto/hash.c:42: undefined reference to `sha256_init_accel'
> > > collect2: error: ld returned 1 exit status
> > > make: *** [Makefile:673: btrfstune] Error 1
>
> I've been chasing this error on various build target and compiler
> combinations. What's the gcc version and architecture?

GCC 13.0.1 on x86_64. I didn't bother trying other architectures yet.



--
真実はいつも一つ！/ Always, there's only one truth!

Re: Btrfs progs release 6.2

[Qu Wenruo]

On 2023/3/1 09:35, David Sterba wrote:
> On Wed, Mar 01, 2023 at 08:30:05AM +0800, Qu Wenruo wrote:
>>
>>
>> On 2023/3/1 08:15, David Sterba wrote:
>>> On Wed, Mar 01, 2023 at 08:17:59AM +0800, Qu Wenruo wrote:
>>>> On 2023/3/1 07:07, Tomasz Kłoczko wrote:
>>>>> On Tue, 28 Feb 2023 at 20:07, David Sterba <dsterba@suse.com> wrote:
>>>> cmds/property.o cmds/filesystem-usage.o cmds/inspect-dump-tree.o
>>>> cmds/inspect-dump-super.o cmds/inspect-tree-stats.o cmds/filesystem-du.o
>>>> cmds/reflink.o mkfs/common.o check/mode-common.o check/mode-lowmem.o
>>>> check/clear-cache.o libbtrfsutil.a  -rdynamic -L.   -luuid  -lblkid
>>>> -ludev  -L. -pthread  -lz  -llzo2 -lzstd
>>>>
>>>> According to the Makefile, it looks like Fedora build is not using the
>>>> built-in crypto code.
>>>>
>>>> If using libsodium, I got the same error, as libsodium goes a different
>>>> name for its blake2b_init (crypto_generichash_blake2b_init).
>>>
>>> Oh right, thanks, I can reproduce it now.
>>
>> And bisection points to the following two patches:
>>
>> bbf703bfd3f68958d33d139eb22057ab397e6c68 btrfs-progs: crypto: call
>> sha256 implementations by pointer
>> d1c366ee42bd3d2abb4fd855ac4a496b720d8bb6 btrfs-progs: crypto: call
>> blake2 implementations by pointer
> 
> Yeah I know, I did some last minute changes to the commits because 32bit
> builds on intel and arm failed due to the flag and feature detection.
> The fix is not straightforward but now I have something that works.

Just one question, why we support external crypto libraries in the first 
place?

IIRC the built-in ones are already utilizing various optimization, and 
I'd argue that performance should not be the critical aspect for btrfs 
anyway, as only btrfs check and btrfs-restore are really involved for 
performance but they are all single-thread workload, far from 
performance critical.

For best compatibility, the external crypto libs can even be a problem.

So clues are appreciated for why we're supporting external crypto 
libraries for hash functions.

Thanks,
Qu

Re: Btrfs progs release 6.2

[David Sterba]

On Wed, Mar 01, 2023 at 09:00:11PM +0800, Qu Wenruo wrote:
> Just one question, why we support external crypto libraries in the first 
> place?

For static build of btrfs-progs we need a fallback implementation
because distros don't provide static versions of the crypto libraries.

External libraries can get a certification so if distro has to meet the
criteria then it must use only the certified libraries. This was
requested e.g. for SLES back then for the authenticated hashes.  More
libraries give users the freedom to choose the one they use.

> IIRC the built-in ones are already utilizing various optimization,

Only crc32c had an optimized version and not the fastest one, so in 6.2
there's at least one acelerated if possible.

> and 
> I'd argue that performance should not be the critical aspect for btrfs 
> anyway, as only btrfs check and btrfs-restore are really involved for 
> performance but they are all single-thread workload, far from 
> performance critical.

The checksums are still verified at read time and on a fast device the
CPU is the bottleneck, not the IO, so this is basically to keep up with
hardware speeds.

That it's single threaded is actually a problem and there's one issue
asking to make 'check --init-csum-tree' parallel. The proper checksum
switch does only checksumming so this should be accelerated and/or
parallelized. If this can shorten time to do eg. check or uuid switch or
other things then it IMO makes sense to implement.

> For best compatibility, the external crypto libs can even be a problem.

Yes, so there's the fallback.

Re: Btrfs progs release 6.2

[David Sterba]

On Tue, Feb 28, 2023 at 08:23:35PM +0100, David Sterba wrote:
> Hi,
> 
> btrfs-progs version 6.2 have been released.
> 
> Changelog:
>    * receive: fix a corruption when decompressing zstd extents
>    * subvol sync: print total number and deletion progress
>    * accelerated hash algorithm implementations in fallback mode on x86_64
>    * fi mkswapfile: new option --uuid
>    * new global option --log=level to set the verbosity level directly
>    * other:
>       * experimental: update checksum conversion (not usable yet)
>       * build actually requires -std=gnu11
>       * refactor help option formatting, auto wrap long lines
> 
> Tarballs: https://www.kernel.org/pub/linux/kernel/people/kdave/btrfs-progs/
> Git: git://git.kernel.org/pub/scm/linux/kernel/git/kdave/btrfs-progs.git

Fixed version is in devel, I did some more checks on the CI images and
the OBS build on different arches also pass so it should be fine. Bugfix
release will be in a day.