From: Qu Wenruo <quwenruo.btrfs@gmx.com>
To: Piotr Balwierz <nikt@sucha.eu>,
"Konstantin V. Gavrilenko" <k.gavrilenko@arhont.com>,
Linux fs Btrfs <linux-btrfs@vger.kernel.org>
Subject: Re: super_total_bytes 32004083023872 mismatch with fs_devices total_rw_bytes 64008166047744
Date: Wed, 20 Mar 2019 08:58:31 +0800 [thread overview]
Message-ID: <9039b84b-9dc6-f012-9e4c-5e8d19cbcad8@gmx.com> (raw)
In-Reply-To: <13f15a18-c659-1cda-e69c-858a50a53401@sucha.eu>
[-- Attachment #1.1: Type: text/plain, Size: 55376 bytes --]
On 2019/3/20 上午6:36, Piotr Balwierz wrote:
> Hello BTRFS experts,
>
> I am reporting/confirming the same problem present in kernel 4.19.0
> (Debian Buster, x86_64).
> To be specific:
>
> 0) The BTRFS partition used to work fine for 2 years until a power cut
> today. It uses lzo compression. Until power cut kernel version was
> probably 4.14.0
>
> 1) BTRFS partition refused to mount during boot with the following
> kernel and systemd errors:
>
> |Mar 19 15:10:52 mamut kernel: BTRFS error (device dm-5): open_ctree
> failed Mar 19 15:10:52 mamut kernel: BTRFS info (device dm-5): use lzo
> compression, level 0 Mar 19 15:10:52 mamut kernel: BTRFS info (device
> dm-5): disk space caching is enabled Mar 19 15:10:52 mamut kernel: BTRFS
> info (device dm-5): has skinny extents Mar 19 15:10:52 mamut systemd[1]:
> mnt-storage.mount: Mount process exited, code=killed, status=15/TERM Mar
> 19 15:10:52 mamut systemd[1]: mnt-storage.mount: Failed with result
> 'timeout'. Mar 19 15:10:52 mamut systemd[1]: Failed to mount
> /mnt/storage. Mar 19 15:10:52 mamut kernel: BTRFS error (device dm-5):
> super_total_bytes 52798547820544 mismatch with fs_devices total_rw_bytes
> 105597095641088 Mar 19 15:10:52 mamut kernel: BTRFS error (device dm-5):
> failed to read chunk tree: -22 Mar 19 15:10:52 mamut kernel: BTRFS error
> (device dm-5): open_ctree failed|
>
> 2)|Note that total_rw_bytes in kernel error message equals exactly twice
> ||super_total_bytes|. super_total_bytes is the correct size of the
> partition (52TB, 48TiB). It equals to the size reported by fdisk.
> Something being counted twice? The partition is on top of a 8-fold
> multipath to a fibre channel-attached disk array (in case it matters).
>
> 3) the first manual mount failed with a generic message (wrong fs type etc)
>
> 4) the first btrfs check (without --repair) fails with an IO error. I
> used to have no IO errors on this device before.
>
> |# btrfs check /dev/mapper/fc_trunk-part3 Opening filesystem to check...
> Checking filesystem on /dev/mapper/fc_trunk-part3 UUID:
> 40a2e65b-f34a-4d33-946d-055d93fe7ffa [1/7] checking root items ERROR:
> failed to repair root items: Input/output error|
>
> 5) the second btrfs check with --repair succeeds. I did *not* run btrfs
> rescue fix-device-size.
>
> btrfs check --repair /dev/mapper/fc_trunk-part3
> enabling repair mode
> Opening filesystem to check...
> Checking filesystem on /dev/mapper/fc_trunk-part3
> UUID: 40a2e65b-f34a-4d33-946d-055d93fe7ffa
> [1/7] checking root items
> Fixed 0 roots.
> [2/7] checking extents
> No device size related problem found
> [3/7] checking free space cache
> cache and super generation don't match, space cache will be invalidated
> [4/7] checking fs roots
> [5/7] checking only csums items (without verifying data)
> [6/7] checking root refs
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 41629742850048 bytes used, no error found
> total csum bytes: 40569609916
> total tree bytes: 76425707520
> total fs tree bytes: 15665889280
> total extent tree bytes: 16241786880
> btree space waste bytes: 4094701258
> file data blocks allocated: 54396052914176
> referenced 67747603632128
If btrfs-check can repair, then your fs is not seriously corrupted,
which is a good news.
>
> 6) The second try of manual mount succeeded, but it took 2 minutes.
Slow mount is related to extent tree size.
> # time mount /mnt/storage
> real 2m9.076s
> user 0m0.005s
> sys 0m2.753s
>
> 7) btrfs inspect-internal dump-super reports only the correct size:
>
> btrfs inspect-internal dump-super -fa /dev/mapper/fc_trunk-part3 |
> grep total
> total_bytes 52798547820544
> dev_item.total_bytes 52798547820544
> backup_total_bytes: 52798547820544
> backup_total_bytes: 52798547820544
> backup_total_bytes: 52798547820544
> backup_total_bytes: 52798547820544
> total_bytes 52798547820544
> dev_item.total_bytes 52798547820544
> backup_total_bytes: 52798547820544
> backup_total_bytes: 52798547820544
> backup_total_bytes: 52798547820544
> backup_total_bytes: 52798547820544
> total_bytes 52798547820544
> dev_item.total_bytes 52798547820544
> backup_total_bytes: 52798547820544
> backup_total_bytes: 52798547820544
> backup_total_bytes: 52798547820544
> backup_total_bytes: 52798547820544
>
>
> 8) I did not try a second reboot.
> btrfs-progs v4.20.1
>
> btrfs fi usage /mnt/storage
> Overall:
> Device size: 48.02TiB
> Device allocated: 37.94TiB
This explains the slow mount.
Unless using the new BG_TREE feature I purposed, the slow mount can't
really be solved.
Thanks,
Qu
> Device unallocated: 10.08TiB
> Device missing: 0.00B
> Used: 37.93TiB
> Free (estimated): 10.08TiB (min: 5.04TiB)
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 512.00MiB (used: 0.00B)
>
> Data,single: Size:37.80TiB, Used:37.79TiB
> /dev/mapper/fc_trunk-part3 37.80TiB
>
> Metadata,DUP: Size:73.00GiB, Used:71.17GiB
> /dev/mapper/fc_trunk-part3 146.00GiB
>
> System,DUP: Size:8.00MiB, Used:4.09MiB
> /dev/mapper/fc_trunk-part3 16.00MiB
>
> Unallocated:
> /dev/mapper/fc_trunk-part3 10.08TiB
>
>
>
> Hope that it will be helpful,
> Piotr J. Balwierz
>
>
>
> On 24/10/2017 20.02, Konstantin V. Gavrilenko wrote:
>> The mentioning of the device scan scode and the fact the total_bytes
>> is double made me try hashing the raid from the fstab.
>> So i booted, run the "inspect-internal dump-super" that confirmed that
>> it is in order.
>> # grep -i total_bytes hashed-inspect-internal
>>
>> total_bytes 32004083023872
>> dev_item.total_bytes 32004083023872
>> backup_total_bytes: 32004083023872
>> backup_total_bytes: 32004083023872
>> backup_total_bytes: 32004083023872
>> backup_total_bytes: 32004083023872
>> total_bytes 32004083023872
>> dev_item.total_bytes 32004083023872
>> backup_total_bytes: 32004083023872
>> backup_total_bytes: 32004083023872
>> backup_total_bytes: 32004083023872
>> backup_total_bytes: 32004083023872
>> total_bytes 32004083023872
>> dev_item.total_bytes 32004083023872
>> backup_total_bytes: 32004083023872
>> backup_total_bytes: 32004083023872
>> backup_total_bytes: 32004083023872
>> backup_total_bytes: 32004083023872
>>
>> then I unhashed the device in fstab, mounted it manually and it
>> successfully mounted.
>>
>> # time mount /mnt/arh-backup1/
>>
>> real 2m49.021s
>> user 0m0.000s
>> sys 0m1.244s
>>
>>
>>
>> With the unhashed device in the fstab, i rebooted and upon reboot I
>> run mount
>>
>> time mount /mnt/arh-backup1/
>> mount: wrong fs type, bad option, bad superblock on /dev/sda,
>> missing codepage or helper program, or other error
>>
>> In some cases useful info is found in syslog - try
>> dmesg | tail or so.
>>
>> real 1m20.499s
>> user 0m0.000s
>> sys 0m0.045s
>>
>> that failed. I further waited for couple of minutes and run the mount
>> again, and it mounted successfully.
>>
>>
>> So it seems that because of the amount of time it takes mount, nearly
>> 3 minutes, to mount the device, there is some sort of race condition,
>> and two device scans are running at the same time, or something similar.
>> I can say one thing for sure, it wasn't happening on 4.10 and I have
>> only observed such behaviour on 4.12 and 4.13
>>
>> p.s. the disk does not mount automatically upon boot, but can be
>> mounted manually later
>>
>>
>> # uptime
>> 19:54:45 up 4 min, 1 user, load average: 0.30, 0.74, 0.39
>>
>> # time mount /mnt/arh-backup1/
>>
>> real 2m52.247s
>> user 0m0.000s
>> sys 0m1.246s
>>
>>
>> Here is the dmesg extract. It seems that for some reason on 204th
>> second the system return "open ctree failed"
>> on 329 second, I started the mount manually.
>>
>> [ 204.389231] BTRFS error (device sda): open_ctree failed
>> [ 329.234613] BTRFS info (device sda): force zlib compression
>> [ 329.234618] BTRFS info (device sda): using free space tree
>> [ 329.234620] BTRFS info (device sda): has skinny extents
>>
>>
>> hope that helps and thanks for your help
>>
>>
>> Yours sincerely,
>> Konstantin V. Gavrilenko
>>
>>
>>
>> ----- Original Message -----
>> From: "Qu Wenruo" <quwenruo.btrfs@gmx.com>
>> To: "Konstantin V. Gavrilenko" <k.gavrilenko@arhont.com>
>> Cc: "Linux fs Btrfs" <linux-btrfs@vger.kernel.org>
>> Sent: Tuesday, 24 October, 2017 3:44:21 PM
>> Subject: Re: super_total_bytes 32004083023872 mismatch with fs_devices
>> total_rw_bytes 64008166047744
>>
>>
>>
>> On 2017年10月24日 19:44, Konstantin V. Gavrilenko wrote:
>>> answers inline marked with KVG:
>>>
>>> Yours sincerely,
>>> Konstantin V. Gavrilenko
>>>
>>>
>>>
>>>
>>> ----- Original Message -----
>>> From: "Qu Wenruo" <quwenruo.btrfs@gmx.com>
>>> To: "Konstantin V. Gavrilenko" <k.gavrilenko@arhont.com>, "Linux fs
>>> Btrfs" <linux-btrfs@vger.kernel.org>
>>> Sent: Tuesday, 24 October, 2017 11:37:56 AM
>>> Subject: Re: super_total_bytes 32004083023872 mismatch with
>>> fs_devices total_rw_bytes 64008166047744
>>>
>>>
>>>
>>> On 2017年10月24日 17:20, Konstantin V. Gavrilenko wrote:
>>>> Hi list,
>>>>
>>>> having installed the recent kernel version I am no longer able to
>>>> mount the btrfs partition with compression on the first attempt.
>>>> Previously on 4.10.0-37-generic everything was working fine, once I
>>>> switched to 4.13.9-041309-generic I started getting the following
>>>> error while trying to mount it with the same options
>>>> "compress-force=zlib,space_cache=v2"
>>>>
>>>> [ 204.596381] BTRFS error (device sda): open_ctree failed
>>>> [ 204.631895] BTRFS info (device sda): force zlib compression
>>>> [ 204.631901] BTRFS info (device sda): using free space tree
>>>> [ 204.631903] BTRFS info (device sda): has skinny extents
>>>> [ 204.890145] BTRFS error (device sda): super_total_bytes
>>>> 32004083023872 mismatch with fs_devices total_rw_bytes 64008166047744
>>>> [ 204.891276] BTRFS error (device sda): failed to read chunk tree: -22
>>>> [ 204.944333] BTRFS error (device sda): open_ctree failed
>>> Such problem can be easily fixed with this branch:
>>> https://github.com/adam900710/btrfs-progs/tree/check_unaligned_dev
>>>
>>> Use "btrfs rescue fix-device-size" should handle it well.
>>>
>>> But the problem is, normally the super_total_bytes should only be less
>>> than 4K smaller than total device size.
>>>
>>> Unless something else went wrong, it should not have such large
>>> difference.
>>>
>>>
>>> KVG: Thanks, will try.
>>> The drive was initially created as RAID5 with 4 x 8tb devices, and
>>> later expanded to included another 8Tb. So now it is 5x8tb RAID5.
>>>
>>>
>>>> For some reason, the super_total_bytes is exactly half of
>>>> total_rw_bytes.
>>>>
>>>>
>>>> however, if after unsuccessful first mount attempt, I mount it with
>>>> minimum number of options "space_cache=v2" the partition mounts.
>>>> Then I umount it, and mount normally, with full set of options
>>>> "compress-force=zlib,space_cache=v2" it mounts without an error.
>>>> I also observed the same error on 4.12.14-041214-generic
>>>> Any ideas why this might be happening?
>>> Would you please provide super dump by:
>>>
>>> # btrfs inspect-internal dump-super -fa /dev/sda
>>>
>>> (Although I don't think it will be very interesting since it can be
>>> mounted later)
>>>
>>>
>>> KVG: here you go
>>>
>>> # btrfs inspect-internal dump-super -fa /dev/sda
>>> superblock: bytenr=65536, device=/dev/sda
>>> ---------------------------------------------------------
>>> csum_type 0 (crc32c)
>>> csum_size 4
>>> csum 0xc4b95762 [match]
>>> bytenr 65536
>>> flags 0x1
>>> ( WRITTEN )
>>> magic _BHRfS_M [match]
>>> fsid 017b587a-3e88-4f08-8616-ec686f8f969f
>>> label arh-backup
>>> generation 145942
>>> root 17334200778752
>>> sys_array_size 129
>>> chunk_root_generation 145699
>>> root_level 1
>>> chunk_root 20971520
>>> chunk_root_level 1
>>> log_root 17334149644288
>>> log_root_transid 0
>>> log_root_level 0
>>> total_bytes 32004083023872
>> Here 32004083023872.
>>> bytes_used 19719727349760
>>> sectorsize 4096
>>> nodesize 16384
>>> leafsize (deprecated) 16384
>>> stripesize 4096
>>> root_dir 6
>>> num_devices 1
>>> compat_flags 0x0
>>> compat_ro_flags 0x3
>>> ( FREE_SPACE_TREE |
>>> FREE_SPACE_TREE_VALID )
>>> incompat_flags 0x169
>>> ( MIXED_BACKREF |
>>> COMPRESS_LZO |
>>> BIG_METADATA |
>>> EXTENDED_IREF |
>>> SKINNY_METADATA )
>>> cache_generation 7
>>> uuid_tree_generation 145942
>>> dev_item.uuid ee632cde-2139-4181-9d57-ff892768f2ef
>>> dev_item.fsid 017b587a-3e88-4f08-8616-ec686f8f969f [match]
>>> dev_item.type 0
>>> dev_item.total_bytes 32004083023872
>> And here, 32004083023872 again.
>>
>> So it matches now.
>>
>> And considering the original report says it's twice the size, I wonder
>> if it's something related to device scan.
>>
>> Like the same disk get counted twice, so it's reporting like that.
>>
>> If the problem happens again, would you please try run "btrfs device
>> scan" and remount it with the same mount option?
>>
>>
>>> dev_item.bytes_used 19832028266496
>>> dev_item.io_align 4096
>>> dev_item.io_width 4096
>>> dev_item.sector_size 4096
>>> dev_item.devid 1
>>> dev_item.dev_group 0
>>> dev_item.seek_speed 0
>>> dev_item.bandwidth 0
>>> dev_item.generation 0
>>> sys_chunk_array[2048]:
>>> item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520)
>>> length 8388608 owner 2 stripe_len 65536 type SYSTEM|DUP
>>> io_align 65536 io_width 65536 sector_size 4096
>>> num_stripes 2 sub_stripes 0
>>> stripe 0 devid 1 offset 20971520
>>> dev_uuid ee632cde-2139-4181-9d57-ff892768f2ef
>>> stripe 1 devid 1 offset 29360128
>>> dev_uuid ee632cde-2139-4181-9d57-ff892768f2ef
>>> backup_roots[4]:
>>> backup 0:
>>> backup_tree_root: 17334200778752 gen:
>>> 145940 level: 1
>>> backup_chunk_root: 20971520 gen:
>>> 145699 level: 1
>>> backup_extent_root: 17334177021952 gen:
>>> 145940 level: 3
>>> backup_fs_root: 17334133260288 gen:
>>> 145941 level: 3
>>> backup_dev_root: 32391168 gen:
>>> 145836 level: 1
>>> backup_csum_root: 17154410381312 gen:
>>> 145925 level: 3
>>> backup_total_bytes: 32004083023872
>>> backup_bytes_used: 19719727349760
>>> backup_num_devices: 1
>>>
>>> backup 1:
>>> backup_tree_root: 17334201057280 gen:
>>> 145941 level: 1
>>> backup_chunk_root: 20971520 gen:
>>> 145699 level: 1
>>> backup_extent_root: 17334177988608 gen:
>>> 145941 level: 3
>>> backup_fs_root: 17334133161984 gen:
>>> 145942 level: 3
>>> backup_dev_root: 32391168 gen:
>>> 145836 level: 1
>>> backup_csum_root: 17154410381312 gen:
>>> 145925 level: 3
>>> backup_total_bytes: 32004083023872
>>> backup_bytes_used: 19719727349760
>>> backup_num_devices: 1
>>>
>>> backup 2:
>>> backup_tree_root: 17334200778752 gen:
>>> 145942 level: 1
>>> backup_chunk_root: 20971520 gen:
>>> 145699 level: 1
>>> backup_extent_root: 17334184558592 gen:
>>> 145942 level: 3
>>> backup_fs_root: 17334135275520 gen:
>>> 145943 level: 3
>>> backup_dev_root: 32391168 gen:
>>> 145836 level: 1
>>> backup_csum_root: 17154410381312 gen:
>>> 145925 level: 3
>>> backup_total_bytes: 32004083023872
>>> backup_bytes_used: 19719727349760
>>> backup_num_devices: 1
>>>
>>> backup 3:
>>> backup_tree_root: 17334201057280 gen:
>>> 145939 level: 1
>>> backup_chunk_root: 20971520 gen:
>>> 145699 level: 1
>>> backup_extent_root: 17334177988608 gen:
>>> 145939 level: 3
>>> backup_fs_root: 17334131884032 gen:
>>> 145940 level: 3
>>> backup_dev_root: 32391168 gen:
>>> 145836 level: 1
>>> backup_csum_root: 17154410381312 gen:
>>> 145925 level: 3
>>> backup_total_bytes: 32004083023872
>>> backup_bytes_used: 19719727349760
>>> backup_num_devices: 1
>>>
>>>
>>> superblock: bytenr=67108864, device=/dev/sda
>>> ---------------------------------------------------------
>>> csum_type 0 (crc32c)
>>> csum_size 4
>>> csum 0xd1c4187f [match]
>>> bytenr 67108864
>>> flags 0x1
>>> ( WRITTEN )
>>> magic _BHRfS_M [match]
>>> fsid 017b587a-3e88-4f08-8616-ec686f8f969f
>>> label arh-backup
>>> generation 145942
>>> root 17334200778752
>>> sys_array_size 129
>>> chunk_root_generation 145699
>>> root_level 1
>>> chunk_root 20971520
>>> chunk_root_level 1
>>> log_root 0
>>> log_root_transid 0
>>> log_root_level 0
>>> total_bytes 32004083023872
>>> bytes_used 19719727349760
>>> sectorsize 4096
>>> nodesize 16384
>>> leafsize (deprecated) 16384
>>> stripesize 4096
>>> root_dir 6
>>> num_devices 1
>>> compat_flags 0x0
>>> compat_ro_flags 0x3
>>> ( FREE_SPACE_TREE |
>>> FREE_SPACE_TREE_VALID )
>>> incompat_flags 0x169
>>> ( MIXED_BACKREF |
>>> COMPRESS_LZO |
>>> BIG_METADATA |
>>> EXTENDED_IREF |
>>> SKINNY_METADATA )
>>> cache_generation 7
>>> uuid_tree_generation 145942
>>> dev_item.uuid ee632cde-2139-4181-9d57-ff892768f2ef
>>> dev_item.fsid 017b587a-3e88-4f08-8616-ec686f8f969f [match]
>>> dev_item.type 0
>>> dev_item.total_bytes 32004083023872
>>> dev_item.bytes_used 19832028266496
>>> dev_item.io_align 4096
>>> dev_item.io_width 4096
>>> dev_item.sector_size 4096
>>> dev_item.devid 1
>>> dev_item.dev_group 0
>>> dev_item.seek_speed 0
>>> dev_item.bandwidth 0
>>> dev_item.generation 0
>>> sys_chunk_array[2048]:
>>> item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520)
>>> length 8388608 owner 2 stripe_len 65536 type SYSTEM|DUP
>>> io_align 65536 io_width 65536 sector_size 4096
>>> num_stripes 2 sub_stripes 0
>>> stripe 0 devid 1 offset 20971520
>>> dev_uuid ee632cde-2139-4181-9d57-ff892768f2ef
>>> stripe 1 devid 1 offset 29360128
>>> dev_uuid ee632cde-2139-4181-9d57-ff892768f2ef
>>> backup_roots[4]:
>>> backup 0:
>>> backup_tree_root: 17334200778752 gen:
>>> 145940 level: 1
>>> backup_chunk_root: 20971520 gen:
>>> 145699 level: 1
>>> backup_extent_root: 17334177021952 gen:
>>> 145940 level: 3
>>> backup_fs_root: 17334133260288 gen:
>>> 145941 level: 3
>>> backup_dev_root: 32391168 gen:
>>> 145836 level: 1
>>> backup_csum_root: 17154410381312 gen:
>>> 145925 level: 3
>>> backup_total_bytes: 32004083023872
>>> backup_bytes_used: 19719727349760
>>> backup_num_devices: 1
>>>
>>> backup 1:
>>> backup_tree_root: 17334201057280 gen:
>>> 145941 level: 1
>>> backup_chunk_root: 20971520 gen:
>>> 145699 level: 1
>>> backup_extent_root: 17334177988608 gen:
>>> 145941 level: 3
>>> backup_fs_root: 17334133161984 gen:
>>> 145942 level: 3
>>> backup_dev_root: 32391168 gen:
>>> 145836 level: 1
>>> backup_csum_root: 17154410381312 gen:
>>> 145925 level: 3
>>> backup_total_bytes: 32004083023872
>>> backup_bytes_used: 19719727349760
>>> backup_num_devices: 1
>>>
>>> backup 2:
>>> backup_tree_root: 17334200778752 gen:
>>> 145942 level: 1
>>> backup_chunk_root: 20971520 gen:
>>> 145699 level: 1
>>> backup_extent_root: 17334184558592 gen:
>>> 145942 level: 3
>>> backup_fs_root: 17334133161984 gen:
>>> 145942 level: 3
>>> backup_dev_root: 32391168 gen:
>>> 145836 level: 1
>>> backup_csum_root: 17154410381312 gen:
>>> 145925 level: 3
>>> backup_total_bytes: 32004083023872
>>> backup_bytes_used: 19719727349760
>>> backup_num_devices: 1
>>>
>>> backup 3:
>>> backup_tree_root: 17334201057280 gen:
>>> 145939 level: 1
>>> backup_chunk_root: 20971520 gen:
>>> 145699 level: 1
>>> backup_extent_root: 17334177988608 gen:
>>> 145939 level: 3
>>> backup_fs_root: 17334131884032 gen:
>>> 145940 level: 3
>>> backup_dev_root: 32391168 gen:
>>> 145836 level: 1
>>> backup_csum_root: 17154410381312 gen:
>>> 145925 level: 3
>>> backup_total_bytes: 32004083023872
>>> backup_bytes_used: 19719727349760
>>> backup_num_devices: 1
>>>
>>>
>>> superblock: bytenr=274877906944, device=/dev/sda
>>> ---------------------------------------------------------
>>> csum_type 0 (crc32c)
>>> csum_size 4
>>> csum 0x2c434e4e [match]
>>> bytenr 274877906944
>>> flags 0x1
>>> ( WRITTEN )
>>> magic _BHRfS_M [match]
>>> fsid 017b587a-3e88-4f08-8616-ec686f8f969f
>>> label arh-backup
>>> generation 145942
>>> root 17334200778752
>>> sys_array_size 129
>>> chunk_root_generation 145699
>>> root_level 1
>>> chunk_root 20971520
>>> chunk_root_level 1
>>> log_root 0
>>> log_root_transid 0
>>> log_root_level 0
>>> total_bytes 32004083023872
>>> bytes_used 19719727349760
>>> sectorsize 4096
>>> nodesize 16384
>>> leafsize (deprecated) 16384
>>> stripesize 4096
>>> root_dir 6
>>> num_devices 1
>>> compat_flags 0x0
>>> compat_ro_flags 0x3
>>> ( FREE_SPACE_TREE |
>>> FREE_SPACE_TREE_VALID )
>>> incompat_flags 0x169
>>> ( MIXED_BACKREF |
>>> COMPRESS_LZO |
>>> BIG_METADATA |
>>> EXTENDED_IREF |
>>> SKINNY_METADATA )
>>> cache_generation 7
>>> uuid_tree_generation 145942
>>> dev_item.uuid ee632cde-2139-4181-9d57-ff892768f2ef
>>> dev_item.fsid 017b587a-3e88-4f08-8616-ec686f8f969f [match]
>>> dev_item.type 0
>>> dev_item.total_bytes 32004083023872
>>> dev_item.bytes_used 19832028266496
>>> dev_item.io_align 4096
>>> dev_item.io_width 4096
>>> dev_item.sector_size 4096
>>> dev_item.devid 1
>>> dev_item.dev_group 0
>>> dev_item.seek_speed 0
>>> dev_item.bandwidth 0
>>> dev_item.generation 0
>>> sys_chunk_array[2048]:
>>> item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520)
>>> length 8388608 owner 2 stripe_len 65536 type SYSTEM|DUP
>>> io_align 65536 io_width 65536 sector_size 4096
>>> num_stripes 2 sub_stripes 0
>>> stripe 0 devid 1 offset 20971520
>>> dev_uuid ee632cde-2139-4181-9d57-ff892768f2ef
>>> stripe 1 devid 1 offset 29360128
>>> dev_uuid ee632cde-2139-4181-9d57-ff892768f2ef
>>> backup_roots[4]:
>>> backup 0:
>>> backup_tree_root: 17334200778752 gen:
>>> 145940 level: 1
>>> backup_chunk_root: 20971520 gen:
>>> 145699 level: 1
>>> backup_extent_root: 17334177021952 gen:
>>> 145940 level: 3
>>> backup_fs_root: 17334133260288 gen:
>>> 145941 level: 3
>>> backup_dev_root: 32391168 gen:
>>> 145836 level: 1
>>> backup_csum_root: 17154410381312 gen:
>>> 145925 level: 3
>>> backup_total_bytes: 32004083023872
>>> backup_bytes_used: 19719727349760
>>> backup_num_devices: 1
>>>
>>> backup 1:
>>> backup_tree_root: 17334201057280 gen:
>>> 145941 level: 1
>>> backup_chunk_root: 20971520 gen:
>>> 145699 level: 1
>>> backup_extent_root: 17334177988608 gen:
>>> 145941 level: 3
>>> backup_fs_root: 17334133161984 gen:
>>> 145942 level: 3
>>> backup_dev_root: 32391168 gen:
>>> 145836 level: 1
>>> backup_csum_root: 17154410381312 gen:
>>> 145925 level: 3
>>> backup_total_bytes: 32004083023872
>>> backup_bytes_used: 19719727349760
>>> backup_num_devices: 1
>>>
>>> backup 2:
>>> backup_tree_root: 17334200778752 gen:
>>> 145942 level: 1
>>> backup_chunk_root: 20971520 gen:
>>> 145699 level: 1
>>> backup_extent_root: 17334184558592 gen:
>>> 145942 level: 3
>>> backup_fs_root: 17334133161984 gen:
>>> 145942 level: 3
>>> backup_dev_root: 32391168 gen:
>>> 145836 level: 1
>>> backup_csum_root: 17154410381312 gen:
>>> 145925 level: 3
>>> backup_total_bytes: 32004083023872
>>> backup_bytes_used: 19719727349760
>>> backup_num_devices: 1
>>>
>>> backup 3:
>>> backup_tree_root: 17334201057280 gen:
>>> 145939 level: 1
>>> backup_chunk_root: 20971520 gen:
>>> 145699 level: 1
>>> backup_extent_root: 17334177988608 gen:
>>> 145939 level: 3
>>> backup_fs_root: 17334131884032 gen:
>>> 145940 level: 3
>>> backup_dev_root: 32391168 gen:
>>> 145836 level: 1
>>> backup_csum_root: 17154410381312 gen:
>>> 145925 level: 3
>>> backup_total_bytes: 32004083023872
>>> backup_bytes_used: 19719727349760
>>> backup_num_devices: 1
>>>
>>>
>>>
>>>
>>>
>>> And device tree dump by:
>>> # btrfs inspect-internal dump-tree -t dev /dev/sda
>>>
>>> KVG: Here you go again
>>>
>>> # btrfs inspect-internal dump-tree -t dev /dev/sda >
>>> /tmp/inspect-internal-dump-tree
>>> parent transid verify failed on 17334203744256 wanted 145954 found
>>> 145956
>>> parent transid verify failed on 17334203744256 wanted 145954 found
>>> 145956
>>> parent transid verify failed on 17334203744256 wanted 145954 found
>>> 145956
>>> parent transid verify failed on 17334203744256 wanted 145954 found
>>> 145956
>>> Ignoring transid failure
>>>
>>> # wc -l <
>>> inspect-internal-dump-tree
>>> 74760
>> That's beyond my expectation, but since the super block matches, I
>> wonder it's related to device scan implemented wrongly or has some race.
>>
>> And in that case, dump-tree output is not that important.
>>
>> BTW, are you running dump-tree with device mounted? If so the transid
>> verification failure may not be a big problem.
>>
>>>
>>> # head -n 150 <
>>> inspect-internal-dump-tree
>>> [13:28:00]
>>> btrfs-progs v4.13.3
>>> device tree key (DEV_TREE ROOT_ITEM 0)
>>> node 32391168 level 1 items 88 free 405 generation 145836 owner 4
>>> fs uuid 017b587a-3e88-4f08-8616-ec686f8f969f
>>> chunk uuid 5820b5e3-9610-4cf3-a37a-b638a9963084
>>> key (0 PERSISTENT_ITEM 1) block 32407552 (1978) gen 145836
>>> key (1 DEV_EXTENT 234113466368) block 231767769088
>>> (14145982) gen 274
>>> key (1 DEV_EXTENT 473557893120) block 484797186048
>>> (29589672) gen 378
>>> key (1 DEV_EXTENT 710854836224) block 861293805568
>>> (52569202) gen 482
>>> key (1 DEV_EXTENT 949225521152) block 1042388156416
>>> (63622324) gen 587
>>> key (1 DEV_EXTENT 1187596206080) block 1221328453632
>>> (74543973) gen 691
>>> key (1 DEV_EXTENT 1424893149184) block 1583808970752
>>> (96668028) gen 764
>>> key (1 DEV_EXTENT 1662190092288) block 1603743318016
>>> (97884724) gen 843
>>> key (1 DEV_EXTENT 1900560777216) block 2231483187200
>>> (136198925) gen 1017
>>> key (1 DEV_EXTENT 2138931462144) block 2241853407232
>>> (136831873) gen 1080
>>> key (1 DEV_EXTENT 2371933437952) block 2268266070016
>>> (138443974) gen 1138
>>> key (1 DEV_EXTENT 2610304122880) block 2311626964992
>>> (141090513) gen 1177
>>> key (1 DEV_EXTENT 2848674807808) block 2590371545088
>>> (158103732) gen 1263
>>> key (1 DEV_EXTENT 3088119234560) block 2837798567936
>>> (173205479) gen 1314
>>> key (1 DEV_EXTENT 3326489919488) block 3114321903616
>>> (190083124) gen 1372
>>> key (1 DEV_EXTENT 3564860604416) block 3591785086976
>>> (219225164) gen 1448
>>> key (1 DEV_EXTENT 3802157547520) block 3718121062400
>>> (226936100) gen 1524
>>> key (1 DEV_EXTENT 4040528232448) block 21927333150720
>>> (1338338205) gen 136290
>>> key (1 DEV_EXTENT 4278898917376) block 4133925863424
>>> (252314811) gen 1724
>>> key (1 DEV_EXTENT 4516195860480) block 22441652109312
>>> (1369729743) gen 133266
>>> key (1 DEV_EXTENT 4755640287232) block 14608468082688
>>> (891630132) gen 133130
>>> key (1 DEV_EXTENT 4992937230336) block 5180103491584
>>> (316168426) gen 2146
>>> key (1 DEV_EXTENT 5231307915264) block 20077315735552
>>> (1225422103) gen 131467
>>> key (1 DEV_EXTENT 5468604858368) block 5866555588608
>>> (358066137) gen 3067
>>> key (1 DEV_EXTENT 5705901801472) block 6367858622464
>>> (388663246) gen 3174
>>> key (1 DEV_EXTENT 5826160885760) block 6367858638848
>>> (388663247) gen 3174
>>> key (1 DEV_EXTENT 6064531570688) block 17709247496192
>>> (1080886688) gen 63205
>>> key (1 DEV_EXTENT 6303975997440) block 17459226411008
>>> (1065626612) gen 145661
>>> key (1 DEV_EXTENT 6500470751232) block 17459593674752
>>> (1065649028) gen 145669
>>> key (1 DEV_EXTENT 6737767694336) block 14608384458752
>>> (891625028) gen 132007
>>> key (1 DEV_EXTENT 6976138379264) block 15230725767168
>>> (929609727) gen 132017
>>> key (1 DEV_EXTENT 7136125911040) block 13657160859648
>>> (833566947) gen 137915
>>> key (1 DEV_EXTENT 7255311253504) block 13657457180672
>>> (833585033) gen 137917
>>> key (1 DEV_EXTENT 7492608196608) block 22441489580032
>>> (1369719823) gen 136294
>>> key (1 DEV_EXTENT 7730978881536) block 13854137253888
>>> (845589432) gen 138616
>>> key (1 DEV_EXTENT 7969349566464) block 7788285968384
>>> (475359251) gen 140545
>>> key (1 DEV_EXTENT 8207720251392) block 14113310818304
>>> (861408131) gen 127092
>>> key (1 DEV_EXTENT 8446090936320) block 8679398998016
>>> (529748474) gen 17985
>>> key (1 DEV_EXTENT 8684461621248) block 8916314619904
>>> (544208656) gen 18574
>>> key (1 DEV_EXTENT 8922832306176) block 29434636730368
>>> (1796547652) gen 144688
>>> key (1 DEV_EXTENT 9157981765632) block 6453626322944
>>> (393898091) gen 145220
>>> key (1 DEV_EXTENT 9361992712192) block 19459087679488
>>> (1187688457) gen 132049
>>> key (1 DEV_EXTENT 9600363397120) block 12893675651072
>>> (786967508) gen 116367
>>> key (1 DEV_EXTENT 9720622481408) block 22441517318144
>>> (1369721516) gen 95603
>>> key (1 DEV_EXTENT 9847860887552) block 22441466626048
>>> (1369718422) gen 117878
>>> key (1 DEV_EXTENT 10082473476096) block 22493359931392
>>> (1372885738) gen 117882
>>> key (1 DEV_EXTENT 10232797331456) block 13983139643392
>>> (853463113) gen 137923
>>> key (1 DEV_EXTENT 10471168016384) block 6959763996672
>>> (424790283) gen 105377
>>> key (1 DEV_EXTENT 10591427100672) block 10067009994752
>>> (614441528) gen 27773
>>> key (1 DEV_EXTENT 10830871527424) block 10072383062016
>>> (614769474) gen 28150
>>> key (1 DEV_EXTENT 11070315954176) block 10072413356032
>>> (614771323) gen 92253
>>> key (1 DEV_EXTENT 11309760380928) block 22493328687104
>>> (1372883831) gen 133268
>>> key (1 DEV_EXTENT 11548131065856) block 1042474778624
>>> (63627611) gen 133909
>>> key (1 DEV_EXTENT 11785428008960) block 15016548761600
>>> (916537400) gen 121951
>>> key (1 DEV_EXTENT 12024872435712) block 12656435298304
>>> (772487506) gen 32985
>>> key (1 DEV_EXTENT 12263243120640) block 12893629710336
>>> (786964704) gen 33745
>>> key (1 DEV_EXTENT 12500540063744) block 16644219928576
>>> (1015882564) gen 112366
>>> key (1 DEV_EXTENT 12738910748672) block 14703563259904
>>> (897434281) gen 145699
>>> key (1 DEV_EXTENT 12976207691776) block 14113684733952
>>> (861430953) gen 137926
>>> key (1 DEV_EXTENT 13214578376704) block 15230715084800
>>> (929609075) gen 116469
>>> key (1 DEV_EXTENT 13452949061632) block 12787035504640
>>> (780458710) gen 119919
>>> key (1 DEV_EXTENT 13690246004736) block 10067511001088
>>> (614472107) gen 119958
>>> key (1 DEV_EXTENT 13928616689664) block 14113506328576
>>> (861420064) gen 106290
>>> key (1 DEV_EXTENT 14166987374592) block 861654532096
>>> (52591219) gen 140596
>>> key (1 DEV_EXTENT 14404284317696) block 6959531130880
>>> (424776070) gen 103496
>>> key (1 DEV_EXTENT 14643728744448) block 484408541184
>>> (29565951) gen 103596
>>> key (1 DEV_EXTENT 14882099429376) block 205519962112
>>> (12543943) gen 137977
>>> key (1 DEV_EXTENT 15118322630656) block 13289788358656
>>> (811144309) gen 145698
>>> key (1 DEV_EXTENT 15356693315584) block 13289789358080
>>> (811144370) gen 145698
>>> key (1 DEV_EXTENT 15596137742336) block 15736233984000
>>> (960463500) gen 117936
>>> key (1 DEV_EXTENT 15833434685440) block 1221260197888
>>> (74539807) gen 138299
>>> key (1 DEV_EXTENT 16071805370368) block 6368249397248
>>> (388687097) gen 124384
>>> key (1 DEV_EXTENT 16310176055296) block 12787123699712
>>> (780464093) gen 138137
>>> key (1 DEV_EXTENT 16548546740224) block 9420764561408
>>> (574997837) gen 124415
>>> key (1 DEV_EXTENT 16787991166976) block 22441589932032
>>> (1369725948) gen 124583
>>> key (1 DEV_EXTENT 17025288110080) block 15016789147648
>>> (916552072) gen 144864
>>> key (1 DEV_EXTENT 17263658795008) block 17459171082240
>>> (1065623235) gen 144951
>>> key (1 DEV_EXTENT 17502029479936) block 673143095296
>>> (41085394) gen 143317
>>> key (1 DEV_EXTENT 17740400164864) block 19010757427200
>>> (1160324550) gen 141153
>>> key (1 DEV_EXTENT 17979844591616) block 10067131596800
>>> (614448950) gen 141191
>>> key (1 DEV_EXTENT 18217141534720) block 205212958720
>>> (12525205) gen 124623
>>> key (1 DEV_EXTENT 18455512219648) block 205287997440
>>> (12529785) gen 124626
>>> key (1 DEV_EXTENT 18692809162752) block 16703827378176
>>> (1019520714) gen 124744
>>> key (1 DEV_EXTENT 18931179847680) block 7268874698752
>>> (443656903) gen 124899
>>> key (1 DEV_EXTENT 19169550532608) block 14291177324544
>>> (872264241) gen 141256
>>> key (1 DEV_EXTENT 19407921217536) block 29134446510080
>>> (1778225495) gen 142791
>>> key (1 DEV_EXTENT 19646291902464) block 6453644148736
>>> (393899179) gen 143236
>>> key (1 DEV_EXTENT 19883588845568) block 15998310219776
>>> (976459364) gen 143270
>>> leaf 32407552 items 223 free space 12 generation 145836 owner 4
>>> leaf 32407552 flags 0x1(WRITTEN) backref revision 1
>>> fs uuid 017b587a-3e88-4f08-8616-ec686f8f969f
>>> chunk uuid 5820b5e3-9610-4cf3-a37a-b638a9963084
>>> item 0 key (0 PERSISTENT_ITEM 1) itemoff 16243 itemsize 40
>>> persistent item objectid 0 offset 1
>>> device stats
>>> write_errs 0 read_errs 0 flush_errs 0
>>> corruption_errs 0 generation 0
>>> item 1 key (1 DEV_EXTENT 20971520) itemoff 16195 itemsize 48
>>> dev extent chunk_tree 3
>>> chunk_objectid 256 chunk_offset 20971520 length 8388608
>>> chunk_tree_uuid 5820b5e3-9610-4cf3-a37a-b638a9963084
>>> item 2 key (1 DEV_EXTENT 29360128) itemoff 16147 itemsize 48
>>> dev extent chunk_tree 3
>>> chunk_objectid 256 chunk_offset 20971520 length 8388608
>>> chunk_tree_uuid 5820b5e3-9610-4cf3-a37a-b638a9963084
>>> item 3 key (1 DEV_EXTENT 37748736) itemoff 16099 itemsize 48
>>> dev extent chunk_tree 3
>>> chunk_objectid 256 chunk_offset 29360128 length
>>> 1073741824
>>> chunk_tree_uuid 5820b5e3-9610-4cf3-a37a-b638a9963084
>>> item 4 key (1 DEV_EXTENT 1111490560) itemoff 16051 itemsize 48
>>> dev extent chunk_tree 3
>>> chunk_objectid 256 chunk_offset 29360128 length
>>> 1073741824
>>> chunk_tree_uuid 5820b5e3-9610-4cf3-a37a-b638a9963084
>>> item 5 key (1 DEV_EXTENT 2185232384) itemoff 16003 itemsize 48
>>> dev extent chunk_tree 3
>>> chunk_objectid 256 chunk_offset 16135487488 length
>>> 1073741824
>>> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>>> item 6 key (1 DEV_EXTENT 3258974208) itemoff 15955 itemsize 48
>>> dev extent chunk_tree 3
>>> chunk_objectid 256 chunk_offset 17209229312 length
>>> 1073741824
>>> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>>> item 7 key (1 DEV_EXTENT 4332716032) itemoff 15907 itemsize 48
>>> dev extent chunk_tree 3
>>> chunk_objectid 256 chunk_offset 18282971136 length
>>> 1073741824
>>> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>>> item 8 key (1 DEV_EXTENT 5406457856) itemoff 15859 itemsize 48
>>> dev extent chunk_tree 3
>>> chunk_objectid 256 chunk_offset 19356712960 length
>>> 1073741824
>>> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>>> item 9 key (1 DEV_EXTENT 6480199680) itemoff 15811 itemsize 48
>>> dev extent chunk_tree 3
>>> chunk_objectid 256 chunk_offset 20430454784 length
>>> 1073741824
>>> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>>> item 10 key (1 DEV_EXTENT 7553941504) itemoff 15763 itemsize 48
>>> dev extent chunk_tree 3
>>> chunk_objectid 256 chunk_offset 21504196608 length
>>> 1073741824
>>> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>>> item 11 key (1 DEV_EXTENT 8627683328) itemoff 15715 itemsize 48
>>> dev extent chunk_tree 3
>>> chunk_objectid 256 chunk_offset 22577938432 length
>>> 1073741824
>>> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>>> item 12 key (1 DEV_EXTENT 9701425152) itemoff 15667 itemsize 48
>>> dev extent chunk_tree 3
>>> chunk_objectid 256 chunk_offset 23651680256 length
>>> 1073741824
>>> chunk_tree_uuid 00000000-0000-0000-0000-000000000000
>>> item 13 key (1 DEV_EXTENT 10775166976) itemoff 15619
>>> itemsize 48
>>>
>>> KVG: and the complete file is attached to this message
>>>
>>>
>>>
>>>
>>> Normally it should not be mountable after v4.6 kernel if
>>> super_total_bytes mismatch, but I'm more interested in how it mounted
>>> successfully.
>>>
>>> KVG: I've no idea why it mounts successfully when the compressions
>>> flags are omitted. No indication in the dmesg.
>> I think it's device scan code, maybe I can check it later.
>>
>> Considering your superblock is good, you won't encounter such problem
>> any longer.
>>
>> And if it really happens again, please try "btrfs device scan" first.
>>
>> Thanks,
>> Qu
>>
>>>
>>> And BTW, are you using x86_64 kernel or x86 kernel?
>>> I don't think it's related in your case, but some reports about 32bit
>>> kernel has strange bugs are in mail list, so just in case.
>>>
>>> KVG: both kernels are 64bit. with 4.10.37 it mounts without a
>>> problem, with newer kernels the problem exis
>>>
>>> [ 0.000000] Linux version 4.10.0-37-generic
>>> (buildd@lgw01-amd64-037) (gcc version 5.4.0 20160609 (Ubuntu
>>> 5.4.0-6ubuntu1~16.04.4) ) #41~16.04.1-Ubuntu SMP Fri Oct 6 22:42:59
>>> UTC 2017 (Ubuntu 4.10.0-37.41~16.04.1-generic 4.10.17)
>>>
>>> [ 0.000000] Linux version 4.13.9-041309-generic (kernel@tangerine)
>>> (gcc version 7.2.0 (Ubuntu 7.2.0-8ubuntu3)) #201710211231 SMP Sat Oct
>>> 21 16:32:44 UTC 2017
>>>
>>>
>>>
>>> Thanks,
>>> Qu
>>>
>>>
>>>>
>>>>
>>>> System information
>>>>
>>>> distribution: Ubuntu 16.04
>>>> btrfs-progs v4.8.1 later upgraded to v4.13.3
>>>>
>>>> # btrfs fi usage /mnt/backup
>>>> Overall:
>>>> Device size: 29.11TiB
>>>> Device allocated: 18.04TiB
>>>> Device unallocated: 11.07TiB
>>>> Device missing: 0.00B
>>>> Used: 17.99TiB
>>>> Free (estimated): 11.12TiB (min: 5.58TiB)
>>>> Data ratio: 1.00
>>>> Metadata ratio: 2.00
>>>> Global reserve: 512.00MiB (used: 0.00B)
>>>>
>>>> Data,single: Size:17.93TiB, Used:17.88TiB
>>>> /dev/sda 17.93TiB
>>>>
>>>> Metadata,DUP: Size:53.50GiB, Used:51.78GiB
>>>> /dev/sda 107.00GiB
>>>>
>>>> System,DUP: Size:8.00MiB, Used:2.30MiB
>>>> /dev/sda 16.00MiB
>>>>
>>>> Unallocated:
>>>> /dev/sda 11.07TiB
>>>>
>>>>
>>>>
>>>>
>>>> Yours sincerely,
>>>> Konstantin V. Gavrilenko
>>>>
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe
>>>> linux-btrfs" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>
>>
>>
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
prev parent reply other threads:[~2019-03-20 0:59 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <7560696.184.1508834006162.JavaMail.gkos@dynomob>
2017-10-24 9:20 ` super_total_bytes 32004083023872 mismatch with fs_devices total_rw_bytes 64008166047744 Konstantin V. Gavrilenko
2017-10-24 9:37 ` Qu Wenruo
[not found] ` <14075786.243.1508845404640.JavaMail.gkos@dynomob>
2017-10-24 13:44 ` Qu Wenruo
2017-10-24 19:02 ` Konstantin V. Gavrilenko
2019-03-19 22:36 ` Piotr Balwierz
2019-03-20 0:58 ` Qu Wenruo [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=9039b84b-9dc6-f012-9e4c-5e8d19cbcad8@gmx.com \
--to=quwenruo.btrfs@gmx.com \
--cc=k.gavrilenko@arhont.com \
--cc=linux-btrfs@vger.kernel.org \
--cc=nikt@sucha.eu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).