* rsync stuck in "D" state short after starting to copy to a newly created filesystem
@ 2024-06-28 11:55 Martin Steigerwald
2024-06-28 14:45 ` Kent Overstreet
2024-06-29 23:09 ` Kent Overstreet
0 siblings, 2 replies; 4+ messages in thread
From: Martin Steigerwald @ 2024-06-28 11:55 UTC (permalink / raw)
To: linux-bcachefs
Hi!
I am ending a migration from a ThinkPad T14 AMD Gen 1 to a ThinkPad T14
AMD Gen 5.
Last filesystem is a BCacheFS with some larger files that I use for testing
BCacheFS. rsync was directly pulling from the older laptop over 1 GBit
link through my local router. All other filesystems are BTRFS and there
have not been an issue with migrating about 1,8 TiB of data to three
BTRFS filesystems via rsync.
Standard Debian Unstable Kernel as of today (on Devuan):
Linux version 6.9.7-amd64 (debian-kernel@lists.debian.org)
(x86_64-linux-gnu-gcc-13 (Debian 13.3.0-1) 13.3.0,
GNU ld (GNU Binutils for Debian) 2.42.50.20240625)
#1 SMP PREEMPT_DYNAMIC Debian 6.9.7-1 (2024-06-27)
% bcachefs version
1.9.1
SSD is 4 TB Samsung 990 Pro. BCacheFS is on LUKS encrypted LVM as the
BTRFS filesystems as well.
I created BCacheFS as follows (this is from a subsequent mkfs.bcachefs.
I do not have initial output anymore as I already overwrote it with the
new successful attempt in my documentation, but other than UUIDs nothing
should have changed I bet, the parameters were identical - see below for
the successful attempt):
% mkfs.bcachefs --data_checksum xxhash --metadata_checksum xxhash
--compression=lz4 /dev/nvme1/daten2
[… identifiers deleted …]
Device index: 0
Label:
Version: 1.7: mi_btree_bitmap
Version upgrade complete: 0.0: (unknown version)
Oldest version on disk: 1.7: mi_btree_bitmap
Created: […]
Sequence number: 0
Time of last write: Thu Jan 1 01:00:00 1970
Superblock size: 976 B/1.00 MiB
Clean: 0
Devices: 1
Sections: members_v1,members_v2
Features:
Compat features:
Options:
block_size: 512 B
btree_node_size: 256 KiB
errors: continue [ro] panic
metadata_replicas: 1
data_replicas: 1
metadata_replicas_required: 1
data_replicas_required: 1
encoded_extent_max: 64.0 KiB
metadata_checksum: none crc32c crc64 [xxhash]
data_checksum: none crc32c crc64 [xxhash]
compression: lz4
background_compression: none
str_hash: crc32c crc64 [siphash]
metadata_target: none
foreground_target: none
background_target: none
promote_target: none
erasure_code: 0
inodes_32bit: 1
shard_inode_numbers: 1
inodes_use_key_cache: 1
gc_reserve_percent: 8
gc_reserve_bytes: 0 B
root_reserve_percent: 0
wide_macs: 0
acl: 1
usrquota: 0
grpquota: 0
prjquota: 0
journal_flush_delay: 1000
journal_flush_disabled: 0
journal_reclaim_delay: 100
journal_transaction_names: 1
version_upgrade: [compatible] incompatible none
nocow: 0
members_v2 (size 160):
Device: 0
Label: (none)
UUID: […]
Size: 300 GiB
read errors: 0
write errors: 0
checksum errors: 0
seqread iops: 0
seqwrite iops: 0
randread iops: 0
randwrite iops: 0
Bucket size: 256 KiB
First bucket: 0
Buckets: 1228800
Last mount: (never)
Last superblock write: 0
State: rw
Data allowed: journal,btree,user
Has data: (none)
Btree allocated bitmap blocksize: 1.00 B
Btree allocated bitmap: 0000000000000000000000000000000000000000000000000000000000000000
Durability: 1
Discard: 0
Freespace initialized: 0
Directly after creating it I mounted it from /etc/fstab:
/dev/nvme1/daten2 /daten2 bcachefs lazytime 0 0
Soon after the copying process started, rsync got stuck in "D" state.
It was within the first 500 MiB or so. Nothing in kernel log. I waited
a bit and then stopped rsync. One rsync process remained in "D" state
and thus did not go away. I tried another time and one the rsync
processes was immediately in "D" state.
Thus I rebooted. Runit hung during reboot. Likely due to processes in
D state. I eventually switched up the laptop by pressing the power
button long enough.
I did an fsck.bcachefs and got:
% fsck.bcachefs /dev/nvme1/daten2
fsck binary is version 1.9: disk_accounting_v2 but filesystem is 1.7: mi_btree_bitmap and kernel is 1.7: mi_btree_bitmap, using kernel fsck
bcachefs (dm-5): mounting version 1.7: mi_btree_bitmap opts=ro,metadata_checksum=xxhash,data_checksum=xxhash,compression=lz4,degraded,fsck,fix_errors=ask,read_only
bcachefs (dm-5): recovering from clean shutdown, journal seq 45
bcachefs (dm-5): journal read done, replaying entries 45-45
bcachefs (dm-5): alloc_read... done
bcachefs (dm-5): stripes_read... done
bcachefs (dm-5): snapshots_read... done
bcachefs (dm-5): check_allocations...key version number higher than recorded: 73014444594 > 0: fix? (y,n, or Y,N for all errors of this type) y
key version number higher than recorded: 81604378807 > 73014444594: fix? (y,n, or Y,N for all errors of this type) y
dev 0 has wrong free buckets: got 0, should be 1220580: fix? (y,n, or Y,N for all errors of this type) y
dev 0 has wrong sb buckets: got 0, should be 13: fix? (y,n, or Y,N for all errors of this type) y
dev 0 has wrong sb sectors: got 0, should be 6152: fix? (y,n, or Y,N for all errors of this type) y
dev 0 has wrong sb fragmented: got 0, should be 504: fix? (y,n, or Y,N for all errors of this type) y
dev 0 has wrong journal buckets: got 0, should be 8192: fix? (y,n, or Y,N for all errors of this type) y
dev 0 has wrong journal sectors: got 0, should be 4194304: fix? (y,n, or Y,N for all errors of this type) y
dev 0 has wrong btree buckets: got 0, should be 15: fix? (y,n, or Y,N for all errors of this type) y
dev 0 has wrong btree sectors: got 0, should be 7680: fix? (y,n, or Y,N for all errors of this type) y
fs has wrong hidden: got 0, should be 4200960: fix? (y,n, or Y,N for all errors of this type) y
fs has wrong btree: got 0, should be 7680: fix? (y,n, or Y,N for all errors of this type) y
fs has wrong nr_inodes: got 20, should be 22: fix? (y,n, or Y,N for all errors of this type) y
fs has wrong btree: 1/1 [0]: got 0, should be 7680: fix? (y,n, or Y,N for all errors of this type) y
done
bcachefs (dm-5): going read-write
bcachefs (dm-5): journal_replay... done
bcachefs (dm-5): check_alloc_info...y done
bcachefs (dm-5): check_lrus... done
bcachefs (dm-5): check_btree_backpointers... done
bcachefs (dm-5): check_backpointers_to_extents... done
bcachefs (dm-5): check_extents_to_backpointers... done
bcachefs (dm-5): check_alloc_to_lru_refs... done
bcachefs (dm-5): check_snapshot_trees... done
bcachefs (dm-5): check_snapshots... done
bcachefs (dm-5): check_subvols... done
bcachefs (dm-5): check_subvol_children... done
bcachefs (dm-5): delete_dead_snapshots... done
bcachefs (dm-5): check_inodes... done
bcachefs (dm-5): check_extents... done
bcachefs (dm-5): check_indirect_extents... done
bcachefs (dm-5): check_dirents... done
bcachefs (dm-5): check_xattrs... done
bcachefs (dm-5): check_root... done
bcachefs (dm-5): check_subvolume_structure... done
bcachefs (dm-5): check_directory_structure... done
bcachefs (dm-5): check_nlinks... done
bcachefs (dm-5): resume_logged_ops... done
bcachefs (dm-5): delete_dead_inodes... done
bcachefs (dm-5): shutdown complete, journal seq 47
dm-5: errors fixed
For a regular unclean shutdown I would not have expected any filesystem
errors. A subsequent call to "fsck.bcachefs" revealed no further errors.
I mounted the filesystem again and tried another time with rsync and
it did not seem to get stuck as before. However I felt uncomfortable
with continuing with a filesystem that has had errors already.
Especially at BCacheFS is still marked experimental.
Also I thought maybe it did not like being mounted without a reboot.
Does not really make much sense to me, but I thought whatever due to
lack of a better idea let's try it.
I recreated the filesystem. Then I rebooted.
Then I started the rsync again.
So far it still runs at maximum speed of the GBit link of around
110 MiB/s.
Let's see whether it completes this time.
It did. Nothing in kernel log. fsck.bcachefs is happy, too.
Wrote about 188 GiB of data without any apparent issues.
I am not sure what to make of this.
The only differences during the second attempt were:
1. I rebooted after mkfs.bcachefs.
2. I did not have a hibernation (suspend to disk) or standby (suspend
to RAM) cycle after the reboot. I was testing those before, but unlike
with the Gen 1 where both hibernation and standby are broken, I found
no issues with hibernation and standby on the new Gen 5.
That were the only differences. However I doubt one of these has to do
with it. Maybe just a strange race condition?
I am not feeling like retrying the whole process at the moment, being
happy with a completed migration to the new laptop.
Not sure whether this report is of any use. But maybe the fsck output
gives an idea of what might have been the cause. If not, feel free to
disregard. I do not have the old filesystem anymore, so in case further
information is missing it will be lost forever.
Best,
--
Martin
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: rsync stuck in "D" state short after starting to copy to a newly created filesystem
2024-06-28 11:55 rsync stuck in "D" state short after starting to copy to a newly created filesystem Martin Steigerwald
@ 2024-06-28 14:45 ` Kent Overstreet
2024-06-29 23:09 ` Kent Overstreet
1 sibling, 0 replies; 4+ messages in thread
From: Kent Overstreet @ 2024-06-28 14:45 UTC (permalink / raw)
To: Martin Steigerwald; +Cc: linux-bcachefs
On Fri, Jun 28, 2024 at 01:55:08PM +0200, Martin Steigerwald wrote:
> Hi!
>
> I am ending a migration from a ThinkPad T14 AMD Gen 1 to a ThinkPad T14
> AMD Gen 5.
>
> Last filesystem is a BCacheFS with some larger files that I use for testing
> BCacheFS. rsync was directly pulling from the older laptop over 1 GBit
> link through my local router. All other filesystems are BTRFS and there
> have not been an issue with migrating about 1,8 TiB of data to three
> BTRFS filesystems via rsync.
>
> Standard Debian Unstable Kernel as of today (on Devuan):
>
> Linux version 6.9.7-amd64 (debian-kernel@lists.debian.org)
> (x86_64-linux-gnu-gcc-13 (Debian 13.3.0-1) 13.3.0,
> GNU ld (GNU Binutils for Debian) 2.42.50.20240625)
> #1 SMP PREEMPT_DYNAMIC Debian 6.9.7-1 (2024-06-27)
>
> % bcachefs version
> 1.9.1
>
> SSD is 4 TB Samsung 990 Pro. BCacheFS is on LUKS encrypted LVM as the
> BTRFS filesystems as well.
>
> I created BCacheFS as follows (this is from a subsequent mkfs.bcachefs.
> I do not have initial output anymore as I already overwrote it with the
> new successful attempt in my documentation, but other than UUIDs nothing
> should have changed I bet, the parameters were identical - see below for
> the successful attempt):
>
> % mkfs.bcachefs --data_checksum xxhash --metadata_checksum xxhash
> --compression=lz4 /dev/nvme1/daten2
> [… identifiers deleted …]
> Device index: 0
> Label:
> Version: 1.7: mi_btree_bitmap
> Version upgrade complete: 0.0: (unknown version)
> Oldest version on disk: 1.7: mi_btree_bitmap
> Created: […]
> Sequence number: 0
> Time of last write: Thu Jan 1 01:00:00 1970
> Superblock size: 976 B/1.00 MiB
> Clean: 0
> Devices: 1
> Sections: members_v1,members_v2
> Features:
> Compat features:
>
> Options:
> block_size: 512 B
> btree_node_size: 256 KiB
> errors: continue [ro] panic
> metadata_replicas: 1
> data_replicas: 1
> metadata_replicas_required: 1
> data_replicas_required: 1
> encoded_extent_max: 64.0 KiB
> metadata_checksum: none crc32c crc64 [xxhash]
> data_checksum: none crc32c crc64 [xxhash]
> compression: lz4
> background_compression: none
> str_hash: crc32c crc64 [siphash]
> metadata_target: none
> foreground_target: none
> background_target: none
> promote_target: none
> erasure_code: 0
> inodes_32bit: 1
> shard_inode_numbers: 1
> inodes_use_key_cache: 1
> gc_reserve_percent: 8
> gc_reserve_bytes: 0 B
> root_reserve_percent: 0
> wide_macs: 0
> acl: 1
> usrquota: 0
> grpquota: 0
> prjquota: 0
> journal_flush_delay: 1000
> journal_flush_disabled: 0
> journal_reclaim_delay: 100
> journal_transaction_names: 1
> version_upgrade: [compatible] incompatible none
> nocow: 0
>
> members_v2 (size 160):
> Device: 0
> Label: (none)
> UUID: […]
> Size: 300 GiB
> read errors: 0
> write errors: 0
> checksum errors: 0
> seqread iops: 0
> seqwrite iops: 0
> randread iops: 0
> randwrite iops: 0
> Bucket size: 256 KiB
> First bucket: 0
> Buckets: 1228800
> Last mount: (never)
> Last superblock write: 0
> State: rw
> Data allowed: journal,btree,user
> Has data: (none)
> Btree allocated bitmap blocksize: 1.00 B
> Btree allocated bitmap: 0000000000000000000000000000000000000000000000000000000000000000
> Durability: 1
> Discard: 0
> Freespace initialized: 0
>
>
> Directly after creating it I mounted it from /etc/fstab:
>
> /dev/nvme1/daten2 /daten2 bcachefs lazytime 0 0
>
> Soon after the copying process started, rsync got stuck in "D" state.
> It was within the first 500 MiB or so. Nothing in kernel log. I waited
> a bit and then stopped rsync. One rsync process remained in "D" state
> and thus did not go away. I tried another time and one the rsync
> processes was immediately in "D" state.
>
> Thus I rebooted. Runit hung during reboot. Likely due to processes in
> D state. I eventually switched up the laptop by pressing the power
> button long enough.
>
> I did an fsck.bcachefs and got:
>
> % fsck.bcachefs /dev/nvme1/daten2
> fsck binary is version 1.9: disk_accounting_v2 but filesystem is 1.7: mi_btree_bitmap and kernel is 1.7: mi_btree_bitmap, using kernel fsck
> bcachefs (dm-5): mounting version 1.7: mi_btree_bitmap opts=ro,metadata_checksum=xxhash,data_checksum=xxhash,compression=lz4,degraded,fsck,fix_errors=ask,read_only
> bcachefs (dm-5): recovering from clean shutdown, journal seq 45
> bcachefs (dm-5): journal read done, replaying entries 45-45
> bcachefs (dm-5): alloc_read... done
> bcachefs (dm-5): stripes_read... done
> bcachefs (dm-5): snapshots_read... done
> bcachefs (dm-5): check_allocations...key version number higher than recorded: 73014444594 > 0: fix? (y,n, or Y,N for all errors of this type) y
> key version number higher than recorded: 81604378807 > 73014444594: fix? (y,n, or Y,N for all errors of this type) y
> dev 0 has wrong free buckets: got 0, should be 1220580: fix? (y,n, or Y,N for all errors of this type) y
> dev 0 has wrong sb buckets: got 0, should be 13: fix? (y,n, or Y,N for all errors of this type) y
> dev 0 has wrong sb sectors: got 0, should be 6152: fix? (y,n, or Y,N for all errors of this type) y
> dev 0 has wrong sb fragmented: got 0, should be 504: fix? (y,n, or Y,N for all errors of this type) y
> dev 0 has wrong journal buckets: got 0, should be 8192: fix? (y,n, or Y,N for all errors of this type) y
> dev 0 has wrong journal sectors: got 0, should be 4194304: fix? (y,n, or Y,N for all errors of this type) y
> dev 0 has wrong btree buckets: got 0, should be 15: fix? (y,n, or Y,N for all errors of this type) y
> dev 0 has wrong btree sectors: got 0, should be 7680: fix? (y,n, or Y,N for all errors of this type) y
> fs has wrong hidden: got 0, should be 4200960: fix? (y,n, or Y,N for all errors of this type) y
> fs has wrong btree: got 0, should be 7680: fix? (y,n, or Y,N for all errors of this type) y
> fs has wrong nr_inodes: got 20, should be 22: fix? (y,n, or Y,N for all errors of this type) y
> fs has wrong btree: 1/1 [0]: got 0, should be 7680: fix? (y,n, or Y,N for all errors of this type) y
> done
> bcachefs (dm-5): going read-write
> bcachefs (dm-5): journal_replay... done
> bcachefs (dm-5): check_alloc_info...y done
> bcachefs (dm-5): check_lrus... done
> bcachefs (dm-5): check_btree_backpointers... done
> bcachefs (dm-5): check_backpointers_to_extents... done
> bcachefs (dm-5): check_extents_to_backpointers... done
> bcachefs (dm-5): check_alloc_to_lru_refs... done
> bcachefs (dm-5): check_snapshot_trees... done
> bcachefs (dm-5): check_snapshots... done
> bcachefs (dm-5): check_subvols... done
> bcachefs (dm-5): check_subvol_children... done
> bcachefs (dm-5): delete_dead_snapshots... done
> bcachefs (dm-5): check_inodes... done
> bcachefs (dm-5): check_extents... done
> bcachefs (dm-5): check_indirect_extents... done
> bcachefs (dm-5): check_dirents... done
> bcachefs (dm-5): check_xattrs... done
> bcachefs (dm-5): check_root... done
> bcachefs (dm-5): check_subvolume_structure... done
> bcachefs (dm-5): check_directory_structure... done
> bcachefs (dm-5): check_nlinks... done
> bcachefs (dm-5): resume_logged_ops... done
> bcachefs (dm-5): delete_dead_inodes... done
> bcachefs (dm-5): shutdown complete, journal seq 47
> dm-5: errors fixed
>
> For a regular unclean shutdown I would not have expected any filesystem
> errors. A subsequent call to "fsck.bcachefs" revealed no further errors.
The errors are because tools is currently at on disk format version 1.9,
but Linus's tree is still at 1.7, and there was a bug in the downgrade
sb section code that caused us to not run the right recovery passes when
downgrading (just sent the fix today).
We usually don't do the filesystem initialization in userspace on
version mismatch, but you probably had bcachefs built as a module.
> I mounted the filesystem again and tried another time with rsync and
> it did not seem to get stuck as before. However I felt uncomfortable
> with continuing with a filesystem that has had errors already.
> Especially at BCacheFS is still marked experimental.
>
> Also I thought maybe it did not like being mounted without a reboot.
> Does not really make much sense to me, but I thought whatever due to
> lack of a better idea let's try it.
>
> I recreated the filesystem. Then I rebooted.
>
> Then I started the rsync again.
>
> So far it still runs at maximum speed of the GBit link of around
> 110 MiB/s.
>
> Let's see whether it completes this time.
>
> It did. Nothing in kernel log. fsck.bcachefs is happy, too.
>
> Wrote about 188 GiB of data without any apparent issues.
>
> I am not sure what to make of this.
Without more info it's hard to say what happened. We have a lot of
runtime debugging info, so it shouldn't be too hard to see what rsync
was stuck on: /proc/pid/stack to start with, then
/sys/fs/bcachefs/*/internal/alloc_debug if it was the allocator, which
is the most likely culprit.
If you're willing to reproduce it and hop on IRC (irc.oftc.net#bcache),
I'd be happy to walk you through grabbing all the relevant info.
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: rsync stuck in "D" state short after starting to copy to a newly created filesystem
2024-06-28 11:55 rsync stuck in "D" state short after starting to copy to a newly created filesystem Martin Steigerwald
2024-06-28 14:45 ` Kent Overstreet
@ 2024-06-29 23:09 ` Kent Overstreet
2024-06-30 8:03 ` Martin Steigerwald
1 sibling, 1 reply; 4+ messages in thread
From: Kent Overstreet @ 2024-06-29 23:09 UTC (permalink / raw)
To: Martin Steigerwald; +Cc: linux-bcachefs
On Fri, Jun 28, 2024 at 01:55:08PM +0200, Martin Steigerwald wrote:
> Hi!
>
> I am ending a migration from a ThinkPad T14 AMD Gen 1 to a ThinkPad T14
> AMD Gen 5.
>
> Last filesystem is a BCacheFS with some larger files that I use for testing
> BCacheFS. rsync was directly pulling from the older laptop over 1 GBit
> link through my local router. All other filesystems are BTRFS and there
> have not been an issue with migrating about 1,8 TiB of data to three
> BTRFS filesystems via rsync.
>
> Standard Debian Unstable Kernel as of today (on Devuan):
>
> Linux version 6.9.7-amd64 (debian-kernel@lists.debian.org)
> (x86_64-linux-gnu-gcc-13 (Debian 13.3.0-1) 13.3.0,
> GNU ld (GNU Binutils for Debian) 2.42.50.20240625)
> #1 SMP PREEMPT_DYNAMIC Debian 6.9.7-1 (2024-06-27)
>
> % bcachefs version
> 1.9.1
>
> SSD is 4 TB Samsung 990 Pro. BCacheFS is on LUKS encrypted LVM as the
> BTRFS filesystems as well.
>
> I created BCacheFS as follows (this is from a subsequent mkfs.bcachefs.
> I do not have initial output anymore as I already overwrote it with the
> new successful attempt in my documentation, but other than UUIDs nothing
> should have changed I bet, the parameters were identical - see below for
> the successful attempt):
>
> % mkfs.bcachefs --data_checksum xxhash --metadata_checksum xxhash
> --compression=lz4 /dev/nvme1/daten2
> [… identifiers deleted …]
> Device index: 0
> Label:
> Version: 1.7: mi_btree_bitmap
> Version upgrade complete: 0.0: (unknown version)
> Oldest version on disk: 1.7: mi_btree_bitmap
> Created: […]
> Sequence number: 0
> Time of last write: Thu Jan 1 01:00:00 1970
> Superblock size: 976 B/1.00 MiB
> Clean: 0
> Devices: 1
> Sections: members_v1,members_v2
> Features:
> Compat features:
>
> Options:
> block_size: 512 B
> btree_node_size: 256 KiB
> errors: continue [ro] panic
> metadata_replicas: 1
> data_replicas: 1
> metadata_replicas_required: 1
> data_replicas_required: 1
> encoded_extent_max: 64.0 KiB
> metadata_checksum: none crc32c crc64 [xxhash]
> data_checksum: none crc32c crc64 [xxhash]
> compression: lz4
> background_compression: none
> str_hash: crc32c crc64 [siphash]
> metadata_target: none
> foreground_target: none
> background_target: none
> promote_target: none
> erasure_code: 0
> inodes_32bit: 1
> shard_inode_numbers: 1
> inodes_use_key_cache: 1
> gc_reserve_percent: 8
> gc_reserve_bytes: 0 B
> root_reserve_percent: 0
> wide_macs: 0
> acl: 1
> usrquota: 0
> grpquota: 0
> prjquota: 0
> journal_flush_delay: 1000
> journal_flush_disabled: 0
> journal_reclaim_delay: 100
> journal_transaction_names: 1
> version_upgrade: [compatible] incompatible none
> nocow: 0
>
> members_v2 (size 160):
> Device: 0
> Label: (none)
> UUID: […]
> Size: 300 GiB
> read errors: 0
> write errors: 0
> checksum errors: 0
> seqread iops: 0
> seqwrite iops: 0
> randread iops: 0
> randwrite iops: 0
> Bucket size: 256 KiB
> First bucket: 0
> Buckets: 1228800
> Last mount: (never)
> Last superblock write: 0
> State: rw
> Data allowed: journal,btree,user
> Has data: (none)
> Btree allocated bitmap blocksize: 1.00 B
> Btree allocated bitmap: 0000000000000000000000000000000000000000000000000000000000000000
> Durability: 1
> Discard: 0
> Freespace initialized: 0
>
>
> Directly after creating it I mounted it from /etc/fstab:
>
> /dev/nvme1/daten2 /daten2 bcachefs lazytime 0 0
>
> Soon after the copying process started, rsync got stuck in "D" state.
> It was within the first 500 MiB or so. Nothing in kernel log. I waited
> a bit and then stopped rsync. One rsync process remained in "D" state
> and thus did not go away. I tried another time and one the rsync
> processes was immediately in "D" state.
>
> Thus I rebooted. Runit hung during reboot. Likely due to processes in
> D state. I eventually switched up the laptop by pressing the power
> button long enough.
>
> I did an fsck.bcachefs and got:
>
> % fsck.bcachefs /dev/nvme1/daten2
> fsck binary is version 1.9: disk_accounting_v2 but filesystem is 1.7: mi_btree_bitmap and kernel is 1.7: mi_btree_bitmap, using kernel fsck
> bcachefs (dm-5): mounting version 1.7: mi_btree_bitmap opts=ro,metadata_checksum=xxhash,data_checksum=xxhash,compression=lz4,degraded,fsck,fix_errors=ask,read_only
> bcachefs (dm-5): recovering from clean shutdown, journal seq 45
> bcachefs (dm-5): journal read done, replaying entries 45-45
> bcachefs (dm-5): alloc_read... done
> bcachefs (dm-5): stripes_read... done
> bcachefs (dm-5): snapshots_read... done
> bcachefs (dm-5): check_allocations...key version number higher than recorded: 73014444594 > 0: fix? (y,n, or Y,N for all errors of this type) y
> key version number higher than recorded: 81604378807 > 73014444594: fix? (y,n, or Y,N for all errors of this type) y
> dev 0 has wrong free buckets: got 0, should be 1220580: fix? (y,n, or Y,N for all errors of this type) y
> dev 0 has wrong sb buckets: got 0, should be 13: fix? (y,n, or Y,N for all errors of this type) y
> dev 0 has wrong sb sectors: got 0, should be 6152: fix? (y,n, or Y,N for all errors of this type) y
> dev 0 has wrong sb fragmented: got 0, should be 504: fix? (y,n, or Y,N for all errors of this type) y
> dev 0 has wrong journal buckets: got 0, should be 8192: fix? (y,n, or Y,N for all errors of this type) y
> dev 0 has wrong journal sectors: got 0, should be 4194304: fix? (y,n, or Y,N for all errors of this type) y
> dev 0 has wrong btree buckets: got 0, should be 15: fix? (y,n, or Y,N for all errors of this type) y
> dev 0 has wrong btree sectors: got 0, should be 7680: fix? (y,n, or Y,N for all errors of this type) y
> fs has wrong hidden: got 0, should be 4200960: fix? (y,n, or Y,N for all errors of this type) y
> fs has wrong btree: got 0, should be 7680: fix? (y,n, or Y,N for all errors of this type) y
> fs has wrong nr_inodes: got 20, should be 22: fix? (y,n, or Y,N for all errors of this type) y
> fs has wrong btree: 1/1 [0]: got 0, should be 7680: fix? (y,n, or Y,N for all errors of this type) y
> done
> bcachefs (dm-5): going read-write
> bcachefs (dm-5): journal_replay... done
> bcachefs (dm-5): check_alloc_info...y done
> bcachefs (dm-5): check_lrus... done
> bcachefs (dm-5): check_btree_backpointers... done
> bcachefs (dm-5): check_backpointers_to_extents... done
> bcachefs (dm-5): check_extents_to_backpointers... done
> bcachefs (dm-5): check_alloc_to_lru_refs... done
> bcachefs (dm-5): check_snapshot_trees... done
> bcachefs (dm-5): check_snapshots... done
> bcachefs (dm-5): check_subvols... done
> bcachefs (dm-5): check_subvol_children... done
> bcachefs (dm-5): delete_dead_snapshots... done
> bcachefs (dm-5): check_inodes... done
> bcachefs (dm-5): check_extents... done
> bcachefs (dm-5): check_indirect_extents... done
> bcachefs (dm-5): check_dirents... done
> bcachefs (dm-5): check_xattrs... done
> bcachefs (dm-5): check_root... done
> bcachefs (dm-5): check_subvolume_structure... done
> bcachefs (dm-5): check_directory_structure... done
> bcachefs (dm-5): check_nlinks... done
> bcachefs (dm-5): resume_logged_ops... done
> bcachefs (dm-5): delete_dead_inodes... done
> bcachefs (dm-5): shutdown complete, journal seq 47
> dm-5: errors fixed
>
> For a regular unclean shutdown I would not have expected any filesystem
> errors. A subsequent call to "fsck.bcachefs" revealed no further errors.
>
> I mounted the filesystem again and tried another time with rsync and
> it did not seem to get stuck as before. However I felt uncomfortable
> with continuing with a filesystem that has had errors already.
> Especially at BCacheFS is still marked experimental.
This should all be fixed in this branch:
https://evilpiepirate.org/git/bcachefs.git/log/?h=bcachefs-for-6.9
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: rsync stuck in "D" state short after starting to copy to a newly created filesystem
2024-06-29 23:09 ` Kent Overstreet
@ 2024-06-30 8:03 ` Martin Steigerwald
0 siblings, 0 replies; 4+ messages in thread
From: Martin Steigerwald @ 2024-06-30 8:03 UTC (permalink / raw)
To: Kent Overstreet; +Cc: linux-bcachefs
Kent Overstreet - 30.06.24, 01:09:04 CEST:
> > For a regular unclean shutdown I would not have expected any
> > filesystem
> > errors. A subsequent call to "fsck.bcachefs" revealed no further
> > errors.
> >
> > I mounted the filesystem again and tried another time with rsync and
> > it did not seem to get stuck as before. However I felt uncomfortable
> > with continuing with a filesystem that has had errors already.
> > Especially at BCacheFS is still marked experimental.
>
> This should all be fixed in this branch:
> https://evilpiepirate.org/git/bcachefs.git/log/?h=bcachefs-for-6.9
You found the issue? Great! Thanks.
Thanks for the detailed description on how to look where things may be
stuck. I will keep it around.
--
Martin
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2024-06-30 8:03 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-06-28 11:55 rsync stuck in "D" state short after starting to copy to a newly created filesystem Martin Steigerwald
2024-06-28 14:45 ` Kent Overstreet
2024-06-29 23:09 ` Kent Overstreet
2024-06-30 8:03 ` Martin Steigerwald
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox