Damaged filesystem - request for support

Damaged filesystem - request for support

[Tobiasz Karoń]

[-- Attachment #1.1: Type: text/plain, Size: 2332 bytes --]

Hi!

I have a damaged filesystem using 2 HDDs in a single configuration.
One HDD is a 6 TB Toshiba, the other is a 4 TB Western Digital.
Together they make a 10 TB FS I use for daily borg backups.

Recently I had borg backup crash with input-/output errors. I tried
scrubbing the FS but that failed, then did a check and it reported errors.

My system info:


root@pve:~# uname -a
Linux pve 6.8.12-15-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-15
(2025-09-12T11:02Z) x86_64 GNU/Linux
root@pve:~# btrfs --version
btrfs-progs v6.2
root@pve:~# btrfs fi show
Label: 'unfa-RollingBackup6G'  uuid: a11787a5-de1d-421c-ac2e-b669f948b1f0
       Total devices 2 FS bytes used 9.08TiB
       devid    1 size 0 used 0 path /dev/sdd MISSING
       devid    2 size 0 used 0 path /dev/sdb MISSING
root@pve:~#  btrfs fi df /mnt/backup/
Data, single: total=9.07TiB, used=9.07TiB
System, RAID1: total=32.00MiB, used=1.19MiB
Metadata, RAID1: total=11.03GiB, used=9.91GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

I am attaching a full zstd-compressed dmesg log.

SMART reports no errors, and I can still mount the FS.

btrfs check returned this:

root@pve:~# btrfs check /dev/sde
Opening filesystem to check...
Checking filesystem on /dev/sde
UUID: a11787a5-de1d-421c-ac2e-b669f948b1f0
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space tree
[4/7] checking fs roots
root 5 inode 32750 errors 2000, link count wrong
ERROR: errors found in fs roots
found 9978823688192 bytes used, error(s) found
total csum bytes: 9734550208
total tree bytes: 10644275200
total fs tree bytes: 169132032
total extent tree bytes: 235536384
btree space waste bytes: 458925672
file data blocks allocated: 9968179412992
referenced 9978512461824

I use a dual-disk USB enclosure to accees these drives - I know it's less
than ideal, but the machine I have is low-power and that's the best I can
do with the budget I have for now.

Can I fix this FS and keep going with it? It does not contain critical
information (that's why no redundancy), but it's a backup of stiff I have
on other machines. Loosing it is not the end of the world, but obviously
I'd want my backup to be reliable.

Please help!
Thank you!

Best regards,
-- 
- Tobiasz 'unfa' Karoń

www.youtube.com/unfa000

[-- Attachment #1.2: Type: text/html, Size: 3372 bytes --]

[-- Attachment #2: dmesg.log.tar.zst --]
[-- Type: application/zstd, Size: 48587 bytes --]

Re: Damaged filesystem - request for support

[Chris Murphy]

On Wed, Oct 22, 2025, at 7:27 AM, Tobiasz Karoń wrote:

> root@pve:~# uname -a 
> Linux pve 6.8.12-15-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-15 (2025-09-12T11:02Z) x86_64 GNU/Linux 
> root@pve:~# btrfs --version 
> btrfs-progs v6.2
> 

This is an EOL kernel, so support would come from the provider of this kernel. The 6.8 kernel was released in early 2024, and isn't receiving stable backports from upstream.

Also the btrfs-progs version is quite old, about 2023. It's safe to run it without --repair but there's too many bug fixes since then to give a summary.

> root@pve:~# btrfs fi show 
> Label: 'unfa-RollingBackup6G'  uuid: a11787a5-de1d-421c-ac2e-b669f948b1f0 
>        Total devices 2 FS bytes used 9.08TiB 
>        devid    1 size 0 used 0 path /dev/sdd MISSING 
>        devid    2 size 0 used 0 path /dev/sdb MISSING
> 

Not sure why these are reported missing.

> 
> root@pve:~#  btrfs fi df /mnt/backup/ 
> Data, single: total=9.07TiB, used=9.07TiB 
> System, RAID1: total=32.00MiB, used=1.19MiB 
> Metadata, RAID1: total=11.03GiB, used=9.91GiB 
> GlobalReserve, single: total=512.00MiB, used=0.00B
> I am attaching a full zstd-compressed dmesg log.
> SMART reports no errors, and I can still mount the FS.
> btrfs check returned this:
> 
> root@pve:~# btrfs check /dev/sde  
> Opening filesystem to check... 
> Checking filesystem on /dev/sde 
> UUID: a11787a5-de1d-421c-ac2e-b669f948b1f0 
> [1/7] checking root items 
> [2/7] checking extents 
> [3/7] checking free space tree 
> [4/7] checking fs roots 
> root 5 inode 32750 errors 2000, link count wrong
> 

This is a minor issue, and should be repairable with --repair using recent btrfs-progs, but I can't recommend it with old btrfs-progs.



> 
> I use a dual-disk USB enclosure to accees these drives - I know it's less than ideal, but the machine I have is low-power and that's the best I can do with the budget I have for now.

Kernel first sees them at:

[70132.171424] BTRFS: device label unfa-RollingBackup6G devid 2 transid 70744 /dev/sdb scanned by (udev-worker) (340477)
[70132.337014] BTRFS: device label unfa-RollingBackup6G devid 1 transid 70744 /dev/sdd scanned by (udev-worker) (340481)

There are command queue failures happening with both drives:

[2075668.586774] sd 2:0:0:0: [sdb] tag#18 uas_eh_abort_handler 0 uas-tag 6 inflight: CMD OUT 
[2075668.586778] sd 2:0:0:0: [sdb] tag#18 CDB: Write(16) 8a 00 00 00 00 01 d1 9f af e0 00 00 00 60 00 00
[2075668.586849] sd 2:0:0:0: [sdb] tag#17 uas_eh_abort_handler 0 uas-tag 5 inflight: CMD OUT 
[2075668.586852] sd 2:0:0:0: [sdb] tag#17 CDB: Write(16) 8a 00 00 00 00 01 d1 9f ae e0 00 00 00 20 00 00

[2075668.086683] sd 2:0:0:1: [sdd] tag#0 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD IN 
[2075668.086694] sd 2:0:0:1: [sdd] tag#0 CDB: Read(16) 88 00 00 00 00 00 0e 0a cc 08 00 00 00 08 00 00
[2075668.176701] sd 2:0:0:1: [sdd] tag#23 uas_eh_abort_handler 0 uas-tag 24 inflight: CMD OUT 
[2075668.176718] sd 2:0:0:1: [sdd] tag#23 CDB: Write(16) 8a 00 00 00 00 02 ba 3f f0 e0 00 00 00 80 00 00

This can mean there are dropped reads and writes at the block layer. My guess is there's a firmware quirk with both enclosures and the quirk isn't being set by this kernel. You might be able to work around it by disabling the uas driver for both drives. That can be done with a udev rule.

I have two enclosures that had similar errors and it was fixed either by disabling uas for both drives, or connecting to an externally powered USB hub. Something with a good bit of power in the spec so both drives can use their max runtime amps at the same time. I advise not powering them up at the same time, the draw is too much, and while they will eventually spin up - in effect the drive is "booting" with a brown out and that could result in indeterminate behavior of the firmware.

Also, best to disable the write cache on both drives. Neither drive supports FUA. That too can be set with a udev rule. Make sure to check the hdparm man page and get the correct flag because the wrong one can brick a drive. 


[70132.042567] sd 2:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[70132.042929] sd 2:0:0:1: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA


This errors suggests that drive sdb dropped writes and it was corrected

[2080584.362151] BTRFS error (device sdd): parent transid verify failed on logical 15136619216896 mirror 2 wanted 74627 found 73734
[2080584.376137] BTRFS info (device sdd): read error corrected: ino 0 off 15136619216896 (dev /dev/sdb sector 7811870432)
[2080584.376260] BTRFS info (device sdd): read error corrected: ino 0 off 15136619220992 (dev /dev/sdb sector 7811870440)
[2080584.376378] BTRFS info (device sdd): read error corrected: ino 0 off 15136619225088 (dev /dev/sdb sector 7811870448)
[2080584.376496] BTRFS info (device sdd): read error corrected: ino 0 off 15136619229184 (dev /dev/sdb sector 7811870456)


More command queue errors

[2086918.503672] sd 2:0:0:1: [sdd] tag#3 uas_eh_abort_handler 0 uas-tag 24 inflight: CMD OUT 
[2086918.503681] sd 2:0:0:1: [sdd] tag#3 CDB: Write(16) 8a 00 00 00 00 02 ba 23 da 20 00 00 00 40 00 00

Device reset

[2086940.343856] usb 4-1.2: reset SuperSpeed USB device number 5 using xhci_hcd

And also the other file systems have issues going on too which is likely related to the hardware/firmware/kernel block layer confusion - therefore all the file systems get confused.

And then enospc issues, unrelated to the above.

[2092753.887210] BTRFS info (device sdd): balance: start -dusage=1 -musage=1 -susage=1
[2092784.472774] BTRFS info (device sdd): 157 enospc errors during balance
[2092784.472782] BTRFS info (device sdd): balance: ended with status: -28

Add more space I assume

[2652031.025503] BTRFS info (device sdd): disk added /dev/loop0

The balance now proceeds. More errors and fixups from good copies on the mirror (lucky since both drives appear to be occasionally dropping writes - just so far not the same writes):

[2652149.062979] BTRFS error (device sdd): bad tree block start, mirror 2 want 15135668535296 have 8901696662615020189
[2652149.184097] BTRFS info (device sdd): read error corrected: ino 0 off 15135668535296 (dev /dev/sdb sector 7810013632)
[2652149.184226] BTRFS info (device sdd): read error corrected: ino 0 off 15135668539392 (dev /dev/sdb sector 7810013640)
[2652149.184370] BTRFS info (device sdd): read error corrected: ino 0 off 15135668543488 (dev /dev/sdb sector 7810013648)
[2652149.184511] BTRFS info (device sdd): read error corrected: ino 0 off 15135668547584 (dev /dev/sdb sector 7810013656)


The balance continues for a long time - so a balance is a very intense process. I'm not sure why balance 90 was indicated for this file system, but it's going to take a long time during which a write could be dropped at any moment. It's pretty risky. But it finally ends OK.

[2655999.361506] BTRFS info (device sdd): balance: ended with status: 0

But I don't see removal of /dev/loop0. There are more errors found though.

[2719403.975693] BTRFS error (device sdd): bad tree block start, mirror 2 want 15136617250816 have 15136619364352
[2719403.996904] BTRFS info (device sdd): read error corrected: ino 0 off 15136617250816 (dev /dev/sdb sector 7811866592)
[2719403.997058] BTRFS info (device sdd): read error corrected: ino 0 off 15136617254912 (dev /dev/sdb sector 7811866600)
[2719403.997347] BTRFS info (device sdd): read error corrected: ino 0 off 15136617259008 (dev /dev/sdb sector 7811866608)
[2719403.997500] BTRFS info (device sdd): read error corrected: ino 0 off 15136617263104 (dev /dev/sdb sector 7811866616)

next, quite a lot of command queue errors again and then  a new error:

[3082064.445514] I/O error, dev sdd, sector 1378595728 op 0x0:(READ) flags 0x84700 phys_seg 128 prio class 0
[3082064.445543] I/O error, dev sdd, sector 1378594800 op 0x0:(READ) flags 0x80700 phys_seg 116 prio class 0
[3082064.445567] I/O error, dev sdd, sector 1378596752 op 0x0:(READ) flags 0x80700 phys_seg 128 prio class 0
...
there's quite a lot of these now

btrfs reports on the effect of these read errors

[3082064.490661] BTRFS error (device sdd): bdev /dev/sdd errs: wr 0, rd 1, flush 0, corrupt 0, gen 0
[3082064.531667] BTRFS error (device sdd): bdev /dev/sdd errs: wr 0, rd 2, flush 0, corrupt 0, gen 0
[3082064.595676] sd 2:0:0:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
[3082064.608670] BTRFS error (device sdd): bdev /dev/sdd errs: wr 0, rd 3, flush 0, corrupt 0, gen 0

Either the devices are unplugged or drop off the bus, I can't tell but then they come back and....

[3082068.348113] BTRFS warning: duplicate device /dev/sde devid 2 generation 80653 scanned by (udev-worker) (1575180)
[3082068.514282] BTRFS warning: duplicate device /dev/sdf devid 1 generation 80653 scanned by (udev-worker) (1575184)
[3082072.149738] kworker/u66:3: attempt to access beyond end of device
                 sdd: rw=1052673, sector=11695499072, nr_sectors = 32 limit=0
[3082072.149747] BTRFS error (device sdd): bdev /dev/sdd errs: wr 1, rd 3, flush 0, corrupt 0, gen 0
[3082072.149754] kworker/u66:3: attempt to access beyond end of device
                 sdb: rw=1052673, sector=1847256896, nr_sectors = 32 limit=0

...
and kaboom

[3082097.750544] BTRFS info (device sdd: state EA): forced readonly

And then clearly the kernel is confused:

[3082919.543755] BTRFS warning: duplicate device /dev/sde devid 2 generation 80653 scanned by (udev-worker) (1578969)
[3082919.728788] BTRFS warning: duplicate device /dev/sdf devid 1 generation 80653 scanned by (udev-worker) (1578973)
[3082994.380016] BTRFS warning: duplicate device /dev/sdf devid 1 generation 80653 scanned by mount (1579331)
[3083009.198896] BTRFS warning: duplicate device /dev/sde devid 2 generation 80653 scanned by (udev-worker) (1579398)
[3083009.605855] BTRFS warning: duplicate device /dev/sdf devid 1 generation 80653 scanned by (udev-worker) (1579398)


As far as I'm aware, when it gets confused about drives dropping off USB bus and then coming back - it reassigns them new nodes instead of connecting them to the nodes they had before - and that's only fixable with a reboot.

And then figure out how to disable uas for these drives.

And disable the write cache for the drives.

And only then should they be scrubbed, and it'll probably work. But with write cache disabled, writes will be slower yet safer.

I don't think the file system is damaged, I think all the problems are related to enclosures having quirky firmware, very common. These quirks get cleaned up by USB hubs. But also, with later kernels at least my USB drives no longer misbehave when directly connected to computers, instead of via an externally powered USB hub.


--
Chris Murphy

Re: Damaged filesystem - request for support

[Chris Murphy]

On Thu, Oct 23, 2025, at 4:53 AM, Tobiasz Karoń wrote:
> Chris, thank you for such a detailed breakdown of what happened.
> I have disconnected the drives once or maybe even twice to try and reset things. I don't recall exact reasoning, but them disconnecting was not due to power loss. The enclosure is powered with it's own 12-V power adapter plugged into mains. The exact model is StarTech SDOCK2U33V.

OK so if the problem isn't power related, it's just a quirk/incompatibility between the kernel and the enclosure firmware. 

If the new kernel doesn't resolve the problem, options are are to disable uas or use a good USB hub which will "normalizes" the USB stream, i.e. the USB stream is not merely rebroadcast to the host, and vice versa.

But in any case, I advise disabling the write cache for both drives, that should ensure write order is honored, and protect in case of crash or power failure or forced reboots. But especially for doing repair because if repair writes aren't fully flushed to disk, and then there's a crash - I expect there'd be a lot of confusion, possibly unrepairable.

I can't tell from the dmesg if the out of order writes are due to the command queue errors. It's actually pretty common that the drive firmware will silently ignore flush fua, and doesn't even always ignore. In which case you never know about it until there's a crash or power failure and then a problem appears.




> 
> I have another Linux system I want to use to fix the Btrfs errors, it's running MX Linux. I have updated btrfs-progs from Debian backports repository so the system is:
> 
> unfa@mx-workstation ~> btrfs --version 
> btrfs-progs v6.14 
> -EXPERIMENTAL -INJECT -STATIC +LZO +ZSTD +UDEV +FSVERITY +ZONED CRYPTO=builtin 
> unfa@mx-workstation ~> uname -a 
> Linux mx-workstation 6.15.11-1-liquorix-amd64 #1 ZEN SMP PREEMPT_DYNAMIC liquorix 6.15-12~mx23ahs (2025-08-23) x86_64 GNU/Linux
> Is that configuration of kernel + btrfs-progs versions safe to run btrfs check --repari on my broken Btrfs FS?

The versions are fine, but again the Btrfs probably isn't broken. The problem reported by the older btrfs check is a minor issue and unrelated to what's going on. It's a good idea to run the `btrfs check` (no repair) with the new version and post that here and see if there are different problems.

Also, `btrfs check --mode=lowmem` might reveal different problems since it's a different implementation of check. So it's worth seeing that output as well.


> Thank you once more for detailed write-up. I'll look into UDEV to disable the write cache for them and the UAS driver. I was hoping UAS can give me better performance over USB 3.2, but if it's buggy it might be not worth the risk.

It's possible the newer kernel will resolve the problem. These sorts of quirks are constantly being fixed in the kernel. But only if users report the problem to kernel usb folks.


--
Chris Murphy

Re: Damaged filesystem - request for support

[Tobiasz Karoń]

I couldn't wait, so I have went ahead and run the check with --repair,
but the system froze after undetermined amount of time.
I was unsure if the repair has finished or not, but I could not regain
any control over the system, so I attempted to gracefully shut it down
with SysRq commands.

I have now re-run the check on an even newer version of btrfs-progs on
an Arch Linux system, and it seemed to find no errors now...

❯ btrfs check -p /dev/sde
Opening filesystem to check...
Checking filesystem on /dev/sde
UUID: a11787a5-de1d-421c-ac2e-b669f948b1f0
[1/8] checking log skipped (none written)
[1/7] checking root items                      (0:00:24 elapsed,
2642123 items checked)
[2/7] checking extents                         (0:03:19 elapsed,
649741 items checked)
[3/7] checking free space tree                 (0:00:14 elapsed, 9306
items checked)
[4/7] checking fs roots                        (0:05:36 elapsed, 10289
items checked)
[5/7] checking csums (without verifying data)  (0:05:37 elapsed,
981538 items checked)
[6/7] checking root refs                       (0:00:00 elapsed, 3
items checked)
[8/8] checking quota groups skipped (not enabled on this FS)
found 9978824622080 bytes used, no error found
total csum bytes: 9734550208
total tree bytes: 10645209088
total fs tree bytes: 169132032
total extent tree bytes: 235716608
btree space waste bytes: 459278040
file data blocks allocated: 9968179412992
referenced 9978512461824

Now the low-memory mode:

❯ btrfs check -p --mode=lowmem /dev/sde
Opening filesystem to check...
Checking filesystem on /dev/sde
UUID: a11787a5-de1d-421c-ac2e-b669f948b1f0
[1/8] checking log skipped (none written)
[1/7] checking root items                      (0:00:24 elapsed,
2642123 items checked)
[2/7] checking extents                         (0:45:28 elapsed,
647281 items checked)
[3/7] checking free space tree                 (0:00:08 elapsed, 9306
items checked)
[4/7] checking fs roots                        (0:04:18 elapsed, 2655
items checked)
[5/7] checking csums (without verifying data)  (0:01:17 elapsed,
981538 items checked)
[7/8] checking root refs done with fs roots in lowmem mode, skipping
[8/8] checking quota groups skipped (not enabled on this FS)
found 9978824622080 bytes used, no error found
total csum bytes: 9734550208
total tree bytes: 10645209088
total fs tree bytes: 169132032
total extent tree bytes: 235716608
btree space waste bytes: 459278040
file data blocks allocated: 9968179412992
referenced 9978512461824


The system I am running this on is:
❯ btrfs --version
btrfs-progs v6.17
-EXPERIMENTAL -INJECT -STATIC +LZO +ZSTD +UDEV +FSVERITY +ZONED CRYPTO=libgcrypt
❯ uname -a
Linux unfa-desktop 6.12.53-1-lts #1 SMP PREEMPT_DYNAMIC Wed, 15 Oct
2025 11:05:48 +0000 x86_64 GNU/Linux


I have started a scrub, since check reported no errors, and it's
working now. Seems like the problem is resolved:

Every 10.0s: btrfs scrub status .

                                 pve: Fri Oct 24 12:04:19 2025

UUID:             a11787a5-de1d-421c-ac2e-b669f948b1f0
Scrub started:    Fri Oct 24 11:21:29 2025
Status:           running
Duration:         0:42:47
Time left:        10:43:39
ETA:              Fri Oct 24 22:47:58 2025
Total to scrub:   9.08TiB
Bytes scrubbed:   579.84GiB  (6.23%)
Rate:             231.30MiB/s
Error summary:    no errors found


On Thu, Oct 23, 2025 at 10:09 PM Chris Murphy <lists@colorremedies.com> wrote:
>
>
>
> On Thu, Oct 23, 2025, at 4:53 AM, Tobiasz Karoń wrote:
> > Chris, thank you for such a detailed breakdown of what happened.
> > I have disconnected the drives once or maybe even twice to try and reset things. I don't recall exact reasoning, but them disconnecting was not due to power loss. The enclosure is powered with it's own 12-V power adapter plugged into mains. The exact model is StarTech SDOCK2U33V.
>
> OK so if the problem isn't power related, it's just a quirk/incompatibility between the kernel and the enclosure firmware.
>
> If the new kernel doesn't resolve the problem, options are are to disable uas or use a good USB hub which will "normalizes" the USB stream, i.e. the USB stream is not merely rebroadcast to the host, and vice versa.
>
> But in any case, I advise disabling the write cache for both drives, that should ensure write order is honored, and protect in case of crash or power failure or forced reboots. But especially for doing repair because if repair writes aren't fully flushed to disk, and then there's a crash - I expect there'd be a lot of confusion, possibly unrepairable.
>
> I can't tell from the dmesg if the out of order writes are due to the command queue errors. It's actually pretty common that the drive firmware will silently ignore flush fua, and doesn't even always ignore. In which case you never know about it until there's a crash or power failure and then a problem appears.
>
>
>
>
> >
> > I have another Linux system I want to use to fix the Btrfs errors, it's running MX Linux. I have updated btrfs-progs from Debian backports repository so the system is:
> >
> > unfa@mx-workstation ~> btrfs --version
> > btrfs-progs v6.14
> > -EXPERIMENTAL -INJECT -STATIC +LZO +ZSTD +UDEV +FSVERITY +ZONED CRYPTO=builtin
> > unfa@mx-workstation ~> uname -a
> > Linux mx-workstation 6.15.11-1-liquorix-amd64 #1 ZEN SMP PREEMPT_DYNAMIC liquorix 6.15-12~mx23ahs (2025-08-23) x86_64 GNU/Linux
> > Is that configuration of kernel + btrfs-progs versions safe to run btrfs check --repari on my broken Btrfs FS?
>
> The versions are fine, but again the Btrfs probably isn't broken. The problem reported by the older btrfs check is a minor issue and unrelated to what's going on. It's a good idea to run the `btrfs check` (no repair) with the new version and post that here and see if there are different problems.
>
> Also, `btrfs check --mode=lowmem` might reveal different problems since it's a different implementation of check. So it's worth seeing that output as well.
>
>
> > Thank you once more for detailed write-up. I'll look into UDEV to disable the write cache for them and the UAS driver. I was hoping UAS can give me better performance over USB 3.2, but if it's buggy it might be not worth the risk.
>
> It's possible the newer kernel will resolve the problem. These sorts of quirks are constantly being fixed in the kernel. But only if users report the problem to kernel usb folks.
>
>
> --
> Chris Murphy



-- 
- Tobiasz 'unfa' Karoń

www.youtube.com/unfa000

Re: Damaged filesystem - request for support

[Tobiasz Karoń]

Here's scrub status:


UUID:             a11787a5-de1d-421c-ac2e-b669f948b1f0
Scrub started:    Fri Oct 24 11:21:29 2025
Status:           finished
Duration:         15:49:35
Total to scrub:   9.09TiB
Rate:             167.21MiB/s
Error summary:    csum=1068
 Corrected:      172
 Uncorrectable:  896
 Unverified:     0

I assume this means I can continue to use the filesystem, accepting
that a number of files will be inaccessible and result in i/o errors
when accessed?
Since the data is replacable (assuming I don't need to go back to an
older backup, which I don't currently have a need for), I can destroy
this filesystem and start anew with a clean borg backup.
I am not sure what would be better.
I don't want the FS to constantly remount in ro mode due to i/o errors.

On Fri, Oct 24, 2025 at 12:26 PM Tobiasz Karoń <unfa00@gmail.com> wrote:
>
> I couldn't wait, so I have went ahead and run the check with --repair,
> but the system froze after undetermined amount of time.
> I was unsure if the repair has finished or not, but I could not regain
> any control over the system, so I attempted to gracefully shut it down
> with SysRq commands.
>
> I have now re-run the check on an even newer version of btrfs-progs on
> an Arch Linux system, and it seemed to find no errors now...
>
> ❯ btrfs check -p /dev/sde
> Opening filesystem to check...
> Checking filesystem on /dev/sde
> UUID: a11787a5-de1d-421c-ac2e-b669f948b1f0
> [1/8] checking log skipped (none written)
> [1/7] checking root items                      (0:00:24 elapsed,
> 2642123 items checked)
> [2/7] checking extents                         (0:03:19 elapsed,
> 649741 items checked)
> [3/7] checking free space tree                 (0:00:14 elapsed, 9306
> items checked)
> [4/7] checking fs roots                        (0:05:36 elapsed, 10289
> items checked)
> [5/7] checking csums (without verifying data)  (0:05:37 elapsed,
> 981538 items checked)
> [6/7] checking root refs                       (0:00:00 elapsed, 3
> items checked)
> [8/8] checking quota groups skipped (not enabled on this FS)
> found 9978824622080 bytes used, no error found
> total csum bytes: 9734550208
> total tree bytes: 10645209088
> total fs tree bytes: 169132032
> total extent tree bytes: 235716608
> btree space waste bytes: 459278040
> file data blocks allocated: 9968179412992
> referenced 9978512461824
>
> Now the low-memory mode:
>
> ❯ btrfs check -p --mode=lowmem /dev/sde
> Opening filesystem to check...
> Checking filesystem on /dev/sde
> UUID: a11787a5-de1d-421c-ac2e-b669f948b1f0
> [1/8] checking log skipped (none written)
> [1/7] checking root items                      (0:00:24 elapsed,
> 2642123 items checked)
> [2/7] checking extents                         (0:45:28 elapsed,
> 647281 items checked)
> [3/7] checking free space tree                 (0:00:08 elapsed, 9306
> items checked)
> [4/7] checking fs roots                        (0:04:18 elapsed, 2655
> items checked)
> [5/7] checking csums (without verifying data)  (0:01:17 elapsed,
> 981538 items checked)
> [7/8] checking root refs done with fs roots in lowmem mode, skipping
> [8/8] checking quota groups skipped (not enabled on this FS)
> found 9978824622080 bytes used, no error found
> total csum bytes: 9734550208
> total tree bytes: 10645209088
> total fs tree bytes: 169132032
> total extent tree bytes: 235716608
> btree space waste bytes: 459278040
> file data blocks allocated: 9968179412992
> referenced 9978512461824
>
>
> The system I am running this on is:
> ❯ btrfs --version
> btrfs-progs v6.17
> -EXPERIMENTAL -INJECT -STATIC +LZO +ZSTD +UDEV +FSVERITY +ZONED CRYPTO=libgcrypt
> ❯ uname -a
> Linux unfa-desktop 6.12.53-1-lts #1 SMP PREEMPT_DYNAMIC Wed, 15 Oct
> 2025 11:05:48 +0000 x86_64 GNU/Linux
>
>
> I have started a scrub, since check reported no errors, and it's
> working now. Seems like the problem is resolved:
>
> Every 10.0s: btrfs scrub status .
>
>                                  pve: Fri Oct 24 12:04:19 2025
>
> UUID:             a11787a5-de1d-421c-ac2e-b669f948b1f0
> Scrub started:    Fri Oct 24 11:21:29 2025
> Status:           running
> Duration:         0:42:47
> Time left:        10:43:39
> ETA:              Fri Oct 24 22:47:58 2025
> Total to scrub:   9.08TiB
> Bytes scrubbed:   579.84GiB  (6.23%)
> Rate:             231.30MiB/s
> Error summary:    no errors found
>
>
> On Thu, Oct 23, 2025 at 10:09 PM Chris Murphy <lists@colorremedies.com> wrote:
> >
> >
> >
> > On Thu, Oct 23, 2025, at 4:53 AM, Tobiasz Karoń wrote:
> > > Chris, thank you for such a detailed breakdown of what happened.
> > > I have disconnected the drives once or maybe even twice to try and reset things. I don't recall exact reasoning, but them disconnecting was not due to power loss. The enclosure is powered with it's own 12-V power adapter plugged into mains. The exact model is StarTech SDOCK2U33V.
> >
> > OK so if the problem isn't power related, it's just a quirk/incompatibility between the kernel and the enclosure firmware.
> >
> > If the new kernel doesn't resolve the problem, options are are to disable uas or use a good USB hub which will "normalizes" the USB stream, i.e. the USB stream is not merely rebroadcast to the host, and vice versa.
> >
> > But in any case, I advise disabling the write cache for both drives, that should ensure write order is honored, and protect in case of crash or power failure or forced reboots. But especially for doing repair because if repair writes aren't fully flushed to disk, and then there's a crash - I expect there'd be a lot of confusion, possibly unrepairable.
> >
> > I can't tell from the dmesg if the out of order writes are due to the command queue errors. It's actually pretty common that the drive firmware will silently ignore flush fua, and doesn't even always ignore. In which case you never know about it until there's a crash or power failure and then a problem appears.
> >
> >
> >
> >
> > >
> > > I have another Linux system I want to use to fix the Btrfs errors, it's running MX Linux. I have updated btrfs-progs from Debian backports repository so the system is:
> > >
> > > unfa@mx-workstation ~> btrfs --version
> > > btrfs-progs v6.14
> > > -EXPERIMENTAL -INJECT -STATIC +LZO +ZSTD +UDEV +FSVERITY +ZONED CRYPTO=builtin
> > > unfa@mx-workstation ~> uname -a
> > > Linux mx-workstation 6.15.11-1-liquorix-amd64 #1 ZEN SMP PREEMPT_DYNAMIC liquorix 6.15-12~mx23ahs (2025-08-23) x86_64 GNU/Linux
> > > Is that configuration of kernel + btrfs-progs versions safe to run btrfs check --repari on my broken Btrfs FS?
> >
> > The versions are fine, but again the Btrfs probably isn't broken. The problem reported by the older btrfs check is a minor issue and unrelated to what's going on. It's a good idea to run the `btrfs check` (no repair) with the new version and post that here and see if there are different problems.
> >
> > Also, `btrfs check --mode=lowmem` might reveal different problems since it's a different implementation of check. So it's worth seeing that output as well.
> >
> >
> > > Thank you once more for detailed write-up. I'll look into UDEV to disable the write cache for them and the UAS driver. I was hoping UAS can give me better performance over USB 3.2, but if it's buggy it might be not worth the risk.
> >
> > It's possible the newer kernel will resolve the problem. These sorts of quirks are constantly being fixed in the kernel. But only if users report the problem to kernel usb folks.
> >
> >
> > --
> > Chris Murphy
>
>
>
> --
> - Tobiasz 'unfa' Karoń
>
> www.youtube.com/unfa000



-- 
- Tobiasz 'unfa' Karoń

www.youtube.com/unfa000

Re: Damaged filesystem - request for support

[Chris Murphy]

On Sat, Oct 25, 2025, at 4:58 AM, Tobiasz Karoń wrote:
> Here's scrub status:
>
>
> UUID:             a11787a5-de1d-421c-ac2e-b669f948b1f0
> Scrub started:    Fri Oct 24 11:21:29 2025
> Status:           finished
> Duration:         15:49:35
> Total to scrub:   9.09TiB
> Rate:             167.21MiB/s
> Error summary:    csum=1068
>  Corrected:      172
>  Uncorrectable:  896
>  Unverified:     0
>
> I assume this means I can continue to use the filesystem, accepting
> that a number of files will be inaccessible and result in i/o errors
> when accessed?

Qu, does `btrfs check` verify both copies of metadata when available? Or only one copy? Does btrfs check ignore csum mismatches?

From first post, this file system is:

#  btrfs fi df /mnt/backup/ 
Data, single: total=9.07TiB, used=9.07TiB 
System, RAID1: total=32.00MiB, used=1.19MiB 
Metadata, RAID1: total=11.03GiB, used=9.91GiB 

Since btrfs check says the fs is clean, but then scrub finds and corrects 172 errors, that must mean those csum mismatches for some of the metadata that btrfs check did not detect? It's a little confusing what state the file system is in now because the results don't explicitly tell us. We have to infer it.

Tobiasz, I think you need to run the btrfs check again (normal mode only is ok I think)  to be sure the metadata is OK. It's a little tedious but that is pretty important as you point out.

Note that the scrub is actually performed by btrfs kernel code and all messages about the scrub will be in dmesg. All of those 1068 csum mismatches will produces at least one message each. What is affected, whether a fixup was attempted, and whether the fixup worked. Metadata csum mismatch error looks different from data csum mismatch error. The data error will show path to the file affected. So the dmesg will leak file names.


> Since the data is replacable (assuming I don't need to go back to an
> older backup, which I don't currently have a need for), I can destroy
> this filesystem and start anew with a clean borg backup.

I'm not super familiar with borg, but I expect that it has an option to verify all data blocks on both sides. It probably takes quite a bit longer since both sides have to read and compare data. Btrfs never hands over corrupt data in normal operation, instead it returns EIO for any blocks that fail checksum verification. The handling in case of EIO is up to the application. What I'd like to think happens is borg will see EIO, know the target file is bad (or missing some blocks) and will replace it with a good copy from the source.


> I am not sure what would be better.
> I don't want the FS to constantly remount in ro mode due to i/o errors.

Right. Which is why the first order of business is to make sure the IO errors aren't happening anymore. Either new kernel has fixed that problem, or disabling uas will fix it, or using a USB hub will fix it (I would borrow a hub before buying one for testing purposes if it comes to that).

-- 
Chris Murphy

Re: Damaged filesystem - request for support

[Tobiasz Karoń]

I have been trying various things, also with other drives.
I purchased a 20 TB Toshiba drive and I am currently attempting to do
a burn-in before using it for backup.

I see issues with the USB exclosure, where it resets ev en with jsut
this one drive in it (so one bay is empty). Dmesg says:

[Wed Nov 12 11:09:31 2025] sd 2:0:0:0: [sdc] tag#19
uas_eh_abort_handler 0 uas-tag 1 inflight: CMD
[Wed Nov 12 11:09:31 2025] sd 2:0:0:0: [sdc] tag#19 CDB: ATA command
pass through(16) 85 06 0c 00 d4 00 00 00 82 00 4f 00 c2 00 b0 00
[Wed Nov 12 11:09:31 2025] scsi host2: uas_eh_device_reset_handler start
[Wed Nov 12 11:09:31 2025] usb 4-1.2: reset SuperSpeed USB device
number 7 using xhci_hcd
[Wed Nov 12 11:09:31 2025] scsi host2: uas_eh_device_reset_handler success

[Wed Nov 12 11:11:01 2025] sd 2:0:0:0: [sdc] tag#5
uas_eh_abort_handler 0 uas-tag 1 inflight: CMD
[Wed Nov 12 11:11:01 2025] sd 2:0:0:0: [sdc] tag#5 CDB: ATA command
pass through(16) 85 06 0c 00 d4 00 00 00 82 00 4f 00 c2 00 b0 00
[Wed Nov 12 11:11:01 2025] scsi host2: uas_eh_device_reset_handler start
[Wed Nov 12 11:11:01 2025] usb 4-1.2: reset SuperSpeed USB device
number 7 using xhci_hcd
[Wed Nov 12 11:11:01 2025] scsi host2: uas_eh_device_reset_handler success

That happens after a minute wheenver I do:
smartctl -l long -C /dev/sdc

So it seems I can't do a long, captive HDD test through this enclosure.
This is the self-test log for the drive:

root@pve:/mnt# smartctl -l xselftest /dev/sdc
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.8.12-16-pve] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining
LifeTime(hours)  LBA_of_first_error
# 1  Extended captive    Interrupted (host reset)      90%        42         -
# 2  Extended captive    Interrupted (host reset)      90%        42         -
# 3  Extended offline    Aborted by host               90%        42         -
# 4  Extended offline    Aborted by host               90%         1         -

I have tested blacklisting the uas driver, but the enclosure would
just not show up (should I load an alternative driver for it?)
lsusb says this about the 2-bay disk enclosure:
Bus 004 Device 007: ID 174c:55aa ASMedia Technology Inc. ASM1051E SATA
6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge,
ASM1153E SATA 6Gb/s bridge

I wonder if there's something that can be done about this enclosure in
terms of patching the driver or something?
Should I contact a different mailing list?
This is by no means Btrfs-specific at this point.

About write cache disabling, I was unable to do it persistently with
hdparm, but I found that there's a way to disable write caching
persistently with smartclt. It's also possible to disable write cache
reordering, which I suppose could be more dangerous fro Btrfs. I have
once used bcache with Btrfs in writeback mode and after a particular
system crash the FS has gone bad. (I've even written a song about
it..)
smartctl -s wcreorder=off,p /dev/sdc

I have attempted to disable autosuspend for all xhci controlers using PowerTop:

  Good          Enable SATA link power management for host0
  Good          Enable Audio codec power management
  Bad           Autosuspend for USB device xHCI Host Controller [usb1]
  Good          Autosuspend for USB device USB2.0 Hub
[VIA Labs, Inc.         ]
  Good          Autosuspend for USB device USB2.0 Hub [1-1]
  Good          Autosuspend for USB device USB2.1 Hub [Generic]
  Bad           Autosuspend for USB device xHCI Host Controller [usb3]
  Good          Autosuspend for USB device USB3.0 Hub
[VIA Labs, Inc.         ]
  Bad           Autosuspend for USB device xHCI Host Controller [usb2]
  Good          Autosuspend for USB device USB3.2 Hub [Generic]
  Bad           Autosuspend for USB device xHCI Host Controller [usb4]

But another attempt at long self-test aborts after a few seconds and
resets the USB controller:

[Wed Nov 12 11:20:55 2025] sd 2:0:0:0: [sdc] tag#15
uas_eh_abort_handler 0 uas-tag 1 inflight: CMD
[Wed Nov 12 11:20:55 2025] sd 2:0:0:0: [sdc] tag#15 CDB: ATA command
pass through(16) 85 06 0c 00 d4 00 00 00 82 00 4f 00 c2 00 b0 00
[Wed Nov 12 11:21:13 2025] sd 2:0:0:0: [sdc] tag#12
uas_eh_abort_handler 0 uas-tag 2 inflight: CMD IN
[Wed Nov 12 11:21:13 2025] sd 2:0:0:0: [sdc] tag#12 CDB: ATA command
pass through(16) 85 08 0e 00 00 00 01 00 00 00 00 00 00 00 ec 00
[Wed Nov 12 11:21:13 2025] scsi host2: uas_eh_device_reset_handler start
[Wed Nov 12 11:21:13 2025] usb 4-1.2: reset SuperSpeed USB device
number 7 using xhci_hcd
[Wed Nov 12 11:21:13 2025] scsi host2: uas_eh_device_reset_handler success


Same with short self-test. I wonder if I can use this disk in the
enclosure at all if this keeps happening...

I'll use f3 to stress the disk and see if this will reset the USB
controller as well...

f3write . && f3read .

So far 300 GB written and read no problem.
I started a borg repository there and I am doing a backup as test...


On Sat, Oct 25, 2025 at 7:06 PM Chris Murphy <lists@colorremedies.com> wrote:
>
>
>
> On Sat, Oct 25, 2025, at 4:58 AM, Tobiasz Karoń wrote:
> > Here's scrub status:
> >
> >
> > UUID:             a11787a5-de1d-421c-ac2e-b669f948b1f0
> > Scrub started:    Fri Oct 24 11:21:29 2025
> > Status:           finished
> > Duration:         15:49:35
> > Total to scrub:   9.09TiB
> > Rate:             167.21MiB/s
> > Error summary:    csum=1068
> >  Corrected:      172
> >  Uncorrectable:  896
> >  Unverified:     0
> >
> > I assume this means I can continue to use the filesystem, accepting
> > that a number of files will be inaccessible and result in i/o errors
> > when accessed?
>
> Qu, does `btrfs check` verify both copies of metadata when available? Or only one copy? Does btrfs check ignore csum mismatches?
>
> From first post, this file system is:
>
> #  btrfs fi df /mnt/backup/
> Data, single: total=9.07TiB, used=9.07TiB
> System, RAID1: total=32.00MiB, used=1.19MiB
> Metadata, RAID1: total=11.03GiB, used=9.91GiB
>
> Since btrfs check says the fs is clean, but then scrub finds and corrects 172 errors, that must mean those csum mismatches for some of the metadata that btrfs check did not detect? It's a little confusing what state the file system is in now because the results don't explicitly tell us. We have to infer it.
>
> Tobiasz, I think you need to run the btrfs check again (normal mode only is ok I think)  to be sure the metadata is OK. It's a little tedious but that is pretty important as you point out.
>
> Note that the scrub is actually performed by btrfs kernel code and all messages about the scrub will be in dmesg. All of those 1068 csum mismatches will produces at least one message each. What is affected, whether a fixup was attempted, and whether the fixup worked. Metadata csum mismatch error looks different from data csum mismatch error. The data error will show path to the file affected. So the dmesg will leak file names.
>
>
> > Since the data is replacable (assuming I don't need to go back to an
> > older backup, which I don't currently have a need for), I can destroy
> > this filesystem and start anew with a clean borg backup.
>
> I'm not super familiar with borg, but I expect that it has an option to verify all data blocks on both sides. It probably takes quite a bit longer since both sides have to read and compare data. Btrfs never hands over corrupt data in normal operation, instead it returns EIO for any blocks that fail checksum verification. The handling in case of EIO is up to the application. What I'd like to think happens is borg will see EIO, know the target file is bad (or missing some blocks) and will replace it with a good copy from the source.
>
>
> > I am not sure what would be better.
> > I don't want the FS to constantly remount in ro mode due to i/o errors.
>
> Right. Which is why the first order of business is to make sure the IO errors aren't happening anymore. Either new kernel has fixed that problem, or disabling uas will fix it, or using a USB hub will fix it (I would borrow a hub before buying one for testing purposes if it comes to that).
>
> --
> Chris Murphy



--
- Tobiasz 'unfa' Karoń

www.youtube.com/unfa000

Re: Damaged filesystem - request for support

[Nicholas D Steeves]

[-- Attachment #1: Type: text/plain, Size: 5148 bytes --]

Hi Tobiasz,

Tobiasz Karoń <unfa00@gmail.com> writes:

> [Wed Nov 12 11:11:01 2025] sd 2:0:0:0: [sdc] tag#5
> uas_eh_abort_handler 0 uas-tag 1 inflight: CMD
> [Wed Nov 12 11:11:01 2025] sd 2:0:0:0: [sdc] tag#5 CDB: ATA command
> pass through(16) 85 06 0c 00 d4 00 00 00 82 00 4f 00 c2 00 b0 00
> [Wed Nov 12 11:11:01 2025] scsi host2: uas_eh_device_reset_handler start
> [Wed Nov 12 11:11:01 2025] usb 4-1.2: reset SuperSpeed USB device
> number 7 using xhci_hcd
> [Wed Nov 12 11:11:01 2025] scsi host2: uas_eh_device_reset_handler success
>
> That happens after a minute wheenver I do:
> smartctl -l long -C /dev/sdc
>
> So it seems I can't do a long, captive HDD test through this enclosure.
> This is the self-test log for the drive:

USB attached disks, even directly attached (like WD Passport*) without a
SATA2USB bridge are unfortunately often...special...this way.  The way
to run a long SMART test on them (if the controller isn't uselessly
buggy) is to use a "while true" loop to do something like dd a block
from /dev/urandom to /mnt/point/delete_me, sleep for a period shorter
than then the enclosure's sleep timeout (IIRC there are bad controllers
that will always sleep even if you tell them not to), sync, allow to
iterate, and interrupt the loop after you've allowed the long test to
complete, plus an hour, plus checked the self-test log (checking the log
will abort a running test!).  I've also, 3× now, run into disks that
marked blocks for reallocation, in firmware, during their first write ->
read_verify-write cycle, wrote to the manufacturer (three different
manufacturers), and was told that was normal--oh brave new world.

> root@pve:/mnt# smartctl -l xselftest /dev/sdc
> smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.8.12-16-pve] (local build)
> Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org
>
> === START OF READ SMART DATA SECTION ===
> SMART Extended Self-test Log Version: 1 (1 sectors)
> Num  Test_Description    Status                  Remaining
> LifeTime(hours)  LBA_of_first_error
> # 1  Extended captive    Interrupted (host reset)      90%        42         -
> # 2  Extended captive    Interrupted (host reset)      90%        42         -
> # 3  Extended offline    Aborted by host               90%        42         -
> # 4  Extended offline    Aborted by host               90%         1         -
>
> I have tested blacklisting the uas driver, but the enclosure would
> just not show up (should I load an alternative driver for it?)
> lsusb says this about the 2-bay disk enclosure:
> Bus 004 Device 007: ID 174c:55aa ASMedia Technology Inc. ASM1051E SATA
> 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge,
> ASM1153E SATA 6Gb/s bridge
>
> I wonder if there's something that can be done about this enclosure in
> terms of patching the driver or something?
> Should I contact a different mailing list?
> This is by no means Btrfs-specific at this point.

I don't have experience with this chipset, and someone[s] in the
following thread maintain[s] that it works fine in UASP mode in 2022 (it
is documented in various places that it was not fine in 2015).
https://forums.raspberrypi.com/viewtopic.php?t=339591&start=25

> About write cache disabling, I was unable to do it persistently with
> hdparm, but I found that there's a way to disable write caching
> persistently with smartclt. It's also possible to disable write cache
> reordering, which I suppose could be more dangerous fro Btrfs. I have
> once used bcache with Btrfs in writeback mode and after a particular
> system crash the FS has gone bad. (I've even written a song about
> it..)
> smartctl -s wcreorder=off,p /dev/sdc
>
> I have attempted to disable autosuspend for all xhci controlers using PowerTop:

Have you tried adding "usbcore.autosuspend=-1" to your kernel arguments?
(ie: /etc/default/grub, then run `update-grub`).

>   Good          Enable SATA link power management for host0
>   Good          Enable Audio codec power management
>   Bad           Autosuspend for USB device xHCI Host Controller [usb1]
>   Good          Autosuspend for USB device USB2.0 Hub
> [VIA Labs, Inc.         ]
>   Good          Autosuspend for USB device USB2.0 Hub [1-1]
>   Good          Autosuspend for USB device USB2.1 Hub [Generic]
>   Bad           Autosuspend for USB device xHCI Host Controller [usb3]
>   Good          Autosuspend for USB device USB3.0 Hub
> [VIA Labs, Inc.         ]
>   Bad           Autosuspend for USB device xHCI Host Controller [usb2]
>   Good          Autosuspend for USB device USB3.2 Hub [Generic]
>   Bad           Autosuspend for USB device xHCI Host Controller [usb4]

How is VIA's xHCI support for UASP?  Missing usb-host quirks could
hypothetically be needed, and USB device manufacturers sometimes assume
Intel to such an extent that there have been cases where the hardware
(I'm thinking of a handful of audio interfaces) didn't work at all on
AMD motherboards of that era.  I assume this class of issue can be
mitigated in-kernel.

Cheers,
Nicholas

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

Re: Damaged filesystem - request for support

[Chris Murphy]

On Wed, Nov 12, 2025, at 7:57 AM, Tobiasz Karoń wrote:
> I have been trying various things, also with other drives.
> I purchased a 20 TB Toshiba drive and I am currently attempting to do
> a burn-in before using it for backup.
>
> I see issues with the USB exclosure, where it resets ev en with jsut
> this one drive in it (so one bay is empty). Dmesg says:
>
> [Wed Nov 12 11:09:31 2025] sd 2:0:0:0: [sdc] tag#19
> uas_eh_abort_handler 0 uas-tag 1 inflight: CMD
> [Wed Nov 12 11:09:31 2025] sd 2:0:0:0: [sdc] tag#19 CDB: ATA command
> pass through(16) 85 06 0c 00 d4 00 00 00 82 00 4f 00 c2 00 b0 00
> [Wed Nov 12 11:09:31 2025] scsi host2: uas_eh_device_reset_handler start
> [Wed Nov 12 11:09:31 2025] usb 4-1.2: reset SuperSpeed USB device
> number 7 using xhci_hcd
> [Wed Nov 12 11:09:31 2025] scsi host2: uas_eh_device_reset_handler success
>
> [Wed Nov 12 11:11:01 2025] sd 2:0:0:0: [sdc] tag#5
> uas_eh_abort_handler 0 uas-tag 1 inflight: CMD
> [Wed Nov 12 11:11:01 2025] sd 2:0:0:0: [sdc] tag#5 CDB: ATA command
> pass through(16) 85 06 0c 00 d4 00 00 00 82 00 4f 00 c2 00 b0 00
> [Wed Nov 12 11:11:01 2025] scsi host2: uas_eh_device_reset_handler start
> [Wed Nov 12 11:11:01 2025] usb 4-1.2: reset SuperSpeed USB device
> number 7 using xhci_hcd
> [Wed Nov 12 11:11:01 2025] scsi host2: uas_eh_device_reset_handler success
>
> That happens after a minute wheenver I do:
> smartctl -l long -C /dev/sdc
>
> So it seems I can't do a long, captive HDD test through this enclosure.

       -C, --captive
              [ATA] Runs self-tests in captive mode.  This has no effect with '-t offline' or if the '-t' option is not used.

              WARNING: Tests run in captive mode may busy out the drive for the length of the test.  Only run captive tests on drives without any mounted partitions!


Maybe use -t long and omit -C


>
> I have tested blacklisting the uas driver, but the enclosure would
> just not show up (should I load an alternative driver for it?)

usb_storage


> lsusb says this about the 2-bay disk enclosure:
> Bus 004 Device 007: ID 174c:55aa ASMedia Technology Inc. ASM1051E SATA
> 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge,
> ASM1153E SATA 6Gb/s bridge
>
> I wonder if there's something that can be done about this enclosure in
> terms of patching the driver or something?
> Should I contact a different mailing list?

linux-usb@vger.kernel.org

http://www.linux-usb.org


> About write cache disabling, I was unable to do it persistently with
> hdparm, but I found that there's a way to disable write caching
> persistently with smartclt. It's also possible to disable write cache
> reordering, which I suppose could be more dangerous fro Btrfs. 

I'm not sure about that. Btrfs has no optimizations for write order based on device topology whereas the drive firmware knows things about that topology and can acceptably do out of order writes for performance reasons that are quite safe. 

Granted, the code for these settings is not available fo us to inspect. 


>
> Same with short self-test. I wonder if I can use this disk in the
> enclosure at all if this keeps happening...

I just wouldn't use the captive test.



-- 
Chris Murphy

Re: Damaged filesystem - request for support

[Tobiasz Karoń]

Nicholas, Chris - thank you for your input

I have let the 20 TB Toshiba run more tests, I have used a while loop
with a 30s pause to keep the bay from sleeping. Eitehr it or something
else (I was writing 2 different streams of backups to the drive at the
time as well) had apparently prevented the disk from being lost by the
system, *despite* the UAS device resetting!

Here's my sneaky write loop (4K as that's the physical block size on
the drive, and I believe the Btrfs filesystem I created on it also
adapted that):
while [[ true ]]; do dd if=/dev/urandom bs=4K count=1 of=keepalive.dd;
sleep 30s; done

While this, a Proxmox VE backup and a network remote borg backup were
being written to the drive I have by mistake run this:

smartctl -t long -C /dev/sdc

I have expected a catastrophic result, and immediatelly tried to back
of, hitting Ctrl+C two times over a couple seconds. At first it seemed
like it's the drive is gonna get lost, and I'll have Btrfs errors in
dmesg - but that was not the case!
There have been a few grueling pauses, but in that time the backup
processes of pve and a remote borg instance (from my desktop ot the
home server) did not crash. The drive and filesystem seemed to have
recovered gracefully with no errors reported in dmesg. Outside of this
half a minute of no response from the HDD, and a suddenly stopped
stream of data from my desktop system I have not noticed any
indication of a possible problem. The backup processes have carried on
without issue after a pause. It is strange. I am still not convinced I
can really trust the enclosure, so I am only using it to work with
non-critical data (if you can treat backups as non-critical...).

Here's the dmesg log of that mistakenly started captive test, and the
aftermath (or lack thereof!):

[Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#11
uas_eh_abort_handler 0 uas-tag 7 inflight: CMD OUT
[Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#11 CDB: Write(16) 8a
00 00 00 00 00 97 88 94 80 00 00 04 00 00 00
[Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#10
uas_eh_abort_handler 0 uas-tag 4 inflight: CMD OUT
[Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#10 CDB: Write(16) 8a
00 00 00 00 00 97 88 90 80 00 00 04 00 00 00
[Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#9
uas_eh_abort_handler 0 uas-tag 3 inflight: CMD OUT
[Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#9 CDB: Write(16) 8a
00 00 00 00 00 97 88 8c 80 00 00 04 00 00 00
[Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#8
uas_eh_abort_handler 0 uas-tag 2 inflight: CMD OUT
[Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#8 CDB: Write(16) 8a
00 00 00 00 00 97 88 88 80 00 00 04 00 00 00
[Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#7
uas_eh_abort_handler 0 uas-tag 8 inflight: CMD OUT
[Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#7 CDB: Write(16) 8a
00 00 00 00 00 97 60 5e c8 00 00 04 00 00 00
[Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#6
uas_eh_abort_handler 0 uas-tag 6 inflight: CMD OUT
[Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#6 CDB: Write(16) 8a
00 00 00 00 00 97 60 5a c8 00 00 04 00 00 00
[Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#5
uas_eh_abort_handler 0 uas-tag 5 inflight: CMD OUT
[Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#5 CDB: Write(16) 8a
00 00 00 00 00 97 60 56 c8 00 00 04 00 00 00
[Sun Nov 16 18:19:15 2025] sd 2:0:0:0: [sdc] tag#13
uas_eh_abort_handler 0 uas-tag 9 inflight: CMD IN
[Sun Nov 16 18:19:15 2025] sd 2:0:0:0: [sdc] tag#13 CDB: Read(16) 88
00 00 00 00 00 7c 44 8d a0 00 00 00 20 00 00
[Sun Nov 16 18:19:17 2025] sd 2:0:0:0: [sdc] tag#14
uas_eh_abort_handler 0 uas-tag 10 inflight: CMD IN
[Sun Nov 16 18:19:17 2025] sd 2:0:0:0: [sdc] tag#14 CDB: Read(16) 88
00 00 00 00 00 7c 44 8b 20 00 00 00 20 00 00
[Sun Nov 16 18:19:30 2025] sd 2:0:0:0: [sdc] tag#4
uas_eh_abort_handler 0 uas-tag 1 inflight: CMD
[Sun Nov 16 18:19:30 2025] sd 2:0:0:0: [sdc] tag#4 CDB: ATA command
pass through(16) 85 06 0c 00 d4 00 00 00 82 00 4f 00 c2 00 b0 00
[Sun Nov 16 18:19:30 2025] scsi host2: uas_eh_device_reset_handler start
[Sun Nov 16 18:19:30 2025] usb 4-1.2: reset SuperSpeed USB device
number 7 using xhci_hcd
[Sun Nov 16 18:19:30 2025] scsi host2: uas_eh_device_reset_handler success

Also - it seems I have managed to get a complete long test (offline,
not a captive one!) thanks to the while-dd-sleep trick. It is clear to
me that it was necessary to use that. I wonder if I should keep that

root@pve:/mnt# smartctl -l xselftest /dev/sdc
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.8.12-16-pve] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining
LifeTime(hours)  LBA_of_first_error
# 1  Extended captive    Interrupted (host reset)      90%       145         -
# 2  Extended offline    Completed without error       00%       118         -
# 3  Short captive       Interrupted (host reset)      40%        43         -
# 4  Extended captive    Interrupted (host reset)      90%        42         -
# 5  Extended offline    Aborted by host               90%        42         -
# 6  Extended captive    Interrupted (host reset)      90%        42         -
# 7  Extended captive    Interrupted (host reset)      90%        42         -
# 8  Extended offline    Aborted by host               90%        42         -
# 9  Extended offline    Aborted by host               90%         1         -

So at least it seems like the 20 TB Toshiba spindle has passed the
initial burn-in.

I should do a btrfs scrub though to make sure the backups that were
paused did not in fact get corrupted right then and there.

I'll report any further findings should anything interesting come up
from this still.
- unfa

On Thu, Nov 13, 2025 at 2:28 AM Chris Murphy <lists@colorremedies.com> wrote:
>
>
>
> On Wed, Nov 12, 2025, at 7:57 AM, Tobiasz Karoń wrote:
> > I have been trying various things, also with other drives.
> > I purchased a 20 TB Toshiba drive and I am currently attempting to do
> > a burn-in before using it for backup.
> >
> > I see issues with the USB exclosure, where it resets ev en with jsut
> > this one drive in it (so one bay is empty). Dmesg says:
> >
> > [Wed Nov 12 11:09:31 2025] sd 2:0:0:0: [sdc] tag#19
> > uas_eh_abort_handler 0 uas-tag 1 inflight: CMD
> > [Wed Nov 12 11:09:31 2025] sd 2:0:0:0: [sdc] tag#19 CDB: ATA command
> > pass through(16) 85 06 0c 00 d4 00 00 00 82 00 4f 00 c2 00 b0 00
> > [Wed Nov 12 11:09:31 2025] scsi host2: uas_eh_device_reset_handler start
> > [Wed Nov 12 11:09:31 2025] usb 4-1.2: reset SuperSpeed USB device
> > number 7 using xhci_hcd
> > [Wed Nov 12 11:09:31 2025] scsi host2: uas_eh_device_reset_handler success
> >
> > [Wed Nov 12 11:11:01 2025] sd 2:0:0:0: [sdc] tag#5
> > uas_eh_abort_handler 0 uas-tag 1 inflight: CMD
> > [Wed Nov 12 11:11:01 2025] sd 2:0:0:0: [sdc] tag#5 CDB: ATA command
> > pass through(16) 85 06 0c 00 d4 00 00 00 82 00 4f 00 c2 00 b0 00
> > [Wed Nov 12 11:11:01 2025] scsi host2: uas_eh_device_reset_handler start
> > [Wed Nov 12 11:11:01 2025] usb 4-1.2: reset SuperSpeed USB device
> > number 7 using xhci_hcd
> > [Wed Nov 12 11:11:01 2025] scsi host2: uas_eh_device_reset_handler success
> >
> > That happens after a minute wheenver I do:
> > smartctl -l long -C /dev/sdc
> >
> > So it seems I can't do a long, captive HDD test through this enclosure.
>
>        -C, --captive
>               [ATA] Runs self-tests in captive mode.  This has no effect with '-t offline' or if the '-t' option is not used.
>
>               WARNING: Tests run in captive mode may busy out the drive for the length of the test.  Only run captive tests on drives without any mounted partitions!
>
>
> Maybe use -t long and omit -C
>
>
> >
> > I have tested blacklisting the uas driver, but the enclosure would
> > just not show up (should I load an alternative driver for it?)
>
> usb_storage
>
>
> > lsusb says this about the 2-bay disk enclosure:
> > Bus 004 Device 007: ID 174c:55aa ASMedia Technology Inc. ASM1051E SATA
> > 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge,
> > ASM1153E SATA 6Gb/s bridge
> >
> > I wonder if there's something that can be done about this enclosure in
> > terms of patching the driver or something?
> > Should I contact a different mailing list?
>
> linux-usb@vger.kernel.org
>
> http://www.linux-usb.org
>
Thank you, I will see if and how I can report this problem there.

>
> > About write cache disabling, I was unable to do it persistently with
> > hdparm, but I found that there's a way to disable write caching
> > persistently with smartclt. It's also possible to disable write cache
> > reordering, which I suppose could be more dangerous fro Btrfs.
>
> I'm not sure about that. Btrfs has no optimizations for write order based on device topology whereas the drive firmware knows things about that topology and can acceptably do out of order writes for performance reasons that are quite safe.

That's most intriguing. I was told in the past (2015?) that btrfs's
filesystem integrity depends on the order of write operations being
preserved. This is why using bcache in writeback mode was not
recommended, as in case of a power failure, the filesystem might fail
to recover. And in my case after about 1 year that has happened. My
system crashed and after rebooting, the btrfs filesystem that I had on
top of bcache in writeback mode, has suffered a collapse and has not
become mountable again. I did loose some recently written files.
I've updated Arch Linux wiki to add a precaution against using bcache
writeback with btrfs after that...
https://wiki.archlinux.org/title/Talk:Bcache#btrfs_on_bcache
I have checked and someone has said that "issues with btrfs+bcache
were fixed 10 years ago". Time flies!
But does that mean the initial assumption about the dangers of btrfs
cached write reordering is wrong?
Or maybe that was a general problem with btrfs's atomic updates, but
is no longer the case?
>
> Granted, the code for these settings is not available fo us to inspect.
>
>
> >
> > Same with short self-test. I wonder if I can use this disk in the
> > enclosure at all if this keeps happening...
>
> I just wouldn't use the captive test.
>
>
>
> --
> Chris Murphy



-- 
- Tobiasz 'unfa' Karoń

www.youtube.com/unfa000

Re: Damaged filesystem - request for support

[Chris Murphy]

On Sun, Nov 16, 2025, at 7:03 PM, Tobiasz Karoń wrote:

> Here's the dmesg log of that mistakenly started captive test, and the
> aftermath (or lack thereof!):
>
> [Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#11
> uas_eh_abort_handler 0 uas-tag 7 inflight: CMD OUT
> [Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#11 CDB: Write(16) 8a
> 00 00 00 00 00 97 88 94 80 00 00 04 00 00 00
> [Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#10
> uas_eh_abort_handler 0 uas-tag 4 inflight: CMD OUT
> [Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#10 CDB: Write(16) 8a
> 00 00 00 00 00 97 88 90 80 00 00 04 00 00 00
> [Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#9
> uas_eh_abort_handler 0 uas-tag 3 inflight: CMD OUT
> [Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#9 CDB: Write(16) 8a
> 00 00 00 00 00 97 88 8c 80 00 00 04 00 00 00
> [Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#8
> uas_eh_abort_handler 0 uas-tag 2 inflight: CMD OUT
> [Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#8 CDB: Write(16) 8a
> 00 00 00 00 00 97 88 88 80 00 00 04 00 00 00
> [Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#7
> uas_eh_abort_handler 0 uas-tag 8 inflight: CMD OUT
> [Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#7 CDB: Write(16) 8a
> 00 00 00 00 00 97 60 5e c8 00 00 04 00 00 00
> [Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#6
> uas_eh_abort_handler 0 uas-tag 6 inflight: CMD OUT
> [Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#6 CDB: Write(16) 8a
> 00 00 00 00 00 97 60 5a c8 00 00 04 00 00 00
> [Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#5
> uas_eh_abort_handler 0 uas-tag 5 inflight: CMD OUT
> [Sun Nov 16 18:19:00 2025] sd 2:0:0:0: [sdc] tag#5 CDB: Write(16) 8a
> 00 00 00 00 00 97 60 56 c8 00 00 04 00 00 00
> [Sun Nov 16 18:19:15 2025] sd 2:0:0:0: [sdc] tag#13
> uas_eh_abort_handler 0 uas-tag 9 inflight: CMD IN
> [Sun Nov 16 18:19:15 2025] sd 2:0:0:0: [sdc] tag#13 CDB: Read(16) 88
> 00 00 00 00 00 7c 44 8d a0 00 00 00 20 00 00
> [Sun Nov 16 18:19:17 2025] sd 2:0:0:0: [sdc] tag#14
> uas_eh_abort_handler 0 uas-tag 10 inflight: CMD IN
> [Sun Nov 16 18:19:17 2025] sd 2:0:0:0: [sdc] tag#14 CDB: Read(16) 88
> 00 00 00 00 00 7c 44 8b 20 00 00 00 20 00 00
> [Sun Nov 16 18:19:30 2025] sd 2:0:0:0: [sdc] tag#4
> uas_eh_abort_handler 0 uas-tag 1 inflight: CMD
> [Sun Nov 16 18:19:30 2025] sd 2:0:0:0: [sdc] tag#4 CDB: ATA command
> pass through(16) 85 06 0c 00 d4 00 00 00 82 00 4f 00 c2 00 b0 00
> [Sun Nov 16 18:19:30 2025] scsi host2: uas_eh_device_reset_handler start
> [Sun Nov 16 18:19:30 2025] usb 4-1.2: reset SuperSpeed USB device
> number 7 using xhci_hcd
> [Sun Nov 16 18:19:30 2025] scsi host2: uas_eh_device_reset_handler success
>
> Also - it seems I have managed to get a complete long test (offline,
> not a captive one!) thanks to the while-dd-sleep trick. It is clear to
> me that it was necessary to use that. I wonder if I should keep that

Sounds like a bad hack, as in, you can't rely on this and shouldn't have to. There's a bug somewhere. I'd retest with mainline and then go shop the problem on the linux-usb list and see if they have a quirk to use or can incorporate a better fix into the kernel. This is not an area of expertise of mine - so it's better to defer to usb upstream.

What I can tell you having experienced similar USB problems with SATA-USB enclosures, all problems were resolved by using a good quality (not expensive, but properly spec'd) USB hub. For whatever reason directly plugging in USB devices into the port on a computer - which itself is a kind of hub - was inadequate.



>> I'm not sure about that. Btrfs has no optimizations for write order based on device topology whereas the drive firmware knows things about that topology and can acceptably do out of order writes for performance reasons that are quite safe.
>
> That's most intriguing. I was told in the past (2015?) that btrfs's
> filesystem integrity depends on the order of write operations being
> preserved. 

There's different kinds of ordering.

There's many separate read and write commands when writing out data from a variety of sources. The vast majority of these can be reordered by the drive firmware for topology reasons. But when Btrfs issues a FLUSH or FUA to separate metadata (b-trees) writes from super block writes - that *must* be honored. 

Btrfs doesn't care if the drive reorders writes within the bracketing provided by flush/fua.


> But does that mean the initial assumption about the dangers of btrfs
> cached write reordering is wrong?
> Or maybe that was a general problem with btrfs's atomic updates, but
> is no longer the case?

I'm not super familiar with bcache writeback mode but if metadata and super block writes only persist on the SSD, and do not pass through to the HDD, then it is not possible to separate the SSD and HDD - the HDD alone is not in a consistent state. So following a crash in writeback mode, it's a requirement that the combined block device (SSD+HDD) are assembled and that bcache device is what gets mounted.

I think where people went astray was not realizing there are critical writes only on the SSD and then decided to mount or repair the btrfs on the HDD alone. But like I said, not super familiar with that issue.



-- 
Chris Murphy