Re: How to fix "BTRFS error (device dm-3): error writing primary super block to device 1"?

[Qu Wenruo <wqu@suse.com>]

在 2025/4/12 01:18, Kai Stian Olstad 写道:
> Kubuntu 24.04
> Kernel 6.8.0-57-generic
> 
> 2 day ago I got a sector error on one of the BTRFS disk
> 
> $ journalctl -k -S 2025-04-09 | grep -A 20 mpt3sas_cm0
> Apr 09 03:16:26 cb kernel: mpt3sas_cm0: log_info(0x31080000): 
> originator(PL), code(0x08), sub_code(0x0000)
> Apr 09 03:16:26 cb kernel: mpt3sas_cm0: log_info(0x31080000): 
> originator(PL), code(0x08), sub_code(0x0000)
> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 FAILED Result: 
> hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=6s
> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Sense Key : 
> Illegal Request [current]
> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Add. Sense: 
> Logical block address out of range
> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 CDB: Write(16) 8a 
> 08 00 00 00 00 00 00 10 80 00 00 00 08 00 00
> Apr 09 03:16:26 cb kernel: critical target error, dev sdd, sector 4224 
> op 0x1:(WRITE) flags 0x23800 phys_seg 1 prio class 0

This error is completely from the lower layer (the block device).

Btrfs nor the LUKS upon the disk can do anything to it.

Thanks,
Qu

> Apr 09 03:16:26 cb kernel: BTRFS warning (device dm-3): lost page write 
> due to IO error on /dev/mapper/cdisk20 (-121)
> Apr 09 03:16:26 cb kernel: BTRFS error (device dm-3): bdev /dev/mapper/ 
> cdisk20 errs: wr 2, rd 1, flush 0, corrupt 0, gen 0
> Apr 09 03:16:26 cb kernel: BTRFS error (device dm-3): error writing 
> primary super block to device 1
> Apr 09 03:17:02 cb kernel: BTRFS error (device dm-3): error writing 
> primary super block to device 1
> Apr 09 03:17:02 cb kernel: BTRFS error (device dm-3): error writing 
> primary super block to device 1
> Apr 09 03:17:02 cb kernel: BTRFS error (device dm-3): error writing 
> primary super block to device 1
> Apr 09 03:17:02 cb kernel: BTRFS error (device dm-3): error writing 
> primary super block to device 1
> Apr 09 03:17:08 cb kernel: BTRFS error (device dm-3): error writing 
> primary super block to device 1
> Apr 09 03:17:10 cb kernel: BTRFS error (device dm-3): error writing 
> primary super block to device 1
> Apr 09 03:17:10 cb kernel: BTRFS error (device dm-3): error writing 
> primary super block to device 1
> Apr 09 03:17:27 cb kernel: BTRFS error (device dm-3): error writing 
> primary super block to device 1
> Apr 09 03:17:58 cb kernel: BTRFS error (device dm-3): error writing 
> primary super block to device 1
> 
> And after that I constantly get "error writing primary super block to 
> device
> 1". Tried to run "btrfs scrub status /data" but that didn't help
> 
> $ sudo btrfs scrub status /data
> UUID:             6554e692-6c1c-4a69-8ff8-cb5615fb9200
> Scrub started:    Thu Apr 10 19:40:56 2025
> Status:           finished
> Duration:         18:54:46
> Total to scrub:   25.08TiB
> Rate:             386.23MiB/s (some device limits set)
> Error summary:    no errors found
> 
> /dev/sdd is /dev/mapper/cdisk20, it's running LUKS on top off sdd.
> 
> Does anyone know how to fix this?
> 
> 
> Some output that might be useful:
> 
> $ sudo btrfs filesystem usage /data
> Overall:
>      Device size:                  29.11TiB
>      Device allocated:             25.09TiB
>      Device unallocated:            4.01TiB
>      Device missing:                  0.00B
>      Device slack:                    0.00B
>      Used:                         25.08TiB
>      Free (estimated):              2.01TiB      (min: 2.01TiB)
>      Free (statfs, df):             2.01TiB
>      Data ratio:                       2.00
>      Metadata ratio:                   2.00
>      Global reserve:              512.00MiB      (used: 0.00B)
>      Multiple profiles:                  no
> 
> Data,RAID1: Size:12.52TiB, Used:12.52TiB (99.96%)
>     /dev/mapper/cdisk20     4.45TiB
>     /dev/mapper/cdisk21     4.45TiB
>     /dev/mapper/cdisk22     8.08TiB
>     /dev/mapper/cdisk23     8.08TiB
> 
> Metadata,RAID1: Size:23.03GiB, Used:19.93GiB (86.56%)
>     /dev/mapper/cdisk20     8.03GiB
>     /dev/mapper/cdisk21     8.03GiB
>     /dev/mapper/cdisk22    15.00GiB
>     /dev/mapper/cdisk23    15.00GiB
> 
> System,RAID1: Size:64.00MiB, Used:1.80MiB (2.81%)
>     /dev/mapper/cdisk22    64.00MiB
>     /dev/mapper/cdisk23    64.00MiB
> 
> Unallocated:
>     /dev/mapper/cdisk20     1.00TiB
>     /dev/mapper/cdisk21     1.00TiB
>     /dev/mapper/cdisk22     1.00TiB
>     /dev/mapper/cdisk23     1.00TiB
> 
> 
> $ sudo btrfs device stats /data | grep -v " 0$"
> [/dev/mapper/cdisk20].write_io_errs    2
> [/dev/mapper/cdisk20].read_io_errs     1
> [/dev/mapper/cdisk21].write_io_errs    1
> [/dev/mapper/cdisk21].read_io_errs     1
> 
>