How to fix "BTRFS error (device dm-3): error writing primary super block to device 1"?

How to fix "BTRFS error (device dm-3): error writing primary super block to device 1"?

[Kai Stian Olstad]

Kubuntu 24.04
Kernel 6.8.0-57-generic

2 day ago I got a sector error on one of the BTRFS disk

$ journalctl -k -S 2025-04-09 | grep -A 20 mpt3sas_cm0
Apr 09 03:16:26 cb kernel: mpt3sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
Apr 09 03:16:26 cb kernel: mpt3sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=6s
Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Sense Key : Illegal Request [current]
Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Add. Sense: Logical block address out of range
Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 CDB: Write(16) 8a 08 00 00 00 00 00 00 10 80 00 00 00 08 00 00
Apr 09 03:16:26 cb kernel: critical target error, dev sdd, sector 4224 op 0x1:(WRITE) flags 0x23800 phys_seg 1 prio class 0
Apr 09 03:16:26 cb kernel: BTRFS warning (device dm-3): lost page write due to IO error on /dev/mapper/cdisk20 (-121)
Apr 09 03:16:26 cb kernel: BTRFS error (device dm-3): bdev /dev/mapper/cdisk20 errs: wr 2, rd 1, flush 0, corrupt 0, gen 0
Apr 09 03:16:26 cb kernel: BTRFS error (device dm-3): error writing primary super block to device 1
Apr 09 03:17:02 cb kernel: BTRFS error (device dm-3): error writing primary super block to device 1
Apr 09 03:17:02 cb kernel: BTRFS error (device dm-3): error writing primary super block to device 1
Apr 09 03:17:02 cb kernel: BTRFS error (device dm-3): error writing primary super block to device 1
Apr 09 03:17:02 cb kernel: BTRFS error (device dm-3): error writing primary super block to device 1
Apr 09 03:17:08 cb kernel: BTRFS error (device dm-3): error writing primary super block to device 1
Apr 09 03:17:10 cb kernel: BTRFS error (device dm-3): error writing primary super block to device 1
Apr 09 03:17:10 cb kernel: BTRFS error (device dm-3): error writing primary super block to device 1
Apr 09 03:17:27 cb kernel: BTRFS error (device dm-3): error writing primary super block to device 1
Apr 09 03:17:58 cb kernel: BTRFS error (device dm-3): error writing primary super block to device 1

And after that I constantly get "error writing primary super block to device
1". Tried to run "btrfs scrub status /data" but that didn't help

$ sudo btrfs scrub status /data
UUID:             6554e692-6c1c-4a69-8ff8-cb5615fb9200
Scrub started:    Thu Apr 10 19:40:56 2025
Status:           finished
Duration:         18:54:46
Total to scrub:   25.08TiB
Rate:             386.23MiB/s (some device limits set)
Error summary:    no errors found

/dev/sdd is /dev/mapper/cdisk20, it's running LUKS on top off sdd.

Does anyone know how to fix this?


Some output that might be useful:

$ sudo btrfs filesystem usage /data
Overall:
     Device size:                  29.11TiB
     Device allocated:             25.09TiB
     Device unallocated:            4.01TiB
     Device missing:                  0.00B
     Device slack:                    0.00B
     Used:                         25.08TiB
     Free (estimated):              2.01TiB      (min: 2.01TiB)
     Free (statfs, df):             2.01TiB
     Data ratio:                       2.00
     Metadata ratio:                   2.00
     Global reserve:              512.00MiB      (used: 0.00B)
     Multiple profiles:                  no

Data,RAID1: Size:12.52TiB, Used:12.52TiB (99.96%)
    /dev/mapper/cdisk20     4.45TiB
    /dev/mapper/cdisk21     4.45TiB
    /dev/mapper/cdisk22     8.08TiB
    /dev/mapper/cdisk23     8.08TiB

Metadata,RAID1: Size:23.03GiB, Used:19.93GiB (86.56%)
    /dev/mapper/cdisk20     8.03GiB
    /dev/mapper/cdisk21     8.03GiB
    /dev/mapper/cdisk22    15.00GiB
    /dev/mapper/cdisk23    15.00GiB

System,RAID1: Size:64.00MiB, Used:1.80MiB (2.81%)
    /dev/mapper/cdisk22    64.00MiB
    /dev/mapper/cdisk23    64.00MiB

Unallocated:
    /dev/mapper/cdisk20     1.00TiB
    /dev/mapper/cdisk21     1.00TiB
    /dev/mapper/cdisk22     1.00TiB
    /dev/mapper/cdisk23     1.00TiB


$ sudo btrfs device stats /data | grep -v " 0$"
[/dev/mapper/cdisk20].write_io_errs    2
[/dev/mapper/cdisk20].read_io_errs     1
[/dev/mapper/cdisk21].write_io_errs    1
[/dev/mapper/cdisk21].read_io_errs     1


-- 
Kai Stian Olstad

Re: How to fix "BTRFS error (device dm-3): error writing primary super block to device 1"?

[Qu Wenruo]

在 2025/4/12 01:18, Kai Stian Olstad 写道:
> Kubuntu 24.04
> Kernel 6.8.0-57-generic
> 
> 2 day ago I got a sector error on one of the BTRFS disk
> 
> $ journalctl -k -S 2025-04-09 | grep -A 20 mpt3sas_cm0
> Apr 09 03:16:26 cb kernel: mpt3sas_cm0: log_info(0x31080000): 
> originator(PL), code(0x08), sub_code(0x0000)
> Apr 09 03:16:26 cb kernel: mpt3sas_cm0: log_info(0x31080000): 
> originator(PL), code(0x08), sub_code(0x0000)
> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 FAILED Result: 
> hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=6s
> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Sense Key : 
> Illegal Request [current]
> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Add. Sense: 
> Logical block address out of range
> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 CDB: Write(16) 8a 
> 08 00 00 00 00 00 00 10 80 00 00 00 08 00 00
> Apr 09 03:16:26 cb kernel: critical target error, dev sdd, sector 4224 
> op 0x1:(WRITE) flags 0x23800 phys_seg 1 prio class 0

This error is completely from the lower layer (the block device).

Btrfs nor the LUKS upon the disk can do anything to it.

Thanks,
Qu

> Apr 09 03:16:26 cb kernel: BTRFS warning (device dm-3): lost page write 
> due to IO error on /dev/mapper/cdisk20 (-121)
> Apr 09 03:16:26 cb kernel: BTRFS error (device dm-3): bdev /dev/mapper/ 
> cdisk20 errs: wr 2, rd 1, flush 0, corrupt 0, gen 0
> Apr 09 03:16:26 cb kernel: BTRFS error (device dm-3): error writing 
> primary super block to device 1
> Apr 09 03:17:02 cb kernel: BTRFS error (device dm-3): error writing 
> primary super block to device 1
> Apr 09 03:17:02 cb kernel: BTRFS error (device dm-3): error writing 
> primary super block to device 1
> Apr 09 03:17:02 cb kernel: BTRFS error (device dm-3): error writing 
> primary super block to device 1
> Apr 09 03:17:02 cb kernel: BTRFS error (device dm-3): error writing 
> primary super block to device 1
> Apr 09 03:17:08 cb kernel: BTRFS error (device dm-3): error writing 
> primary super block to device 1
> Apr 09 03:17:10 cb kernel: BTRFS error (device dm-3): error writing 
> primary super block to device 1
> Apr 09 03:17:10 cb kernel: BTRFS error (device dm-3): error writing 
> primary super block to device 1
> Apr 09 03:17:27 cb kernel: BTRFS error (device dm-3): error writing 
> primary super block to device 1
> Apr 09 03:17:58 cb kernel: BTRFS error (device dm-3): error writing 
> primary super block to device 1
> 
> And after that I constantly get "error writing primary super block to 
> device
> 1". Tried to run "btrfs scrub status /data" but that didn't help
> 
> $ sudo btrfs scrub status /data
> UUID:             6554e692-6c1c-4a69-8ff8-cb5615fb9200
> Scrub started:    Thu Apr 10 19:40:56 2025
> Status:           finished
> Duration:         18:54:46
> Total to scrub:   25.08TiB
> Rate:             386.23MiB/s (some device limits set)
> Error summary:    no errors found
> 
> /dev/sdd is /dev/mapper/cdisk20, it's running LUKS on top off sdd.
> 
> Does anyone know how to fix this?
> 
> 
> Some output that might be useful:
> 
> $ sudo btrfs filesystem usage /data
> Overall:
>      Device size:                  29.11TiB
>      Device allocated:             25.09TiB
>      Device unallocated:            4.01TiB
>      Device missing:                  0.00B
>      Device slack:                    0.00B
>      Used:                         25.08TiB
>      Free (estimated):              2.01TiB      (min: 2.01TiB)
>      Free (statfs, df):             2.01TiB
>      Data ratio:                       2.00
>      Metadata ratio:                   2.00
>      Global reserve:              512.00MiB      (used: 0.00B)
>      Multiple profiles:                  no
> 
> Data,RAID1: Size:12.52TiB, Used:12.52TiB (99.96%)
>     /dev/mapper/cdisk20     4.45TiB
>     /dev/mapper/cdisk21     4.45TiB
>     /dev/mapper/cdisk22     8.08TiB
>     /dev/mapper/cdisk23     8.08TiB
> 
> Metadata,RAID1: Size:23.03GiB, Used:19.93GiB (86.56%)
>     /dev/mapper/cdisk20     8.03GiB
>     /dev/mapper/cdisk21     8.03GiB
>     /dev/mapper/cdisk22    15.00GiB
>     /dev/mapper/cdisk23    15.00GiB
> 
> System,RAID1: Size:64.00MiB, Used:1.80MiB (2.81%)
>     /dev/mapper/cdisk22    64.00MiB
>     /dev/mapper/cdisk23    64.00MiB
> 
> Unallocated:
>     /dev/mapper/cdisk20     1.00TiB
>     /dev/mapper/cdisk21     1.00TiB
>     /dev/mapper/cdisk22     1.00TiB
>     /dev/mapper/cdisk23     1.00TiB
> 
> 
> $ sudo btrfs device stats /data | grep -v " 0$"
> [/dev/mapper/cdisk20].write_io_errs    2
> [/dev/mapper/cdisk20].read_io_errs     1
> [/dev/mapper/cdisk21].write_io_errs    1
> [/dev/mapper/cdisk21].read_io_errs     1
> 
> 

Re: How to fix "BTRFS error (device dm-3): error writing primary super block to device 1"?

[Kai Stian Olstad]

On 12.04.2025 00:10, Qu Wenruo wrote:
> 在 2025/4/12 01:18, Kai Stian Olstad 写道:
>> Kubuntu 24.04
>> Kernel 6.8.0-57-generic
>> 
>> 2 day ago I got a sector error on one of the BTRFS disk
>> 
>> $ journalctl -k -S 2025-04-09 | grep -A 20 mpt3sas_cm0
>> Apr 09 03:16:26 cb kernel: mpt3sas_cm0: log_info(0x31080000): 
>> originator(PL), code(0x08), sub_code(0x0000)
>> Apr 09 03:16:26 cb kernel: mpt3sas_cm0: log_info(0x31080000): 
>> originator(PL), code(0x08), sub_code(0x0000)
>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 FAILED Result: 
>> hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=6s
>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Sense Key : 
>> Illegal Request [current]
>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Add. Sense: 
>> Logical block address out of range
>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 CDB: Write(16) 
>> 8a 08 00 00 00 00 00 00 10 80 00 00 00 08 00 00
>> Apr 09 03:16:26 cb kernel: critical target error, dev sdd, sector 4224 
>> op 0x1:(WRITE) flags 0x23800 phys_seg 1 prio class 0
> 
> This error is completely from the lower layer (the block device).
> 
> Btrfs nor the LUKS upon the disk can do anything to it.

Thank you for the response.

This disk support scterc

$ sudo smartctl -l scterc /dev/sdd
SCT Error Recovery Control:
            Read:     70 (7.0 seconds)
           Write:     70 (7.0 seconds)

Doesn't that mean that the disk gives up after 7 seconds, and then the 
sector i mapped to a spare.
So if Btrfs does a write to the sector again it will be written to the 
spare?

I've experienced numerous sector errors throughout the years with mdadm 
and they have been fixed with a check.
Also a few with Btrfs I think, but they have been fixed automatically.

So why not this time?
To me this looks like an ordinary faulty sector that can be "fixed" with 
a write?

-- 
Kai Stian Olstad

Re: How to fix "BTRFS error (device dm-3): error writing primary super block to device 1"?

[Qu Wenruo]

在 2025/4/12 09:59, Kai Stian Olstad 写道:
> On 12.04.2025 00:10, Qu Wenruo wrote:
>> 在 2025/4/12 01:18, Kai Stian Olstad 写道:
>>> Kubuntu 24.04
>>> Kernel 6.8.0-57-generic
>>>
>>> 2 day ago I got a sector error on one of the BTRFS disk
>>>
>>> $ journalctl -k -S 2025-04-09 | grep -A 20 mpt3sas_cm0
>>> Apr 09 03:16:26 cb kernel: mpt3sas_cm0: log_info(0x31080000):
>>> originator(PL), code(0x08), sub_code(0x0000)
>>> Apr 09 03:16:26 cb kernel: mpt3sas_cm0: log_info(0x31080000):
>>> originator(PL), code(0x08), sub_code(0x0000)
>>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 FAILED Result:
>>> hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=6s
>>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Sense Key :
>>> Illegal Request [current]
>>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Add. Sense:
>>> Logical block address out of range
>>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 CDB: Write(16)
>>> 8a 08 00 00 00 00 00 00 10 80 00 00 00 08 00 00
>>> Apr 09 03:16:26 cb kernel: critical target error, dev sdd, sector
>>> 4224 op 0x1:(WRITE) flags 0x23800 phys_seg 1 prio class 0
>>
>> This error is completely from the lower layer (the block device).
>>
>> Btrfs nor the LUKS upon the disk can do anything to it.
>
> Thank you for the response.
>
> This disk support scterc
>
> $ sudo smartctl -l scterc /dev/sdd
> SCT Error Recovery Control:
>             Read:     70 (7.0 seconds)
>            Write:     70 (7.0 seconds)
>
> Doesn't that mean that the disk gives up after 7 seconds, and then the
> sector i mapped to a spare.
> So if Btrfs does a write to the sector again it will be written to the
> spare?
>
> I've experienced numerous sector errors throughout the years with mdadm
> and they have been fixed with a check.
> Also a few with Btrfs I think, but they have been fixed automatically.

Whatever the feature is, it's block device driver's behavior.

Btrfs only errors out because the disk reported the write failed.

For the detailed reason you should check these lines:

 > Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 FAILED Result:
hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=6s
 > Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Sense Key :
Illegal Request [current]
 > Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Add. Sense:
Logical block address out of range

>
> So why not this time?
> To me this looks like an ordinary faulty sector that can be "fixed" with
> a write?
>
I'm not sure what ever the "SCT Error recovery control" feature is, but
if it is designed to re-map a write, it should not return -EIO for the
initial write failure, but OK as long as eventually the write succeeded.

It should not require any upper layer to do any extra work.

But since the write eventually failed, there is nothing upper layer can
do, unless the dm or fs layer has some extra recovery mechanism.

Thanks,
Qu

Re: How to fix "BTRFS error (device dm-3): error writing primary super block to device 1"?

[Kai Stian Olstad]

On 12.04.2025 02:43, Qu Wenruo wrote:
> 在 2025/4/12 09:59, Kai Stian Olstad 写道:
>> On 12.04.2025 00:10, Qu Wenruo wrote:
>>> 在 2025/4/12 01:18, Kai Stian Olstad 写道:
>>>> Kubuntu 24.04
>>>> Kernel 6.8.0-57-generic
>>>> 
>>>> 2 day ago I got a sector error on one of the BTRFS disk
>>>> 
>>>> $ journalctl -k -S 2025-04-09 | grep -A 20 mpt3sas_cm0
>>>> Apr 09 03:16:26 cb kernel: mpt3sas_cm0: log_info(0x31080000):
>>>> originator(PL), code(0x08), sub_code(0x0000)
>>>> Apr 09 03:16:26 cb kernel: mpt3sas_cm0: log_info(0x31080000):
>>>> originator(PL), code(0x08), sub_code(0x0000)
>>>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 FAILED Result:
>>>> hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=6s
>>>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Sense Key :
>>>> Illegal Request [current]
>>>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Add. Sense:
>>>> Logical block address out of range
>>>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 CDB: Write(16)
>>>> 8a 08 00 00 00 00 00 00 10 80 00 00 00 08 00 00
>>>> Apr 09 03:16:26 cb kernel: critical target error, dev sdd, sector
>>>> 4224 op 0x1:(WRITE) flags 0x23800 phys_seg 1 prio class 0
>>> 
>>> This error is completely from the lower layer (the block device).
>>> 
>>> Btrfs nor the LUKS upon the disk can do anything to it.
>> 
>> Thank you for the response.
>> 
>> This disk support scterc
>> 
>> $ sudo smartctl -l scterc /dev/sdd
>> SCT Error Recovery Control:
>>             Read:     70 (7.0 seconds)
>>            Write:     70 (7.0 seconds)
>> 
>> Doesn't that mean that the disk gives up after 7 seconds, and then the
>> sector i mapped to a spare.
>> So if Btrfs does a write to the sector again it will be written to the
>> spare?
>> 
>> I've experienced numerous sector errors throughout the years with 
>> mdadm
>> and they have been fixed with a check.
>> Also a few with Btrfs I think, but they have been fixed automatically.
> 
> Whatever the feature is, it's block device driver's behavior.
> 
> Btrfs only errors out because the disk reported the write failed.
> 
> For the detailed reason you should check these lines:
> 
>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 FAILED Result:
> hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=6s
>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Sense Key :
> Illegal Request [current]
>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Add. Sense:
> Logical block address out of range

I'll check them but this is what I usually sees when a disk have a 
sector error.


>> So why not this time?
>> To me this looks like an ordinary faulty sector that can be "fixed" 
>> with
>> a write?
>> 
> I'm not sure what ever the "SCT Error recovery control" feature is, but
> if it is designed to re-map a write, it should not return -EIO for the
> initial write failure, but OK as long as eventually the write 
> succeeded.
> 
> It should not require any upper layer to do any extra work.
> 
> But since the write eventually failed, there is nothing upper layer can
> do, unless the dm or fs layer has some extra recovery mechanism.

Now I'm confused, I'm running RAID1 an only one disk has/had 1 sector 
failure.
Shouldn't Btrfs manage to to write this data, it should exist on one of 
the other drives because of RAID1?
And shouldn't a scrub fix it?

Since I don't get any other error from the block layer, the sector is 
either fixed/remapped or Btrfs doesn't try to fix the data in scrub?
If it had tried and the sector is still bad I should get sector error 
from the disk multiple time.
But this error is only in the logs that one time.

What is the purpose of Btrfs RAID1 if it doesn't try to fix this, by 
writing the data again from the good copy?

-- 
Kai Stian Olstad

Re: How to fix "BTRFS error (device dm-3): error writing primary super block to device 1"?

[Qu Wenruo]

在 2025/4/12 10:32, Kai Stian Olstad 写道:
> On 12.04.2025 02:43, Qu Wenruo wrote:
>> 在 2025/4/12 09:59, Kai Stian Olstad 写道:
>>> On 12.04.2025 00:10, Qu Wenruo wrote:
>>>> 在 2025/4/12 01:18, Kai Stian Olstad 写道:
>>>>> Kubuntu 24.04
>>>>> Kernel 6.8.0-57-generic
>>>>>
>>>>> 2 day ago I got a sector error on one of the BTRFS disk
>>>>>
>>>>> $ journalctl -k -S 2025-04-09 | grep -A 20 mpt3sas_cm0
>>>>> Apr 09 03:16:26 cb kernel: mpt3sas_cm0: log_info(0x31080000):
>>>>> originator(PL), code(0x08), sub_code(0x0000)
>>>>> Apr 09 03:16:26 cb kernel: mpt3sas_cm0: log_info(0x31080000):
>>>>> originator(PL), code(0x08), sub_code(0x0000)
>>>>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 FAILED Result:
>>>>> hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=6s
>>>>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Sense Key :
>>>>> Illegal Request [current]
>>>>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Add. Sense:
>>>>> Logical block address out of range
>>>>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 CDB: Write(16)
>>>>> 8a 08 00 00 00 00 00 00 10 80 00 00 00 08 00 00
>>>>> Apr 09 03:16:26 cb kernel: critical target error, dev sdd, sector
>>>>> 4224 op 0x1:(WRITE) flags 0x23800 phys_seg 1 prio class 0
>>>>
>>>> This error is completely from the lower layer (the block device).
>>>>
>>>> Btrfs nor the LUKS upon the disk can do anything to it.
>>>
>>> Thank you for the response.
>>>
>>> This disk support scterc
>>>
>>> $ sudo smartctl -l scterc /dev/sdd
>>> SCT Error Recovery Control:
>>>             Read:     70 (7.0 seconds)
>>>            Write:     70 (7.0 seconds)
>>>
>>> Doesn't that mean that the disk gives up after 7 seconds, and then the
>>> sector i mapped to a spare.
>>> So if Btrfs does a write to the sector again it will be written to the
>>> spare?
>>>
>>> I've experienced numerous sector errors throughout the years with mdadm
>>> and they have been fixed with a check.
>>> Also a few with Btrfs I think, but they have been fixed automatically.
>>
>> Whatever the feature is, it's block device driver's behavior.
>>
>> Btrfs only errors out because the disk reported the write failed.
>>
>> For the detailed reason you should check these lines:
>>
>>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 FAILED Result:
>> hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=6s
>>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Sense Key :
>> Illegal Request [current]
>>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Add. Sense:
>> Logical block address out of range
>
> I'll check them but this is what I usually sees when a disk have a
> sector error.
>
>
>>> So why not this time?
>>> To me this looks like an ordinary faulty sector that can be "fixed" with
>>> a write?
>>>
>> I'm not sure what ever the "SCT Error recovery control" feature is, but
>> if it is designed to re-map a write, it should not return -EIO for the
>> initial write failure, but OK as long as eventually the write succeeded.
>>
>> It should not require any upper layer to do any extra work.
>>
>> But since the write eventually failed, there is nothing upper layer can
>> do, unless the dm or fs layer has some extra recovery mechanism.
>
> Now I'm confused, I'm running RAID1 an only one disk has/had 1 sector
> failure.
> Shouldn't Btrfs manage to to write this data, it should exist on one of
> the other drives because of RAID1?
> And shouldn't a scrub fix it?

Sorry, I finally got your concern that, it's not about the initial write
failure, but the future errors messages.

It turns out to be a bug in the older kernels, that after one super
block write back error, the folio keeps its error flag without clearing
it up, thus it always shows an error message.

And since it's RAID1, btrfs continues the fs (thus your fs is still
running, not flipping into read-only).

Scrub won't solve it because there is nothing to resolve, everything is
fine, except the false warning messages.


In upstream it's fixed by a rework patch, upstream commit bc00965dbff7
("btrfs: count super block write errors in device instead of tracking
folio error state") fixes the bug by going a completely different path
counting the super block write back errors.

Unfortunately that commit is only in v6.10, and since it's not
explicitly marked as a bug fix (even it indeed fixes a hidden bug), it's
not backported to any older kernel. (BTW, 6.8 kernel is already EOL)

Please update to v6.12 or newer LTS kernels.

Or just unmount and remount the fs and pray no more super block
writeback errors happen again...

Thanks,
Qu

>
> Since I don't get any other error from the block layer, the sector is
> either fixed/remapped or Btrfs doesn't try to fix the data in scrub?
> If it had tried and the sector is still bad I should get sector error
> from the disk multiple time.
> But this error is only in the logs that one time.
>
> What is the purpose of Btrfs RAID1 if it doesn't try to fix this, by
> writing the data again from the good copy?
>

Re: How to fix "BTRFS error (device dm-3): error writing primary super block to device 1"?

[Kai Stian Olstad]

On 12.04.2025 05:15, Qu Wenruo wrote:
> 在 2025/4/12 10:32, Kai Stian Olstad 写道:
>> On 12.04.2025 02:43, Qu Wenruo wrote:
>>> 在 2025/4/12 09:59, Kai Stian Olstad 写道:
>>>> On 12.04.2025 00:10, Qu Wenruo wrote:
>>>>> 在 2025/4/12 01:18, Kai Stian Olstad 写道:
>>>>>> Kubuntu 24.04
>>>>>> Kernel 6.8.0-57-generic
>>>>>> 
>>>>>> 2 day ago I got a sector error on one of the BTRFS disk
>>>>>> 
>>>>>> $ journalctl -k -S 2025-04-09 | grep -A 20 mpt3sas_cm0
>>>>>> Apr 09 03:16:26 cb kernel: mpt3sas_cm0: log_info(0x31080000):
>>>>>> originator(PL), code(0x08), sub_code(0x0000)
>>>>>> Apr 09 03:16:26 cb kernel: mpt3sas_cm0: log_info(0x31080000):
>>>>>> originator(PL), code(0x08), sub_code(0x0000)
>>>>>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 FAILED 
>>>>>> Result:
>>>>>> hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=6s
>>>>>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Sense Key :
>>>>>> Illegal Request [current]
>>>>>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Add. Sense:
>>>>>> Logical block address out of range
>>>>>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 CDB: 
>>>>>> Write(16)
>>>>>> 8a 08 00 00 00 00 00 00 10 80 00 00 00 08 00 00
>>>>>> Apr 09 03:16:26 cb kernel: critical target error, dev sdd, sector
>>>>>> 4224 op 0x1:(WRITE) flags 0x23800 phys_seg 1 prio class 0
>>>>> 
>>>>> This error is completely from the lower layer (the block device).
>>>>> 
>>>>> Btrfs nor the LUKS upon the disk can do anything to it.
>>>> 
>>>> Thank you for the response.
>>>> 
>>>> This disk support scterc
>>>> 
>>>> $ sudo smartctl -l scterc /dev/sdd
>>>> SCT Error Recovery Control:
>>>>             Read:     70 (7.0 seconds)
>>>>            Write:     70 (7.0 seconds)
>>>> 
>>>> Doesn't that mean that the disk gives up after 7 seconds, and then 
>>>> the
>>>> sector i mapped to a spare.
>>>> So if Btrfs does a write to the sector again it will be written to 
>>>> the
>>>> spare?
>>>> 
>>>> I've experienced numerous sector errors throughout the years with 
>>>> mdadm
>>>> and they have been fixed with a check.
>>>> Also a few with Btrfs I think, but they have been fixed 
>>>> automatically.
>>> 
>>> Whatever the feature is, it's block device driver's behavior.
>>> 
>>> Btrfs only errors out because the disk reported the write failed.
>>> 
>>> For the detailed reason you should check these lines:
>>> 
>>>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 FAILED Result:
>>> hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=6s
>>>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Sense Key :
>>> Illegal Request [current]
>>>> Apr 09 03:16:26 cb kernel: sd 4:0:1:0: [sdd] tag#5552 Add. Sense:
>>> Logical block address out of range
>> 
>> I'll check them but this is what I usually sees when a disk have a
>> sector error.
>> 
>> 
>>>> So why not this time?
>>>> To me this looks like an ordinary faulty sector that can be "fixed" 
>>>> with
>>>> a write?
>>>> 
>>> I'm not sure what ever the "SCT Error recovery control" feature is, 
>>> but
>>> if it is designed to re-map a write, it should not return -EIO for 
>>> the
>>> initial write failure, but OK as long as eventually the write 
>>> succeeded.
>>> 
>>> It should not require any upper layer to do any extra work.
>>> 
>>> But since the write eventually failed, there is nothing upper layer 
>>> can
>>> do, unless the dm or fs layer has some extra recovery mechanism.
>> 
>> Now I'm confused, I'm running RAID1 an only one disk has/had 1 sector
>> failure.
>> Shouldn't Btrfs manage to to write this data, it should exist on one 
>> of
>> the other drives because of RAID1?
>> And shouldn't a scrub fix it?
> 
> Sorry, I finally got your concern that, it's not about the initial 
> write
> failure, but the future errors messages.
> 
> It turns out to be a bug in the older kernels, that after one super
> block write back error, the folio keeps its error flag without clearing
> it up, thus it always shows an error message.
> 
> And since it's RAID1, btrfs continues the fs (thus your fs is still
> running, not flipping into read-only).
> 
> Scrub won't solve it because there is nothing to resolve, everything is
> fine, except the false warning messages.
> 
> 
> In upstream it's fixed by a rework patch, upstream commit bc00965dbff7
> ("btrfs: count super block write errors in device instead of tracking
> folio error state") fixes the bug by going a completely different path
> counting the super block write back errors.
> 
> Unfortunately that commit is only in v6.10, and since it's not
> explicitly marked as a bug fix (even it indeed fixes a hidden bug), 
> it's
> not backported to any older kernel. (BTW, 6.8 kernel is already EOL)
> 
> Please update to v6.12 or newer LTS kernels.
> 
> Or just unmount and remount the fs and pray no more super block
> writeback errors happen again...

Unfortunately Canonical is shipping 6.8 with there latest LTS release 
and managing backports inhouse i guess.
But they have an option to upgrade to 6.11 with there Hardware 
Enablement (HWE) program.

At least I get that patch upgrading to 6.11.

Thank you for your help Qu.

-- 
Kai Stian Olstad