Fixing a corrupted file system

Fixing a corrupted file system

[Amit Shah]

Hello folks,

I had a msata drive that started showing signs of failure - error
reading sectors.  I had a single LUKS partition on it (for encryption)
with btrfs as the file system taking up the entire space (1 TB).

I got all the raw data from the drive via ddrescue, and have dumped it
into a second drive.  However, it looks like before I noticed the disk
errors, btrfs tried correcting them based on the erroneous reads on the
first disk, causing those errors to propagate on the recovered copy:

# mount /dev/mapper/luks-ad4cc8e3-17c1-41f8-8d9d-bb831bfcef23
/mnt/msatassd/
mount: /mnt/msatassd: can't read superblock on /dev/mapper/luks-
ad4cc8e3-17c1-41f8-8d9d-bb831bfcef23.
       dmesg(1) may have more information after failed mount system
call.

# dmesg|tail
[960464.502742] BTRFS: device label msatafs devid 1 transid 772442
/dev/mapper/luks-ad4cc8e3-17c1-41f8-8d9d-bb831bfcef23 scanned by mount
(10021)
[960464.516714] BTRFS info (device dm-0): first mount of filesystem
28fa4955-eadf-4f12-a34d-fb7405bbd0a5
[960464.526014] BTRFS info (device dm-0): using crc32c (crc32c-generic)
checksum algorithm
[960464.534065] BTRFS info (device dm-0): disk space caching is enabled
[960464.559255] BTRFS info (device dm-0): bdev /dev/mapper/luks-
ad4cc8e3-17c1-41f8-8d9d-bb831bfcef23 errs: wr 0, rd 0, flush 0, corrupt
100, gen 0
[960464.607016] BTRFS error (device dm-0): bad tree block start, mirror
1 want 1116258304 have 5031253079992706228
[960464.617170] BTRFS error (device dm-0): failed to read block groups:
-5
[960464.626009] BTRFS error (device dm-0): open_ctree failed: -5


(I also have the raw file that I can loop-mount instead of using this
other drive, but for this experiment, working on the drive is simpler
to iterate.)

So questions:

1. Can I do anything to fix the metadata for this filesystem, and have
it mount, recognize and recover the data on there?

2. Does it make sense to drop down the filesystem to read-only when the
underlying drive starts exhibiting errors?

Thanks,
		Amit

Re: Fixing a corrupted file system

[Qu Wenruo]

在 2026/4/2 02:28, Amit Shah 写道:
> Hello folks,
> 
> I had a msata drive that started showing signs of failure - error
> reading sectors.  I had a single LUKS partition on it (for encryption)
> with btrfs as the file system taking up the entire space (1 TB).
> 
> I got all the raw data from the drive via ddrescue, and have dumped it
> into a second drive.  However, it looks like before I noticed the disk
> errors, btrfs tried correcting them based on the erroneous reads on the
> first disk, causing those errors to propagate on the recovered copy:

Nope, btrfs will only try extra mirrors, but in your case your metadata 
doesn't even have extra mirrors.
Which is not common, as the default btrfs mkfs profiles will go DUP for 
metadata.

So you only have one chance and lost that chance.

> 
> # mount /dev/mapper/luks-ad4cc8e3-17c1-41f8-8d9d-bb831bfcef23
> /mnt/msatassd/
> mount: /mnt/msatassd: can't read superblock on /dev/mapper/luks-
> ad4cc8e3-17c1-41f8-8d9d-bb831bfcef23.
>         dmesg(1) may have more information after failed mount system
> call.
> 
> # dmesg|tail
> [960464.502742] BTRFS: device label msatafs devid 1 transid 772442
> /dev/mapper/luks-ad4cc8e3-17c1-41f8-8d9d-bb831bfcef23 scanned by mount
> (10021)
> [960464.516714] BTRFS info (device dm-0): first mount of filesystem
> 28fa4955-eadf-4f12-a34d-fb7405bbd0a5
> [960464.526014] BTRFS info (device dm-0): using crc32c (crc32c-generic)
> checksum algorithm
> [960464.534065] BTRFS info (device dm-0): disk space caching is enabled
> [960464.559255] BTRFS info (device dm-0): bdev /dev/mapper/luks-
> ad4cc8e3-17c1-41f8-8d9d-bb831bfcef23 errs: wr 0, rd 0, flush 0, corrupt
> 100, gen 0
> [960464.607016] BTRFS error (device dm-0): bad tree block start, mirror
> 1 want 1116258304 have 5031253079992706228
> [960464.617170] BTRFS error (device dm-0): failed to read block groups:
> -5
> [960464.626009] BTRFS error (device dm-0): open_ctree failed: -5
> 
> 
> (I also have the raw file that I can loop-mount instead of using this
> other drive, but for this experiment, working on the drive is simpler
> to iterate.)
> 
> So questions:
> 
> 1. Can I do anything to fix the metadata for this filesystem, and have
> it mount, recognize and recover the data on there?

Not really. You only have one copy or metadata, and it's corrupted.

It may be possible to repair, but it's definitely not reliable and I 
won't recommend to do it.

All you can do is try to salvage the data, by mounting with 
"rescue=all,ro" which should give you some chance to read out some data.

> 
> 2. Does it make sense to drop down the filesystem to read-only when the
> underlying drive starts exhibiting errors?

For your case, it makes no difference.

Your metadata is only SINGLE, and that's your choice, so there is no 
repair anyway.

Furthermore even with extra mirrors (e.g. DUP or RAID1), btrfs will only 
write the correct data/metadata into the bad one, thus it should not 
make things worse unless your hardware is totally screwed up.

Thanks,
Qu

> 
> Thanks,
> 		Amit
> 

Re: Fixing a corrupted file system

[Amit Shah]

Hello Qu,


On Thu, 2026-04-02 at 07:51 +1030, Qu Wenruo wrote:
> 
> 
> 在 2026/4/2 02:28, Amit Shah 写道:
> > Hello folks,
> > 
> > I had a msata drive that started showing signs of failure - error
> > reading sectors.  I had a single LUKS partition on it (for
> > encryption)
> > with btrfs as the file system taking up the entire space (1 TB).
> > 
> > I got all the raw data from the drive via ddrescue, and have dumped
> > it
> > into a second drive.  However, it looks like before I noticed the
> > disk
> > errors, btrfs tried correcting them based on the erroneous reads on
> > the
> > first disk, causing those errors to propagate on the recovered
> > copy:
> 
> Nope, btrfs will only try extra mirrors, but in your case your
> metadata 
> doesn't even have extra mirrors.
> Which is not common, as the default btrfs mkfs profiles will go DUP
> for 
> metadata.

Hm, looks like the functionality for auto-enabling metadata DUP only
landed in 5.15, and perhaps my install was from before that?  I didn't
go out of my way to select non-default entries.

> So you only have one chance and lost that chance.

[...]

> > 1. Can I do anything to fix the metadata for this filesystem, and
> > have
> > it mount, recognize and recover the data on there?
> 
> Not really. You only have one copy or metadata, and it's corrupted.
> 
> It may be possible to repair, but it's definitely not reliable and I 
> won't recommend to do it.
> 
> All you can do is try to salvage the data, by mounting with 
> "rescue=all,ro" which should give you some chance to read out some
> data.

Oh this is very useful, thanks.  I tried btrfs-rescue as a separate
command and only could reach a handful files.  Using it as a mount
option does give me a more complete-looking directory tree. As
expected, a few files have errors reading them, but most of them do
open fine.

> > 2. Does it make sense to drop down the filesystem to read-only when
> > the
> > underlying drive starts exhibiting errors?
> 
> For your case, it makes no difference.

Oh sure.  This isn't a question to fix anything for me now.  This is
just a way of throwing out a suggestion for a future feature - whether
something like that makes sense.

> Your metadata is only SINGLE, and that's your choice, so there is no 
> repair anyway.
> 
> Furthermore even with extra mirrors (e.g. DUP or RAID1), btrfs will
> only 
> write the correct data/metadata into the bad one, thus it should not 
> make things worse unless your hardware is totally screwed up.

Yea, that's understandable, of course.

Thanks a lot for the mount option hint, it has already helped me poke
around the file system.

		Amit

Re: Fixing a corrupted file system

[remi]

On Wed, Apr 1, 2026, at 5:21 PM, Qu Wenruo wrote:

>
> Nope, btrfs will only try extra mirrors, but in your case your metadata 
> doesn't even have extra mirrors.
> Which is not common, as the default btrfs mkfs profiles will go DUP for 
> metadata.

I forget what mkfs version, but the default behaviour changed a few years ago so the default profile is Single on SSD's. (Correct me if this has since been reverted.)  Something about the way SSD's copy on write, the DUP entries would end up on the same physical block, so the redundancy was not effective.

Re: Fixing a corrupted file system

[Amit Shah]

On Thu, 2026-04-02 at 14:32 -0400, remi@georgianit.com wrote:
> 
> 
> On Wed, Apr 1, 2026, at 5:21 PM, Qu Wenruo wrote:
> 
> > 
> > Nope, btrfs will only try extra mirrors, but in your case your
> > metadata 
> > doesn't even have extra mirrors.
> > Which is not common, as the default btrfs mkfs profiles will go DUP
> > for 
> > metadata.
> 
> I forget what mkfs version, but the default behaviour changed a few
> years ago so the default profile is Single on SSD's. (Correct me if
> this has since been reverted.)  Something about the way SSD's copy on
> write, the DUP entries would end up on the same physical block, so
> the redundancy was not effective.

Right, the man page continues to mention that now as well.

		Amit

Re: Fixing a corrupted file system

[Forza]

---- From: remi@georgianit.com -- Sent: 2026-04-02 - 20:32 ----

> 
> 
> On Wed, Apr 1, 2026, at 5:21 PM, Qu Wenruo wrote:
> 
>>
>> Nope, btrfs will only try extra mirrors, but in your case your metadata 
>> doesn't even have extra mirrors.
>> Which is not common, as the default btrfs mkfs profiles will go DUP for 
>> metadata.
> 
> I forget what mkfs version, but the default behaviour changed a few years ago so the default profile is Single on SSD's. (Correct me if this has since been reverted.)  Something about the way SSD's copy on write, the DUP entries would end up on the same physical block, so the redundancy was not effective.
> 

It is the reverse. Since btrfs-progs 5.15, the default was changed to DUP for single device fs.

https://github.com/kdave/btrfs-progs/commit/65181c273e67bd48d01fc79f00826dce38b93c4c

~ Forza

Re: Fixing a corrupted file system

[Ulli Horlacher]

On Thu 2026-04-02 (21:07), Forza wrote:

> > I forget what mkfs version, but the default behaviour changed a few years ago so the default profile is Single on SSD's. (Correct me if this has since been reverted.)  Something about the way SSD's copy on write, the DUP entries would end up on the same physical block, so the redundancy was not effective.
> 
> It is the reverse. Since btrfs-progs 5.15, the default was changed to DUP for single device fs.
> 
> https://github.com/kdave/btrfs-progs/commit/65181c273e67bd48d01fc79f00826dce38b93c4c

I have some old btrfs filesystems with single metadata:

root@moep:~# btrfs filesystem df /
Data, single: total=23.01GiB, used=12.51GiB
System, DUP: total=8.00MiB, used=16.00KiB
Metadata, DUP: total=1.75GiB, used=930.19MiB
GlobalReserve, single: total=42.06MiB, used=0.00B

root@moep:~# btrfs filesystem df /home
Data, single: total=95.01GiB, used=54.27GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=1.00GiB, used=293.70MiB
GlobalReserve, single: total=80.53MiB, used=0.00B

(How) can I convert /home to DUP for System and Metadata?
(/ looks good)

-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    https://www.tik.uni-stuttgart.de/
REF:<87d1aca.ae41f579.19d4f977d95@tnonline.net>

Re: Fixing a corrupted file system

[Roman Mamedov]

On Fri, 24 Apr 2026 10:23:00 +0200
Ulli Horlacher <framstag@rus.uni-stuttgart.de> wrote:

> root@moep:~# btrfs filesystem df /home
> Data, single: total=95.01GiB, used=54.27GiB
> System, single: total=32.00MiB, used=16.00KiB
> Metadata, single: total=1.00GiB, used=293.70MiB
> GlobalReserve, single: total=80.53MiB, used=0.00B
> 
> (How) can I convert /home to DUP for System and Metadata?
> (/ looks good)
> 

See https://btrfs.readthedocs.io/en/latest/Balance.html

  btrfs balance start -mprofiles=single -mconvert=dup /home

-- 
With respect,
Roman

How to corrupt a btrfs fs [was Re: Fixing a corrupted file system]

[Nicholas D Steeves]

[-- Attachment #1: Type: text/plain, Size: 847 bytes --]

Hi,

Roman Mamedov <rm@romanrm.net> writes:

> On Fri, 24 Apr 2026 10:23:00 +0200
> Ulli Horlacher <framstag@rus.uni-stuttgart.de> wrote:
>
>> root@moep:~# btrfs filesystem df /home
>> Data, single: total=95.01GiB, used=54.27GiB
>> System, single: total=32.00MiB, used=16.00KiB
>> Metadata, single: total=1.00GiB, used=293.70MiB
>> GlobalReserve, single: total=80.53MiB, used=0.00B
>> 
>> (How) can I convert /home to DUP for System and Metadata?
>> (/ looks good)
>> 
>
> See https://btrfs.readthedocs.io/en/latest/Balance.html
>
>   btrfs balance start -mprofiles=single -mconvert=dup /home

FYI, I just tried this with linux-6.12.57, btrfs-progs_6.17.1 and it
corrupted my filesystem.  Luckily I only lose this morning's and
afternoon's work...

I am happy to provide post-mortem debugging info.

Regards,
Nicholas

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 861 bytes --]

Re: How to corrupt a btrfs fs [was Re: Fixing a corrupted file system]

[Roman Mamedov]

On Fri, 24 Apr 2026 14:10:17 -0400
Nicholas D Steeves <nsteeves@gmail.com> wrote:

> > See https://btrfs.readthedocs.io/en/latest/Balance.html
> >
> >   btrfs balance start -mprofiles=single -mconvert=dup /home
> 
> FYI, I just tried this with linux-6.12.57, btrfs-progs_6.17.1 and it
> corrupted my filesystem.  Luckily I only lose this morning's and
> afternoon's work...
> 
> I am happy to provide post-mortem debugging info.

This is the correct command and I've used it successfully in the past.

Perhaps there's a new bug in your kernel or progs version. In that case it
would be helpful for developers to see a dmesg of what happened, along with
all the other stuff, such as "filesystem check" (no --repair for now).

-- 
With respect,
Roman