[f2fs-dev] Resize metadata corruption

[f2fs-dev] Resize metadata corruption

[uplinkr--- via Linux-f2fs-devel]

Hello everyone,

I am having trouble with F2FS. Specifically, I believe metadata got 
corrupted when I resized it. I have a 512 GB drive. My F2FS partition 
was approximately located on 369-497 GB (128 GB size). Using GParted, I 
resized it to 0.5-497 GB. While the partition resizing went through 
successfully, filesystem resizing initially failed with "Mount unclean 
image to replay log". I have done that and retried resizing.

Afterwards, however, fsck started giving out a lot of errors, at one 
point it asked if I wished to restore lost data, which I agreed to. Logs 
specified a lot of my files (which I could tell by filenames) and 
mid-way through the process, it segfaulted. Now, when I run fsck, no 
files are asked to be restored, and it completes successfully. However, 
when I attempt to mount it, I get an error saying "Structure needs 
cleaning".

Could someone help me restore my metadata (at least, long enough to 
extract my files)? Thanks.

dmesg logs:

[   96.184127] F2FS-fs (nvme0n1p5): Mismatch valid blocks 769 vs. 68
[   96.188050] F2FS-fs (nvme0n1p5): Failed to initialize F2FS segment 
manager (-117)

fsck.f2fs --dry-run -d 3 logs: https://arter.com/.f2fs-20250406/fsck.log


_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] Resize metadata corruption

[Juhyung Park]

Hi all,

I also encountered something similar a while back with resizing but
didn't report it and just manually migrated the files.
I assisted him to ensure using the latest kernel/f2fs-tools, but his
metadata seems pretty bad right now and I suggested him to ask the
mailing list directly.

The URL there should be https://arter97.com/.f2fs-20250406/fsck.log ,
a little typo there.

Can we have some sort of CI/automated testing for the resizing as well?

Thanks.

On Sun, Apr 6, 2025 at 1:26 AM uplinkr--- via Linux-f2fs-devel
<linux-f2fs-devel@lists.sourceforge.net> wrote:
>
> Hello everyone,
>
> I am having trouble with F2FS. Specifically, I believe metadata got
> corrupted when I resized it. I have a 512 GB drive. My F2FS partition
> was approximately located on 369-497 GB (128 GB size). Using GParted, I
> resized it to 0.5-497 GB. While the partition resizing went through
> successfully, filesystem resizing initially failed with "Mount unclean
> image to replay log". I have done that and retried resizing.
>
> Afterwards, however, fsck started giving out a lot of errors, at one
> point it asked if I wished to restore lost data, which I agreed to. Logs
> specified a lot of my files (which I could tell by filenames) and
> mid-way through the process, it segfaulted. Now, when I run fsck, no
> files are asked to be restored, and it completes successfully. However,
> when I attempt to mount it, I get an error saying "Structure needs
> cleaning".
>
> Could someone help me restore my metadata (at least, long enough to
> extract my files)? Thanks.
>
> dmesg logs:
>
> [   96.184127] F2FS-fs (nvme0n1p5): Mismatch valid blocks 769 vs. 68
> [   96.188050] F2FS-fs (nvme0n1p5): Failed to initialize F2FS segment
> manager (-117)
>
> fsck.f2fs --dry-run -d 3 logs: https://arter.com/.f2fs-20250406/fsck.log
>
>
> _______________________________________________
> Linux-f2fs-devel mailing list
> Linux-f2fs-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel


_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] Resize metadata corruption

[uplinkr--- via Linux-f2fs-devel]

Hello everyone,

I have also decided to take a look at the partition with data recovery 
tools such as Photorec. It seems all my data is intact! As such, I was 
considering an alternate data recovery solution.

Judging by F2FS's documentation, it is blocks who keep track of their 
path rather than the checkpoint. If I could export all directory and 
file names, perhaps I could recover my data manually. Is this the 
simpler solution, or should fsck be explicitly patched to resolve both 
the corruption recovery failure and resize block misalignment?

Thanks.

On 2025-04-06 11:30, Juhyung Park wrote:
> Hi all,
> 
> I also encountered something similar a while back with resizing but
> didn't report it and just manually migrated the files.
> I assisted him to ensure using the latest kernel/f2fs-tools, but his
> metadata seems pretty bad right now and I suggested him to ask the
> mailing list directly.
> 
> The URL there should be https://arter97.com/.f2fs-20250406/fsck.log ,
> a little typo there.
> 
> Can we have some sort of CI/automated testing for the resizing as well?
> 
> Thanks.
> 
> On Sun, Apr 6, 2025 at 1:26 AM uplinkr--- via Linux-f2fs-devel
> <linux-f2fs-devel@lists.sourceforge.net> wrote:
>> 
>> Hello everyone,
>> 
>> I am having trouble with F2FS. Specifically, I believe metadata got
>> corrupted when I resized it. I have a 512 GB drive. My F2FS partition
>> was approximately located on 369-497 GB (128 GB size). Using GParted, 
>> I
>> resized it to 0.5-497 GB. While the partition resizing went through
>> successfully, filesystem resizing initially failed with "Mount unclean
>> image to replay log". I have done that and retried resizing.
>> 
>> Afterwards, however, fsck started giving out a lot of errors, at one
>> point it asked if I wished to restore lost data, which I agreed to. 
>> Logs
>> specified a lot of my files (which I could tell by filenames) and
>> mid-way through the process, it segfaulted. Now, when I run fsck, no
>> files are asked to be restored, and it completes successfully. 
>> However,
>> when I attempt to mount it, I get an error saying "Structure needs
>> cleaning".
>> 
>> Could someone help me restore my metadata (at least, long enough to
>> extract my files)? Thanks.
>> 
>> dmesg logs:
>> 
>> [   96.184127] F2FS-fs (nvme0n1p5): Mismatch valid blocks 769 vs. 68
>> [   96.188050] F2FS-fs (nvme0n1p5): Failed to initialize F2FS segment
>> manager (-117)
>> 
>> fsck.f2fs --dry-run -d 3 logs: 
>> https://arter.com/.f2fs-20250406/fsck.log
>> 
>> 
>> _______________________________________________
>> Linux-f2fs-devel mailing list
>> Linux-f2fs-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel


_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] Resize metadata corruption

[Chao Yu via Linux-f2fs-devel]

On 4/6/25 16:30, Juhyung Park wrote:
> Hi all,
> 
> I also encountered something similar a while back with resizing but
> didn't report it and just manually migrated the files.

Hi Juhyung,

Did you develop an individual tool to migrate specified inode to target
block address?

> I assisted him to ensure using the latest kernel/f2fs-tools, but his
> metadata seems pretty bad right now and I suggested him to ask the
> mailing list directly.

I checked the log, I guess it actually seems pretty bad... I guess we
need to find out those file which has not been migrated correctly, and
try to correct them, may be w/ a new tool.

To Jaegeuk, any thoughts about this problem?

> 
> The URL there should be https://arter97.com/.f2fs-20250406/fsck.log ,
> a little typo there.

Thanks, I didn't notice this previously.

> 
> Can we have some sort of CI/automated testing for the resizing as well?

Agreed, will work on some testcases for resize.f2fs when I get some free
slots.

Thanks,

> 
> Thanks.
> 
> On Sun, Apr 6, 2025 at 1:26 AM uplinkr--- via Linux-f2fs-devel
> <linux-f2fs-devel@lists.sourceforge.net> wrote:
>>
>> Hello everyone,
>>
>> I am having trouble with F2FS. Specifically, I believe metadata got
>> corrupted when I resized it. I have a 512 GB drive. My F2FS partition
>> was approximately located on 369-497 GB (128 GB size). Using GParted, I
>> resized it to 0.5-497 GB. While the partition resizing went through
>> successfully, filesystem resizing initially failed with "Mount unclean
>> image to replay log". I have done that and retried resizing.
>>
>> Afterwards, however, fsck started giving out a lot of errors, at one
>> point it asked if I wished to restore lost data, which I agreed to. Logs
>> specified a lot of my files (which I could tell by filenames) and
>> mid-way through the process, it segfaulted. Now, when I run fsck, no
>> files are asked to be restored, and it completes successfully. However,
>> when I attempt to mount it, I get an error saying "Structure needs
>> cleaning".
>>
>> Could someone help me restore my metadata (at least, long enough to
>> extract my files)? Thanks.
>>
>> dmesg logs:
>>
>> [   96.184127] F2FS-fs (nvme0n1p5): Mismatch valid blocks 769 vs. 68
>> [   96.188050] F2FS-fs (nvme0n1p5): Failed to initialize F2FS segment
>> manager (-117)
>>
>> fsck.f2fs --dry-run -d 3 logs: https://arter.com/.f2fs-20250406/fsck.log
>>
>>
>> _______________________________________________
>> Linux-f2fs-devel mailing list
>> Linux-f2fs-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
> 
> 
> _______________________________________________
> Linux-f2fs-devel mailing list
> Linux-f2fs-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel



_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] Resize metadata corruption

[Juhyung Park]

Hi Chao,

On Wed, Apr 9, 2025 at 10:52 PM Chao Yu <chao@kernel.org> wrote:
>
> On 4/6/25 16:30, Juhyung Park wrote:
> > Hi all,
> >
> > I also encountered something similar a while back with resizing but
> > didn't report it and just manually migrated the files.
>
> Hi Juhyung,
>
> Did you develop an individual tool to migrate specified inode to target
> block address?

Nope. And neither did @uplinkr use any custom tools to mess around the
f2fs partition.

>
> > I assisted him to ensure using the latest kernel/f2fs-tools, but his
> > metadata seems pretty bad right now and I suggested him to ask the
> > mailing list directly.
>
> I checked the log, I guess it actually seems pretty bad... I guess we
> need to find out those file which has not been migrated correctly, and
> try to correct them, may be w/ a new tool.

Yeah, having fsck.f2fs segfault mid-way post resize won't help either.

>
> To Jaegeuk, any thoughts about this problem?
>
> >
> > The URL there should be https://arter97.com/.f2fs-20250406/fsck.log ,
> > a little typo there.
>
> Thanks, I didn't notice this previously.
>
> >
> > Can we have some sort of CI/automated testing for the resizing as well?
>
> Agreed, will work on some testcases for resize.f2fs when I get some free
> slots.

Should we mark this experimental for the time being?

Thanks.

>
> Thanks,
>
> >
> > Thanks.
> >
> > On Sun, Apr 6, 2025 at 1:26 AM uplinkr--- via Linux-f2fs-devel
> > <linux-f2fs-devel@lists.sourceforge.net> wrote:
> >>
> >> Hello everyone,
> >>
> >> I am having trouble with F2FS. Specifically, I believe metadata got
> >> corrupted when I resized it. I have a 512 GB drive. My F2FS partition
> >> was approximately located on 369-497 GB (128 GB size). Using GParted, I
> >> resized it to 0.5-497 GB. While the partition resizing went through
> >> successfully, filesystem resizing initially failed with "Mount unclean
> >> image to replay log". I have done that and retried resizing.
> >>
> >> Afterwards, however, fsck started giving out a lot of errors, at one
> >> point it asked if I wished to restore lost data, which I agreed to. Logs
> >> specified a lot of my files (which I could tell by filenames) and
> >> mid-way through the process, it segfaulted. Now, when I run fsck, no
> >> files are asked to be restored, and it completes successfully. However,
> >> when I attempt to mount it, I get an error saying "Structure needs
> >> cleaning".
> >>
> >> Could someone help me restore my metadata (at least, long enough to
> >> extract my files)? Thanks.
> >>
> >> dmesg logs:
> >>
> >> [   96.184127] F2FS-fs (nvme0n1p5): Mismatch valid blocks 769 vs. 68
> >> [   96.188050] F2FS-fs (nvme0n1p5): Failed to initialize F2FS segment
> >> manager (-117)
> >>
> >> fsck.f2fs --dry-run -d 3 logs: https://arter.com/.f2fs-20250406/fsck.log
> >>
> >>
> >> _______________________________________________
> >> Linux-f2fs-devel mailing list
> >> Linux-f2fs-devel@lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
> >
> >
> > _______________________________________________
> > Linux-f2fs-devel mailing list
> > Linux-f2fs-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
>


_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] Resize metadata corruption

[Chao Yu via Linux-f2fs-devel]

On 4/10/25 13:57, Juhyung Park wrote:
> Hi Chao,
> 
> On Wed, Apr 9, 2025 at 10:52 PM Chao Yu <chao@kernel.org> wrote:
>>
>> On 4/6/25 16:30, Juhyung Park wrote:
>>> Hi all,
>>>
>>> I also encountered something similar a while back with resizing but
>>> didn't report it and just manually migrated the files.
>>
>> Hi Juhyung,
>>
>> Did you develop an individual tool to migrate specified inode to target
>> block address?
> 
> Nope. And neither did @uplinkr use any custom tools to mess around the
> f2fs partition.

Oh, so how did you migrate files manually?

> 
>>
>>> I assisted him to ensure using the latest kernel/f2fs-tools, but his
>>> metadata seems pretty bad right now and I suggested him to ask the
>>> mailing list directly.
>>
>> I checked the log, I guess it actually seems pretty bad... I guess we
>> need to find out those file which has not been migrated correctly, and
>> try to correct them, may be w/ a new tool.
> 
> Yeah, having fsck.f2fs segfault mid-way post resize won't help either.
> 
>>
>> To Jaegeuk, any thoughts about this problem?
>>
>>>
>>> The URL there should be https://arter97.com/.f2fs-20250406/fsck.log ,
>>> a little typo there.
>>
>> Thanks, I didn't notice this previously.
>>
>>>
>>> Can we have some sort of CI/automated testing for the resizing as well?
>>
>> Agreed, will work on some testcases for resize.f2fs when I get some free
>> slots.
> 
> Should we mark this experimental for the time being?

Agreed.

Thanks,

> 
> Thanks.
> 
>>
>> Thanks,
>>
>>>
>>> Thanks.
>>>
>>> On Sun, Apr 6, 2025 at 1:26 AM uplinkr--- via Linux-f2fs-devel
>>> <linux-f2fs-devel@lists.sourceforge.net> wrote:
>>>>
>>>> Hello everyone,
>>>>
>>>> I am having trouble with F2FS. Specifically, I believe metadata got
>>>> corrupted when I resized it. I have a 512 GB drive. My F2FS partition
>>>> was approximately located on 369-497 GB (128 GB size). Using GParted, I
>>>> resized it to 0.5-497 GB. While the partition resizing went through
>>>> successfully, filesystem resizing initially failed with "Mount unclean
>>>> image to replay log". I have done that and retried resizing.
>>>>
>>>> Afterwards, however, fsck started giving out a lot of errors, at one
>>>> point it asked if I wished to restore lost data, which I agreed to. Logs
>>>> specified a lot of my files (which I could tell by filenames) and
>>>> mid-way through the process, it segfaulted. Now, when I run fsck, no
>>>> files are asked to be restored, and it completes successfully. However,
>>>> when I attempt to mount it, I get an error saying "Structure needs
>>>> cleaning".
>>>>
>>>> Could someone help me restore my metadata (at least, long enough to
>>>> extract my files)? Thanks.
>>>>
>>>> dmesg logs:
>>>>
>>>> [   96.184127] F2FS-fs (nvme0n1p5): Mismatch valid blocks 769 vs. 68
>>>> [   96.188050] F2FS-fs (nvme0n1p5): Failed to initialize F2FS segment
>>>> manager (-117)
>>>>
>>>> fsck.f2fs --dry-run -d 3 logs: https://arter.com/.f2fs-20250406/fsck.log
>>>>
>>>>
>>>> _______________________________________________
>>>> Linux-f2fs-devel mailing list
>>>> Linux-f2fs-devel@lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
>>>
>>>
>>> _______________________________________________
>>> Linux-f2fs-devel mailing list
>>> Linux-f2fs-devel@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
>>



_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] Resize metadata corruption

[Juhyung Park]

On Thu, Apr 10, 2025 at 12:00 AM Chao Yu <chao@kernel.org> wrote:
>
> On 4/10/25 13:57, Juhyung Park wrote:
> > Hi Chao,
> >
> > On Wed, Apr 9, 2025 at 10:52 PM Chao Yu <chao@kernel.org> wrote:
> >>
> >> On 4/6/25 16:30, Juhyung Park wrote:
> >>> Hi all,
> >>>
> >>> I also encountered something similar a while back with resizing but
> >>> didn't report it and just manually migrated the files.
> >>
> >> Hi Juhyung,
> >>
> >> Did you develop an individual tool to migrate specified inode to target
> >> block address?
> >
> > Nope. And neither did @uplinkr use any custom tools to mess around the
> > f2fs partition.
>
> Oh, so how did you migrate files manually?

You remember the extended node bitmap fiasco :)
After that, whenever I run fsck (or in this case, resize), I wrap the
entire raw block device to a dm-snapshot so that original stays intact
and run if on top of the dm, so that I don't run into corner cases.

When I was migrating my personal f2fs setup to another SSD, I had
similar resize/fsck warnings/errors. (I should've reported this back
then, sorry.)
So I decided to just rsync the whole partition after mounting it and
migrate under VFS and not risk the potential corner case.

So for my case, it was before the damage was done (with dm-snapshot)
unlike @uplinkr.

>
> >
> >>
> >>> I assisted him to ensure using the latest kernel/f2fs-tools, but his
> >>> metadata seems pretty bad right now and I suggested him to ask the
> >>> mailing list directly.
> >>
> >> I checked the log, I guess it actually seems pretty bad... I guess we
> >> need to find out those file which has not been migrated correctly, and
> >> try to correct them, may be w/ a new tool.
> >
> > Yeah, having fsck.f2fs segfault mid-way post resize won't help either.
> >
> >>
> >> To Jaegeuk, any thoughts about this problem?
> >>
> >>>
> >>> The URL there should be https://arter97.com/.f2fs-20250406/fsck.log ,
> >>> a little typo there.
> >>
> >> Thanks, I didn't notice this previously.
> >>
> >>>
> >>> Can we have some sort of CI/automated testing for the resizing as well?
> >>
> >> Agreed, will work on some testcases for resize.f2fs when I get some free
> >> slots.
> >
> > Should we mark this experimental for the time being?
>
> Agreed.
>
> Thanks,
>
> >
> > Thanks.
> >
> >>
> >> Thanks,
> >>
> >>>
> >>> Thanks.
> >>>
> >>> On Sun, Apr 6, 2025 at 1:26 AM uplinkr--- via Linux-f2fs-devel
> >>> <linux-f2fs-devel@lists.sourceforge.net> wrote:
> >>>>
> >>>> Hello everyone,
> >>>>
> >>>> I am having trouble with F2FS. Specifically, I believe metadata got
> >>>> corrupted when I resized it. I have a 512 GB drive. My F2FS partition
> >>>> was approximately located on 369-497 GB (128 GB size). Using GParted, I
> >>>> resized it to 0.5-497 GB. While the partition resizing went through
> >>>> successfully, filesystem resizing initially failed with "Mount unclean
> >>>> image to replay log". I have done that and retried resizing.
> >>>>
> >>>> Afterwards, however, fsck started giving out a lot of errors, at one
> >>>> point it asked if I wished to restore lost data, which I agreed to. Logs
> >>>> specified a lot of my files (which I could tell by filenames) and
> >>>> mid-way through the process, it segfaulted. Now, when I run fsck, no
> >>>> files are asked to be restored, and it completes successfully. However,
> >>>> when I attempt to mount it, I get an error saying "Structure needs
> >>>> cleaning".
> >>>>
> >>>> Could someone help me restore my metadata (at least, long enough to
> >>>> extract my files)? Thanks.
> >>>>
> >>>> dmesg logs:
> >>>>
> >>>> [   96.184127] F2FS-fs (nvme0n1p5): Mismatch valid blocks 769 vs. 68
> >>>> [   96.188050] F2FS-fs (nvme0n1p5): Failed to initialize F2FS segment
> >>>> manager (-117)
> >>>>
> >>>> fsck.f2fs --dry-run -d 3 logs: https://arter.com/.f2fs-20250406/fsck.log
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Linux-f2fs-devel mailing list
> >>>> Linux-f2fs-devel@lists.sourceforge.net
> >>>> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
> >>>
> >>>
> >>> _______________________________________________
> >>> Linux-f2fs-devel mailing list
> >>> Linux-f2fs-devel@lists.sourceforge.net
> >>> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
> >>
>


_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] Resize metadata corruption

[Chao Yu via Linux-f2fs-devel]

On 4/10/25 15:08, Juhyung Park wrote:
> On Thu, Apr 10, 2025 at 12:00 AM Chao Yu <chao@kernel.org> wrote:
>>
>> On 4/10/25 13:57, Juhyung Park wrote:
>>> Hi Chao,
>>>
>>> On Wed, Apr 9, 2025 at 10:52 PM Chao Yu <chao@kernel.org> wrote:
>>>>
>>>> On 4/6/25 16:30, Juhyung Park wrote:
>>>>> Hi all,
>>>>>
>>>>> I also encountered something similar a while back with resizing but
>>>>> didn't report it and just manually migrated the files.
>>>>
>>>> Hi Juhyung,
>>>>
>>>> Did you develop an individual tool to migrate specified inode to target
>>>> block address?
>>>
>>> Nope. And neither did @uplinkr use any custom tools to mess around the
>>> f2fs partition.
>>
>> Oh, so how did you migrate files manually?
> 
> You remember the extended node bitmap fiasco :)

Oh, yes, luckily, you reported that bug so quickly, so that we can fix
that in time, thank you!

> After that, whenever I run fsck (or in this case, resize), I wrap the
> entire raw block device to a dm-snapshot so that original stays intact
> and run if on top of the dm, so that I don't run into corner cases.

I think it's a good habit to always make important data being backuped.

> 
> When I was migrating my personal f2fs setup to another SSD, I had
> similar resize/fsck warnings/errors. (I should've reported this back
> then, sorry.)

Alright, anyway, I'd like to say please feel free to report any bugs of
f2fs you encounter later, will be appreciate for that. :)

> So I decided to just rsync the whole partition after mounting it and
> migrate under VFS and not risk the potential corner case.
> 
> So for my case, it was before the damage was done (with dm-snapshot)
> unlike @uplinkr.

Okay, got it, thanks for the details.

Thanks,

> 
>>
>>>
>>>>
>>>>> I assisted him to ensure using the latest kernel/f2fs-tools, but his
>>>>> metadata seems pretty bad right now and I suggested him to ask the
>>>>> mailing list directly.
>>>>
>>>> I checked the log, I guess it actually seems pretty bad... I guess we
>>>> need to find out those file which has not been migrated correctly, and
>>>> try to correct them, may be w/ a new tool.
>>>
>>> Yeah, having fsck.f2fs segfault mid-way post resize won't help either.
>>>
>>>>
>>>> To Jaegeuk, any thoughts about this problem?
>>>>
>>>>>
>>>>> The URL there should be https://arter97.com/.f2fs-20250406/fsck.log ,
>>>>> a little typo there.
>>>>
>>>> Thanks, I didn't notice this previously.
>>>>
>>>>>
>>>>> Can we have some sort of CI/automated testing for the resizing as well?
>>>>
>>>> Agreed, will work on some testcases for resize.f2fs when I get some free
>>>> slots.
>>>
>>> Should we mark this experimental for the time being?
>>
>> Agreed.
>>
>> Thanks,
>>
>>>
>>> Thanks.
>>>
>>>>
>>>> Thanks,
>>>>
>>>>>
>>>>> Thanks.
>>>>>
>>>>> On Sun, Apr 6, 2025 at 1:26 AM uplinkr--- via Linux-f2fs-devel
>>>>> <linux-f2fs-devel@lists.sourceforge.net> wrote:
>>>>>>
>>>>>> Hello everyone,
>>>>>>
>>>>>> I am having trouble with F2FS. Specifically, I believe metadata got
>>>>>> corrupted when I resized it. I have a 512 GB drive. My F2FS partition
>>>>>> was approximately located on 369-497 GB (128 GB size). Using GParted, I
>>>>>> resized it to 0.5-497 GB. While the partition resizing went through
>>>>>> successfully, filesystem resizing initially failed with "Mount unclean
>>>>>> image to replay log". I have done that and retried resizing.
>>>>>>
>>>>>> Afterwards, however, fsck started giving out a lot of errors, at one
>>>>>> point it asked if I wished to restore lost data, which I agreed to. Logs
>>>>>> specified a lot of my files (which I could tell by filenames) and
>>>>>> mid-way through the process, it segfaulted. Now, when I run fsck, no
>>>>>> files are asked to be restored, and it completes successfully. However,
>>>>>> when I attempt to mount it, I get an error saying "Structure needs
>>>>>> cleaning".
>>>>>>
>>>>>> Could someone help me restore my metadata (at least, long enough to
>>>>>> extract my files)? Thanks.
>>>>>>
>>>>>> dmesg logs:
>>>>>>
>>>>>> [   96.184127] F2FS-fs (nvme0n1p5): Mismatch valid blocks 769 vs. 68
>>>>>> [   96.188050] F2FS-fs (nvme0n1p5): Failed to initialize F2FS segment
>>>>>> manager (-117)
>>>>>>
>>>>>> fsck.f2fs --dry-run -d 3 logs: https://arter.com/.f2fs-20250406/fsck.log
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Linux-f2fs-devel mailing list
>>>>>> Linux-f2fs-devel@lists.sourceforge.net
>>>>>> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Linux-f2fs-devel mailing list
>>>>> Linux-f2fs-devel@lists.sourceforge.net
>>>>> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
>>>>
>>



_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] Resize metadata corruption

[uplinkr--- via Linux-f2fs-devel]

On 2025-04-10 08:52, Chao Yu wrote:
> I checked the log, I guess it actually seems pretty bad... I guess we
> need to find out those file which has not been migrated correctly, and
> try to correct them, may be w/ a new tool.

Hello,

The issue is the corrupt partition in question contains a lot of 
unbackupped, valuable data for me. I wasn't aware or informed of the 
potential issues resizing on F2FS has (the ArchWiki listed none) and as 
such recovery of this partition is a lifeline for me.

Could you please write this tool or a patch that I can try in fsck?

Thanks.


_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] Resize metadata corruption

[Chao Yu via Linux-f2fs-devel]

On 4/10/25 14:19, uplinkr@airmail.cc wrote:
> On 2025-04-10 08:52, Chao Yu wrote:
>> I checked the log, I guess it actually seems pretty bad... I guess we
>> need to find out those file which has not been migrated correctly, and
>> try to correct them, may be w/ a new tool.
> 
> Hello,
> 
> The issue is the corrupt partition in question contains a lot of unbackupped, valuable data for me. I wasn't aware or informed of the potential issues resizing on F2FS has (the ArchWiki listed none)
> and as such recovery of this partition is a lifeline for me.

Sorry about this, we should give a explicit caution about resize tool
use.

But still I didn't get why we can run into this situation, since as you
said, resize went through successfully. Could you please provide more
details about process of resize? Parameters for resize? Logs you kept
during resize? etc.

> 
> Could you please write this tool or a patch that I can try in fsck?

I will try my best.

> 
> Thanks.



_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] Resize metadata corruption

[uplinkr--- via Linux-f2fs-devel]

On 2025-04-10 10:17, Chao Yu wrote:
> But still I didn't get why we can run into this situation, since as you
> said, resize went through successfully. Could you please provide more
> details about process of resize? Parameters for resize? Logs you kept
> during resize? etc.

Hello,

I'm afraid I don't have any logs beyond fsck because I was running all 
operations off a live ISO and didn't have the foresight to save the 
logs. I will attempt to replicate my actions on a VM to have further 
insight.

I started the resize in GParted. It initially went through with its GPT 
workflow (resize the GPT partition, move F2FS to the left, exact steps 
are unclear). Proceeding to growing the filesystem, however, it errored 
with "Mount unclean image to replay log". I initially didn't take this 
at face value and rebooted into the system to see my filesystem 
completely intact (regrettably, not taking a backup).

Afterwards, I rebooted back into the live ISO, mounted and unmounted the 
partition and ran "Check" in GParted. It first triggered an fsck (which 
suggested it the filesystem could grow), launched resize.f2fs (which 
went through successfully) and ran fsck again. This is where the fun 
stopped: GParted freezed dead. I killed it and ran fsck in a terminal 
with the same args as GParted (-f -a) and regrettably pressed y on 
restoring lost files. It listed a lot of my files (which I recognized by 
filenames) and went on for about two minutes before erroring with 
"Segmentation Error".

I hope this will be helpful. Thanks.


_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] Resize metadata corruption

[Juhyung Park]

Hi Chao.

Good news!

It appears that by some miracle, @uplinkr managed to dodge all odds:
1. gparted did not perform shrink as they didn't write the fool-proof
shrink size calculation code yet
2. The new location within the partition table did not overlap with
the old partition
3. The faulty fsck run did not touch the old partition's area

We managed to get the older partition's location via testdisk (which
looks for f2fs magic code) and restore the entire partition.

I insisted that he use this opportunity to reproduce the issue in a
safe fashion, communicate with upstream and make sure no one else has
to go through the same thing (after a full backup, obviously).

I'll walk him through setting up a dm-snapshot and gather the
initial/proper resize, mount, fsck log after the backup.

One thing to note here is that he said that the utilization was very
full: only 8G left. Maybe this is the corner case that we need to look
out for?

Thanks,

On Thu, Apr 10, 2025 at 12:17 AM Chao Yu <chao@kernel.org> wrote:
>
> On 4/10/25 14:19, uplinkr@airmail.cc wrote:
> > On 2025-04-10 08:52, Chao Yu wrote:
> >> I checked the log, I guess it actually seems pretty bad... I guess we
> >> need to find out those file which has not been migrated correctly, and
> >> try to correct them, may be w/ a new tool.
> >
> > Hello,
> >
> > The issue is the corrupt partition in question contains a lot of unbackupped, valuable data for me. I wasn't aware or informed of the potential issues resizing on F2FS has (the ArchWiki listed none)
> > and as such recovery of this partition is a lifeline for me.
>
> Sorry about this, we should give a explicit caution about resize tool
> use.
>
> But still I didn't get why we can run into this situation, since as you
> said, resize went through successfully. Could you please provide more
> details about process of resize? Parameters for resize? Logs you kept
> during resize? etc.
>
> >
> > Could you please write this tool or a patch that I can try in fsck?
>
> I will try my best.
>
> >
> > Thanks.
>


_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] Resize metadata corruption

[Chao Yu via Linux-f2fs-devel]

On 2025/4/11 13:26, Juhyung Park wrote:
> Hi Chao.
> 
> Good news!
> 
> It appears that by some miracle, @uplinkr managed to dodge all odds:
> 1. gparted did not perform shrink as they didn't write the fool-proof
> shrink size calculation code yet
> 2. The new location within the partition table did not overlap with
> the old partition
> 3. The faulty fsck run did not touch the old partition's area
> 
> We managed to get the older partition's location via testdisk (which
> looks for f2fs magic code) and restore the entire partition.

Juhyung,

Great! That's really good news.

I'm too focused on the bug/repair to miss that point, anyway that's really
good point to try to restore data by checking and reading from original
location.

> 
> I insisted that he use this opportunity to reproduce the issue in a
> safe fashion, communicate with upstream and make sure no one else has
> to go through the same thing (after a full backup, obviously).

Thanks, it's really important for us and other uses.

I'm planning to post revert patch for -i support today, and also I'm thinking
about adding a -f option to allow distros user to forcing running resize, w/o
-f option, let it only print caution message of the risk.

> 
> I'll walk him through setting up a dm-snapshot and gather the
> initial/proper resize, mount, fsck log after the backup.
> 
> One thing to note here is that he said that the utilization was very
> full: only 8G left. Maybe this is the corner case that we need to look
> out for?

Thanks for the hint, let me create testcases based on such condition to
see whether it can trigger the bug.

Thanks,

> 
> Thanks,
> 
> On Thu, Apr 10, 2025 at 12:17 AM Chao Yu <chao@kernel.org> wrote:
>>
>> On 4/10/25 14:19, uplinkr@airmail.cc wrote:
>>> On 2025-04-10 08:52, Chao Yu wrote:
>>>> I checked the log, I guess it actually seems pretty bad... I guess we
>>>> need to find out those file which has not been migrated correctly, and
>>>> try to correct them, may be w/ a new tool.
>>>
>>> Hello,
>>>
>>> The issue is the corrupt partition in question contains a lot of unbackupped, valuable data for me. I wasn't aware or informed of the potential issues resizing on F2FS has (the ArchWiki listed none)
>>> and as such recovery of this partition is a lifeline for me.
>>
>> Sorry about this, we should give a explicit caution about resize tool
>> use.
>>
>> But still I didn't get why we can run into this situation, since as you
>> said, resize went through successfully. Could you please provide more
>> details about process of resize? Parameters for resize? Logs you kept
>> during resize? etc.
>>
>>>
>>> Could you please write this tool or a patch that I can try in fsck?
>>
>> I will try my best.
>>
>>>
>>> Thanks.
>>



_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] Resize metadata corruption

[uplinkr--- via Linux-f2fs-devel]

Hello everyone,

I'm sorry to not have anything constructive to add, but I'm about as 
flabbergasted as I could be. I retraced all my steps (exactly as I went 
through with them before!) and was unable to replicate the corruption. 
On the contrary, the filesystem grew completely as expected.

After restoring my partition and backing it up, I ran fsck on it. fsck 
reported no issues with the data, so I pressed on. I opened up GParted 
and expanded it like last time. This time, however, I didn't end up with 
no "Mount unclean image to replay log" error, and the resize went 
through as expected. The partition mounted perfectly well, and when I 
ran fsck, it too reported no corruption.

I recall that the "Mount unclean image to replay log" error was present 
through reboots and live ISOs. Perhaps that's the culprit?

Thanks, sincerely.


_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] Resize metadata corruption

[Chao Yu via Linux-f2fs-devel]

On 4/11/25 23:49, uplinkr@airmail.cc wrote:
> Hello everyone,
> 
> I'm sorry to not have anything constructive to add, but I'm about as flabbergasted as I could be. I retraced all my steps (exactly as I went through with them before!) and was unable to replicate the
> corruption. On the contrary, the filesystem grew completely as expected.

Hi uplinkr,

Thanks a lot for helping to reproduce this bug.

> 
> After restoring my partition and backing it up, I ran fsck on it. fsck reported no issues with the data, so I pressed on. I opened up GParted and expanded it like last time. This time, however, I
> didn't end up with no "Mount unclean image to replay log" error, and the resize went through as expected. The partition mounted perfectly well, and when I ran fsck, it too reported no corruption.
> 
> I recall that the "Mount unclean image to replay log" error was present through reboots and live ISOs. Perhaps that's the culprit?

	if (c.func != FSCK && c.func != DUMP && c.func != INJECT &&
		!is_set_ckpt_flags(F2FS_CKPT(sbi), CP_UMOUNT_FLAG)) {
		ERR_MSG("Mount unclean image to replay log first\n");
		return -1;
	}

If we run resize on an uncleaned image (suffer sudden power cut), it will
reports "Mount unclean image to replay log first" message, once you mount
this image, and umount it normally, then we can run resize on it successfully.

So I doubt it's not the key point of resize bug, one thing I suspect more
is space usage, as Juhyung reported:

"One thing to note here is that he said that the utilization was very
full: only 8G left. Maybe this is the corner case that we need to look
out for?"

Maybe, we can try resizing w/ almost fully used image to check whether
there is any corner case we missed to handle.

Please let me know, if you find more special status of your data.

Thanks,

> 
> Thanks, sincerely.



_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] Resize metadata corruption

[Chao Yu via Linux-f2fs-devel]

On 4/6/25 16:04, uplinkr--- via Linux-f2fs-devel wrote:
> Hello everyone,
> 
> I am having trouble with F2FS. Specifically, I believe metadata got corrupted when I resized it. I have a 512 GB drive. My F2FS partition was approximately located on 369-497 GB (128 GB size). Using
> GParted, I resized it to 0.5-497 GB. While the partition resizing went through successfully, filesystem resizing initially failed with "Mount unclean image to replay log". I have done that and retried
> resizing.
> 
> Afterwards, however, fsck started giving out a lot of errors, at one point it asked if I wished to restore lost data, which I agreed to. Logs specified a lot of my files (which I could tell by
> filenames) and mid-way through the process, it segfaulted. Now, when I run fsck, no files are asked to be restored, and it completes successfully. However, when I attempt to mount it, I get an error
> saying "Structure needs cleaning".
> 
> Could someone help me restore my metadata (at least, long enough to extract my files)? Thanks.
> 
> dmesg logs:
> 
> [   96.184127] F2FS-fs (nvme0n1p5): Mismatch valid blocks 769 vs. 68
> [   96.188050] F2FS-fs (nvme0n1p5): Failed to initialize F2FS segment manager (-117)
> 
> fsck.f2fs --dry-run -d 3 logs: https://arter.com/.f2fs-20250406/fsck.log

Hello,

Could you please upload fsck.log in bugzilla.kernel.org, or somewhere else?
I can not download it from above link address, sorry.

Thanks,

> 
> 
> _______________________________________________
> Linux-f2fs-devel mailing list
> Linux-f2fs-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel



_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] Resize metadata corruption

[uplinkr--- via Linux-f2fs-devel]

Hello,

I made a typo in the URL earlier. It should be 
https://arter97.com/.f2fs-20250406/fsck.log . Could you try it, please?

Thanks.

On 2025-04-08 08:43, Chao Yu wrote:
> On 4/6/25 16:04, uplinkr--- via Linux-f2fs-devel wrote:
>> Hello everyone,
>> 
>> I am having trouble with F2FS. Specifically, I believe metadata got 
>> corrupted when I resized it. I have a 512 GB drive. My F2FS partition 
>> was approximately located on 369-497 GB (128 GB size). Using
>> GParted, I resized it to 0.5-497 GB. While the partition resizing went 
>> through successfully, filesystem resizing initially failed with "Mount 
>> unclean image to replay log". I have done that and retried
>> resizing.
>> 
>> Afterwards, however, fsck started giving out a lot of errors, at one 
>> point it asked if I wished to restore lost data, which I agreed to. 
>> Logs specified a lot of my files (which I could tell by
>> filenames) and mid-way through the process, it segfaulted. Now, when I 
>> run fsck, no files are asked to be restored, and it completes 
>> successfully. However, when I attempt to mount it, I get an error
>> saying "Structure needs cleaning".
>> 
>> Could someone help me restore my metadata (at least, long enough to 
>> extract my files)? Thanks.
>> 
>> dmesg logs:
>> 
>> [   96.184127] F2FS-fs (nvme0n1p5): Mismatch valid blocks 769 vs. 68
>> [   96.188050] F2FS-fs (nvme0n1p5): Failed to initialize F2FS segment 
>> manager (-117)
>> 
>> fsck.f2fs --dry-run -d 3 logs: 
>> https://arter.com/.f2fs-20250406/fsck.log
> 
> Hello,
> 
> Could you please upload fsck.log in bugzilla.kernel.org, or somewhere 
> else?
> I can not download it from above link address, sorry.
> 
> Thanks,
> 
>> 
>> 
>> _______________________________________________
>> Linux-f2fs-devel mailing list
>> Linux-f2fs-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel


_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] Resize metadata corruption

[Chao Yu via Linux-f2fs-devel]

On 4/8/25 14:10, uplinkr@airmail.cc wrote:
> Hello,
> 
> I made a typo in the URL earlier. It should be https://arter97.com/.f2fs-20250406/fsck.log . Could you try it, please?

Ah, alright.

Now, I can access it, thank you. Let me check the details in the log.

Thanks,

> 
> Thanks.
> 
> On 2025-04-08 08:43, Chao Yu wrote:
>> On 4/6/25 16:04, uplinkr--- via Linux-f2fs-devel wrote:
>>> Hello everyone,
>>>
>>> I am having trouble with F2FS. Specifically, I believe metadata got corrupted when I resized it. I have a 512 GB drive. My F2FS partition was approximately located on 369-497 GB (128 GB size). Using
>>> GParted, I resized it to 0.5-497 GB. While the partition resizing went through successfully, filesystem resizing initially failed with "Mount unclean image to replay log". I have done that and retried
>>> resizing.
>>>
>>> Afterwards, however, fsck started giving out a lot of errors, at one point it asked if I wished to restore lost data, which I agreed to. Logs specified a lot of my files (which I could tell by
>>> filenames) and mid-way through the process, it segfaulted. Now, when I run fsck, no files are asked to be restored, and it completes successfully. However, when I attempt to mount it, I get an error
>>> saying "Structure needs cleaning".
>>>
>>> Could someone help me restore my metadata (at least, long enough to extract my files)? Thanks.
>>>
>>> dmesg logs:
>>>
>>> [   96.184127] F2FS-fs (nvme0n1p5): Mismatch valid blocks 769 vs. 68
>>> [   96.188050] F2FS-fs (nvme0n1p5): Failed to initialize F2FS segment manager (-117)
>>>
>>> fsck.f2fs --dry-run -d 3 logs: https://arter.com/.f2fs-20250406/fsck.log
>>
>> Hello,
>>
>> Could you please upload fsck.log in bugzilla.kernel.org, or somewhere else?
>> I can not download it from above link address, sorry.
>>
>> Thanks,
>>
>>>
>>>
>>> _______________________________________________
>>> Linux-f2fs-devel mailing list
>>> Linux-f2fs-devel@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel



_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel