[f2fs-dev] Reproducing resize corruption

[f2fs-dev] Reproducing resize corruption

[Juhyung Park]

Hi everyone.

Besides salvaging @uplinkr's data [1], I figured it's important for us
to understand why resize corrupts data in the first place.

I took some time today to have my laptop's 1.8TiB f2fs partition
resized slightly and managed to reproduce the data corruption.

I'm not necessarily sure whether this would be the same symptoms as
@uplinkr's, but it's probably likely.

Here's what I did:
1. Remounted to checkpoint=disable
2. Create a dm-snapshot of the current root device
3. Mount snapshot to replay the log
4. Unmount
5. Resize sector 487248896 to 486539264
# ./resize.f2fs -d 3 -s -i /dev/mapper/snap -t 486539264

Latest f2fs-tools was used:
33c5b9539af2 f2fs_io: add fragread command to evaluate fragmented
buffer for reads

This reproduced the mount and fsck failure.

Mount log:
[442431.020594] F2FS-fs (dm-2): invalid crc_offset: 0
[442431.130055] F2FS-fs (dm-2): SIT is corrupted node# 13615201 vs 13616290
[442431.139684] F2FS-fs (dm-2): Failed to initialize F2FS segment manager (-117)

debugfs & resize log:
https://arter97.com/.f2fs-20250410/

The fsck log was way too large, 8.21GiB without "-d" flag and it
contained many sensitive data, so I'm not uploading it for now.

From looking at the dm stats, the fsck also wrote 2138 MiB to the
snapshot device.

Chao, can we start from here?
Thanks.

[1] https://lore.kernel.org/linux-f2fs-devel/608f23ce-56ef-4faf-bee1-3aa89786ed41@kernel.org/T/#mc628f8f3ca6c73178ffa1716f927755527856d4b


_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] Reproducing resize corruption

[Chao Yu via Linux-f2fs-devel]

On 2025/4/10 16:32, Juhyung Park wrote:
> Hi everyone.
> 
> Besides salvaging @uplinkr's data [1], I figured it's important for us
> to understand why resize corrupts data in the first place.

Yes, I feel the same with you.

> 
> I took some time today to have my laptop's 1.8TiB f2fs partition
> resized slightly and managed to reproduce the data corruption.
> 
> I'm not necessarily sure whether this would be the same symptoms as
> @uplinkr's, but it's probably likely.

Thanks for the quick reproducing, and good catch!

> 
> Here's what I did:
> 1. Remounted to checkpoint=disable
> 2. Create a dm-snapshot of the current root device
> 3. Mount snapshot to replay the log
> 4. Unmount
> 5. Resize sector 487248896 to 486539264

@uplinkr's case is expanding partition, not the same w/ shrink one.

> # ./resize.f2fs -d 3 -s -i /dev/mapper/snap -t 486539264

Also, I didn't see large_nat_bitmap flag in his fsck.log.

> 
> Latest f2fs-tools was used:
> 33c5b9539af2 f2fs_io: add fragread command to evaluate fragmented
> buffer for reads
> 
> This reproduced the mount and fsck failure.
> 
> Mount log:
> [442431.020594] F2FS-fs (dm-2): invalid crc_offset: 0
> [442431.130055] F2FS-fs (dm-2): SIT is corrupted node# 13615201 vs 13616290
> [442431.139684] F2FS-fs (dm-2): Failed to initialize F2FS segment manager (-117)

I tried:

a) resize.f2fs -d 3 -s -i img -t $sectors
b) resize.f2fs -d 3 -s img -t $sectors

Only a) can reproduce the bug, we need to revert -i support for resize
first.

Thanks,

> 
> debugfs & resize log:
> https://arter97.com/.f2fs-20250410/
> 
> The fsck log was way too large, 8.21GiB without "-d" flag and it
> contained many sensitive data, so I'm not uploading it for now.
> 
>  From looking at the dm stats, the fsck also wrote 2138 MiB to the
> snapshot device.
> 
> Chao, can we start from here?
> Thanks.
> 
> [1] https://lore.kernel.org/linux-f2fs-devel/608f23ce-56ef-4faf-bee1-3aa89786ed41@kernel.org/T/#mc628f8f3ca6c73178ffa1716f927755527856d4b



_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] Reproducing resize corruption

[Juhyung Park]

Hi Chao,

On Thu, Apr 10, 2025 at 7:56 AM Chao Yu <chao@kernel.org> wrote:
>
> On 2025/4/10 16:32, Juhyung Park wrote:
> > Hi everyone.
> >
> > Besides salvaging @uplinkr's data [1], I figured it's important for us
> > to understand why resize corrupts data in the first place.
>
> Yes, I feel the same with you.
>
> >
> > I took some time today to have my laptop's 1.8TiB f2fs partition
> > resized slightly and managed to reproduce the data corruption.
> >
> > I'm not necessarily sure whether this would be the same symptoms as
> > @uplinkr's, but it's probably likely.
>
> Thanks for the quick reproducing, and good catch!
>
> >
> > Here's what I did:
> > 1. Remounted to checkpoint=disable
> > 2. Create a dm-snapshot of the current root device
> > 3. Mount snapshot to replay the log
> > 4. Unmount
> > 5. Resize sector 487248896 to 486539264
>
> @uplinkr's case is expanding partition, not the same w/ shrink one.

It must have gone through shrink and expand, that's how gparted works iirc.

To be specific, his case was expanding the partition to a different
offset. To reduce time required for this, gparted first shrinks the
partition as much as it can inplace, and then move it to the new
offset, and then expand it back to full size. Maybe this scenario
produces some bug?

>
> > # ./resize.f2fs -d 3 -s -i /dev/mapper/snap -t 486539264
>
> Also, I didn't see large_nat_bitmap flag in his fsck.log.
>
> >
> > Latest f2fs-tools was used:
> > 33c5b9539af2 f2fs_io: add fragread command to evaluate fragmented
> > buffer for reads
> >
> > This reproduced the mount and fsck failure.
> >
> > Mount log:
> > [442431.020594] F2FS-fs (dm-2): invalid crc_offset: 0
> > [442431.130055] F2FS-fs (dm-2): SIT is corrupted node# 13615201 vs 13616290
> > [442431.139684] F2FS-fs (dm-2): Failed to initialize F2FS segment manager (-117)
>
> I tried:
>
> a) resize.f2fs -d 3 -s -i img -t $sectors
> b) resize.f2fs -d 3 -s img -t $sectors
>
> Only a) can reproduce the bug, we need to revert -i support for resize
> first.

Ugh, I suspected this could be the determining factor :(

So @uplinkr's case is a different culprit one. I'll try to reproduce
something with my phone which doesn't use large_nat_bitmap.

>
> Thanks,
>
> >
> > debugfs & resize log:
> > https://arter97.com/.f2fs-20250410/
> >
> > The fsck log was way too large, 8.21GiB without "-d" flag and it
> > contained many sensitive data, so I'm not uploading it for now.
> >
> >  From looking at the dm stats, the fsck also wrote 2138 MiB to the
> > snapshot device.
> >
> > Chao, can we start from here?
> > Thanks.
> >
> > [1] https://lore.kernel.org/linux-f2fs-devel/608f23ce-56ef-4faf-bee1-3aa89786ed41@kernel.org/T/#mc628f8f3ca6c73178ffa1716f927755527856d4b
>


_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] Reproducing resize corruption

[Chao Yu via Linux-f2fs-devel]

On 2025/4/11 3:08, Juhyung Park wrote:
> Hi Chao,
> 
> On Thu, Apr 10, 2025 at 7:56 AM Chao Yu <chao@kernel.org> wrote:
>>
>> On 2025/4/10 16:32, Juhyung Park wrote:
>>> Hi everyone.
>>>
>>> Besides salvaging @uplinkr's data [1], I figured it's important for us
>>> to understand why resize corrupts data in the first place.
>>
>> Yes, I feel the same with you.
>>
>>>
>>> I took some time today to have my laptop's 1.8TiB f2fs partition
>>> resized slightly and managed to reproduce the data corruption.
>>>
>>> I'm not necessarily sure whether this would be the same symptoms as
>>> @uplinkr's, but it's probably likely.
>>
>> Thanks for the quick reproducing, and good catch!
>>
>>>
>>> Here's what I did:
>>> 1. Remounted to checkpoint=disable
>>> 2. Create a dm-snapshot of the current root device
>>> 3. Mount snapshot to replay the log
>>> 4. Unmount
>>> 5. Resize sector 487248896 to 486539264
>>
>> @uplinkr's case is expanding partition, not the same w/ shrink one.
> 
> It must have gone through shrink and expand, that's how gparted works iirc.
> 
> To be specific, his case was expanding the partition to a different
> offset. To reduce time required for this, gparted first shrinks the
> partition as much as it can inplace, and then move it to the new
> offset, and then expand it back to full size. Maybe this scenario
> produces some bug?
> 
>>
>>> # ./resize.f2fs -d 3 -s -i /dev/mapper/snap -t 486539264
>>
>> Also, I didn't see large_nat_bitmap flag in his fsck.log.
>>
>>>
>>> Latest f2fs-tools was used:
>>> 33c5b9539af2 f2fs_io: add fragread command to evaluate fragmented
>>> buffer for reads
>>>
>>> This reproduced the mount and fsck failure.
>>>
>>> Mount log:
>>> [442431.020594] F2FS-fs (dm-2): invalid crc_offset: 0
>>> [442431.130055] F2FS-fs (dm-2): SIT is corrupted node# 13615201 vs 13616290
>>> [442431.139684] F2FS-fs (dm-2): Failed to initialize F2FS segment manager (-117)
>>
>> I tried:
>>
>> a) resize.f2fs -d 3 -s -i img -t $sectors
>> b) resize.f2fs -d 3 -s img -t $sectors

c) resize.f2fs -d 3 -i img -t $sectors (expanding), the root cause seems
conflict between -i and -s option in shrink mode resize.

Thanks,

>>
>> Only a) can reproduce the bug, we need to revert -i support for resize
>> first.
> 
> Ugh, I suspected this could be the determining factor :(
> 
> So @uplinkr's case is a different culprit one. I'll try to reproduce
> something with my phone which doesn't use large_nat_bitmap.
> 
>>
>> Thanks,
>>
>>>
>>> debugfs & resize log:
>>> https://arter97.com/.f2fs-20250410/
>>>
>>> The fsck log was way too large, 8.21GiB without "-d" flag and it
>>> contained many sensitive data, so I'm not uploading it for now.
>>>
>>>   From looking at the dm stats, the fsck also wrote 2138 MiB to the
>>> snapshot device.
>>>
>>> Chao, can we start from here?
>>> Thanks.
>>>
>>> [1] https://lore.kernel.org/linux-f2fs-devel/608f23ce-56ef-4faf-bee1-3aa89786ed41@kernel.org/T/#mc628f8f3ca6c73178ffa1716f927755527856d4b
>>



_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

Re: [f2fs-dev] Reproducing resize corruption

[Chao Yu via Linux-f2fs-devel]

On 2025/4/11 15:08, Chao Yu wrote:
> On 2025/4/11 3:08, Juhyung Park wrote:
>> Hi Chao,
>>
>> On Thu, Apr 10, 2025 at 7:56 AM Chao Yu <chao@kernel.org> wrote:
>>>
>>> On 2025/4/10 16:32, Juhyung Park wrote:
>>>> Hi everyone.
>>>>
>>>> Besides salvaging @uplinkr's data [1], I figured it's important for us
>>>> to understand why resize corrupts data in the first place.
>>>
>>> Yes, I feel the same with you.
>>>
>>>>
>>>> I took some time today to have my laptop's 1.8TiB f2fs partition
>>>> resized slightly and managed to reproduce the data corruption.
>>>>
>>>> I'm not necessarily sure whether this would be the same symptoms as
>>>> @uplinkr's, but it's probably likely.
>>>
>>> Thanks for the quick reproducing, and good catch!
>>>
>>>>
>>>> Here's what I did:
>>>> 1. Remounted to checkpoint=disable
>>>> 2. Create a dm-snapshot of the current root device
>>>> 3. Mount snapshot to replay the log
>>>> 4. Unmount
>>>> 5. Resize sector 487248896 to 486539264
>>>
>>> @uplinkr's case is expanding partition, not the same w/ shrink one.
>>
>> It must have gone through shrink and expand, that's how gparted works iirc.
>>
>> To be specific, his case was expanding the partition to a different
>> offset. To reduce time required for this, gparted first shrinks the
>> partition as much as it can inplace, and then move it to the new
>> offset, and then expand it back to full size. Maybe this scenario
>> produces some bug?
>>
>>>
>>>> # ./resize.f2fs -d 3 -s -i /dev/mapper/snap -t 486539264
>>>
>>> Also, I didn't see large_nat_bitmap flag in his fsck.log.
>>>
>>>>
>>>> Latest f2fs-tools was used:
>>>> 33c5b9539af2 f2fs_io: add fragread command to evaluate fragmented
>>>> buffer for reads
>>>>
>>>> This reproduced the mount and fsck failure.
>>>>
>>>> Mount log:
>>>> [442431.020594] F2FS-fs (dm-2): invalid crc_offset: 0
>>>> [442431.130055] F2FS-fs (dm-2): SIT is corrupted node# 13615201 vs 13616290
>>>> [442431.139684] F2FS-fs (dm-2): Failed to initialize F2FS segment manager (-117)
>>>
>>> I tried:
>>>
>>> a) resize.f2fs -d 3 -s -i img -t $sectors
>>> b) resize.f2fs -d 3 -s img -t $sectors
> 
> c) resize.f2fs -d 3 -i img -t $sectors (expanding), the root cause seems
> conflict between -i and -s option in shrink mode resize.

Sorry,

c) resize.f2fs -d 3 -i img -t $sectors (expanding) won't trigger the bug.

Thanks,

> 
> Thanks,
> 
>>>
>>> Only a) can reproduce the bug, we need to revert -i support for resize
>>> first.
>>
>> Ugh, I suspected this could be the determining factor :(
>>
>> So @uplinkr's case is a different culprit one. I'll try to reproduce
>> something with my phone which doesn't use large_nat_bitmap.
>>
>>>
>>> Thanks,
>>>
>>>>
>>>> debugfs & resize log:
>>>> https://arter97.com/.f2fs-20250410/
>>>>
>>>> The fsck log was way too large, 8.21GiB without "-d" flag and it
>>>> contained many sensitive data, so I'm not uploading it for now.
>>>>
>>>>    From looking at the dm stats, the fsck also wrote 2138 MiB to the
>>>> snapshot device.
>>>>
>>>> Chao, can we start from here?
>>>> Thanks.
>>>>
>>>> [1] https://lore.kernel.org/linux-f2fs-devel/608f23ce-56ef-4faf-bee1-3aa89786ed41@kernel.org/T/#mc628f8f3ca6c73178ffa1716f927755527856d4b
>>>
> 



_______________________________________________
Linux-f2fs-devel mailing list
Linux-f2fs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel