Sector size changes creating filesystem problems

Sector size changes creating filesystem problems

[Filipe Maia]

Hi,

When a 4Kn disk is added to an mdadm array with sector size 512, its
sector size changes to 4096 to accommodate the new disk.

Here's an example:

```
truncate -s 1G /tmp/loop512a
truncate -s 1G /tmp/loop512b
truncate -s 1G /tmp/loop512c
truncate -s 1G /tmp/loop4Ka
losetup --sector-size 512  --direct-io=on /dev/loop0  /tmp/loop512a
losetup --sector-size 512  --direct-io=on /dev/loop1  /tmp/loop512b
losetup --sector-size 512  --direct-io=on /dev/loop2  /tmp/loop512c
losetup --sector-size 4096  --direct-io=on /dev/loop3  /tmp/loop4Ka
mdadm --create /dev/md2 --level=5 --raid-devices=3 /dev/loop[0-2]
# blockdev returns 512
blockdev --getss /dev/md2
mdadm /dev/md2 -a /dev/loop3
mdadm /dev/md2 -f /dev/loop2
# blockdev still returns 512
blockdev --getss /dev/md2
mdadm -S /dev/md2
mdadm -A /dev/md2 /dev/loop0 /dev/loop1 /dev/loop3
# blockdev now returns 4096
blockdev --getss /dev/md2
```

This breaks filesystems like XFS, with new mounts failing with:
`mount: /mnt: mount(2) system call failed: Function not implemented.`

Shouldn't the user be warned when this can happen?

Cheers,
Filipe

Re: Sector size changes creating filesystem problems

[Yu Kuai]

Hi,

在 2025/7/16 22:30, Filipe Maia 写道:
> Hi,
>
> When a 4Kn disk is added to an mdadm array with sector size 512, its
> sector size changes to 4096 to accommodate the new disk.
>
> Here's an example:
>
> ```
> truncate -s 1G /tmp/loop512a
> truncate -s 1G /tmp/loop512b
> truncate -s 1G /tmp/loop512c
> truncate -s 1G /tmp/loop4Ka
> losetup --sector-size 512  --direct-io=on /dev/loop0  /tmp/loop512a
> losetup --sector-size 512  --direct-io=on /dev/loop1  /tmp/loop512b
> losetup --sector-size 512  --direct-io=on /dev/loop2  /tmp/loop512c
> losetup --sector-size 4096  --direct-io=on /dev/loop3  /tmp/loop4Ka
> mdadm --create /dev/md2 --level=5 --raid-devices=3 /dev/loop[0-2]
> # blockdev returns 512
> blockdev --getss /dev/md2
> mdadm /dev/md2 -a /dev/loop3
> mdadm /dev/md2 -f /dev/loop2
> # blockdev still returns 512
> blockdev --getss /dev/md2
> mdadm -S /dev/md2
> mdadm -A /dev/md2 /dev/loop0 /dev/loop1 /dev/loop3
> # blockdev now returns 4096
> blockdev --getss /dev/md2
> ```
>
> This breaks filesystems like XFS, with new mounts failing with:
> `mount: /mnt: mount(2) system call failed: Function not implemented.`
Yes, this is a known problem. Li Nan was trying to fix this long time ago.
>
> Shouldn't the user be warned when this can happen?
Kernel should forbid larger lbs new disk, however, this is still a missing
feature for now.

Thanks,

Kuai
>
> Cheers,
> Filipe
>

Re: Sector size changes creating filesystem problems

[Roman Mamedov]

On Wed, 16 Jul 2025 15:30:20 +0100
Filipe Maia <filipe.c.maia@gmail.com> wrote:

> Hi,
> 
> When a 4Kn disk is added to an mdadm array with sector size 512, its
> sector size changes to 4096 to accommodate the new disk.
> 
> Here's an example:
> 
> ```
> truncate -s 1G /tmp/loop512a
> truncate -s 1G /tmp/loop512b
> truncate -s 1G /tmp/loop512c
> truncate -s 1G /tmp/loop4Ka
> losetup --sector-size 512  --direct-io=on /dev/loop0  /tmp/loop512a
> losetup --sector-size 512  --direct-io=on /dev/loop1  /tmp/loop512b
> losetup --sector-size 512  --direct-io=on /dev/loop2  /tmp/loop512c
> losetup --sector-size 4096  --direct-io=on /dev/loop3  /tmp/loop4Ka
> mdadm --create /dev/md2 --level=5 --raid-devices=3 /dev/loop[0-2]
> # blockdev returns 512
> blockdev --getss /dev/md2
> mdadm /dev/md2 -a /dev/loop3
> mdadm /dev/md2 -f /dev/loop2
> # blockdev still returns 512
> blockdev --getss /dev/md2
> mdadm -S /dev/md2
> mdadm -A /dev/md2 /dev/loop0 /dev/loop1 /dev/loop3
> # blockdev now returns 4096
> blockdev --getss /dev/md2
> ```
> 
> This breaks filesystems like XFS, with new mounts failing with:
> `mount: /mnt: mount(2) system call failed: Function not implemented.`

If you dd the XFS image from an old 512b disk onto a newly bought large
4K-sector HDD, would it also stop mounting on the new disk in the same way?

Perhaps something to be improved on the XFS side?

-- 
With respect,
Roman

Re: Sector size changes creating filesystem problems

[Filipe Maia]

> If you dd the XFS image from an old 512b disk onto a newly bought large
> 4K-sector HDD, would it also stop mounting on the new disk in the same way?
Yes, it causes exactly the same problem.
>
> Perhaps something to be improved on the XFS side?
Indeed!

Cheers,
Filipe

Re: Sector size changes creating filesystem problems

[Yu Kuai]

+CC Li Nan

在 2025/07/17 0:21, Yu Kuai 写道:
> Hi,
> 
> 在 2025/7/16 22:30, Filipe Maia 写道:
>> Hi,
>>
>> When a 4Kn disk is added to an mdadm array with sector size 512, its
>> sector size changes to 4096 to accommodate the new disk.
>>
>> Here's an example:
>>
>> ```
>> truncate -s 1G /tmp/loop512a
>> truncate -s 1G /tmp/loop512b
>> truncate -s 1G /tmp/loop512c
>> truncate -s 1G /tmp/loop4Ka
>> losetup --sector-size 512  --direct-io=on /dev/loop0  /tmp/loop512a
>> losetup --sector-size 512  --direct-io=on /dev/loop1  /tmp/loop512b
>> losetup --sector-size 512  --direct-io=on /dev/loop2  /tmp/loop512c
>> losetup --sector-size 4096  --direct-io=on /dev/loop3  /tmp/loop4Ka
>> mdadm --create /dev/md2 --level=5 --raid-devices=3 /dev/loop[0-2]
>> # blockdev returns 512
>> blockdev --getss /dev/md2
>> mdadm /dev/md2 -a /dev/loop3
>> mdadm /dev/md2 -f /dev/loop2
>> # blockdev still returns 512
>> blockdev --getss /dev/md2
>> mdadm -S /dev/md2
>> mdadm -A /dev/md2 /dev/loop0 /dev/loop1 /dev/loop3
>> # blockdev now returns 4096
>> blockdev --getss /dev/md2
>> ```
>>
>> This breaks filesystems like XFS, with new mounts failing with:
>> `mount: /mnt: mount(2) system call failed: Function not implemented.`
> Yes, this is a known problem. Li Nan was trying to fix this long time ago.
>>
>> Shouldn't the user be warned when this can happen?
> Kernel should forbid larger lbs new disk, however, this is still a missing
> feature for now.

You can try this set:

https://lore.kernel.org/all/20250304121918.3159388-1-linan666@huaweicloud.com/

> 
> Thanks,
> 
> Kuai
>>
>> Cheers,
>> Filipe
>>
> 
> .
> 

Re: Sector size changes creating filesystem problems

[Christoph Hellwig]

On Wed, Jul 16, 2025 at 10:10:03PM +0500, Roman Mamedov wrote:
> If you dd the XFS image from an old 512b disk onto a newly bought large
> 4K-sector HDD, would it also stop mounting on the new disk in the same way?
> 
> Perhaps something to be improved on the XFS side?

Nothing to be improved, as the sector size obviously matters.  It is
part of the hardware geometry that is fixed, as XFS can read and write
down to the sector size boundary.  Both for internal metadata, and
through direct I/O also for applications.  The latter is true for other
file systems to, so even when you can change the sector size for some
file systems, you will be subtly breaking applications instead.

Re: Sector size changes creating filesystem problems

[Martin K. Petersen]

>> Shouldn't the user be warned when this can happen?

> Kernel should forbid larger lbs new disk,

This is really something mdadm should validate prior to adding a new
device to an existing array.

-- 
Martin K. Petersen