Executable loading issues with erofs on arm?

Executable loading issues with erofs on arm?

[Jan Kiszka]

Hi all,

for some days, I'm trying to understand if we have an integration issue
with erofs or rather some upstream bug. After playing with various
parameters, it rather looks like the latter:

$ ls -l erofs-dir/
total 132
-rwxr-xr-x 1 1000 users 132868 Jul  8 10:50 dash
(from Debian bookworm)
$ mkfs.erofs -z lz4hc erofs.img erofs-dir/
mkfs.erofs 1.8.6 (trixie version, but same happens with bookworm 1.5)
Build completed.
------
Filesystem UUID: aae0b2f0-4ee4-4850-af49-3c1aad7fa30c
Filesystem total blocks: 17 (of 4096-byte blocks)
Filesystem total inodes: 2
Filesystem total metadata blocks: 1
Filesystem total deduplicated bytes (of source files): 0

Now I have 6.15-rc5 and a defconfig-close setting for the 32-bit ARM
target BeagleBone Black. When booting into init=/bin/sh, then running

# mount -t erofs /dev/mmcblk0p1 /mnt
erofs (device mmcblk0p1): mounted with root inode @ nid 36.
# /mnt/dash
Segmentation fault

I once also got this:

Alignment trap: not handling instruction 2b00 at [<004debc0>]
8<--- cut here ---
Unhandled fault: alignment exception (0x001) at 0x000004d9
[000004d9] *pgd=00000000
Bus error

All is fine if I
 - run the command once more
 - dump the file first (cat /mnt/dash > /dev/null; /mnt/dash)
 - boot a full Debian system and then mount & run the command
 - do not compress erofs

Also broken is -z zstd, so the decompression algorithm itself should not
be the reason. I furthermore tested older kernels as well, namely
stable-derived 6.1-cip and 6.12-cip, and those are equally affected.

Any ideas? I have CONFIG_EROFS_FS_DEBUG=y, but that does not trigger
anything. Is there anything I could instrument?

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

Re: Executable loading issues with erofs on arm?

[Jan Kiszka]

On 08.07.25 14:41, Jan Kiszka wrote:
> Hi all,
> 
> for some days, I'm trying to understand if we have an integration issue
> with erofs or rather some upstream bug. After playing with various
> parameters, it rather looks like the latter:
> 
> $ ls -l erofs-dir/
> total 132
> -rwxr-xr-x 1 1000 users 132868 Jul  8 10:50 dash
> (from Debian bookworm)
> $ mkfs.erofs -z lz4hc erofs.img erofs-dir/
> mkfs.erofs 1.8.6 (trixie version, but same happens with bookworm 1.5)
> Build completed.
> ------
> Filesystem UUID: aae0b2f0-4ee4-4850-af49-3c1aad7fa30c
> Filesystem total blocks: 17 (of 4096-byte blocks)
> Filesystem total inodes: 2
> Filesystem total metadata blocks: 1
> Filesystem total deduplicated bytes (of source files): 0
> 
> Now I have 6.15-rc5 and a defconfig-close setting for the 32-bit ARM
> target BeagleBone Black. When booting into init=/bin/sh, then running
> 
> # mount -t erofs /dev/mmcblk0p1 /mnt
> erofs (device mmcblk0p1): mounted with root inode @ nid 36.
> # /mnt/dash
> Segmentation fault
> 
> I once also got this:
> 
> Alignment trap: not handling instruction 2b00 at [<004debc0>]
> 8<--- cut here ---
> Unhandled fault: alignment exception (0x001) at 0x000004d9
> [000004d9] *pgd=00000000
> Bus error
> 
> All is fine if I
>  - run the command once more
>  - dump the file first (cat /mnt/dash > /dev/null; /mnt/dash)

Forgot to mention: That first dump when done via md5sum or so actually
gives the right checksum. So pure reading of the binary is also ok, just
trying to load it for execution fails on the first attempt.

Jan

>  - boot a full Debian system and then mount & run the command
>  - do not compress erofs
> 
> Also broken is -z zstd, so the decompression algorithm itself should not
> be the reason. I furthermore tested older kernels as well, namely
> stable-derived 6.1-cip and 6.12-cip, and those are equally affected.
> 
> Any ideas? I have CONFIG_EROFS_FS_DEBUG=y, but that does not trigger
> anything. Is there anything I could instrument?
> 
> Jan
> 

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

Re: Executable loading issues with erofs on arm?

[Gao Xiang]

Hi Jan,

On 2025/7/8 20:43, Jan Kiszka wrote:
> On 08.07.25 14:41, Jan Kiszka wrote:
>> Hi all,
>>
>> for some days, I'm trying to understand if we have an integration issue
>> with erofs or rather some upstream bug. After playing with various
>> parameters, it rather looks like the latter:
>>
>> $ ls -l erofs-dir/
>> total 132
>> -rwxr-xr-x 1 1000 users 132868 Jul  8 10:50 dash
>> (from Debian bookworm)
>> $ mkfs.erofs -z lz4hc erofs.img erofs-dir/
>> mkfs.erofs 1.8.6 (trixie version, but same happens with bookworm 1.5)
>> Build completed.
>> ------
>> Filesystem UUID: aae0b2f0-4ee4-4850-af49-3c1aad7fa30c
>> Filesystem total blocks: 17 (of 4096-byte blocks)
>> Filesystem total inodes: 2
>> Filesystem total metadata blocks: 1
>> Filesystem total deduplicated bytes (of source files): 0
>>
>> Now I have 6.15-rc5 and a defconfig-close setting for the 32-bit ARM
>> target BeagleBone Black. When booting into init=/bin/sh, then running
>>
>> # mount -t erofs /dev/mmcblk0p1 /mnt
>> erofs (device mmcblk0p1): mounted with root inode @ nid 36.
>> # /mnt/dash
>> Segmentation fault
>>
>> I once also got this:
>>
>> Alignment trap: not handling instruction 2b00 at [<004debc0>]
>> 8<--- cut here ---
>> Unhandled fault: alignment exception (0x001) at 0x000004d9
>> [000004d9] *pgd=00000000
>> Bus error
>>
>> All is fine if I
>>   - run the command once more
>>   - dump the file first (cat /mnt/dash > /dev/null; /mnt/dash)
> 
> Forgot to mention: That first dump when done via md5sum or so actually
> gives the right checksum. So pure reading of the binary is also ok, just
> trying to load it for execution fails on the first attempt.

Thanks for your report.  I rarely take care arm32 platform
because I don't have such setup.

but could you share a reproducible rootfs image and
I wonder if qemu could reproduce this?

Otherwise it's hard for me to debug this issue...

Thanks,
Gao Xiang

> 
> Jan
> 
>>   - boot a full Debian system and then mount & run the command
>>   - do not compress erofs
>>
>> Also broken is -z zstd, so the decompression algorithm itself should not
>> be the reason. I furthermore tested older kernels as well, namely
>> stable-derived 6.1-cip and 6.12-cip, and those are equally affected.
>>
>> Any ideas? I have CONFIG_EROFS_FS_DEBUG=y, but that does not trigger
>> anything. Is there anything I could instrument?
>>
>> Jan
>>
> 

Re: Executable loading issues with erofs on arm?

[Jan Kiszka]

On 08.07.25 17:12, Gao Xiang wrote:
> Hi Jan,
> 
> On 2025/7/8 20:43, Jan Kiszka wrote:
>> On 08.07.25 14:41, Jan Kiszka wrote:
>>> Hi all,
>>>
>>> for some days, I'm trying to understand if we have an integration issue
>>> with erofs or rather some upstream bug. After playing with various
>>> parameters, it rather looks like the latter:
>>>
>>> $ ls -l erofs-dir/
>>> total 132
>>> -rwxr-xr-x 1 1000 users 132868 Jul  8 10:50 dash
>>> (from Debian bookworm)
>>> $ mkfs.erofs -z lz4hc erofs.img erofs-dir/
>>> mkfs.erofs 1.8.6 (trixie version, but same happens with bookworm 1.5)
>>> Build completed.
>>> ------
>>> Filesystem UUID: aae0b2f0-4ee4-4850-af49-3c1aad7fa30c
>>> Filesystem total blocks: 17 (of 4096-byte blocks)
>>> Filesystem total inodes: 2
>>> Filesystem total metadata blocks: 1
>>> Filesystem total deduplicated bytes (of source files): 0
>>>
>>> Now I have 6.15-rc5 and a defconfig-close setting for the 32-bit ARM
>>> target BeagleBone Black. When booting into init=/bin/sh, then running
>>>
>>> # mount -t erofs /dev/mmcblk0p1 /mnt
>>> erofs (device mmcblk0p1): mounted with root inode @ nid 36.
>>> # /mnt/dash
>>> Segmentation fault
>>>
>>> I once also got this:
>>>
>>> Alignment trap: not handling instruction 2b00 at [<004debc0>]
>>> 8<--- cut here ---
>>> Unhandled fault: alignment exception (0x001) at 0x000004d9
>>> [000004d9] *pgd=00000000
>>> Bus error
>>>
>>> All is fine if I
>>>   - run the command once more
>>>   - dump the file first (cat /mnt/dash > /dev/null; /mnt/dash)
>>
>> Forgot to mention: That first dump when done via md5sum or so actually
>> gives the right checksum. So pure reading of the binary is also ok, just
>> trying to load it for execution fails on the first attempt.
> 
> Thanks for your report.  I rarely take care arm32 platform
> because I don't have such setup.
> 
> but could you share a reproducible rootfs image and
> I wonder if qemu could reproduce this?

The image can be generated from isar-cip-core
(https://gitlab.com/cip-project/cip-core/isar-cip-core), bbb image with
swupdate extension and erofs as immutable rootfs. As I wrote, those will
be 6.12 or 6.1 based, but I also injected a mainline kernel into that
with the same result. But all that only helps if you have some
beaglebone black in reach right now.

The same configuration, just for qemuarm as target, unfortunately does
not reproduce the issue.

> 
> Otherwise it's hard for me to debug this issue...

If you tell me how I could do that, I'm happy to instrument and analyze.
I just have no understanding of erofs yet, specifically how reading
files might be different from loading executables.

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

Re: Executable loading issues with erofs on arm?

[Gao Xiang]

On 2025/7/8 23:22, Jan Kiszka wrote:
> On 08.07.25 17:12, Gao Xiang wrote:
>> Hi Jan,
>>
>> On 2025/7/8 20:43, Jan Kiszka wrote:
>>> On 08.07.25 14:41, Jan Kiszka wrote:
>>>> Hi all,
>>>>
>>>> for some days, I'm trying to understand if we have an integration issue
>>>> with erofs or rather some upstream bug. After playing with various
>>>> parameters, it rather looks like the latter:
>>>>
>>>> $ ls -l erofs-dir/
>>>> total 132
>>>> -rwxr-xr-x 1 1000 users 132868 Jul  8 10:50 dash
>>>> (from Debian bookworm)
>>>> $ mkfs.erofs -z lz4hc erofs.img erofs-dir/
>>>> mkfs.erofs 1.8.6 (trixie version, but same happens with bookworm 1.5)
>>>> Build completed.
>>>> ------
>>>> Filesystem UUID: aae0b2f0-4ee4-4850-af49-3c1aad7fa30c
>>>> Filesystem total blocks: 17 (of 4096-byte blocks)
>>>> Filesystem total inodes: 2
>>>> Filesystem total metadata blocks: 1
>>>> Filesystem total deduplicated bytes (of source files): 0
>>>>
>>>> Now I have 6.15-rc5 and a defconfig-close setting for the 32-bit ARM
>>>> target BeagleBone Black. When booting into init=/bin/sh, then running
>>>>
>>>> # mount -t erofs /dev/mmcblk0p1 /mnt
>>>> erofs (device mmcblk0p1): mounted with root inode @ nid 36.
>>>> # /mnt/dash
>>>> Segmentation fault
>>>>
>>>> I once also got this:
>>>>
>>>> Alignment trap: not handling instruction 2b00 at [<004debc0>]
>>>> 8<--- cut here ---
>>>> Unhandled fault: alignment exception (0x001) at 0x000004d9
>>>> [000004d9] *pgd=00000000
>>>> Bus error
>>>>
>>>> All is fine if I
>>>>    - run the command once more
>>>>    - dump the file first (cat /mnt/dash > /dev/null; /mnt/dash)
>>>
>>> Forgot to mention: That first dump when done via md5sum or so actually
>>> gives the right checksum. So pure reading of the binary is also ok, just
>>> trying to load it for execution fails on the first attempt.
>>
>> Thanks for your report.  I rarely take care arm32 platform
>> because I don't have such setup.
>>
>> but could you share a reproducible rootfs image and
>> I wonder if qemu could reproduce this?
> 
> The image can be generated from isar-cip-core
> (https://gitlab.com/cip-project/cip-core/isar-cip-core), bbb image with
> swupdate extension and erofs as immutable rootfs. As I wrote, those will
> be 6.12 or 6.1 based, but I also injected a mainline kernel into that
> with the same result. But all that only helps if you have some
> beaglebone black in reach right now.

Could you check 5.4 lts, 5.15 lts, 5.10 lts if possible?

I wonder if it's a regression or it does not work from
the beginning. Anyway, I have no chance to look after arm32
due to lack of use cases and hardware resource.

> 
> The same configuration, just for qemuarm as target, unfortunately does
> not reproduce the issue.

That is too bad for me honestly because I do think it's much
easier for me to quickly shooting down with a reproducable
environment on my side...

> 
>>
>> Otherwise it's hard for me to debug this issue...
> 
> If you tell me how I could do that, I'm happy to instrument and analyze.
> I just have no understanding of erofs yet, specifically how reading
> files might be different from loading executables.

I have no idea too.  But it seems this issue is quite similar to
https://lore.kernel.org/r/76f0ea6f-d4c2-434a-8ca5-4bd93921209f@linux.alibaba.com/T/#t

I've given several ways to help finding the cause, but it
seems that data checksum is all good, and that issue is
still not resolved now either.

Thanks,
Gao Xiang

> 
> Jan
> 

Re: Executable loading issues with erofs on arm?

[Gao Xiang]

On 2025/7/8 23:32, Gao Xiang wrote:
> 
> 
> On 2025/7/8 23:22, Jan Kiszka wrote:
>> On 08.07.25 17:12, Gao Xiang wrote:
>>> Hi Jan,
>>>
>>> On 2025/7/8 20:43, Jan Kiszka wrote:
>>>> On 08.07.25 14:41, Jan Kiszka wrote:
>>>>> Hi all,
>>>>>
>>>>> for some days, I'm trying to understand if we have an integration issue
>>>>> with erofs or rather some upstream bug. After playing with various
>>>>> parameters, it rather looks like the latter:
>>>>>
>>>>> $ ls -l erofs-dir/
>>>>> total 132
>>>>> -rwxr-xr-x 1 1000 users 132868 Jul  8 10:50 dash
>>>>> (from Debian bookworm)
>>>>> $ mkfs.erofs -z lz4hc erofs.img erofs-dir/
>>>>> mkfs.erofs 1.8.6 (trixie version, but same happens with bookworm 1.5)
>>>>> Build completed.
>>>>> ------
>>>>> Filesystem UUID: aae0b2f0-4ee4-4850-af49-3c1aad7fa30c
>>>>> Filesystem total blocks: 17 (of 4096-byte blocks)
>>>>> Filesystem total inodes: 2
>>>>> Filesystem total metadata blocks: 1
>>>>> Filesystem total deduplicated bytes (of source files): 0
>>>>>
>>>>> Now I have 6.15-rc5 and a defconfig-close setting for the 32-bit ARM
>>>>> target BeagleBone Black. When booting into init=/bin/sh, then running
>>>>>
>>>>> # mount -t erofs /dev/mmcblk0p1 /mnt
>>>>> erofs (device mmcblk0p1): mounted with root inode @ nid 36.
>>>>> # /mnt/dash
>>>>> Segmentation fault

Two extra quick questions:
  - If the segfault happens, then if you run /mnt/dash again, does
    segfault still happen?

  - If the /mnt/dash segfault happens, then if you run
      cat /mnt/dash > /dev/null
      /mnt/dash
    does segfault still happen?

Thanks,
Gao Xiang

Re: Executable loading issues with erofs on arm?

[Gao Xiang]

On 2025/7/8 23:36, Gao Xiang wrote:
> 
> 
> On 2025/7/8 23:32, Gao Xiang wrote:
>>
>>
>> On 2025/7/8 23:22, Jan Kiszka wrote:
>>> On 08.07.25 17:12, Gao Xiang wrote:
>>>> Hi Jan,
>>>>
>>>> On 2025/7/8 20:43, Jan Kiszka wrote:
>>>>> On 08.07.25 14:41, Jan Kiszka wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> for some days, I'm trying to understand if we have an integration issue
>>>>>> with erofs or rather some upstream bug. After playing with various
>>>>>> parameters, it rather looks like the latter:
>>>>>>
>>>>>> $ ls -l erofs-dir/
>>>>>> total 132
>>>>>> -rwxr-xr-x 1 1000 users 132868 Jul  8 10:50 dash
>>>>>> (from Debian bookworm)
>>>>>> $ mkfs.erofs -z lz4hc erofs.img erofs-dir/
>>>>>> mkfs.erofs 1.8.6 (trixie version, but same happens with bookworm 1.5)
>>>>>> Build completed.
>>>>>> ------
>>>>>> Filesystem UUID: aae0b2f0-4ee4-4850-af49-3c1aad7fa30c
>>>>>> Filesystem total blocks: 17 (of 4096-byte blocks)
>>>>>> Filesystem total inodes: 2
>>>>>> Filesystem total metadata blocks: 1
>>>>>> Filesystem total deduplicated bytes (of source files): 0
>>>>>>
>>>>>> Now I have 6.15-rc5 and a defconfig-close setting for the 32-bit ARM
>>>>>> target BeagleBone Black. When booting into init=/bin/sh, then running
>>>>>>
>>>>>> # mount -t erofs /dev/mmcblk0p1 /mnt
>>>>>> erofs (device mmcblk0p1): mounted with root inode @ nid 36.
>>>>>> # /mnt/dash
>>>>>> Segmentation fault
> 
> Two extra quick questions:
>   - If the segfault happens, then if you run /mnt/dash again, does
>     segfault still happen?
> 
>   - If the /mnt/dash segfault happens, then if you run
>       cat /mnt/dash > /dev/null
>       /mnt/dash
>     does segfault still happen?

Oh, sorry I didn't read the full hints, could you check if
the following patch resolve the issue (space-damaged)?

diff --git a/fs/erofs/data.c b/fs/erofs/data.c
index 6a329c329f43..701490b3ef7d 100644
--- a/fs/erofs/data.c
+++ b/fs/erofs/data.c
@@ -245,6 +245,7 @@ void erofs_onlinefolio_end(struct folio *folio, int err)
         if (v & ~EROFS_ONLINEFOLIO_EIO)
                 return;
         folio->private = 0;
+       flush_dcache_folio(folio);
         folio_end_read(folio, !(v & EROFS_ONLINEFOLIO_EIO));
  }

Thanks,
Gao Xiang

Re: Executable loading issues with erofs on arm?

[Jan Kiszka]

On 08.07.25 17:57, Gao Xiang wrote:
> 
> 
> On 2025/7/8 23:36, Gao Xiang wrote:
>>
>>
>> On 2025/7/8 23:32, Gao Xiang wrote:
>>>
>>>
>>> On 2025/7/8 23:22, Jan Kiszka wrote:
>>>> On 08.07.25 17:12, Gao Xiang wrote:
>>>>> Hi Jan,
>>>>>
>>>>> On 2025/7/8 20:43, Jan Kiszka wrote:
>>>>>> On 08.07.25 14:41, Jan Kiszka wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> for some days, I'm trying to understand if we have an integration
>>>>>>> issue
>>>>>>> with erofs or rather some upstream bug. After playing with various
>>>>>>> parameters, it rather looks like the latter:
>>>>>>>
>>>>>>> $ ls -l erofs-dir/
>>>>>>> total 132
>>>>>>> -rwxr-xr-x 1 1000 users 132868 Jul  8 10:50 dash
>>>>>>> (from Debian bookworm)
>>>>>>> $ mkfs.erofs -z lz4hc erofs.img erofs-dir/
>>>>>>> mkfs.erofs 1.8.6 (trixie version, but same happens with bookworm
>>>>>>> 1.5)
>>>>>>> Build completed.
>>>>>>> ------
>>>>>>> Filesystem UUID: aae0b2f0-4ee4-4850-af49-3c1aad7fa30c
>>>>>>> Filesystem total blocks: 17 (of 4096-byte blocks)
>>>>>>> Filesystem total inodes: 2
>>>>>>> Filesystem total metadata blocks: 1
>>>>>>> Filesystem total deduplicated bytes (of source files): 0
>>>>>>>
>>>>>>> Now I have 6.15-rc5 and a defconfig-close setting for the 32-bit ARM
>>>>>>> target BeagleBone Black. When booting into init=/bin/sh, then
>>>>>>> running
>>>>>>>
>>>>>>> # mount -t erofs /dev/mmcblk0p1 /mnt
>>>>>>> erofs (device mmcblk0p1): mounted with root inode @ nid 36.
>>>>>>> # /mnt/dash
>>>>>>> Segmentation fault
>>
>> Two extra quick questions:
>>   - If the segfault happens, then if you run /mnt/dash again, does
>>     segfault still happen?
>>
>>   - If the /mnt/dash segfault happens, then if you run
>>       cat /mnt/dash > /dev/null
>>       /mnt/dash
>>     does segfault still happen?
> 
> Oh, sorry I didn't read the full hints, could you check if
> the following patch resolve the issue (space-damaged)?
> 
> diff --git a/fs/erofs/data.c b/fs/erofs/data.c
> index 6a329c329f43..701490b3ef7d 100644
> --- a/fs/erofs/data.c
> +++ b/fs/erofs/data.c
> @@ -245,6 +245,7 @@ void erofs_onlinefolio_end(struct folio *folio, int
> err)
>         if (v & ~EROFS_ONLINEFOLIO_EIO)
>                 return;
>         folio->private = 0;
> +       flush_dcache_folio(folio);
>         folio_end_read(folio, !(v & EROFS_ONLINEFOLIO_EIO));
>  }
> 

Yeah, indeed that seem to have helped with the minimal test. Will do the
full scenario test (complete rootfs) next.

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

Re: Executable loading issues with erofs on arm?

[Jan Kiszka]

On 08.07.25 18:39, Jan Kiszka wrote:
> On 08.07.25 17:57, Gao Xiang wrote:
>>
>>
>> On 2025/7/8 23:36, Gao Xiang wrote:
>>>
>>>
>>> On 2025/7/8 23:32, Gao Xiang wrote:
>>>>
>>>>
>>>> On 2025/7/8 23:22, Jan Kiszka wrote:
>>>>> On 08.07.25 17:12, Gao Xiang wrote:
>>>>>> Hi Jan,
>>>>>>
>>>>>> On 2025/7/8 20:43, Jan Kiszka wrote:
>>>>>>> On 08.07.25 14:41, Jan Kiszka wrote:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> for some days, I'm trying to understand if we have an integration
>>>>>>>> issue
>>>>>>>> with erofs or rather some upstream bug. After playing with various
>>>>>>>> parameters, it rather looks like the latter:
>>>>>>>>
>>>>>>>> $ ls -l erofs-dir/
>>>>>>>> total 132
>>>>>>>> -rwxr-xr-x 1 1000 users 132868 Jul  8 10:50 dash
>>>>>>>> (from Debian bookworm)
>>>>>>>> $ mkfs.erofs -z lz4hc erofs.img erofs-dir/
>>>>>>>> mkfs.erofs 1.8.6 (trixie version, but same happens with bookworm
>>>>>>>> 1.5)
>>>>>>>> Build completed.
>>>>>>>> ------
>>>>>>>> Filesystem UUID: aae0b2f0-4ee4-4850-af49-3c1aad7fa30c
>>>>>>>> Filesystem total blocks: 17 (of 4096-byte blocks)
>>>>>>>> Filesystem total inodes: 2
>>>>>>>> Filesystem total metadata blocks: 1
>>>>>>>> Filesystem total deduplicated bytes (of source files): 0
>>>>>>>>
>>>>>>>> Now I have 6.15-rc5 and a defconfig-close setting for the 32-bit ARM
>>>>>>>> target BeagleBone Black. When booting into init=/bin/sh, then
>>>>>>>> running
>>>>>>>>
>>>>>>>> # mount -t erofs /dev/mmcblk0p1 /mnt
>>>>>>>> erofs (device mmcblk0p1): mounted with root inode @ nid 36.
>>>>>>>> # /mnt/dash
>>>>>>>> Segmentation fault
>>>
>>> Two extra quick questions:
>>>   - If the segfault happens, then if you run /mnt/dash again, does
>>>     segfault still happen?
>>>
>>>   - If the /mnt/dash segfault happens, then if you run
>>>       cat /mnt/dash > /dev/null
>>>       /mnt/dash
>>>     does segfault still happen?
>>
>> Oh, sorry I didn't read the full hints, could you check if
>> the following patch resolve the issue (space-damaged)?
>>
>> diff --git a/fs/erofs/data.c b/fs/erofs/data.c
>> index 6a329c329f43..701490b3ef7d 100644
>> --- a/fs/erofs/data.c
>> +++ b/fs/erofs/data.c
>> @@ -245,6 +245,7 @@ void erofs_onlinefolio_end(struct folio *folio, int
>> err)
>>         if (v & ~EROFS_ONLINEFOLIO_EIO)
>>                 return;
>>         folio->private = 0;
>> +       flush_dcache_folio(folio);
>>         folio_end_read(folio, !(v & EROFS_ONLINEFOLIO_EIO));
>>  }
>>
> 
> Yeah, indeed that seem to have helped with the minimal test. Will do the
> full scenario test (complete rootfs) next.
> 

And that looks good as! Thanks a lot for that quick fix - hoping that is
the real solution already.

BTW, that change does not look very specific to the armhf arch, rather
like we were lucky that it didn't hit elsewhere, right?

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

Re: Executable loading issues with erofs on arm?

[Gao Xiang]

On 2025/7/9 01:01, Jan Kiszka wrote:
> On 08.07.25 18:39, Jan Kiszka wrote:
>> On 08.07.25 17:57, Gao Xiang wrote:
>>>
>>>
>>> On 2025/7/8 23:36, Gao Xiang wrote:
>>>>
>>>>
>>>> On 2025/7/8 23:32, Gao Xiang wrote:
>>>>>
>>>>>
>>>>> On 2025/7/8 23:22, Jan Kiszka wrote:
>>>>>> On 08.07.25 17:12, Gao Xiang wrote:
>>>>>>> Hi Jan,
>>>>>>>
>>>>>>> On 2025/7/8 20:43, Jan Kiszka wrote:
>>>>>>>> On 08.07.25 14:41, Jan Kiszka wrote:
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> for some days, I'm trying to understand if we have an integration
>>>>>>>>> issue
>>>>>>>>> with erofs or rather some upstream bug. After playing with various
>>>>>>>>> parameters, it rather looks like the latter:
>>>>>>>>>
>>>>>>>>> $ ls -l erofs-dir/
>>>>>>>>> total 132
>>>>>>>>> -rwxr-xr-x 1 1000 users 132868 Jul  8 10:50 dash
>>>>>>>>> (from Debian bookworm)
>>>>>>>>> $ mkfs.erofs -z lz4hc erofs.img erofs-dir/
>>>>>>>>> mkfs.erofs 1.8.6 (trixie version, but same happens with bookworm
>>>>>>>>> 1.5)
>>>>>>>>> Build completed.
>>>>>>>>> ------
>>>>>>>>> Filesystem UUID: aae0b2f0-4ee4-4850-af49-3c1aad7fa30c
>>>>>>>>> Filesystem total blocks: 17 (of 4096-byte blocks)
>>>>>>>>> Filesystem total inodes: 2
>>>>>>>>> Filesystem total metadata blocks: 1
>>>>>>>>> Filesystem total deduplicated bytes (of source files): 0
>>>>>>>>>
>>>>>>>>> Now I have 6.15-rc5 and a defconfig-close setting for the 32-bit ARM
>>>>>>>>> target BeagleBone Black. When booting into init=/bin/sh, then
>>>>>>>>> running
>>>>>>>>>
>>>>>>>>> # mount -t erofs /dev/mmcblk0p1 /mnt
>>>>>>>>> erofs (device mmcblk0p1): mounted with root inode @ nid 36.
>>>>>>>>> # /mnt/dash
>>>>>>>>> Segmentation fault
>>>>
>>>> Two extra quick questions:
>>>>    - If the segfault happens, then if you run /mnt/dash again, does
>>>>      segfault still happen?
>>>>
>>>>    - If the /mnt/dash segfault happens, then if you run
>>>>        cat /mnt/dash > /dev/null
>>>>        /mnt/dash
>>>>      does segfault still happen?
>>>
>>> Oh, sorry I didn't read the full hints, could you check if
>>> the following patch resolve the issue (space-damaged)?
>>>
>>> diff --git a/fs/erofs/data.c b/fs/erofs/data.c
>>> index 6a329c329f43..701490b3ef7d 100644
>>> --- a/fs/erofs/data.c
>>> +++ b/fs/erofs/data.c
>>> @@ -245,6 +245,7 @@ void erofs_onlinefolio_end(struct folio *folio, int
>>> err)
>>>          if (v & ~EROFS_ONLINEFOLIO_EIO)
>>>                  return;
>>>          folio->private = 0;
>>> +       flush_dcache_folio(folio);
>>>          folio_end_read(folio, !(v & EROFS_ONLINEFOLIO_EIO));
>>>   }
>>>
>>
>> Yeah, indeed that seem to have helped with the minimal test. Will do the
>> full scenario test (complete rootfs) next.
>>
> 
> And that looks good as! Thanks a lot for that quick fix - hoping that is
> the real solution already.
> 
> BTW, that change does not look very specific to the armhf arch, rather
> like we were lucky that it didn't hit elsewhere, right?

I may submit a formal patch tomorrow.

This issue doesn't impact x86 and arm64. For example on arm64,
PG_dcache_clean is clear when it's a new page cache folio.

But it seems on arm platform flush_dcache_folio() does more
to handle D-cache aliasing so some caching setup may be
impacted.

Thanks,
Gao Xiang

> 
> Jan
> 

Re: Executable loading issues with erofs on arm?

[Jan Kiszka]

On 08.07.25 19:09, Gao Xiang wrote:
> 
> 
> On 2025/7/9 01:01, Jan Kiszka wrote:
>> On 08.07.25 18:39, Jan Kiszka wrote:
>>> On 08.07.25 17:57, Gao Xiang wrote:
>>>>
>>>>
>>>> On 2025/7/8 23:36, Gao Xiang wrote:
>>>>>
>>>>>
>>>>> On 2025/7/8 23:32, Gao Xiang wrote:
>>>>>>
>>>>>>
>>>>>> On 2025/7/8 23:22, Jan Kiszka wrote:
>>>>>>> On 08.07.25 17:12, Gao Xiang wrote:
>>>>>>>> Hi Jan,
>>>>>>>>
>>>>>>>> On 2025/7/8 20:43, Jan Kiszka wrote:
>>>>>>>>> On 08.07.25 14:41, Jan Kiszka wrote:
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> for some days, I'm trying to understand if we have an integration
>>>>>>>>>> issue
>>>>>>>>>> with erofs or rather some upstream bug. After playing with
>>>>>>>>>> various
>>>>>>>>>> parameters, it rather looks like the latter:
>>>>>>>>>>
>>>>>>>>>> $ ls -l erofs-dir/
>>>>>>>>>> total 132
>>>>>>>>>> -rwxr-xr-x 1 1000 users 132868 Jul  8 10:50 dash
>>>>>>>>>> (from Debian bookworm)
>>>>>>>>>> $ mkfs.erofs -z lz4hc erofs.img erofs-dir/
>>>>>>>>>> mkfs.erofs 1.8.6 (trixie version, but same happens with bookworm
>>>>>>>>>> 1.5)
>>>>>>>>>> Build completed.
>>>>>>>>>> ------
>>>>>>>>>> Filesystem UUID: aae0b2f0-4ee4-4850-af49-3c1aad7fa30c
>>>>>>>>>> Filesystem total blocks: 17 (of 4096-byte blocks)
>>>>>>>>>> Filesystem total inodes: 2
>>>>>>>>>> Filesystem total metadata blocks: 1
>>>>>>>>>> Filesystem total deduplicated bytes (of source files): 0
>>>>>>>>>>
>>>>>>>>>> Now I have 6.15-rc5 and a defconfig-close setting for the 32-
>>>>>>>>>> bit ARM
>>>>>>>>>> target BeagleBone Black. When booting into init=/bin/sh, then
>>>>>>>>>> running
>>>>>>>>>>
>>>>>>>>>> # mount -t erofs /dev/mmcblk0p1 /mnt
>>>>>>>>>> erofs (device mmcblk0p1): mounted with root inode @ nid 36.
>>>>>>>>>> # /mnt/dash
>>>>>>>>>> Segmentation fault
>>>>>
>>>>> Two extra quick questions:
>>>>>    - If the segfault happens, then if you run /mnt/dash again, does
>>>>>      segfault still happen?
>>>>>
>>>>>    - If the /mnt/dash segfault happens, then if you run
>>>>>        cat /mnt/dash > /dev/null
>>>>>        /mnt/dash
>>>>>      does segfault still happen?
>>>>
>>>> Oh, sorry I didn't read the full hints, could you check if
>>>> the following patch resolve the issue (space-damaged)?
>>>>
>>>> diff --git a/fs/erofs/data.c b/fs/erofs/data.c
>>>> index 6a329c329f43..701490b3ef7d 100644
>>>> --- a/fs/erofs/data.c
>>>> +++ b/fs/erofs/data.c
>>>> @@ -245,6 +245,7 @@ void erofs_onlinefolio_end(struct folio *folio, int
>>>> err)
>>>>          if (v & ~EROFS_ONLINEFOLIO_EIO)
>>>>                  return;
>>>>          folio->private = 0;
>>>> +       flush_dcache_folio(folio);
>>>>          folio_end_read(folio, !(v & EROFS_ONLINEFOLIO_EIO));
>>>>   }
>>>>
>>>
>>> Yeah, indeed that seem to have helped with the minimal test. Will do the
>>> full scenario test (complete rootfs) next.
>>>
>>
>> And that looks good as! Thanks a lot for that quick fix - hoping that is
>> the real solution already.
>>
>> BTW, that change does not look very specific to the armhf arch, rather
>> like we were lucky that it didn't hit elsewhere, right?
> 
> I may submit a formal patch tomorrow.
> 

Great thanks. I quickly checked backports, and it fits cleanly on 6.12,
but at least 6.1 requires more work to find a home there as well.

> This issue doesn't impact x86 and arm64. For example on arm64,
> PG_dcache_clean is clear when it's a new page cache folio.
> 
> But it seems on arm platform flush_dcache_folio() does more
> to handle D-cache aliasing so some caching setup may be
> impacted.

Yeah, that would explain it. And Stefan (on CC) was on an arm32 as well
back then.

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center