EROFS: decompressed page cache returns valid file content from the wrong offset under sustained reclaim

EROFS: decompressed page cache returns valid file content from the wrong offset under sustained reclaim

[Walther, Jens-Uwe]

[-- Attachment #1.1: Type: text/plain, Size: 5535 bytes --]

Hey folks,

We have two independent occurrences of silent data corruption on EROFS roots, on
two machines run by two different operators, and we cannot reproduce it locally.
The corruption is not random: a contiguous region of a read-only file comes back
containing that same file's own bytes taken from a different offset. Details in
case the pattern is recognisable.

Environment
  EROFS root image built with mkfs.erofs --all-root -z lz4hc,12 -C 262144
  mounted ro, cache_strategy=readaround, dm-verity below with
  restart_on_corruption
  kernels 6.18.38 (case A) and 6.12.95 (case B), x86_64
  8 GiB and 16 GiB machines; swap is zram only (1 GiB, lz4), no disk swap
  failures appeared after 9d5h and 5d16h uptime

Symptom
  A 3,627,816-byte shared library reads back with one contiguous region
  replaced. It has a self-integrity check, which is how it was noticed.
  Everything outside the region is byte-identical to a known-good copy, and the
  region reconstructs 100% from that copy as exactly two contiguous runs taken
  from elsewhere in the same file:

  case A   region 0x239000-0x23afff (8,192 B)
           run 1  at 0x239000  3,694 B  from +16,786
           run 2  at 0x239e6e  4,498 B  from +8,393

  case B   region 0x271d0c-0x273fff (8,948 B)
           run 1  at 0x271d0c  7,436 B  from +17,896
           run 2  at 0x273a18  1,512 B  from +8,948

  In both cases the first shift is exactly twice the second. Neither is a
  multiple of PAGE_SIZE (8,393 = 2*4096 + 201), so this is not a page- or
  folio-level mixup.

Location
  dump.erofs -e places both corrupted ranges and both of their source ranges
  inside a single extent: logical 2,177,202..2,779,162 (601,960 B) decompressed
  from one 262,144-byte pcluster. Nothing sits near an extent boundary. The
  error is therefore produced within one decompression, not by selecting the
  wrong pcluster.

Ruled out
  on-disk corruption - dm-verity reported no error across 15.92 days spanning
    the fault, with restart_on_corruption armed
  a stale mapping - a read() of the file and the process mapping return the same
    wrong bytes, so the page cache itself holds them
  hardware - two operators, different CPU vendors, and the result is structured
  folio migration as the copying agent - migration moves whole pages, the shifts
    are not page-aligned

Conditions
  Both machines were under sustained page-cache reclaim. Case A: 956M
  pgsteal_kswapd and 559M workingset_refault_file over 17.7 days. Case B: 71M
  and 16M over 7.74 days. A comparable machine on the same image with no reclaim
  shows pgscan_kswapd and pgsteal_kswapd exactly 0. Since the only swap is zram,
  clean page cache - i.e. the EROFS image - is the only memory the kernel can
  genuinely free, so all pressure lands on evicting and re-reading these pages.

Geometry in this configuration
  bpftrace on z_erofs_lz4_decompress, same image and mount options, while the
  library was evicted and re-read about a million times:
    rq->inplace_io was 0 for all 963,196 calls
    74.7% had rq->partial_decoding = 1
    inplace_io=1 occurs only when the compressed pages are absent from the
      managed cache; with a warm managed cache we never observed it
  So the failing path is very likely a managed-cache decode, mostly partial.

Not reproduced
  ~33M real decompressions of the same file, same image and mount options, under
  heavier reclaim than either failing machine (66x the pgsteal_kswapd rate, 1.4x
  the pgmigrate_success rate), produced no mismatch. Also tried: 92M single-page
  faults against a purpose-built image with an identical extent map, forced
  compaction, and cache_strategy=disabled. No mismatch in any configuration.

Questions
  1. Is a byte-granular displacement of this shape known in the partial decode
     path with big pclusters - output written at rq->pageofs_out into a
     partially populated rq->out[] set?
  2. Does the first shift being exactly twice the second suggest particular
     index or length arithmetic worth looking at?
  3. Could z_erofs_lz4_prepare_dstpages() or the fillgaps handling place output
     at the wrong offset when only a subset of out[] is requested?
  4. If field data would help, what would you want captured? We have a detector
     running that hashes the file periodically and, when it fires, preserves the
     machine along with the corrupted copy, /proc/vmstat and PSI. We can also
     run additional tracing or a debug patch on affected fleets.


Thank you in advance.

Best regards
Jens-Uwe Walther
Senior Specialist TAM Containers
AWS Enterprise Support
[CKA badge]<https://www.credly.com/badges/b4f55949-4357-4062-93a9-6d0f0f5f3833/linked_in>[CKAD badge]<https://www.credly.com/badges/85b75f03-91d4-4887-8150-cfb9ea39d272> [CKS: Certified Kubernetes Security Specialist] <https://www.credly.com/badges/c26e3b3c-cfc7-46d7-8b7b-7a7a59045c60/>
e: waltju@amazon.com<mailto:waltju@amazon.com> m: +49 152 38870547
[cid:image004.png@01D9D4D4.4DF69060]<http://www.aws.amazon.com/>



Amazon Web Services EMEA SARL
38 avenue John F. Kennedy, L-1855 Luxembourg
Sitz der Gesellschaft: L-1855 Luxemburg
eingetragen im Luxemburgischen Handelsregister unter R.C.S. B186284

Amazon Web Services EMEA SARL, Niederlassung Deutschland
Anni-Albers-Str. 21, D-80807 Muenchen
Sitz der Zweigniederlassung: Muenchen
eingetragen im Handelsregister des Amtsgerichts Muenchen unter HRB 242240
USt-ID DE317013094

[-- Attachment #1.2: Type: text/html, Size: 9952 bytes --]

[-- Attachment #2: image001.png --]
[-- Type: image/png, Size: 5216 bytes --]

[-- Attachment #3: image002.png --]
[-- Type: image/png, Size: 5563 bytes --]

[-- Attachment #4: image003.png --]
[-- Type: image/png, Size: 7022 bytes --]

[-- Attachment #5: image004.png --]
[-- Type: image/png, Size: 3073 bytes --]

Re: EROFS: decompressed page cache returns valid file content from the wrong offset under sustained reclaim

[Gao Xiang]

Hi Walther,

On Fri, Aug 28, 2026 at 12:01:22PM +0000, Walther, Jens-Uwe wrote:
> Hey folks,
> 
> We have two independent occurrences of silent data corruption on EROFS roots, on
> two machines run by two different operators, and we cannot reproduce it locally.
> The corruption is not random: a contiguous region of a read-only file comes back
> containing that same file's own bytes taken from a different offset. Details in
> case the pattern is recognisable.

So, so far there is only two individual reports, and it is
unreproducible locally?

> 
> Environment
>   EROFS root image built with mkfs.erofs --all-root -z lz4hc,12 -C 262144
>   mounted ro, cache_strategy=readaround, dm-verity below with
>   restart_on_corruption
>   kernels 6.18.38 (case A) and 6.12.95 (case B), x86_64
>   8 GiB and 16 GiB machines; swap is zram only (1 GiB, lz4), no disk swap
>   failures appeared after 9d5h and 5d16h uptime
> 
> Symptom
>   A 3,627,816-byte shared library reads back with one contiguous region
>   replaced. It has a self-integrity check, which is how it was noticed.
>   Everything outside the region is byte-identical to a known-good copy, and the
>   region reconstructs 100% from that copy as exactly two contiguous runs taken
>   from elsewhere in the same file:
> 
>   case A   region 0x239000-0x23afff (8,192 B)
>            run 1  at 0x239000  3,694 B  from +16,786
>            run 2  at 0x239e6e  4,498 B  from +8,393
> 
>   case B   region 0x271d0c-0x273fff (8,948 B)
>            run 1  at 0x271d0c  7,436 B  from +17,896
>            run 2  at 0x273a18  1,512 B  from +8,948
> 
>   In both cases the first shift is exactly twice the second. Neither is a
>   multiple of PAGE_SIZE (8,393 = 2*4096 + 201), so this is not a page- or
>   folio-level mixup.

I'm not sure I could parse it correctly (for example, I don't quite get
the idea what shift means), but if the shared library can be shared with
me, could you just send the binary to me directly?

Or when the wrong data was read, did you find the same data in the
image?

> 
> Location
>   dump.erofs -e places both corrupted ranges and both of their source ranges
>   inside a single extent: logical 2,177,202..2,779,162 (601,960 B) decompressed
>   from one 262,144-byte pcluster. Nothing sits near an extent boundary. The
>   error is therefore produced within one decompression, not by selecting the
>   wrong pcluster.
> 
> Ruled out
>   on-disk corruption - dm-verity reported no error across 15.92 days spanning
>     the fault, with restart_on_corruption armed
>   a stale mapping - a read() of the file and the process mapping return the same
>     wrong bytes, so the page cache itself holds them
>   hardware - two operators, different CPU vendors, and the result is structured
>   folio migration as the copying agent - migration moves whole pages, the shifts
>     are not page-aligned
> 
> Conditions
>   Both machines were under sustained page-cache reclaim. Case A: 956M
>   pgsteal_kswapd and 559M workingset_refault_file over 17.7 days. Case B: 71M
>   and 16M over 7.74 days. A comparable machine on the same image with no reclaim
>   shows pgscan_kswapd and pgsteal_kswapd exactly 0. Since the only swap is zram,
>   clean page cache - i.e. the EROFS image - is the only memory the kernel can
>   genuinely free, so all pressure lands on evicting and re-reading these pages.
> 
> Geometry in this configuration
>   bpftrace on z_erofs_lz4_decompress, same image and mount options, while the
>   library was evicted and re-read about a million times:
>     rq->inplace_io was 0 for all 963,196 calls
>     74.7% had rq->partial_decoding = 1
>     inplace_io=1 occurs only when the compressed pages are absent from the
>       managed cache; with a warm managed cache we never observed it
>   So the failing path is very likely a managed-cache decode, mostly partial.
> 
> Not reproduced
>   ~33M real decompressions of the same file, same image and mount options, under
>   heavier reclaim than either failing machine (66x the pgsteal_kswapd rate, 1.4x
>   the pgmigrate_success rate), produced no mismatch. Also tried: 92M single-page
>   faults against a purpose-built image with an identical extent map, forced
>   compaction, and cache_strategy=disabled. No mismatch in any configuration.
> 
> Questions
>   1. Is a byte-granular displacement of this shape known in the partial decode
>      path with big pclusters - output written at rq->pageofs_out into a
>      partially populated rq->out[] set?
>   2. Does the first shift being exactly twice the second suggest particular
>      index or length arithmetic worth looking at?
>   3. Could z_erofs_lz4_prepare_dstpages() or the fillgaps handling place output
>      at the wrong offset when only a subset of out[] is requested?

I don't think answering the details of 1,2,3 helps since
 - it involves many erofs implementation details and hard to write here
   in simple words, for example fillgaps should have nothing with your
   set since it's only related to mkfs.erofs `-Ededupe` feature;

 - the decompression subsystem is basically unchanged from Linux 6.1
   to Linux 6.12, except that:
     - it adds subpage block support (I think it's unrelated to your
       case);
     - it adds large folio support (suspicious);
     - the lz4 doesn't rely on some page refcount anymore (see below).

 - If you didn't observe it on Linux 6.1/6.6, I guess it may due to the
   following changes:
     - (6.6~6.12) it implements large folios, so we could try to see if
       the fleets can still be reproduced with large folios disabled;

     - we could try to revert 1001042e54ef ("erofs: avoid refcounting
       short-lived pages") since I think it's a significant change from
       Linux 6.1/6.6.

But anyway, if it only reproduced twice so far and no reproducer, I think
first we need to find a better way to reproduce it in a short time; on
the other hand, I will review and reproduce myself too, but as I said,
the binary is helpful for me to get more clues.

Thanks,
Gao Xiang

>   4. If field data would help, what would you want captured? We have a detector
>      running that hashes the file periodically and, when it fires, preserves the
>      machine along with the corrupted copy, /proc/vmstat and PSI. We can also
>      run additional tracing or a debug patch on affected fleets.
> 
> 
> Thank you in advance.
> 
> Best regards
> Jens-Uwe Walther
> Senior Specialist TAM Containers
> AWS Enterprise Support
> [CKA badge]<https://www.credly.com/badges/b4f55949-4357-4062-93a9-6d0f0f5f3833/linked_in>[CKAD badge]<https://www.credly.com/badges/85b75f03-91d4-4887-8150-cfb9ea39d272> [CKS: Certified Kubernetes Security Specialist] <https://www.credly.com/badges/c26e3b3c-cfc7-46d7-8b7b-7a7a59045c60/>
> e: waltju@amazon.com<mailto:waltju@amazon.com> m: +49 152 38870547
> [cid:image004.png@01D9D4D4.4DF69060]<http://www.aws.amazon.com/>
> 
> 
> 
> Amazon Web Services EMEA SARL
> 38 avenue John F. Kennedy, L-1855 Luxembourg
> Sitz der Gesellschaft: L-1855 Luxemburg
> eingetragen im Luxemburgischen Handelsregister unter R.C.S. B186284
> 
> Amazon Web Services EMEA SARL, Niederlassung Deutschland
> Anni-Albers-Str. 21, D-80807 Muenchen
> Sitz der Zweigniederlassung: Muenchen
> eingetragen im Handelsregister des Amtsgerichts Muenchen unter HRB 242240
> USt-ID DE317013094