* Re: EROFS: decompressed page cache returns valid file content from the wrong offset under sustained reclaim
2026-08-28 12:01 EROFS: decompressed page cache returns valid file content from the wrong offset under sustained reclaim Walther, Jens-Uwe
@ 2026-08-28 13:29 ` Gao Xiang
0 siblings, 0 replies; 2+ messages in thread
From: Gao Xiang @ 2026-08-28 13:29 UTC (permalink / raw)
To: Walther, Jens-Uwe
Cc: linux-erofs@lists.ozlabs.org, xiang@kernel.org, chao@kernel.org
Hi Walther,
On Fri, Aug 28, 2026 at 12:01:22PM +0000, Walther, Jens-Uwe wrote:
> Hey folks,
>
> We have two independent occurrences of silent data corruption on EROFS roots, on
> two machines run by two different operators, and we cannot reproduce it locally.
> The corruption is not random: a contiguous region of a read-only file comes back
> containing that same file's own bytes taken from a different offset. Details in
> case the pattern is recognisable.
So, so far there is only two individual reports, and it is
unreproducible locally?
>
> Environment
> EROFS root image built with mkfs.erofs --all-root -z lz4hc,12 -C 262144
> mounted ro, cache_strategy=readaround, dm-verity below with
> restart_on_corruption
> kernels 6.18.38 (case A) and 6.12.95 (case B), x86_64
> 8 GiB and 16 GiB machines; swap is zram only (1 GiB, lz4), no disk swap
> failures appeared after 9d5h and 5d16h uptime
>
> Symptom
> A 3,627,816-byte shared library reads back with one contiguous region
> replaced. It has a self-integrity check, which is how it was noticed.
> Everything outside the region is byte-identical to a known-good copy, and the
> region reconstructs 100% from that copy as exactly two contiguous runs taken
> from elsewhere in the same file:
>
> case A region 0x239000-0x23afff (8,192 B)
> run 1 at 0x239000 3,694 B from +16,786
> run 2 at 0x239e6e 4,498 B from +8,393
>
> case B region 0x271d0c-0x273fff (8,948 B)
> run 1 at 0x271d0c 7,436 B from +17,896
> run 2 at 0x273a18 1,512 B from +8,948
>
> In both cases the first shift is exactly twice the second. Neither is a
> multiple of PAGE_SIZE (8,393 = 2*4096 + 201), so this is not a page- or
> folio-level mixup.
I'm not sure I could parse it correctly (for example, I don't quite get
the idea what shift means), but if the shared library can be shared with
me, could you just send the binary to me directly?
Or when the wrong data was read, did you find the same data in the
image?
>
> Location
> dump.erofs -e places both corrupted ranges and both of their source ranges
> inside a single extent: logical 2,177,202..2,779,162 (601,960 B) decompressed
> from one 262,144-byte pcluster. Nothing sits near an extent boundary. The
> error is therefore produced within one decompression, not by selecting the
> wrong pcluster.
>
> Ruled out
> on-disk corruption - dm-verity reported no error across 15.92 days spanning
> the fault, with restart_on_corruption armed
> a stale mapping - a read() of the file and the process mapping return the same
> wrong bytes, so the page cache itself holds them
> hardware - two operators, different CPU vendors, and the result is structured
> folio migration as the copying agent - migration moves whole pages, the shifts
> are not page-aligned
>
> Conditions
> Both machines were under sustained page-cache reclaim. Case A: 956M
> pgsteal_kswapd and 559M workingset_refault_file over 17.7 days. Case B: 71M
> and 16M over 7.74 days. A comparable machine on the same image with no reclaim
> shows pgscan_kswapd and pgsteal_kswapd exactly 0. Since the only swap is zram,
> clean page cache - i.e. the EROFS image - is the only memory the kernel can
> genuinely free, so all pressure lands on evicting and re-reading these pages.
>
> Geometry in this configuration
> bpftrace on z_erofs_lz4_decompress, same image and mount options, while the
> library was evicted and re-read about a million times:
> rq->inplace_io was 0 for all 963,196 calls
> 74.7% had rq->partial_decoding = 1
> inplace_io=1 occurs only when the compressed pages are absent from the
> managed cache; with a warm managed cache we never observed it
> So the failing path is very likely a managed-cache decode, mostly partial.
>
> Not reproduced
> ~33M real decompressions of the same file, same image and mount options, under
> heavier reclaim than either failing machine (66x the pgsteal_kswapd rate, 1.4x
> the pgmigrate_success rate), produced no mismatch. Also tried: 92M single-page
> faults against a purpose-built image with an identical extent map, forced
> compaction, and cache_strategy=disabled. No mismatch in any configuration.
>
> Questions
> 1. Is a byte-granular displacement of this shape known in the partial decode
> path with big pclusters - output written at rq->pageofs_out into a
> partially populated rq->out[] set?
> 2. Does the first shift being exactly twice the second suggest particular
> index or length arithmetic worth looking at?
> 3. Could z_erofs_lz4_prepare_dstpages() or the fillgaps handling place output
> at the wrong offset when only a subset of out[] is requested?
I don't think answering the details of 1,2,3 helps since
- it involves many erofs implementation details and hard to write here
in simple words, for example fillgaps should have nothing with your
set since it's only related to mkfs.erofs `-Ededupe` feature;
- the decompression subsystem is basically unchanged from Linux 6.1
to Linux 6.12, except that:
- it adds subpage block support (I think it's unrelated to your
case);
- it adds large folio support (suspicious);
- the lz4 doesn't rely on some page refcount anymore (see below).
- If you didn't observe it on Linux 6.1/6.6, I guess it may due to the
following changes:
- (6.6~6.12) it implements large folios, so we could try to see if
the fleets can still be reproduced with large folios disabled;
- we could try to revert 1001042e54ef ("erofs: avoid refcounting
short-lived pages") since I think it's a significant change from
Linux 6.1/6.6.
But anyway, if it only reproduced twice so far and no reproducer, I think
first we need to find a better way to reproduce it in a short time; on
the other hand, I will review and reproduce myself too, but as I said,
the binary is helpful for me to get more clues.
Thanks,
Gao Xiang
> 4. If field data would help, what would you want captured? We have a detector
> running that hashes the file periodically and, when it fires, preserves the
> machine along with the corrupted copy, /proc/vmstat and PSI. We can also
> run additional tracing or a debug patch on affected fleets.
>
>
> Thank you in advance.
>
> Best regards
> Jens-Uwe Walther
> Senior Specialist TAM Containers
> AWS Enterprise Support
> [CKA badge]<https://www.credly.com/badges/b4f55949-4357-4062-93a9-6d0f0f5f3833/linked_in>[CKAD badge]<https://www.credly.com/badges/85b75f03-91d4-4887-8150-cfb9ea39d272> [CKS: Certified Kubernetes Security Specialist] <https://www.credly.com/badges/c26e3b3c-cfc7-46d7-8b7b-7a7a59045c60/>
> e: waltju@amazon.com<mailto:waltju@amazon.com> m: +49 152 38870547
> [cid:image004.png@01D9D4D4.4DF69060]<http://www.aws.amazon.com/>
>
>
>
> Amazon Web Services EMEA SARL
> 38 avenue John F. Kennedy, L-1855 Luxembourg
> Sitz der Gesellschaft: L-1855 Luxemburg
> eingetragen im Luxemburgischen Handelsregister unter R.C.S. B186284
>
> Amazon Web Services EMEA SARL, Niederlassung Deutschland
> Anni-Albers-Str. 21, D-80807 Muenchen
> Sitz der Zweigniederlassung: Muenchen
> eingetragen im Handelsregister des Amtsgerichts Muenchen unter HRB 242240
> USt-ID DE317013094
^ permalink raw reply [flat|nested] 2+ messages in thread