[LTP] [PATCH] ltp: syscalls/madvise09: Reset cgroup memory limits before test retries

[Wake Liu via ltp <ltp@lists.linux.it>]

### Background
When executing `madvise09`, the test intermittently fails with:

```
madvise09.c:163: TFAIL: MADV_FREE pages were freed immediately
...
madvise09.c:116: TINFO: 0x7476845000 unexpected   (0) at 1 expected 'a'
madvise09.c:232: TFAIL: Found 2 corrupted page(s)
```

---

### Root Cause Analysis
By analyzing the failure logs, we discovered that these failures only occurred when the test entered its retry block ("Both children killed, retrying...").

1. **Why does it retry? (OOM Race Condition)**
   The test relies on inducing memory pressure to trigger the OOM killer on the `memory_pressure_child`. However, because the entire cgroup is constrained by a strict `memory.max` of 8MB (`MEM_LIMIT`), we observed instances where the parent process (`child()`) was also terminated by the OOM killer.
   Specifically, when the memory-pressure child is killed, the parent process wakes up from `SAFE_WAIT(&status)` (which invokes `wait4`). While writing the exit status to the user-space stack pointer `&status`, the parent triggers a swap-in page fault (`do_swap_page` -> `mem_cgroup_swapin_charge_folio`).
   Since the memory-pressure child was just killed and its memory charges have not yet been fully freed, the parent's page fault hits the strict 8MB limit, invoking the OOM killer on the parent process. The test framework catches `WIFSIGNALED(status)` in `run()` and jumps to `retry:`.

2. **Why does the retry fail? (Cgroup Limit Leakage)**
   When retrying, `run()` forks a new `child()`. However, the cgroup `memory.max` (8MB) and `memory.swap.max` (16MB) set in the previous run were never reset.
   As a result, the new child starts executing in a cgroup that is already heavily constrained by the inherited 8MB limit.
   - When the child allocates pages, writes `'a'`s, and calls `madvise(..., MADV_FREE)`, the kernel immediately drops and reclaims the pages to relieve the 8MB memory pressure. This triggers the first failure: `MADV_FREE pages were freed immediately`.
   - Subsequently, when the test writes `'b'` to the first byte of `TOUCHED_PAGE1` (offset 0), it triggers a Copy-on-Write (CoW) on a zero-page (since the page was already reclaimed). The page is refaulted with `'b'` at offset 0, but offsets 1-4095 remain `0` (instead of the original `'a'`).
   - Finally, `check_page_baaa()` reads offset 1, finds `0` instead of `'a'`, and reports false-positive "page corruption" failures.

---

### Fix
This patch resets both `memory.max` and `memory.swap.max` to `"max"` (unconstrained) at the beginning of the `child()` function. This guarantees that each retry run starts in a clean cgroup environment. The strict memory limits will only be applied later in the test flow after `MADV_FREE` has been marked and initial page edits are completed.

Signed-off-by: Wake Liu <wakel@google.com>
---
 testcases/kernel/syscalls/madvise/madvise09.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/testcases/kernel/syscalls/madvise/madvise09.c b/testcases/kernel/syscalls/madvise/madvise09.c
index 87fe096fb..e6fb74fdb 100644
--- a/testcases/kernel/syscalls/madvise/madvise09.c
+++ b/testcases/kernel/syscalls/madvise/madvise09.c
@@ -146,6 +146,16 @@ static void child(void)
 
 	SAFE_CG_PRINTF(tst_cg, "cgroup.procs", "%d", getpid());
 
+	/*
+	 * Reset cgroup memory limits to default ("max") in case this is a retry run.
+	 * Otherwise, the retried child inherits the strict MEM_LIMIT from the previous
+	 * run, causing MADV_FREE pages to be dropped immediately before we touch them.
+	 */
+	if (SAFE_CG_HAS(tst_cg, "memory.max"))
+		SAFE_CG_PRINTF(tst_cg, "memory.max", "max");
+	if (swap_accounting_enabled && SAFE_CG_HAS(tst_cg, "memory.swap.max"))
+		SAFE_CG_PRINTF(tst_cg, "memory.swap.max", "max");
+
 	ptr = SAFE_MMAP(NULL, PAGES * page_size, PROT_READ | PROT_WRITE,
 			MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
 
-- 
2.55.0.795.g602f6c329a-goog


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp