cgroup/test_zswap failed with "zswpout does not increase after test program"

cgroup/test_zswap failed with "zswpout does not increase after test program"

[Zenghui Yu]

Hi,

Running cgroup/test_zswap on my arm64 box failed immediately with:

  [root@localhost cgroup]# ./test_zswap 
  TAP version 13
  1..8
  # zswpout does not increase after test program
  not ok 1 test_zswap_usage
  [...]

I'm sure that pages are successfully written into zswap by checking the
count_memcg_events(.., idx=ZSWPOUT, ..) trace events. But "zswpout_after"
in test_zswap_usage() is 0 and results in this failure.

I guess the problem is that (in this particular case) the memcg stats has
not been flushed when userspace reads it.

 memcg_stat_format()
   mem_cgroup_flush_stats()
     __mem_cgroup_flush_stats(.., force=false)
       needs_flush = memcg_vmstats_needs_flush();

 static bool memcg_vmstats_needs_flush(struct memcg_vmstats *vmstats)
 {
 	return atomic_long_read(&vmstats->stats_updates) >
 		MEMCG_CHARGE_BATCH * num_online_cpus();
 }

I can image that memcg_vmstats_needs_flush() will return false because I'm
testing a 16k-page-size kernel on a box with 96 cpus..

As we have a periodic flusher flushed all the stats every 2 seconds, I use
the following diff to wait the flusher to expose the accurate stats to
userspace.

diff --git a/tools/testing/selftests/cgroup/lib/cgroup_util.c b/tools/testing/selftests/cgroup/lib/cgroup_util.c
index 3ce134509041..9596f294da0b 100644
--- a/tools/testing/selftests/cgroup/lib/cgroup_util.c
+++ b/tools/testing/selftests/cgroup/lib/cgroup_util.c
@@ -95,6 +95,8 @@ int cg_read(const char *cgroup, const char *control, char *buf, size_t len)
 
 	snprintf(path, sizeof(path), "%s/%s", cgroup, control);
 
+	sleep(2);
+
 	ret = read_text(path, buf, len);
 	return ret >= 0 ? 0 : ret;
 }

I have no idea how to "fix" it properly. Please have a look!

Thanks,
Zenghui

Re: cgroup/test_zswap failed with "zswpout does not increase after test program"

[Michal Koutný]

[-- Attachment #1: Type: text/plain, Size: 593 bytes --]

Hello Zenghui.

On Tue, Jul 07, 2026 at 05:38:13PM +0800, Zenghui Yu <zenghui.yu@linux.dev> wrote:
> Hi,
> 
> Running cgroup/test_zswap on my arm64 box failed immediately with:
> 
>   [root@localhost cgroup]# ./test_zswap 
>   TAP version 13
>   1..8
>   # zswpout does not increase after test program
>   not ok 1 test_zswap_usage
>   [...]

What version of the tests do you run?
Namely, does it have the recent patches from Li [1]?

Thanks,
Michal

[1] 83476cc97bc63 ("Merge tag 'cgroup-for-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup") v7.2-rc1~136

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 265 bytes --]

Re: cgroup/test_zswap failed with "zswpout does not increase after test program"

[Zenghui Yu]

Hi Michal,

On 7/7/26 8:01 PM, Michal Koutný wrote:
> Hello Zenghui.
> 
> On Tue, Jul 07, 2026 at 05:38:13PM +0800, Zenghui Yu <zenghui.yu@linux.dev> wrote:
> > Hi,
> >
> > Running cgroup/test_zswap on my arm64 box failed immediately with:
> >
> >   [root@localhost cgroup]# ./test_zswap 
> >   TAP version 13
> >   1..8
> >   # zswpout does not increase after test program
> >   not ok 1 test_zswap_usage
> >   [...]
> 
> What version of the tests do you run?
> Namely, does it have the recent patches from Li [1]?

I use today's mm-new [*]

> 
> [1] 83476cc97bc63 ("Merge tag 'cgroup-for-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup") v7.2-rc1~136

... so it should already be included.

[*] https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git/log/?h=mm-new

Thanks,
Zenghui

Re: cgroup/test_zswap failed with "zswpout does not increase after test program"

[Yosry Ahmed]

On Tue, Jul 7, 2026 at 2:38 AM Zenghui Yu <zenghui.yu@linux.dev> wrote:
>
> Hi,
>
> Running cgroup/test_zswap on my arm64 box failed immediately with:
>
>   [root@localhost cgroup]# ./test_zswap
>   TAP version 13
>   1..8
>   # zswpout does not increase after test program
>   not ok 1 test_zswap_usage
>   [...]
>
> I'm sure that pages are successfully written into zswap by checking the
> count_memcg_events(.., idx=ZSWPOUT, ..) trace events. But "zswpout_after"
> in test_zswap_usage() is 0 and results in this failure.
>
> I guess the problem is that (in this particular case) the memcg stats has
> not been flushed when userspace reads it.
>
>  memcg_stat_format()
>    mem_cgroup_flush_stats()
>      __mem_cgroup_flush_stats(.., force=false)
>        needs_flush = memcg_vmstats_needs_flush();
>
>  static bool memcg_vmstats_needs_flush(struct memcg_vmstats *vmstats)
>  {
>         return atomic_long_read(&vmstats->stats_updates) >
>                 MEMCG_CHARGE_BATCH * num_online_cpus();
>  }
>
> I can image that memcg_vmstats_needs_flush() will return false because I'm
> testing a 16k-page-size kernel on a box with 96 cpus..
>
> As we have a periodic flusher flushed all the stats every 2 seconds, I use
> the following diff to wait the flusher to expose the accurate stats to
> userspace.
>
> diff --git a/tools/testing/selftests/cgroup/lib/cgroup_util.c b/tools/testing/selftests/cgroup/lib/cgroup_util.c
> index 3ce134509041..9596f294da0b 100644
> --- a/tools/testing/selftests/cgroup/lib/cgroup_util.c
> +++ b/tools/testing/selftests/cgroup/lib/cgroup_util.c
> @@ -95,6 +95,8 @@ int cg_read(const char *cgroup, const char *control, char *buf, size_t len)
>
>         snprintf(path, sizeof(path), "%s/%s", cgroup, control);
>
> +       sleep(2);
> +
>         ret = read_text(path, buf, len);
>         return ret >= 0 ? 0 : ret;
>  }
>
> I have no idea how to "fix" it properly. Please have a look!

We were discussing a way for userspace to explicitly trigger a flush
before, which would come in handy for testing. However, we decided not
to expose flushing as a concept to userspace.

Unfortunately I think the only way to "fix" the test is to allocate
more memory, enough to trigger a flush on most interesting setups.
Perhaps we should scale the amount of memory with the number of CPUs
so that we don't have to keep playing whack-a-mole.

Re: cgroup/test_zswap failed with "zswpout does not increase after test program"

[Nhat Pham]

On Tue, Jul 7, 2026 at 11:25 AM Yosry Ahmed <yosry@kernel.org> wrote:
>
> On Tue, Jul 7, 2026 at 2:38 AM Zenghui Yu <zenghui.yu@linux.dev> wrote:
>
> We were discussing a way for userspace to explicitly trigger a flush
> before, which would come in handy for testing. However, we decided not
> to expose flushing as a concept to userspace.
>
> Unfortunately I think the only way to "fix" the test is to allocate
> more memory, enough to trigger a flush on most interesting setups.
> Perhaps we should scale the amount of memory with the number of CPUs
> so that we don't have to keep playing whack-a-mole.

I don't have a good idea for writeback, but for zswap out, would
MADV_PAGEOUT work here?

Re: cgroup/test_zswap failed with "zswpout does not increase after test program"

[Yosry Ahmed]

On Tue, Jul 7, 2026 at 2:35 PM Nhat Pham <nphamcs@gmail.com> wrote:
>
> On Tue, Jul 7, 2026 at 11:25 AM Yosry Ahmed <yosry@kernel.org> wrote:
> >
> > On Tue, Jul 7, 2026 at 2:38 AM Zenghui Yu <zenghui.yu@linux.dev> wrote:
> >
> > We were discussing a way for userspace to explicitly trigger a flush
> > before, which would come in handy for testing. However, we decided not
> > to expose flushing as a concept to userspace.
> >
> > Unfortunately I think the only way to "fix" the test is to allocate
> > more memory, enough to trigger a flush on most interesting setups.
> > Perhaps we should scale the amount of memory with the number of CPUs
> > so that we don't have to keep playing whack-a-mole.
>
> I don't have a good idea for writeback, but for zswap out, would
> MADV_PAGEOUT work here?

I don't think the reclaim mechanism is the problem, but the fact that
we don't have enough pending updates to flush the stats. Am I missing
something?

Re: cgroup/test_zswap failed with "zswpout does not increase after test program"

[Nhat Pham]

On Tue, Jul 7, 2026 at 2:36 PM Yosry Ahmed <yosry@kernel.org> wrote:
>
> On Tue, Jul 7, 2026 at 2:35 PM Nhat Pham <nphamcs@gmail.com> wrote:
> >
> > On Tue, Jul 7, 2026 at 11:25 AM Yosry Ahmed <yosry@kernel.org> wrote:
> > >
> > > On Tue, Jul 7, 2026 at 2:38 AM Zenghui Yu <zenghui.yu@linux.dev> wrote:
> > >
> > > We were discussing a way for userspace to explicitly trigger a flush
> > > before, which would come in handy for testing. However, we decided not
> > > to expose flushing as a concept to userspace.
> > >
> > > Unfortunately I think the only way to "fix" the test is to allocate
> > > more memory, enough to trigger a flush on most interesting setups.
> > > Perhaps we should scale the amount of memory with the number of CPUs
> > > so that we don't have to keep playing whack-a-mole.
> >
> > I don't have a good idea for writeback, but for zswap out, would
> > MADV_PAGEOUT work here?
>
> I don't think the reclaim mechanism is the problem, but the fact that
> we don't have enough pending updates to flush the stats. Am I missing
> something?

Ah yeah, you're right. Hmmm.

Yeah it sucks, but maybe sleeping (more than the flush period) before
read is the only way. Which
is terribly implementation-dependent :(

Re: cgroup/test_zswap failed with "zswpout does not increase after test program"

[Yosry Ahmed]

On Tue, Jul 7, 2026 at 2:47 PM Nhat Pham <nphamcs@gmail.com> wrote:
>
> On Tue, Jul 7, 2026 at 2:36 PM Yosry Ahmed <yosry@kernel.org> wrote:
> >
> > On Tue, Jul 7, 2026 at 2:35 PM Nhat Pham <nphamcs@gmail.com> wrote:
> > >
> > > On Tue, Jul 7, 2026 at 11:25 AM Yosry Ahmed <yosry@kernel.org> wrote:
> > > >
> > > > On Tue, Jul 7, 2026 at 2:38 AM Zenghui Yu <zenghui.yu@linux.dev> wrote:
> > > >
> > > > We were discussing a way for userspace to explicitly trigger a flush
> > > > before, which would come in handy for testing. However, we decided not
> > > > to expose flushing as a concept to userspace.
> > > >
> > > > Unfortunately I think the only way to "fix" the test is to allocate
> > > > more memory, enough to trigger a flush on most interesting setups.
> > > > Perhaps we should scale the amount of memory with the number of CPUs
> > > > so that we don't have to keep playing whack-a-mole.
> > >
> > > I don't have a good idea for writeback, but for zswap out, would
> > > MADV_PAGEOUT work here?
> >
> > I don't think the reclaim mechanism is the problem, but the fact that
> > we don't have enough pending updates to flush the stats. Am I missing
> > something?
>
> Ah yeah, you're right. Hmmm.
>
> Yeah it sucks, but maybe sleeping (more than the flush period) before
> read is the only way. Which
> is terribly implementation-dependent :(

I would honestly rather use more memory. I think there might be cases
where the flusher is delayed. The flush being slightly delayed is not
technically a bug that we want to see a failure for, but if a large
stats change is not visible that's a user-noticeable behavior that we
want a failure for.

WDYT?