Re: [RFC] memcg rstat flushing optimization

[Tejun Heo <tj@kernel.org>]

Hello,

On Wed, Oct 05, 2022 at 10:20:54AM -0700, Yosry Ahmed wrote:
> > How long were the stalls? Given that rstats are usually flushed by its
> 
> I think 10 seconds while interrupts are disabled is what we need for a
> hard lockup, right?

Oh man, that's a long while. I'd really like to learn more about the
numbers. How many cgroups are being flushed across how many CPUs?

> IIUC you mean that the caller of cgroup_rstat_flush() can call a
> different variant that only flushes a part of the rstat tree then
> returns, and the caller makes several calls interleaved by re-enabling
> irq, right? Because the flushing code seems to already do this
> internally if the non irqsafe version is used.

I was thinking more that being done inside the flush function.

> I think this might be tricky. In this case the path that caused the
> lockup was memcg_check_events()->mem_cgroup_threshold()->__mem_cgroup_threshold()->mem_cgroup_usage()->mem_cgroup_flush_stats().
> Interrupts are disabled by callers of memcg_check_events(), but the
> rstat flush call is made much deeper in the call stack. Whoever is
> disabling interrupts doesn't have access to pause/resume flushing.

Hmm.... yeah I guess it's worthwhile to experiment with selective flushing
for specific paths. That said, we'd still need to address the whole flush
taking long too.

> There are also other code paths that used to use
> cgroup_rstat_flush_irqsafe() directly before mem_cgroup_flush_stats()
> was introduced like mem_cgroup_wb_stats() [1].
> 
> This is why I suggested a selective flushing variant of
> cgroup_rstat_flush_irqsafe(), so that flushers that need irq disabled
> have the ability to only flush a subset of the stats to avoid long
> stalls if possible.

I have nothing against selective flushing but it's not a free thing to do
both in terms of complexity and runtime overhead, so let's get some numbers
on how much time is spent where.

Thanks.

-- 
tejun