[RFC] memory.high auto limit daemon

[RFC] memory.high auto limit daemon

[Andrew Moise]

Hi all. So I wrote a daemon which attempts to manage memory.high limits
to keep interactive processes usable during thrashing (Rust compiles,
NixOS builds, etc) without the user needing to do workload-specific
setup. I'd be interested in feedback on idea and implementation, if
anyone is interested to take a look.

Code: https://codeberg.org/andyman/destressd

Basically what it's doing is:

* We watch the PSI stall percentages to see if programs are starting
  to stall because of memory pressure.
* When they are, we try to roughly estimate how much memory each
  cgroup was trying to touch during the last 5 seconds of its
  computation time, if PSI waiting time hadn't been a factor.
* Whatever is the weightiest cgroup by that metric (*), we start
  bumping down its memory.high allotment incrementally until we stop
  seeing PSI-indicated memory stalls in other cgroups.

(*) oversimplifying a bit

To me this seems like experimentally a feasible approach. I did some
testing of how long it takes to do simple browser tasks under pretty
punishing thrashing caused by Rust compiles. There are details in
the tool's README, but overall the results on my machine averaged
over five runs are these:

  phase                    normal   thrashing  thrashing+destressd
  --------------------- ---------  ----------  -------------------
  browser_startup          0.78s       21.63s               14.95s
  reddit_load              2.56s       27.57s               10.02s
  scroll_20               20.99s       39.91s               27.05s

The test is not super-scientific, just an attempt to give numbers to
my subjective feeling that "when we set memory.high on the thrashing
process the system gets a lot more responsive." Of course that outcome
is not surprising, but having a daemon that will adaptively set
memory.high when needed to make it happen seems useful to me.

I would also very much like to get the "thrashing+destressd" numbers
down so they're more similar to the unthrashing case. I probably will
keep working, seeing if there's a way of better containing the
thrashing cgroup so that the interactive stuff just works similarly
to how it does if the system's not loaded. But that up there is
the performance right now with the current approach.

Thoughts? Feedback? I saw a talk where
the idea of limiting memory.high was briefly mentioned
(https://lpc.events/event/7/contributions/649/attachments/546/965/2020-lpc-resource-management.pdf),
and in the tool's README I list some prior art of similar approaches
that I was able to find, but I couldn't find much that's using this
specific memory.high limiting approach, and to me it seems like a good
way. But like I say I'm curious what the people who are in this space
think about it.

-Andrew