Re: [PATCH] misc: introduce strim-memory qapi to support free memory trimming

["Daniel P. Berrangé" <berrange@redhat.com>]

On Sat, Jul 27, 2024 at 01:18:32PM +0800, Guoyi Tu wrote:
> On 2024/7/25 19:57, Daniel P. Berrangé wrote:
> > On Thu, Jul 25, 2024 at 01:35:21PM +0200, Markus Armbruster wrote:
> > > Guoyi Tu <tugy@chinatelecom.cn> writes:
> > > 
> > > > In the test environment, we conducted IO stress tests on all storage disks
> > > > within a virtual machine that had five storage devices mounted.During
> > > > testing,
> > > > we found that the qemu process allocated a large amount of memory (~800MB)
> > > > to handle these IO operations.
> > > > 
> > > > When the test ended, although qemu called free() to release the allocated
> > > > memory, the memory was not actually returned to the operating system, as
> > > > observed via the top command.
> > > > 
> > > > Upon researching the glibc memory management mechanism, we found that when
> > > > small chunks of memory are allocated in user space and then released with
> > > > free(),  the glibc memory management mechanism does not necessarily return
> > > > this memory to the operating system. Instead, it retains the memory until
> > > > certain conditions are met for release.
> > > 
> > > Yes.
> > 
> > Looking at mallopt(3) man page, the M_TRIM_THRESHOLD is said to control
> > when glibc releases the top of the heap back to the OS. It is said to
> > default to 128 kb.
> Yes, the M_TRIM_THRESHOLD option can control glibc to release the free
> memory at the top of the heap, but glibc will not release the free
> memory in the middle of the heap.
> 
> > I'm curious how we get from that default, to 800 MB of unused memory > Is it related to the number of distinct malloc arenas that are in use ?
> 
> At least 600MB of memory is free, and this memory might be in the middle of
> the heap and cannot be automatically released.
> 
> > I'm curious what malloc_stats() would report before & after malloc_trim
> > when QEMU is in this situation with lots of wasted memory.
> Here is the test case:

snip

That looks like an artifical reproducer, rather than the real world
QEMU scenario.

What's the actual I/O stress test scenario you use to reproduce the
problem in QEMU, and how have you configured QEMU (ie what CLI args) ?

I'm fairly inclined to suggest that having such a huge amount of
freed memory is a glibc bug, but to escalate this to glibc requires
us to provide them better real world examples of the problems.

> > The above usage is automatic, while this proposal requires that
> > an external mgmt app monitor QEMU and tell it to free memory.
> > I'm wondering if the latter is really desirable, or whether QEMU
> > can call this itself when reasonable ?
> 
> Yes, I have also considered implementing an automatic memory release
> function within qemu. This approach would require qemu to periodically
> monitor the IO load of all backend storage, and if the IO load is very
> low over a period of time, it would proactively release memory.

I would note that in systemd they have logic which is monitoring
either /proc/pressure/memory or $CGROUP/memory.pressure, and in
response to events on that, it will call malloc_trim

  https://github.com/systemd/systemd/blob/main/docs/MEMORY_PRESSURE.md
  https://docs.kernel.org/accounting/psi.html

Something like that might be better, as it lets us hide the specific
design & impl choices inside QEMU, letting us change/evolve them at
will without impacting public API designs.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|