Re: [PATCH 7/7] module: add debug stats to help identify memory pressure

[Luis Chamberlain <mcgrof@kernel.org>]

On Wed, Mar 29, 2023 at 07:46:22AM +0200, Greg KH wrote:
> On Tue, Mar 28, 2023 at 10:31:49PM -0700, Luis Chamberlain wrote:
> > Loading modules with finit_module() can end up using vmalloc(), vmap()
> > and vmalloc() again, for a total of up to 3 separate allocations in the
> > worse case for a single module. We always kernel_read*() the module,
> > that's a vmalloc(). Then vmap() is used for the module decompression,
> > and if so the last read buffer is freed as we use the now decompressed
> > module buffer to stuff data into our copy module. The last one is
> > specific to architectures but pretty much that's generally a series
> > of vmalloc() for different ELF sections...
> > 
> > Evaluation with new stress-ng module support [1] with just 100 ops
> > us proving that you can end up using GiBs of data easily even if
> > we are trying to be very careful not to load modules which are already
> > loaded. 100 ops seems to resemble the sort of pressure a system with
> > about 400 CPUs can create on modules. Although those issues for so
> > many concurrent loads per CPU is silly and are being fixed, we lack
> > proper tooling to help diagnose easily what happened, when it happened
> > and what likely are the culprits -- userspace or kernel module
> > autoloading.
> > 
> > Provide an initial set of stats for debugfs which let us easily scrape
> > post-boot information about failed loads. This sort of information can
> > be used on production worklaods to try to optimize *avoiding* redundant
> > memory pressure using finit_module().
> > 
> > Screen shot:
> > 
> > root@kmod ~ # cat /sys/kernel/debug/modules/stats
> >            Modules loaded       67
> 
> Is this "loaded now", or "ever successfully loaded"?  As in a
> modprobe/rmmod/modprobe would bump this by 2, right?

Ah, the later, so "how modules have I ever loaded". Maybe

Modules ever loaded

?

Will fix the nits, thanks!

> > diff --git a/kernel/module/debug.c b/kernel/module/debug.c
> 
> Why is this a whole separate file?

It's just a style preference, no real hard reason other than
module.c was huge before and now its split up. I find that
easier to review / manage. Certainly overkill for such as
simple thing but if its debug I think I rather see that
then some ifdef eyesore. But that's just preference.

> And as MODULE_DEBUG does not reference debugfs,

That should be fixed thanks.

> > diff --git a/kernel/module/internal.h b/kernel/module/internal.h
> > index 6ae29bb8836f..a645cb3fafc7 100644
> > --- a/kernel/module/internal.h
> > +++ b/kernel/module/internal.h
> > @@ -143,6 +143,41 @@ static inline bool set_livepatch_module(struct module *mod)
> >  #endif
> >  }
> >  
> > +#ifdef CONFIG_MODULE_STATS
> > +
> > +#define mod_stat_add64(count, var) atomic64_add(count, var)
> > +#define mod_stat_inc(name) atomic_inc(name)
> 
> Ok, but:
> 
> > +#define mod_stat_inc(name) atomic_inc(name)
> 
> Why do you still increment the variable here if the option is not
> enabled?

Whoops, will fix!

> Also, didn't we have some sort of "we want to use an atomic variable as
> statistics" type somewhere in the kernel? 

I didn't get the memo, nor do I recall, so it's not on my radar.

> Or did that never get accepted?

Not sure.

> And do all of these really need to be atomic variables?  Don't you have
> locks for some of this to not need the atomic-ness of them?  I guess it
> doesn't matter much as this isn't that fast of a code-path.

That was actually intentional, as this only *grows* I just care its not 0
so to help divide by the total number of modules to get average module
length and average module .text length. I used atomics and made it only
grow precisely to not have to lock anywhere.

  Luis