Re: [PATCH bpf-next v1 RESEND 1/5] vmalloc: introduce vmalloc_exec, vfree_exec, and vcopy_exec

[Luis Chamberlain <mcgrof@kernel.org>]

On Mon, Nov 07, 2022 at 08:58:17AM +0200, Mike Rapoport wrote:
> On Thu, Nov 03, 2022 at 11:59:48AM -0700, Luis Chamberlain wrote:
> > On Thu, Nov 03, 2022 at 05:51:57PM +0200, Mike Rapoport wrote:
> > 
> > > I had to put this project on a backburner for $VARIOUS_REASONS, but I still
> > > think that we need a generic allocator for memory with non-default
> > > permissions in the direct map and that code allocation should build on that
> > > allocator.
> > 
> > It seems this generalization of the bpf prog pack to possibly be used
> > for modules / kprobes / ftrace is a small step in that direction.
> > 
> > > All that said, the direct map fragmentation problem is currently relevant
> > > only to x86 because it's the only architecture that supports splitting of
> > > the large pages in the direct map.
> > 
> > I was thinking even more long term too, using this as a proof of concept. If
> > this practice in general helps with fragmentation, could it be used for
> > experimetnation with compound pages later, as a way to reduce possible
> > fragmentation.
> 
> As Rick already mentioned, these patches help with the direct map
> fragmentation only indirectly. With these patches memory is freed in
> PMD_SIZE chunks and this makes the changes to the direct map in
> vm_remove_mappings() to happen in in PMD_SIZE units and this is pretty much
> the only effect of this series on the direct map layout.

I understand that is what *this* series does. I was wondering is similar
scheme may be useful to study to see if it helps with aggregating say
something like 32 x 64 kb for compound page allocations of order 16 (64 Kib)
to see if it may help with possible fragmentation concerns for that world
where that may be useful in the future (completely unrelated to page
permissions).

> A bit unrelated, but I'm wondering now if we want to have the direct map
> alias of the pages allocated for code also to be read-only...
> 
> > > Whenever a large page in the direct map is split, all
> > > kernel accesses via the direct map will use small pages which requires
> > > dealing with 512 page table entries instead of one for 2M range.
> > > 
> > > Since small pages in the direct map are never collapsed back to large
> > > pages, long living system that heavily uses eBPF programs will have its
> > > direct map severely fragmented, higher TLB miss rate and worse overall
> > > performance. 
> > 
> > Shouldn't compaction help with those situations?
> 
> Compaction helps to reduce fragmentation of the physical memory, it tries
> to bring free physical pages next to each other to create large contiguous
> chunks, but it does not change the virtual addresses the users of the
> underlying data see.

Sorry I understood that 'bpf prog pack' only only used *one* 2 MiB huge
page for *all* eBPF JIT programs, and so the fragmentation issue prior
to 'bpf prog pack' I thought was the fact that as eBPF programs are
still alive, we have fragmentation. In the world without 'bpf prog pack'
I thought no huge pages were used.

> Changing permissions of a small page in the direct map causes
> "discontinuity" in the virtual space. E.g. if we have 2M mapped RW with a
> single PMD changing several page in the middle of those 2M to R+X will
> require to remap that range with 512 PTEs.

Makes sense, thanks.

  Luis