Re: [PATCH bpf-next] libbpf: support module BTF for BPF_TYPE_ID_TARGET CO-RE relocation

[Alan Maguire <alan.maguire@oracle.com>]

On Mon, 7 Dec 2020, Andrii Nakryiko wrote:

> On Mon, Dec 7, 2020 at 7:12 PM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > On Mon, Dec 07, 2020 at 04:38:16PM +0000, Alan Maguire wrote:
> > > Sorry about this Andrii, but I'm a bit stuck here.
> > >
> > > I'm struggling to get tests working where the obj fd is used to designate
> > > the module BTF. Unless I'm missing something there are a few problems:
> > >
> > > - the fd association is removed by libbpf when the BPF program has loaded;
> > > the module fds are closed and the module BTF is discarded.  However even if
> > > that isn't done (and as you mentioned, we could hold onto BTF that is in
> > > use, and I commented out the code that does that to test) - there's
> > > another problem:
> > > - I can't see a way to use the object fd value we set here later in BPF
> > > program context; btf_get_by_fd() returns -EBADF as the fd is associated
> > > with the module BTF in the test's process context, not necessarily in
> > > the context that the BPF program is running.  Would it be possible in this
> > > case to use object id? Or is there another way to handle the fd->module
> > > BTF association that we need to make in BPF program context that I'm
> > > missing?
> > > - A more long-term issue; if we use fds to specify module BTFs and write
> > > the object fd into the program, we can pin the BPF program such that it
> > > outlives fds that refer to its associated BTF.  So unless we pinned the
> > > BTF too, any code that assumed the BTF fd-> module mapping was valid would
> > > start to break once the user-space side went away and the pinned program
> > > persisted.
> >
> > All of the above are not issues. They are features of FD based approach.
> > When the program refers to btf via fd the verifier needs to increment btf's refcnt
> > so it won't go away while the prog is running. For module's BTF it means
> > that the module can be unloaded, but its BTF may stay around if there is a prog
> > that needs to access it.
> > I think the missing piece in the above is that btf_get_by_fd() should be
> > done at load time instead of program run-time.
> > Everything FD based needs to behave similar to map_fds where ld_imm64 insn
> > contains map_fd that gets converted to map_ptr by the verifier at load time.
> 
> Right. I was going to extend verifier to do the same for all used BTF
> objects as part of ksym support for module BTFs. So totally agree.
> Just didn't need it so far.
> 

Does this approach prevent more complex run-time specification of BTF 
object fd though?  For example, I've been working on a simple tracer 
focused on kernel debugging; it uses a BPF map entry for each kernel 
function that is traced. User-space populates the map entry with BTF type 
ids for the function arguments/return value, and when the BPF program 
runs it uses the instruction pointer to look up the map entry for that
function, and uses bpf_snprintf_btf() to write the string representations 
of the function arguments/return values.  I'll send out an RFC soon, 
but longer-term I was hoping to extend it to support module-specific 
types.  Would a dynamic case like that - where the BTF module fd is looked 
up in a map entry during program execution (rather than derived via 
__btf_builtin_type_id()) work too? Thanks!

Alan