Re: Questions about querying map object information

[Jiri Olsa <olsajiri@gmail.com>]

On Mon, Jul 11, 2022 at 04:35:19PM -0700, Hao Luo wrote:
> Hi Hao Xiang,
> 
> On Mon, Jul 11, 2022 at 3:50 PM Hao Xiang . <hao.xiang@bytedance.com> wrote:
> >
> > Ping...
> >
> > Can someone please help to shed some light on this?
> >
> > Thanks, Hao
> >
> > On Sun, Jul 3, 2022 at 3:33 PM Hao Xiang . <hao.xiang@bytedance.com> wrote:
> > >
> > > Hi everyone,
> > >
> > > I am super new to bpf and the open source community in general. Please
> > > bear with me asking some basic questions.
> > > We are working on a bpf monitoring tool to track the CPU and memory
> > > usage for all bpf programs loaded in the system. We were able to get
> > > CPU usage per bpf program with the BPF_OBJ_GET_INFO_BY_ID syscall on a
> > > bpf prog object. We are trying to do the same on a map object to query
> > > for per map memory usage. The information returned from bpf_map_info
> > > only contains things like max_entries, key_size, value_size, which can
> > > be used to calculate estimated memory allocation size. But we are also
> > > interested in knowing how much memory is actually being used by our
> > > program. For instance, one of our bpf program uses a map with type
> > > hashtable. The hashtable is created with a chunk of pre-allocated
> > > memory based on the max_entres, key_size and value size. The
> > > pre-allocated size is useful information to know but so is the current
> > > number of entries in the hashtable. We used to run into a performance
> > > issue where our bpf map's max_entries is set to be too small and we
> > > end up totally exhausting the pre-allocated memory. So knowing things
> > > like current entry count VS max entry count of a hashtable is useful
> > > information for us.
> > > With that being said, we have a few questions and hopefully we can get
> > > some help from the community.
> > > 1) We couldn't find anything in bpf_map_info to give us the current
> > > entry count of a hashtable. I read that bpf_map_info returns
> > > information about a map object in general. So it makes total sense to
> > > not have information of a particular map type. But is there an
> > > existing place we can get the per map type information (eg, the
> > > current entry count of a hashtable, the number of elements pushed to a
> > > stack, etc)?
> 
> cc more BPF experts for their comments.
> 
> I agree with you that knowing the current space usage of a map is
> quite helpful. In my understanding, a naive and inefficient way to
> estimate space usage is iterating the map and counting the map
> elements. That's doable, but may not be the best method though.
> Regarding auto-adjusting map size, I remember Andrii talked about
> resizable hash maps, maybe he can tell you more [1].

we keep entries count for hash map, maybe we could just add
'current count' to bpf_map_info, it seems generic enough

jirka

> 
> [1] http://vger.kernel.org/bpfconf2022_material/lsfmmbpf2022-algs.pdf
> 
> > > 2) If there isn't an existing place to return map type specific
> > > information, would it make sense to extend the structure bpf_map_info
> > > with a union at the end and have that union to contain per map type
> > > specific information?
> > >
> > > Thanks, Hao