RTE mempool "used count" steadily goes down to zero despite steady packet throughput

RTE mempool "used count" steadily goes down to zero despite steady packet throughput

[Kamraan Nasim]

Hello,

I have ~15Gbps of traffic flowing through two 10GE ports and been profiling
the rte mempool(or rather the pktmbuf mempool) memory consumption:

I have per lcore caching disabled(cache_size is 0)

I have noticed that:
- Mempool FREE cnt(as given byt rte_mempool_free_count()) increases
- Mempool USED cnt(as given by rte_mempool_used_count() decreases and
eventually drops to 0. When this happens, mempool reports itself as EMPTY
- rx_nombuf stats for the eth ports start climbing
- Valgrind Memcheck does not indicate any obvious leaks in RTE mempool or
my application.


I was wondering if others have come across this issue?  Or if people here
have used ways, besides Valgrind to profile the mempool or the pkt mbuf
pool?

Thanks,
Kam

Re: RTE mempool "used count" steadily goes down to zero despite steady packet throughput

[Olivier MATZ]

Hi Kam,

On 11/28/2014 06:34 PM, Kamraan Nasim wrote:
> I have ~15Gbps of traffic flowing through two 10GE ports and been profiling
> the rte mempool(or rather the pktmbuf mempool) memory consumption:
> 
> I have per lcore caching disabled(cache_size is 0)
> 
> I have noticed that:
> - Mempool FREE cnt(as given byt rte_mempool_free_count()) increases
> - Mempool USED cnt(as given by rte_mempool_used_count() decreases and
> eventually drops to 0. When this happens, mempool reports itself as EMPTY
> - rx_nombuf stats for the eth ports start climbing
> - Valgrind Memcheck does not indicate any obvious leaks in RTE mempool or
> my application.
> 
> 
> I was wondering if others have come across this issue?  Or if people here
> have used ways, besides Valgrind to profile the mempool or the pkt mbuf
> pool?

First, be careful with rte_mempool_free_count(): it returns the number
of free entries in the internal ring of the mempool, which actually
corresponds to the number of allocated objects from the mempool point
of view. See:
http://dpdk.org/browse/dpdk/tree/lib/librte_mempool/rte_mempool.h

If you have the number of allocated objects increasing, and the
statistics rx_nombuf increasing when the mbuf pool is empty, it means
that you have a mbuf leak in your application. Valgrind won't see it
since it does not know about mempool alloc/free functions.

Regards,
Olivier

Re: RTE mempool "used count" steadily goes down to zero despite steady packet throughput

[Kamraan Nasim]

Thanks Oliver for the helpful tip, that settles the confusion as to why
Free count grows and used shrinks.

I've modified the rte_mempool library to maintain atomic pktmbuf malloc and
free stats, hopefully that would help pinpoint the leak.

Thanks again for your help!

--Kam

On Mon, Dec 1, 2014 at 4:58 AM, Olivier MATZ <olivier.matz-pdR9zngts4EAvxtiuMwx3w@public.gmane.org> wrote:

> Hi Kam,
>
> On 11/28/2014 06:34 PM, Kamraan Nasim wrote:
> > I have ~15Gbps of traffic flowing through two 10GE ports and been
> profiling
> > the rte mempool(or rather the pktmbuf mempool) memory consumption:
> >
> > I have per lcore caching disabled(cache_size is 0)
> >
> > I have noticed that:
> > - Mempool FREE cnt(as given byt rte_mempool_free_count()) increases
> > - Mempool USED cnt(as given by rte_mempool_used_count() decreases and
> > eventually drops to 0. When this happens, mempool reports itself as EMPTY
> > - rx_nombuf stats for the eth ports start climbing
> > - Valgrind Memcheck does not indicate any obvious leaks in RTE mempool or
> > my application.
> >
> >
> > I was wondering if others have come across this issue?  Or if people here
> > have used ways, besides Valgrind to profile the mempool or the pkt mbuf
> > pool?
>
> First, be careful with rte_mempool_free_count(): it returns the number
> of free entries in the internal ring of the mempool, which actually
> corresponds to the number of allocated objects from the mempool point
> of view. See:
> http://dpdk.org/browse/dpdk/tree/lib/librte_mempool/rte_mempool.h
>
> If you have the number of allocated objects increasing, and the
> statistics rx_nombuf increasing when the mbuf pool is empty, it means
> that you have a mbuf leak in your application. Valgrind won't see it
> since it does not know about mempool alloc/free functions.
>
> Regards,
> Olivier
>