Re: [RFC 0/5] fib: shared and resizable tbl8 pool

["Robin Jarry" <rjarry@redhat.com>]

Hi Maxime,

Maxime Leroy, Mar 31, 2026 at 23:41:
> This RFC proposes an optional shared tbl8 pool for FIB/FIB6,
> to address the difficulty of sizing num_tbl8 upfront.
>
> In practice, tbl8 usage depends on prefix distribution and
> evolves over time. In multi-VRF environments, some VRFs are
> elephants (full table, thousands of tbl8 groups) while others
> consume very little (mostly /24 or shorter). Per-FIB sizing
> forces each instance to provision for its worst case, leading
> to significant memory waste.
>
> A shared pool solves this: all FIBs draw from the same tbl8
> memory, so elephant VRFs use what they need while light VRFs
> cost almost nothing. The sharing granularity is flexible: one pool per
> VRF, per address family, a global pool, or no sharing at all.
>
> This series adds:
>
>   - A shared tbl8 pool, replacing per-backend allocation
>     (bitmap in dir24_8, stack in trie) with a common
>     refcounted O(1) stack allocator.
>   - An optional resizable mode (grow via alloc + copy + QSBR
>     synchronize), removing the need to guess peak usage at
>     creation time.
>   - A stats API (rte_fib_tbl8_pool_get_stats()) exposing
>     used/total/max counters.
>
> All features are opt-in:
>
>   - Existing per-FIB allocation remains the default.
>   - Shared pool is enabled via the tbl8_pool config field.
>   - Resize is enabled by setting max_tbl8 > 0 with QSBR.

The shared pool is nice, but dynamic resize is awesome.

I have gone over the implementation and it seems sane to me. The only
concern I might have is the change of tbl8 pool allocator for DIR24_8
from a O(n/64) slab to O(1) stack. I don't know if it can have
a performance impact on lookup or if it only affects the control plane
operations (add/del).

> Shrinking (reducing pool capacity after usage drops) is not
> part of this series. It would always be best-effort since
> there is no compaction: if any tbl8 group near the end of the
> pool is still in use, the pool cannot shrink. The current LIFO
> free-list makes this less likely by immediately reusing freed
> high indices, which prevents a contiguous free tail from
> forming. A different allocation strategy (e.g. a min-heap
> favoring low indices) could improve shrink opportunities, but
> is better addressed separately.

Shrinking would be nice to have but not critical in my opinion. I would
prefer if we could add a dynamic resize feature (and possibly RIB node
mempool sharing) for rte_rib* as well so that FIB objects can really be
scaled up on demand. For now, if you run out of space in the RIB, you
will get an ENOSPC error even if the FIB tbl8 pool still has room.

Nice work, thanks!

> A working integration in Grout is available:
> https://github.com/DPDK/grout/pull/581 (still a draft)
>
> Maxime Leroy (5):
>   test/fib6: zero-initialize config struct
>   fib: share tbl8 definitions between fib and fib6
>   fib: add shared tbl8 pool
>   fib: add resizable tbl8 pool
>   fib: add tbl8 pool stats API
>
>  app/test/test_fib6.c        |  10 +-
>  lib/fib/dir24_8.c           | 234 ++++++++++---------------
>  lib/fib/dir24_8.h           |  17 +-
>  lib/fib/fib_tbl8.h          |  50 ++++++
>  lib/fib/fib_tbl8_pool.c     | 337 ++++++++++++++++++++++++++++++++++++
>  lib/fib/fib_tbl8_pool.h     | 113 ++++++++++++
>  lib/fib/meson.build         |   5 +-
>  lib/fib/rte_fib.h           |   3 +
>  lib/fib/rte_fib6.h          |   3 +
>  lib/fib/rte_fib_tbl8_pool.h | 149 ++++++++++++++++
>  lib/fib/trie.c              | 230 +++++++++---------------
>  lib/fib/trie.h              |  15 +-
>  12 files changed, 844 insertions(+), 322 deletions(-)
>  create mode 100644 lib/fib/fib_tbl8.h
>  create mode 100644 lib/fib/fib_tbl8_pool.c
>  create mode 100644 lib/fib/fib_tbl8_pool.h
>  create mode 100644 lib/fib/rte_fib_tbl8_pool.h


-- 
Robin

> For recreational use only.