* [RFC PATCH 0/4] VRF support in FIB library
@ 2026-03-22 15:42 Vladimir Medvedkin
2026-03-22 15:42 ` [RFC PATCH 1/4] fib: add multi-VRF support Vladimir Medvedkin
` (8 more replies)
0 siblings, 9 replies; 45+ messages in thread
From: Vladimir Medvedkin @ 2026-03-22 15:42 UTC (permalink / raw)
To: dev; +Cc: rjarry, nsaxena16, mb, adwivedi, jerinjacobk
This series adds multi-VRF support to both IPv4 and IPv6 FIB paths by
allowing a single FIB instance to host multiple isolated routing domains.
Currently FIB instance represents one routing instance. For workloads that
need multiple VRFs, the only option is to create multiple FIB objects. In a
burst oriented datapath, packets in the same batch can belong to different VRFs, so
the application either does per-packet lookup in different FIB instances or
regroups packets by VRF before lookup. Both approaches are expensive.
To remove that cost, this series keeps all VRFs inside one FIB instance and
extends lookup input with per-packet VRF IDs.
The design follows the existing fast-path structure for both families. IPv4 and
IPv6 use multi-ary trees with a 2^24 associativity on a first level (tbl24). The
first-level table scales per configured VRF. This increases memory usage, but
keeps performance and lookup complexity on par with non-VRF implementation.
Vladimir Medvedkin (4):
fib: add multi-VRF support
fib: add VRF functional and unit tests
fib6: add multi-VRF support
fib6: add VRF functional and unit tests
app/test-fib/main.c | 257 ++++++++++++++++++++++--
app/test/test_fib.c | 298 +++++++++++++++++++++++++++
app/test/test_fib6.c | 319 ++++++++++++++++++++++++++++-
lib/fib/dir24_8.c | 241 ++++++++++++++++------
lib/fib/dir24_8.h | 255 ++++++++++++++++--------
lib/fib/dir24_8_avx512.c | 420 +++++++++++++++++++++++++++++++--------
lib/fib/dir24_8_avx512.h | 80 +++++++-
lib/fib/rte_fib.c | 158 ++++++++++++---
lib/fib/rte_fib.h | 94 ++++++++-
lib/fib/rte_fib6.c | 166 +++++++++++++---
lib/fib/rte_fib6.h | 88 +++++++-
lib/fib/trie.c | 158 +++++++++++----
lib/fib/trie.h | 51 +++--
lib/fib/trie_avx512.c | 225 +++++++++++++++++++--
lib/fib/trie_avx512.h | 39 +++-
15 files changed, 2453 insertions(+), 396 deletions(-)
--
2.43.0
^ permalink raw reply [flat|nested] 45+ messages in thread
* [RFC PATCH 1/4] fib: add multi-VRF support
2026-03-22 15:42 [RFC PATCH 0/4] VRF support in FIB library Vladimir Medvedkin
@ 2026-03-22 15:42 ` Vladimir Medvedkin
2026-03-23 15:48 ` Konstantin Ananyev
2026-03-22 15:42 ` [RFC PATCH 2/4] fib: add VRF functional and unit tests Vladimir Medvedkin
` (7 subsequent siblings)
8 siblings, 1 reply; 45+ messages in thread
From: Vladimir Medvedkin @ 2026-03-22 15:42 UTC (permalink / raw)
To: dev; +Cc: rjarry, nsaxena16, mb, adwivedi, jerinjacobk
Add VRF (Virtual Routing and Forwarding) support to the IPv4
FIB library, allowing multiple independent routing tables
within a single FIB instance.
Introduce max_vrfs and vrf_default_nh fields in rte_fib_conf
to configure the number of VRFs and per-VRF default nexthops.
Add four new experimental APIs:
- rte_fib_vrf_add() and rte_fib_vrf_delete() to manage routes
per VRF
- rte_fib_vrf_lookup_bulk() for multi-VRF bulk lookups
- rte_fib_vrf_get_rib() to retrieve a per-VRF RIB handle
Signed-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
---
lib/fib/dir24_8.c | 241 ++++++++++++++++------
lib/fib/dir24_8.h | 255 ++++++++++++++++--------
lib/fib/dir24_8_avx512.c | 420 +++++++++++++++++++++++++++++++--------
lib/fib/dir24_8_avx512.h | 80 +++++++-
lib/fib/rte_fib.c | 158 ++++++++++++---
lib/fib/rte_fib.h | 94 ++++++++-
6 files changed, 988 insertions(+), 260 deletions(-)
diff --git a/lib/fib/dir24_8.c b/lib/fib/dir24_8.c
index 489d2ef427..ad295c5f16 100644
--- a/lib/fib/dir24_8.c
+++ b/lib/fib/dir24_8.c
@@ -32,41 +32,80 @@
#define ROUNDUP(x, y) RTE_ALIGN_CEIL(x, (1 << (32 - y)))
static inline rte_fib_lookup_fn_t
-get_scalar_fn(enum rte_fib_dir24_8_nh_sz nh_sz, bool be_addr)
+get_scalar_fn(const struct dir24_8_tbl *dp, enum rte_fib_dir24_8_nh_sz nh_sz,
+ bool be_addr)
{
+ bool single_vrf = dp->num_vrfs <= 1;
+
switch (nh_sz) {
case RTE_FIB_DIR24_8_1B:
- return be_addr ? dir24_8_lookup_bulk_1b_be : dir24_8_lookup_bulk_1b;
+ if (single_vrf)
+ return be_addr ? dir24_8_lookup_bulk_1b_be :
+ dir24_8_lookup_bulk_1b;
+ return be_addr ? dir24_8_lookup_bulk_vrf_1b_be :
+ dir24_8_lookup_bulk_vrf_1b;
case RTE_FIB_DIR24_8_2B:
- return be_addr ? dir24_8_lookup_bulk_2b_be : dir24_8_lookup_bulk_2b;
+ if (single_vrf)
+ return be_addr ? dir24_8_lookup_bulk_2b_be :
+ dir24_8_lookup_bulk_2b;
+ return be_addr ? dir24_8_lookup_bulk_vrf_2b_be :
+ dir24_8_lookup_bulk_vrf_2b;
case RTE_FIB_DIR24_8_4B:
- return be_addr ? dir24_8_lookup_bulk_4b_be : dir24_8_lookup_bulk_4b;
+ if (single_vrf)
+ return be_addr ? dir24_8_lookup_bulk_4b_be :
+ dir24_8_lookup_bulk_4b;
+ return be_addr ? dir24_8_lookup_bulk_vrf_4b_be :
+ dir24_8_lookup_bulk_vrf_4b;
case RTE_FIB_DIR24_8_8B:
- return be_addr ? dir24_8_lookup_bulk_8b_be : dir24_8_lookup_bulk_8b;
+ if (single_vrf)
+ return be_addr ? dir24_8_lookup_bulk_8b_be :
+ dir24_8_lookup_bulk_8b;
+ return be_addr ? dir24_8_lookup_bulk_vrf_8b_be :
+ dir24_8_lookup_bulk_vrf_8b;
default:
return NULL;
}
}
static inline rte_fib_lookup_fn_t
-get_scalar_fn_inlined(enum rte_fib_dir24_8_nh_sz nh_sz, bool be_addr)
+get_scalar_fn_inlined(const struct dir24_8_tbl *dp,
+ enum rte_fib_dir24_8_nh_sz nh_sz, bool be_addr)
{
+ bool single_vrf = dp->num_vrfs <= 1;
+
switch (nh_sz) {
case RTE_FIB_DIR24_8_1B:
- return be_addr ? dir24_8_lookup_bulk_0_be : dir24_8_lookup_bulk_0;
+ if (single_vrf)
+ return be_addr ? dir24_8_lookup_bulk_0_be :
+ dir24_8_lookup_bulk_0;
+ return be_addr ? dir24_8_lookup_bulk_vrf_0_be :
+ dir24_8_lookup_bulk_vrf_0;
case RTE_FIB_DIR24_8_2B:
- return be_addr ? dir24_8_lookup_bulk_1_be : dir24_8_lookup_bulk_1;
+ if (single_vrf)
+ return be_addr ? dir24_8_lookup_bulk_1_be :
+ dir24_8_lookup_bulk_1;
+ return be_addr ? dir24_8_lookup_bulk_vrf_1_be :
+ dir24_8_lookup_bulk_vrf_1;
case RTE_FIB_DIR24_8_4B:
- return be_addr ? dir24_8_lookup_bulk_2_be : dir24_8_lookup_bulk_2;
+ if (single_vrf)
+ return be_addr ? dir24_8_lookup_bulk_2_be :
+ dir24_8_lookup_bulk_2;
+ return be_addr ? dir24_8_lookup_bulk_vrf_2_be :
+ dir24_8_lookup_bulk_vrf_2;
case RTE_FIB_DIR24_8_8B:
- return be_addr ? dir24_8_lookup_bulk_3_be : dir24_8_lookup_bulk_3;
+ if (single_vrf)
+ return be_addr ? dir24_8_lookup_bulk_3_be :
+ dir24_8_lookup_bulk_3;
+ return be_addr ? dir24_8_lookup_bulk_vrf_3_be :
+ dir24_8_lookup_bulk_vrf_3;
default:
return NULL;
}
}
static inline rte_fib_lookup_fn_t
-get_vector_fn(enum rte_fib_dir24_8_nh_sz nh_sz, bool be_addr)
+get_vector_fn(const struct dir24_8_tbl *dp, enum rte_fib_dir24_8_nh_sz nh_sz,
+ bool be_addr)
{
#ifdef CC_AVX512_SUPPORT
if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512F) <= 0 ||
@@ -77,24 +116,63 @@ get_vector_fn(enum rte_fib_dir24_8_nh_sz nh_sz, bool be_addr)
if (be_addr && rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512BW) <= 0)
return NULL;
+ if (dp->num_vrfs <= 1) {
+ switch (nh_sz) {
+ case RTE_FIB_DIR24_8_1B:
+ return be_addr ? rte_dir24_8_vec_lookup_bulk_1b_be :
+ rte_dir24_8_vec_lookup_bulk_1b;
+ case RTE_FIB_DIR24_8_2B:
+ return be_addr ? rte_dir24_8_vec_lookup_bulk_2b_be :
+ rte_dir24_8_vec_lookup_bulk_2b;
+ case RTE_FIB_DIR24_8_4B:
+ return be_addr ? rte_dir24_8_vec_lookup_bulk_4b_be :
+ rte_dir24_8_vec_lookup_bulk_4b;
+ case RTE_FIB_DIR24_8_8B:
+ return be_addr ? rte_dir24_8_vec_lookup_bulk_8b_be :
+ rte_dir24_8_vec_lookup_bulk_8b;
+ default:
+ return NULL;
+ }
+ }
+
+ if (dp->num_vrfs >= 256) {
+ switch (nh_sz) {
+ case RTE_FIB_DIR24_8_1B:
+ return be_addr ? rte_dir24_8_vec_lookup_bulk_vrf_1b_be_large :
+ rte_dir24_8_vec_lookup_bulk_vrf_1b_large;
+ case RTE_FIB_DIR24_8_2B:
+ return be_addr ? rte_dir24_8_vec_lookup_bulk_vrf_2b_be_large :
+ rte_dir24_8_vec_lookup_bulk_vrf_2b_large;
+ case RTE_FIB_DIR24_8_4B:
+ return be_addr ? rte_dir24_8_vec_lookup_bulk_vrf_4b_be_large :
+ rte_dir24_8_vec_lookup_bulk_vrf_4b_large;
+ case RTE_FIB_DIR24_8_8B:
+ return be_addr ? rte_dir24_8_vec_lookup_bulk_vrf_8b_be_large :
+ rte_dir24_8_vec_lookup_bulk_vrf_8b_large;
+ default:
+ return NULL;
+ }
+ }
+
switch (nh_sz) {
case RTE_FIB_DIR24_8_1B:
- return be_addr ? rte_dir24_8_vec_lookup_bulk_1b_be :
- rte_dir24_8_vec_lookup_bulk_1b;
+ return be_addr ? rte_dir24_8_vec_lookup_bulk_vrf_1b_be :
+ rte_dir24_8_vec_lookup_bulk_vrf_1b;
case RTE_FIB_DIR24_8_2B:
- return be_addr ? rte_dir24_8_vec_lookup_bulk_2b_be :
- rte_dir24_8_vec_lookup_bulk_2b;
+ return be_addr ? rte_dir24_8_vec_lookup_bulk_vrf_2b_be :
+ rte_dir24_8_vec_lookup_bulk_vrf_2b;
case RTE_FIB_DIR24_8_4B:
- return be_addr ? rte_dir24_8_vec_lookup_bulk_4b_be :
- rte_dir24_8_vec_lookup_bulk_4b;
+ return be_addr ? rte_dir24_8_vec_lookup_bulk_vrf_4b_be :
+ rte_dir24_8_vec_lookup_bulk_vrf_4b;
case RTE_FIB_DIR24_8_8B:
- return be_addr ? rte_dir24_8_vec_lookup_bulk_8b_be :
- rte_dir24_8_vec_lookup_bulk_8b;
+ return be_addr ? rte_dir24_8_vec_lookup_bulk_vrf_8b_be :
+ rte_dir24_8_vec_lookup_bulk_vrf_8b;
default:
return NULL;
}
#elif defined(RTE_RISCV_FEATURE_V)
RTE_SET_USED(be_addr);
+ RTE_SET_USED(dp);
if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_RISCV_ISA_V) <= 0)
return NULL;
switch (nh_sz) {
@@ -130,16 +208,17 @@ dir24_8_get_lookup_fn(void *p, enum rte_fib_lookup_type type, bool be_addr)
switch (type) {
case RTE_FIB_LOOKUP_DIR24_8_SCALAR_MACRO:
- return get_scalar_fn(nh_sz, be_addr);
+ return get_scalar_fn(dp, nh_sz, be_addr);
case RTE_FIB_LOOKUP_DIR24_8_SCALAR_INLINE:
- return get_scalar_fn_inlined(nh_sz, be_addr);
+ return get_scalar_fn_inlined(dp, nh_sz, be_addr);
case RTE_FIB_LOOKUP_DIR24_8_SCALAR_UNI:
- return be_addr ? dir24_8_lookup_bulk_uni_be : dir24_8_lookup_bulk_uni;
+ return be_addr ? dir24_8_lookup_bulk_uni_be :
+ dir24_8_lookup_bulk_uni;
case RTE_FIB_LOOKUP_DIR24_8_VECTOR_AVX512:
- return get_vector_fn(nh_sz, be_addr);
+ return get_vector_fn(dp, nh_sz, be_addr);
case RTE_FIB_LOOKUP_DEFAULT:
- ret_fn = get_vector_fn(nh_sz, be_addr);
- return ret_fn != NULL ? ret_fn : get_scalar_fn(nh_sz, be_addr);
+ ret_fn = get_vector_fn(dp, nh_sz, be_addr);
+ return ret_fn != NULL ? ret_fn : get_scalar_fn(dp, nh_sz, be_addr);
default:
return NULL;
}
@@ -246,15 +325,18 @@ __rcu_qsbr_free_resource(void *p, void *data, unsigned int n __rte_unused)
}
static void
-tbl8_recycle(struct dir24_8_tbl *dp, uint32_t ip, uint64_t tbl8_idx)
+tbl8_recycle(struct dir24_8_tbl *dp, uint16_t vrf_id, uint32_t ip, uint64_t tbl8_idx)
{
uint32_t i;
uint64_t nh;
+ uint64_t tbl24_idx;
uint8_t *ptr8;
uint16_t *ptr16;
uint32_t *ptr32;
uint64_t *ptr64;
+ tbl24_idx = get_tbl24_idx(vrf_id, ip);
+
switch (dp->nh_sz) {
case RTE_FIB_DIR24_8_1B:
ptr8 = &((uint8_t *)dp->tbl8)[tbl8_idx *
@@ -264,7 +346,7 @@ tbl8_recycle(struct dir24_8_tbl *dp, uint32_t ip, uint64_t tbl8_idx)
if (nh != ptr8[i])
return;
}
- ((uint8_t *)dp->tbl24)[ip >> 8] =
+ ((uint8_t *)dp->tbl24)[tbl24_idx] =
nh & ~DIR24_8_EXT_ENT;
break;
case RTE_FIB_DIR24_8_2B:
@@ -275,7 +357,7 @@ tbl8_recycle(struct dir24_8_tbl *dp, uint32_t ip, uint64_t tbl8_idx)
if (nh != ptr16[i])
return;
}
- ((uint16_t *)dp->tbl24)[ip >> 8] =
+ ((uint16_t *)dp->tbl24)[tbl24_idx] =
nh & ~DIR24_8_EXT_ENT;
break;
case RTE_FIB_DIR24_8_4B:
@@ -286,7 +368,7 @@ tbl8_recycle(struct dir24_8_tbl *dp, uint32_t ip, uint64_t tbl8_idx)
if (nh != ptr32[i])
return;
}
- ((uint32_t *)dp->tbl24)[ip >> 8] =
+ ((uint32_t *)dp->tbl24)[tbl24_idx] =
nh & ~DIR24_8_EXT_ENT;
break;
case RTE_FIB_DIR24_8_8B:
@@ -297,7 +379,7 @@ tbl8_recycle(struct dir24_8_tbl *dp, uint32_t ip, uint64_t tbl8_idx)
if (nh != ptr64[i])
return;
}
- ((uint64_t *)dp->tbl24)[ip >> 8] =
+ ((uint64_t *)dp->tbl24)[tbl24_idx] =
nh & ~DIR24_8_EXT_ENT;
break;
}
@@ -314,7 +396,7 @@ tbl8_recycle(struct dir24_8_tbl *dp, uint32_t ip, uint64_t tbl8_idx)
}
static int
-install_to_fib(struct dir24_8_tbl *dp, uint32_t ledge, uint32_t redge,
+install_to_fib(struct dir24_8_tbl *dp, uint16_t vrf_id, uint32_t ledge, uint32_t redge,
uint64_t next_hop)
{
uint64_t tbl24_tmp;
@@ -328,7 +410,7 @@ install_to_fib(struct dir24_8_tbl *dp, uint32_t ledge, uint32_t redge,
if (((ledge >> 8) != (redge >> 8)) || (len == 1 << 24)) {
if ((ROUNDUP(ledge, 24) - ledge) != 0) {
- tbl24_tmp = get_tbl24(dp, ledge, dp->nh_sz);
+ tbl24_tmp = get_tbl24(dp, vrf_id, ledge, dp->nh_sz);
if ((tbl24_tmp & DIR24_8_EXT_ENT) !=
DIR24_8_EXT_ENT) {
/**
@@ -346,7 +428,7 @@ install_to_fib(struct dir24_8_tbl *dp, uint32_t ledge, uint32_t redge,
}
tbl8_free_idx(dp, tmp_tbl8_idx);
/*update dir24 entry with tbl8 index*/
- write_to_fib(get_tbl24_p(dp, ledge,
+ write_to_fib(get_tbl24_p(dp, vrf_id, ledge,
dp->nh_sz), (tbl8_idx << 1)|
DIR24_8_EXT_ENT,
dp->nh_sz, 1);
@@ -360,19 +442,19 @@ install_to_fib(struct dir24_8_tbl *dp, uint32_t ledge, uint32_t redge,
write_to_fib((void *)tbl8_ptr, (next_hop << 1)|
DIR24_8_EXT_ENT,
dp->nh_sz, ROUNDUP(ledge, 24) - ledge);
- tbl8_recycle(dp, ledge, tbl8_idx);
+ tbl8_recycle(dp, vrf_id, ledge, tbl8_idx);
}
- write_to_fib(get_tbl24_p(dp, ROUNDUP(ledge, 24), dp->nh_sz),
+ write_to_fib(get_tbl24_p(dp, vrf_id, ROUNDUP(ledge, 24), dp->nh_sz),
next_hop << 1, dp->nh_sz, len);
if (redge & ~DIR24_8_TBL24_MASK) {
- tbl24_tmp = get_tbl24(dp, redge, dp->nh_sz);
+ tbl24_tmp = get_tbl24(dp, vrf_id, redge, dp->nh_sz);
if ((tbl24_tmp & DIR24_8_EXT_ENT) !=
DIR24_8_EXT_ENT) {
tbl8_idx = tbl8_alloc(dp, tbl24_tmp);
if (tbl8_idx < 0)
return -ENOSPC;
/*update dir24 entry with tbl8 index*/
- write_to_fib(get_tbl24_p(dp, redge,
+ write_to_fib(get_tbl24_p(dp, vrf_id, redge,
dp->nh_sz), (tbl8_idx << 1)|
DIR24_8_EXT_ENT,
dp->nh_sz, 1);
@@ -385,17 +467,17 @@ install_to_fib(struct dir24_8_tbl *dp, uint32_t ledge, uint32_t redge,
write_to_fib((void *)tbl8_ptr, (next_hop << 1)|
DIR24_8_EXT_ENT,
dp->nh_sz, redge & ~DIR24_8_TBL24_MASK);
- tbl8_recycle(dp, redge, tbl8_idx);
+ tbl8_recycle(dp, vrf_id, redge, tbl8_idx);
}
} else if ((redge - ledge) != 0) {
- tbl24_tmp = get_tbl24(dp, ledge, dp->nh_sz);
+ tbl24_tmp = get_tbl24(dp, vrf_id, ledge, dp->nh_sz);
if ((tbl24_tmp & DIR24_8_EXT_ENT) !=
DIR24_8_EXT_ENT) {
tbl8_idx = tbl8_alloc(dp, tbl24_tmp);
if (tbl8_idx < 0)
return -ENOSPC;
/*update dir24 entry with tbl8 index*/
- write_to_fib(get_tbl24_p(dp, ledge, dp->nh_sz),
+ write_to_fib(get_tbl24_p(dp, vrf_id, ledge, dp->nh_sz),
(tbl8_idx << 1)|
DIR24_8_EXT_ENT,
dp->nh_sz, 1);
@@ -409,13 +491,13 @@ install_to_fib(struct dir24_8_tbl *dp, uint32_t ledge, uint32_t redge,
write_to_fib((void *)tbl8_ptr, (next_hop << 1)|
DIR24_8_EXT_ENT,
dp->nh_sz, redge - ledge);
- tbl8_recycle(dp, ledge, tbl8_idx);
+ tbl8_recycle(dp, vrf_id, ledge, tbl8_idx);
}
return 0;
}
static int
-modify_fib(struct dir24_8_tbl *dp, struct rte_rib *rib, uint32_t ip,
+modify_fib(struct dir24_8_tbl *dp, struct rte_rib *rib, uint16_t vrf_id, uint32_t ip,
uint8_t depth, uint64_t next_hop)
{
struct rte_rib_node *tmp = NULL;
@@ -438,7 +520,7 @@ modify_fib(struct dir24_8_tbl *dp, struct rte_rib *rib, uint32_t ip,
(uint32_t)(1ULL << (32 - tmp_depth));
continue;
}
- ret = install_to_fib(dp, ledge, redge,
+ ret = install_to_fib(dp, vrf_id, ledge, redge,
next_hop);
if (ret != 0)
return ret;
@@ -454,7 +536,7 @@ modify_fib(struct dir24_8_tbl *dp, struct rte_rib *rib, uint32_t ip,
redge = ip + (uint32_t)(1ULL << (32 - depth));
if (ledge == redge && ledge != 0)
break;
- ret = install_to_fib(dp, ledge, redge,
+ ret = install_to_fib(dp, vrf_id, ledge, redge,
next_hop);
if (ret != 0)
return ret;
@@ -465,7 +547,7 @@ modify_fib(struct dir24_8_tbl *dp, struct rte_rib *rib, uint32_t ip,
}
int
-dir24_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t depth,
+dir24_8_modify(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip, uint8_t depth,
uint64_t next_hop, int op)
{
struct dir24_8_tbl *dp;
@@ -480,8 +562,13 @@ dir24_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t depth,
return -EINVAL;
dp = rte_fib_get_dp(fib);
- rib = rte_fib_get_rib(fib);
- RTE_ASSERT((dp != NULL) && (rib != NULL));
+ RTE_ASSERT(dp != NULL);
+
+ if (vrf_id >= dp->num_vrfs)
+ return -EINVAL;
+
+ rib = rte_fib_vrf_get_rib(fib, vrf_id);
+ RTE_ASSERT(rib != NULL);
if (next_hop > get_max_nh(dp->nh_sz))
return -EINVAL;
@@ -495,7 +582,7 @@ dir24_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t depth,
rte_rib_get_nh(node, &node_nh);
if (node_nh == next_hop)
return 0;
- ret = modify_fib(dp, rib, ip, depth, next_hop);
+ ret = modify_fib(dp, rib, vrf_id, ip, depth, next_hop);
if (ret == 0)
rte_rib_set_nh(node, next_hop);
return 0;
@@ -518,7 +605,7 @@ dir24_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t depth,
if (par_nh == next_hop)
goto successfully_added;
}
- ret = modify_fib(dp, rib, ip, depth, next_hop);
+ ret = modify_fib(dp, rib, vrf_id, ip, depth, next_hop);
if (ret != 0) {
rte_rib_remove(rib, ip, depth);
return ret;
@@ -536,9 +623,9 @@ dir24_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t depth,
rte_rib_get_nh(parent, &par_nh);
rte_rib_get_nh(node, &node_nh);
if (par_nh != node_nh)
- ret = modify_fib(dp, rib, ip, depth, par_nh);
+ ret = modify_fib(dp, rib, vrf_id, ip, depth, par_nh);
} else
- ret = modify_fib(dp, rib, ip, depth, dp->def_nh);
+ ret = modify_fib(dp, rib, vrf_id, ip, depth, dp->def_nh[vrf_id]);
if (ret == 0) {
rte_rib_remove(rib, ip, depth);
if (depth > 24) {
@@ -562,7 +649,10 @@ dir24_8_create(const char *name, int socket_id, struct rte_fib_conf *fib_conf)
struct dir24_8_tbl *dp;
uint64_t def_nh;
uint32_t num_tbl8;
+ uint16_t num_vrfs;
enum rte_fib_dir24_8_nh_sz nh_sz;
+ uint64_t tbl24_sz;
+ uint16_t vrf;
if ((name == NULL) || (fib_conf == NULL) ||
(fib_conf->dir24_8.nh_sz < RTE_FIB_DIR24_8_1B) ||
@@ -580,19 +670,56 @@ dir24_8_create(const char *name, int socket_id, struct rte_fib_conf *fib_conf)
nh_sz = fib_conf->dir24_8.nh_sz;
num_tbl8 = RTE_ALIGN_CEIL(fib_conf->dir24_8.num_tbl8,
BITMAP_SLAB_BIT_SIZE);
+ num_vrfs = (fib_conf->max_vrfs == 0) ? 1 : fib_conf->max_vrfs;
+
+ /* Validate per-VRF default nexthops if provided */
+ if (fib_conf->vrf_default_nh != NULL) {
+ for (vrf = 0; vrf < num_vrfs; vrf++) {
+ if (fib_conf->vrf_default_nh[vrf] > get_max_nh(nh_sz)) {
+ rte_errno = EINVAL;
+ return NULL;
+ }
+ }
+ }
+
+ tbl24_sz = (uint64_t)num_vrfs * DIR24_8_TBL24_NUM_ENT * (1 << nh_sz);
snprintf(mem_name, sizeof(mem_name), "DP_%s", name);
dp = rte_zmalloc_socket(name, sizeof(struct dir24_8_tbl) +
- DIR24_8_TBL24_NUM_ENT * (1 << nh_sz) + sizeof(uint32_t),
+ tbl24_sz + sizeof(uint32_t),
RTE_CACHE_LINE_SIZE, socket_id);
if (dp == NULL) {
rte_errno = ENOMEM;
return NULL;
}
- /* Init table with default value */
- write_to_fib(dp->tbl24, (def_nh << 1), nh_sz, 1 << 24);
+ dp->num_vrfs = num_vrfs;
+ dp->nh_sz = nh_sz;
+ dp->number_tbl8s = num_tbl8;
+
+ /* Allocate per-VRF default nexthop array */
+ snprintf(mem_name, sizeof(mem_name), "DEFNH_%p", dp);
+ dp->def_nh = rte_zmalloc_socket(mem_name, num_vrfs * sizeof(uint64_t),
+ RTE_CACHE_LINE_SIZE, socket_id);
+ if (dp->def_nh == NULL) {
+ rte_errno = ENOMEM;
+ rte_free(dp);
+ return NULL;
+ }
+
+ /* Initialize all VRFs with default nexthop */
+ for (vrf = 0; vrf < num_vrfs; vrf++) {
+ uint64_t vrf_def_nh = (fib_conf->vrf_default_nh != NULL) ?
+ fib_conf->vrf_default_nh[vrf] : def_nh;
+ dp->def_nh[vrf] = vrf_def_nh;
+ /* Init TBL24 for this VRF with default value */
+ uint64_t vrf_offset = (uint64_t)vrf * DIR24_8_TBL24_NUM_ENT;
+ void *vrf_tbl24 = (void *)&((uint8_t *)dp->tbl24)[vrf_offset << nh_sz];
+ write_to_fib(vrf_tbl24, (vrf_def_nh << 1), nh_sz, 1 << 24);
+ }
+
+ /* Allocate shared TBL8 for all VRFs */
snprintf(mem_name, sizeof(mem_name), "TBL8_%p", dp);
uint64_t tbl8_sz = DIR24_8_TBL8_GRP_NUM_ENT * (1ULL << nh_sz) *
(num_tbl8 + 1);
@@ -600,12 +727,10 @@ dir24_8_create(const char *name, int socket_id, struct rte_fib_conf *fib_conf)
RTE_CACHE_LINE_SIZE, socket_id);
if (dp->tbl8 == NULL) {
rte_errno = ENOMEM;
+ rte_free(dp->def_nh);
rte_free(dp);
return NULL;
}
- dp->def_nh = def_nh;
- dp->nh_sz = nh_sz;
- dp->number_tbl8s = num_tbl8;
snprintf(mem_name, sizeof(mem_name), "TBL8_idxes_%p", dp);
dp->tbl8_idxes = rte_zmalloc_socket(mem_name,
@@ -614,6 +739,7 @@ dir24_8_create(const char *name, int socket_id, struct rte_fib_conf *fib_conf)
if (dp->tbl8_idxes == NULL) {
rte_errno = ENOMEM;
rte_free(dp->tbl8);
+ rte_free(dp->def_nh);
rte_free(dp);
return NULL;
}
@@ -629,6 +755,7 @@ dir24_8_free(void *p)
rte_rcu_qsbr_dq_delete(dp->dq);
rte_free(dp->tbl8_idxes);
rte_free(dp->tbl8);
+ rte_free(dp->def_nh);
rte_free(dp);
}
diff --git a/lib/fib/dir24_8.h b/lib/fib/dir24_8.h
index b343b5d686..37a73a3cc2 100644
--- a/lib/fib/dir24_8.h
+++ b/lib/fib/dir24_8.h
@@ -12,6 +12,7 @@
#include <rte_byteorder.h>
#include <rte_prefetch.h>
#include <rte_branch_prediction.h>
+#include <rte_debug.h>
#include <rte_rcu_qsbr.h>
/**
@@ -32,24 +33,19 @@ struct dir24_8_tbl {
uint32_t number_tbl8s; /**< Total number of tbl8s */
uint32_t rsvd_tbl8s; /**< Number of reserved tbl8s */
uint32_t cur_tbl8s; /**< Current number of tbl8s */
+ uint16_t num_vrfs; /**< Number of VRFs */
enum rte_fib_dir24_8_nh_sz nh_sz; /**< Size of nexthop entry */
/* RCU config. */
enum rte_fib_qsbr_mode rcu_mode;/* Blocking, defer queue. */
struct rte_rcu_qsbr *v; /* RCU QSBR variable. */
struct rte_rcu_qsbr_dq *dq; /* RCU QSBR defer queue. */
- uint64_t def_nh; /**< Default next hop */
+ uint64_t *def_nh; /**< Per-VRF default next hop array */
uint64_t *tbl8; /**< tbl8 table. */
uint64_t *tbl8_idxes; /**< bitmap containing free tbl8 idxes*/
/* tbl24 table. */
alignas(RTE_CACHE_LINE_SIZE) uint64_t tbl24[];
};
-static inline void *
-get_tbl24_p(struct dir24_8_tbl *dp, uint32_t ip, uint8_t nh_sz)
-{
- return (void *)&((uint8_t *)dp->tbl24)[(ip &
- DIR24_8_TBL24_MASK) >> (8 - nh_sz)];
-}
static inline uint8_t
bits_in_nh(uint8_t nh_sz)
@@ -63,14 +59,21 @@ get_max_nh(uint8_t nh_sz)
return ((1ULL << (bits_in_nh(nh_sz) - 1)) - 1);
}
-static inline uint32_t
-get_tbl24_idx(uint32_t ip)
+static inline uint64_t
+get_tbl24_idx(uint16_t vrf_id, uint32_t ip)
+{
+ return ((uint64_t)vrf_id << 24) + (ip >> 8);
+}
+
+static inline void *
+get_tbl24_p(struct dir24_8_tbl *dp, uint16_t vrf_id, uint32_t ip, uint8_t nh_sz)
{
- return ip >> 8;
+ uint64_t idx = get_tbl24_idx(vrf_id, ip);
+ return (void *)&((uint8_t *)dp->tbl24)[idx << nh_sz];
}
-static inline uint32_t
-get_tbl8_idx(uint32_t res, uint32_t ip)
+static inline uint64_t
+get_tbl8_idx(uint64_t res, uint32_t ip)
{
return (res >> 1) * DIR24_8_TBL8_GRP_NUM_ENT + (uint8_t)ip;
}
@@ -87,17 +90,18 @@ get_psd_idx(uint32_t val, uint8_t nh_sz)
return val & ((1 << (3 - nh_sz)) - 1);
}
-static inline uint32_t
-get_tbl_idx(uint32_t val, uint8_t nh_sz)
+static inline uint64_t
+get_tbl_idx(uint64_t val, uint8_t nh_sz)
{
return val >> (3 - nh_sz);
}
static inline uint64_t
-get_tbl24(struct dir24_8_tbl *dp, uint32_t ip, uint8_t nh_sz)
+get_tbl24(struct dir24_8_tbl *dp, uint16_t vrf_id, uint32_t ip, uint8_t nh_sz)
{
- return ((dp->tbl24[get_tbl_idx(get_tbl24_idx(ip), nh_sz)] >>
- (get_psd_idx(get_tbl24_idx(ip), nh_sz) *
+ uint64_t idx = get_tbl24_idx(vrf_id, ip);
+ return ((dp->tbl24[get_tbl_idx(idx, nh_sz)] >>
+ (get_psd_idx(idx, nh_sz) *
bits_in_nh(nh_sz))) & lookup_msk(nh_sz));
}
@@ -115,62 +119,92 @@ is_entry_extended(uint64_t ent)
return (ent & DIR24_8_EXT_ENT) == DIR24_8_EXT_ENT;
}
-#define LOOKUP_FUNC(suffix, type, bulk_prefetch, nh_sz) \
-static inline void dir24_8_lookup_bulk_##suffix(void *p, const uint32_t *ips, \
- uint64_t *next_hops, const unsigned int n) \
-{ \
- struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p; \
- uint64_t tmp; \
- uint32_t i; \
- uint32_t prefetch_offset = \
- RTE_MIN((unsigned int)bulk_prefetch, n); \
- \
- for (i = 0; i < prefetch_offset; i++) \
- rte_prefetch0(get_tbl24_p(dp, ips[i], nh_sz)); \
- for (i = 0; i < (n - prefetch_offset); i++) { \
- rte_prefetch0(get_tbl24_p(dp, \
- ips[i + prefetch_offset], nh_sz)); \
- tmp = ((type *)dp->tbl24)[ips[i] >> 8]; \
- if (unlikely(is_entry_extended(tmp))) \
- tmp = ((type *)dp->tbl8)[(uint8_t)ips[i] + \
- ((tmp >> 1) * DIR24_8_TBL8_GRP_NUM_ENT)]; \
- next_hops[i] = tmp >> 1; \
- } \
- for (; i < n; i++) { \
- tmp = ((type *)dp->tbl24)[ips[i] >> 8]; \
- if (unlikely(is_entry_extended(tmp))) \
- tmp = ((type *)dp->tbl8)[(uint8_t)ips[i] + \
- ((tmp >> 1) * DIR24_8_TBL8_GRP_NUM_ENT)]; \
- next_hops[i] = tmp >> 1; \
- } \
-} \
-
-LOOKUP_FUNC(1b, uint8_t, 5, 0)
-LOOKUP_FUNC(2b, uint16_t, 6, 1)
-LOOKUP_FUNC(4b, uint32_t, 15, 2)
-LOOKUP_FUNC(8b, uint64_t, 12, 3)
+
+#define LOOKUP_FUNC(suffix, type, bulk_prefetch, nh_sz, is_vrf) \
+static inline void dir24_8_lookup_bulk_##suffix(void *p, \
+ const uint16_t *vrf_ids, const uint32_t *ips, \
+ uint64_t *next_hops, const unsigned int n) \
+{ \
+ struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p; \
+ uint64_t tmp; \
+ uint32_t i; \
+ uint32_t prefetch_offset = RTE_MIN((unsigned int)bulk_prefetch, n); \
+ \
+ if (!is_vrf) \
+ RTE_SET_USED(vrf_ids); \
+ \
+ for (i = 0; i < prefetch_offset; i++) { \
+ uint16_t vid = is_vrf ? vrf_ids[i] : 0; \
+ RTE_ASSERT(vid < dp->num_vrfs); \
+ rte_prefetch0(get_tbl24_p(dp, vid, ips[i], nh_sz)); \
+ } \
+ for (i = 0; i < (n - prefetch_offset); i++) { \
+ uint16_t vid = is_vrf ? vrf_ids[i] : 0; \
+ uint16_t vid_next = is_vrf ? vrf_ids[i + prefetch_offset] : 0; \
+ RTE_ASSERT(vid < dp->num_vrfs); \
+ RTE_ASSERT(vid_next < dp->num_vrfs); \
+ rte_prefetch0(get_tbl24_p(dp, vid_next, \
+ ips[i + prefetch_offset], nh_sz)); \
+ tmp = ((type *)dp->tbl24)[get_tbl24_idx(vid, ips[i])]; \
+ if (unlikely(is_entry_extended(tmp))) \
+ tmp = ((type *)dp->tbl8)[(uint8_t)ips[i] + \
+ ((tmp >> 1) * DIR24_8_TBL8_GRP_NUM_ENT)]; \
+ next_hops[i] = tmp >> 1; \
+ } \
+ for (; i < n; i++) { \
+ uint16_t vid = is_vrf ? vrf_ids[i] : 0; \
+ RTE_ASSERT(vid < dp->num_vrfs); \
+ tmp = ((type *)dp->tbl24)[get_tbl24_idx(vid, ips[i])]; \
+ if (unlikely(is_entry_extended(tmp))) \
+ tmp = ((type *)dp->tbl8)[(uint8_t)ips[i] + \
+ ((tmp >> 1) * DIR24_8_TBL8_GRP_NUM_ENT)]; \
+ next_hops[i] = tmp >> 1; \
+ } \
+}
+
+LOOKUP_FUNC(1b, uint8_t, 5, 0, false)
+LOOKUP_FUNC(2b, uint16_t, 6, 1, false)
+LOOKUP_FUNC(4b, uint32_t, 15, 2, false)
+LOOKUP_FUNC(8b, uint64_t, 12, 3, false)
+LOOKUP_FUNC(vrf_1b, uint8_t, 5, 0, true)
+LOOKUP_FUNC(vrf_2b, uint16_t, 6, 1, true)
+LOOKUP_FUNC(vrf_4b, uint32_t, 15, 2, true)
+LOOKUP_FUNC(vrf_8b, uint64_t, 12, 3, true)
static inline void
-dir24_8_lookup_bulk(struct dir24_8_tbl *dp, const uint32_t *ips,
- uint64_t *next_hops, const unsigned int n, uint8_t nh_sz)
+__dir24_8_lookup_bulk(struct dir24_8_tbl *dp, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n,
+ uint8_t nh_sz, bool is_vrf)
{
uint64_t tmp;
uint32_t i;
uint32_t prefetch_offset = RTE_MIN(15U, n);
- for (i = 0; i < prefetch_offset; i++)
- rte_prefetch0(get_tbl24_p(dp, ips[i], nh_sz));
+ if (!is_vrf)
+ RTE_SET_USED(vrf_ids);
+
+ for (i = 0; i < prefetch_offset; i++) {
+ uint16_t vid = is_vrf ? vrf_ids[i] : 0;
+ RTE_ASSERT(vid < dp->num_vrfs);
+ rte_prefetch0(get_tbl24_p(dp, vid, ips[i], nh_sz));
+ }
for (i = 0; i < (n - prefetch_offset); i++) {
- rte_prefetch0(get_tbl24_p(dp, ips[i + prefetch_offset],
- nh_sz));
- tmp = get_tbl24(dp, ips[i], nh_sz);
+ uint16_t vid = is_vrf ? vrf_ids[i] : 0;
+ uint16_t vid_next = is_vrf ? vrf_ids[i + prefetch_offset] : 0;
+ RTE_ASSERT(vid < dp->num_vrfs);
+ RTE_ASSERT(vid_next < dp->num_vrfs);
+ rte_prefetch0(get_tbl24_p(dp, vid_next,
+ ips[i + prefetch_offset], nh_sz));
+ tmp = get_tbl24(dp, vid, ips[i], nh_sz);
if (unlikely(is_entry_extended(tmp)))
tmp = get_tbl8(dp, tmp, ips[i], nh_sz);
next_hops[i] = tmp >> 1;
}
for (; i < n; i++) {
- tmp = get_tbl24(dp, ips[i], nh_sz);
+ uint16_t vid = is_vrf ? vrf_ids[i] : 0;
+ RTE_ASSERT(vid < dp->num_vrfs);
+ tmp = get_tbl24(dp, vid, ips[i], nh_sz);
if (unlikely(is_entry_extended(tmp)))
tmp = get_tbl8(dp, tmp, ips[i], nh_sz);
@@ -179,43 +213,79 @@ dir24_8_lookup_bulk(struct dir24_8_tbl *dp, const uint32_t *ips,
}
static inline void
-dir24_8_lookup_bulk_0(void *p, const uint32_t *ips,
+dir24_8_lookup_bulk_0(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
uint64_t *next_hops, const unsigned int n)
{
struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
- dir24_8_lookup_bulk(dp, ips, next_hops, n, 0);
+ __dir24_8_lookup_bulk(dp, vrf_ids, ips, next_hops, n, 0, false);
+}
+
+static inline void
+dir24_8_lookup_bulk_vrf_0(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
+{
+ struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
+
+ __dir24_8_lookup_bulk(dp, vrf_ids, ips, next_hops, n, 0, true);
}
static inline void
-dir24_8_lookup_bulk_1(void *p, const uint32_t *ips,
+dir24_8_lookup_bulk_1(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
uint64_t *next_hops, const unsigned int n)
{
struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
- dir24_8_lookup_bulk(dp, ips, next_hops, n, 1);
+ __dir24_8_lookup_bulk(dp, vrf_ids, ips, next_hops, n, 1, false);
}
static inline void
-dir24_8_lookup_bulk_2(void *p, const uint32_t *ips,
+dir24_8_lookup_bulk_vrf_1(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
+{
+ struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
+
+ __dir24_8_lookup_bulk(dp, vrf_ids, ips, next_hops, n, 1, true);
+}
+
+static inline void
+dir24_8_lookup_bulk_2(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
uint64_t *next_hops, const unsigned int n)
{
struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
- dir24_8_lookup_bulk(dp, ips, next_hops, n, 2);
+ __dir24_8_lookup_bulk(dp, vrf_ids, ips, next_hops, n, 2, false);
}
static inline void
-dir24_8_lookup_bulk_3(void *p, const uint32_t *ips,
+dir24_8_lookup_bulk_vrf_2(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
+{
+ struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
+
+ __dir24_8_lookup_bulk(dp, vrf_ids, ips, next_hops, n, 2, true);
+}
+
+static inline void
+dir24_8_lookup_bulk_3(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
uint64_t *next_hops, const unsigned int n)
{
struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
- dir24_8_lookup_bulk(dp, ips, next_hops, n, 3);
+ __dir24_8_lookup_bulk(dp, vrf_ids, ips, next_hops, n, 3, false);
}
static inline void
-dir24_8_lookup_bulk_uni(void *p, const uint32_t *ips,
+dir24_8_lookup_bulk_vrf_3(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
+{
+ struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
+
+ __dir24_8_lookup_bulk(dp, vrf_ids, ips, next_hops, n, 3, true);
+}
+
+static inline void
+dir24_8_lookup_bulk_uni(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
uint64_t *next_hops, const unsigned int n)
{
struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
@@ -224,66 +294,83 @@ dir24_8_lookup_bulk_uni(void *p, const uint32_t *ips,
uint32_t prefetch_offset = RTE_MIN(15U, n);
uint8_t nh_sz = dp->nh_sz;
- for (i = 0; i < prefetch_offset; i++)
- rte_prefetch0(get_tbl24_p(dp, ips[i], nh_sz));
+ for (i = 0; i < prefetch_offset; i++) {
+ uint16_t vid = vrf_ids[i];
+ RTE_ASSERT(vid < dp->num_vrfs);
+ rte_prefetch0(get_tbl24_p(dp, vid, ips[i], nh_sz));
+ }
for (i = 0; i < (n - prefetch_offset); i++) {
- rte_prefetch0(get_tbl24_p(dp, ips[i + prefetch_offset],
- nh_sz));
- tmp = get_tbl24(dp, ips[i], nh_sz);
+ uint16_t vid = vrf_ids[i];
+ uint16_t vid_next = vrf_ids[i + prefetch_offset];
+ RTE_ASSERT(vid < dp->num_vrfs);
+ RTE_ASSERT(vid_next < dp->num_vrfs);
+ rte_prefetch0(get_tbl24_p(dp, vid_next,
+ ips[i + prefetch_offset], nh_sz));
+ tmp = get_tbl24(dp, vid, ips[i], nh_sz);
if (unlikely(is_entry_extended(tmp)))
tmp = get_tbl8(dp, tmp, ips[i], nh_sz);
next_hops[i] = tmp >> 1;
}
for (; i < n; i++) {
- tmp = get_tbl24(dp, ips[i], nh_sz);
+ uint16_t vid = vrf_ids[i];
+ RTE_ASSERT(vid < dp->num_vrfs);
+ tmp = get_tbl24(dp, vid, ips[i], nh_sz);
if (unlikely(is_entry_extended(tmp)))
tmp = get_tbl8(dp, tmp, ips[i], nh_sz);
next_hops[i] = tmp >> 1;
}
}
-
#define BSWAP_MAX_LENGTH 64
-typedef void (*dir24_8_lookup_bulk_be_cb)(void *p, const uint32_t *ips, uint64_t *next_hops,
- const unsigned int n);
+typedef void (*dir24_8_lookup_bulk_be_cb)(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
static inline void
-dir24_8_lookup_bulk_be(void *p, const uint32_t *ips, uint64_t *next_hops, const unsigned int n,
- dir24_8_lookup_bulk_be_cb cb)
+dir24_8_lookup_bulk_be(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
+ uint64_t *next_hops, const unsigned int n, dir24_8_lookup_bulk_be_cb cb)
{
uint32_t le_ips[BSWAP_MAX_LENGTH];
unsigned int i;
#if RTE_BYTE_ORDER == RTE_BIG_ENDIAN
- cb(p, ips, next_hops, n);
+ cb(p, vrf_ids, ips, next_hops, n);
#else
for (i = 0; i < n; i += BSWAP_MAX_LENGTH) {
int j;
for (j = 0; j < BSWAP_MAX_LENGTH && i + j < n; j++)
le_ips[j] = rte_be_to_cpu_32(ips[i + j]);
- cb(p, le_ips, next_hops + i, j);
+ cb(p, vrf_ids + i, le_ips, next_hops + i, j);
}
#endif
}
#define DECLARE_BE_LOOKUP_FN(name) \
static inline void \
-name##_be(void *p, const uint32_t *ips, uint64_t *next_hops, const unsigned int n) \
+name##_be(void *p, const uint16_t *vrf_ids, const uint32_t *ips, \
+ uint64_t *next_hops, const unsigned int n) \
{ \
- dir24_8_lookup_bulk_be(p, ips, next_hops, n, name); \
+ dir24_8_lookup_bulk_be(p, vrf_ids, ips, next_hops, n, name); \
}
DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_1b)
DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_2b)
DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_4b)
DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_8b)
+DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_vrf_1b)
+DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_vrf_2b)
+DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_vrf_4b)
+DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_vrf_8b)
DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_0)
DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_1)
DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_2)
DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_3)
+DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_vrf_0)
+DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_vrf_1)
+DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_vrf_2)
+DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_vrf_3)
DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_uni)
void *
@@ -296,7 +383,7 @@ rte_fib_lookup_fn_t
dir24_8_get_lookup_fn(void *p, enum rte_fib_lookup_type type, bool be_addr);
int
-dir24_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t depth,
+dir24_8_modify(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip, uint8_t depth,
uint64_t next_hop, int op);
int
diff --git a/lib/fib/dir24_8_avx512.c b/lib/fib/dir24_8_avx512.c
index 89b43583c7..3e576e410e 100644
--- a/lib/fib/dir24_8_avx512.c
+++ b/lib/fib/dir24_8_avx512.c
@@ -4,75 +4,132 @@
#include <rte_vect.h>
#include <rte_fib.h>
+#include <rte_debug.h>
#include "dir24_8.h"
#include "dir24_8_avx512.h"
+enum vrf_scale {
+ VRF_SCALE_SINGLE = 0,
+ VRF_SCALE_SMALL = 1,
+ VRF_SCALE_LARGE = 2,
+};
+
static __rte_always_inline void
-dir24_8_vec_lookup_x16(void *p, const uint32_t *ips,
- uint64_t *next_hops, int size, bool be_addr)
+dir24_8_vec_lookup_x8_64b_path(struct dir24_8_tbl *dp, __m256i ip_vec_256,
+ __m256i vrf32_256, uint64_t *next_hops, int size)
{
- struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
- __mmask16 msk_ext;
- __mmask16 exp_msk = 0x5555;
- __m512i ip_vec, idxes, res, bytes;
- const __m512i zero = _mm512_set1_epi32(0);
- const __m512i lsb = _mm512_set1_epi32(1);
- const __m512i lsbyte_msk = _mm512_set1_epi32(0xff);
- __m512i tmp1, tmp2, res_msk;
- __m256i tmp256;
- /* used to mask gather values if size is 1/2 (8/16 bit next hops) */
+ const __m512i zero_64 = _mm512_set1_epi64(0);
+ const __m512i lsb_64 = _mm512_set1_epi64(1);
+ const __m512i lsbyte_msk_64 = _mm512_set1_epi64(0xff);
+ __m512i res_msk_64, vrf64, idxes_64, res, bytes_64;
+ __mmask8 msk_ext_64;
+
if (size == sizeof(uint8_t))
- res_msk = _mm512_set1_epi32(UINT8_MAX);
+ res_msk_64 = _mm512_set1_epi64(UINT8_MAX);
else if (size == sizeof(uint16_t))
- res_msk = _mm512_set1_epi32(UINT16_MAX);
+ res_msk_64 = _mm512_set1_epi64(UINT16_MAX);
+ else if (size == sizeof(uint32_t))
+ res_msk_64 = _mm512_set1_epi64(UINT32_MAX);
- ip_vec = _mm512_loadu_si512(ips);
- if (be_addr) {
- const __m512i bswap32 = _mm512_set_epi32(
- 0x0c0d0e0f, 0x08090a0b, 0x04050607, 0x00010203,
- 0x0c0d0e0f, 0x08090a0b, 0x04050607, 0x00010203,
- 0x0c0d0e0f, 0x08090a0b, 0x04050607, 0x00010203,
- 0x0c0d0e0f, 0x08090a0b, 0x04050607, 0x00010203
- );
- ip_vec = _mm512_shuffle_epi8(ip_vec, bswap32);
+ vrf64 = _mm512_cvtepu32_epi64(vrf32_256);
+
+ /* Compute index: (vrf_id << 24) + (ip >> 8) using 64-bit shift */
+ idxes_64 = _mm512_slli_epi64(vrf64, 24);
+ idxes_64 = _mm512_add_epi64(idxes_64, _mm512_cvtepu32_epi64(
+ _mm256_srli_epi32(ip_vec_256, 8)));
+
+ /* lookup in tbl24 */
+ if (size == sizeof(uint8_t)) {
+ res = _mm512_i64gather_epi64(idxes_64, (const void *)dp->tbl24, 1);
+ res = _mm512_and_epi64(res, res_msk_64);
+ } else if (size == sizeof(uint16_t)) {
+ res = _mm512_i64gather_epi64(idxes_64, (const void *)dp->tbl24, 2);
+ res = _mm512_and_epi64(res, res_msk_64);
+ } else {
+ res = _mm512_i64gather_epi64(idxes_64, (const void *)dp->tbl24, 4);
+ res = _mm512_and_epi64(res, res_msk_64);
+ }
+
+ /* get extended entries indexes */
+ msk_ext_64 = _mm512_test_epi64_mask(res, lsb_64);
+
+ if (msk_ext_64 != 0) {
+ bytes_64 = _mm512_cvtepu32_epi64(ip_vec_256);
+ idxes_64 = _mm512_srli_epi64(res, 1);
+ idxes_64 = _mm512_slli_epi64(idxes_64, 8);
+ bytes_64 = _mm512_and_epi64(bytes_64, lsbyte_msk_64);
+ idxes_64 = _mm512_maskz_add_epi64(msk_ext_64, idxes_64, bytes_64);
+
+ if (size == sizeof(uint8_t))
+ idxes_64 = _mm512_mask_i64gather_epi64(zero_64, msk_ext_64,
+ idxes_64, (const void *)dp->tbl8, 1);
+ else if (size == sizeof(uint16_t))
+ idxes_64 = _mm512_mask_i64gather_epi64(zero_64, msk_ext_64,
+ idxes_64, (const void *)dp->tbl8, 2);
+ else
+ idxes_64 = _mm512_mask_i64gather_epi64(zero_64, msk_ext_64,
+ idxes_64, (const void *)dp->tbl8, 4);
+
+ res = _mm512_mask_blend_epi64(msk_ext_64, res, idxes_64);
}
- /* mask 24 most significant bits */
- idxes = _mm512_srli_epi32(ip_vec, 8);
+ res = _mm512_srli_epi64(res, 1);
+ _mm512_storeu_si512(next_hops, res);
+}
+
+static __rte_always_inline void
+dir24_8_vec_lookup_x16_32b_path(struct dir24_8_tbl *dp, __m512i ip_vec,
+ __m512i idxes, uint64_t *next_hops, int size)
+{
+ __mmask16 msk_ext;
+ const __mmask16 exp_msk = 0x5555;
+ const __m512i zero_32 = _mm512_set1_epi32(0);
+ const __m512i lsb_32 = _mm512_set1_epi32(1);
+ const __m512i lsbyte_msk_32 = _mm512_set1_epi32(0xff);
+ __m512i res, bytes, tmp1, tmp2;
+ __m256i tmp256;
+ __m512i res_msk_32;
+
+ if (size == sizeof(uint8_t))
+ res_msk_32 = _mm512_set1_epi32(UINT8_MAX);
+ else if (size == sizeof(uint16_t))
+ res_msk_32 = _mm512_set1_epi32(UINT16_MAX);
- /**
+ /*
* lookup in tbl24
* Put it inside branch to make compiler happy with -O0
*/
if (size == sizeof(uint8_t)) {
res = _mm512_i32gather_epi32(idxes, (const int *)dp->tbl24, 1);
- res = _mm512_and_epi32(res, res_msk);
+ res = _mm512_and_epi32(res, res_msk_32);
} else if (size == sizeof(uint16_t)) {
res = _mm512_i32gather_epi32(idxes, (const int *)dp->tbl24, 2);
- res = _mm512_and_epi32(res, res_msk);
- } else
+ res = _mm512_and_epi32(res, res_msk_32);
+ } else {
res = _mm512_i32gather_epi32(idxes, (const int *)dp->tbl24, 4);
+ }
/* get extended entries indexes */
- msk_ext = _mm512_test_epi32_mask(res, lsb);
+ msk_ext = _mm512_test_epi32_mask(res, lsb_32);
if (msk_ext != 0) {
idxes = _mm512_srli_epi32(res, 1);
idxes = _mm512_slli_epi32(idxes, 8);
- bytes = _mm512_and_epi32(ip_vec, lsbyte_msk);
+ bytes = _mm512_and_epi32(ip_vec, lsbyte_msk_32);
idxes = _mm512_maskz_add_epi32(msk_ext, idxes, bytes);
if (size == sizeof(uint8_t)) {
- idxes = _mm512_mask_i32gather_epi32(zero, msk_ext,
+ idxes = _mm512_mask_i32gather_epi32(zero_32, msk_ext,
idxes, (const int *)dp->tbl8, 1);
- idxes = _mm512_and_epi32(idxes, res_msk);
+ idxes = _mm512_and_epi32(idxes, res_msk_32);
} else if (size == sizeof(uint16_t)) {
- idxes = _mm512_mask_i32gather_epi32(zero, msk_ext,
+ idxes = _mm512_mask_i32gather_epi32(zero_32, msk_ext,
idxes, (const int *)dp->tbl8, 2);
- idxes = _mm512_and_epi32(idxes, res_msk);
- } else
- idxes = _mm512_mask_i32gather_epi32(zero, msk_ext,
+ idxes = _mm512_and_epi32(idxes, res_msk_32);
+ } else {
+ idxes = _mm512_mask_i32gather_epi32(zero_32, msk_ext,
idxes, (const int *)dp->tbl8, 4);
+ }
res = _mm512_mask_blend_epi32(msk_ext, res, idxes);
}
@@ -86,16 +143,74 @@ dir24_8_vec_lookup_x16(void *p, const uint32_t *ips,
_mm512_storeu_si512(next_hops + 8, tmp2);
}
+/* Unified function with vrf_scale parameter similar to be_addr */
+static __rte_always_inline void
+dir24_8_vec_lookup_x16(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
+ uint64_t *next_hops, int size, bool be_addr, enum vrf_scale vrf_scale)
+{
+ struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
+ __m512i ip_vec, idxes;
+ __m256i ip_vec_256, vrf32_256;
+
+ ip_vec = _mm512_loadu_si512(ips);
+ if (be_addr) {
+ const __m512i bswap32 = _mm512_set_epi32(
+ 0x0c0d0e0f, 0x08090a0b, 0x04050607, 0x00010203,
+ 0x0c0d0e0f, 0x08090a0b, 0x04050607, 0x00010203,
+ 0x0c0d0e0f, 0x08090a0b, 0x04050607, 0x00010203,
+ 0x0c0d0e0f, 0x08090a0b, 0x04050607, 0x00010203
+ );
+ ip_vec = _mm512_shuffle_epi8(ip_vec, bswap32);
+ }
+
+ if (vrf_scale == VRF_SCALE_SINGLE) {
+ /* mask 24 most significant bits */
+ idxes = _mm512_srli_epi32(ip_vec, 8);
+ dir24_8_vec_lookup_x16_32b_path(dp, ip_vec, idxes, next_hops, size);
+ } else if (vrf_scale == VRF_SCALE_SMALL) {
+ /* For < 256 VRFs: use 32-bit indices with 32-bit shift */
+ __m512i vrf32;
+ uint32_t i;
+
+ for (i = 0; i < 16; i++)
+ RTE_ASSERT(vrf_ids[i] < dp->num_vrfs);
+
+ vrf32 = _mm512_cvtepu16_epi32(_mm256_loadu_si256((const void *)vrf_ids));
+
+ /* mask 24 most significant bits */
+ idxes = _mm512_srli_epi32(ip_vec, 8);
+ idxes = _mm512_add_epi32(idxes, _mm512_slli_epi32(vrf32, 24));
+ dir24_8_vec_lookup_x16_32b_path(dp, ip_vec, idxes, next_hops, size);
+ } else {
+ /* For >= 256 VRFs: use 64-bit indices to avoid overflow */
+ uint32_t i;
+
+ for (i = 0; i < 16; i++)
+ RTE_ASSERT(vrf_ids[i] < dp->num_vrfs);
+
+ /* Extract first 8 IPs and VRF IDs */
+ ip_vec_256 = _mm512_castsi512_si256(ip_vec);
+ vrf32_256 = _mm256_cvtepu16_epi32(_mm_loadu_si128((const void *)vrf_ids));
+ dir24_8_vec_lookup_x8_64b_path(dp, ip_vec_256, vrf32_256, next_hops, size);
+
+ /* Process next 8 IPs from the second half of the vector */
+ ip_vec_256 = _mm512_extracti32x8_epi32(ip_vec, 1);
+ vrf32_256 = _mm256_cvtepu16_epi32(_mm_loadu_si128((const void *)(vrf_ids + 8)));
+ dir24_8_vec_lookup_x8_64b_path(dp, ip_vec_256, vrf32_256, next_hops + 8, size);
+ }
+}
+
+/* Unified function with vrf_scale parameter */
static __rte_always_inline void
-dir24_8_vec_lookup_x8_8b(void *p, const uint32_t *ips,
- uint64_t *next_hops, bool be_addr)
+dir24_8_vec_lookup_x8_8b(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, bool be_addr, enum vrf_scale vrf_scale)
{
struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
- const __m512i zero = _mm512_set1_epi32(0);
- const __m512i lsbyte_msk = _mm512_set1_epi64(0xff);
- const __m512i lsb = _mm512_set1_epi64(1);
+ const __m512i zero_64 = _mm512_set1_epi64(0);
+ const __m512i lsbyte_msk_64 = _mm512_set1_epi64(0xff);
+ const __m512i lsb_64 = _mm512_set1_epi64(1);
__m512i res, idxes, bytes;
- __m256i idxes_256, ip_vec;
+ __m256i ip_vec, vrf32_256;
__mmask8 msk_ext;
ip_vec = _mm256_loadu_si256((const void *)ips);
@@ -106,66 +221,207 @@ dir24_8_vec_lookup_x8_8b(void *p, const uint32_t *ips,
);
ip_vec = _mm256_shuffle_epi8(ip_vec, bswap32);
}
- /* mask 24 most significant bits */
- idxes_256 = _mm256_srli_epi32(ip_vec, 8);
- /* lookup in tbl24 */
- res = _mm512_i32gather_epi64(idxes_256, (const void *)dp->tbl24, 8);
+ if (vrf_scale == VRF_SCALE_SINGLE) {
+ /* For single VRF: use 32-bit indices without vrf_ids */
+ __m256i idxes_256;
- /* get extended entries indexes */
- msk_ext = _mm512_test_epi64_mask(res, lsb);
+ /* mask 24 most significant bits */
+ idxes_256 = _mm256_srli_epi32(ip_vec, 8);
- if (msk_ext != 0) {
- bytes = _mm512_cvtepi32_epi64(ip_vec);
- idxes = _mm512_srli_epi64(res, 1);
- idxes = _mm512_slli_epi64(idxes, 8);
- bytes = _mm512_and_epi64(bytes, lsbyte_msk);
- idxes = _mm512_maskz_add_epi64(msk_ext, idxes, bytes);
- idxes = _mm512_mask_i64gather_epi64(zero, msk_ext, idxes,
- (const void *)dp->tbl8, 8);
-
- res = _mm512_mask_blend_epi64(msk_ext, res, idxes);
- }
+ /* lookup in tbl24 */
+ res = _mm512_i32gather_epi64(idxes_256, (const void *)dp->tbl24, 8);
- res = _mm512_srli_epi64(res, 1);
- _mm512_storeu_si512(next_hops, res);
+ /* get extended entries indexes */
+ msk_ext = _mm512_test_epi64_mask(res, lsb_64);
+
+ if (msk_ext != 0) {
+ bytes = _mm512_cvtepu32_epi64(ip_vec);
+ idxes = _mm512_srli_epi64(res, 1);
+ idxes = _mm512_slli_epi64(idxes, 8);
+ bytes = _mm512_and_epi64(bytes, lsbyte_msk_64);
+ idxes = _mm512_maskz_add_epi64(msk_ext, idxes, bytes);
+ idxes = _mm512_mask_i64gather_epi64(zero_64, msk_ext, idxes,
+ (const void *)dp->tbl8, 8);
+
+ res = _mm512_mask_blend_epi64(msk_ext, res, idxes);
+ }
+
+ res = _mm512_srli_epi64(res, 1);
+ _mm512_storeu_si512(next_hops, res);
+ } else if (vrf_scale == VRF_SCALE_SMALL) {
+ /* For < 256 VRFs: use 32-bit indices with 32-bit shift */
+ __m256i idxes_256;
+ uint32_t i;
+
+ for (i = 0; i < 8; i++)
+ RTE_ASSERT(vrf_ids[i] < dp->num_vrfs);
+
+ /* mask 24 most significant bits */
+ idxes_256 = _mm256_srli_epi32(ip_vec, 8);
+ vrf32_256 = _mm256_cvtepu16_epi32(_mm_loadu_si128((const void *)vrf_ids));
+ idxes_256 = _mm256_add_epi32(idxes_256, _mm256_slli_epi32(vrf32_256, 24));
+
+ /* lookup in tbl24 */
+ res = _mm512_i32gather_epi64(idxes_256, (const void *)dp->tbl24, 8);
+
+ /* get extended entries indexes */
+ msk_ext = _mm512_test_epi64_mask(res, lsb_64);
+
+ if (msk_ext != 0) {
+ bytes = _mm512_cvtepu32_epi64(ip_vec);
+ idxes = _mm512_srli_epi64(res, 1);
+ idxes = _mm512_slli_epi64(idxes, 8);
+ bytes = _mm512_and_epi64(bytes, lsbyte_msk_64);
+ idxes = _mm512_maskz_add_epi64(msk_ext, idxes, bytes);
+ idxes = _mm512_mask_i64gather_epi64(zero_64, msk_ext, idxes,
+ (const void *)dp->tbl8, 8);
+
+ res = _mm512_mask_blend_epi64(msk_ext, res, idxes);
+ }
+
+ res = _mm512_srli_epi64(res, 1);
+ _mm512_storeu_si512(next_hops, res);
+ } else {
+ /* For >= 256 VRFs: use 64-bit indices to avoid overflow */
+ uint32_t i;
+
+ for (i = 0; i < 8; i++)
+ RTE_ASSERT(vrf_ids[i] < dp->num_vrfs);
+
+ vrf32_256 = _mm256_cvtepu16_epi32(_mm_loadu_si128((const void *)vrf_ids));
+ __m512i vrf64 = _mm512_cvtepu32_epi64(vrf32_256);
+
+ /* Compute index: (vrf_id << 24) + (ip >> 8) using 64-bit arithmetic */
+ idxes = _mm512_slli_epi64(vrf64, 24);
+ idxes = _mm512_add_epi64(idxes, _mm512_cvtepu32_epi64(
+ _mm256_srli_epi32(ip_vec, 8)));
+
+ /* lookup in tbl24 with 64-bit gather */
+ res = _mm512_i64gather_epi64(idxes, (const void *)dp->tbl24, 8);
+
+ /* get extended entries indexes */
+ msk_ext = _mm512_test_epi64_mask(res, lsb_64);
+
+ if (msk_ext != 0) {
+ bytes = _mm512_cvtepu32_epi64(ip_vec);
+ idxes = _mm512_srli_epi64(res, 1);
+ idxes = _mm512_slli_epi64(idxes, 8);
+ bytes = _mm512_and_epi64(bytes, lsbyte_msk_64);
+ idxes = _mm512_maskz_add_epi64(msk_ext, idxes, bytes);
+ idxes = _mm512_mask_i64gather_epi64(zero_64, msk_ext, idxes,
+ (const void *)dp->tbl8, 8);
+
+ res = _mm512_mask_blend_epi64(msk_ext, res, idxes);
+ }
+
+ res = _mm512_srli_epi64(res, 1);
+ _mm512_storeu_si512(next_hops, res);
+ }
}
-#define DECLARE_VECTOR_FN(suffix, nh_type, be_addr) \
+#define DECLARE_VECTOR_FN(suffix, scalar_suffix, nh_type, be_addr, vrf_scale) \
void \
-rte_dir24_8_vec_lookup_bulk_##suffix(void *p, const uint32_t *ips, uint64_t *next_hops, \
- const unsigned int n) \
+rte_dir24_8_vec_lookup_bulk_##suffix(void *p, const uint16_t *vrf_ids, \
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n) \
{ \
uint32_t i; \
for (i = 0; i < (n / 16); i++) \
- dir24_8_vec_lookup_x16(p, ips + i * 16, next_hops + i * 16, sizeof(nh_type), \
- be_addr); \
- dir24_8_lookup_bulk_##suffix(p, ips + i * 16, next_hops + i * 16, n - i * 16); \
+ dir24_8_vec_lookup_x16(p, vrf_ids + i * 16, ips + i * 16, \
+ next_hops + i * 16, sizeof(nh_type), be_addr, vrf_scale); \
+ dir24_8_lookup_bulk_##scalar_suffix(p, vrf_ids + i * 16, ips + i * 16, \
+ next_hops + i * 16, n - i * 16); \
+}
+
+DECLARE_VECTOR_FN(1b, 1b, uint8_t, false, VRF_SCALE_SINGLE)
+DECLARE_VECTOR_FN(1b_be, 1b_be, uint8_t, true, VRF_SCALE_SINGLE)
+DECLARE_VECTOR_FN(2b, 2b, uint16_t, false, VRF_SCALE_SINGLE)
+DECLARE_VECTOR_FN(2b_be, 2b_be, uint16_t, true, VRF_SCALE_SINGLE)
+DECLARE_VECTOR_FN(4b, 4b, uint32_t, false, VRF_SCALE_SINGLE)
+DECLARE_VECTOR_FN(4b_be, 4b_be, uint32_t, true, VRF_SCALE_SINGLE)
+
+DECLARE_VECTOR_FN(vrf_1b, vrf_1b, uint8_t, false, VRF_SCALE_SMALL)
+DECLARE_VECTOR_FN(vrf_1b_be, vrf_1b_be, uint8_t, true, VRF_SCALE_SMALL)
+DECLARE_VECTOR_FN(vrf_2b, vrf_2b, uint16_t, false, VRF_SCALE_SMALL)
+DECLARE_VECTOR_FN(vrf_2b_be, vrf_2b_be, uint16_t, true, VRF_SCALE_SMALL)
+DECLARE_VECTOR_FN(vrf_4b, vrf_4b, uint32_t, false, VRF_SCALE_SMALL)
+DECLARE_VECTOR_FN(vrf_4b_be, vrf_4b_be, uint32_t, true, VRF_SCALE_SMALL)
+
+DECLARE_VECTOR_FN(vrf_1b_large, vrf_1b, uint8_t, false, VRF_SCALE_LARGE)
+DECLARE_VECTOR_FN(vrf_1b_be_large, vrf_1b_be, uint8_t, true, VRF_SCALE_LARGE)
+DECLARE_VECTOR_FN(vrf_2b_large, vrf_2b, uint16_t, false, VRF_SCALE_LARGE)
+DECLARE_VECTOR_FN(vrf_2b_be_large, vrf_2b_be, uint16_t, true, VRF_SCALE_LARGE)
+DECLARE_VECTOR_FN(vrf_4b_large, vrf_4b, uint32_t, false, VRF_SCALE_LARGE)
+DECLARE_VECTOR_FN(vrf_4b_be_large, vrf_4b_be, uint32_t, true, VRF_SCALE_LARGE)
+
+void
+rte_dir24_8_vec_lookup_bulk_8b(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
+{
+ uint32_t i;
+ for (i = 0; i < (n / 8); i++)
+ dir24_8_vec_lookup_x8_8b(p, vrf_ids + i * 8, ips + i * 8,
+ next_hops + i * 8, false, VRF_SCALE_SINGLE);
+ dir24_8_lookup_bulk_8b(p, vrf_ids + i * 8, ips + i * 8,
+ next_hops + i * 8, n - i * 8);
+}
+
+void
+rte_dir24_8_vec_lookup_bulk_8b_be(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
+{
+ uint32_t i;
+ for (i = 0; i < (n / 8); i++)
+ dir24_8_vec_lookup_x8_8b(p, vrf_ids + i * 8, ips + i * 8,
+ next_hops + i * 8, true, VRF_SCALE_SINGLE);
+ dir24_8_lookup_bulk_8b_be(p, vrf_ids + i * 8, ips + i * 8,
+ next_hops + i * 8, n - i * 8);
+}
+
+void
+rte_dir24_8_vec_lookup_bulk_vrf_8b(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
+{
+ uint32_t i;
+ for (i = 0; i < (n / 8); i++)
+ dir24_8_vec_lookup_x8_8b(p, vrf_ids + i * 8, ips + i * 8,
+ next_hops + i * 8, false, VRF_SCALE_SMALL);
+ dir24_8_lookup_bulk_vrf_8b(p, vrf_ids + i * 8, ips + i * 8,
+ next_hops + i * 8, n - i * 8);
}
-DECLARE_VECTOR_FN(1b, uint8_t, false)
-DECLARE_VECTOR_FN(1b_be, uint8_t, true)
-DECLARE_VECTOR_FN(2b, uint16_t, false)
-DECLARE_VECTOR_FN(2b_be, uint16_t, true)
-DECLARE_VECTOR_FN(4b, uint32_t, false)
-DECLARE_VECTOR_FN(4b_be, uint32_t, true)
+void
+rte_dir24_8_vec_lookup_bulk_vrf_8b_be(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
+{
+ uint32_t i;
+ for (i = 0; i < (n / 8); i++)
+ dir24_8_vec_lookup_x8_8b(p, vrf_ids + i * 8, ips + i * 8,
+ next_hops + i * 8, true, VRF_SCALE_SMALL);
+ dir24_8_lookup_bulk_vrf_8b_be(p, vrf_ids + i * 8, ips + i * 8,
+ next_hops + i * 8, n - i * 8);
+}
void
-rte_dir24_8_vec_lookup_bulk_8b(void *p, const uint32_t *ips,
- uint64_t *next_hops, const unsigned int n)
+rte_dir24_8_vec_lookup_bulk_vrf_8b_large(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
{
uint32_t i;
for (i = 0; i < (n / 8); i++)
- dir24_8_vec_lookup_x8_8b(p, ips + i * 8, next_hops + i * 8, false);
- dir24_8_lookup_bulk_8b(p, ips + i * 8, next_hops + i * 8, n - i * 8);
+ dir24_8_vec_lookup_x8_8b(p, vrf_ids + i * 8, ips + i * 8,
+ next_hops + i * 8, false, VRF_SCALE_LARGE);
+ dir24_8_lookup_bulk_vrf_8b(p, vrf_ids + i * 8, ips + i * 8,
+ next_hops + i * 8, n - i * 8);
}
void
-rte_dir24_8_vec_lookup_bulk_8b_be(void *p, const uint32_t *ips,
- uint64_t *next_hops, const unsigned int n)
+rte_dir24_8_vec_lookup_bulk_vrf_8b_be_large(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
{
uint32_t i;
for (i = 0; i < (n / 8); i++)
- dir24_8_vec_lookup_x8_8b(p, ips + i * 8, next_hops + i * 8, true);
- dir24_8_lookup_bulk_8b_be(p, ips + i * 8, next_hops + i * 8, n - i * 8);
+ dir24_8_vec_lookup_x8_8b(p, vrf_ids + i * 8, ips + i * 8,
+ next_hops + i * 8, true, VRF_SCALE_LARGE);
+ dir24_8_lookup_bulk_vrf_8b_be(p, vrf_ids + i * 8, ips + i * 8,
+ next_hops + i * 8, n - i * 8);
}
diff --git a/lib/fib/dir24_8_avx512.h b/lib/fib/dir24_8_avx512.h
index 3e2bbc2490..d42ef1d17f 100644
--- a/lib/fib/dir24_8_avx512.h
+++ b/lib/fib/dir24_8_avx512.h
@@ -6,35 +6,99 @@
#define _DIR248_AVX512_H_
void
-rte_dir24_8_vec_lookup_bulk_1b(void *p, const uint32_t *ips,
+rte_dir24_8_vec_lookup_bulk_1b(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
+
+void
+rte_dir24_8_vec_lookup_bulk_1b_be(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
+
+void
+rte_dir24_8_vec_lookup_bulk_vrf_1b(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
uint64_t *next_hops, const unsigned int n);
void
-rte_dir24_8_vec_lookup_bulk_2b(void *p, const uint32_t *ips,
+rte_dir24_8_vec_lookup_bulk_vrf_1b_be(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
uint64_t *next_hops, const unsigned int n);
void
-rte_dir24_8_vec_lookup_bulk_4b(void *p, const uint32_t *ips,
+rte_dir24_8_vec_lookup_bulk_vrf_1b_large(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
+
+void
+rte_dir24_8_vec_lookup_bulk_vrf_1b_be_large(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
+
+void
+rte_dir24_8_vec_lookup_bulk_2b(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
+
+void
+rte_dir24_8_vec_lookup_bulk_2b_be(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
+
+void
+rte_dir24_8_vec_lookup_bulk_vrf_2b(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
uint64_t *next_hops, const unsigned int n);
void
-rte_dir24_8_vec_lookup_bulk_8b(void *p, const uint32_t *ips,
+rte_dir24_8_vec_lookup_bulk_vrf_2b_be(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
uint64_t *next_hops, const unsigned int n);
void
-rte_dir24_8_vec_lookup_bulk_1b_be(void *p, const uint32_t *ips,
+rte_dir24_8_vec_lookup_bulk_vrf_2b_large(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
+
+void
+rte_dir24_8_vec_lookup_bulk_vrf_2b_be_large(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
+
+void
+rte_dir24_8_vec_lookup_bulk_4b(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
+
+void
+rte_dir24_8_vec_lookup_bulk_4b_be(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
+
+void
+rte_dir24_8_vec_lookup_bulk_vrf_4b(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
uint64_t *next_hops, const unsigned int n);
void
-rte_dir24_8_vec_lookup_bulk_2b_be(void *p, const uint32_t *ips,
+rte_dir24_8_vec_lookup_bulk_vrf_4b_be(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
uint64_t *next_hops, const unsigned int n);
void
-rte_dir24_8_vec_lookup_bulk_4b_be(void *p, const uint32_t *ips,
+rte_dir24_8_vec_lookup_bulk_vrf_4b_large(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
+
+void
+rte_dir24_8_vec_lookup_bulk_vrf_4b_be_large(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
+
+void
+rte_dir24_8_vec_lookup_bulk_8b(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
+
+void
+rte_dir24_8_vec_lookup_bulk_8b_be(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
+
+void
+rte_dir24_8_vec_lookup_bulk_vrf_8b(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
uint64_t *next_hops, const unsigned int n);
void
-rte_dir24_8_vec_lookup_bulk_8b_be(void *p, const uint32_t *ips,
+rte_dir24_8_vec_lookup_bulk_vrf_8b_be(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
uint64_t *next_hops, const unsigned int n);
+void
+rte_dir24_8_vec_lookup_bulk_vrf_8b_large(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
+
+void
+rte_dir24_8_vec_lookup_bulk_vrf_8b_be_large(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
+
#endif /* _DIR248_AVX512_H_ */
diff --git a/lib/fib/rte_fib.c b/lib/fib/rte_fib.c
index 184210f380..efc0595a7f 100644
--- a/lib/fib/rte_fib.c
+++ b/lib/fib/rte_fib.c
@@ -14,12 +14,15 @@
#include <rte_string_fns.h>
#include <rte_tailq.h>
+#include <rte_debug.h>
#include <rte_rib.h>
#include <rte_fib.h>
#include "dir24_8.h"
#include "fib_log.h"
+#define FIB_MAX_LOOKUP_BULK 64U
+
RTE_LOG_REGISTER_DEFAULT(fib_logtype, INFO);
TAILQ_HEAD(rte_fib_list, rte_tailq_entry);
@@ -40,52 +43,61 @@ EAL_REGISTER_TAILQ(rte_fib_tailq)
struct rte_fib {
char name[RTE_FIB_NAMESIZE];
enum rte_fib_type type; /**< Type of FIB struct */
- unsigned int flags; /**< Flags */
- struct rte_rib *rib; /**< RIB helper datastructure */
+ uint16_t flags; /**< Flags */
+ uint16_t num_vrfs;/**< Number of VRFs */
+ struct rte_rib **ribs; /**< RIB helper datastructures per VRF */
void *dp; /**< pointer to the dataplane struct*/
rte_fib_lookup_fn_t lookup; /**< FIB lookup function */
rte_fib_modify_fn_t modify; /**< modify FIB datastructure */
- uint64_t def_nh;
+ uint64_t *def_nh;/**< Per-VRF default next hop array */
};
static void
-dummy_lookup(void *fib_p, const uint32_t *ips, uint64_t *next_hops,
- const unsigned int n)
+dummy_lookup(void *fib_p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
{
unsigned int i;
struct rte_fib *fib = fib_p;
struct rte_rib_node *node;
+ struct rte_rib *rib;
for (i = 0; i < n; i++) {
- node = rte_rib_lookup(fib->rib, ips[i]);
+ RTE_ASSERT(vrf_ids[i] < fib->num_vrfs);
+ rib = rte_fib_vrf_get_rib(fib, vrf_ids[i]);
+ node = rte_rib_lookup(rib, ips[i]);
if (node != NULL)
rte_rib_get_nh(node, &next_hops[i]);
else
- next_hops[i] = fib->def_nh;
+ next_hops[i] = fib->def_nh[vrf_ids[i]];
}
}
static int
-dummy_modify(struct rte_fib *fib, uint32_t ip, uint8_t depth,
- uint64_t next_hop, int op)
+dummy_modify(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip,
+ uint8_t depth, uint64_t next_hop, int op)
{
struct rte_rib_node *node;
+ struct rte_rib *rib;
if ((fib == NULL) || (depth > RTE_FIB_MAXDEPTH))
return -EINVAL;
- node = rte_rib_lookup_exact(fib->rib, ip, depth);
+ rib = rte_fib_vrf_get_rib(fib, vrf_id);
+ if (rib == NULL)
+ return -EINVAL;
+
+ node = rte_rib_lookup_exact(rib, ip, depth);
switch (op) {
case RTE_FIB_ADD:
if (node == NULL)
- node = rte_rib_insert(fib->rib, ip, depth);
+ node = rte_rib_insert(rib, ip, depth);
if (node == NULL)
return -rte_errno;
return rte_rib_set_nh(node, next_hop);
case RTE_FIB_DEL:
if (node == NULL)
return -ENOENT;
- rte_rib_remove(fib->rib, ip, depth);
+ rte_rib_remove(rib, ip, depth);
return 0;
}
return -EINVAL;
@@ -125,7 +137,7 @@ rte_fib_add(struct rte_fib *fib, uint32_t ip, uint8_t depth, uint64_t next_hop)
if ((fib == NULL) || (fib->modify == NULL) ||
(depth > RTE_FIB_MAXDEPTH))
return -EINVAL;
- return fib->modify(fib, ip, depth, next_hop, RTE_FIB_ADD);
+ return fib->modify(fib, 0, ip, depth, next_hop, RTE_FIB_ADD);
}
RTE_EXPORT_SYMBOL(rte_fib_delete)
@@ -135,7 +147,7 @@ rte_fib_delete(struct rte_fib *fib, uint32_t ip, uint8_t depth)
if ((fib == NULL) || (fib->modify == NULL) ||
(depth > RTE_FIB_MAXDEPTH))
return -EINVAL;
- return fib->modify(fib, ip, depth, 0, RTE_FIB_DEL);
+ return fib->modify(fib, 0, ip, depth, 0, RTE_FIB_DEL);
}
RTE_EXPORT_SYMBOL(rte_fib_lookup_bulk)
@@ -143,24 +155,73 @@ int
rte_fib_lookup_bulk(struct rte_fib *fib, uint32_t *ips,
uint64_t *next_hops, int n)
{
+ static const uint16_t zero_vrf_ids[FIB_MAX_LOOKUP_BULK];
+ unsigned int off = 0;
+ unsigned int total = (unsigned int)n;
+
FIB_RETURN_IF_TRUE(((fib == NULL) || (ips == NULL) ||
(next_hops == NULL) || (fib->lookup == NULL)), -EINVAL);
- fib->lookup(fib->dp, ips, next_hops, n);
+ while (off < total) {
+ unsigned int chunk = RTE_MIN(total - off,
+ FIB_MAX_LOOKUP_BULK);
+ fib->lookup(fib->dp, zero_vrf_ids, ips + off,
+ next_hops + off, chunk);
+ off += chunk;
+ }
+
+ return 0;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fib_vrf_lookup_bulk, 26.07)
+int
+rte_fib_vrf_lookup_bulk(struct rte_fib *fib, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, int n)
+{
+ FIB_RETURN_IF_TRUE(((fib == NULL) || (vrf_ids == NULL) ||
+ (ips == NULL) || (next_hops == NULL) ||
+ (fib->lookup == NULL)), -EINVAL);
+
+ fib->lookup(fib->dp, vrf_ids, ips, next_hops, (unsigned int)n);
return 0;
}
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fib_vrf_add, 26.07)
+int
+rte_fib_vrf_add(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip,
+ uint8_t depth, uint64_t next_hop)
+{
+ if ((fib == NULL) || (fib->modify == NULL) ||
+ (depth > RTE_FIB_MAXDEPTH))
+ return -EINVAL;
+ return fib->modify(fib, vrf_id, ip, depth, next_hop, RTE_FIB_ADD);
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fib_vrf_delete, 26.07)
+int
+rte_fib_vrf_delete(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip,
+ uint8_t depth)
+{
+ if ((fib == NULL) || (fib->modify == NULL) ||
+ (depth > RTE_FIB_MAXDEPTH))
+ return -EINVAL;
+ return fib->modify(fib, vrf_id, ip, depth, 0, RTE_FIB_DEL);
+}
+
RTE_EXPORT_SYMBOL(rte_fib_create)
struct rte_fib *
rte_fib_create(const char *name, int socket_id, struct rte_fib_conf *conf)
{
char mem_name[RTE_FIB_NAMESIZE];
+ char rib_name[RTE_FIB_NAMESIZE];
int ret;
struct rte_fib *fib = NULL;
struct rte_rib *rib = NULL;
struct rte_tailq_entry *te;
struct rte_fib_list *fib_list;
struct rte_rib_conf rib_conf;
+ uint16_t num_vrfs;
+ uint16_t vrf;
/* Check user arguments. */
if ((name == NULL) || (conf == NULL) || (conf->max_routes < 0) ||
@@ -170,16 +231,42 @@ rte_fib_create(const char *name, int socket_id, struct rte_fib_conf *conf)
return NULL;
}
+ num_vrfs = (conf->max_vrfs == 0) ? 1 : conf->max_vrfs;
rib_conf.ext_sz = conf->rib_ext_sz;
rib_conf.max_nodes = conf->max_routes * 2;
- rib = rte_rib_create(name, socket_id, &rib_conf);
- if (rib == NULL) {
- FIB_LOG(ERR,
- "Can not allocate RIB %s", name);
+ struct rte_rib **ribs = rte_zmalloc_socket("FIB_RIBS",
+ num_vrfs * sizeof(*fib->ribs), RTE_CACHE_LINE_SIZE, socket_id);
+ if (ribs == NULL) {
+ FIB_LOG(ERR, "FIB %s RIB array allocation failed", name);
+ rte_errno = ENOMEM;
return NULL;
}
+ uint64_t *def_nh = rte_zmalloc_socket("FIB_DEF_NH",
+ num_vrfs * sizeof(*def_nh), RTE_CACHE_LINE_SIZE, socket_id);
+ if (def_nh == NULL) {
+ FIB_LOG(ERR, "FIB %s default nexthop array allocation failed", name);
+ rte_errno = ENOMEM;
+ rte_free(ribs);
+ return NULL;
+ }
+
+ for (vrf = 0; vrf < num_vrfs; vrf++) {
+ if (num_vrfs == 1)
+ snprintf(rib_name, sizeof(rib_name), "%s", name);
+ else
+ snprintf(rib_name, sizeof(rib_name), "%s_vrf%u", name, vrf);
+ rib = rte_rib_create(rib_name, socket_id, &rib_conf);
+ if (rib == NULL) {
+ FIB_LOG(ERR, "Can not allocate RIB %s", rib_name);
+ goto free_ribs;
+ }
+ ribs[vrf] = rib;
+ def_nh[vrf] = (conf->vrf_default_nh != NULL) ?
+ conf->vrf_default_nh[vrf] : conf->default_nh;
+ }
+
snprintf(mem_name, sizeof(mem_name), "FIB_%s", name);
fib_list = RTE_TAILQ_CAST(rte_fib_tailq.head, rte_fib_list);
@@ -215,11 +302,13 @@ rte_fib_create(const char *name, int socket_id, struct rte_fib_conf *conf)
goto free_te;
}
+ fib->num_vrfs = num_vrfs;
+ fib->ribs = ribs;
+ fib->def_nh = def_nh;
+
rte_strlcpy(fib->name, name, sizeof(fib->name));
- fib->rib = rib;
fib->type = conf->type;
fib->flags = conf->flags;
- fib->def_nh = conf->default_nh;
ret = init_dataplane(fib, socket_id, conf);
if (ret < 0) {
FIB_LOG(ERR,
@@ -242,8 +331,12 @@ rte_fib_create(const char *name, int socket_id, struct rte_fib_conf *conf)
rte_free(te);
exit:
rte_mcfg_tailq_write_unlock();
- rte_rib_free(rib);
+free_ribs:
+ for (vrf = 0; vrf < num_vrfs; vrf++)
+ rte_rib_free(ribs[vrf]);
+ rte_free(def_nh);
+ rte_free(ribs);
return NULL;
}
@@ -311,7 +404,13 @@ rte_fib_free(struct rte_fib *fib)
rte_mcfg_tailq_write_unlock();
free_dataplane(fib);
- rte_rib_free(fib->rib);
+ if (fib->ribs != NULL) {
+ uint16_t vrf;
+ for (vrf = 0; vrf < fib->num_vrfs; vrf++)
+ rte_rib_free(fib->ribs[vrf]);
+ }
+ rte_free(fib->ribs);
+ rte_free(fib->def_nh);
rte_free(fib);
rte_free(te);
}
@@ -327,7 +426,18 @@ RTE_EXPORT_SYMBOL(rte_fib_get_rib)
struct rte_rib *
rte_fib_get_rib(struct rte_fib *fib)
{
- return (fib == NULL) ? NULL : fib->rib;
+ return (fib == NULL || fib->ribs == NULL) ? NULL : fib->ribs[0];
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fib_vrf_get_rib, 26.07)
+struct rte_rib *
+rte_fib_vrf_get_rib(struct rte_fib *fib, uint16_t vrf_id)
+{
+ if (fib == NULL || fib->ribs == NULL)
+ return NULL;
+ if (vrf_id >= fib->num_vrfs)
+ return NULL;
+ return fib->ribs[vrf_id];
}
RTE_EXPORT_SYMBOL(rte_fib_select_lookup)
diff --git a/lib/fib/rte_fib.h b/lib/fib/rte_fib.h
index b16a653535..883195c7d6 100644
--- a/lib/fib/rte_fib.h
+++ b/lib/fib/rte_fib.h
@@ -53,11 +53,11 @@ enum rte_fib_type {
};
/** Modify FIB function */
-typedef int (*rte_fib_modify_fn_t)(struct rte_fib *fib, uint32_t ip,
- uint8_t depth, uint64_t next_hop, int op);
+typedef int (*rte_fib_modify_fn_t)(struct rte_fib *fib, uint16_t vrf_id,
+ uint32_t ip, uint8_t depth, uint64_t next_hop, int op);
/** FIB bulk lookup function */
-typedef void (*rte_fib_lookup_fn_t)(void *fib, const uint32_t *ips,
- uint64_t *next_hops, const unsigned int n);
+typedef void (*rte_fib_lookup_fn_t)(void *fib, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
enum rte_fib_op {
RTE_FIB_ADD,
@@ -110,6 +110,10 @@ struct rte_fib_conf {
} dir24_8;
};
unsigned int flags; /**< Optional feature flags from RTE_FIB_F_* */
+ /** Number of VRFs to support (0 or 1 = single VRF for backward compat) */
+ uint16_t max_vrfs;
+ /** Per-VRF default nexthops (NULL = use default_nh for all) */
+ uint64_t *vrf_default_nh;
};
/** FIB RCU QSBR configuration structure. */
@@ -224,6 +228,71 @@ rte_fib_delete(struct rte_fib *fib, uint32_t ip, uint8_t depth);
int
rte_fib_lookup_bulk(struct rte_fib *fib, uint32_t *ips,
uint64_t *next_hops, int n);
+
+/**
+ * Add a route to the FIB with VRF ID.
+ *
+ * @param fib
+ * FIB object handle
+ * @param vrf_id
+ * VRF ID (0 to max_vrfs-1)
+ * @param ip
+ * IPv4 prefix address to be added to the FIB
+ * @param depth
+ * Prefix length
+ * @param next_hop
+ * Next hop to be added to the FIB
+ * @return
+ * 0 on success, negative value otherwise
+ */
+__rte_experimental
+int
+rte_fib_vrf_add(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip,
+ uint8_t depth, uint64_t next_hop);
+
+/**
+ * Delete a rule from the FIB with VRF ID.
+ *
+ * @param fib
+ * FIB object handle
+ * @param vrf_id
+ * VRF ID (0 to max_vrfs-1)
+ * @param ip
+ * IPv4 prefix address to be deleted from the FIB
+ * @param depth
+ * Prefix length
+ * @return
+ * 0 on success, negative value otherwise
+ */
+__rte_experimental
+int
+rte_fib_vrf_delete(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip,
+ uint8_t depth);
+
+/**
+ * Lookup multiple IP addresses in the FIB with per-packet VRF IDs.
+ *
+ * @param fib
+ * FIB object handle
+ * @param vrf_ids
+ * Array of VRF IDs
+ * @param ips
+ * Array of IPs to be looked up in the FIB
+ * @param next_hops
+ * Next hop of the most specific rule found for IP in the corresponding VRF.
+ * This is an array of eight byte values.
+ * If the lookup for the given IP failed, then corresponding element would
+ * contain default nexthop value configured for that VRF.
+ * @param n
+ * Number of elements in vrf_ids, ips (and next_hops) arrays to lookup.
+ * @return
+ * -EINVAL for incorrect arguments, otherwise 0
+ */
+__rte_experimental
+int
+rte_fib_vrf_lookup_bulk(struct rte_fib *fib, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, int n);
+
/**
* Get pointer to the dataplane specific struct
*
@@ -237,7 +306,7 @@ void *
rte_fib_get_dp(struct rte_fib *fib);
/**
- * Get pointer to the RIB
+ * Get pointer to the RIB for VRF 0
*
* @param fib
* FIB object handle
@@ -248,6 +317,21 @@ rte_fib_get_dp(struct rte_fib *fib);
struct rte_rib *
rte_fib_get_rib(struct rte_fib *fib);
+/**
+ * Get pointer to the RIB for a specific VRF
+ *
+ * @param fib
+ * FIB object handle
+ * @param vrf_id
+ * VRF ID (0 to max_vrfs-1)
+ * @return
+ * Pointer on the RIB on success
+ * NULL otherwise
+ */
+__rte_experimental
+struct rte_rib *
+rte_fib_vrf_get_rib(struct rte_fib *fib, uint16_t vrf_id);
+
/**
* Set lookup function based on type
*
--
2.43.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [RFC PATCH 2/4] fib: add VRF functional and unit tests
2026-03-22 15:42 [RFC PATCH 0/4] VRF support in FIB library Vladimir Medvedkin
2026-03-22 15:42 ` [RFC PATCH 1/4] fib: add multi-VRF support Vladimir Medvedkin
@ 2026-03-22 15:42 ` Vladimir Medvedkin
2026-03-22 16:40 ` Stephen Hemminger
2026-03-22 16:41 ` Stephen Hemminger
2026-03-22 15:42 ` [RFC PATCH 3/4] fib6: add multi-VRF support Vladimir Medvedkin
` (6 subsequent siblings)
8 siblings, 2 replies; 45+ messages in thread
From: Vladimir Medvedkin @ 2026-03-22 15:42 UTC (permalink / raw)
To: dev; +Cc: rjarry, nsaxena16, mb, adwivedi, jerinjacobk
Add test coverage for the multi-VRF IPv4 FIB API.
The app/test-fib functional harness exercises VRF-aware routes
with nexthop encoding, bulk VRF lookup, and nexthop validation
across multiple VRFs (enabled with the -V option).
The app/test unit tests cover VRF table creation, per-VRF
route add and delete, lookup correctness, inter-VRF isolation,
and all supported nexthop sizes.
Signed-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
---
app/test-fib/main.c | 167 +++++++++++++++++++++++--
app/test/test_fib.c | 298 ++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 454 insertions(+), 11 deletions(-)
diff --git a/app/test-fib/main.c b/app/test-fib/main.c
index dd1a6d7297..5593fdd47e 100644
--- a/app/test-fib/main.c
+++ b/app/test-fib/main.c
@@ -82,6 +82,7 @@ static struct {
uint32_t nb_routes_per_depth[128 + 1];
uint32_t flags;
uint32_t tbl8;
+ uint32_t nb_vrfs;
uint8_t ent_sz;
uint8_t rnd_lookup_ips_ratio;
uint8_t print_fract;
@@ -95,6 +96,7 @@ static struct {
.nb_routes_per_depth = {0},
.flags = FIB_V4_DIR_TYPE,
.tbl8 = DEFAULT_LPM_TBL8,
+ .nb_vrfs = 1,
.ent_sz = 4,
.rnd_lookup_ips_ratio = 0,
.print_fract = 10,
@@ -136,6 +138,45 @@ get_max_nh(uint8_t nh_sz)
(1ULL << 21) - 1);
}
+/* Get number of bits needed to encode VRF IDs */
+static __rte_always_inline __rte_pure uint8_t
+get_vrf_bits(uint32_t nb_vrfs)
+{
+ uint8_t bits = 0;
+ uint32_t vrfs = nb_vrfs > 1 ? nb_vrfs : 2; /* At least 1 bit */
+
+ /* Round up to next power of 2 */
+ vrfs--;
+ while (vrfs > 0) {
+ bits++;
+ vrfs >>= 1;
+ }
+ return bits;
+}
+
+/* Encode VRF ID into nexthop (high bits) */
+static __rte_always_inline uint64_t
+encode_vrf_nh(uint16_t vrf_id, uint64_t nh, uint8_t nh_sz)
+{
+ uint8_t vrf_bits = get_vrf_bits(config.nb_vrfs);
+ /* +1 to account for ext bit in nexthop */
+ uint8_t vrf_shift = bits_in_nh(nh_sz) - (vrf_bits + 1);
+ uint64_t nh_mask = (1ULL << vrf_shift) - 1;
+
+ return (nh & nh_mask) | ((uint64_t)vrf_id << vrf_shift);
+}
+
+/* Decode VRF ID from nexthop (high bits) */
+static __rte_always_inline uint16_t
+decode_vrf_nh(uint64_t nh, uint8_t nh_sz)
+{
+ uint8_t vrf_bits = get_vrf_bits(config.nb_vrfs);
+ /* +1 to account for ext bit in nexthop */
+ uint8_t vrf_shift = bits_in_nh(nh_sz) - (vrf_bits + 1);
+
+ return (uint16_t)(nh >> vrf_shift);
+}
+
static int
get_fib_type(void)
{
@@ -629,7 +670,8 @@ print_usage(void)
"[-v <type of lookup function:"
"\ts1, s2, s3 (3 types of scalar), v (vector) -"
" for DIR24_8 based FIB\n"
- "\ts, v - for TRIE based ipv6 FIB>]\n",
+ "\ts, v - for TRIE based ipv6 FIB>]\n"
+ "[-V <number of VRFs (default 1)>]\n",
config.prgname);
}
@@ -652,6 +694,11 @@ check_config(void)
return -1;
}
+ if ((config.flags & CMP_FLAG) && (config.nb_vrfs > 1)) {
+ printf("-c option can not be used with -V > 1\n");
+ return -1;
+ }
+
if (!((config.ent_sz == 1) || (config.ent_sz == 2) ||
(config.ent_sz == 4) || (config.ent_sz == 8))) {
printf("wrong -e option %d, can be 1 or 2 or 4 or 8\n",
@@ -663,6 +710,24 @@ check_config(void)
printf("-e 1 is valid only for ipv4\n");
return -1;
}
+
+ /*
+ * For multi-VRF mode, VRF IDs are packed into the high bits of the
+ * nexthop for validation. Ensure there are enough bits:
+ * get_vrf_bits(nb_vrfs) must be strictly less than
+ * the total nexthop width.
+ */
+ if ((config.nb_vrfs > 1) && !(config.flags & IPV6_FLAG)) {
+ uint8_t nh_sz = rte_ctz32(config.ent_sz);
+ uint8_t vrf_bits = get_vrf_bits(config.nb_vrfs);
+ /* - 2 to leave at least 1 bit for nexthop and 1 bit for ext_ent flag */
+ if (vrf_bits >= bits_in_nh(nh_sz) - 2) {
+ printf("%u VRFs cannot be encoded in a %u-byte nexthop; "
+ "use a wider -e entry size\n",
+ config.nb_vrfs, config.ent_sz);
+ return -1;
+ }
+ }
return 0;
}
@@ -672,7 +737,7 @@ parse_opts(int argc, char **argv)
int opt;
char *endptr;
- while ((opt = getopt(argc, argv, "f:t:n:d:l:r:c6ab:e:g:w:u:sv:")) !=
+ while ((opt = getopt(argc, argv, "f:t:n:d:l:r:c6ab:e:g:w:u:sv:V:")) !=
-1) {
switch (opt) {
case 'f':
@@ -781,6 +846,16 @@ parse_opts(int argc, char **argv)
}
print_usage();
rte_exit(-EINVAL, "Invalid option -v %s\n", optarg);
+ case 'V':
+ errno = 0;
+ config.nb_vrfs = strtoul(optarg, &endptr, 10);
+ /* VRF IDs are uint16_t, max valid VRF is 65535 */
+ if ((errno != 0) || (config.nb_vrfs == 0) ||
+ (config.nb_vrfs > UINT16_MAX)) {
+ print_usage();
+ rte_exit(-EINVAL, "Invalid option -V: must be 1..65535\n");
+ }
+ break;
default:
print_usage();
rte_exit(-EINVAL, "Invalid options\n");
@@ -820,6 +895,7 @@ run_v4(void)
{
uint64_t start, acc;
uint64_t def_nh = 0;
+ uint8_t nh_sz = rte_ctz32(config.ent_sz);
struct rte_fib *fib;
struct rte_fib_conf conf = {0};
struct rt_rule_4 *rt;
@@ -830,6 +906,7 @@ run_v4(void)
uint32_t *tbl4 = config.lookup_tbl;
uint64_t fib_nh[BURST_SZ];
uint32_t lpm_nh[BURST_SZ];
+ uint16_t *vrf_ids = NULL;
rt = (struct rt_rule_4 *)config.rt;
@@ -843,16 +920,38 @@ run_v4(void)
return ret;
}
+ /* Allocate VRF IDs array for lookups if using multiple VRFs */
+ if (config.nb_vrfs > 1) {
+ vrf_ids = rte_malloc(NULL, sizeof(uint16_t) * config.nb_lookup_ips, 0);
+ if (vrf_ids == NULL) {
+ printf("Can not alloc VRF IDs array\n");
+ return -ENOMEM;
+ }
+ /* Generate random VRF IDs for each lookup */
+ for (i = 0; i < config.nb_lookup_ips; i++)
+ vrf_ids[i] = rte_rand() % config.nb_vrfs;
+ }
+
conf.type = get_fib_type();
conf.default_nh = def_nh;
conf.max_routes = config.nb_routes * 2;
conf.rib_ext_sz = 0;
+ conf.max_vrfs = config.nb_vrfs;
+ conf.vrf_default_nh = NULL; /* Use global default for single VRF */
if (conf.type == RTE_FIB_DIR24_8) {
conf.dir24_8.nh_sz = rte_ctz32(config.ent_sz);
conf.dir24_8.num_tbl8 = RTE_MIN(config.tbl8,
get_max_nh(conf.dir24_8.nh_sz));
}
+ conf.vrf_default_nh = rte_malloc(NULL, conf.max_vrfs * sizeof(uint64_t), 0);
+ if (conf.vrf_default_nh == NULL) {
+ printf("Can not alloc VRF default nexthops array\n");
+ return -ENOMEM;
+ }
+ for (i = 0; i < conf.max_vrfs; i++)
+ conf.vrf_default_nh[i] = encode_vrf_nh(i, def_nh, nh_sz);
+
fib = rte_fib_create("test", -1, &conf);
if (fib == NULL) {
printf("Can not alloc FIB, err %d\n", rte_errno);
@@ -883,12 +982,27 @@ run_v4(void)
for (k = config.print_fract, i = 0; k > 0; k--) {
start = rte_rdtsc_precise();
for (j = 0; j < (config.nb_routes - i) / k; j++) {
- ret = rte_fib_add(fib, rt[i + j].addr, rt[i + j].depth,
- rt[i + j].nh);
- if (unlikely(ret != 0)) {
- printf("Can not add a route to FIB, err %d\n",
- ret);
- return -ret;
+ uint32_t idx = i + j;
+ if (config.nb_vrfs > 1) {
+ uint16_t vrf_id;
+ for (vrf_id = 0; vrf_id < config.nb_vrfs; vrf_id++) {
+ uint64_t nh = encode_vrf_nh(vrf_id, rt[idx].nh, nh_sz);
+ ret = rte_fib_vrf_add(fib, vrf_id, rt[idx].addr,
+ rt[idx].depth, nh);
+ if (unlikely(ret != 0)) {
+ printf("Can not add a route to FIB, err %d\n",
+ ret);
+ return -ret;
+ }
+ }
+ } else {
+ ret = rte_fib_add(fib, rt[idx].addr, rt[idx].depth,
+ rt[idx].nh);
+ if (unlikely(ret != 0)) {
+ printf("Can not add a route to FIB, err %d\n",
+ ret);
+ return -ret;
+ }
}
}
printf("AVG FIB add %"PRIu64"\n",
@@ -928,14 +1042,33 @@ run_v4(void)
acc = 0;
for (i = 0; i < config.nb_lookup_ips; i += BURST_SZ) {
start = rte_rdtsc_precise();
- ret = rte_fib_lookup_bulk(fib, tbl4 + i, fib_nh, BURST_SZ);
+ if (config.nb_vrfs > 1)
+ ret = rte_fib_vrf_lookup_bulk(fib, vrf_ids + i,
+ tbl4 + i, fib_nh, BURST_SZ);
+ else
+ ret = rte_fib_lookup_bulk(fib, tbl4 + i, fib_nh,
+ BURST_SZ);
acc += rte_rdtsc_precise() - start;
if (ret != 0) {
printf("FIB lookup fails, err %d\n", ret);
return -ret;
}
+ /* Validate VRF IDs in returned nexthops */
+ if (config.nb_vrfs > 1) {
+ for (j = 0; j < BURST_SZ; j++) {
+ uint16_t returned_vrf = decode_vrf_nh(fib_nh[j], nh_sz);
+ if (returned_vrf != vrf_ids[i + j]) {
+ printf("VRF validation failed: "
+ "expected VRF %u, got %u\n",
+ vrf_ids[i + j], returned_vrf);
+ return -1;
+ }
+ }
+ }
}
printf("AVG FIB lookup %.1f\n", (double)acc / (double)i);
+ if (config.nb_vrfs > 1)
+ printf("VRF validation passed\n");
if (config.flags & CMP_FLAG) {
acc = 0;
@@ -970,8 +1103,17 @@ run_v4(void)
for (k = config.print_fract, i = 0; k > 0; k--) {
start = rte_rdtsc_precise();
- for (j = 0; j < (config.nb_routes - i) / k; j++)
- rte_fib_delete(fib, rt[i + j].addr, rt[i + j].depth);
+ for (j = 0; j < (config.nb_routes - i) / k; j++) {
+ uint32_t idx = i + j;
+ if (config.nb_vrfs > 1) {
+ uint16_t vrf_id;
+ for (vrf_id = 0; vrf_id < config.nb_vrfs; vrf_id++)
+ rte_fib_vrf_delete(fib, vrf_id, rt[idx].addr,
+ rt[idx].depth);
+ } else {
+ rte_fib_delete(fib, rt[idx].addr, rt[idx].depth);
+ }
+ }
printf("AVG FIB delete %"PRIu64"\n",
(rte_rdtsc_precise() - start) / j);
@@ -991,6 +1133,9 @@ run_v4(void)
}
}
+ if (vrf_ids != NULL)
+ rte_free(vrf_ids);
+
return 0;
}
diff --git a/app/test/test_fib.c b/app/test/test_fib.c
index bd73399d56..6a5d9836de 100644
--- a/app/test/test_fib.c
+++ b/app/test/test_fib.c
@@ -24,6 +24,11 @@ static int32_t test_get_invalid(void);
static int32_t test_lookup(void);
static int32_t test_invalid_rcu(void);
static int32_t test_fib_rcu_sync_rw(void);
+static int32_t test_create_vrf(void);
+static int32_t test_vrf_add_del(void);
+static int32_t test_vrf_lookup(void);
+static int32_t test_vrf_isolation(void);
+static int32_t test_vrf_all_nh_sizes(void);
#define MAX_ROUTES (1 << 16)
#define MAX_TBL8 (1 << 15)
@@ -588,6 +593,294 @@ test_fib_rcu_sync_rw(void)
return status == 0 ? TEST_SUCCESS : TEST_FAILED;
}
+/*
+ * Test VRF creation and basic operations
+ */
+static int32_t
+test_create_vrf(void)
+{
+ struct rte_fib *fib = NULL;
+ struct rte_fib_conf config = { 0 };
+ uint64_t def_nh = 100;
+ uint64_t vrf_def_nh[4] = {100, 200, 300, 400};
+
+ config.max_routes = MAX_ROUTES;
+ config.rib_ext_sz = 0;
+ config.default_nh = def_nh;
+ config.type = RTE_FIB_DIR24_8;
+ config.dir24_8.nh_sz = RTE_FIB_DIR24_8_4B;
+ config.dir24_8.num_tbl8 = MAX_TBL8;
+
+ /* Test single VRF (backward compat) */
+ config.max_vrfs = 0;
+ config.vrf_default_nh = NULL;
+ fib = rte_fib_create(__func__, SOCKET_ID_ANY, &config);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create FIB with max_vrfs=0\n");
+ rte_fib_free(fib);
+
+ /* Test single VRF explicitly */
+ config.max_vrfs = 1;
+ fib = rte_fib_create(__func__, SOCKET_ID_ANY, &config);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create FIB with max_vrfs=1\n");
+ rte_fib_free(fib);
+
+ /* Test multi-VRF with per-VRF defaults */
+ config.max_vrfs = 4;
+ config.vrf_default_nh = vrf_def_nh;
+ fib = rte_fib_create(__func__, SOCKET_ID_ANY, &config);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create FIB with max_vrfs=4\n");
+ rte_fib_free(fib);
+
+ return TEST_SUCCESS;
+}
+
+/*
+ * Test VRF route add/delete operations
+ */
+static int32_t
+test_vrf_add_del(void)
+{
+ struct rte_fib *fib = NULL;
+ struct rte_fib_conf config = { 0 };
+ uint64_t def_nh = 100;
+ uint64_t vrf_def_nh[4] = {100, 200, 300, 400};
+ uint32_t ip = RTE_IPV4(192, 168, 1, 0);
+ uint8_t depth = 24;
+ uint64_t nh = 1000;
+ int ret;
+
+ config.max_routes = MAX_ROUTES;
+ config.rib_ext_sz = 0;
+ config.default_nh = def_nh;
+ config.type = RTE_FIB_DIR24_8;
+ config.dir24_8.nh_sz = RTE_FIB_DIR24_8_4B;
+ config.dir24_8.num_tbl8 = MAX_TBL8;
+ config.max_vrfs = 4;
+ config.vrf_default_nh = vrf_def_nh;
+
+ fib = rte_fib_create(__func__, SOCKET_ID_ANY, &config);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create FIB\n");
+
+ /* Add route to VRF 0 */
+ ret = rte_fib_vrf_add(fib, 0, ip, depth, nh);
+ RTE_TEST_ASSERT(ret == 0, "Failed to add route to VRF 0\n");
+
+ /* Add route to VRF 1 with different nexthop */
+ ret = rte_fib_vrf_add(fib, 1, ip, depth, nh + 1);
+ RTE_TEST_ASSERT(ret == 0, "Failed to add route to VRF 1\n");
+
+ /* Add route to VRF 2 */
+ ret = rte_fib_vrf_add(fib, 2, ip, depth, nh + 2);
+ RTE_TEST_ASSERT(ret == 0, "Failed to add route to VRF 2\n");
+
+ /* Test invalid VRF ID */
+ ret = rte_fib_vrf_add(fib, 10, ip, depth, nh);
+ RTE_TEST_ASSERT(ret != 0, "Should fail with invalid VRF ID\n");
+
+ /* Delete route from VRF 1 */
+ ret = rte_fib_vrf_delete(fib, 1, ip, depth);
+ RTE_TEST_ASSERT(ret == 0, "Failed to delete route from VRF 1\n");
+
+ /* Delete non-existent route - implementation may return error */
+ ret = rte_fib_vrf_delete(fib, 3, ip, depth);
+ (void)ret; /* Accept any return value */
+
+ rte_fib_free(fib);
+ return TEST_SUCCESS;
+}
+
+/*
+ * Test VRF lookup functionality
+ */
+static int32_t
+test_vrf_lookup(void)
+{
+ struct rte_fib *fib = NULL;
+ struct rte_fib_conf config = { 0 };
+ uint64_t def_nh = 100;
+ uint64_t vrf_def_nh[4] = {1000, 2000, 3000, 4000};
+ uint32_t ip_base = RTE_IPV4(10, 0, 0, 0);
+ uint16_t vrf_ids[8];
+ uint32_t ips[8];
+ uint64_t next_hops[8];
+ int ret;
+ uint32_t i;
+
+ config.max_routes = MAX_ROUTES;
+ config.rib_ext_sz = 0;
+ config.default_nh = def_nh;
+ config.type = RTE_FIB_DIR24_8;
+ config.dir24_8.nh_sz = RTE_FIB_DIR24_8_4B;
+ config.dir24_8.num_tbl8 = MAX_TBL8;
+ config.max_vrfs = 4;
+ config.vrf_default_nh = vrf_def_nh;
+
+ fib = rte_fib_create(__func__, SOCKET_ID_ANY, &config);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create FIB\n");
+
+ /* Add routes to different VRFs with VRF-specific nexthops */
+ for (i = 0; i < 4; i++) {
+ ret = rte_fib_vrf_add(fib, i, ip_base + (i << 16), 16, 100 + i);
+ RTE_TEST_ASSERT(ret == 0, "Failed to add route to VRF %u\n", i);
+ }
+
+ /* Prepare lookup: each IP should match its VRF-specific route */
+ for (i = 0; i < 4; i++) {
+ vrf_ids[i] = i;
+ ips[i] = ip_base + (i << 16) + 0x1234; /* Within the /16 */
+ }
+
+ /* Lookup should return VRF-specific nexthops */
+ ret = rte_fib_vrf_lookup_bulk(fib, vrf_ids, ips, next_hops, 4);
+ RTE_TEST_ASSERT(ret == 0, "VRF lookup failed\n");
+
+ for (i = 0; i < 4; i++) {
+ RTE_TEST_ASSERT(next_hops[i] == 100 + i,
+ "Wrong nexthop for VRF %u: expected %"PRIu64", got %"PRIu64"\n",
+ i, (uint64_t)(100 + i), next_hops[i]);
+ }
+
+ /* Test default nexthops for unmatched IPs */
+ for (i = 0; i < 4; i++) {
+ vrf_ids[i] = i;
+ ips[i] = RTE_IPV4(192, 168, i, 1); /* No route for these */
+ }
+
+ ret = rte_fib_vrf_lookup_bulk(fib, vrf_ids, ips, next_hops, 4);
+ RTE_TEST_ASSERT(ret == 0, "VRF lookup failed\n");
+
+ for (i = 0; i < 4; i++) {
+ RTE_TEST_ASSERT(next_hops[i] == vrf_def_nh[i],
+ "Wrong default nexthop for VRF %u: expected %"PRIu64", got %"PRIu64"\n",
+ i, vrf_def_nh[i], next_hops[i]);
+ }
+
+ rte_fib_free(fib);
+ return TEST_SUCCESS;
+}
+
+/*
+ * Test VRF isolation - routes in one VRF shouldn't affect others
+ */
+static int32_t
+test_vrf_isolation(void)
+{
+ struct rte_fib *fib = NULL;
+ struct rte_fib_conf config = { 0 };
+ uint64_t vrf_def_nh[3] = {100, 200, 300};
+ uint32_t ip = RTE_IPV4(10, 10, 10, 0);
+ uint16_t vrf_ids[3] = {0, 1, 2};
+ uint32_t ips[3];
+ uint64_t next_hops[3];
+ int ret;
+ uint32_t i;
+
+ config.max_routes = MAX_ROUTES;
+ config.rib_ext_sz = 0;
+ config.default_nh = 0;
+ config.type = RTE_FIB_DIR24_8;
+ config.dir24_8.nh_sz = RTE_FIB_DIR24_8_4B;
+ config.dir24_8.num_tbl8 = MAX_TBL8;
+ config.max_vrfs = 3;
+ config.vrf_default_nh = vrf_def_nh;
+
+ fib = rte_fib_create("test_vrfisol", SOCKET_ID_ANY, &config);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create FIB\n");
+
+ /* Add route only to VRF 1 */
+ ret = rte_fib_vrf_add(fib, 1, ip, 24, 777);
+ RTE_TEST_ASSERT(ret == 0, "Failed to add route to VRF 1\n");
+
+ /* Lookup same IP in all three VRFs */
+ for (i = 0; i < 3; i++)
+ ips[i] = ip + 15; /* Within /24 */
+
+ ret = rte_fib_vrf_lookup_bulk(fib, vrf_ids, ips, next_hops, 3);
+ RTE_TEST_ASSERT(ret == 0, "VRF lookup failed\n");
+
+ /* VRF 0 should get default */
+ RTE_TEST_ASSERT(next_hops[0] == vrf_def_nh[0],
+ "VRF 0 should return default nexthop\n");
+
+ /* VRF 1 should get the route */
+ RTE_TEST_ASSERT(next_hops[1] == 777,
+ "VRF 1 should return route nexthop 777, got %"PRIu64"\n", next_hops[1]);
+
+ /* VRF 2 should get default */
+ RTE_TEST_ASSERT(next_hops[2] == vrf_def_nh[2],
+ "VRF 2 should return default nexthop\n");
+
+ rte_fib_free(fib);
+ return TEST_SUCCESS;
+}
+
+/*
+ * Test multi-VRF with all nexthop sizes
+ */
+static int32_t
+test_vrf_all_nh_sizes(void)
+{
+ struct rte_fib *fib = NULL;
+ struct rte_fib_conf config = { 0 };
+ uint64_t vrf_def_nh[2] = {10, 20};
+ uint32_t ip = RTE_IPV4(172, 16, 0, 0);
+ uint16_t vrf_ids[2] = {0, 1};
+ uint32_t ips[2];
+ uint64_t next_hops[2];
+ int ret;
+ enum rte_fib_dir24_8_nh_sz nh_sizes[] = {
+ RTE_FIB_DIR24_8_1B,
+ RTE_FIB_DIR24_8_2B,
+ RTE_FIB_DIR24_8_4B,
+ RTE_FIB_DIR24_8_8B
+ };
+ uint64_t max_nhs[] = {127, 32767, 2147483647ULL, 9223372036854775807ULL};
+ int i;
+
+ config.max_routes = MAX_ROUTES;
+ config.rib_ext_sz = 0;
+ config.default_nh = 0;
+ config.type = RTE_FIB_DIR24_8;
+ config.dir24_8.num_tbl8 = 127;
+ config.max_vrfs = 2;
+ config.vrf_default_nh = vrf_def_nh;
+
+ for (i = 0; i < (int)RTE_DIM(nh_sizes); i++) {
+ char name[32];
+ config.dir24_8.nh_sz = nh_sizes[i];
+ snprintf(name, sizeof(name), "vrf_nh%d", i);
+
+ fib = rte_fib_create(name, SOCKET_ID_ANY, &config);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create FIB\n");
+
+ /* Add routes with max nexthop for this size */
+ ret = rte_fib_vrf_add(fib, 0, ip, 16, max_nhs[i]);
+ RTE_TEST_ASSERT(ret == 0,
+ "Failed to add route to VRF 0 with nh_sz=%d\n", nh_sizes[i]);
+
+ ret = rte_fib_vrf_add(fib, 1, ip, 16, max_nhs[i] - 1);
+ RTE_TEST_ASSERT(ret == 0,
+ "Failed to add route to VRF 1 with nh_sz=%d\n", nh_sizes[i]);
+
+ /* Lookup */
+ ips[0] = ip + 0x100;
+ ips[1] = ip + 0x200;
+
+ ret = rte_fib_vrf_lookup_bulk(fib, vrf_ids, ips, next_hops, 2);
+ RTE_TEST_ASSERT(ret == 0, "VRF lookup failed with nh_sz=%d\n", nh_sizes[i]);
+
+ RTE_TEST_ASSERT(next_hops[0] == max_nhs[i],
+ "Wrong nexthop for VRF 0 with nh_sz=%d\n", nh_sizes[i]);
+ RTE_TEST_ASSERT(next_hops[1] == max_nhs[i] - 1,
+ "Wrong nexthop for VRF 1 with nh_sz=%d\n", nh_sizes[i]);
+
+ rte_fib_free(fib);
+ fib = NULL;
+ }
+
+ return TEST_SUCCESS;
+}
+
static struct unit_test_suite fib_fast_tests = {
.suite_name = "fib autotest",
.setup = NULL,
@@ -600,6 +893,11 @@ static struct unit_test_suite fib_fast_tests = {
TEST_CASE(test_lookup),
TEST_CASE(test_invalid_rcu),
TEST_CASE(test_fib_rcu_sync_rw),
+ TEST_CASE(test_create_vrf),
+ TEST_CASE(test_vrf_add_del),
+ TEST_CASE(test_vrf_lookup),
+ TEST_CASE(test_vrf_isolation),
+ TEST_CASE(test_vrf_all_nh_sizes),
TEST_CASES_END()
}
};
--
2.43.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [RFC PATCH 3/4] fib6: add multi-VRF support
2026-03-22 15:42 [RFC PATCH 0/4] VRF support in FIB library Vladimir Medvedkin
2026-03-22 15:42 ` [RFC PATCH 1/4] fib: add multi-VRF support Vladimir Medvedkin
2026-03-22 15:42 ` [RFC PATCH 2/4] fib: add VRF functional and unit tests Vladimir Medvedkin
@ 2026-03-22 15:42 ` Vladimir Medvedkin
2026-03-22 15:42 ` [RFC PATCH 4/4] fib6: add VRF functional and unit tests Vladimir Medvedkin
` (5 subsequent siblings)
8 siblings, 0 replies; 45+ messages in thread
From: Vladimir Medvedkin @ 2026-03-22 15:42 UTC (permalink / raw)
To: dev; +Cc: rjarry, nsaxena16, mb, adwivedi, jerinjacobk
Add VRF (Virtual Routing and Forwarding) support to the IPv6
FIB library, allowing multiple independent routing tables
within a single FIB instance.
Introduce max_vrfs and vrf_default_nh in rte_fib6_conf and
add four new experimental APIs:
- rte_fib6_vrf_add() and rte_fib6_vrf_delete() for per-VRF
route management
- rte_fib6_vrf_lookup_bulk() for multi-VRF bulk lookups
- rte_fib6_vrf_get_rib() to retrieve a per-VRF RIB handle
Signed-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
---
lib/fib/rte_fib6.c | 166 ++++++++++++++++++++++++++-----
lib/fib/rte_fib6.h | 88 ++++++++++++++++-
lib/fib/trie.c | 158 +++++++++++++++++++++--------
lib/fib/trie.h | 51 +++++++---
lib/fib/trie_avx512.c | 225 ++++++++++++++++++++++++++++++++++++++----
lib/fib/trie_avx512.h | 39 +++++++-
6 files changed, 617 insertions(+), 110 deletions(-)
diff --git a/lib/fib/rte_fib6.c b/lib/fib/rte_fib6.c
index 770becdb61..0d2b2927d5 100644
--- a/lib/fib/rte_fib6.c
+++ b/lib/fib/rte_fib6.c
@@ -22,6 +22,8 @@
#include "trie.h"
#include "fib_log.h"
+#define FIB6_MAX_LOOKUP_BULK 64U
+
TAILQ_HEAD(rte_fib6_list, rte_tailq_entry);
static struct rte_tailq_elem rte_fib6_tailq = {
.name = "RTE_FIB6",
@@ -40,51 +42,61 @@ EAL_REGISTER_TAILQ(rte_fib6_tailq)
struct rte_fib6 {
char name[RTE_FIB6_NAMESIZE];
enum rte_fib6_type type; /**< Type of FIB struct */
- struct rte_rib6 *rib; /**< RIB helper datastructure */
- void *dp; /**< pointer to the dataplane struct*/
- rte_fib6_lookup_fn_t lookup; /**< FIB lookup function */
+ uint16_t num_vrfs; /**< Number of VRFs */
+ struct rte_rib6 **ribs; /**< RIB helper datastructures per VRF */
+ void *dp; /**< pointer to the dataplane struct */
+ rte_fib6_lookup_fn_t lookup; /**< lookup function */
rte_fib6_modify_fn_t modify; /**< modify FIB datastructure */
- uint64_t def_nh;
+ uint64_t *def_nh; /**< Per-VRF default next hop array */
};
static void
-dummy_lookup(void *fib_p, const struct rte_ipv6_addr *ips,
+dummy_lookup(void *fib_p, const uint16_t *vrf_ids,
+ const struct rte_ipv6_addr *ips,
uint64_t *next_hops, const unsigned int n)
{
unsigned int i;
struct rte_fib6 *fib = fib_p;
struct rte_rib6_node *node;
+ struct rte_rib6 *rib;
for (i = 0; i < n; i++) {
- node = rte_rib6_lookup(fib->rib, &ips[i]);
+ RTE_ASSERT(vrf_ids[i] < fib->num_vrfs);
+ rib = rte_fib6_vrf_get_rib(fib, vrf_ids[i]);
+ node = rte_rib6_lookup(rib, &ips[i]);
if (node != NULL)
rte_rib6_get_nh(node, &next_hops[i]);
else
- next_hops[i] = fib->def_nh;
+ next_hops[i] = fib->def_nh[vrf_ids[i]];
}
}
static int
-dummy_modify(struct rte_fib6 *fib, const struct rte_ipv6_addr *ip,
- uint8_t depth, uint64_t next_hop, int op)
+dummy_modify(struct rte_fib6 *fib, uint16_t vrf_id,
+ const struct rte_ipv6_addr *ip, uint8_t depth,
+ uint64_t next_hop, int op)
{
struct rte_rib6_node *node;
+ struct rte_rib6 *rib;
if ((fib == NULL) || (depth > RTE_IPV6_MAX_DEPTH))
return -EINVAL;
+ rib = rte_fib6_vrf_get_rib(fib, vrf_id);
+ if (rib == NULL)
+ return -EINVAL;
- node = rte_rib6_lookup_exact(fib->rib, ip, depth);
+ node = rte_rib6_lookup_exact(rib, ip, depth);
switch (op) {
case RTE_FIB6_ADD:
if (node == NULL)
- node = rte_rib6_insert(fib->rib, ip, depth);
+ node = rte_rib6_insert(rib, ip, depth);
if (node == NULL)
return -rte_errno;
return rte_rib6_set_nh(node, next_hop);
case RTE_FIB6_DEL:
if (node == NULL)
return -ENOENT;
- rte_rib6_remove(fib->rib, ip, depth);
+ rte_rib6_remove(rib, ip, depth);
return 0;
}
return -EINVAL;
@@ -113,7 +125,6 @@ init_dataplane(struct rte_fib6 *fib, __rte_unused int socket_id,
default:
return -EINVAL;
}
- return 0;
}
RTE_EXPORT_SYMBOL(rte_fib6_add)
@@ -124,7 +135,7 @@ rte_fib6_add(struct rte_fib6 *fib, const struct rte_ipv6_addr *ip,
if ((fib == NULL) || (ip == NULL) || (fib->modify == NULL) ||
(depth > RTE_IPV6_MAX_DEPTH))
return -EINVAL;
- return fib->modify(fib, ip, depth, next_hop, RTE_FIB6_ADD);
+ return fib->modify(fib, 0, ip, depth, next_hop, RTE_FIB6_ADD);
}
RTE_EXPORT_SYMBOL(rte_fib6_delete)
@@ -135,7 +146,7 @@ rte_fib6_delete(struct rte_fib6 *fib, const struct rte_ipv6_addr *ip,
if ((fib == NULL) || (ip == NULL) || (fib->modify == NULL) ||
(depth > RTE_IPV6_MAX_DEPTH))
return -EINVAL;
- return fib->modify(fib, ip, depth, 0, RTE_FIB6_DEL);
+ return fib->modify(fib, 0, ip, depth, 0, RTE_FIB6_DEL);
}
RTE_EXPORT_SYMBOL(rte_fib6_lookup_bulk)
@@ -144,23 +155,72 @@ rte_fib6_lookup_bulk(struct rte_fib6 *fib,
const struct rte_ipv6_addr *ips,
uint64_t *next_hops, int n)
{
+ static const uint16_t zero_vrf_ids[FIB6_MAX_LOOKUP_BULK];
+ unsigned int off = 0;
+ unsigned int total = (unsigned int)n;
+
FIB6_RETURN_IF_TRUE((fib == NULL) || (ips == NULL) ||
(next_hops == NULL) || (fib->lookup == NULL), -EINVAL);
- fib->lookup(fib->dp, ips, next_hops, n);
+
+ while (off < total) {
+ unsigned int chunk = RTE_MIN(total - off,
+ FIB6_MAX_LOOKUP_BULK);
+ fib->lookup(fib->dp, zero_vrf_ids, ips + off,
+ next_hops + off, chunk);
+ off += chunk;
+ }
+ return 0;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fib6_vrf_lookup_bulk, 26.07)
+int
+rte_fib6_vrf_lookup_bulk(struct rte_fib6 *fib, const uint16_t *vrf_ids,
+ const struct rte_ipv6_addr *ips, uint64_t *next_hops, int n)
+{
+ FIB6_RETURN_IF_TRUE((fib == NULL) || (vrf_ids == NULL) || (ips == NULL) ||
+ (next_hops == NULL) || (fib->lookup == NULL), -EINVAL);
+
+ fib->lookup(fib->dp, vrf_ids, ips, next_hops, (unsigned int)n);
+
return 0;
}
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fib6_vrf_add, 26.07)
+int
+rte_fib6_vrf_add(struct rte_fib6 *fib, uint16_t vrf_id,
+ const struct rte_ipv6_addr *ip, uint8_t depth, uint64_t next_hop)
+{
+ if ((fib == NULL) || (ip == NULL) || (fib->modify == NULL) ||
+ (depth > RTE_IPV6_MAX_DEPTH))
+ return -EINVAL;
+ return fib->modify(fib, vrf_id, ip, depth, next_hop, RTE_FIB6_ADD);
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fib6_vrf_delete, 26.07)
+int
+rte_fib6_vrf_delete(struct rte_fib6 *fib, uint16_t vrf_id,
+ const struct rte_ipv6_addr *ip, uint8_t depth)
+{
+ if ((fib == NULL) || (ip == NULL) || (fib->modify == NULL) ||
+ (depth > RTE_IPV6_MAX_DEPTH))
+ return -EINVAL;
+ return fib->modify(fib, vrf_id, ip, depth, 0, RTE_FIB6_DEL);
+}
+
RTE_EXPORT_SYMBOL(rte_fib6_create)
struct rte_fib6 *
rte_fib6_create(const char *name, int socket_id, struct rte_fib6_conf *conf)
{
char mem_name[RTE_FIB6_NAMESIZE];
+ char rib_name[RTE_FIB6_NAMESIZE];
int ret;
struct rte_fib6 *fib = NULL;
struct rte_rib6 *rib = NULL;
struct rte_tailq_entry *te;
struct rte_fib6_list *fib_list;
struct rte_rib6_conf rib_conf;
+ uint16_t num_vrfs;
+ uint16_t vrf;
/* Check user arguments. */
if ((name == NULL) || (conf == NULL) || (conf->max_routes < 0) ||
@@ -172,13 +232,41 @@ rte_fib6_create(const char *name, int socket_id, struct rte_fib6_conf *conf)
rib_conf.ext_sz = conf->rib_ext_sz;
rib_conf.max_nodes = conf->max_routes * 2;
- rib = rte_rib6_create(name, socket_id, &rib_conf);
- if (rib == NULL) {
- FIB_LOG(ERR,
- "Can not allocate RIB %s", name);
+ num_vrfs = (conf->max_vrfs == 0) ? 1 : conf->max_vrfs;
+
+ struct rte_rib6 **ribs = rte_zmalloc_socket("FIB6_RIBS",
+ num_vrfs * sizeof(*ribs), RTE_CACHE_LINE_SIZE, socket_id);
+ if (ribs == NULL) {
+ FIB_LOG(ERR, "FIB6 %s RIB array allocation failed", name);
+ rte_errno = ENOMEM;
+ return NULL;
+ }
+
+ uint64_t *def_nh = rte_zmalloc_socket("FIB6_DEF_NH",
+ num_vrfs * sizeof(*def_nh), RTE_CACHE_LINE_SIZE, socket_id);
+ if (def_nh == NULL) {
+ FIB_LOG(ERR, "FIB6 %s default nexthop array allocation failed", name);
+ rte_errno = ENOMEM;
+ rte_free(ribs);
return NULL;
}
+ for (vrf = 0; vrf < num_vrfs; vrf++) {
+ if (num_vrfs == 1)
+ snprintf(rib_name, sizeof(rib_name), "%s", name);
+ else
+ snprintf(rib_name, sizeof(rib_name), "%s_vrf%u", name, vrf);
+ rib = rte_rib6_create(rib_name, socket_id, &rib_conf);
+ if (rib == NULL) {
+ FIB_LOG(ERR, "Can not allocate RIB %s", rib_name);
+ rte_errno = ENOMEM;
+ goto free_ribs;
+ }
+ ribs[vrf] = rib;
+ def_nh[vrf] = (conf->vrf_default_nh != NULL) ?
+ conf->vrf_default_nh[vrf] : conf->default_nh;
+ }
+
snprintf(mem_name, sizeof(mem_name), "FIB6_%s", name);
fib_list = RTE_TAILQ_CAST(rte_fib6_tailq.head, rte_fib6_list);
@@ -214,15 +302,17 @@ rte_fib6_create(const char *name, int socket_id, struct rte_fib6_conf *conf)
goto free_te;
}
+ fib->num_vrfs = num_vrfs;
+ fib->ribs = ribs;
+ fib->def_nh = def_nh;
+
rte_strlcpy(fib->name, name, sizeof(fib->name));
- fib->rib = rib;
fib->type = conf->type;
- fib->def_nh = conf->default_nh;
ret = init_dataplane(fib, socket_id, conf);
if (ret < 0) {
FIB_LOG(ERR,
- "FIB dataplane struct %s memory allocation failed",
- name);
+ "FIB dataplane struct %s memory allocation failed with err %d",
+ name, ret);
rte_errno = -ret;
goto free_fib;
}
@@ -240,7 +330,12 @@ rte_fib6_create(const char *name, int socket_id, struct rte_fib6_conf *conf)
rte_free(te);
exit:
rte_mcfg_tailq_write_unlock();
- rte_rib6_free(rib);
+free_ribs:
+ for (vrf = 0; vrf < num_vrfs; vrf++)
+ rte_rib6_free(ribs[vrf]);
+
+ rte_free(def_nh);
+ rte_free(ribs);
return NULL;
}
@@ -309,7 +404,13 @@ rte_fib6_free(struct rte_fib6 *fib)
rte_mcfg_tailq_write_unlock();
free_dataplane(fib);
- rte_rib6_free(fib->rib);
+ if (fib->ribs != NULL) {
+ uint16_t vrf;
+ for (vrf = 0; vrf < fib->num_vrfs; vrf++)
+ rte_rib6_free(fib->ribs[vrf]);
+ }
+ rte_free(fib->ribs);
+ rte_free(fib->def_nh);
rte_free(fib);
rte_free(te);
}
@@ -325,7 +426,18 @@ RTE_EXPORT_SYMBOL(rte_fib6_get_rib)
struct rte_rib6 *
rte_fib6_get_rib(struct rte_fib6 *fib)
{
- return (fib == NULL) ? NULL : fib->rib;
+ return (fib == NULL || fib->ribs == NULL) ? NULL : fib->ribs[0];
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fib6_vrf_get_rib, 26.07)
+struct rte_rib6 *
+rte_fib6_vrf_get_rib(struct rte_fib6 *fib, uint16_t vrf_id)
+{
+ if (fib == NULL || fib->ribs == NULL)
+ return NULL;
+ if (vrf_id >= fib->num_vrfs)
+ return NULL;
+ return fib->ribs[vrf_id];
}
RTE_EXPORT_SYMBOL(rte_fib6_select_lookup)
diff --git a/lib/fib/rte_fib6.h b/lib/fib/rte_fib6.h
index 4527328bf0..864ec08c4e 100644
--- a/lib/fib/rte_fib6.h
+++ b/lib/fib/rte_fib6.h
@@ -55,11 +55,11 @@ enum rte_fib6_type {
};
/** Modify FIB function */
-typedef int (*rte_fib6_modify_fn_t)(struct rte_fib6 *fib,
+typedef int (*rte_fib6_modify_fn_t)(struct rte_fib6 *fib, uint16_t vrf_id,
const struct rte_ipv6_addr *ip, uint8_t depth,
uint64_t next_hop, int op);
/** FIB bulk lookup function */
-typedef void (*rte_fib6_lookup_fn_t)(void *fib,
+typedef void (*rte_fib6_lookup_fn_t)(void *fib, const uint16_t *vrf_ids,
const struct rte_ipv6_addr *ips,
uint64_t *next_hops, const unsigned int n);
@@ -97,6 +97,10 @@ struct rte_fib6_conf {
uint32_t num_tbl8;
} trie;
};
+ /** Number of VRFs to support (0 or 1 = single VRF for backward compat) */
+ uint16_t max_vrfs;
+ /** Per-VRF default nexthops (NULL = use default_nh for all) */
+ uint64_t *vrf_default_nh;
};
/** FIB RCU QSBR configuration structure. */
@@ -215,6 +219,70 @@ rte_fib6_lookup_bulk(struct rte_fib6 *fib,
const struct rte_ipv6_addr *ips,
uint64_t *next_hops, int n);
+/**
+ * Add a route to the FIB with VRF ID.
+ *
+ * @param fib
+ * FIB object handle
+ * @param vrf_id
+ * VRF ID (0 to max_vrfs-1)
+ * @param ip
+ * IPv6 prefix address to be added to the FIB
+ * @param depth
+ * Prefix length
+ * @param next_hop
+ * Next hop to be added to the FIB
+ * @return
+ * 0 on success, negative value otherwise
+ */
+__rte_experimental
+int
+rte_fib6_vrf_add(struct rte_fib6 *fib, uint16_t vrf_id,
+ const struct rte_ipv6_addr *ip, uint8_t depth, uint64_t next_hop);
+
+/**
+ * Delete a rule from the FIB with VRF ID.
+ *
+ * @param fib
+ * FIB object handle
+ * @param vrf_id
+ * VRF ID (0 to max_vrfs-1)
+ * @param ip
+ * IPv6 prefix address to be deleted from the FIB
+ * @param depth
+ * Prefix length
+ * @return
+ * 0 on success, negative value otherwise
+ */
+__rte_experimental
+int
+rte_fib6_vrf_delete(struct rte_fib6 *fib, uint16_t vrf_id,
+ const struct rte_ipv6_addr *ip, uint8_t depth);
+
+/**
+ * Lookup multiple IP addresses in the FIB with per-packet VRF IDs.
+ *
+ * @param fib
+ * FIB object handle
+ * @param vrf_ids
+ * Array of VRF IDs corresponding to ips[] (0 to max_vrfs-1)
+ * @param ips
+ * Array of IPv6s to be looked up in the FIB
+ * @param next_hops
+ * Next hop of the most specific rule found for IP.
+ * This is an array of eight byte values.
+ * If the lookup for the given IP failed, then corresponding element would
+ * contain default nexthop value configured for that VRF.
+ * @param n
+ * Number of elements in vrf_ids/ips/next_hops arrays to lookup.
+ * @return
+ * -EINVAL for incorrect arguments, otherwise 0
+ */
+__rte_experimental
+int
+rte_fib6_vrf_lookup_bulk(struct rte_fib6 *fib, const uint16_t *vrf_ids,
+ const struct rte_ipv6_addr *ips, uint64_t *next_hops, int n);
+
/**
* Get pointer to the dataplane specific struct
*
@@ -228,7 +296,7 @@ void *
rte_fib6_get_dp(struct rte_fib6 *fib);
/**
- * Get pointer to the RIB6
+ * Get pointer to the RIB6 for VRF 0
*
* @param fib
* FIB object handle
@@ -239,6 +307,20 @@ rte_fib6_get_dp(struct rte_fib6 *fib);
struct rte_rib6 *
rte_fib6_get_rib(struct rte_fib6 *fib);
+/**
+ * Get the RIB for a specific VRF.
+ *
+ * @param fib
+ * FIB object handle
+ * @param vrf_id
+ * VRF ID (0 to max_vrfs-1)
+ * @return
+ * RIB for the specified VRF or NULL on error.
+ */
+__rte_experimental
+struct rte_rib6 *
+rte_fib6_vrf_get_rib(struct rte_fib6 *fib, uint16_t vrf_id);
+
/**
* Set lookup function based on type
*
diff --git a/lib/fib/trie.c b/lib/fib/trie.c
index fa5d9ec6b0..2acc9d9526 100644
--- a/lib/fib/trie.c
+++ b/lib/fib/trie.c
@@ -30,22 +30,27 @@ enum edge {
};
static inline rte_fib6_lookup_fn_t
-get_scalar_fn(enum rte_fib_trie_nh_sz nh_sz)
+get_scalar_fn(const struct rte_trie_tbl *dp, enum rte_fib_trie_nh_sz nh_sz)
{
+ bool single_vrf = dp->num_vrfs <= 1;
+
switch (nh_sz) {
case RTE_FIB6_TRIE_2B:
- return rte_trie_lookup_bulk_2b;
+ return single_vrf ? rte_trie_lookup_bulk_2b :
+ rte_trie_lookup_bulk_vrf_2b;
case RTE_FIB6_TRIE_4B:
- return rte_trie_lookup_bulk_4b;
+ return single_vrf ? rte_trie_lookup_bulk_4b :
+ rte_trie_lookup_bulk_vrf_4b;
case RTE_FIB6_TRIE_8B:
- return rte_trie_lookup_bulk_8b;
+ return single_vrf ? rte_trie_lookup_bulk_8b :
+ rte_trie_lookup_bulk_vrf_8b;
default:
return NULL;
}
}
static inline rte_fib6_lookup_fn_t
-get_vector_fn(enum rte_fib_trie_nh_sz nh_sz)
+get_vector_fn(const struct rte_trie_tbl *dp, enum rte_fib_trie_nh_sz nh_sz)
{
#ifdef CC_AVX512_SUPPORT
if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512F) <= 0 ||
@@ -53,13 +58,40 @@ get_vector_fn(enum rte_fib_trie_nh_sz nh_sz)
rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512BW) <= 0 ||
rte_vect_get_max_simd_bitwidth() < RTE_VECT_SIMD_512)
return NULL;
+
+ if (dp->num_vrfs <= 1) {
+ switch (nh_sz) {
+ case RTE_FIB6_TRIE_2B:
+ return rte_trie_vec_lookup_bulk_2b;
+ case RTE_FIB6_TRIE_4B:
+ return rte_trie_vec_lookup_bulk_4b;
+ case RTE_FIB6_TRIE_8B:
+ return rte_trie_vec_lookup_bulk_8b;
+ default:
+ return NULL;
+ }
+ }
+
+ if (dp->num_vrfs >= 256) {
+ switch (nh_sz) {
+ case RTE_FIB6_TRIE_2B:
+ return rte_trie_vec_lookup_bulk_vrf_2b_large;
+ case RTE_FIB6_TRIE_4B:
+ return rte_trie_vec_lookup_bulk_vrf_4b_large;
+ case RTE_FIB6_TRIE_8B:
+ return rte_trie_vec_lookup_bulk_vrf_8b_large;
+ default:
+ return NULL;
+ }
+ }
+
switch (nh_sz) {
case RTE_FIB6_TRIE_2B:
- return rte_trie_vec_lookup_bulk_2b;
+ return rte_trie_vec_lookup_bulk_vrf_2b;
case RTE_FIB6_TRIE_4B:
- return rte_trie_vec_lookup_bulk_4b;
+ return rte_trie_vec_lookup_bulk_vrf_4b;
case RTE_FIB6_TRIE_8B:
- return rte_trie_vec_lookup_bulk_8b;
+ return rte_trie_vec_lookup_bulk_vrf_8b;
default:
return NULL;
}
@@ -83,12 +115,12 @@ trie_get_lookup_fn(void *p, enum rte_fib6_lookup_type type)
switch (type) {
case RTE_FIB6_LOOKUP_TRIE_SCALAR:
- return get_scalar_fn(nh_sz);
+ return get_scalar_fn(dp, nh_sz);
case RTE_FIB6_LOOKUP_TRIE_VECTOR_AVX512:
- return get_vector_fn(nh_sz);
+ return get_vector_fn(dp, nh_sz);
case RTE_FIB6_LOOKUP_DEFAULT:
- ret_fn = get_vector_fn(nh_sz);
- return (ret_fn != NULL) ? ret_fn : get_scalar_fn(nh_sz);
+ ret_fn = get_vector_fn(dp, nh_sz);
+ return (ret_fn != NULL) ? ret_fn : get_scalar_fn(dp, nh_sz);
default:
return NULL;
}
@@ -310,19 +342,22 @@ recycle_root_path(struct rte_trie_tbl *dp, const uint8_t *ip_part,
}
static inline int
-build_common_root(struct rte_trie_tbl *dp, const struct rte_ipv6_addr *ip,
- int common_bytes, void **tbl)
+build_common_root(struct rte_trie_tbl *dp, uint16_t vrf_id,
+ const struct rte_ipv6_addr *ip, int common_bytes, void **tbl)
{
void *tbl_ptr = NULL;
uint64_t *cur_tbl;
uint64_t val;
int i, j, idx, prev_idx = 0;
+ uint64_t idx_tbl;
+ uint64_t tbl24_base = (uint64_t)vrf_id * TRIE_TBL24_NUM_ENT;
cur_tbl = dp->tbl24;
for (i = 3, j = 0; i <= common_bytes; i++) {
idx = get_idx(ip, prev_idx, i - j, j);
- val = get_tbl_val_by_idx(cur_tbl, idx, dp->nh_sz);
- tbl_ptr = get_tbl_p_by_idx(cur_tbl, idx, dp->nh_sz);
+ idx_tbl = (cur_tbl == dp->tbl24) ? idx + tbl24_base : (uint32_t)idx;
+ val = get_tbl_val_by_idx(cur_tbl, idx_tbl, dp->nh_sz);
+ tbl_ptr = get_tbl_p_by_idx(cur_tbl, idx_tbl, dp->nh_sz);
if ((val & TRIE_EXT_ENT) != TRIE_EXT_ENT) {
idx = tbl8_alloc(dp, val);
if (unlikely(idx < 0))
@@ -336,8 +371,11 @@ build_common_root(struct rte_trie_tbl *dp, const struct rte_ipv6_addr *ip,
j = i;
cur_tbl = dp->tbl8;
}
- *tbl = get_tbl_p_by_idx(cur_tbl, prev_idx * TRIE_TBL8_GRP_NUM_ENT,
- dp->nh_sz);
+
+ uint64_t final_idx = (cur_tbl == dp->tbl24) ?
+ (prev_idx * TRIE_TBL8_GRP_NUM_ENT + tbl24_base) :
+ (prev_idx * TRIE_TBL8_GRP_NUM_ENT);
+ *tbl = get_tbl_p_by_idx(cur_tbl, final_idx, dp->nh_sz);
return 0;
}
@@ -385,7 +423,8 @@ write_edge(struct rte_trie_tbl *dp, const uint8_t *ip_part, uint64_t next_hop,
#define TBL8_LEN (RTE_IPV6_ADDR_SIZE - TBL24_BYTES)
static int
-install_to_dp(struct rte_trie_tbl *dp, const struct rte_ipv6_addr *ledge,
+install_to_dp(struct rte_trie_tbl *dp, uint16_t vrf_id,
+ const struct rte_ipv6_addr *ledge,
const struct rte_ipv6_addr *r, uint64_t next_hop)
{
void *common_root_tbl;
@@ -409,7 +448,7 @@ install_to_dp(struct rte_trie_tbl *dp, const struct rte_ipv6_addr *ledge,
break;
}
- ret = build_common_root(dp, ledge, common_bytes, &common_root_tbl);
+ ret = build_common_root(dp, vrf_id, ledge, common_bytes, &common_root_tbl);
if (unlikely(ret != 0))
return ret;
/*first uncommon tbl8 byte idx*/
@@ -455,7 +494,7 @@ install_to_dp(struct rte_trie_tbl *dp, const struct rte_ipv6_addr *ledge,
uint8_t common_tbl8 = (common_bytes < TBL24_BYTES) ?
0 : common_bytes - (TBL24_BYTES - 1);
- ent = get_tbl24_p(dp, ledge, dp->nh_sz);
+ ent = get_tbl24_p(dp, vrf_id, ledge, dp->nh_sz);
recycle_root_path(dp, ledge->a + TBL24_BYTES, common_tbl8, ent);
return 0;
}
@@ -482,9 +521,8 @@ get_nxt_net(struct rte_ipv6_addr *ip, uint8_t depth)
}
static int
-modify_dp(struct rte_trie_tbl *dp, struct rte_rib6 *rib,
- const struct rte_ipv6_addr *ip,
- uint8_t depth, uint64_t next_hop)
+modify_dp(struct rte_trie_tbl *dp, struct rte_rib6 *rib, uint16_t vrf_id,
+ const struct rte_ipv6_addr *ip, uint8_t depth, uint64_t next_hop)
{
struct rte_rib6_node *tmp = NULL;
struct rte_ipv6_addr ledge, redge;
@@ -507,7 +545,7 @@ modify_dp(struct rte_trie_tbl *dp, struct rte_rib6 *rib,
get_nxt_net(&ledge, tmp_depth);
continue;
}
- ret = install_to_dp(dp, &ledge, &redge, next_hop);
+ ret = install_to_dp(dp, vrf_id, &ledge, &redge, next_hop);
if (ret != 0)
return ret;
get_nxt_net(&redge, tmp_depth);
@@ -525,7 +563,7 @@ modify_dp(struct rte_trie_tbl *dp, struct rte_rib6 *rib,
!rte_ipv6_addr_is_unspec(&ledge))
break;
- ret = install_to_dp(dp, &ledge, &redge, next_hop);
+ ret = install_to_dp(dp, vrf_id, &ledge, &redge, next_hop);
if (ret != 0)
return ret;
}
@@ -535,7 +573,8 @@ modify_dp(struct rte_trie_tbl *dp, struct rte_rib6 *rib,
}
int
-trie_modify(struct rte_fib6 *fib, const struct rte_ipv6_addr *ip,
+trie_modify(struct rte_fib6 *fib, uint16_t vrf_id,
+ const struct rte_ipv6_addr *ip,
uint8_t depth, uint64_t next_hop, int op)
{
struct rte_trie_tbl *dp;
@@ -552,9 +591,11 @@ trie_modify(struct rte_fib6 *fib, const struct rte_ipv6_addr *ip,
return -EINVAL;
dp = rte_fib6_get_dp(fib);
- RTE_ASSERT(dp);
- rib = rte_fib6_get_rib(fib);
- RTE_ASSERT(rib);
+ rib = rte_fib6_vrf_get_rib(fib, vrf_id);
+ RTE_ASSERT((dp != NULL) && (rib != NULL));
+
+ if (vrf_id >= dp->num_vrfs)
+ return -EINVAL;
ip_masked = *ip;
rte_ipv6_addr_mask(&ip_masked, depth);
@@ -597,7 +638,7 @@ trie_modify(struct rte_fib6 *fib, const struct rte_ipv6_addr *ip,
rte_rib6_get_nh(node, &node_nh);
if (node_nh == next_hop)
return 0;
- ret = modify_dp(dp, rib, &ip_masked, depth, next_hop);
+ ret = modify_dp(dp, rib, vrf_id, &ip_masked, depth, next_hop);
if (ret == 0)
rte_rib6_set_nh(node, next_hop);
return 0;
@@ -616,7 +657,7 @@ trie_modify(struct rte_fib6 *fib, const struct rte_ipv6_addr *ip,
if (par_nh == next_hop)
goto successfully_added;
}
- ret = modify_dp(dp, rib, &ip_masked, depth, next_hop);
+ ret = modify_dp(dp, rib, vrf_id, &ip_masked, depth, next_hop);
if (ret != 0) {
rte_rib6_remove(rib, &ip_masked, depth);
return ret;
@@ -633,10 +674,11 @@ trie_modify(struct rte_fib6 *fib, const struct rte_ipv6_addr *ip,
rte_rib6_get_nh(parent, &par_nh);
rte_rib6_get_nh(node, &node_nh);
if (par_nh != node_nh)
- ret = modify_dp(dp, rib, &ip_masked, depth,
+ ret = modify_dp(dp, rib, vrf_id, &ip_masked, depth,
par_nh);
} else
- ret = modify_dp(dp, rib, &ip_masked, depth, dp->def_nh);
+ ret = modify_dp(dp, rib, vrf_id, &ip_masked, depth,
+ dp->def_nh[vrf_id]);
if (ret != 0)
return ret;
@@ -656,9 +698,11 @@ trie_create(const char *name, int socket_id,
{
char mem_name[TRIE_NAMESIZE];
struct rte_trie_tbl *dp = NULL;
- uint64_t def_nh;
uint32_t num_tbl8;
enum rte_fib_trie_nh_sz nh_sz;
+ uint16_t num_vrfs;
+ uint16_t vrf;
+ uint64_t tbl24_sz;
if ((name == NULL) || (conf == NULL) ||
(conf->trie.nh_sz < RTE_FIB6_TRIE_2B) ||
@@ -673,21 +717,28 @@ trie_create(const char *name, int socket_id,
return NULL;
}
- def_nh = conf->default_nh;
nh_sz = conf->trie.nh_sz;
num_tbl8 = conf->trie.num_tbl8;
+ num_vrfs = (conf->max_vrfs == 0) ? 1 : conf->max_vrfs;
+ tbl24_sz = (uint64_t)num_vrfs * TRIE_TBL24_NUM_ENT * (1 << nh_sz);
+
+ if (conf->vrf_default_nh != NULL) {
+ for (vrf = 0; vrf < num_vrfs; vrf++) {
+ if (conf->vrf_default_nh[vrf] > get_max_nh(nh_sz)) {
+ rte_errno = EINVAL;
+ return NULL;
+ }
+ }
+ }
snprintf(mem_name, sizeof(mem_name), "DP_%s", name);
- dp = rte_zmalloc_socket(name, sizeof(struct rte_trie_tbl) +
- TRIE_TBL24_NUM_ENT * (1 << nh_sz) + sizeof(uint32_t),
+ dp = rte_zmalloc_socket(name, sizeof(struct rte_trie_tbl) + tbl24_sz,
RTE_CACHE_LINE_SIZE, socket_id);
if (dp == NULL) {
rte_errno = ENOMEM;
return dp;
}
- write_to_dp(&dp->tbl24, (def_nh << 1), nh_sz, 1 << 24);
-
snprintf(mem_name, sizeof(mem_name), "TBL8_%p", dp);
dp->tbl8 = rte_zmalloc_socket(mem_name, TRIE_TBL8_GRP_NUM_ENT *
(1ll << nh_sz) * (num_tbl8 + 1),
@@ -697,9 +748,32 @@ trie_create(const char *name, int socket_id,
rte_free(dp);
return NULL;
}
- dp->def_nh = def_nh;
+
+ snprintf(mem_name, sizeof(mem_name), "DEF_NH_%p", dp);
+ dp->def_nh = rte_zmalloc_socket(mem_name,
+ num_vrfs * sizeof(*dp->def_nh),
+ RTE_CACHE_LINE_SIZE, socket_id);
+ if (dp->def_nh == NULL) {
+ rte_errno = ENOMEM;
+ rte_free(dp->tbl8);
+ rte_free(dp);
+ return NULL;
+ }
+
+ for (vrf = 0; vrf < num_vrfs; vrf++) {
+ uint64_t vrf_def = (conf->vrf_default_nh != NULL) ?
+ conf->vrf_default_nh[vrf] : conf->default_nh;
+ uint8_t *tbl24_ptr = (uint8_t *)dp->tbl24 +
+ ((uint64_t)vrf * TRIE_TBL24_NUM_ENT << nh_sz);
+
+ dp->def_nh[vrf] = vrf_def;
+ write_to_dp((void *)tbl24_ptr, (vrf_def << 1), nh_sz,
+ TRIE_TBL24_NUM_ENT);
+ }
+
dp->nh_sz = nh_sz;
dp->number_tbl8s = num_tbl8;
+ dp->num_vrfs = num_vrfs;
snprintf(mem_name, sizeof(mem_name), "TBL8_idxes_%p", dp);
dp->tbl8_pool = rte_zmalloc_socket(mem_name,
@@ -707,6 +781,7 @@ trie_create(const char *name, int socket_id,
RTE_CACHE_LINE_SIZE, socket_id);
if (dp->tbl8_pool == NULL) {
rte_errno = ENOMEM;
+ rte_free(dp->def_nh);
rte_free(dp->tbl8);
rte_free(dp);
return NULL;
@@ -725,6 +800,7 @@ trie_free(void *p)
rte_rcu_qsbr_dq_delete(dp->dq);
rte_free(dp->tbl8_pool);
rte_free(dp->tbl8);
+ rte_free(dp->def_nh);
rte_free(dp);
}
diff --git a/lib/fib/trie.h b/lib/fib/trie.h
index c34cc2c057..ef9a1d50c6 100644
--- a/lib/fib/trie.h
+++ b/lib/fib/trie.h
@@ -9,6 +9,7 @@
#include <stdalign.h>
#include <rte_common.h>
+#include <rte_debug.h>
#include <rte_fib6.h>
/**
@@ -32,18 +33,19 @@
struct rte_trie_tbl {
uint32_t number_tbl8s; /**< Total number of tbl8s */
uint32_t rsvd_tbl8s; /**< Number of reserved tbl8s */
- uint32_t cur_tbl8s; /**< Current cumber of tbl8s */
- uint64_t def_nh; /**< Default next hop */
+ uint32_t cur_tbl8s; /**< Current number of tbl8s */
+ uint16_t num_vrfs; /**< Number of VRFs */
enum rte_fib_trie_nh_sz nh_sz; /**< Size of nexthop entry */
- uint64_t *tbl8; /**< tbl8 table. */
- uint32_t *tbl8_pool; /**< bitmap containing free tbl8 idxes*/
- uint32_t tbl8_pool_pos;
/* RCU config. */
enum rte_fib6_qsbr_mode rcu_mode; /**< Blocking, defer queue. */
struct rte_rcu_qsbr *v; /**< RCU QSBR variable. */
struct rte_rcu_qsbr_dq *dq; /**< RCU QSBR defer queue. */
+ uint64_t *def_nh; /**< Per-VRF default next hop array */
+ uint64_t *tbl8; /**< tbl8 table for all VRFs */
+ uint32_t *tbl8_pool; /**< bitmap containing free tbl8 idxes */
+ uint32_t tbl8_pool_pos;
/* tbl24 table. */
- alignas(RTE_CACHE_LINE_SIZE) uint64_t tbl24[];
+ alignas(RTE_CACHE_LINE_SIZE) uint64_t tbl24[];
};
static inline uint32_t
@@ -53,12 +55,15 @@ get_tbl24_idx(const struct rte_ipv6_addr *ip)
}
static inline void *
-get_tbl24_p(struct rte_trie_tbl *dp, const struct rte_ipv6_addr *ip, uint8_t nh_sz)
+get_tbl24_p(struct rte_trie_tbl *dp, uint16_t vrf_id,
+ const struct rte_ipv6_addr *ip, uint8_t nh_sz)
{
uint32_t tbl24_idx;
+ uint64_t base;
tbl24_idx = get_tbl24_idx(ip);
- return (void *)&((uint8_t *)dp->tbl24)[tbl24_idx << nh_sz];
+ base = (uint64_t)vrf_id * TRIE_TBL24_NUM_ENT;
+ return (void *)&((uint8_t *)dp->tbl24)[(base + tbl24_idx) << nh_sz];
}
static inline uint8_t
@@ -110,17 +115,26 @@ is_entry_extended(uint64_t ent)
return (ent & TRIE_EXT_ENT) == TRIE_EXT_ENT;
}
-#define LOOKUP_FUNC(suffix, type, nh_sz) \
+#define LOOKUP_FUNC(suffix, type, is_vrf) \
static inline void rte_trie_lookup_bulk_##suffix(void *p, \
- const struct rte_ipv6_addr *ips, \
+ const uint16_t *vrf_ids, const struct rte_ipv6_addr *ips, \
uint64_t *next_hops, const unsigned int n) \
-{ \
+{\
struct rte_trie_tbl *dp = (struct rte_trie_tbl *)p; \
uint64_t tmp; \
uint32_t i, j; \
+ uint32_t tbl24_idx; \
+ uint64_t base; \
+ \
+ if (!is_vrf) \
+ RTE_SET_USED(vrf_ids); \
\
for (i = 0; i < n; i++) { \
- tmp = ((type *)dp->tbl24)[get_tbl24_idx(&ips[i])]; \
+ uint16_t vrf_id = is_vrf ? vrf_ids[i] : 0; \
+ RTE_ASSERT(vrf_id < dp->num_vrfs); \
+ base = (uint64_t)vrf_id * TRIE_TBL24_NUM_ENT; \
+ tbl24_idx = get_tbl24_idx(&ips[i]); \
+ tmp = ((type *)dp->tbl24)[base + tbl24_idx]; \
j = 3; \
while (is_entry_extended(tmp)) { \
tmp = ((type *)dp->tbl8)[ips[i].a[j++] + \
@@ -129,9 +143,13 @@ static inline void rte_trie_lookup_bulk_##suffix(void *p, \
next_hops[i] = tmp >> 1; \
} \
}
-LOOKUP_FUNC(2b, uint16_t, 1)
-LOOKUP_FUNC(4b, uint32_t, 2)
-LOOKUP_FUNC(8b, uint64_t, 3)
+
+LOOKUP_FUNC(2b, uint16_t, false)
+LOOKUP_FUNC(4b, uint32_t, false)
+LOOKUP_FUNC(8b, uint64_t, false)
+LOOKUP_FUNC(vrf_2b, uint16_t, true)
+LOOKUP_FUNC(vrf_4b, uint32_t, true)
+LOOKUP_FUNC(vrf_8b, uint64_t, true)
void
trie_free(void *p);
@@ -144,7 +162,8 @@ rte_fib6_lookup_fn_t
trie_get_lookup_fn(void *p, enum rte_fib6_lookup_type type);
int
-trie_modify(struct rte_fib6 *fib, const struct rte_ipv6_addr *ip,
+trie_modify(struct rte_fib6 *fib, uint16_t vrf_id,
+ const struct rte_ipv6_addr *ip,
uint8_t depth, uint64_t next_hop, int op);
int
diff --git a/lib/fib/trie_avx512.c b/lib/fib/trie_avx512.c
index f49482a95d..19cd69e69c 100644
--- a/lib/fib/trie_avx512.c
+++ b/lib/fib/trie_avx512.c
@@ -8,6 +8,12 @@
#include "trie.h"
#include "trie_avx512.h"
+enum vrf_scale {
+ VRF_SCALE_SINGLE = 0,
+ VRF_SCALE_SMALL = 1,
+ VRF_SCALE_LARGE = 2,
+};
+
static __rte_always_inline void
transpose_x16(const struct rte_ipv6_addr *ips,
__m512i *first, __m512i *second, __m512i *third, __m512i *fourth)
@@ -67,8 +73,9 @@ transpose_x8(const struct rte_ipv6_addr *ips,
}
static __rte_always_inline void
-trie_vec_lookup_x16x2(void *p, const struct rte_ipv6_addr *ips,
- uint64_t *next_hops, int size)
+trie_vec_lookup_x16x2(void *p, const uint16_t *vrf_ids,
+ const struct rte_ipv6_addr *ips, uint64_t *next_hops, int size,
+ enum vrf_scale vrf_scale)
{
struct rte_trie_tbl *dp = (struct rte_trie_tbl *)p;
const __m512i zero = _mm512_set1_epi32(0);
@@ -79,6 +86,7 @@ trie_vec_lookup_x16x2(void *p, const struct rte_ipv6_addr *ips,
__m512i first_2, second_2, third_2, fourth_2;
__m512i idxes_1, res_1;
__m512i idxes_2, res_2;
+ __m512i vrf32_1, vrf32_2;
__m512i shuf_idxes;
__m512i tmp_1, tmp2_1, bytes_1, byte_chunk_1;
__m512i tmp_2, tmp2_2, bytes_2, byte_chunk_2;
@@ -109,6 +117,24 @@ trie_vec_lookup_x16x2(void *p, const struct rte_ipv6_addr *ips,
idxes_1 = _mm512_shuffle_epi8(first_1, bswap.z);
idxes_2 = _mm512_shuffle_epi8(first_2, bswap.z);
+ if (vrf_scale == VRF_SCALE_SINGLE) {
+ RTE_SET_USED(vrf_ids);
+ } else {
+ uint32_t j;
+
+ for (j = 0; j < 32; j++)
+ RTE_ASSERT(vrf_ids[j] < dp->num_vrfs);
+
+ vrf32_1 = _mm512_cvtepu16_epi32(
+ _mm256_loadu_si256((const void *)vrf_ids));
+ vrf32_2 = _mm512_cvtepu16_epi32(
+ _mm256_loadu_si256((const void *)(vrf_ids + 16)));
+ idxes_1 = _mm512_add_epi32(idxes_1,
+ _mm512_slli_epi32(vrf32_1, 24));
+ idxes_2 = _mm512_add_epi32(idxes_2,
+ _mm512_slli_epi32(vrf32_2, 24));
+ }
+
/**
* lookup in tbl24
* Put it inside branch to make compiller happy with -O0
@@ -213,13 +239,15 @@ trie_vec_lookup_x16x2(void *p, const struct rte_ipv6_addr *ips,
}
static void
-trie_vec_lookup_x8x2_8b(void *p, const struct rte_ipv6_addr *ips,
- uint64_t *next_hops)
+trie_vec_lookup_x8x2(void *p, const uint16_t *vrf_ids,
+ const struct rte_ipv6_addr *ips, uint64_t *next_hops, int size,
+ enum vrf_scale vrf_scale)
{
struct rte_trie_tbl *dp = (struct rte_trie_tbl *)p;
const __m512i zero = _mm512_set1_epi32(0);
const __m512i lsb = _mm512_set1_epi32(1);
const __m512i three_lsb = _mm512_set1_epi32(7);
+ __m512i res_msk;
/* IPv6 eight byte chunks */
__m512i first_1, second_1;
__m512i first_2, second_2;
@@ -228,6 +256,7 @@ trie_vec_lookup_x8x2_8b(void *p, const struct rte_ipv6_addr *ips,
__m512i shuf_idxes, base_idxes;
__m512i tmp_1, bytes_1, byte_chunk_1;
__m512i tmp_2, bytes_2, byte_chunk_2;
+ __m512i vrf64_1, vrf64_2;
const __rte_x86_zmm_t bswap = {
.u8 = { 2, 1, 0, 255, 255, 255, 255, 255,
10, 9, 8, 255, 255, 255, 255, 255,
@@ -244,6 +273,11 @@ trie_vec_lookup_x8x2_8b(void *p, const struct rte_ipv6_addr *ips,
__mmask8 msk_ext_1, new_msk_1;
__mmask8 msk_ext_2, new_msk_2;
+ if (size == sizeof(uint16_t))
+ res_msk = _mm512_set1_epi64(UINT16_MAX);
+ else if (size == sizeof(uint32_t))
+ res_msk = _mm512_set1_epi64(UINT32_MAX);
+
transpose_x8(ips, &first_1, &second_1);
transpose_x8(ips + 8, &first_2, &second_2);
@@ -251,9 +285,39 @@ trie_vec_lookup_x8x2_8b(void *p, const struct rte_ipv6_addr *ips,
idxes_1 = _mm512_shuffle_epi8(first_1, bswap.z);
idxes_2 = _mm512_shuffle_epi8(first_2, bswap.z);
+ if (vrf_scale == VRF_SCALE_SINGLE) {
+ RTE_SET_USED(vrf_ids);
+ } else {
+ uint32_t j;
+
+ for (j = 0; j < 16; j++)
+ RTE_ASSERT(vrf_ids[j] < dp->num_vrfs);
+
+ vrf64_1 = _mm512_cvtepu16_epi64(
+ _mm_loadu_si128((const void *)vrf_ids));
+ vrf64_2 = _mm512_cvtepu16_epi64(
+ _mm_loadu_si128((const void *)(vrf_ids + 8)));
+ idxes_1 = _mm512_add_epi64(idxes_1,
+ _mm512_slli_epi64(vrf64_1, 24));
+ idxes_2 = _mm512_add_epi64(idxes_2,
+ _mm512_slli_epi64(vrf64_2, 24));
+ }
+
/* lookup in tbl24 */
- res_1 = _mm512_i64gather_epi64(idxes_1, (const void *)dp->tbl24, 8);
- res_2 = _mm512_i64gather_epi64(idxes_2, (const void *)dp->tbl24, 8);
+ if (size == sizeof(uint16_t)) {
+ res_1 = _mm512_i64gather_epi64(idxes_1, (const void *)dp->tbl24, 2);
+ res_2 = _mm512_i64gather_epi64(idxes_2, (const void *)dp->tbl24, 2);
+ res_1 = _mm512_and_epi64(res_1, res_msk);
+ res_2 = _mm512_and_epi64(res_2, res_msk);
+ } else if (size == sizeof(uint32_t)) {
+ res_1 = _mm512_i64gather_epi64(idxes_1, (const void *)dp->tbl24, 4);
+ res_2 = _mm512_i64gather_epi64(idxes_2, (const void *)dp->tbl24, 4);
+ res_1 = _mm512_and_epi64(res_1, res_msk);
+ res_2 = _mm512_and_epi64(res_2, res_msk);
+ } else {
+ res_1 = _mm512_i64gather_epi64(idxes_1, (const void *)dp->tbl24, 8);
+ res_2 = _mm512_i64gather_epi64(idxes_2, (const void *)dp->tbl24, 8);
+ }
/* get extended entries indexes */
msk_ext_1 = _mm512_test_epi64_mask(res_1, lsb);
msk_ext_2 = _mm512_test_epi64_mask(res_2, lsb);
@@ -278,10 +342,26 @@ trie_vec_lookup_x8x2_8b(void *p, const struct rte_ipv6_addr *ips,
shuf_idxes);
idxes_1 = _mm512_maskz_add_epi64(msk_ext_1, idxes_1, bytes_1);
idxes_2 = _mm512_maskz_add_epi64(msk_ext_2, idxes_2, bytes_2);
- tmp_1 = _mm512_mask_i64gather_epi64(zero, msk_ext_1,
+ if (size == sizeof(uint16_t)) {
+ tmp_1 = _mm512_mask_i64gather_epi64(zero, msk_ext_1,
+ idxes_1, (const void *)dp->tbl8, 2);
+ tmp_2 = _mm512_mask_i64gather_epi64(zero, msk_ext_2,
+ idxes_2, (const void *)dp->tbl8, 2);
+ tmp_1 = _mm512_and_epi64(tmp_1, res_msk);
+ tmp_2 = _mm512_and_epi64(tmp_2, res_msk);
+ } else if (size == sizeof(uint32_t)) {
+ tmp_1 = _mm512_mask_i64gather_epi64(zero, msk_ext_1,
+ idxes_1, (const void *)dp->tbl8, 4);
+ tmp_2 = _mm512_mask_i64gather_epi64(zero, msk_ext_2,
+ idxes_2, (const void *)dp->tbl8, 4);
+ tmp_1 = _mm512_and_epi64(tmp_1, res_msk);
+ tmp_2 = _mm512_and_epi64(tmp_2, res_msk);
+ } else {
+ tmp_1 = _mm512_mask_i64gather_epi64(zero, msk_ext_1,
idxes_1, (const void *)dp->tbl8, 8);
- tmp_2 = _mm512_mask_i64gather_epi64(zero, msk_ext_2,
+ tmp_2 = _mm512_mask_i64gather_epi64(zero, msk_ext_2,
idxes_2, (const void *)dp->tbl8, 8);
+ }
new_msk_1 = _mm512_test_epi64_mask(tmp_1, lsb);
new_msk_2 = _mm512_test_epi64_mask(tmp_2, lsb);
res_1 = _mm512_mask_blend_epi64(msk_ext_1 ^ new_msk_1, res_1,
@@ -306,40 +386,145 @@ trie_vec_lookup_x8x2_8b(void *p, const struct rte_ipv6_addr *ips,
}
void
-rte_trie_vec_lookup_bulk_2b(void *p, const struct rte_ipv6_addr *ips,
+rte_trie_vec_lookup_bulk_2b(void *p, const uint16_t *vrf_ids,
+ const struct rte_ipv6_addr *ips,
uint64_t *next_hops, const unsigned int n)
{
uint32_t i;
+
for (i = 0; i < (n / 32); i++) {
- trie_vec_lookup_x16x2(p, &ips[i * 32],
- next_hops + i * 32, sizeof(uint16_t));
+ trie_vec_lookup_x16x2(p, vrf_ids + i * 32, &ips[i * 32],
+ next_hops + i * 32, sizeof(uint16_t),
+ VRF_SCALE_SINGLE);
}
- rte_trie_lookup_bulk_2b(p, &ips[i * 32],
+ rte_trie_lookup_bulk_2b(p, vrf_ids + i * 32, &ips[i * 32],
next_hops + i * 32, n - i * 32);
}
void
-rte_trie_vec_lookup_bulk_4b(void *p, const struct rte_ipv6_addr *ips,
+rte_trie_vec_lookup_bulk_vrf_2b(void *p, const uint16_t *vrf_ids,
+ const struct rte_ipv6_addr *ips,
uint64_t *next_hops, const unsigned int n)
{
uint32_t i;
+
for (i = 0; i < (n / 32); i++) {
- trie_vec_lookup_x16x2(p, &ips[i * 32],
- next_hops + i * 32, sizeof(uint32_t));
+ trie_vec_lookup_x16x2(p, vrf_ids + i * 32, &ips[i * 32],
+ next_hops + i * 32, sizeof(uint16_t),
+ VRF_SCALE_SMALL);
}
- rte_trie_lookup_bulk_4b(p, &ips[i * 32],
+ rte_trie_lookup_bulk_vrf_2b(p, vrf_ids + i * 32, &ips[i * 32],
next_hops + i * 32, n - i * 32);
}
void
-rte_trie_vec_lookup_bulk_8b(void *p, const struct rte_ipv6_addr *ips,
+rte_trie_vec_lookup_bulk_vrf_2b_large(void *p, const uint16_t *vrf_ids,
+ const struct rte_ipv6_addr *ips,
uint64_t *next_hops, const unsigned int n)
{
uint32_t i;
+
+ for (i = 0; i < (n / 16); i++) {
+ trie_vec_lookup_x8x2(p, vrf_ids + i * 16, &ips[i * 16],
+ next_hops + i * 16, sizeof(uint16_t),
+ VRF_SCALE_LARGE);
+ }
+ rte_trie_lookup_bulk_vrf_2b(p, vrf_ids + i * 16, &ips[i * 16],
+ next_hops + i * 16, n - i * 16);
+}
+
+void
+rte_trie_vec_lookup_bulk_4b(void *p, const uint16_t *vrf_ids,
+ const struct rte_ipv6_addr *ips,
+ uint64_t *next_hops, const unsigned int n)
+{
+ uint32_t i;
+
+ for (i = 0; i < (n / 32); i++) {
+ trie_vec_lookup_x16x2(p, vrf_ids + i * 32, &ips[i * 32],
+ next_hops + i * 32, sizeof(uint32_t),
+ VRF_SCALE_SINGLE);
+ }
+ rte_trie_lookup_bulk_4b(p, vrf_ids + i * 32, &ips[i * 32],
+ next_hops + i * 32, n - i * 32);
+}
+
+void
+rte_trie_vec_lookup_bulk_vrf_4b(void *p, const uint16_t *vrf_ids,
+ const struct rte_ipv6_addr *ips,
+ uint64_t *next_hops, const unsigned int n)
+{
+ uint32_t i;
+
+ for (i = 0; i < (n / 32); i++) {
+ trie_vec_lookup_x16x2(p, vrf_ids + i * 32, &ips[i * 32],
+ next_hops + i * 32, sizeof(uint32_t),
+ VRF_SCALE_SMALL);
+ }
+ rte_trie_lookup_bulk_vrf_4b(p, vrf_ids + i * 32, &ips[i * 32],
+ next_hops + i * 32, n - i * 32);
+}
+
+void
+rte_trie_vec_lookup_bulk_vrf_4b_large(void *p, const uint16_t *vrf_ids,
+ const struct rte_ipv6_addr *ips,
+ uint64_t *next_hops, const unsigned int n)
+{
+ uint32_t i;
+
+ for (i = 0; i < (n / 16); i++) {
+ trie_vec_lookup_x8x2(p, vrf_ids + i * 16, &ips[i * 16],
+ next_hops + i * 16, sizeof(uint32_t),
+ VRF_SCALE_LARGE);
+ }
+ rte_trie_lookup_bulk_vrf_4b(p, vrf_ids + i * 16, &ips[i * 16],
+ next_hops + i * 16, n - i * 16);
+}
+
+void
+rte_trie_vec_lookup_bulk_8b(void *p, const uint16_t *vrf_ids,
+ const struct rte_ipv6_addr *ips,
+ uint64_t *next_hops, const unsigned int n)
+{
+ uint32_t i;
+
+ for (i = 0; i < (n / 16); i++) {
+ trie_vec_lookup_x8x2(p, vrf_ids + i * 16, &ips[i * 16],
+ next_hops + i * 16, sizeof(uint64_t),
+ VRF_SCALE_SINGLE);
+ }
+ rte_trie_lookup_bulk_8b(p, vrf_ids + i * 16, &ips[i * 16],
+ next_hops + i * 16, n - i * 16);
+}
+
+void
+rte_trie_vec_lookup_bulk_vrf_8b(void *p, const uint16_t *vrf_ids,
+ const struct rte_ipv6_addr *ips,
+ uint64_t *next_hops, const unsigned int n)
+{
+ uint32_t i;
+
+ for (i = 0; i < (n / 16); i++) {
+ trie_vec_lookup_x8x2(p, vrf_ids + i * 16, &ips[i * 16],
+ next_hops + i * 16, sizeof(uint64_t),
+ VRF_SCALE_SMALL);
+ }
+ rte_trie_lookup_bulk_vrf_8b(p, vrf_ids + i * 16, &ips[i * 16],
+ next_hops + i * 16, n - i * 16);
+}
+
+void
+rte_trie_vec_lookup_bulk_vrf_8b_large(void *p, const uint16_t *vrf_ids,
+ const struct rte_ipv6_addr *ips,
+ uint64_t *next_hops, const unsigned int n)
+{
+ uint32_t i;
+
for (i = 0; i < (n / 16); i++) {
- trie_vec_lookup_x8x2_8b(p, &ips[i * 16],
- next_hops + i * 16);
+ trie_vec_lookup_x8x2(p, vrf_ids + i * 16, &ips[i * 16],
+ next_hops + i * 16, sizeof(uint64_t),
+ VRF_SCALE_LARGE);
}
- rte_trie_lookup_bulk_8b(p, &ips[i * 16],
+ rte_trie_lookup_bulk_vrf_8b(p, vrf_ids + i * 16, &ips[i * 16],
next_hops + i * 16, n - i * 16);
}
diff --git a/lib/fib/trie_avx512.h b/lib/fib/trie_avx512.h
index 1028a4899f..190a5c5aa4 100644
--- a/lib/fib/trie_avx512.h
+++ b/lib/fib/trie_avx512.h
@@ -10,15 +10,48 @@
struct rte_ipv6_addr;
void
-rte_trie_vec_lookup_bulk_2b(void *p, const struct rte_ipv6_addr *ips,
+rte_trie_vec_lookup_bulk_2b(void *p, const uint16_t *vrf_ids,
+ const struct rte_ipv6_addr *ips,
uint64_t *next_hops, const unsigned int n);
void
-rte_trie_vec_lookup_bulk_4b(void *p, const struct rte_ipv6_addr *ips,
+rte_trie_vec_lookup_bulk_vrf_2b(void *p, const uint16_t *vrf_ids,
+ const struct rte_ipv6_addr *ips,
uint64_t *next_hops, const unsigned int n);
void
-rte_trie_vec_lookup_bulk_8b(void *p, const struct rte_ipv6_addr *ips,
+rte_trie_vec_lookup_bulk_vrf_2b_large(void *p, const uint16_t *vrf_ids,
+ const struct rte_ipv6_addr *ips,
+ uint64_t *next_hops, const unsigned int n);
+
+void
+rte_trie_vec_lookup_bulk_4b(void *p, const uint16_t *vrf_ids,
+ const struct rte_ipv6_addr *ips,
+ uint64_t *next_hops, const unsigned int n);
+
+void
+rte_trie_vec_lookup_bulk_vrf_4b(void *p, const uint16_t *vrf_ids,
+ const struct rte_ipv6_addr *ips,
+ uint64_t *next_hops, const unsigned int n);
+
+void
+rte_trie_vec_lookup_bulk_vrf_4b_large(void *p, const uint16_t *vrf_ids,
+ const struct rte_ipv6_addr *ips,
+ uint64_t *next_hops, const unsigned int n);
+
+void
+rte_trie_vec_lookup_bulk_8b(void *p, const uint16_t *vrf_ids,
+ const struct rte_ipv6_addr *ips,
+ uint64_t *next_hops, const unsigned int n);
+
+void
+rte_trie_vec_lookup_bulk_vrf_8b(void *p, const uint16_t *vrf_ids,
+ const struct rte_ipv6_addr *ips,
+ uint64_t *next_hops, const unsigned int n);
+
+void
+rte_trie_vec_lookup_bulk_vrf_8b_large(void *p, const uint16_t *vrf_ids,
+ const struct rte_ipv6_addr *ips,
uint64_t *next_hops, const unsigned int n);
#endif /* _TRIE_AVX512_H_ */
--
2.43.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [RFC PATCH 4/4] fib6: add VRF functional and unit tests
2026-03-22 15:42 [RFC PATCH 0/4] VRF support in FIB library Vladimir Medvedkin
` (2 preceding siblings ...)
2026-03-22 15:42 ` [RFC PATCH 3/4] fib6: add multi-VRF support Vladimir Medvedkin
@ 2026-03-22 15:42 ` Vladimir Medvedkin
2026-03-22 16:45 ` Stephen Hemminger
2026-03-22 16:43 ` [RFC PATCH 0/4] VRF support in FIB library Stephen Hemminger
` (4 subsequent siblings)
8 siblings, 1 reply; 45+ messages in thread
From: Vladimir Medvedkin @ 2026-03-22 15:42 UTC (permalink / raw)
To: dev; +Cc: rjarry, nsaxena16, mb, adwivedi, jerinjacobk
Add test coverage for the multi-VRF IPv6 FIB API.
Signed-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
---
app/test-fib/main.c | 92 +++++++++++--
app/test/test_fib6.c | 319 ++++++++++++++++++++++++++++++++++++++++++-
2 files changed, 395 insertions(+), 16 deletions(-)
diff --git a/app/test-fib/main.c b/app/test-fib/main.c
index 5593fdd47e..0feac72b87 100644
--- a/app/test-fib/main.c
+++ b/app/test-fib/main.c
@@ -717,7 +717,7 @@ check_config(void)
* get_vrf_bits(nb_vrfs) must be strictly less than
* the total nexthop width.
*/
- if ((config.nb_vrfs > 1) && !(config.flags & IPV6_FLAG)) {
+ if (config.nb_vrfs > 1) {
uint8_t nh_sz = rte_ctz32(config.ent_sz);
uint8_t vrf_bits = get_vrf_bits(config.nb_vrfs);
/* - 2 to leave at least 1 bit for nexthop and 1 bit for ext_ent flag */
@@ -1165,6 +1165,7 @@ run_v6(void)
{
uint64_t start, acc;
uint64_t def_nh = 0;
+ uint8_t nh_sz = rte_ctz32(config.ent_sz);
struct rte_fib6 *fib;
struct rte_fib6_conf conf = {0};
struct rt_rule_6 *rt;
@@ -1175,6 +1176,7 @@ run_v6(void)
struct rte_ipv6_addr *tbl6;
uint64_t fib_nh[BURST_SZ];
int32_t lpm_nh[BURST_SZ];
+ uint16_t *vrf_ids = NULL;
rt = (struct rt_rule_6 *)config.rt;
tbl6 = config.lookup_tbl;
@@ -1189,16 +1191,38 @@ run_v6(void)
return ret;
}
+ /* Allocate VRF IDs array for lookups if using multiple VRFs */
+ if (config.nb_vrfs > 1) {
+ vrf_ids = rte_malloc(NULL, sizeof(uint16_t) * config.nb_lookup_ips, 0);
+ if (vrf_ids == NULL) {
+ printf("Can not alloc VRF IDs array\n");
+ return -ENOMEM;
+ }
+ /* Generate random VRF IDs for each lookup */
+ for (i = 0; i < config.nb_lookup_ips; i++)
+ vrf_ids[i] = rte_rand() % config.nb_vrfs;
+ }
+
conf.type = get_fib_type();
conf.default_nh = def_nh;
conf.max_routes = config.nb_routes * 2;
conf.rib_ext_sz = 0;
+ conf.max_vrfs = config.nb_vrfs;
+ conf.vrf_default_nh = NULL;
if (conf.type == RTE_FIB6_TRIE) {
conf.trie.nh_sz = rte_ctz32(config.ent_sz);
conf.trie.num_tbl8 = RTE_MIN(config.tbl8,
get_max_nh(conf.trie.nh_sz));
}
+ conf.vrf_default_nh = rte_malloc(NULL, conf.max_vrfs * sizeof(uint64_t), 0);
+ if (conf.vrf_default_nh == NULL) {
+ printf("Can not alloc VRF default nexthops array\n");
+ return -ENOMEM;
+ }
+ for (i = 0; i < conf.max_vrfs; i++)
+ conf.vrf_default_nh[i] = encode_vrf_nh(i, def_nh, nh_sz);
+
fib = rte_fib6_create("test", -1, &conf);
if (fib == NULL) {
printf("Can not alloc FIB, err %d\n", rte_errno);
@@ -1223,12 +1247,28 @@ run_v6(void)
for (k = config.print_fract, i = 0; k > 0; k--) {
start = rte_rdtsc_precise();
for (j = 0; j < (config.nb_routes - i) / k; j++) {
- ret = rte_fib6_add(fib, &rt[i + j].addr,
- rt[i + j].depth, rt[i + j].nh);
- if (unlikely(ret != 0)) {
- printf("Can not add a route to FIB, err %d\n",
- ret);
- return -ret;
+ uint32_t idx = i + j;
+ if (config.nb_vrfs > 1) {
+ uint16_t vrf_id;
+ for (vrf_id = 0; vrf_id < config.nb_vrfs; vrf_id++) {
+ uint64_t nh = encode_vrf_nh(vrf_id, rt[idx].nh,
+ nh_sz);
+ ret = rte_fib6_vrf_add(fib, vrf_id, &rt[idx].addr,
+ rt[idx].depth, nh);
+ if (unlikely(ret != 0)) {
+ printf("Can not add a route to FIB, err %d\n",
+ ret);
+ return -ret;
+ }
+ }
+ } else {
+ ret = rte_fib6_add(fib, &rt[idx].addr,
+ rt[idx].depth, rt[idx].nh);
+ if (unlikely(ret != 0)) {
+ printf("Can not add a route to FIB, err %d\n",
+ ret);
+ return -ret;
+ }
}
}
printf("AVG FIB add %"PRIu64"\n",
@@ -1268,15 +1308,33 @@ run_v6(void)
acc = 0;
for (i = 0; i < config.nb_lookup_ips; i += BURST_SZ) {
start = rte_rdtsc_precise();
- ret = rte_fib6_lookup_bulk(fib, &tbl6[i],
- fib_nh, BURST_SZ);
+ if (config.nb_vrfs > 1)
+ ret = rte_fib6_vrf_lookup_bulk(fib, vrf_ids + i,
+ &tbl6[i], fib_nh, BURST_SZ);
+ else
+ ret = rte_fib6_lookup_bulk(fib, &tbl6[i],
+ fib_nh, BURST_SZ);
acc += rte_rdtsc_precise() - start;
if (ret != 0) {
printf("FIB lookup fails, err %d\n", ret);
return -ret;
}
+ /* Validate VRF IDs in returned nexthops */
+ if (config.nb_vrfs > 1) {
+ for (j = 0; j < BURST_SZ; j++) {
+ uint16_t returned_vrf = decode_vrf_nh(fib_nh[j],
+ nh_sz);
+ if (returned_vrf != vrf_ids[i + j]) {
+ printf("VRF validation failed: expected VRF %u, got %u\n",
+ vrf_ids[i + j], returned_vrf);
+ return -1;
+ }
+ }
+ }
}
printf("AVG FIB lookup %.1f\n", (double)acc / (double)i);
+ if (config.nb_vrfs > 1)
+ printf("VRF validation passed\n");
if (config.flags & CMP_FLAG) {
acc = 0;
@@ -1314,8 +1372,17 @@ run_v6(void)
for (k = config.print_fract, i = 0; k > 0; k--) {
start = rte_rdtsc_precise();
- for (j = 0; j < (config.nb_routes - i) / k; j++)
- rte_fib6_delete(fib, &rt[i + j].addr, rt[i + j].depth);
+ for (j = 0; j < (config.nb_routes - i) / k; j++) {
+ uint32_t idx = i + j;
+ if (config.nb_vrfs > 1) {
+ uint16_t vrf_id;
+ for (vrf_id = 0; vrf_id < config.nb_vrfs; vrf_id++)
+ rte_fib6_vrf_delete(fib, vrf_id, &rt[idx].addr,
+ rt[idx].depth);
+ } else {
+ rte_fib6_delete(fib, &rt[idx].addr, rt[idx].depth);
+ }
+ }
printf("AVG FIB delete %"PRIu64"\n",
(rte_rdtsc_precise() - start) / j);
@@ -1334,6 +1401,9 @@ run_v6(void)
i += j;
}
}
+
+ if (vrf_ids != NULL)
+ rte_free(vrf_ids);
return 0;
}
diff --git a/app/test/test_fib6.c b/app/test/test_fib6.c
index fffb590dbf..1143a338e6 100644
--- a/app/test/test_fib6.c
+++ b/app/test/test_fib6.c
@@ -6,6 +6,7 @@
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
+#include <inttypes.h>
#include <rte_memory.h>
#include <rte_log.h>
@@ -25,6 +26,11 @@ static int32_t test_get_invalid(void);
static int32_t test_lookup(void);
static int32_t test_invalid_rcu(void);
static int32_t test_fib_rcu_sync_rw(void);
+static int32_t test_create_vrf(void);
+static int32_t test_vrf_add_del(void);
+static int32_t test_vrf_lookup(void);
+static int32_t test_vrf_isolation(void);
+static int32_t test_vrf_all_nh_sizes(void);
#define MAX_ROUTES (1 << 16)
/** Maximum number of tbl8 for 2-byte entries */
@@ -38,7 +44,7 @@ int32_t
test_create_invalid(void)
{
struct rte_fib6 *fib = NULL;
- struct rte_fib6_conf config;
+ struct rte_fib6_conf config = { 0 };
config.max_routes = MAX_ROUTES;
config.rib_ext_sz = 0;
@@ -97,7 +103,7 @@ int32_t
test_multiple_create(void)
{
struct rte_fib6 *fib = NULL;
- struct rte_fib6_conf config;
+ struct rte_fib6_conf config = { 0 };
int32_t i;
config.rib_ext_sz = 0;
@@ -124,7 +130,7 @@ int32_t
test_free_null(void)
{
struct rte_fib6 *fib = NULL;
- struct rte_fib6_conf config;
+ struct rte_fib6_conf config = { 0 };
config.max_routes = MAX_ROUTES;
config.rib_ext_sz = 0;
@@ -148,7 +154,7 @@ int32_t
test_add_del_invalid(void)
{
struct rte_fib6 *fib = NULL;
- struct rte_fib6_conf config;
+ struct rte_fib6_conf config = { 0 };
uint64_t nh = 100;
struct rte_ipv6_addr ip = RTE_IPV6_ADDR_UNSPEC;
int ret;
@@ -342,7 +348,7 @@ int32_t
test_lookup(void)
{
struct rte_fib6 *fib = NULL;
- struct rte_fib6_conf config;
+ struct rte_fib6_conf config = { 0 };
uint64_t def_nh = 100;
int ret;
@@ -599,6 +605,304 @@ test_fib_rcu_sync_rw(void)
return status == 0 ? TEST_SUCCESS : TEST_FAILED;
}
+/*
+ * Test VRF creation and basic operations
+ */
+static int32_t
+test_create_vrf(void)
+{
+ struct rte_fib6 *fib = NULL;
+ struct rte_fib6_conf config = { 0 };
+ uint64_t def_nh = 100;
+ uint64_t vrf_def_nh[4] = {100, 200, 300, 400};
+
+ config.max_routes = MAX_ROUTES;
+ config.rib_ext_sz = 0;
+ config.default_nh = def_nh;
+ config.type = RTE_FIB6_TRIE;
+ config.trie.nh_sz = RTE_FIB6_TRIE_4B;
+ config.trie.num_tbl8 = MAX_TBL8;
+
+ /* Test single VRF (backward compat) */
+ config.max_vrfs = 0;
+ config.vrf_default_nh = NULL;
+ fib = rte_fib6_create(__func__, SOCKET_ID_ANY, &config);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create FIB with max_vrfs=0\n");
+ rte_fib6_free(fib);
+
+ /* Test single VRF explicitly */
+ config.max_vrfs = 1;
+ fib = rte_fib6_create(__func__, SOCKET_ID_ANY, &config);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create FIB with max_vrfs=1\n");
+ rte_fib6_free(fib);
+
+ /* Test multi-VRF with per-VRF defaults */
+ config.max_vrfs = 4;
+ config.vrf_default_nh = vrf_def_nh;
+ fib = rte_fib6_create(__func__, SOCKET_ID_ANY, &config);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create FIB with max_vrfs=4\n");
+ rte_fib6_free(fib);
+
+ return TEST_SUCCESS;
+}
+
+/*
+ * Test VRF route add/delete operations
+ */
+static int32_t
+test_vrf_add_del(void)
+{
+ struct rte_fib6 *fib = NULL;
+ struct rte_fib6_conf config = { 0 };
+ uint64_t def_nh = 100;
+ uint64_t vrf_def_nh[4] = {100, 200, 300, 400};
+ struct rte_ipv6_addr ip = RTE_IPV6(0x2001, 0, 0, 0, 0, 0, 0, 0);
+ uint8_t depth = 64;
+ uint64_t nh = 1000;
+ int ret;
+
+ config.max_routes = MAX_ROUTES;
+ config.rib_ext_sz = 0;
+ config.default_nh = def_nh;
+ config.type = RTE_FIB6_TRIE;
+ config.trie.nh_sz = RTE_FIB6_TRIE_4B;
+ config.trie.num_tbl8 = MAX_TBL8;
+ config.max_vrfs = 4;
+ config.vrf_default_nh = vrf_def_nh;
+
+ fib = rte_fib6_create(__func__, SOCKET_ID_ANY, &config);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create FIB\n");
+
+ /* Add route to VRF 0 */
+ ret = rte_fib6_vrf_add(fib, 0, &ip, depth, nh);
+ RTE_TEST_ASSERT(ret == 0, "Failed to add route to VRF 0\n");
+
+ /* Add route to VRF 1 with different nexthop */
+ ret = rte_fib6_vrf_add(fib, 1, &ip, depth, nh + 1);
+ RTE_TEST_ASSERT(ret == 0, "Failed to add route to VRF 1\n");
+
+ /* Add route to VRF 2 */
+ ret = rte_fib6_vrf_add(fib, 2, &ip, depth, nh + 2);
+ RTE_TEST_ASSERT(ret == 0, "Failed to add route to VRF 2\n");
+
+ /* Test invalid VRF ID */
+ ret = rte_fib6_vrf_add(fib, 10, &ip, depth, nh);
+ RTE_TEST_ASSERT(ret != 0, "Should fail with invalid VRF ID\n");
+
+ /* Delete route from VRF 1 */
+ ret = rte_fib6_vrf_delete(fib, 1, &ip, depth);
+ RTE_TEST_ASSERT(ret == 0, "Failed to delete route from VRF 1\n");
+
+ /* Delete non-existent route - implementation may return error */
+ ret = rte_fib6_vrf_delete(fib, 3, &ip, depth);
+ (void)ret; /* Accept any return value */
+
+ rte_fib6_free(fib);
+ return TEST_SUCCESS;
+}
+
+/*
+ * Test VRF lookup functionality
+ */
+static int32_t
+test_vrf_lookup(void)
+{
+ struct rte_fib6 *fib = NULL;
+ struct rte_fib6_conf config = { 0 };
+ uint64_t def_nh = 100;
+ uint64_t vrf_def_nh[4] = {1000, 2000, 3000, 4000};
+ struct rte_ipv6_addr ip_base = RTE_IPV6(0x2001, 0, 0, 0, 0, 0, 0, 0);
+ uint16_t vrf_ids[4];
+ struct rte_ipv6_addr ips[4];
+ uint64_t next_hops[4];
+ int ret;
+ uint32_t i;
+
+ config.max_routes = MAX_ROUTES;
+ config.rib_ext_sz = 0;
+ config.default_nh = def_nh;
+ config.type = RTE_FIB6_TRIE;
+ config.trie.nh_sz = RTE_FIB6_TRIE_4B;
+ config.trie.num_tbl8 = MAX_TBL8;
+ config.max_vrfs = 4;
+ config.vrf_default_nh = vrf_def_nh;
+
+ fib = rte_fib6_create(__func__, SOCKET_ID_ANY, &config);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create FIB\n");
+
+ /* Add routes to different VRFs with VRF-specific nexthops */
+ for (i = 0; i < 4; i++) {
+ struct rte_ipv6_addr ip = ip_base;
+ ip.a[1] = (uint8_t)i;
+ ret = rte_fib6_vrf_add(fib, i, &ip, 64, 100 + i);
+ RTE_TEST_ASSERT(ret == 0, "Failed to add route to VRF %u\n", i);
+ }
+
+ /* Prepare lookup: each IP should match its VRF-specific route */
+ for (i = 0; i < 4; i++) {
+ vrf_ids[i] = i;
+ ips[i] = ip_base;
+ ips[i].a[1] = (uint8_t)i;
+ ips[i].a[15] = 0x34; /* within /64 */
+ }
+
+ /* Lookup should return VRF-specific nexthops */
+ ret = rte_fib6_vrf_lookup_bulk(fib, vrf_ids, ips, next_hops, 4);
+ RTE_TEST_ASSERT(ret == 0, "VRF lookup failed\n");
+
+ for (i = 0; i < 4; i++) {
+ RTE_TEST_ASSERT(next_hops[i] == 100 + i,
+ "Wrong nexthop for VRF %u: expected %"PRIu64", got %"PRIu64"\n",
+ i, (uint64_t)(100 + i), next_hops[i]);
+ }
+
+ /* Test default nexthops for unmatched IPs */
+ {
+ struct rte_ipv6_addr ip_unmatch = RTE_IPV6(0x3001, 0, 0, 0, 0, 0, 0, 1);
+ for (i = 0; i < 4; i++) {
+ vrf_ids[i] = i;
+ ips[i] = ip_unmatch;
+ }
+ }
+
+ ret = rte_fib6_vrf_lookup_bulk(fib, vrf_ids, ips, next_hops, 4);
+ RTE_TEST_ASSERT(ret == 0, "VRF lookup failed\n");
+
+ for (i = 0; i < 4; i++) {
+ RTE_TEST_ASSERT(next_hops[i] == vrf_def_nh[i],
+ "Wrong default nexthop for VRF %u: expected %"PRIu64", got %"PRIu64"\n",
+ i, vrf_def_nh[i], next_hops[i]);
+ }
+
+ rte_fib6_free(fib);
+ return TEST_SUCCESS;
+}
+
+/*
+ * Test VRF isolation - routes in one VRF shouldn't affect others
+ */
+static int32_t
+test_vrf_isolation(void)
+{
+ struct rte_fib6 *fib = NULL;
+ struct rte_fib6_conf config = { 0 };
+ uint64_t vrf_def_nh[3] = {100, 200, 300};
+ struct rte_ipv6_addr ip = RTE_IPV6(0x2001, 0, 0, 0, 0, 0, 0, 0);
+ uint16_t vrf_ids[3] = {0, 1, 2};
+ struct rte_ipv6_addr ips[3];
+ uint64_t next_hops[3];
+ int ret;
+ uint32_t i;
+
+ config.max_routes = MAX_ROUTES;
+ config.rib_ext_sz = 0;
+ config.default_nh = 0;
+ config.type = RTE_FIB6_TRIE;
+ config.trie.nh_sz = RTE_FIB6_TRIE_4B;
+ config.trie.num_tbl8 = MAX_TBL8;
+ config.max_vrfs = 3;
+ config.vrf_default_nh = vrf_def_nh;
+
+ fib = rte_fib6_create("test_vrf6_isol", SOCKET_ID_ANY, &config);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create FIB\n");
+
+ /* Add route only to VRF 1 */
+ ret = rte_fib6_vrf_add(fib, 1, &ip, 64, 777);
+ RTE_TEST_ASSERT(ret == 0, "Failed to add route to VRF 1\n");
+
+ /* Lookup same IP in all three VRFs */
+ for (i = 0; i < 3; i++) {
+ ips[i] = ip;
+ ips[i].a[15] = 0x22; /* within /64 */
+ }
+
+ ret = rte_fib6_vrf_lookup_bulk(fib, vrf_ids, ips, next_hops, 3);
+ RTE_TEST_ASSERT(ret == 0, "VRF lookup failed\n");
+
+ /* VRF 0 should get default */
+ RTE_TEST_ASSERT(next_hops[0] == vrf_def_nh[0],
+ "VRF 0 should return default nexthop\n");
+
+ /* VRF 1 should get the route */
+ RTE_TEST_ASSERT(next_hops[1] == 777,
+ "VRF 1 should return route nexthop 777, got %"PRIu64"\n", next_hops[1]);
+
+ /* VRF 2 should get default */
+ RTE_TEST_ASSERT(next_hops[2] == vrf_def_nh[2],
+ "VRF 2 should return default nexthop\n");
+
+ rte_fib6_free(fib);
+ return TEST_SUCCESS;
+}
+
+/*
+ * Test multi-VRF with all nexthop sizes
+ */
+static int32_t
+test_vrf_all_nh_sizes(void)
+{
+ struct rte_fib6 *fib = NULL;
+ struct rte_fib6_conf config = { 0 };
+ uint64_t vrf_def_nh[2] = {10, 20};
+ struct rte_ipv6_addr ip = RTE_IPV6(0x2001, 0, 0, 0, 0, 0, 0, 0);
+ uint16_t vrf_ids[2] = {0, 1};
+ struct rte_ipv6_addr ips[2];
+ uint64_t next_hops[2];
+ int ret;
+ enum rte_fib_trie_nh_sz nh_sizes[] = {
+ RTE_FIB6_TRIE_2B,
+ RTE_FIB6_TRIE_4B,
+ RTE_FIB6_TRIE_8B
+ };
+ uint64_t max_nhs[] = {32767, 2147483647ULL, 9223372036854775807ULL};
+ int i;
+
+ config.max_routes = MAX_ROUTES;
+ config.rib_ext_sz = 0;
+ config.default_nh = 0;
+ config.type = RTE_FIB6_TRIE;
+ config.trie.num_tbl8 = MAX_TBL8 - 1;
+ config.max_vrfs = 2;
+ config.vrf_default_nh = vrf_def_nh;
+
+ for (i = 0; i < (int)RTE_DIM(nh_sizes); i++) {
+ char name[32];
+ config.trie.nh_sz = nh_sizes[i];
+ snprintf(name, sizeof(name), "vrf6_nh%d", i);
+
+ fib = rte_fib6_create(name, SOCKET_ID_ANY, &config);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create FIB\n");
+
+ /* Add routes with max nexthop for this size */
+ ret = rte_fib6_vrf_add(fib, 0, &ip, 64, max_nhs[i]);
+ RTE_TEST_ASSERT(ret == 0,
+ "Failed to add route to VRF 0 with nh_sz=%d\n", nh_sizes[i]);
+
+ ret = rte_fib6_vrf_add(fib, 1, &ip, 64, max_nhs[i] - 1);
+ RTE_TEST_ASSERT(ret == 0,
+ "Failed to add route to VRF 1 with nh_sz=%d\n", nh_sizes[i]);
+
+ /* Lookup */
+ ips[0] = ip;
+ ips[1] = ip;
+ ips[0].a[15] = 0x11;
+ ips[1].a[15] = 0x22;
+
+ ret = rte_fib6_vrf_lookup_bulk(fib, vrf_ids, ips, next_hops, 2);
+ RTE_TEST_ASSERT(ret == 0, "VRF lookup failed with nh_sz=%d\n", nh_sizes[i]);
+
+ RTE_TEST_ASSERT(next_hops[0] == max_nhs[i],
+ "Wrong nexthop for VRF 0 with nh_sz=%d\n", nh_sizes[i]);
+ RTE_TEST_ASSERT(next_hops[1] == max_nhs[i] - 1,
+ "Wrong nexthop for VRF 1 with nh_sz=%d\n", nh_sizes[i]);
+
+ rte_fib6_free(fib);
+ fib = NULL;
+ }
+
+ return TEST_SUCCESS;
+}
+
static struct unit_test_suite fib6_fast_tests = {
.suite_name = "fib6 autotest",
.setup = NULL,
@@ -611,6 +915,11 @@ static struct unit_test_suite fib6_fast_tests = {
TEST_CASE(test_lookup),
TEST_CASE(test_invalid_rcu),
TEST_CASE(test_fib_rcu_sync_rw),
+ TEST_CASE(test_create_vrf),
+ TEST_CASE(test_vrf_add_del),
+ TEST_CASE(test_vrf_lookup),
+ TEST_CASE(test_vrf_isolation),
+ TEST_CASE(test_vrf_all_nh_sizes),
TEST_CASES_END()
}
};
--
2.43.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* Re: [RFC PATCH 2/4] fib: add VRF functional and unit tests
2026-03-22 15:42 ` [RFC PATCH 2/4] fib: add VRF functional and unit tests Vladimir Medvedkin
@ 2026-03-22 16:40 ` Stephen Hemminger
2026-03-22 16:41 ` Stephen Hemminger
1 sibling, 0 replies; 45+ messages in thread
From: Stephen Hemminger @ 2026-03-22 16:40 UTC (permalink / raw)
To: Vladimir Medvedkin; +Cc: dev, rjarry, nsaxena16, mb, adwivedi, jerinjacobk
On Sun, 22 Mar 2026 15:42:13 +0000
Vladimir Medvedkin <vladimir.medvedkin@intel.com> wrote:
> +/* Get number of bits needed to encode VRF IDs */
> +static __rte_always_inline __rte_pure uint8_t
> +get_vrf_bits(uint32_t nb_vrfs)
This is test code, no need for __rte_always_inline.
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH 2/4] fib: add VRF functional and unit tests
2026-03-22 15:42 ` [RFC PATCH 2/4] fib: add VRF functional and unit tests Vladimir Medvedkin
2026-03-22 16:40 ` Stephen Hemminger
@ 2026-03-22 16:41 ` Stephen Hemminger
1 sibling, 0 replies; 45+ messages in thread
From: Stephen Hemminger @ 2026-03-22 16:41 UTC (permalink / raw)
To: Vladimir Medvedkin; +Cc: dev, rjarry, nsaxena16, mb, adwivedi, jerinjacobk
On Sun, 22 Mar 2026 15:42:13 +0000
Vladimir Medvedkin <vladimir.medvedkin@intel.com> wrote:
> + case 'V':
> + errno = 0;
> + config.nb_vrfs = strtoul(optarg, &endptr, 10);
> + /* VRF IDs are uint16_t, max valid VRF is 65535 */
> + if ((errno != 0) || (config.nb_vrfs == 0) ||
> + (config.nb_vrfs > UINT16_MAX)) {
> + print_usage();
Should also check endptr. Right now "-V1xx" would be accepted.
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH 0/4] VRF support in FIB library
2026-03-22 15:42 [RFC PATCH 0/4] VRF support in FIB library Vladimir Medvedkin
` (3 preceding siblings ...)
2026-03-22 15:42 ` [RFC PATCH 4/4] fib6: add VRF functional and unit tests Vladimir Medvedkin
@ 2026-03-22 16:43 ` Stephen Hemminger
2026-03-23 9:01 ` Morten Brørup
2026-03-23 11:16 ` Medvedkin, Vladimir
2026-03-23 9:54 ` Robin Jarry
` (3 subsequent siblings)
8 siblings, 2 replies; 45+ messages in thread
From: Stephen Hemminger @ 2026-03-22 16:43 UTC (permalink / raw)
To: Vladimir Medvedkin; +Cc: dev, rjarry, nsaxena16, mb, adwivedi, jerinjacobk
On Sun, 22 Mar 2026 15:42:11 +0000
Vladimir Medvedkin <vladimir.medvedkin@intel.com> wrote:
> This series adds multi-VRF support to both IPv4 and IPv6 FIB paths by
> allowing a single FIB instance to host multiple isolated routing domains.
>
> Currently FIB instance represents one routing instance. For workloads that
> need multiple VRFs, the only option is to create multiple FIB objects. In a
> burst oriented datapath, packets in the same batch can belong to different VRFs, so
> the application either does per-packet lookup in different FIB instances or
> regroups packets by VRF before lookup. Both approaches are expensive.
>
> To remove that cost, this series keeps all VRFs inside one FIB instance and
> extends lookup input with per-packet VRF IDs.
>
> The design follows the existing fast-path structure for both families. IPv4 and
> IPv6 use multi-ary trees with a 2^24 associativity on a first level (tbl24). The
> first-level table scales per configured VRF. This increases memory usage, but
> keeps performance and lookup complexity on par with non-VRF implementation.
>
> Vladimir Medvedkin (4):
> fib: add multi-VRF support
> fib: add VRF functional and unit tests
> fib6: add multi-VRF support
> fib6: add VRF functional and unit tests
>
> app/test-fib/main.c | 257 ++++++++++++++++++++++--
> app/test/test_fib.c | 298 +++++++++++++++++++++++++++
> app/test/test_fib6.c | 319 ++++++++++++++++++++++++++++-
> lib/fib/dir24_8.c | 241 ++++++++++++++++------
> lib/fib/dir24_8.h | 255 ++++++++++++++++--------
> lib/fib/dir24_8_avx512.c | 420 +++++++++++++++++++++++++++++++--------
> lib/fib/dir24_8_avx512.h | 80 +++++++-
> lib/fib/rte_fib.c | 158 ++++++++++++---
> lib/fib/rte_fib.h | 94 ++++++++-
> lib/fib/rte_fib6.c | 166 +++++++++++++---
> lib/fib/rte_fib6.h | 88 +++++++-
> lib/fib/trie.c | 158 +++++++++++----
> lib/fib/trie.h | 51 +++--
> lib/fib/trie_avx512.c | 225 +++++++++++++++++++--
> lib/fib/trie_avx512.h | 39 +++-
> 15 files changed, 2453 insertions(+), 396 deletions(-)
Not sure at all if this the right way to do VRF.
There are multiple ways to do VRF, the Linux way, the Cisco way, ...
This needs way more documentation and also an example.
Like an option to l3fwd. And also an implementation in testpmd.
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH 4/4] fib6: add VRF functional and unit tests
2026-03-22 15:42 ` [RFC PATCH 4/4] fib6: add VRF functional and unit tests Vladimir Medvedkin
@ 2026-03-22 16:45 ` Stephen Hemminger
0 siblings, 0 replies; 45+ messages in thread
From: Stephen Hemminger @ 2026-03-22 16:45 UTC (permalink / raw)
To: Vladimir Medvedkin; +Cc: dev, rjarry, nsaxena16, mb, adwivedi, jerinjacobk
On Sun, 22 Mar 2026 15:42:15 +0000
Vladimir Medvedkin <vladimir.medvedkin@intel.com> wrote:
>
> + /* Allocate VRF IDs array for lookups if using multiple VRFs */
> + if (config.nb_vrfs > 1) {
> + vrf_ids = rte_malloc(NULL, sizeof(uint16_t) * config.nb_lookup_ips, 0);
Since this won't be shared across processes, use regular malloc, not hugepages.
> + if (vrf_ids == NULL) {
> + printf("Can not alloc VRF IDs array\n");
> + return -ENOMEM;
> + }
> + /* Generate random VRF IDs for each lookup */
> + for (i = 0; i < config.nb_lookup_ips; i++)
> + vrf_ids[i] = rte_rand() % config.nb_vrfs;
> +
Use rte_rand_max here.
^ permalink raw reply [flat|nested] 45+ messages in thread
* RE: [RFC PATCH 0/4] VRF support in FIB library
2026-03-22 16:43 ` [RFC PATCH 0/4] VRF support in FIB library Stephen Hemminger
@ 2026-03-23 9:01 ` Morten Brørup
2026-03-23 11:32 ` Medvedkin, Vladimir
2026-03-23 11:16 ` Medvedkin, Vladimir
1 sibling, 1 reply; 45+ messages in thread
From: Morten Brørup @ 2026-03-23 9:01 UTC (permalink / raw)
To: Stephen Hemminger, Vladimir Medvedkin
Cc: dev, rjarry, nsaxena16, adwivedi, jerinjacobk
> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> Sent: Sunday, 22 March 2026 17.44
>
> On Sun, 22 Mar 2026 15:42:11 +0000
> Vladimir Medvedkin <vladimir.medvedkin@intel.com> wrote:
>
> > This series adds multi-VRF support to both IPv4 and IPv6 FIB paths by
> > allowing a single FIB instance to host multiple isolated routing
> domains.
> >
> > Currently FIB instance represents one routing instance. For workloads
> that
> > need multiple VRFs, the only option is to create multiple FIB
> objects. In a
> > burst oriented datapath, packets in the same batch can belong to
> different VRFs, so
> > the application either does per-packet lookup in different FIB
> instances or
> > regroups packets by VRF before lookup. Both approaches are expensive.
> >
> > To remove that cost, this series keeps all VRFs inside one FIB
> instance and
> > extends lookup input with per-packet VRF IDs.
> >
> > The design follows the existing fast-path structure for both
> families. IPv4 and
> > IPv6 use multi-ary trees with a 2^24 associativity on a first level
> (tbl24). The
> > first-level table scales per configured VRF. This increases memory
> usage, but
> > keeps performance and lookup complexity on par with non-VRF
> implementation.
> >
I noticed the suggested API uses separate parameters for the VRF and IP.
How about using one parameter, a structure containing the {VRF, IP} tuple, instead?
I'm mainly thinking about the bulk operations, where passing one array seems more intuitive than passing two arrays.
>
>
> Not sure at all if this the right way to do VRF.
> There are multiple ways to do VRF, the Linux way, the Cisco way, ...
I think a shared table operating on the {VRF, IP} tuple makes sense.
If a table instance per VRF is preferred, that is still supported.
Can you elaborate what Linux and Cisco does differently than this?
>
>
>
> This needs way more documentation and also an example.
+1
> Like an option to l3fwd. And also an implementation in testpmd.
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH 0/4] VRF support in FIB library
2026-03-22 15:42 [RFC PATCH 0/4] VRF support in FIB library Vladimir Medvedkin
` (4 preceding siblings ...)
2026-03-22 16:43 ` [RFC PATCH 0/4] VRF support in FIB library Stephen Hemminger
@ 2026-03-23 9:54 ` Robin Jarry
2026-03-23 11:34 ` Medvedkin, Vladimir
2026-03-23 11:27 ` Maxime Leroy
` (2 subsequent siblings)
8 siblings, 1 reply; 45+ messages in thread
From: Robin Jarry @ 2026-03-23 9:54 UTC (permalink / raw)
To: Vladimir Medvedkin, dev; +Cc: nsaxena16, mb, adwivedi, jerinjacobk
Vladimir Medvedkin, Mar 22, 2026 at 16:42:
> This series adds multi-VRF support to both IPv4 and IPv6 FIB paths by
> allowing a single FIB instance to host multiple isolated routing domains.
>
> Currently FIB instance represents one routing instance. For workloads that
> need multiple VRFs, the only option is to create multiple FIB objects. In a
> burst oriented datapath, packets in the same batch can belong to different VRFs, so
> the application either does per-packet lookup in different FIB instances or
> regroups packets by VRF before lookup. Both approaches are expensive.
>
> To remove that cost, this series keeps all VRFs inside one FIB instance and
> extends lookup input with per-packet VRF IDs.
>
> The design follows the existing fast-path structure for both families. IPv4 and
> IPv6 use multi-ary trees with a 2^24 associativity on a first level (tbl24). The
> first-level table scales per configured VRF. This increases memory usage, but
> keeps performance and lookup complexity on par with non-VRF implementation.
>
> Vladimir Medvedkin (4):
> fib: add multi-VRF support
> fib: add VRF functional and unit tests
> fib6: add multi-VRF support
> fib6: add VRF functional and unit tests
Hey Vladimir,
Thanks for the series, this is an interesting approach. Does this allow
sharing the tbl8 arrays amongst VRFs?
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH 0/4] VRF support in FIB library
2026-03-22 16:43 ` [RFC PATCH 0/4] VRF support in FIB library Stephen Hemminger
2026-03-23 9:01 ` Morten Brørup
@ 2026-03-23 11:16 ` Medvedkin, Vladimir
1 sibling, 0 replies; 45+ messages in thread
From: Medvedkin, Vladimir @ 2026-03-23 11:16 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: dev, rjarry, nsaxena16, mb, adwivedi, jerinjacobk
Hi Stephen,
On 3/22/2026 4:43 PM, Stephen Hemminger wrote:
> On Sun, 22 Mar 2026 15:42:11 +0000
> Vladimir Medvedkin <vladimir.medvedkin@intel.com> wrote:
>
>> This series adds multi-VRF support to both IPv4 and IPv6 FIB paths by
>> allowing a single FIB instance to host multiple isolated routing domains.
>>
>> Currently FIB instance represents one routing instance. For workloads that
>> need multiple VRFs, the only option is to create multiple FIB objects. In a
>> burst oriented datapath, packets in the same batch can belong to different VRFs, so
>> the application either does per-packet lookup in different FIB instances or
>> regroups packets by VRF before lookup. Both approaches are expensive.
>>
>> To remove that cost, this series keeps all VRFs inside one FIB instance and
>> extends lookup input with per-packet VRF IDs.
>>
>> The design follows the existing fast-path structure for both families. IPv4 and
>> IPv6 use multi-ary trees with a 2^24 associativity on a first level (tbl24). The
>> first-level table scales per configured VRF. This increases memory usage, but
>> keeps performance and lookup complexity on par with non-VRF implementation.
>>
>> Vladimir Medvedkin (4):
>> fib: add multi-VRF support
>> fib: add VRF functional and unit tests
>> fib6: add multi-VRF support
>> fib6: add VRF functional and unit tests
>>
>> app/test-fib/main.c | 257 ++++++++++++++++++++++--
>> app/test/test_fib.c | 298 +++++++++++++++++++++++++++
>> app/test/test_fib6.c | 319 ++++++++++++++++++++++++++++-
>> lib/fib/dir24_8.c | 241 ++++++++++++++++------
>> lib/fib/dir24_8.h | 255 ++++++++++++++++--------
>> lib/fib/dir24_8_avx512.c | 420 +++++++++++++++++++++++++++++++--------
>> lib/fib/dir24_8_avx512.h | 80 +++++++-
>> lib/fib/rte_fib.c | 158 ++++++++++++---
>> lib/fib/rte_fib.h | 94 ++++++++-
>> lib/fib/rte_fib6.c | 166 +++++++++++++---
>> lib/fib/rte_fib6.h | 88 +++++++-
>> lib/fib/trie.c | 158 +++++++++++----
>> lib/fib/trie.h | 51 +++--
>> lib/fib/trie_avx512.c | 225 +++++++++++++++++++--
>> lib/fib/trie_avx512.h | 39 +++-
>> 15 files changed, 2453 insertions(+), 396 deletions(-)
>
> Not sure at all if this the right way to do VRF.
I'd like to hear the reason why this is not the right way, what is the
right way and why
> There are multiple ways to do VRF, the Linux way, the Cisco way, ...
I'm not 100% sure what defines a "way", but looking at VPP (since Cisco
was mentioned) their way is pretty much the same as Linux - i.e. it has
a separate handle per VRF(fib_index):
https://github.com/FDio/vpp/blob/c2f6a88b43d6175730d5d19364cb446e128065da/src/vnet/fib/ip4_fib.h#L128
In grout with the current FIB design the situation is the same - FIB
instance per VRF(vrf_id):
https://github.com/DPDK/grout/blob/0f7b91c287f7f54cd75a1b64da7ade505b346bf5/modules/ip/control/route.c#L147
I wrote about the problems of this approach in the cover letter,
different options were considered, and this is the design I have arrived at.
After all, why should we have to follow someone else's way? I'm
proposing our own "DPDK way" :)
>
>
>
> This needs way more documentation and also an example.
> Like an option to l3fwd. And also an implementation in testpmd.
--
Regards,
Vladimir
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH 0/4] VRF support in FIB library
2026-03-22 15:42 [RFC PATCH 0/4] VRF support in FIB library Vladimir Medvedkin
` (5 preceding siblings ...)
2026-03-23 9:54 ` Robin Jarry
@ 2026-03-23 11:27 ` Maxime Leroy
2026-03-23 12:49 ` Medvedkin, Vladimir
2026-03-23 19:05 ` Stephen Hemminger
2026-10-14 18:39 ` [RFC PATCH v2 0/7] " Vladimir Medvedkin
8 siblings, 1 reply; 45+ messages in thread
From: Maxime Leroy @ 2026-03-23 11:27 UTC (permalink / raw)
To: Vladimir Medvedkin; +Cc: dev, rjarry, nsaxena16, mb, adwivedi, jerinjacobk
Hi Vladimir,
On Sun, Mar 22, 2026 at 4:42 PM Vladimir Medvedkin
<vladimir.medvedkin@intel.com> wrote:
>
> This series adds multi-VRF support to both IPv4 and IPv6 FIB paths by
> allowing a single FIB instance to host multiple isolated routing domains.
>
> Currently FIB instance represents one routing instance. For workloads that
> need multiple VRFs, the only option is to create multiple FIB objects. In a
> burst oriented datapath, packets in the same batch can belong to different VRFs, so
> the application either does per-packet lookup in different FIB instances or
> regroups packets by VRF before lookup. Both approaches are expensive.
>
> To remove that cost, this series keeps all VRFs inside one FIB instance and
> extends lookup input with per-packet VRF IDs.
>
> The design follows the existing fast-path structure for both families. IPv4 and
> IPv6 use multi-ary trees with a 2^24 associativity on a first level (tbl24). The
> first-level table scales per configured VRF. This increases memory usage, but
> keeps performance and lookup complexity on par with non-VRF implementation.
>
Thanks for the RFC. Some thoughts below.
Memory cost: the flat TBL24 replicates the entire table for every VRF
(num_vrfs * 2^24 * nh_size). With 256 VRFs and 8B nexthops that is
32 GB for TBL24 alone. In grout we support up to 256 VRFs allocated
on demand -- this approach forces the full cost upfront even if most
VRFs are empty.
Per-packet VRF lookup: Rx bursts come from one port, thus one VRF.
Mixed-VRF bulk lookups do not occur in practice. The three AVX512
code paths add complexity for a scenario that does not exist, at
least for a classic router. Am I missing a use-case?
I am not too familiar with DPDK FIB internals, but would it be
possible to keep a separate TBL24 per VRF and only share the TBL8
pool? Something like pre-allocating an array of max_vrfs TBL24
pointers, allocating each TBL24 on demand at VRF add time, and
having them all point into a shared TBL8 pool. The TBL8 index in
TBL24 entries seems to already be global, so would that work without
encoding changes?
Going further: could the same idea extend to IPv6? The dir24_8 and
trie seem to use the same TBL8 block format (256 entries, same
(nh << 1) | ext_bit encoding, same size). Would unifying the TBL8
allocator allow a single pool shared across IPv4, IPv6, and all
VRFs? That could be a bigger win for /32-heavy and /128-heavy tables
and maybe a good first step before multi-VRF.
Regards,
Maxime Leroy
> Vladimir Medvedkin (4):
> fib: add multi-VRF support
> fib: add VRF functional and unit tests
> fib6: add multi-VRF support
> fib6: add VRF functional and unit tests
>
> app/test-fib/main.c | 257 ++++++++++++++++++++++--
> app/test/test_fib.c | 298 +++++++++++++++++++++++++++
> app/test/test_fib6.c | 319 ++++++++++++++++++++++++++++-
> lib/fib/dir24_8.c | 241 ++++++++++++++++------
> lib/fib/dir24_8.h | 255 ++++++++++++++++--------
> lib/fib/dir24_8_avx512.c | 420 +++++++++++++++++++++++++++++++--------
> lib/fib/dir24_8_avx512.h | 80 +++++++-
> lib/fib/rte_fib.c | 158 ++++++++++++---
> lib/fib/rte_fib.h | 94 ++++++++-
> lib/fib/rte_fib6.c | 166 +++++++++++++---
> lib/fib/rte_fib6.h | 88 +++++++-
> lib/fib/trie.c | 158 +++++++++++----
> lib/fib/trie.h | 51 +++--
> lib/fib/trie_avx512.c | 225 +++++++++++++++++++--
> lib/fib/trie_avx512.h | 39 +++-
> 15 files changed, 2453 insertions(+), 396 deletions(-)
>
> --
> 2.43.0
>
--
-------------------------------
Maxime Leroy
maxime@leroys.fr
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH 0/4] VRF support in FIB library
2026-03-23 9:01 ` Morten Brørup
@ 2026-03-23 11:32 ` Medvedkin, Vladimir
0 siblings, 0 replies; 45+ messages in thread
From: Medvedkin, Vladimir @ 2026-03-23 11:32 UTC (permalink / raw)
To: Morten Brørup, Stephen Hemminger
Cc: dev, rjarry, nsaxena16, adwivedi, jerinjacobk
Hi Morten,
On 3/23/2026 9:01 AM, Morten Brørup wrote:
>> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
>> Sent: Sunday, 22 March 2026 17.44
>>
>> On Sun, 22 Mar 2026 15:42:11 +0000
>> Vladimir Medvedkin <vladimir.medvedkin@intel.com> wrote:
>>
>>> This series adds multi-VRF support to both IPv4 and IPv6 FIB paths by
>>> allowing a single FIB instance to host multiple isolated routing
>> domains.
>>> Currently FIB instance represents one routing instance. For workloads
>> that
>>> need multiple VRFs, the only option is to create multiple FIB
>> objects. In a
>>> burst oriented datapath, packets in the same batch can belong to
>> different VRFs, so
>>> the application either does per-packet lookup in different FIB
>> instances or
>>> regroups packets by VRF before lookup. Both approaches are expensive.
>>>
>>> To remove that cost, this series keeps all VRFs inside one FIB
>> instance and
>>> extends lookup input with per-packet VRF IDs.
>>>
>>> The design follows the existing fast-path structure for both
>> families. IPv4 and
>>> IPv6 use multi-ary trees with a 2^24 associativity on a first level
>> (tbl24). The
>>> first-level table scales per configured VRF. This increases memory
>> usage, but
>>> keeps performance and lookup complexity on par with non-VRF
>> implementation.
> I noticed the suggested API uses separate parameters for the VRF and IP.
> How about using one parameter, a structure containing the {VRF, IP} tuple, instead?
> I'm mainly thinking about the bulk operations, where passing one array seems more intuitive than passing two arrays.
I found this design to be more intuitive and kind of backward
compatible, many apps already create an array of addresses, adding an
extra array with corresponding VRFs maynotbecounterintuitive IMO.
ButwhatIfindmoreimportantisperformance,atleastthisapproachis
moreconvenientforvectorization.
>
>>
>> Not sure at all if this the right way to do VRF.
>> There are multiple ways to do VRF, the Linux way, the Cisco way, ...
> I think a shared table operating on the {VRF, IP} tuple makes sense.
> If a table instance per VRF is preferred, that is still supported.
>
> Can you elaborate what Linux and Cisco does differently than this?
>
>>
>>
>> This needs way more documentation and also an example.
> +1
>
>> Like an option to l3fwd. And also an implementation in testpmd.
--
Regards,
Vladimir
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH 0/4] VRF support in FIB library
2026-03-23 9:54 ` Robin Jarry
@ 2026-03-23 11:34 ` Medvedkin, Vladimir
0 siblings, 0 replies; 45+ messages in thread
From: Medvedkin, Vladimir @ 2026-03-23 11:34 UTC (permalink / raw)
To: Robin Jarry, dev; +Cc: nsaxena16, mb, adwivedi, jerinjacobk
Hi Robin,
On 3/23/2026 9:54 AM, Robin Jarry wrote:
> Vladimir Medvedkin, Mar 22, 2026 at 16:42:
>> This series adds multi-VRF support to both IPv4 and IPv6 FIB paths by
>> allowing a single FIB instance to host multiple isolated routing domains.
>>
>> Currently FIB instance represents one routing instance. For workloads that
>> need multiple VRFs, the only option is to create multiple FIB objects. In a
>> burst oriented datapath, packets in the same batch can belong to different VRFs, so
>> the application either does per-packet lookup in different FIB instances or
>> regroups packets by VRF before lookup. Both approaches are expensive.
>>
>> To remove that cost, this series keeps all VRFs inside one FIB instance and
>> extends lookup input with per-packet VRF IDs.
>>
>> The design follows the existing fast-path structure for both families. IPv4 and
>> IPv6 use multi-ary trees with a 2^24 associativity on a first level (tbl24). The
>> first-level table scales per configured VRF. This increases memory usage, but
>> keeps performance and lookup complexity on par with non-VRF implementation.
>>
>> Vladimir Medvedkin (4):
>> fib: add multi-VRF support
>> fib: add VRF functional and unit tests
>> fib6: add multi-VRF support
>> fib6: add VRF functional and unit tests
> Hey Vladimir,
>
> Thanks for the series, this is an interesting approach. Does this allow
> sharing the tbl8 arrays amongst VRFs?
Yes! tbl8 array is shared for all VRFs in both IPv4(dir24_8) and IPv6(trie).
--
Regards,
Vladimir
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH 0/4] VRF support in FIB library
2026-03-23 11:27 ` Maxime Leroy
@ 2026-03-23 12:49 ` Medvedkin, Vladimir
2026-03-23 14:53 ` Maxime Leroy
0 siblings, 1 reply; 45+ messages in thread
From: Medvedkin, Vladimir @ 2026-03-23 12:49 UTC (permalink / raw)
To: Maxime Leroy; +Cc: dev, rjarry, nsaxena16, mb, adwivedi, jerinjacobk
Hi Maxime,
On 3/23/2026 11:27 AM, Maxime Leroy wrote:
> Hi Vladimir,
>
>
> On Sun, Mar 22, 2026 at 4:42 PM Vladimir Medvedkin
> <vladimir.medvedkin@intel.com> wrote:
>> This series adds multi-VRF support to both IPv4 and IPv6 FIB paths by
>> allowing a single FIB instance to host multiple isolated routing domains.
>>
>> Currently FIB instance represents one routing instance. For workloads that
>> need multiple VRFs, the only option is to create multiple FIB objects. In a
>> burst oriented datapath, packets in the same batch can belong to different VRFs, so
>> the application either does per-packet lookup in different FIB instances or
>> regroups packets by VRF before lookup. Both approaches are expensive.
>>
>> To remove that cost, this series keeps all VRFs inside one FIB instance and
>> extends lookup input with per-packet VRF IDs.
>>
>> The design follows the existing fast-path structure for both families. IPv4 and
>> IPv6 use multi-ary trees with a 2^24 associativity on a first level (tbl24). The
>> first-level table scales per configured VRF. This increases memory usage, but
>> keeps performance and lookup complexity on par with non-VRF implementation.
>>
> Thanks for the RFC. Some thoughts below.
>
> Memory cost: the flat TBL24 replicates the entire table for every VRF
> (num_vrfs * 2^24 * nh_size). With 256 VRFs and 8B nexthops that is
> 32 GB for TBL24 alone. In grout we support up to 256 VRFs allocated
> on demand -- this approach forces the full cost upfront even if most
> VRFs are empty.
Yes, increased memory consumption is the
trade-off.WemakethischoiceinDPDKquite often,such as pre-allocatedmbufs,
mempoolsand many other stuff allocated in advance to gain performance.
For FIB, I chose to replicate TBL24 per VRF for this same reason.
And, as Morten mentioned earlier, if memory is the priority, a table
instance per VRF allocated on-demand is still supported.
The high memory cost stems from TBL24's design: for IPv4, it was
justified by the BGP filtering convention (no prefixes more specific
than /24 in BGPv4 full view), ensuring most lookups hit with just one
random memory access. For IPv6, we should likely switch to a 16-bit TRIE
scheme on all layers. For IPv4, alternative algorithms with smaller
footprints (like DXR or DIR16-8-8, as used in VPP) may be worth
exploring if BGP full view is not required for those VRFs.
>
> Per-packet VRF lookup: Rx bursts come from one port, thus one VRF.
> Mixed-VRF bulk lookups do not occur in practice. The three AVX512
> code paths add complexity for a scenario that does not exist, at
> least for a classic router. Am I missing a use-case?
That's not true, you're missing out on a lot of established core use
cases that are at least 2 decades old:
- VLAN subinterface abstraction. Each subinterface may belong to a
separate VRF
- MPLS VPN
- Policy based routing
>
> I am not too familiar with DPDK FIB internals, but would it be
> possible to keep a separate TBL24 per VRF and only share the TBL8
> pool?
it is how it is implemented right now with one note - TBL24 are pre
allocated.
> Something like pre-allocating an array of max_vrfs TBL24
> pointers, allocating each TBL24 on demand at VRF add time,
and you suggesting to allocate TBL24 on demand by adding an extra
indirection layer. Thiswill leadtolowerperformance,whichIwouldliketo avoid.
> and
> having them all point into a shared TBL8 pool. The TBL8 index in
> TBL24 entries seems to already be global, so would that work without
> encoding changes?
>
> Going further: could the same idea extend to IPv6? The dir24_8 and
> trie seem to use the same TBL8 block format (256 entries, same
> (nh << 1) | ext_bit encoding, same size). Would unifying the TBL8
> allocator allow a single pool shared across IPv4, IPv6, and all
> VRFs? That could be a bigger win for /32-heavy and /128-heavy tables
> and maybe a good first step before multi-VRF.
So, you are suggesting merging IPv4 and IPv6 into a single unified FIB?
I'm not sure how this can be a bigger win, could you please elaborate
more on this?
> Regards,
>
> Maxime Leroy
>
>> Vladimir Medvedkin (4):
>> fib: add multi-VRF support
>> fib: add VRF functional and unit tests
>> fib6: add multi-VRF support
>> fib6: add VRF functional and unit tests
>>
>> app/test-fib/main.c | 257 ++++++++++++++++++++++--
>> app/test/test_fib.c | 298 +++++++++++++++++++++++++++
>> app/test/test_fib6.c | 319 ++++++++++++++++++++++++++++-
>> lib/fib/dir24_8.c | 241 ++++++++++++++++------
>> lib/fib/dir24_8.h | 255 ++++++++++++++++--------
>> lib/fib/dir24_8_avx512.c | 420 +++++++++++++++++++++++++++++++--------
>> lib/fib/dir24_8_avx512.h | 80 +++++++-
>> lib/fib/rte_fib.c | 158 ++++++++++++---
>> lib/fib/rte_fib.h | 94 ++++++++-
>> lib/fib/rte_fib6.c | 166 +++++++++++++---
>> lib/fib/rte_fib6.h | 88 +++++++-
>> lib/fib/trie.c | 158 +++++++++++----
>> lib/fib/trie.h | 51 +++--
>> lib/fib/trie_avx512.c | 225 +++++++++++++++++++--
>> lib/fib/trie_avx512.h | 39 +++-
>> 15 files changed, 2453 insertions(+), 396 deletions(-)
>>
>> --
>> 2.43.0
>>
>
--
Regards,
Vladimir
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH 0/4] VRF support in FIB library
2026-03-23 12:49 ` Medvedkin, Vladimir
@ 2026-03-23 14:53 ` Maxime Leroy
2026-03-23 15:08 ` Robin Jarry
2026-03-23 18:42 ` Medvedkin, Vladimir
0 siblings, 2 replies; 45+ messages in thread
From: Maxime Leroy @ 2026-03-23 14:53 UTC (permalink / raw)
To: Medvedkin, Vladimir; +Cc: dev, rjarry, nsaxena16, mb, adwivedi, jerinjacobk
On Mon, Mar 23, 2026 at 1:49 PM Medvedkin, Vladimir
<vladimir.medvedkin@intel.com> wrote:
>
> Hi Maxime,
>
> On 3/23/2026 11:27 AM, Maxime Leroy wrote:
> > Hi Vladimir,
> >
> >
> > On Sun, Mar 22, 2026 at 4:42 PM Vladimir Medvedkin
> > <vladimir.medvedkin@intel.com> wrote:
> >> This series adds multi-VRF support to both IPv4 and IPv6 FIB paths by
> >> allowing a single FIB instance to host multiple isolated routing domains.
> >>
> >> Currently FIB instance represents one routing instance. For workloads that
> >> need multiple VRFs, the only option is to create multiple FIB objects. In a
> >> burst oriented datapath, packets in the same batch can belong to different VRFs, so
> >> the application either does per-packet lookup in different FIB instances or
> >> regroups packets by VRF before lookup. Both approaches are expensive.
> >>
> >> To remove that cost, this series keeps all VRFs inside one FIB instance and
> >> extends lookup input with per-packet VRF IDs.
> >>
> >> The design follows the existing fast-path structure for both families. IPv4 and
> >> IPv6 use multi-ary trees with a 2^24 associativity on a first level (tbl24). The
> >> first-level table scales per configured VRF. This increases memory usage, but
> >> keeps performance and lookup complexity on par with non-VRF implementation.
> >>
> > Thanks for the RFC. Some thoughts below.
> >
> > Memory cost: the flat TBL24 replicates the entire table for every VRF
> > (num_vrfs * 2^24 * nh_size). With 256 VRFs and 8B nexthops that is
> > 32 GB for TBL24 alone. In grout we support up to 256 VRFs allocated
> > on demand -- this approach forces the full cost upfront even if most
> > VRFs are empty.
>
> Yes, increased memory consumption is the
> trade-off.WemakethischoiceinDPDKquite often,such as pre-allocatedmbufs,
> mempoolsand many other stuff allocated in advance to gain performance.
> For FIB, I chose to replicate TBL24 per VRF for this same reason.
>
> And, as Morten mentioned earlier, if memory is the priority, a table
> instance per VRF allocated on-demand is still supported.
>
> The high memory cost stems from TBL24's design: for IPv4, it was
> justified by the BGP filtering convention (no prefixes more specific
> than /24 in BGPv4 full view), ensuring most lookups hit with just one
> random memory access. For IPv6, we should likely switch to a 16-bit TRIE
> scheme on all layers. For IPv4, alternative algorithms with smaller
> footprints (like DXR or DIR16-8-8, as used in VPP) may be worth
> exploring if BGP full view is not required for those VRFs.
>
> >
> > Per-packet VRF lookup: Rx bursts come from one port, thus one VRF.
> > Mixed-VRF bulk lookups do not occur in practice. The three AVX512
> > code paths add complexity for a scenario that does not exist, at
> > least for a classic router. Am I missing a use-case?
>
> That's not true, you're missing out on a lot of established core use
> cases that are at least 2 decades old:
>
> - VLAN subinterface abstraction. Each subinterface may belong to a
> separate VRF
>
> - MPLS VPN
>
> - Policy based routing
>
Fair point on VLAN subinterfaces and MPLS VPN. SRv6 L3VPN (End.DT4/
End.DT6) also fits that pattern after decap.
I agree DPDK often pre-allocates for performance, but I wonder if the
flat TBL24 actually helps here. Each VRF's working set is spread
128 MB apart in the flat table. Would regrouping packets by VRF and
doing one bulk lookup per VRF with separate contiguous TBL24s be
more cache-friendly than a single mixed-VRF gather? Do you have
benchmarks comparing the two approaches?
On the memory trade-off and VRF ID mapping: the API uses vrf_id as
a direct index (0 to max_vrfs-1). With 256 VRFs and 8B nexthops,
TBL24 alone costs 32 GB for IPv4 and 32 GB for IPv6 -- 64 GB total
at startup. In grout, VRF IDs are interface IDs that can be any
uint16_t, so we would also need to maintain a mapping between our
VRF IDs and FIB slot indices. We would need to introduce a max_vrfs
limit, which forces a bad trade-off: either set it low (e.g. 16)
and limit deployments, or set it high (e.g. 256) and pay 64 GB at
startup even with a single VRF. With separate FIB instances per VRF,
we only allocate what we use.
> >
> > I am not too familiar with DPDK FIB internals, but would it be
> > possible to keep a separate TBL24 per VRF and only share the TBL8
> > pool?
> it is how it is implemented right now with one note - TBL24 are pre
> allocated.
> > Something like pre-allocating an array of max_vrfs TBL24
> > pointers, allocating each TBL24 on demand at VRF add time,
> and you suggesting to allocate TBL24 on demand by adding an extra
> indirection layer. Thiswill leadtolowerperformance,whichIwouldliketo avoid.
> > and
> > having them all point into a shared TBL8 pool. The TBL8 index in
> > TBL24 entries seems to already be global, so would that work without
> > encoding changes?
> >
> > Going further: could the same idea extend to IPv6? The dir24_8 and
> > trie seem to use the same TBL8 block format (256 entries, same
> > (nh << 1) | ext_bit encoding, same size). Would unifying the TBL8
> > allocator allow a single pool shared across IPv4, IPv6, and all
> > VRFs? That could be a bigger win for /32-heavy and /128-heavy tables
> > and maybe a good first step before multi-VRF.
>
> So, you are suggesting merging IPv4 and IPv6 into a single unified FIB?
> I'm not sure how this can be a bigger win, could you please elaborate
> more on this?
On the IPv4/IPv6 TBL8 pool: I was not suggesting merging FIBs, just
sharing the TBL8 block allocator between separate FIB instances.
This is possible since dir24_8 and trie use the same TBL8 block
format (256 entries, same encoding, same size).
Would it be possible to pass a shared TBL8 pool at rte_fib_create()
time? Each FIB keeps its own TBL24 and RIB, but TBL8 is shared
across all FIBs and potentially across IPv4/IPv6. Users would no
longer have to guess num_tbl8 per FIB.
>
> > Regards,
> >
> > Maxime Leroy
> >
> >> Vladimir Medvedkin (4):
> >> fib: add multi-VRF support
> >> fib: add VRF functional and unit tests
> >> fib6: add multi-VRF support
> >> fib6: add VRF functional and unit tests
> >>
> >> app/test-fib/main.c | 257 ++++++++++++++++++++++--
> >> app/test/test_fib.c | 298 +++++++++++++++++++++++++++
> >> app/test/test_fib6.c | 319 ++++++++++++++++++++++++++++-
> >> lib/fib/dir24_8.c | 241 ++++++++++++++++------
> >> lib/fib/dir24_8.h | 255 ++++++++++++++++--------
> >> lib/fib/dir24_8_avx512.c | 420 +++++++++++++++++++++++++++++++--------
> >> lib/fib/dir24_8_avx512.h | 80 +++++++-
> >> lib/fib/rte_fib.c | 158 ++++++++++++---
> >> lib/fib/rte_fib.h | 94 ++++++++-
> >> lib/fib/rte_fib6.c | 166 +++++++++++++---
> >> lib/fib/rte_fib6.h | 88 +++++++-
> >> lib/fib/trie.c | 158 +++++++++++----
> >> lib/fib/trie.h | 51 +++--
> >> lib/fib/trie_avx512.c | 225 +++++++++++++++++++--
> >> lib/fib/trie_avx512.h | 39 +++-
> >> 15 files changed, 2453 insertions(+), 396 deletions(-)
> >>
> >> --
> >> 2.43.0
> >>
> >
> --
> Regards,
> Vladimir
>
--
-------------------------------
Maxime Leroy
maxime@leroys.fr
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH 0/4] VRF support in FIB library
2026-03-23 14:53 ` Maxime Leroy
@ 2026-03-23 15:08 ` Robin Jarry
2026-03-23 15:27 ` Morten Brørup
2026-03-23 18:42 ` Medvedkin, Vladimir
1 sibling, 1 reply; 45+ messages in thread
From: Robin Jarry @ 2026-03-23 15:08 UTC (permalink / raw)
To: Maxime Leroy, Medvedkin, Vladimir
Cc: dev, nsaxena16, mb, adwivedi, jerinjacobk
Hey folks,
Maxime Leroy, Mar 23, 2026 at 15:53:
> Fair point on VLAN subinterfaces and MPLS VPN. SRv6 L3VPN (End.DT4/
> End.DT6) also fits that pattern after decap.
>
> I agree DPDK often pre-allocates for performance, but I wonder if the
> flat TBL24 actually helps here. Each VRF's working set is spread
> 128 MB apart in the flat table. Would regrouping packets by VRF and
> doing one bulk lookup per VRF with separate contiguous TBL24s be
> more cache-friendly than a single mixed-VRF gather? Do you have
> benchmarks comparing the two approaches?
> On the memory trade-off and VRF ID mapping: the API uses vrf_id as
> a direct index (0 to max_vrfs-1). With 256 VRFs and 8B nexthops,
> TBL24 alone costs 32 GB for IPv4 and 32 GB for IPv6 -- 64 GB total
> at startup. In grout, VRF IDs are interface IDs that can be any
> uint16_t, so we would also need to maintain a mapping between our
> VRF IDs and FIB slot indices. We would need to introduce a max_vrfs
> limit, which forces a bad trade-off: either set it low (e.g. 16)
> and limit deployments, or set it high (e.g. 256) and pay 64 GB at
> startup even with a single VRF. With separate FIB instances per VRF,
> we only allocate what we use.
I am also concerned about the global memory consumption. Taking grout as
a live example, we currently support up to 1024 VRFs (each VRF is
an interface so the upper limit is just the number of interfaces).
Pre-allocating 1024 rte_fib and rte_fib6 is virtually impossible.
> On the IPv4/IPv6 TBL8 pool: I was not suggesting merging FIBs, just
> sharing the TBL8 block allocator between separate FIB instances.
> This is possible since dir24_8 and trie use the same TBL8 block
> format (256 entries, same encoding, same size).
>
> Would it be possible to pass a shared TBL8 pool at rte_fib_create()
> time? Each FIB keeps its own TBL24 and RIB, but TBL8 is shared
> across all FIBs and potentially across IPv4/IPv6. Users would no
> longer have to guess num_tbl8 per FIB.
+1 to this. That would help a lot to have a common tbl8 pool. That way
we could keep the single VRF per fib/rib but have a global tbl8 pool
that we can tune to our use case.
Cheers,
^ permalink raw reply [flat|nested] 45+ messages in thread
* RE: [RFC PATCH 0/4] VRF support in FIB library
2026-03-23 15:08 ` Robin Jarry
@ 2026-03-23 15:27 ` Morten Brørup
2026-03-23 18:52 ` Medvedkin, Vladimir
0 siblings, 1 reply; 45+ messages in thread
From: Morten Brørup @ 2026-03-23 15:27 UTC (permalink / raw)
To: Robin Jarry, Maxime Leroy, Medvedkin, Vladimir
Cc: dev, nsaxena16, adwivedi, jerinjacobk, Stephen Hemminger
Let's take a level up in abstraction, and consider one of the key design goals:
What are the typical use cases we want VRF support designed for?
I can imagine a use case with one or very few VRFs with a very large (internet) route table, and many VRFs with small (private) route tables.
PS: I have no experience with the DPDK FIB library, so my feedback might be completely off.
^ permalink raw reply [flat|nested] 45+ messages in thread
* RE: [RFC PATCH 1/4] fib: add multi-VRF support
2026-03-22 15:42 ` [RFC PATCH 1/4] fib: add multi-VRF support Vladimir Medvedkin
@ 2026-03-23 15:48 ` Konstantin Ananyev
2026-03-23 19:06 ` Medvedkin, Vladimir
0 siblings, 1 reply; 45+ messages in thread
From: Konstantin Ananyev @ 2026-03-23 15:48 UTC (permalink / raw)
To: Vladimir Medvedkin, dev@dpdk.org
Cc: rjarry@redhat.com, nsaxena16@gmail.com, mb@smartsharesystems.com,
adwivedi@marvell.com, jerinjacobk@gmail.com, Maxime Leroy
> Add VRF (Virtual Routing and Forwarding) support to the IPv4
> FIB library, allowing multiple independent routing tables
> within a single FIB instance.
>
> Introduce max_vrfs and vrf_default_nh fields in rte_fib_conf
> to configure the number of VRFs and per-VRF default nexthops.
Thanks Vladimir, allowing multiple VRFs per same LPM table will
definitely be a useful thing to have.
Though, I have the same concern as Maxime:
memory requirements are just overwhelming.
Stupid q - why just not to store a pointer to a vector of next-hops
within the table entry?
And we can provide to the user with ability to specify custom
alloc/free function for these vectors.
That would help to avoid allocating huge chunks of memory at startup.
I understand that it will be one extra memory dereference,
but probably it will be not that critical in terms of performance .
Again for bulk function we might be able to pipeline lookups and
de-references and hide that extra load latency.
> Add four new experimental APIs:
> - rte_fib_vrf_add() and rte_fib_vrf_delete() to manage routes
> per VRF
> - rte_fib_vrf_lookup_bulk() for multi-VRF bulk lookups
> - rte_fib_vrf_get_rib() to retrieve a per-VRF RIB handle
>
> Signed-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
> ---
> lib/fib/dir24_8.c | 241 ++++++++++++++++------
> lib/fib/dir24_8.h | 255 ++++++++++++++++--------
> lib/fib/dir24_8_avx512.c | 420 +++++++++++++++++++++++++++++++--------
> lib/fib/dir24_8_avx512.h | 80 +++++++-
> lib/fib/rte_fib.c | 158 ++++++++++++---
> lib/fib/rte_fib.h | 94 ++++++++-
> 6 files changed, 988 insertions(+), 260 deletions(-)
>
> diff --git a/lib/fib/dir24_8.c b/lib/fib/dir24_8.c
> index 489d2ef427..ad295c5f16 100644
> --- a/lib/fib/dir24_8.c
> +++ b/lib/fib/dir24_8.c
> @@ -32,41 +32,80 @@
> #define ROUNDUP(x, y) RTE_ALIGN_CEIL(x, (1 << (32 - y)))
>
> static inline rte_fib_lookup_fn_t
> -get_scalar_fn(enum rte_fib_dir24_8_nh_sz nh_sz, bool be_addr)
> +get_scalar_fn(const struct dir24_8_tbl *dp, enum rte_fib_dir24_8_nh_sz nh_sz,
> + bool be_addr)
> {
> + bool single_vrf = dp->num_vrfs <= 1;
> +
> switch (nh_sz) {
> case RTE_FIB_DIR24_8_1B:
> - return be_addr ? dir24_8_lookup_bulk_1b_be :
> dir24_8_lookup_bulk_1b;
> + if (single_vrf)
> + return be_addr ? dir24_8_lookup_bulk_1b_be :
> + dir24_8_lookup_bulk_1b;
> + return be_addr ? dir24_8_lookup_bulk_vrf_1b_be :
> + dir24_8_lookup_bulk_vrf_1b;
> case RTE_FIB_DIR24_8_2B:
> - return be_addr ? dir24_8_lookup_bulk_2b_be :
> dir24_8_lookup_bulk_2b;
> + if (single_vrf)
> + return be_addr ? dir24_8_lookup_bulk_2b_be :
> + dir24_8_lookup_bulk_2b;
> + return be_addr ? dir24_8_lookup_bulk_vrf_2b_be :
> + dir24_8_lookup_bulk_vrf_2b;
> case RTE_FIB_DIR24_8_4B:
> - return be_addr ? dir24_8_lookup_bulk_4b_be :
> dir24_8_lookup_bulk_4b;
> + if (single_vrf)
> + return be_addr ? dir24_8_lookup_bulk_4b_be :
> + dir24_8_lookup_bulk_4b;
> + return be_addr ? dir24_8_lookup_bulk_vrf_4b_be :
> + dir24_8_lookup_bulk_vrf_4b;
> case RTE_FIB_DIR24_8_8B:
> - return be_addr ? dir24_8_lookup_bulk_8b_be :
> dir24_8_lookup_bulk_8b;
> + if (single_vrf)
> + return be_addr ? dir24_8_lookup_bulk_8b_be :
> + dir24_8_lookup_bulk_8b;
> + return be_addr ? dir24_8_lookup_bulk_vrf_8b_be :
> + dir24_8_lookup_bulk_vrf_8b;
> default:
> return NULL;
> }
> }
>
> static inline rte_fib_lookup_fn_t
> -get_scalar_fn_inlined(enum rte_fib_dir24_8_nh_sz nh_sz, bool be_addr)
> +get_scalar_fn_inlined(const struct dir24_8_tbl *dp,
> + enum rte_fib_dir24_8_nh_sz nh_sz, bool be_addr)
> {
> + bool single_vrf = dp->num_vrfs <= 1;
> +
> switch (nh_sz) {
> case RTE_FIB_DIR24_8_1B:
> - return be_addr ? dir24_8_lookup_bulk_0_be :
> dir24_8_lookup_bulk_0;
> + if (single_vrf)
> + return be_addr ? dir24_8_lookup_bulk_0_be :
> + dir24_8_lookup_bulk_0;
> + return be_addr ? dir24_8_lookup_bulk_vrf_0_be :
> + dir24_8_lookup_bulk_vrf_0;
> case RTE_FIB_DIR24_8_2B:
> - return be_addr ? dir24_8_lookup_bulk_1_be :
> dir24_8_lookup_bulk_1;
> + if (single_vrf)
> + return be_addr ? dir24_8_lookup_bulk_1_be :
> + dir24_8_lookup_bulk_1;
> + return be_addr ? dir24_8_lookup_bulk_vrf_1_be :
> + dir24_8_lookup_bulk_vrf_1;
> case RTE_FIB_DIR24_8_4B:
> - return be_addr ? dir24_8_lookup_bulk_2_be :
> dir24_8_lookup_bulk_2;
> + if (single_vrf)
> + return be_addr ? dir24_8_lookup_bulk_2_be :
> + dir24_8_lookup_bulk_2;
> + return be_addr ? dir24_8_lookup_bulk_vrf_2_be :
> + dir24_8_lookup_bulk_vrf_2;
> case RTE_FIB_DIR24_8_8B:
> - return be_addr ? dir24_8_lookup_bulk_3_be :
> dir24_8_lookup_bulk_3;
> + if (single_vrf)
> + return be_addr ? dir24_8_lookup_bulk_3_be :
> + dir24_8_lookup_bulk_3;
> + return be_addr ? dir24_8_lookup_bulk_vrf_3_be :
> + dir24_8_lookup_bulk_vrf_3;
> default:
> return NULL;
> }
> }
>
> static inline rte_fib_lookup_fn_t
> -get_vector_fn(enum rte_fib_dir24_8_nh_sz nh_sz, bool be_addr)
> +get_vector_fn(const struct dir24_8_tbl *dp, enum rte_fib_dir24_8_nh_sz nh_sz,
> + bool be_addr)
> {
> #ifdef CC_AVX512_SUPPORT
> if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512F) <= 0 ||
> @@ -77,24 +116,63 @@ get_vector_fn(enum rte_fib_dir24_8_nh_sz nh_sz, bool
> be_addr)
> if (be_addr && rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512BW) <=
> 0)
> return NULL;
>
> + if (dp->num_vrfs <= 1) {
> + switch (nh_sz) {
> + case RTE_FIB_DIR24_8_1B:
> + return be_addr ? rte_dir24_8_vec_lookup_bulk_1b_be :
> + rte_dir24_8_vec_lookup_bulk_1b;
> + case RTE_FIB_DIR24_8_2B:
> + return be_addr ? rte_dir24_8_vec_lookup_bulk_2b_be :
> + rte_dir24_8_vec_lookup_bulk_2b;
> + case RTE_FIB_DIR24_8_4B:
> + return be_addr ? rte_dir24_8_vec_lookup_bulk_4b_be :
> + rte_dir24_8_vec_lookup_bulk_4b;
> + case RTE_FIB_DIR24_8_8B:
> + return be_addr ? rte_dir24_8_vec_lookup_bulk_8b_be :
> + rte_dir24_8_vec_lookup_bulk_8b;
> + default:
> + return NULL;
> + }
> + }
> +
> + if (dp->num_vrfs >= 256) {
> + switch (nh_sz) {
> + case RTE_FIB_DIR24_8_1B:
> + return be_addr ?
> rte_dir24_8_vec_lookup_bulk_vrf_1b_be_large :
> + rte_dir24_8_vec_lookup_bulk_vrf_1b_large;
> + case RTE_FIB_DIR24_8_2B:
> + return be_addr ?
> rte_dir24_8_vec_lookup_bulk_vrf_2b_be_large :
> + rte_dir24_8_vec_lookup_bulk_vrf_2b_large;
> + case RTE_FIB_DIR24_8_4B:
> + return be_addr ?
> rte_dir24_8_vec_lookup_bulk_vrf_4b_be_large :
> + rte_dir24_8_vec_lookup_bulk_vrf_4b_large;
> + case RTE_FIB_DIR24_8_8B:
> + return be_addr ?
> rte_dir24_8_vec_lookup_bulk_vrf_8b_be_large :
> + rte_dir24_8_vec_lookup_bulk_vrf_8b_large;
> + default:
> + return NULL;
> + }
> + }
> +
> switch (nh_sz) {
> case RTE_FIB_DIR24_8_1B:
> - return be_addr ? rte_dir24_8_vec_lookup_bulk_1b_be :
> - rte_dir24_8_vec_lookup_bulk_1b;
> + return be_addr ? rte_dir24_8_vec_lookup_bulk_vrf_1b_be :
> + rte_dir24_8_vec_lookup_bulk_vrf_1b;
> case RTE_FIB_DIR24_8_2B:
> - return be_addr ? rte_dir24_8_vec_lookup_bulk_2b_be :
> - rte_dir24_8_vec_lookup_bulk_2b;
> + return be_addr ? rte_dir24_8_vec_lookup_bulk_vrf_2b_be :
> + rte_dir24_8_vec_lookup_bulk_vrf_2b;
> case RTE_FIB_DIR24_8_4B:
> - return be_addr ? rte_dir24_8_vec_lookup_bulk_4b_be :
> - rte_dir24_8_vec_lookup_bulk_4b;
> + return be_addr ? rte_dir24_8_vec_lookup_bulk_vrf_4b_be :
> + rte_dir24_8_vec_lookup_bulk_vrf_4b;
> case RTE_FIB_DIR24_8_8B:
> - return be_addr ? rte_dir24_8_vec_lookup_bulk_8b_be :
> - rte_dir24_8_vec_lookup_bulk_8b;
> + return be_addr ? rte_dir24_8_vec_lookup_bulk_vrf_8b_be :
> + rte_dir24_8_vec_lookup_bulk_vrf_8b;
> default:
> return NULL;
> }
> #elif defined(RTE_RISCV_FEATURE_V)
> RTE_SET_USED(be_addr);
> + RTE_SET_USED(dp);
> if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_RISCV_ISA_V) <= 0)
> return NULL;
> switch (nh_sz) {
> @@ -130,16 +208,17 @@ dir24_8_get_lookup_fn(void *p, enum
> rte_fib_lookup_type type, bool be_addr)
>
> switch (type) {
> case RTE_FIB_LOOKUP_DIR24_8_SCALAR_MACRO:
> - return get_scalar_fn(nh_sz, be_addr);
> + return get_scalar_fn(dp, nh_sz, be_addr);
> case RTE_FIB_LOOKUP_DIR24_8_SCALAR_INLINE:
> - return get_scalar_fn_inlined(nh_sz, be_addr);
> + return get_scalar_fn_inlined(dp, nh_sz, be_addr);
> case RTE_FIB_LOOKUP_DIR24_8_SCALAR_UNI:
> - return be_addr ? dir24_8_lookup_bulk_uni_be :
> dir24_8_lookup_bulk_uni;
> + return be_addr ? dir24_8_lookup_bulk_uni_be :
> + dir24_8_lookup_bulk_uni;
> case RTE_FIB_LOOKUP_DIR24_8_VECTOR_AVX512:
> - return get_vector_fn(nh_sz, be_addr);
> + return get_vector_fn(dp, nh_sz, be_addr);
> case RTE_FIB_LOOKUP_DEFAULT:
> - ret_fn = get_vector_fn(nh_sz, be_addr);
> - return ret_fn != NULL ? ret_fn : get_scalar_fn(nh_sz, be_addr);
> + ret_fn = get_vector_fn(dp, nh_sz, be_addr);
> + return ret_fn != NULL ? ret_fn : get_scalar_fn(dp, nh_sz,
> be_addr);
> default:
> return NULL;
> }
> @@ -246,15 +325,18 @@ __rcu_qsbr_free_resource(void *p, void *data,
> unsigned int n __rte_unused)
> }
>
> static void
> -tbl8_recycle(struct dir24_8_tbl *dp, uint32_t ip, uint64_t tbl8_idx)
> +tbl8_recycle(struct dir24_8_tbl *dp, uint16_t vrf_id, uint32_t ip, uint64_t
> tbl8_idx)
> {
> uint32_t i;
> uint64_t nh;
> + uint64_t tbl24_idx;
> uint8_t *ptr8;
> uint16_t *ptr16;
> uint32_t *ptr32;
> uint64_t *ptr64;
>
> + tbl24_idx = get_tbl24_idx(vrf_id, ip);
> +
> switch (dp->nh_sz) {
> case RTE_FIB_DIR24_8_1B:
> ptr8 = &((uint8_t *)dp->tbl8)[tbl8_idx *
> @@ -264,7 +346,7 @@ tbl8_recycle(struct dir24_8_tbl *dp, uint32_t ip, uint64_t
> tbl8_idx)
> if (nh != ptr8[i])
> return;
> }
> - ((uint8_t *)dp->tbl24)[ip >> 8] =
> + ((uint8_t *)dp->tbl24)[tbl24_idx] =
> nh & ~DIR24_8_EXT_ENT;
> break;
> case RTE_FIB_DIR24_8_2B:
> @@ -275,7 +357,7 @@ tbl8_recycle(struct dir24_8_tbl *dp, uint32_t ip, uint64_t
> tbl8_idx)
> if (nh != ptr16[i])
> return;
> }
> - ((uint16_t *)dp->tbl24)[ip >> 8] =
> + ((uint16_t *)dp->tbl24)[tbl24_idx] =
> nh & ~DIR24_8_EXT_ENT;
> break;
> case RTE_FIB_DIR24_8_4B:
> @@ -286,7 +368,7 @@ tbl8_recycle(struct dir24_8_tbl *dp, uint32_t ip, uint64_t
> tbl8_idx)
> if (nh != ptr32[i])
> return;
> }
> - ((uint32_t *)dp->tbl24)[ip >> 8] =
> + ((uint32_t *)dp->tbl24)[tbl24_idx] =
> nh & ~DIR24_8_EXT_ENT;
> break;
> case RTE_FIB_DIR24_8_8B:
> @@ -297,7 +379,7 @@ tbl8_recycle(struct dir24_8_tbl *dp, uint32_t ip, uint64_t
> tbl8_idx)
> if (nh != ptr64[i])
> return;
> }
> - ((uint64_t *)dp->tbl24)[ip >> 8] =
> + ((uint64_t *)dp->tbl24)[tbl24_idx] =
> nh & ~DIR24_8_EXT_ENT;
> break;
> }
> @@ -314,7 +396,7 @@ tbl8_recycle(struct dir24_8_tbl *dp, uint32_t ip, uint64_t
> tbl8_idx)
> }
>
> static int
> -install_to_fib(struct dir24_8_tbl *dp, uint32_t ledge, uint32_t redge,
> +install_to_fib(struct dir24_8_tbl *dp, uint16_t vrf_id, uint32_t ledge, uint32_t
> redge,
> uint64_t next_hop)
> {
> uint64_t tbl24_tmp;
> @@ -328,7 +410,7 @@ install_to_fib(struct dir24_8_tbl *dp, uint32_t ledge,
> uint32_t redge,
>
> if (((ledge >> 8) != (redge >> 8)) || (len == 1 << 24)) {
> if ((ROUNDUP(ledge, 24) - ledge) != 0) {
> - tbl24_tmp = get_tbl24(dp, ledge, dp->nh_sz);
> + tbl24_tmp = get_tbl24(dp, vrf_id, ledge, dp->nh_sz);
> if ((tbl24_tmp & DIR24_8_EXT_ENT) !=
> DIR24_8_EXT_ENT) {
> /**
> @@ -346,7 +428,7 @@ install_to_fib(struct dir24_8_tbl *dp, uint32_t ledge,
> uint32_t redge,
> }
> tbl8_free_idx(dp, tmp_tbl8_idx);
> /*update dir24 entry with tbl8 index*/
> - write_to_fib(get_tbl24_p(dp, ledge,
> + write_to_fib(get_tbl24_p(dp, vrf_id, ledge,
> dp->nh_sz), (tbl8_idx << 1)|
> DIR24_8_EXT_ENT,
> dp->nh_sz, 1);
> @@ -360,19 +442,19 @@ install_to_fib(struct dir24_8_tbl *dp, uint32_t ledge,
> uint32_t redge,
> write_to_fib((void *)tbl8_ptr, (next_hop << 1)|
> DIR24_8_EXT_ENT,
> dp->nh_sz, ROUNDUP(ledge, 24) - ledge);
> - tbl8_recycle(dp, ledge, tbl8_idx);
> + tbl8_recycle(dp, vrf_id, ledge, tbl8_idx);
> }
> - write_to_fib(get_tbl24_p(dp, ROUNDUP(ledge, 24), dp->nh_sz),
> + write_to_fib(get_tbl24_p(dp, vrf_id, ROUNDUP(ledge, 24), dp-
> >nh_sz),
> next_hop << 1, dp->nh_sz, len);
> if (redge & ~DIR24_8_TBL24_MASK) {
> - tbl24_tmp = get_tbl24(dp, redge, dp->nh_sz);
> + tbl24_tmp = get_tbl24(dp, vrf_id, redge, dp->nh_sz);
> if ((tbl24_tmp & DIR24_8_EXT_ENT) !=
> DIR24_8_EXT_ENT) {
> tbl8_idx = tbl8_alloc(dp, tbl24_tmp);
> if (tbl8_idx < 0)
> return -ENOSPC;
> /*update dir24 entry with tbl8 index*/
> - write_to_fib(get_tbl24_p(dp, redge,
> + write_to_fib(get_tbl24_p(dp, vrf_id, redge,
> dp->nh_sz), (tbl8_idx << 1)|
> DIR24_8_EXT_ENT,
> dp->nh_sz, 1);
> @@ -385,17 +467,17 @@ install_to_fib(struct dir24_8_tbl *dp, uint32_t ledge,
> uint32_t redge,
> write_to_fib((void *)tbl8_ptr, (next_hop << 1)|
> DIR24_8_EXT_ENT,
> dp->nh_sz, redge & ~DIR24_8_TBL24_MASK);
> - tbl8_recycle(dp, redge, tbl8_idx);
> + tbl8_recycle(dp, vrf_id, redge, tbl8_idx);
> }
> } else if ((redge - ledge) != 0) {
> - tbl24_tmp = get_tbl24(dp, ledge, dp->nh_sz);
> + tbl24_tmp = get_tbl24(dp, vrf_id, ledge, dp->nh_sz);
> if ((tbl24_tmp & DIR24_8_EXT_ENT) !=
> DIR24_8_EXT_ENT) {
> tbl8_idx = tbl8_alloc(dp, tbl24_tmp);
> if (tbl8_idx < 0)
> return -ENOSPC;
> /*update dir24 entry with tbl8 index*/
> - write_to_fib(get_tbl24_p(dp, ledge, dp->nh_sz),
> + write_to_fib(get_tbl24_p(dp, vrf_id, ledge, dp->nh_sz),
> (tbl8_idx << 1)|
> DIR24_8_EXT_ENT,
> dp->nh_sz, 1);
> @@ -409,13 +491,13 @@ install_to_fib(struct dir24_8_tbl *dp, uint32_t ledge,
> uint32_t redge,
> write_to_fib((void *)tbl8_ptr, (next_hop << 1)|
> DIR24_8_EXT_ENT,
> dp->nh_sz, redge - ledge);
> - tbl8_recycle(dp, ledge, tbl8_idx);
> + tbl8_recycle(dp, vrf_id, ledge, tbl8_idx);
> }
> return 0;
> }
>
> static int
> -modify_fib(struct dir24_8_tbl *dp, struct rte_rib *rib, uint32_t ip,
> +modify_fib(struct dir24_8_tbl *dp, struct rte_rib *rib, uint16_t vrf_id, uint32_t
> ip,
> uint8_t depth, uint64_t next_hop)
> {
> struct rte_rib_node *tmp = NULL;
> @@ -438,7 +520,7 @@ modify_fib(struct dir24_8_tbl *dp, struct rte_rib *rib,
> uint32_t ip,
> (uint32_t)(1ULL << (32 - tmp_depth));
> continue;
> }
> - ret = install_to_fib(dp, ledge, redge,
> + ret = install_to_fib(dp, vrf_id, ledge, redge,
> next_hop);
> if (ret != 0)
> return ret;
> @@ -454,7 +536,7 @@ modify_fib(struct dir24_8_tbl *dp, struct rte_rib *rib,
> uint32_t ip,
> redge = ip + (uint32_t)(1ULL << (32 - depth));
> if (ledge == redge && ledge != 0)
> break;
> - ret = install_to_fib(dp, ledge, redge,
> + ret = install_to_fib(dp, vrf_id, ledge, redge,
> next_hop);
> if (ret != 0)
> return ret;
> @@ -465,7 +547,7 @@ modify_fib(struct dir24_8_tbl *dp, struct rte_rib *rib,
> uint32_t ip,
> }
>
> int
> -dir24_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t depth,
> +dir24_8_modify(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip, uint8_t depth,
> uint64_t next_hop, int op)
> {
> struct dir24_8_tbl *dp;
> @@ -480,8 +562,13 @@ dir24_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t
> depth,
> return -EINVAL;
>
> dp = rte_fib_get_dp(fib);
> - rib = rte_fib_get_rib(fib);
> - RTE_ASSERT((dp != NULL) && (rib != NULL));
> + RTE_ASSERT(dp != NULL);
> +
> + if (vrf_id >= dp->num_vrfs)
> + return -EINVAL;
> +
> + rib = rte_fib_vrf_get_rib(fib, vrf_id);
> + RTE_ASSERT(rib != NULL);
>
> if (next_hop > get_max_nh(dp->nh_sz))
> return -EINVAL;
> @@ -495,7 +582,7 @@ dir24_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t
> depth,
> rte_rib_get_nh(node, &node_nh);
> if (node_nh == next_hop)
> return 0;
> - ret = modify_fib(dp, rib, ip, depth, next_hop);
> + ret = modify_fib(dp, rib, vrf_id, ip, depth, next_hop);
> if (ret == 0)
> rte_rib_set_nh(node, next_hop);
> return 0;
> @@ -518,7 +605,7 @@ dir24_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t
> depth,
> if (par_nh == next_hop)
> goto successfully_added;
> }
> - ret = modify_fib(dp, rib, ip, depth, next_hop);
> + ret = modify_fib(dp, rib, vrf_id, ip, depth, next_hop);
> if (ret != 0) {
> rte_rib_remove(rib, ip, depth);
> return ret;
> @@ -536,9 +623,9 @@ dir24_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t
> depth,
> rte_rib_get_nh(parent, &par_nh);
> rte_rib_get_nh(node, &node_nh);
> if (par_nh != node_nh)
> - ret = modify_fib(dp, rib, ip, depth, par_nh);
> + ret = modify_fib(dp, rib, vrf_id, ip, depth,
> par_nh);
> } else
> - ret = modify_fib(dp, rib, ip, depth, dp->def_nh);
> + ret = modify_fib(dp, rib, vrf_id, ip, depth, dp-
> >def_nh[vrf_id]);
> if (ret == 0) {
> rte_rib_remove(rib, ip, depth);
> if (depth > 24) {
> @@ -562,7 +649,10 @@ dir24_8_create(const char *name, int socket_id, struct
> rte_fib_conf *fib_conf)
> struct dir24_8_tbl *dp;
> uint64_t def_nh;
> uint32_t num_tbl8;
> + uint16_t num_vrfs;
> enum rte_fib_dir24_8_nh_sz nh_sz;
> + uint64_t tbl24_sz;
> + uint16_t vrf;
>
> if ((name == NULL) || (fib_conf == NULL) ||
> (fib_conf->dir24_8.nh_sz < RTE_FIB_DIR24_8_1B) ||
> @@ -580,19 +670,56 @@ dir24_8_create(const char *name, int socket_id, struct
> rte_fib_conf *fib_conf)
> nh_sz = fib_conf->dir24_8.nh_sz;
> num_tbl8 = RTE_ALIGN_CEIL(fib_conf->dir24_8.num_tbl8,
> BITMAP_SLAB_BIT_SIZE);
> + num_vrfs = (fib_conf->max_vrfs == 0) ? 1 : fib_conf->max_vrfs;
> +
> + /* Validate per-VRF default nexthops if provided */
> + if (fib_conf->vrf_default_nh != NULL) {
> + for (vrf = 0; vrf < num_vrfs; vrf++) {
> + if (fib_conf->vrf_default_nh[vrf] > get_max_nh(nh_sz)) {
> + rte_errno = EINVAL;
> + return NULL;
> + }
> + }
> + }
> +
> + tbl24_sz = (uint64_t)num_vrfs * DIR24_8_TBL24_NUM_ENT * (1 <<
> nh_sz);
>
> snprintf(mem_name, sizeof(mem_name), "DP_%s", name);
> dp = rte_zmalloc_socket(name, sizeof(struct dir24_8_tbl) +
> - DIR24_8_TBL24_NUM_ENT * (1 << nh_sz) + sizeof(uint32_t),
> + tbl24_sz + sizeof(uint32_t),
> RTE_CACHE_LINE_SIZE, socket_id);
> if (dp == NULL) {
> rte_errno = ENOMEM;
> return NULL;
> }
>
> - /* Init table with default value */
> - write_to_fib(dp->tbl24, (def_nh << 1), nh_sz, 1 << 24);
> + dp->num_vrfs = num_vrfs;
> + dp->nh_sz = nh_sz;
> + dp->number_tbl8s = num_tbl8;
> +
> + /* Allocate per-VRF default nexthop array */
> + snprintf(mem_name, sizeof(mem_name), "DEFNH_%p", dp);
> + dp->def_nh = rte_zmalloc_socket(mem_name, num_vrfs *
> sizeof(uint64_t),
> + RTE_CACHE_LINE_SIZE, socket_id);
> + if (dp->def_nh == NULL) {
> + rte_errno = ENOMEM;
> + rte_free(dp);
> + return NULL;
> + }
> +
> + /* Initialize all VRFs with default nexthop */
> + for (vrf = 0; vrf < num_vrfs; vrf++) {
> + uint64_t vrf_def_nh = (fib_conf->vrf_default_nh != NULL) ?
> + fib_conf->vrf_default_nh[vrf] : def_nh;
> + dp->def_nh[vrf] = vrf_def_nh;
>
> + /* Init TBL24 for this VRF with default value */
> + uint64_t vrf_offset = (uint64_t)vrf * DIR24_8_TBL24_NUM_ENT;
> + void *vrf_tbl24 = (void *)&((uint8_t *)dp->tbl24)[vrf_offset <<
> nh_sz];
> + write_to_fib(vrf_tbl24, (vrf_def_nh << 1), nh_sz, 1 << 24);
> + }
> +
> + /* Allocate shared TBL8 for all VRFs */
> snprintf(mem_name, sizeof(mem_name), "TBL8_%p", dp);
> uint64_t tbl8_sz = DIR24_8_TBL8_GRP_NUM_ENT * (1ULL << nh_sz) *
> (num_tbl8 + 1);
> @@ -600,12 +727,10 @@ dir24_8_create(const char *name, int socket_id, struct
> rte_fib_conf *fib_conf)
> RTE_CACHE_LINE_SIZE, socket_id);
> if (dp->tbl8 == NULL) {
> rte_errno = ENOMEM;
> + rte_free(dp->def_nh);
> rte_free(dp);
> return NULL;
> }
> - dp->def_nh = def_nh;
> - dp->nh_sz = nh_sz;
> - dp->number_tbl8s = num_tbl8;
>
> snprintf(mem_name, sizeof(mem_name), "TBL8_idxes_%p", dp);
> dp->tbl8_idxes = rte_zmalloc_socket(mem_name,
> @@ -614,6 +739,7 @@ dir24_8_create(const char *name, int socket_id, struct
> rte_fib_conf *fib_conf)
> if (dp->tbl8_idxes == NULL) {
> rte_errno = ENOMEM;
> rte_free(dp->tbl8);
> + rte_free(dp->def_nh);
> rte_free(dp);
> return NULL;
> }
> @@ -629,6 +755,7 @@ dir24_8_free(void *p)
> rte_rcu_qsbr_dq_delete(dp->dq);
> rte_free(dp->tbl8_idxes);
> rte_free(dp->tbl8);
> + rte_free(dp->def_nh);
> rte_free(dp);
> }
>
> diff --git a/lib/fib/dir24_8.h b/lib/fib/dir24_8.h
> index b343b5d686..37a73a3cc2 100644
> --- a/lib/fib/dir24_8.h
> +++ b/lib/fib/dir24_8.h
> @@ -12,6 +12,7 @@
> #include <rte_byteorder.h>
> #include <rte_prefetch.h>
> #include <rte_branch_prediction.h>
> +#include <rte_debug.h>
> #include <rte_rcu_qsbr.h>
>
> /**
> @@ -32,24 +33,19 @@ struct dir24_8_tbl {
> uint32_t number_tbl8s; /**< Total number of tbl8s */
> uint32_t rsvd_tbl8s; /**< Number of reserved tbl8s */
> uint32_t cur_tbl8s; /**< Current number of tbl8s */
> + uint16_t num_vrfs; /**< Number of VRFs */
> enum rte_fib_dir24_8_nh_sz nh_sz; /**< Size of nexthop entry */
> /* RCU config. */
> enum rte_fib_qsbr_mode rcu_mode;/* Blocking, defer queue. */
> struct rte_rcu_qsbr *v; /* RCU QSBR variable. */
> struct rte_rcu_qsbr_dq *dq; /* RCU QSBR defer queue. */
> - uint64_t def_nh; /**< Default next hop */
> + uint64_t *def_nh; /**< Per-VRF default next hop array */
> uint64_t *tbl8; /**< tbl8 table. */
> uint64_t *tbl8_idxes; /**< bitmap containing free tbl8 idxes*/
> /* tbl24 table. */
> alignas(RTE_CACHE_LINE_SIZE) uint64_t tbl24[];
> };
>
> -static inline void *
> -get_tbl24_p(struct dir24_8_tbl *dp, uint32_t ip, uint8_t nh_sz)
> -{
> - return (void *)&((uint8_t *)dp->tbl24)[(ip &
> - DIR24_8_TBL24_MASK) >> (8 - nh_sz)];
> -}
>
> static inline uint8_t
> bits_in_nh(uint8_t nh_sz)
> @@ -63,14 +59,21 @@ get_max_nh(uint8_t nh_sz)
> return ((1ULL << (bits_in_nh(nh_sz) - 1)) - 1);
> }
>
> -static inline uint32_t
> -get_tbl24_idx(uint32_t ip)
> +static inline uint64_t
> +get_tbl24_idx(uint16_t vrf_id, uint32_t ip)
> +{
> + return ((uint64_t)vrf_id << 24) + (ip >> 8);
> +}
> +
> +static inline void *
> +get_tbl24_p(struct dir24_8_tbl *dp, uint16_t vrf_id, uint32_t ip, uint8_t nh_sz)
> {
> - return ip >> 8;
> + uint64_t idx = get_tbl24_idx(vrf_id, ip);
> + return (void *)&((uint8_t *)dp->tbl24)[idx << nh_sz];
> }
>
> -static inline uint32_t
> -get_tbl8_idx(uint32_t res, uint32_t ip)
> +static inline uint64_t
> +get_tbl8_idx(uint64_t res, uint32_t ip)
> {
> return (res >> 1) * DIR24_8_TBL8_GRP_NUM_ENT + (uint8_t)ip;
> }
> @@ -87,17 +90,18 @@ get_psd_idx(uint32_t val, uint8_t nh_sz)
> return val & ((1 << (3 - nh_sz)) - 1);
> }
>
> -static inline uint32_t
> -get_tbl_idx(uint32_t val, uint8_t nh_sz)
> +static inline uint64_t
> +get_tbl_idx(uint64_t val, uint8_t nh_sz)
> {
> return val >> (3 - nh_sz);
> }
>
> static inline uint64_t
> -get_tbl24(struct dir24_8_tbl *dp, uint32_t ip, uint8_t nh_sz)
> +get_tbl24(struct dir24_8_tbl *dp, uint16_t vrf_id, uint32_t ip, uint8_t nh_sz)
> {
> - return ((dp->tbl24[get_tbl_idx(get_tbl24_idx(ip), nh_sz)] >>
> - (get_psd_idx(get_tbl24_idx(ip), nh_sz) *
> + uint64_t idx = get_tbl24_idx(vrf_id, ip);
> + return ((dp->tbl24[get_tbl_idx(idx, nh_sz)] >>
> + (get_psd_idx(idx, nh_sz) *
> bits_in_nh(nh_sz))) & lookup_msk(nh_sz));
> }
>
> @@ -115,62 +119,92 @@ is_entry_extended(uint64_t ent)
> return (ent & DIR24_8_EXT_ENT) == DIR24_8_EXT_ENT;
> }
>
> -#define LOOKUP_FUNC(suffix, type, bulk_prefetch, nh_sz) \
> -static inline void dir24_8_lookup_bulk_##suffix(void *p, const uint32_t *ips, \
> - uint64_t *next_hops, const unsigned int n) \
> -{ \
> - struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p; \
> - uint64_t tmp; \
> - uint32_t i; \
> - uint32_t prefetch_offset = \
> - RTE_MIN((unsigned int)bulk_prefetch, n); \
> - \
> - for (i = 0; i < prefetch_offset; i++) \
> - rte_prefetch0(get_tbl24_p(dp, ips[i], nh_sz)); \
> - for (i = 0; i < (n - prefetch_offset); i++) { \
> - rte_prefetch0(get_tbl24_p(dp, \
> - ips[i + prefetch_offset], nh_sz)); \
> - tmp = ((type *)dp->tbl24)[ips[i] >> 8]; \
> - if (unlikely(is_entry_extended(tmp))) \
> - tmp = ((type *)dp->tbl8)[(uint8_t)ips[i] + \
> - ((tmp >> 1) * DIR24_8_TBL8_GRP_NUM_ENT)]; \
> - next_hops[i] = tmp >> 1; \
> - } \
> - for (; i < n; i++) { \
> - tmp = ((type *)dp->tbl24)[ips[i] >> 8]; \
> - if (unlikely(is_entry_extended(tmp))) \
> - tmp = ((type *)dp->tbl8)[(uint8_t)ips[i] + \
> - ((tmp >> 1) * DIR24_8_TBL8_GRP_NUM_ENT)]; \
> - next_hops[i] = tmp >> 1; \
> - } \
> -} \
> -
> -LOOKUP_FUNC(1b, uint8_t, 5, 0)
> -LOOKUP_FUNC(2b, uint16_t, 6, 1)
> -LOOKUP_FUNC(4b, uint32_t, 15, 2)
> -LOOKUP_FUNC(8b, uint64_t, 12, 3)
> +
> +#define LOOKUP_FUNC(suffix, type, bulk_prefetch, nh_sz, is_vrf)
> \
> +static inline void dir24_8_lookup_bulk_##suffix(void *p, \
> + const uint16_t *vrf_ids, const uint32_t *ips, \
> + uint64_t *next_hops, const unsigned int n) \
> +{ \
> + struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p; \
> + uint64_t tmp; \
> + uint32_t i; \
> + uint32_t prefetch_offset = RTE_MIN((unsigned int)bulk_prefetch, n); \
> + \
> + if (!is_vrf) \
> + RTE_SET_USED(vrf_ids);
> \
> + \
> + for (i = 0; i < prefetch_offset; i++) { \
> + uint16_t vid = is_vrf ? vrf_ids[i] : 0; \
> + RTE_ASSERT(vid < dp->num_vrfs);
> \
> + rte_prefetch0(get_tbl24_p(dp, vid, ips[i], nh_sz)); \
> + } \
> + for (i = 0; i < (n - prefetch_offset); i++) { \
> + uint16_t vid = is_vrf ? vrf_ids[i] : 0; \
> + uint16_t vid_next = is_vrf ? vrf_ids[i + prefetch_offset] : 0; \
> + RTE_ASSERT(vid < dp->num_vrfs);
> \
> + RTE_ASSERT(vid_next < dp->num_vrfs);
> \
> + rte_prefetch0(get_tbl24_p(dp, vid_next,
> \
> + ips[i + prefetch_offset], nh_sz)); \
> + tmp = ((type *)dp->tbl24)[get_tbl24_idx(vid, ips[i])]; \
> + if (unlikely(is_entry_extended(tmp))) \
> + tmp = ((type *)dp->tbl8)[(uint8_t)ips[i] + \
> + ((tmp >> 1) * DIR24_8_TBL8_GRP_NUM_ENT)]; \
> + next_hops[i] = tmp >> 1; \
> + } \
> + for (; i < n; i++) { \
> + uint16_t vid = is_vrf ? vrf_ids[i] : 0; \
> + RTE_ASSERT(vid < dp->num_vrfs); \
> + tmp = ((type *)dp->tbl24)[get_tbl24_idx(vid, ips[i])]; \
> + if (unlikely(is_entry_extended(tmp))) \
> + tmp = ((type *)dp->tbl8)[(uint8_t)ips[i] + \
> + ((tmp >> 1) * DIR24_8_TBL8_GRP_NUM_ENT)]; \
> + next_hops[i] = tmp >> 1; \
> + } \
> +}
> +
> +LOOKUP_FUNC(1b, uint8_t, 5, 0, false)
> +LOOKUP_FUNC(2b, uint16_t, 6, 1, false)
> +LOOKUP_FUNC(4b, uint32_t, 15, 2, false)
> +LOOKUP_FUNC(8b, uint64_t, 12, 3, false)
> +LOOKUP_FUNC(vrf_1b, uint8_t, 5, 0, true)
> +LOOKUP_FUNC(vrf_2b, uint16_t, 6, 1, true)
> +LOOKUP_FUNC(vrf_4b, uint32_t, 15, 2, true)
> +LOOKUP_FUNC(vrf_8b, uint64_t, 12, 3, true)
>
> static inline void
> -dir24_8_lookup_bulk(struct dir24_8_tbl *dp, const uint32_t *ips,
> - uint64_t *next_hops, const unsigned int n, uint8_t nh_sz)
> +__dir24_8_lookup_bulk(struct dir24_8_tbl *dp, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n,
> + uint8_t nh_sz, bool is_vrf)
> {
> uint64_t tmp;
> uint32_t i;
> uint32_t prefetch_offset = RTE_MIN(15U, n);
>
> - for (i = 0; i < prefetch_offset; i++)
> - rte_prefetch0(get_tbl24_p(dp, ips[i], nh_sz));
> + if (!is_vrf)
> + RTE_SET_USED(vrf_ids);
> +
> + for (i = 0; i < prefetch_offset; i++) {
> + uint16_t vid = is_vrf ? vrf_ids[i] : 0;
> + RTE_ASSERT(vid < dp->num_vrfs);
> + rte_prefetch0(get_tbl24_p(dp, vid, ips[i], nh_sz));
> + }
> for (i = 0; i < (n - prefetch_offset); i++) {
> - rte_prefetch0(get_tbl24_p(dp, ips[i + prefetch_offset],
> - nh_sz));
> - tmp = get_tbl24(dp, ips[i], nh_sz);
> + uint16_t vid = is_vrf ? vrf_ids[i] : 0;
> + uint16_t vid_next = is_vrf ? vrf_ids[i + prefetch_offset] : 0;
> + RTE_ASSERT(vid < dp->num_vrfs);
> + RTE_ASSERT(vid_next < dp->num_vrfs);
> + rte_prefetch0(get_tbl24_p(dp, vid_next,
> + ips[i + prefetch_offset], nh_sz));
> + tmp = get_tbl24(dp, vid, ips[i], nh_sz);
> if (unlikely(is_entry_extended(tmp)))
> tmp = get_tbl8(dp, tmp, ips[i], nh_sz);
>
> next_hops[i] = tmp >> 1;
> }
> for (; i < n; i++) {
> - tmp = get_tbl24(dp, ips[i], nh_sz);
> + uint16_t vid = is_vrf ? vrf_ids[i] : 0;
> + RTE_ASSERT(vid < dp->num_vrfs);
> + tmp = get_tbl24(dp, vid, ips[i], nh_sz);
> if (unlikely(is_entry_extended(tmp)))
> tmp = get_tbl8(dp, tmp, ips[i], nh_sz);
>
> @@ -179,43 +213,79 @@ dir24_8_lookup_bulk(struct dir24_8_tbl *dp, const
> uint32_t *ips,
> }
>
> static inline void
> -dir24_8_lookup_bulk_0(void *p, const uint32_t *ips,
> +dir24_8_lookup_bulk_0(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
> uint64_t *next_hops, const unsigned int n)
> {
> struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
>
> - dir24_8_lookup_bulk(dp, ips, next_hops, n, 0);
> + __dir24_8_lookup_bulk(dp, vrf_ids, ips, next_hops, n, 0, false);
> +}
> +
> +static inline void
> +dir24_8_lookup_bulk_vrf_0(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
> +{
> + struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
> +
> + __dir24_8_lookup_bulk(dp, vrf_ids, ips, next_hops, n, 0, true);
> }
>
> static inline void
> -dir24_8_lookup_bulk_1(void *p, const uint32_t *ips,
> +dir24_8_lookup_bulk_1(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
> uint64_t *next_hops, const unsigned int n)
> {
> struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
>
> - dir24_8_lookup_bulk(dp, ips, next_hops, n, 1);
> + __dir24_8_lookup_bulk(dp, vrf_ids, ips, next_hops, n, 1, false);
> }
>
> static inline void
> -dir24_8_lookup_bulk_2(void *p, const uint32_t *ips,
> +dir24_8_lookup_bulk_vrf_1(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
> +{
> + struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
> +
> + __dir24_8_lookup_bulk(dp, vrf_ids, ips, next_hops, n, 1, true);
> +}
> +
> +static inline void
> +dir24_8_lookup_bulk_2(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
> uint64_t *next_hops, const unsigned int n)
> {
> struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
>
> - dir24_8_lookup_bulk(dp, ips, next_hops, n, 2);
> + __dir24_8_lookup_bulk(dp, vrf_ids, ips, next_hops, n, 2, false);
> }
>
> static inline void
> -dir24_8_lookup_bulk_3(void *p, const uint32_t *ips,
> +dir24_8_lookup_bulk_vrf_2(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
> +{
> + struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
> +
> + __dir24_8_lookup_bulk(dp, vrf_ids, ips, next_hops, n, 2, true);
> +}
> +
> +static inline void
> +dir24_8_lookup_bulk_3(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
> uint64_t *next_hops, const unsigned int n)
> {
> struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
>
> - dir24_8_lookup_bulk(dp, ips, next_hops, n, 3);
> + __dir24_8_lookup_bulk(dp, vrf_ids, ips, next_hops, n, 3, false);
> }
>
> static inline void
> -dir24_8_lookup_bulk_uni(void *p, const uint32_t *ips,
> +dir24_8_lookup_bulk_vrf_3(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
> +{
> + struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
> +
> + __dir24_8_lookup_bulk(dp, vrf_ids, ips, next_hops, n, 3, true);
> +}
> +
> +static inline void
> +dir24_8_lookup_bulk_uni(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
> uint64_t *next_hops, const unsigned int n)
> {
> struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
> @@ -224,66 +294,83 @@ dir24_8_lookup_bulk_uni(void *p, const uint32_t *ips,
> uint32_t prefetch_offset = RTE_MIN(15U, n);
> uint8_t nh_sz = dp->nh_sz;
>
> - for (i = 0; i < prefetch_offset; i++)
> - rte_prefetch0(get_tbl24_p(dp, ips[i], nh_sz));
> + for (i = 0; i < prefetch_offset; i++) {
> + uint16_t vid = vrf_ids[i];
> + RTE_ASSERT(vid < dp->num_vrfs);
> + rte_prefetch0(get_tbl24_p(dp, vid, ips[i], nh_sz));
> + }
> for (i = 0; i < (n - prefetch_offset); i++) {
> - rte_prefetch0(get_tbl24_p(dp, ips[i + prefetch_offset],
> - nh_sz));
> - tmp = get_tbl24(dp, ips[i], nh_sz);
> + uint16_t vid = vrf_ids[i];
> + uint16_t vid_next = vrf_ids[i + prefetch_offset];
> + RTE_ASSERT(vid < dp->num_vrfs);
> + RTE_ASSERT(vid_next < dp->num_vrfs);
> + rte_prefetch0(get_tbl24_p(dp, vid_next,
> + ips[i + prefetch_offset], nh_sz));
> + tmp = get_tbl24(dp, vid, ips[i], nh_sz);
> if (unlikely(is_entry_extended(tmp)))
> tmp = get_tbl8(dp, tmp, ips[i], nh_sz);
>
> next_hops[i] = tmp >> 1;
> }
> for (; i < n; i++) {
> - tmp = get_tbl24(dp, ips[i], nh_sz);
> + uint16_t vid = vrf_ids[i];
> + RTE_ASSERT(vid < dp->num_vrfs);
> + tmp = get_tbl24(dp, vid, ips[i], nh_sz);
> if (unlikely(is_entry_extended(tmp)))
> tmp = get_tbl8(dp, tmp, ips[i], nh_sz);
>
> next_hops[i] = tmp >> 1;
> }
> }
> -
> #define BSWAP_MAX_LENGTH 64
>
> -typedef void (*dir24_8_lookup_bulk_be_cb)(void *p, const uint32_t *ips,
> uint64_t *next_hops,
> - const unsigned int n);
> +typedef void (*dir24_8_lookup_bulk_be_cb)(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
>
> static inline void
> -dir24_8_lookup_bulk_be(void *p, const uint32_t *ips, uint64_t *next_hops,
> const unsigned int n,
> - dir24_8_lookup_bulk_be_cb cb)
> +dir24_8_lookup_bulk_be(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
> + uint64_t *next_hops, const unsigned int n, dir24_8_lookup_bulk_be_cb
> cb)
> {
> uint32_t le_ips[BSWAP_MAX_LENGTH];
> unsigned int i;
>
> #if RTE_BYTE_ORDER == RTE_BIG_ENDIAN
> - cb(p, ips, next_hops, n);
> + cb(p, vrf_ids, ips, next_hops, n);
> #else
> for (i = 0; i < n; i += BSWAP_MAX_LENGTH) {
> int j;
> for (j = 0; j < BSWAP_MAX_LENGTH && i + j < n; j++)
> le_ips[j] = rte_be_to_cpu_32(ips[i + j]);
>
> - cb(p, le_ips, next_hops + i, j);
> + cb(p, vrf_ids + i, le_ips, next_hops + i, j);
> }
> #endif
> }
>
> #define DECLARE_BE_LOOKUP_FN(name) \
> static inline void \
> -name##_be(void *p, const uint32_t *ips, uint64_t *next_hops, const unsigned
> int n) \
> +name##_be(void *p, const uint16_t *vrf_ids, const uint32_t *ips, \
> + uint64_t *next_hops, const unsigned int n) \
> { \
> - dir24_8_lookup_bulk_be(p, ips, next_hops, n, name); \
> + dir24_8_lookup_bulk_be(p, vrf_ids, ips, next_hops, n, name); \
> }
>
> DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_1b)
> DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_2b)
> DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_4b)
> DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_8b)
> +DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_vrf_1b)
> +DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_vrf_2b)
> +DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_vrf_4b)
> +DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_vrf_8b)
> DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_0)
> DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_1)
> DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_2)
> DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_3)
> +DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_vrf_0)
> +DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_vrf_1)
> +DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_vrf_2)
> +DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_vrf_3)
> DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_uni)
>
> void *
> @@ -296,7 +383,7 @@ rte_fib_lookup_fn_t
> dir24_8_get_lookup_fn(void *p, enum rte_fib_lookup_type type, bool be_addr);
>
> int
> -dir24_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t depth,
> +dir24_8_modify(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip, uint8_t depth,
> uint64_t next_hop, int op);
>
> int
> diff --git a/lib/fib/dir24_8_avx512.c b/lib/fib/dir24_8_avx512.c
> index 89b43583c7..3e576e410e 100644
> --- a/lib/fib/dir24_8_avx512.c
> +++ b/lib/fib/dir24_8_avx512.c
> @@ -4,75 +4,132 @@
>
> #include <rte_vect.h>
> #include <rte_fib.h>
> +#include <rte_debug.h>
>
> #include "dir24_8.h"
> #include "dir24_8_avx512.h"
>
> +enum vrf_scale {
> + VRF_SCALE_SINGLE = 0,
> + VRF_SCALE_SMALL = 1,
> + VRF_SCALE_LARGE = 2,
> +};
> +
> static __rte_always_inline void
> -dir24_8_vec_lookup_x16(void *p, const uint32_t *ips,
> - uint64_t *next_hops, int size, bool be_addr)
> +dir24_8_vec_lookup_x8_64b_path(struct dir24_8_tbl *dp, __m256i ip_vec_256,
> + __m256i vrf32_256, uint64_t *next_hops, int size)
> {
> - struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
> - __mmask16 msk_ext;
> - __mmask16 exp_msk = 0x5555;
> - __m512i ip_vec, idxes, res, bytes;
> - const __m512i zero = _mm512_set1_epi32(0);
> - const __m512i lsb = _mm512_set1_epi32(1);
> - const __m512i lsbyte_msk = _mm512_set1_epi32(0xff);
> - __m512i tmp1, tmp2, res_msk;
> - __m256i tmp256;
> - /* used to mask gather values if size is 1/2 (8/16 bit next hops) */
> + const __m512i zero_64 = _mm512_set1_epi64(0);
> + const __m512i lsb_64 = _mm512_set1_epi64(1);
> + const __m512i lsbyte_msk_64 = _mm512_set1_epi64(0xff);
> + __m512i res_msk_64, vrf64, idxes_64, res, bytes_64;
> + __mmask8 msk_ext_64;
> +
> if (size == sizeof(uint8_t))
> - res_msk = _mm512_set1_epi32(UINT8_MAX);
> + res_msk_64 = _mm512_set1_epi64(UINT8_MAX);
> else if (size == sizeof(uint16_t))
> - res_msk = _mm512_set1_epi32(UINT16_MAX);
> + res_msk_64 = _mm512_set1_epi64(UINT16_MAX);
> + else if (size == sizeof(uint32_t))
> + res_msk_64 = _mm512_set1_epi64(UINT32_MAX);
>
> - ip_vec = _mm512_loadu_si512(ips);
> - if (be_addr) {
> - const __m512i bswap32 = _mm512_set_epi32(
> - 0x0c0d0e0f, 0x08090a0b, 0x04050607, 0x00010203,
> - 0x0c0d0e0f, 0x08090a0b, 0x04050607, 0x00010203,
> - 0x0c0d0e0f, 0x08090a0b, 0x04050607, 0x00010203,
> - 0x0c0d0e0f, 0x08090a0b, 0x04050607, 0x00010203
> - );
> - ip_vec = _mm512_shuffle_epi8(ip_vec, bswap32);
> + vrf64 = _mm512_cvtepu32_epi64(vrf32_256);
> +
> + /* Compute index: (vrf_id << 24) + (ip >> 8) using 64-bit shift */
> + idxes_64 = _mm512_slli_epi64(vrf64, 24);
> + idxes_64 = _mm512_add_epi64(idxes_64, _mm512_cvtepu32_epi64(
> + _mm256_srli_epi32(ip_vec_256, 8)));
> +
> + /* lookup in tbl24 */
> + if (size == sizeof(uint8_t)) {
> + res = _mm512_i64gather_epi64(idxes_64, (const void *)dp-
> >tbl24, 1);
> + res = _mm512_and_epi64(res, res_msk_64);
> + } else if (size == sizeof(uint16_t)) {
> + res = _mm512_i64gather_epi64(idxes_64, (const void *)dp-
> >tbl24, 2);
> + res = _mm512_and_epi64(res, res_msk_64);
> + } else {
> + res = _mm512_i64gather_epi64(idxes_64, (const void *)dp-
> >tbl24, 4);
> + res = _mm512_and_epi64(res, res_msk_64);
> + }
> +
> + /* get extended entries indexes */
> + msk_ext_64 = _mm512_test_epi64_mask(res, lsb_64);
> +
> + if (msk_ext_64 != 0) {
> + bytes_64 = _mm512_cvtepu32_epi64(ip_vec_256);
> + idxes_64 = _mm512_srli_epi64(res, 1);
> + idxes_64 = _mm512_slli_epi64(idxes_64, 8);
> + bytes_64 = _mm512_and_epi64(bytes_64, lsbyte_msk_64);
> + idxes_64 = _mm512_maskz_add_epi64(msk_ext_64, idxes_64,
> bytes_64);
> +
> + if (size == sizeof(uint8_t))
> + idxes_64 = _mm512_mask_i64gather_epi64(zero_64,
> msk_ext_64,
> + idxes_64, (const void *)dp->tbl8, 1);
> + else if (size == sizeof(uint16_t))
> + idxes_64 = _mm512_mask_i64gather_epi64(zero_64,
> msk_ext_64,
> + idxes_64, (const void *)dp->tbl8, 2);
> + else
> + idxes_64 = _mm512_mask_i64gather_epi64(zero_64,
> msk_ext_64,
> + idxes_64, (const void *)dp->tbl8, 4);
> +
> + res = _mm512_mask_blend_epi64(msk_ext_64, res, idxes_64);
> }
>
> - /* mask 24 most significant bits */
> - idxes = _mm512_srli_epi32(ip_vec, 8);
> + res = _mm512_srli_epi64(res, 1);
> + _mm512_storeu_si512(next_hops, res);
> +}
> +
> +static __rte_always_inline void
> +dir24_8_vec_lookup_x16_32b_path(struct dir24_8_tbl *dp, __m512i ip_vec,
> + __m512i idxes, uint64_t *next_hops, int size)
> +{
> + __mmask16 msk_ext;
> + const __mmask16 exp_msk = 0x5555;
> + const __m512i zero_32 = _mm512_set1_epi32(0);
> + const __m512i lsb_32 = _mm512_set1_epi32(1);
> + const __m512i lsbyte_msk_32 = _mm512_set1_epi32(0xff);
> + __m512i res, bytes, tmp1, tmp2;
> + __m256i tmp256;
> + __m512i res_msk_32;
> +
> + if (size == sizeof(uint8_t))
> + res_msk_32 = _mm512_set1_epi32(UINT8_MAX);
> + else if (size == sizeof(uint16_t))
> + res_msk_32 = _mm512_set1_epi32(UINT16_MAX);
>
> - /**
> + /*
> * lookup in tbl24
> * Put it inside branch to make compiler happy with -O0
> */
> if (size == sizeof(uint8_t)) {
> res = _mm512_i32gather_epi32(idxes, (const int *)dp->tbl24, 1);
> - res = _mm512_and_epi32(res, res_msk);
> + res = _mm512_and_epi32(res, res_msk_32);
> } else if (size == sizeof(uint16_t)) {
> res = _mm512_i32gather_epi32(idxes, (const int *)dp->tbl24, 2);
> - res = _mm512_and_epi32(res, res_msk);
> - } else
> + res = _mm512_and_epi32(res, res_msk_32);
> + } else {
> res = _mm512_i32gather_epi32(idxes, (const int *)dp->tbl24, 4);
> + }
>
> /* get extended entries indexes */
> - msk_ext = _mm512_test_epi32_mask(res, lsb);
> + msk_ext = _mm512_test_epi32_mask(res, lsb_32);
>
> if (msk_ext != 0) {
> idxes = _mm512_srli_epi32(res, 1);
> idxes = _mm512_slli_epi32(idxes, 8);
> - bytes = _mm512_and_epi32(ip_vec, lsbyte_msk);
> + bytes = _mm512_and_epi32(ip_vec, lsbyte_msk_32);
> idxes = _mm512_maskz_add_epi32(msk_ext, idxes, bytes);
> if (size == sizeof(uint8_t)) {
> - idxes = _mm512_mask_i32gather_epi32(zero, msk_ext,
> + idxes = _mm512_mask_i32gather_epi32(zero_32,
> msk_ext,
> idxes, (const int *)dp->tbl8, 1);
> - idxes = _mm512_and_epi32(idxes, res_msk);
> + idxes = _mm512_and_epi32(idxes, res_msk_32);
> } else if (size == sizeof(uint16_t)) {
> - idxes = _mm512_mask_i32gather_epi32(zero, msk_ext,
> + idxes = _mm512_mask_i32gather_epi32(zero_32,
> msk_ext,
> idxes, (const int *)dp->tbl8, 2);
> - idxes = _mm512_and_epi32(idxes, res_msk);
> - } else
> - idxes = _mm512_mask_i32gather_epi32(zero, msk_ext,
> + idxes = _mm512_and_epi32(idxes, res_msk_32);
> + } else {
> + idxes = _mm512_mask_i32gather_epi32(zero_32,
> msk_ext,
> idxes, (const int *)dp->tbl8, 4);
> + }
>
> res = _mm512_mask_blend_epi32(msk_ext, res, idxes);
> }
> @@ -86,16 +143,74 @@ dir24_8_vec_lookup_x16(void *p, const uint32_t *ips,
> _mm512_storeu_si512(next_hops + 8, tmp2);
> }
>
> +/* Unified function with vrf_scale parameter similar to be_addr */
> +static __rte_always_inline void
> +dir24_8_vec_lookup_x16(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
> + uint64_t *next_hops, int size, bool be_addr, enum vrf_scale vrf_scale)
> +{
> + struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
> + __m512i ip_vec, idxes;
> + __m256i ip_vec_256, vrf32_256;
> +
> + ip_vec = _mm512_loadu_si512(ips);
> + if (be_addr) {
> + const __m512i bswap32 = _mm512_set_epi32(
> + 0x0c0d0e0f, 0x08090a0b, 0x04050607, 0x00010203,
> + 0x0c0d0e0f, 0x08090a0b, 0x04050607, 0x00010203,
> + 0x0c0d0e0f, 0x08090a0b, 0x04050607, 0x00010203,
> + 0x0c0d0e0f, 0x08090a0b, 0x04050607, 0x00010203
> + );
> + ip_vec = _mm512_shuffle_epi8(ip_vec, bswap32);
> + }
> +
> + if (vrf_scale == VRF_SCALE_SINGLE) {
> + /* mask 24 most significant bits */
> + idxes = _mm512_srli_epi32(ip_vec, 8);
> + dir24_8_vec_lookup_x16_32b_path(dp, ip_vec, idxes, next_hops,
> size);
> + } else if (vrf_scale == VRF_SCALE_SMALL) {
> + /* For < 256 VRFs: use 32-bit indices with 32-bit shift */
> + __m512i vrf32;
> + uint32_t i;
> +
> + for (i = 0; i < 16; i++)
> + RTE_ASSERT(vrf_ids[i] < dp->num_vrfs);
> +
> + vrf32 = _mm512_cvtepu16_epi32(_mm256_loadu_si256((const
> void *)vrf_ids));
> +
> + /* mask 24 most significant bits */
> + idxes = _mm512_srli_epi32(ip_vec, 8);
> + idxes = _mm512_add_epi32(idxes, _mm512_slli_epi32(vrf32,
> 24));
> + dir24_8_vec_lookup_x16_32b_path(dp, ip_vec, idxes, next_hops,
> size);
> + } else {
> + /* For >= 256 VRFs: use 64-bit indices to avoid overflow */
> + uint32_t i;
> +
> + for (i = 0; i < 16; i++)
> + RTE_ASSERT(vrf_ids[i] < dp->num_vrfs);
> +
> + /* Extract first 8 IPs and VRF IDs */
> + ip_vec_256 = _mm512_castsi512_si256(ip_vec);
> + vrf32_256 = _mm256_cvtepu16_epi32(_mm_loadu_si128((const
> void *)vrf_ids));
> + dir24_8_vec_lookup_x8_64b_path(dp, ip_vec_256, vrf32_256,
> next_hops, size);
> +
> + /* Process next 8 IPs from the second half of the vector */
> + ip_vec_256 = _mm512_extracti32x8_epi32(ip_vec, 1);
> + vrf32_256 = _mm256_cvtepu16_epi32(_mm_loadu_si128((const
> void *)(vrf_ids + 8)));
> + dir24_8_vec_lookup_x8_64b_path(dp, ip_vec_256, vrf32_256,
> next_hops + 8, size);
> + }
> +}
> +
> +/* Unified function with vrf_scale parameter */
> static __rte_always_inline void
> -dir24_8_vec_lookup_x8_8b(void *p, const uint32_t *ips,
> - uint64_t *next_hops, bool be_addr)
> +dir24_8_vec_lookup_x8_8b(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, bool be_addr, enum vrf_scale
> vrf_scale)
> {
> struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
> - const __m512i zero = _mm512_set1_epi32(0);
> - const __m512i lsbyte_msk = _mm512_set1_epi64(0xff);
> - const __m512i lsb = _mm512_set1_epi64(1);
> + const __m512i zero_64 = _mm512_set1_epi64(0);
> + const __m512i lsbyte_msk_64 = _mm512_set1_epi64(0xff);
> + const __m512i lsb_64 = _mm512_set1_epi64(1);
> __m512i res, idxes, bytes;
> - __m256i idxes_256, ip_vec;
> + __m256i ip_vec, vrf32_256;
> __mmask8 msk_ext;
>
> ip_vec = _mm256_loadu_si256((const void *)ips);
> @@ -106,66 +221,207 @@ dir24_8_vec_lookup_x8_8b(void *p, const uint32_t
> *ips,
> );
> ip_vec = _mm256_shuffle_epi8(ip_vec, bswap32);
> }
> - /* mask 24 most significant bits */
> - idxes_256 = _mm256_srli_epi32(ip_vec, 8);
>
> - /* lookup in tbl24 */
> - res = _mm512_i32gather_epi64(idxes_256, (const void *)dp->tbl24, 8);
> + if (vrf_scale == VRF_SCALE_SINGLE) {
> + /* For single VRF: use 32-bit indices without vrf_ids */
> + __m256i idxes_256;
>
> - /* get extended entries indexes */
> - msk_ext = _mm512_test_epi64_mask(res, lsb);
> + /* mask 24 most significant bits */
> + idxes_256 = _mm256_srli_epi32(ip_vec, 8);
>
> - if (msk_ext != 0) {
> - bytes = _mm512_cvtepi32_epi64(ip_vec);
> - idxes = _mm512_srli_epi64(res, 1);
> - idxes = _mm512_slli_epi64(idxes, 8);
> - bytes = _mm512_and_epi64(bytes, lsbyte_msk);
> - idxes = _mm512_maskz_add_epi64(msk_ext, idxes, bytes);
> - idxes = _mm512_mask_i64gather_epi64(zero, msk_ext, idxes,
> - (const void *)dp->tbl8, 8);
> -
> - res = _mm512_mask_blend_epi64(msk_ext, res, idxes);
> - }
> + /* lookup in tbl24 */
> + res = _mm512_i32gather_epi64(idxes_256, (const void *)dp-
> >tbl24, 8);
>
> - res = _mm512_srli_epi64(res, 1);
> - _mm512_storeu_si512(next_hops, res);
> + /* get extended entries indexes */
> + msk_ext = _mm512_test_epi64_mask(res, lsb_64);
> +
> + if (msk_ext != 0) {
> + bytes = _mm512_cvtepu32_epi64(ip_vec);
> + idxes = _mm512_srli_epi64(res, 1);
> + idxes = _mm512_slli_epi64(idxes, 8);
> + bytes = _mm512_and_epi64(bytes, lsbyte_msk_64);
> + idxes = _mm512_maskz_add_epi64(msk_ext, idxes,
> bytes);
> + idxes = _mm512_mask_i64gather_epi64(zero_64,
> msk_ext, idxes,
> + (const void *)dp->tbl8, 8);
> +
> + res = _mm512_mask_blend_epi64(msk_ext, res, idxes);
> + }
> +
> + res = _mm512_srli_epi64(res, 1);
> + _mm512_storeu_si512(next_hops, res);
> + } else if (vrf_scale == VRF_SCALE_SMALL) {
> + /* For < 256 VRFs: use 32-bit indices with 32-bit shift */
> + __m256i idxes_256;
> + uint32_t i;
> +
> + for (i = 0; i < 8; i++)
> + RTE_ASSERT(vrf_ids[i] < dp->num_vrfs);
> +
> + /* mask 24 most significant bits */
> + idxes_256 = _mm256_srli_epi32(ip_vec, 8);
> + vrf32_256 = _mm256_cvtepu16_epi32(_mm_loadu_si128((const
> void *)vrf_ids));
> + idxes_256 = _mm256_add_epi32(idxes_256,
> _mm256_slli_epi32(vrf32_256, 24));
> +
> + /* lookup in tbl24 */
> + res = _mm512_i32gather_epi64(idxes_256, (const void *)dp-
> >tbl24, 8);
> +
> + /* get extended entries indexes */
> + msk_ext = _mm512_test_epi64_mask(res, lsb_64);
> +
> + if (msk_ext != 0) {
> + bytes = _mm512_cvtepu32_epi64(ip_vec);
> + idxes = _mm512_srli_epi64(res, 1);
> + idxes = _mm512_slli_epi64(idxes, 8);
> + bytes = _mm512_and_epi64(bytes, lsbyte_msk_64);
> + idxes = _mm512_maskz_add_epi64(msk_ext, idxes,
> bytes);
> + idxes = _mm512_mask_i64gather_epi64(zero_64,
> msk_ext, idxes,
> + (const void *)dp->tbl8, 8);
> +
> + res = _mm512_mask_blend_epi64(msk_ext, res, idxes);
> + }
> +
> + res = _mm512_srli_epi64(res, 1);
> + _mm512_storeu_si512(next_hops, res);
> + } else {
> + /* For >= 256 VRFs: use 64-bit indices to avoid overflow */
> + uint32_t i;
> +
> + for (i = 0; i < 8; i++)
> + RTE_ASSERT(vrf_ids[i] < dp->num_vrfs);
> +
> + vrf32_256 = _mm256_cvtepu16_epi32(_mm_loadu_si128((const
> void *)vrf_ids));
> + __m512i vrf64 = _mm512_cvtepu32_epi64(vrf32_256);
> +
> + /* Compute index: (vrf_id << 24) + (ip >> 8) using 64-bit
> arithmetic */
> + idxes = _mm512_slli_epi64(vrf64, 24);
> + idxes = _mm512_add_epi64(idxes, _mm512_cvtepu32_epi64(
> + _mm256_srli_epi32(ip_vec, 8)));
> +
> + /* lookup in tbl24 with 64-bit gather */
> + res = _mm512_i64gather_epi64(idxes, (const void *)dp->tbl24, 8);
> +
> + /* get extended entries indexes */
> + msk_ext = _mm512_test_epi64_mask(res, lsb_64);
> +
> + if (msk_ext != 0) {
> + bytes = _mm512_cvtepu32_epi64(ip_vec);
> + idxes = _mm512_srli_epi64(res, 1);
> + idxes = _mm512_slli_epi64(idxes, 8);
> + bytes = _mm512_and_epi64(bytes, lsbyte_msk_64);
> + idxes = _mm512_maskz_add_epi64(msk_ext, idxes,
> bytes);
> + idxes = _mm512_mask_i64gather_epi64(zero_64,
> msk_ext, idxes,
> + (const void *)dp->tbl8, 8);
> +
> + res = _mm512_mask_blend_epi64(msk_ext, res, idxes);
> + }
> +
> + res = _mm512_srli_epi64(res, 1);
> + _mm512_storeu_si512(next_hops, res);
> + }
> }
>
> -#define DECLARE_VECTOR_FN(suffix, nh_type, be_addr) \
> +#define DECLARE_VECTOR_FN(suffix, scalar_suffix, nh_type, be_addr, vrf_scale)
> \
> void \
> -rte_dir24_8_vec_lookup_bulk_##suffix(void *p, const uint32_t *ips, uint64_t
> *next_hops, \
> - const unsigned int n) \
> +rte_dir24_8_vec_lookup_bulk_##suffix(void *p, const uint16_t *vrf_ids, \
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n) \
> { \
> uint32_t i; \
> for (i = 0; i < (n / 16); i++) \
> - dir24_8_vec_lookup_x16(p, ips + i * 16, next_hops + i * 16,
> sizeof(nh_type), \
> - be_addr); \
> - dir24_8_lookup_bulk_##suffix(p, ips + i * 16, next_hops + i * 16, n - i *
> 16); \
> + dir24_8_vec_lookup_x16(p, vrf_ids + i * 16, ips + i * 16, \
> + next_hops + i * 16, sizeof(nh_type), be_addr, vrf_scale); \
> + dir24_8_lookup_bulk_##scalar_suffix(p, vrf_ids + i * 16, ips + i * 16, \
> + next_hops + i * 16, n - i * 16); \
> +}
> +
> +DECLARE_VECTOR_FN(1b, 1b, uint8_t, false, VRF_SCALE_SINGLE)
> +DECLARE_VECTOR_FN(1b_be, 1b_be, uint8_t, true, VRF_SCALE_SINGLE)
> +DECLARE_VECTOR_FN(2b, 2b, uint16_t, false, VRF_SCALE_SINGLE)
> +DECLARE_VECTOR_FN(2b_be, 2b_be, uint16_t, true, VRF_SCALE_SINGLE)
> +DECLARE_VECTOR_FN(4b, 4b, uint32_t, false, VRF_SCALE_SINGLE)
> +DECLARE_VECTOR_FN(4b_be, 4b_be, uint32_t, true, VRF_SCALE_SINGLE)
> +
> +DECLARE_VECTOR_FN(vrf_1b, vrf_1b, uint8_t, false, VRF_SCALE_SMALL)
> +DECLARE_VECTOR_FN(vrf_1b_be, vrf_1b_be, uint8_t, true, VRF_SCALE_SMALL)
> +DECLARE_VECTOR_FN(vrf_2b, vrf_2b, uint16_t, false, VRF_SCALE_SMALL)
> +DECLARE_VECTOR_FN(vrf_2b_be, vrf_2b_be, uint16_t, true, VRF_SCALE_SMALL)
> +DECLARE_VECTOR_FN(vrf_4b, vrf_4b, uint32_t, false, VRF_SCALE_SMALL)
> +DECLARE_VECTOR_FN(vrf_4b_be, vrf_4b_be, uint32_t, true, VRF_SCALE_SMALL)
> +
> +DECLARE_VECTOR_FN(vrf_1b_large, vrf_1b, uint8_t, false, VRF_SCALE_LARGE)
> +DECLARE_VECTOR_FN(vrf_1b_be_large, vrf_1b_be, uint8_t, true,
> VRF_SCALE_LARGE)
> +DECLARE_VECTOR_FN(vrf_2b_large, vrf_2b, uint16_t, false, VRF_SCALE_LARGE)
> +DECLARE_VECTOR_FN(vrf_2b_be_large, vrf_2b_be, uint16_t, true,
> VRF_SCALE_LARGE)
> +DECLARE_VECTOR_FN(vrf_4b_large, vrf_4b, uint32_t, false, VRF_SCALE_LARGE)
> +DECLARE_VECTOR_FN(vrf_4b_be_large, vrf_4b_be, uint32_t, true,
> VRF_SCALE_LARGE)
> +
> +void
> +rte_dir24_8_vec_lookup_bulk_8b(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
> +{
> + uint32_t i;
> + for (i = 0; i < (n / 8); i++)
> + dir24_8_vec_lookup_x8_8b(p, vrf_ids + i * 8, ips + i * 8,
> + next_hops + i * 8, false, VRF_SCALE_SINGLE);
> + dir24_8_lookup_bulk_8b(p, vrf_ids + i * 8, ips + i * 8,
> + next_hops + i * 8, n - i * 8);
> +}
> +
> +void
> +rte_dir24_8_vec_lookup_bulk_8b_be(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
> +{
> + uint32_t i;
> + for (i = 0; i < (n / 8); i++)
> + dir24_8_vec_lookup_x8_8b(p, vrf_ids + i * 8, ips + i * 8,
> + next_hops + i * 8, true, VRF_SCALE_SINGLE);
> + dir24_8_lookup_bulk_8b_be(p, vrf_ids + i * 8, ips + i * 8,
> + next_hops + i * 8, n - i * 8);
> +}
> +
> +void
> +rte_dir24_8_vec_lookup_bulk_vrf_8b(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
> +{
> + uint32_t i;
> + for (i = 0; i < (n / 8); i++)
> + dir24_8_vec_lookup_x8_8b(p, vrf_ids + i * 8, ips + i * 8,
> + next_hops + i * 8, false, VRF_SCALE_SMALL);
> + dir24_8_lookup_bulk_vrf_8b(p, vrf_ids + i * 8, ips + i * 8,
> + next_hops + i * 8, n - i * 8);
> }
>
> -DECLARE_VECTOR_FN(1b, uint8_t, false)
> -DECLARE_VECTOR_FN(1b_be, uint8_t, true)
> -DECLARE_VECTOR_FN(2b, uint16_t, false)
> -DECLARE_VECTOR_FN(2b_be, uint16_t, true)
> -DECLARE_VECTOR_FN(4b, uint32_t, false)
> -DECLARE_VECTOR_FN(4b_be, uint32_t, true)
> +void
> +rte_dir24_8_vec_lookup_bulk_vrf_8b_be(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
> +{
> + uint32_t i;
> + for (i = 0; i < (n / 8); i++)
> + dir24_8_vec_lookup_x8_8b(p, vrf_ids + i * 8, ips + i * 8,
> + next_hops + i * 8, true, VRF_SCALE_SMALL);
> + dir24_8_lookup_bulk_vrf_8b_be(p, vrf_ids + i * 8, ips + i * 8,
> + next_hops + i * 8, n - i * 8);
> +}
>
> void
> -rte_dir24_8_vec_lookup_bulk_8b(void *p, const uint32_t *ips,
> - uint64_t *next_hops, const unsigned int n)
> +rte_dir24_8_vec_lookup_bulk_vrf_8b_large(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
> {
> uint32_t i;
> for (i = 0; i < (n / 8); i++)
> - dir24_8_vec_lookup_x8_8b(p, ips + i * 8, next_hops + i * 8, false);
> - dir24_8_lookup_bulk_8b(p, ips + i * 8, next_hops + i * 8, n - i * 8);
> + dir24_8_vec_lookup_x8_8b(p, vrf_ids + i * 8, ips + i * 8,
> + next_hops + i * 8, false, VRF_SCALE_LARGE);
> + dir24_8_lookup_bulk_vrf_8b(p, vrf_ids + i * 8, ips + i * 8,
> + next_hops + i * 8, n - i * 8);
> }
>
> void
> -rte_dir24_8_vec_lookup_bulk_8b_be(void *p, const uint32_t *ips,
> - uint64_t *next_hops, const unsigned int n)
> +rte_dir24_8_vec_lookup_bulk_vrf_8b_be_large(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
> {
> uint32_t i;
> for (i = 0; i < (n / 8); i++)
> - dir24_8_vec_lookup_x8_8b(p, ips + i * 8, next_hops + i * 8, true);
> - dir24_8_lookup_bulk_8b_be(p, ips + i * 8, next_hops + i * 8, n - i * 8);
> + dir24_8_vec_lookup_x8_8b(p, vrf_ids + i * 8, ips + i * 8,
> + next_hops + i * 8, true, VRF_SCALE_LARGE);
> + dir24_8_lookup_bulk_vrf_8b_be(p, vrf_ids + i * 8, ips + i * 8,
> + next_hops + i * 8, n - i * 8);
> }
> diff --git a/lib/fib/dir24_8_avx512.h b/lib/fib/dir24_8_avx512.h
> index 3e2bbc2490..d42ef1d17f 100644
> --- a/lib/fib/dir24_8_avx512.h
> +++ b/lib/fib/dir24_8_avx512.h
> @@ -6,35 +6,99 @@
> #define _DIR248_AVX512_H_
>
> void
> -rte_dir24_8_vec_lookup_bulk_1b(void *p, const uint32_t *ips,
> +rte_dir24_8_vec_lookup_bulk_1b(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
> +
> +void
> +rte_dir24_8_vec_lookup_bulk_1b_be(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
> +
> +void
> +rte_dir24_8_vec_lookup_bulk_vrf_1b(void *p, const uint16_t *vrf_ids, const
> uint32_t *ips,
> uint64_t *next_hops, const unsigned int n);
>
> void
> -rte_dir24_8_vec_lookup_bulk_2b(void *p, const uint32_t *ips,
> +rte_dir24_8_vec_lookup_bulk_vrf_1b_be(void *p, const uint16_t *vrf_ids, const
> uint32_t *ips,
> uint64_t *next_hops, const unsigned int n);
>
> void
> -rte_dir24_8_vec_lookup_bulk_4b(void *p, const uint32_t *ips,
> +rte_dir24_8_vec_lookup_bulk_vrf_1b_large(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
> +
> +void
> +rte_dir24_8_vec_lookup_bulk_vrf_1b_be_large(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
> +
> +void
> +rte_dir24_8_vec_lookup_bulk_2b(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
> +
> +void
> +rte_dir24_8_vec_lookup_bulk_2b_be(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
> +
> +void
> +rte_dir24_8_vec_lookup_bulk_vrf_2b(void *p, const uint16_t *vrf_ids, const
> uint32_t *ips,
> uint64_t *next_hops, const unsigned int n);
>
> void
> -rte_dir24_8_vec_lookup_bulk_8b(void *p, const uint32_t *ips,
> +rte_dir24_8_vec_lookup_bulk_vrf_2b_be(void *p, const uint16_t *vrf_ids, const
> uint32_t *ips,
> uint64_t *next_hops, const unsigned int n);
>
> void
> -rte_dir24_8_vec_lookup_bulk_1b_be(void *p, const uint32_t *ips,
> +rte_dir24_8_vec_lookup_bulk_vrf_2b_large(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
> +
> +void
> +rte_dir24_8_vec_lookup_bulk_vrf_2b_be_large(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
> +
> +void
> +rte_dir24_8_vec_lookup_bulk_4b(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
> +
> +void
> +rte_dir24_8_vec_lookup_bulk_4b_be(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
> +
> +void
> +rte_dir24_8_vec_lookup_bulk_vrf_4b(void *p, const uint16_t *vrf_ids, const
> uint32_t *ips,
> uint64_t *next_hops, const unsigned int n);
>
> void
> -rte_dir24_8_vec_lookup_bulk_2b_be(void *p, const uint32_t *ips,
> +rte_dir24_8_vec_lookup_bulk_vrf_4b_be(void *p, const uint16_t *vrf_ids, const
> uint32_t *ips,
> uint64_t *next_hops, const unsigned int n);
>
> void
> -rte_dir24_8_vec_lookup_bulk_4b_be(void *p, const uint32_t *ips,
> +rte_dir24_8_vec_lookup_bulk_vrf_4b_large(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
> +
> +void
> +rte_dir24_8_vec_lookup_bulk_vrf_4b_be_large(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
> +
> +void
> +rte_dir24_8_vec_lookup_bulk_8b(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
> +
> +void
> +rte_dir24_8_vec_lookup_bulk_8b_be(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
> +
> +void
> +rte_dir24_8_vec_lookup_bulk_vrf_8b(void *p, const uint16_t *vrf_ids, const
> uint32_t *ips,
> uint64_t *next_hops, const unsigned int n);
>
> void
> -rte_dir24_8_vec_lookup_bulk_8b_be(void *p, const uint32_t *ips,
> +rte_dir24_8_vec_lookup_bulk_vrf_8b_be(void *p, const uint16_t *vrf_ids, const
> uint32_t *ips,
> uint64_t *next_hops, const unsigned int n);
>
> +void
> +rte_dir24_8_vec_lookup_bulk_vrf_8b_large(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
> +
> +void
> +rte_dir24_8_vec_lookup_bulk_vrf_8b_be_large(void *p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
> +
> #endif /* _DIR248_AVX512_H_ */
> diff --git a/lib/fib/rte_fib.c b/lib/fib/rte_fib.c
> index 184210f380..efc0595a7f 100644
> --- a/lib/fib/rte_fib.c
> +++ b/lib/fib/rte_fib.c
> @@ -14,12 +14,15 @@
> #include <rte_string_fns.h>
> #include <rte_tailq.h>
>
> +#include <rte_debug.h>
> #include <rte_rib.h>
> #include <rte_fib.h>
>
> #include "dir24_8.h"
> #include "fib_log.h"
>
> +#define FIB_MAX_LOOKUP_BULK 64U
> +
> RTE_LOG_REGISTER_DEFAULT(fib_logtype, INFO);
>
> TAILQ_HEAD(rte_fib_list, rte_tailq_entry);
> @@ -40,52 +43,61 @@ EAL_REGISTER_TAILQ(rte_fib_tailq)
> struct rte_fib {
> char name[RTE_FIB_NAMESIZE];
> enum rte_fib_type type; /**< Type of FIB struct */
> - unsigned int flags; /**< Flags */
> - struct rte_rib *rib; /**< RIB helper datastructure */
> + uint16_t flags; /**< Flags */
> + uint16_t num_vrfs;/**< Number of VRFs */
> + struct rte_rib **ribs; /**< RIB helper datastructures per VRF */
> void *dp; /**< pointer to the dataplane struct*/
> rte_fib_lookup_fn_t lookup; /**< FIB lookup function */
> rte_fib_modify_fn_t modify; /**< modify FIB datastructure */
> - uint64_t def_nh;
> + uint64_t *def_nh;/**< Per-VRF default next hop array */
> };
>
> static void
> -dummy_lookup(void *fib_p, const uint32_t *ips, uint64_t *next_hops,
> - const unsigned int n)
> +dummy_lookup(void *fib_p, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
> {
> unsigned int i;
> struct rte_fib *fib = fib_p;
> struct rte_rib_node *node;
> + struct rte_rib *rib;
>
> for (i = 0; i < n; i++) {
> - node = rte_rib_lookup(fib->rib, ips[i]);
> + RTE_ASSERT(vrf_ids[i] < fib->num_vrfs);
> + rib = rte_fib_vrf_get_rib(fib, vrf_ids[i]);
> + node = rte_rib_lookup(rib, ips[i]);
> if (node != NULL)
> rte_rib_get_nh(node, &next_hops[i]);
> else
> - next_hops[i] = fib->def_nh;
> + next_hops[i] = fib->def_nh[vrf_ids[i]];
> }
> }
>
> static int
> -dummy_modify(struct rte_fib *fib, uint32_t ip, uint8_t depth,
> - uint64_t next_hop, int op)
> +dummy_modify(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip,
> + uint8_t depth, uint64_t next_hop, int op)
> {
> struct rte_rib_node *node;
> + struct rte_rib *rib;
> if ((fib == NULL) || (depth > RTE_FIB_MAXDEPTH))
> return -EINVAL;
>
> - node = rte_rib_lookup_exact(fib->rib, ip, depth);
> + rib = rte_fib_vrf_get_rib(fib, vrf_id);
> + if (rib == NULL)
> + return -EINVAL;
> +
> + node = rte_rib_lookup_exact(rib, ip, depth);
>
> switch (op) {
> case RTE_FIB_ADD:
> if (node == NULL)
> - node = rte_rib_insert(fib->rib, ip, depth);
> + node = rte_rib_insert(rib, ip, depth);
> if (node == NULL)
> return -rte_errno;
> return rte_rib_set_nh(node, next_hop);
> case RTE_FIB_DEL:
> if (node == NULL)
> return -ENOENT;
> - rte_rib_remove(fib->rib, ip, depth);
> + rte_rib_remove(rib, ip, depth);
> return 0;
> }
> return -EINVAL;
> @@ -125,7 +137,7 @@ rte_fib_add(struct rte_fib *fib, uint32_t ip, uint8_t depth,
> uint64_t next_hop)
> if ((fib == NULL) || (fib->modify == NULL) ||
> (depth > RTE_FIB_MAXDEPTH))
> return -EINVAL;
> - return fib->modify(fib, ip, depth, next_hop, RTE_FIB_ADD);
> + return fib->modify(fib, 0, ip, depth, next_hop, RTE_FIB_ADD);
> }
>
> RTE_EXPORT_SYMBOL(rte_fib_delete)
> @@ -135,7 +147,7 @@ rte_fib_delete(struct rte_fib *fib, uint32_t ip, uint8_t
> depth)
> if ((fib == NULL) || (fib->modify == NULL) ||
> (depth > RTE_FIB_MAXDEPTH))
> return -EINVAL;
> - return fib->modify(fib, ip, depth, 0, RTE_FIB_DEL);
> + return fib->modify(fib, 0, ip, depth, 0, RTE_FIB_DEL);
> }
>
> RTE_EXPORT_SYMBOL(rte_fib_lookup_bulk)
> @@ -143,24 +155,73 @@ int
> rte_fib_lookup_bulk(struct rte_fib *fib, uint32_t *ips,
> uint64_t *next_hops, int n)
> {
> + static const uint16_t zero_vrf_ids[FIB_MAX_LOOKUP_BULK];
> + unsigned int off = 0;
> + unsigned int total = (unsigned int)n;
> +
> FIB_RETURN_IF_TRUE(((fib == NULL) || (ips == NULL) ||
> (next_hops == NULL) || (fib->lookup == NULL)), -EINVAL);
>
> - fib->lookup(fib->dp, ips, next_hops, n);
> + while (off < total) {
> + unsigned int chunk = RTE_MIN(total - off,
> + FIB_MAX_LOOKUP_BULK);
> + fib->lookup(fib->dp, zero_vrf_ids, ips + off,
> + next_hops + off, chunk);
> + off += chunk;
> + }
> +
> + return 0;
> +}
> +
> +RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fib_vrf_lookup_bulk, 26.07)
> +int
> +rte_fib_vrf_lookup_bulk(struct rte_fib *fib, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, int n)
> +{
> + FIB_RETURN_IF_TRUE(((fib == NULL) || (vrf_ids == NULL) ||
> + (ips == NULL) || (next_hops == NULL) ||
> + (fib->lookup == NULL)), -EINVAL);
> +
> + fib->lookup(fib->dp, vrf_ids, ips, next_hops, (unsigned int)n);
> return 0;
> }
>
> +RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fib_vrf_add, 26.07)
> +int
> +rte_fib_vrf_add(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip,
> + uint8_t depth, uint64_t next_hop)
> +{
> + if ((fib == NULL) || (fib->modify == NULL) ||
> + (depth > RTE_FIB_MAXDEPTH))
> + return -EINVAL;
> + return fib->modify(fib, vrf_id, ip, depth, next_hop, RTE_FIB_ADD);
> +}
> +
> +RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fib_vrf_delete, 26.07)
> +int
> +rte_fib_vrf_delete(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip,
> + uint8_t depth)
> +{
> + if ((fib == NULL) || (fib->modify == NULL) ||
> + (depth > RTE_FIB_MAXDEPTH))
> + return -EINVAL;
> + return fib->modify(fib, vrf_id, ip, depth, 0, RTE_FIB_DEL);
> +}
> +
> RTE_EXPORT_SYMBOL(rte_fib_create)
> struct rte_fib *
> rte_fib_create(const char *name, int socket_id, struct rte_fib_conf *conf)
> {
> char mem_name[RTE_FIB_NAMESIZE];
> + char rib_name[RTE_FIB_NAMESIZE];
> int ret;
> struct rte_fib *fib = NULL;
> struct rte_rib *rib = NULL;
> struct rte_tailq_entry *te;
> struct rte_fib_list *fib_list;
> struct rte_rib_conf rib_conf;
> + uint16_t num_vrfs;
> + uint16_t vrf;
>
> /* Check user arguments. */
> if ((name == NULL) || (conf == NULL) || (conf->max_routes < 0) ||
> @@ -170,16 +231,42 @@ rte_fib_create(const char *name, int socket_id, struct
> rte_fib_conf *conf)
> return NULL;
> }
>
> + num_vrfs = (conf->max_vrfs == 0) ? 1 : conf->max_vrfs;
> rib_conf.ext_sz = conf->rib_ext_sz;
> rib_conf.max_nodes = conf->max_routes * 2;
>
> - rib = rte_rib_create(name, socket_id, &rib_conf);
> - if (rib == NULL) {
> - FIB_LOG(ERR,
> - "Can not allocate RIB %s", name);
> + struct rte_rib **ribs = rte_zmalloc_socket("FIB_RIBS",
> + num_vrfs * sizeof(*fib->ribs), RTE_CACHE_LINE_SIZE, socket_id);
> + if (ribs == NULL) {
> + FIB_LOG(ERR, "FIB %s RIB array allocation failed", name);
> + rte_errno = ENOMEM;
> return NULL;
> }
>
> + uint64_t *def_nh = rte_zmalloc_socket("FIB_DEF_NH",
> + num_vrfs * sizeof(*def_nh), RTE_CACHE_LINE_SIZE, socket_id);
> + if (def_nh == NULL) {
> + FIB_LOG(ERR, "FIB %s default nexthop array allocation failed",
> name);
> + rte_errno = ENOMEM;
> + rte_free(ribs);
> + return NULL;
> + }
> +
> + for (vrf = 0; vrf < num_vrfs; vrf++) {
> + if (num_vrfs == 1)
> + snprintf(rib_name, sizeof(rib_name), "%s", name);
> + else
> + snprintf(rib_name, sizeof(rib_name), "%s_vrf%u", name,
> vrf);
> + rib = rte_rib_create(rib_name, socket_id, &rib_conf);
> + if (rib == NULL) {
> + FIB_LOG(ERR, "Can not allocate RIB %s", rib_name);
> + goto free_ribs;
> + }
> + ribs[vrf] = rib;
> + def_nh[vrf] = (conf->vrf_default_nh != NULL) ?
> + conf->vrf_default_nh[vrf] : conf->default_nh;
> + }
> +
> snprintf(mem_name, sizeof(mem_name), "FIB_%s", name);
> fib_list = RTE_TAILQ_CAST(rte_fib_tailq.head, rte_fib_list);
>
> @@ -215,11 +302,13 @@ rte_fib_create(const char *name, int socket_id, struct
> rte_fib_conf *conf)
> goto free_te;
> }
>
> + fib->num_vrfs = num_vrfs;
> + fib->ribs = ribs;
> + fib->def_nh = def_nh;
> +
> rte_strlcpy(fib->name, name, sizeof(fib->name));
> - fib->rib = rib;
> fib->type = conf->type;
> fib->flags = conf->flags;
> - fib->def_nh = conf->default_nh;
> ret = init_dataplane(fib, socket_id, conf);
> if (ret < 0) {
> FIB_LOG(ERR,
> @@ -242,8 +331,12 @@ rte_fib_create(const char *name, int socket_id, struct
> rte_fib_conf *conf)
> rte_free(te);
> exit:
> rte_mcfg_tailq_write_unlock();
> - rte_rib_free(rib);
> +free_ribs:
> + for (vrf = 0; vrf < num_vrfs; vrf++)
> + rte_rib_free(ribs[vrf]);
>
> + rte_free(def_nh);
> + rte_free(ribs);
> return NULL;
> }
>
> @@ -311,7 +404,13 @@ rte_fib_free(struct rte_fib *fib)
> rte_mcfg_tailq_write_unlock();
>
> free_dataplane(fib);
> - rte_rib_free(fib->rib);
> + if (fib->ribs != NULL) {
> + uint16_t vrf;
> + for (vrf = 0; vrf < fib->num_vrfs; vrf++)
> + rte_rib_free(fib->ribs[vrf]);
> + }
> + rte_free(fib->ribs);
> + rte_free(fib->def_nh);
> rte_free(fib);
> rte_free(te);
> }
> @@ -327,7 +426,18 @@ RTE_EXPORT_SYMBOL(rte_fib_get_rib)
> struct rte_rib *
> rte_fib_get_rib(struct rte_fib *fib)
> {
> - return (fib == NULL) ? NULL : fib->rib;
> + return (fib == NULL || fib->ribs == NULL) ? NULL : fib->ribs[0];
> +}
> +
> +RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fib_vrf_get_rib, 26.07)
> +struct rte_rib *
> +rte_fib_vrf_get_rib(struct rte_fib *fib, uint16_t vrf_id)
> +{
> + if (fib == NULL || fib->ribs == NULL)
> + return NULL;
> + if (vrf_id >= fib->num_vrfs)
> + return NULL;
> + return fib->ribs[vrf_id];
> }
>
> RTE_EXPORT_SYMBOL(rte_fib_select_lookup)
> diff --git a/lib/fib/rte_fib.h b/lib/fib/rte_fib.h
> index b16a653535..883195c7d6 100644
> --- a/lib/fib/rte_fib.h
> +++ b/lib/fib/rte_fib.h
> @@ -53,11 +53,11 @@ enum rte_fib_type {
> };
>
> /** Modify FIB function */
> -typedef int (*rte_fib_modify_fn_t)(struct rte_fib *fib, uint32_t ip,
> - uint8_t depth, uint64_t next_hop, int op);
> +typedef int (*rte_fib_modify_fn_t)(struct rte_fib *fib, uint16_t vrf_id,
> + uint32_t ip, uint8_t depth, uint64_t next_hop, int op);
> /** FIB bulk lookup function */
> -typedef void (*rte_fib_lookup_fn_t)(void *fib, const uint32_t *ips,
> - uint64_t *next_hops, const unsigned int n);
> +typedef void (*rte_fib_lookup_fn_t)(void *fib, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
>
> enum rte_fib_op {
> RTE_FIB_ADD,
> @@ -110,6 +110,10 @@ struct rte_fib_conf {
> } dir24_8;
> };
> unsigned int flags; /**< Optional feature flags from RTE_FIB_F_* */
> + /** Number of VRFs to support (0 or 1 = single VRF for backward compat)
> */
> + uint16_t max_vrfs;
> + /** Per-VRF default nexthops (NULL = use default_nh for all) */
> + uint64_t *vrf_default_nh;
> };
>
> /** FIB RCU QSBR configuration structure. */
> @@ -224,6 +228,71 @@ rte_fib_delete(struct rte_fib *fib, uint32_t ip, uint8_t
> depth);
> int
> rte_fib_lookup_bulk(struct rte_fib *fib, uint32_t *ips,
> uint64_t *next_hops, int n);
> +
> +/**
> + * Add a route to the FIB with VRF ID.
> + *
> + * @param fib
> + * FIB object handle
> + * @param vrf_id
> + * VRF ID (0 to max_vrfs-1)
> + * @param ip
> + * IPv4 prefix address to be added to the FIB
> + * @param depth
> + * Prefix length
> + * @param next_hop
> + * Next hop to be added to the FIB
> + * @return
> + * 0 on success, negative value otherwise
> + */
> +__rte_experimental
> +int
> +rte_fib_vrf_add(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip,
> + uint8_t depth, uint64_t next_hop);
> +
> +/**
> + * Delete a rule from the FIB with VRF ID.
> + *
> + * @param fib
> + * FIB object handle
> + * @param vrf_id
> + * VRF ID (0 to max_vrfs-1)
> + * @param ip
> + * IPv4 prefix address to be deleted from the FIB
> + * @param depth
> + * Prefix length
> + * @return
> + * 0 on success, negative value otherwise
> + */
> +__rte_experimental
> +int
> +rte_fib_vrf_delete(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip,
> + uint8_t depth);
> +
> +/**
> + * Lookup multiple IP addresses in the FIB with per-packet VRF IDs.
> + *
> + * @param fib
> + * FIB object handle
> + * @param vrf_ids
> + * Array of VRF IDs
> + * @param ips
> + * Array of IPs to be looked up in the FIB
> + * @param next_hops
> + * Next hop of the most specific rule found for IP in the corresponding VRF.
> + * This is an array of eight byte values.
> + * If the lookup for the given IP failed, then corresponding element would
> + * contain default nexthop value configured for that VRF.
> + * @param n
> + * Number of elements in vrf_ids, ips (and next_hops) arrays to lookup.
> + * @return
> + * -EINVAL for incorrect arguments, otherwise 0
> + */
> +__rte_experimental
> +int
> +rte_fib_vrf_lookup_bulk(struct rte_fib *fib, const uint16_t *vrf_ids,
> + const uint32_t *ips, uint64_t *next_hops, int n);
> +
> /**
> * Get pointer to the dataplane specific struct
> *
> @@ -237,7 +306,7 @@ void *
> rte_fib_get_dp(struct rte_fib *fib);
>
> /**
> - * Get pointer to the RIB
> + * Get pointer to the RIB for VRF 0
> *
> * @param fib
> * FIB object handle
> @@ -248,6 +317,21 @@ rte_fib_get_dp(struct rte_fib *fib);
> struct rte_rib *
> rte_fib_get_rib(struct rte_fib *fib);
>
> +/**
> + * Get pointer to the RIB for a specific VRF
> + *
> + * @param fib
> + * FIB object handle
> + * @param vrf_id
> + * VRF ID (0 to max_vrfs-1)
> + * @return
> + * Pointer on the RIB on success
> + * NULL otherwise
> + */
> +__rte_experimental
> +struct rte_rib *
> +rte_fib_vrf_get_rib(struct rte_fib *fib, uint16_t vrf_id);
> +
> /**
> * Set lookup function based on type
> *
> --
> 2.43.0
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH 0/4] VRF support in FIB library
2026-03-23 14:53 ` Maxime Leroy
2026-03-23 15:08 ` Robin Jarry
@ 2026-03-23 18:42 ` Medvedkin, Vladimir
2026-03-24 9:19 ` Maxime Leroy
1 sibling, 1 reply; 45+ messages in thread
From: Medvedkin, Vladimir @ 2026-03-23 18:42 UTC (permalink / raw)
To: Maxime Leroy; +Cc: dev, rjarry, nsaxena16, mb, adwivedi, jerinjacobk
On 3/23/2026 2:53 PM, Maxime Leroy wrote:
> On Mon, Mar 23, 2026 at 1:49 PM Medvedkin, Vladimir
> <vladimir.medvedkin@intel.com> wrote:
>> Hi Maxime,
>>
>> On 3/23/2026 11:27 AM, Maxime Leroy wrote:
>>> Hi Vladimir,
>>>
>>>
>>> On Sun, Mar 22, 2026 at 4:42 PM Vladimir Medvedkin
>>> <vladimir.medvedkin@intel.com> wrote:
>>>> This series adds multi-VRF support to both IPv4 and IPv6 FIB paths by
>>>> allowing a single FIB instance to host multiple isolated routing domains.
>>>>
>>>> Currently FIB instance represents one routing instance. For workloads that
>>>> need multiple VRFs, the only option is to create multiple FIB objects. In a
>>>> burst oriented datapath, packets in the same batch can belong to different VRFs, so
>>>> the application either does per-packet lookup in different FIB instances or
>>>> regroups packets by VRF before lookup. Both approaches are expensive.
>>>>
>>>> To remove that cost, this series keeps all VRFs inside one FIB instance and
>>>> extends lookup input with per-packet VRF IDs.
>>>>
>>>> The design follows the existing fast-path structure for both families. IPv4 and
>>>> IPv6 use multi-ary trees with a 2^24 associativity on a first level (tbl24). The
>>>> first-level table scales per configured VRF. This increases memory usage, but
>>>> keeps performance and lookup complexity on par with non-VRF implementation.
>>>>
>>> Thanks for the RFC. Some thoughts below.
>>>
>>> Memory cost: the flat TBL24 replicates the entire table for every VRF
>>> (num_vrfs * 2^24 * nh_size). With 256 VRFs and 8B nexthops that is
>>> 32 GB for TBL24 alone. In grout we support up to 256 VRFs allocated
>>> on demand -- this approach forces the full cost upfront even if most
>>> VRFs are empty.
>> Yes, increased memory consumption is the
>> trade-off.WemakethischoiceinDPDKquite often,such as pre-allocatedmbufs,
>> mempoolsand many other stuff allocated in advance to gain performance.
>> For FIB, I chose to replicate TBL24 per VRF for this same reason.
>>
>> And, as Morten mentioned earlier, if memory is the priority, a table
>> instance per VRF allocated on-demand is still supported.
>>
>> The high memory cost stems from TBL24's design: for IPv4, it was
>> justified by the BGP filtering convention (no prefixes more specific
>> than /24 in BGPv4 full view), ensuring most lookups hit with just one
>> random memory access. For IPv6, we should likely switch to a 16-bit TRIE
>> scheme on all layers. For IPv4, alternative algorithms with smaller
>> footprints (like DXR or DIR16-8-8, as used in VPP) may be worth
>> exploring if BGP full view is not required for those VRFs.
>>
>>> Per-packet VRF lookup: Rx bursts come from one port, thus one VRF.
>>> Mixed-VRF bulk lookups do not occur in practice. The three AVX512
>>> code paths add complexity for a scenario that does not exist, at
>>> least for a classic router. Am I missing a use-case?
>> That's not true, you're missing out on a lot of established core use
>> cases that are at least 2 decades old:
>>
>> - VLAN subinterface abstraction. Each subinterface may belong to a
>> separate VRF
>>
>> - MPLS VPN
>>
>> - Policy based routing
>>
> Fair point on VLAN subinterfaces and MPLS VPN. SRv6 L3VPN (End.DT4/
> End.DT6) also fits that pattern after decap.
>
> I agree DPDK often pre-allocates for performance, but I wonder if the
> flat TBL24 actually helps here. Each VRF's working set is spread
> 128 MB apart in the flat table. Would regrouping packets by VRF and
> doing one bulk lookup per VRF with separate contiguous TBL24s be
> more cache-friendly than a single mixed-VRF gather? Do you have
> benchmarks comparing the two approaches?
It depends. Generally, if we assume that we are working with wide
internet traffic, then even for a single VRF we most likely will miss
the cache for TLB24, thus, regardless of the size of the tbl24, each
memory access will be performed directly to DRAM.
And if the addresses are localized (i.e. most traffic is internal), then
having multiple TBL24 won'tmake the situationmuchworse.
I don't have any benchmarks for regrouping, however I have 2 things to
consider:
1. lookup is relatively fast (for IPv4 it is about 10 cycles per
address, and I don't really want to slow it down)
2. incoming addresses and their corresponding VRFs are not controlled by
"us", so this is a random set. Regrouping effectively is sorting. I'm
not really happy to have nlogn complexity on a fast path :)
>
> On the memory trade-off and VRF ID mapping: the API uses vrf_id as
> a direct index (0 to max_vrfs-1). With 256 VRFs and 8B nexthops,
> TBL24 alone costs 32 GB for IPv4 and 32 GB for IPv6 -- 64 GB total
> at startup. In grout, VRF IDs are interface IDs that can be any
> uint16_t, so we would also need to maintain a mapping between our
> VRF IDs and FIB slot indices.
of course, this is an application responsibility. In FIB VRFs are in
continuous range.
> We would need to introduce a max_vrfs
> limit, which forces a bad trade-off: either set it low (e.g. 16)
> and limit deployments, or set it high (e.g. 256) and pay 64 GB at
> startup even with a single VRF. With separate FIB instances per VRF,
> we only allocate what we use.
Yes, I understand this. In the end, if the user wants to use 256 VRFs,
the amount of memory footprint will be at least 64Gb anyway.
As a trade-off for a bad trade-off ;) I can suggest to allocate it in
chunks. Let's say you are starting with 16 VRFs, and during runtime, if
the user wants to increase the number of VRFs above this limit, you can
allocate another 16xVRF FIB. Then, of course, you need to split
addresses into 2 bursts each for each FIB handle.
>
>>> I am not too familiar with DPDK FIB internals, but would it be
>>> possible to keep a separate TBL24 per VRF and only share the TBL8
>>> pool?
>> it is how it is implemented right now with one note - TBL24 are pre
>> allocated.
>>> Something like pre-allocating an array of max_vrfs TBL24
>>> pointers, allocating each TBL24 on demand at VRF add time,
>> and you suggesting to allocate TBL24 on demand by adding an extra
>> indirection layer. Thiswill leadtolowerperformance,whichIwouldliketo avoid.
>>> and
>>> having them all point into a shared TBL8 pool. The TBL8 index in
>>> TBL24 entries seems to already be global, so would that work without
>>> encoding changes?
>>>
>>> Going further: could the same idea extend to IPv6? The dir24_8 and
>>> trie seem to use the same TBL8 block format (256 entries, same
>>> (nh << 1) | ext_bit encoding, same size). Would unifying the TBL8
>>> allocator allow a single pool shared across IPv4, IPv6, and all
>>> VRFs? That could be a bigger win for /32-heavy and /128-heavy tables
>>> and maybe a good first step before multi-VRF.
>> So, you are suggesting merging IPv4 and IPv6 into a single unified FIB?
>> I'm not sure how this can be a bigger win, could you please elaborate
>> more on this?
> On the IPv4/IPv6 TBL8 pool: I was not suggesting merging FIBs, just
> sharing the TBL8 block allocator between separate FIB instances.
> This is possible since dir24_8 and trie use the same TBL8 block
> format (256 entries, same encoding, same size).
>
> Would it be possible to pass a shared TBL8 pool at rte_fib_create()
> time? Each FIB keeps its own TBL24 and RIB, but TBL8 is shared
> across all FIBs and potentially across IPv4/IPv6. Users would no
> longer have to guess num_tbl8 per FIB.
Yes, this is possible. However, this will significantly complicate the
work with the library, solving a not so big problem.
>>> Regards,
>>>
>>> Maxime Leroy
>>>
>>>> Vladimir Medvedkin (4):
>>>> fib: add multi-VRF support
>>>> fib: add VRF functional and unit tests
>>>> fib6: add multi-VRF support
>>>> fib6: add VRF functional and unit tests
>>>>
>>>> app/test-fib/main.c | 257 ++++++++++++++++++++++--
>>>> app/test/test_fib.c | 298 +++++++++++++++++++++++++++
>>>> app/test/test_fib6.c | 319 ++++++++++++++++++++++++++++-
>>>> lib/fib/dir24_8.c | 241 ++++++++++++++++------
>>>> lib/fib/dir24_8.h | 255 ++++++++++++++++--------
>>>> lib/fib/dir24_8_avx512.c | 420 +++++++++++++++++++++++++++++++--------
>>>> lib/fib/dir24_8_avx512.h | 80 +++++++-
>>>> lib/fib/rte_fib.c | 158 ++++++++++++---
>>>> lib/fib/rte_fib.h | 94 ++++++++-
>>>> lib/fib/rte_fib6.c | 166 +++++++++++++---
>>>> lib/fib/rte_fib6.h | 88 +++++++-
>>>> lib/fib/trie.c | 158 +++++++++++----
>>>> lib/fib/trie.h | 51 +++--
>>>> lib/fib/trie_avx512.c | 225 +++++++++++++++++++--
>>>> lib/fib/trie_avx512.h | 39 +++-
>>>> 15 files changed, 2453 insertions(+), 396 deletions(-)
>>>>
>>>> --
>>>> 2.43.0
>>>>
>> --
>> Regards,
>> Vladimir
>>
>
--
Regards,
Vladimir
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH 0/4] VRF support in FIB library
2026-03-23 15:27 ` Morten Brørup
@ 2026-03-23 18:52 ` Medvedkin, Vladimir
0 siblings, 0 replies; 45+ messages in thread
From: Medvedkin, Vladimir @ 2026-03-23 18:52 UTC (permalink / raw)
To: Morten Brørup, Robin Jarry, Maxime Leroy
Cc: dev, nsaxena16, adwivedi, jerinjacobk, Stephen Hemminger
On 3/23/2026 3:27 PM, Morten Brørup wrote:
> Let's take a level up in abstraction, and consider one of the key design goals:
> What are the typical use cases we want VRF support designed for?
>
> I can imagine a use case with one or very few VRFs with a very large (internet) route table, and many VRFs with small (private) route tables.
>
> PS: I have no experience with the DPDK FIB library, so my feedback might be completely off.
No, you completely correct :) In real world scenarios only a few VRFs
will have full view, so I considered that number of VRFs would be small.
For small (private) route tables, it would be good to use either smaller
nexthops (e.g. 2bytes), or some other LPM algorithm that is not so
greedy for memory like DIR24_8.
>
--
Regards,
Vladimir
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH 0/4] VRF support in FIB library
2026-03-22 15:42 [RFC PATCH 0/4] VRF support in FIB library Vladimir Medvedkin
` (6 preceding siblings ...)
2026-03-23 11:27 ` Maxime Leroy
@ 2026-03-23 19:05 ` Stephen Hemminger
2026-10-14 18:39 ` [RFC PATCH v2 0/7] " Vladimir Medvedkin
8 siblings, 0 replies; 45+ messages in thread
From: Stephen Hemminger @ 2026-03-23 19:05 UTC (permalink / raw)
To: Vladimir Medvedkin; +Cc: dev, rjarry, nsaxena16, mb, adwivedi, jerinjacobk
On Sun, 22 Mar 2026 15:42:11 +0000
Vladimir Medvedkin <vladimir.medvedkin@intel.com> wrote:
> This series adds multi-VRF support to both IPv4 and IPv6 FIB paths by
> allowing a single FIB instance to host multiple isolated routing domains.
>
> Currently FIB instance represents one routing instance. For workloads that
> need multiple VRFs, the only option is to create multiple FIB objects. In a
> burst oriented datapath, packets in the same batch can belong to different VRFs, so
> the application either does per-packet lookup in different FIB instances or
> regroups packets by VRF before lookup. Both approaches are expensive.
>
> To remove that cost, this series keeps all VRFs inside one FIB instance and
> extends lookup input with per-packet VRF IDs.
>
> The design follows the existing fast-path structure for both families. IPv4 and
> IPv6 use multi-ary trees with a 2^24 associativity on a first level (tbl24). The
> first-level table scales per configured VRF. This increases memory usage, but
> keeps performance and lookup complexity on par with non-VRF implementation.
>
> Vladimir Medvedkin (4):
> fib: add multi-VRF support
> fib: add VRF functional and unit tests
> fib6: add multi-VRF support
> fib6: add VRF functional and unit tests
>
> app/test-fib/main.c | 257 ++++++++++++++++++++++--
> app/test/test_fib.c | 298 +++++++++++++++++++++++++++
> app/test/test_fib6.c | 319 ++++++++++++++++++++++++++++-
> lib/fib/dir24_8.c | 241 ++++++++++++++++------
> lib/fib/dir24_8.h | 255 ++++++++++++++++--------
> lib/fib/dir24_8_avx512.c | 420 +++++++++++++++++++++++++++++++--------
> lib/fib/dir24_8_avx512.h | 80 +++++++-
> lib/fib/rte_fib.c | 158 ++++++++++++---
> lib/fib/rte_fib.h | 94 ++++++++-
> lib/fib/rte_fib6.c | 166 +++++++++++++---
> lib/fib/rte_fib6.h | 88 +++++++-
> lib/fib/trie.c | 158 +++++++++++----
> lib/fib/trie.h | 51 +++--
> lib/fib/trie_avx512.c | 225 +++++++++++++++++++--
> lib/fib/trie_avx512.h | 39 +++-
> 15 files changed, 2453 insertions(+), 396 deletions(-)
>
AI review found several things
Review: [RFC PATCH 1/4] fib: add multi-VRF support
[RFC PATCH 2/4] fib: add VRF functional and unit tests
[RFC PATCH 3/4] fib6: add multi-VRF support
[RFC PATCH 4/4] fib6: add VRF functional and unit tests
Overall this is a well-structured RFC that adds multi-VRF support
to both the IPv4 and IPv6 FIB libraries with AVX512-optimized
lookup paths and comprehensive test coverage. There is one
significant correctness bug in the AVX512 gather paths, several
design points worth discussing, and some minor issues.
Patch 1/4 - fib: add multi-VRF support
Error: Signed overflow in AVX512 32-bit gather for VRF IDs >= 128
The VRF_SCALE_SMALL path (num_vrfs in [2, 255]) computes the
tbl24 index in 32-bit arithmetic as (vrf_id << 24) + (ip >> 8).
For vrf_id >= 128, vrf_id << 24 sets bit 31, making the result
negative when interpreted as int32. The _mm512_i32gather_epi32
and _mm512_i32gather_epi64 intrinsics sign-extend 32-bit indices
to compute byte offsets, so a negative index produces a read
before the start of tbl24 -- a buffer underflow.
Example: vrf_id=128, ip=0 gives index 0x08000000 << 24 =
0x80000000 = -2147483648 as signed int32.
This affects all nexthop sizes in both dir24_8_avx512.c and
trie_avx512.c.
Fix: Either lower the VRF_SCALE_SMALL ceiling from 256 to 128
(so VRFs 128-255 use the 64-bit path), or switch to unsigned
gather by pre-scaling the indices into byte offsets and using
scale=1 with unsigned arithmetic.
In dir24_8_avx512.c get_vector_fn():
if (dp->num_vrfs >= 256) {
should be:
if (dp->num_vrfs >= 128) {
Same change needed in trie.c get_vector_fn().
Warning: ABI break -- public function pointer typedefs changed
rte_fib_lookup_fn_t and rte_fib_modify_fn_t in rte_fib.h (and
the corresponding fib6 typedefs in rte_fib6.h) have new
parameters (vrf_ids/vrf_id). These are installed header typedefs
used by applications setting custom lookup functions via
rte_fib_select_lookup(). Changing them is an ABI break that needs
deprecation notice or ABI versioning.
Similarly, adding max_vrfs and vrf_default_nh to rte_fib_conf and
rte_fib6_conf changes the struct layout.
Since this is RFC, this is expected, but it will need to be
addressed before non-RFC submission.
Warning: No release notes for new experimental APIs
Eight new experimental APIs are added (rte_fib_vrf_add,
rte_fib_vrf_delete, rte_fib_vrf_lookup_bulk, rte_fib_vrf_get_rib
plus the fib6 equivalents). These need entries in
doc/guides/rel_notes/.
Warning: No testpmd hooks for new APIs
Per DPDK policy, new APIs should have hooks in app/testpmd.
Patch 2/4 - fib: add VRF functional and unit tests
Warning: Resource leak in run_v4() -- conf.vrf_default_nh not freed
In app/test-fib/main.c run_v4(), conf.vrf_default_nh is allocated
via rte_malloc() but never freed on any path (success or failure).
Same issue in run_v6() in patch 4/4.
Patch 3/4 - fib6: add multi-VRF support
Error: Same signed-overflow AVX512 gather bug as patch 1/4
The trie_avx512.c VRF_SCALE_SMALL path has the identical issue:
_mm512_slli_epi32(vrf32, 24) produces a negative signed index
for vrf_id >= 128, causing the 32-bit gather to read from a
negative offset.
In trie.c get_vector_fn():
if (dp->num_vrfs >= 256) {
should be:
if (dp->num_vrfs >= 128) {
Warning: Potential 32-bit truncation in trie helper functions
build_common_root() computes idx_tbl as uint64_t but passes it
to get_tbl_val_by_idx() and get_tbl_p_by_idx(). If those helpers
take uint32_t index parameters (the original code used 32-bit
indices), the upper bits will be silently truncated for large VRF
counts. The helpers should be widened to accept uint64_t, or
confirm they already do.
In practice, large VRF counts (hundreds+) with IPv6 trie tbl24
would require terabytes of memory, so this is unlikely to
manifest, but it is a latent correctness issue.
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH 1/4] fib: add multi-VRF support
2026-03-23 15:48 ` Konstantin Ananyev
@ 2026-03-23 19:06 ` Medvedkin, Vladimir
2026-03-23 22:22 ` Konstantin Ananyev
0 siblings, 1 reply; 45+ messages in thread
From: Medvedkin, Vladimir @ 2026-03-23 19:06 UTC (permalink / raw)
To: Konstantin Ananyev, dev@dpdk.org
Cc: rjarry@redhat.com, nsaxena16@gmail.com, mb@smartsharesystems.com,
adwivedi@marvell.com, jerinjacobk@gmail.com, Maxime Leroy
Hi Konstantin,
On 3/23/2026 3:48 PM, Konstantin Ananyev wrote:
>
>> Add VRF (Virtual Routing and Forwarding) support to the IPv4
>> FIB library, allowing multiple independent routing tables
>> within a single FIB instance.
>>
>> Introduce max_vrfs and vrf_default_nh fields in rte_fib_conf
>> to configure the number of VRFs and per-VRF default nexthops.
> Thanks Vladimir, allowing multiple VRFs per same LPM table will
> definitely be a useful thing to have.
> Though, I have the same concern as Maxime:
> memory requirements are just overwhelming.
> Stupid q - why just not to store a pointer to a vector of next-hops
> within the table entry?
Am I understand correctly, a vector with max_number_of_vrfs entries and
use vrf id to address a nexthop?
Yes, this may work.
But, if we are going to do an extra memory access, I'd better to
maintain an internal hash table with 5 byte keys {24_bits_from_LPM,
16_bits_vrf_id} to retrieve a nexthop.
> And we can provide to the user with ability to specify custom
> alloc/free function for these vectors.
> That would help to avoid allocating huge chunks of memory at startup.
> I understand that it will be one extra memory dereference,
> but probably it will be not that critical in terms of performance .
> Again for bulk function we might be able to pipeline lookups and
> de-references and hide that extra load latency.
>
>> Add four new experimental APIs:
>> - rte_fib_vrf_add() and rte_fib_vrf_delete() to manage routes
>> per VRF
>> - rte_fib_vrf_lookup_bulk() for multi-VRF bulk lookups
>> - rte_fib_vrf_get_rib() to retrieve a per-VRF RIB handle
>>
>> Signed-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
>> ---
>> lib/fib/dir24_8.c | 241 ++++++++++++++++------
>> lib/fib/dir24_8.h | 255 ++++++++++++++++--------
>> lib/fib/dir24_8_avx512.c | 420 +++++++++++++++++++++++++++++++--------
>> lib/fib/dir24_8_avx512.h | 80 +++++++-
>> lib/fib/rte_fib.c | 158 ++++++++++++---
>> lib/fib/rte_fib.h | 94 ++++++++-
>> 6 files changed, 988 insertions(+), 260 deletions(-)
>>
<snip>
--
Regards,
Vladimir
^ permalink raw reply [flat|nested] 45+ messages in thread
* RE: [RFC PATCH 1/4] fib: add multi-VRF support
2026-03-23 19:06 ` Medvedkin, Vladimir
@ 2026-03-23 22:22 ` Konstantin Ananyev
2026-03-25 14:09 ` Medvedkin, Vladimir
0 siblings, 1 reply; 45+ messages in thread
From: Konstantin Ananyev @ 2026-03-23 22:22 UTC (permalink / raw)
To: Medvedkin, Vladimir, dev@dpdk.org
Cc: rjarry@redhat.com, nsaxena16@gmail.com, mb@smartsharesystems.com,
adwivedi@marvell.com, jerinjacobk@gmail.com, Maxime Leroy
> >> Add VRF (Virtual Routing and Forwarding) support to the IPv4
> >> FIB library, allowing multiple independent routing tables
> >> within a single FIB instance.
> >>
> >> Introduce max_vrfs and vrf_default_nh fields in rte_fib_conf
> >> to configure the number of VRFs and per-VRF default nexthops.
> > Thanks Vladimir, allowing multiple VRFs per same LPM table will
> > definitely be a useful thing to have.
> > Though, I have the same concern as Maxime:
> > memory requirements are just overwhelming.
> > Stupid q - why just not to store a pointer to a vector of next-hops
> > within the table entry?
>
> Am I understand correctly, a vector with max_number_of_vrfs entries and
> use vrf id to address a nexthop?
Yes.
> Yes, this may work.
> But, if we are going to do an extra memory access, I'd better to
> maintain an internal hash table with 5 byte keys {24_bits_from_LPM,
> 16_bits_vrf_id} to retrieve a nexthop.
Hmm... and what to do with entries in tbl8, I mean what will be the key for them?
Or you don't plan to put entries from tbl8 to that hash table?
> > And we can provide to the user with ability to specify custom
> > alloc/free function for these vectors.
> > That would help to avoid allocating huge chunks of memory at startup.
> > I understand that it will be one extra memory dereference,
> > but probably it will be not that critical in terms of performance .
> > Again for bulk function we might be able to pipeline lookups and
> > de-references and hide that extra load latency.
> >
> >> Add four new experimental APIs:
> >> - rte_fib_vrf_add() and rte_fib_vrf_delete() to manage routes
> >> per VRF
> >> - rte_fib_vrf_lookup_bulk() for multi-VRF bulk lookups
> >> - rte_fib_vrf_get_rib() to retrieve a per-VRF RIB handle
> >>
> >> Signed-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
> >> ---
> >> lib/fib/dir24_8.c | 241 ++++++++++++++++------
> >> lib/fib/dir24_8.h | 255 ++++++++++++++++--------
> >> lib/fib/dir24_8_avx512.c | 420 +++++++++++++++++++++++++++++++--------
> >> lib/fib/dir24_8_avx512.h | 80 +++++++-
> >> lib/fib/rte_fib.c | 158 ++++++++++++---
> >> lib/fib/rte_fib.h | 94 ++++++++-
> >> 6 files changed, 988 insertions(+), 260 deletions(-)
> >>
> <snip>
>
> --
> Regards,
> Vladimir
>
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH 0/4] VRF support in FIB library
2026-03-23 18:42 ` Medvedkin, Vladimir
@ 2026-03-24 9:19 ` Maxime Leroy
2026-03-25 15:56 ` Medvedkin, Vladimir
0 siblings, 1 reply; 45+ messages in thread
From: Maxime Leroy @ 2026-03-24 9:19 UTC (permalink / raw)
To: Medvedkin, Vladimir; +Cc: dev, rjarry, nsaxena16, mb, adwivedi, jerinjacobk
Hi Vladimir,
On Mon, Mar 23, 2026 at 7:46 PM Medvedkin, Vladimir
<vladimir.medvedkin@intel.com> wrote:
>
>
> On 3/23/2026 2:53 PM, Maxime Leroy wrote:
> > On Mon, Mar 23, 2026 at 1:49 PM Medvedkin, Vladimir
> > <vladimir.medvedkin@intel.com> wrote:
> >> Hi Maxime,
> >>
> >> On 3/23/2026 11:27 AM, Maxime Leroy wrote:
> >>> Hi Vladimir,
> >>>
> >>>
> >>> On Sun, Mar 22, 2026 at 4:42 PM Vladimir Medvedkin
> >>> <vladimir.medvedkin@intel.com> wrote:
> >>>> This series adds multi-VRF support to both IPv4 and IPv6 FIB paths by
> >>>> allowing a single FIB instance to host multiple isolated routing domains.
> >>>>
> >>>> Currently FIB instance represents one routing instance. For workloads that
> >>>> need multiple VRFs, the only option is to create multiple FIB objects. In a
> >>>> burst oriented datapath, packets in the same batch can belong to different VRFs, so
> >>>> the application either does per-packet lookup in different FIB instances or
> >>>> regroups packets by VRF before lookup. Both approaches are expensive.
> >>>>
> >>>> To remove that cost, this series keeps all VRFs inside one FIB instance and
> >>>> extends lookup input with per-packet VRF IDs.
> >>>>
> >>>> The design follows the existing fast-path structure for both families. IPv4 and
> >>>> IPv6 use multi-ary trees with a 2^24 associativity on a first level (tbl24). The
> >>>> first-level table scales per configured VRF. This increases memory usage, but
> >>>> keeps performance and lookup complexity on par with non-VRF implementation.
> >>>>
> >>> Thanks for the RFC. Some thoughts below.
> >>>
> >>> Memory cost: the flat TBL24 replicates the entire table for every VRF
> >>> (num_vrfs * 2^24 * nh_size). With 256 VRFs and 8B nexthops that is
> >>> 32 GB for TBL24 alone. In grout we support up to 256 VRFs allocated
> >>> on demand -- this approach forces the full cost upfront even if most
> >>> VRFs are empty.
> >> Yes, increased memory consumption is the
> >> trade-off.WemakethischoiceinDPDKquite often,such as pre-allocatedmbufs,
> >> mempoolsand many other stuff allocated in advance to gain performance.
> >> For FIB, I chose to replicate TBL24 per VRF for this same reason.
> >>
> >> And, as Morten mentioned earlier, if memory is the priority, a table
> >> instance per VRF allocated on-demand is still supported.
> >>
> >> The high memory cost stems from TBL24's design: for IPv4, it was
> >> justified by the BGP filtering convention (no prefixes more specific
> >> than /24 in BGPv4 full view), ensuring most lookups hit with just one
> >> random memory access. For IPv6, we should likely switch to a 16-bit TRIE
> >> scheme on all layers. For IPv4, alternative algorithms with smaller
> >> footprints (like DXR or DIR16-8-8, as used in VPP) may be worth
> >> exploring if BGP full view is not required for those VRFs.
> >>
> >>> Per-packet VRF lookup: Rx bursts come from one port, thus one VRF.
> >>> Mixed-VRF bulk lookups do not occur in practice. The three AVX512
> >>> code paths add complexity for a scenario that does not exist, at
> >>> least for a classic router. Am I missing a use-case?
> >> That's not true, you're missing out on a lot of established core use
> >> cases that are at least 2 decades old:
> >>
> >> - VLAN subinterface abstraction. Each subinterface may belong to a
> >> separate VRF
> >>
> >> - MPLS VPN
> >>
> >> - Policy based routing
> >>
> > Fair point on VLAN subinterfaces and MPLS VPN. SRv6 L3VPN (End.DT4/
> > End.DT6) also fits that pattern after decap.
> >
> > I agree DPDK often pre-allocates for performance, but I wonder if the
> > flat TBL24 actually helps here. Each VRF's working set is spread
> > 128 MB apart in the flat table. Would regrouping packets by VRF and
> > doing one bulk lookup per VRF with separate contiguous TBL24s be
> > more cache-friendly than a single mixed-VRF gather? Do you have
> > benchmarks comparing the two approaches?
>
> It depends. Generally, if we assume that we are working with wide
> internet traffic, then even for a single VRF we most likely will miss
> the cache for TLB24, thus, regardless of the size of the tbl24, each
> memory access will be performed directly to DRAM.
If the lookup is DRAM-bound anyway, then the 10 cycles/addr cost
is dominated by memory latency, not CPU. The CPU cost of a bucket
sort on 32-64 packets is negligible next to a DRAM access (~80-100
ns per cache miss). That actually makes the case for regroup +
per-VRF lookup: the regrouping is pure CPU work hidden behind
memory stalls, and each per-VRF bulk lookup hits a contiguous
TBL24 instead of scattering across 128 MB-apart VRF regions.
> And if the addresses are localized (i.e. most traffic is internal), then
> having multiple TBL24 won'tmake the situationmuchworse.
>
With localized traffic, regrouping by VRF + per-VRF lookup on
contiguous TBL24s would benefit from cache locality, while the
flat multi-VRF table spreads hot entries 128 MB apart. The flat
approach may actually be worse in that scenario
> I don't have any benchmarks for regrouping, however I have 2 things to
> consider:
>
> 1. lookup is relatively fast (for IPv4 it is about 10 cycles per
> address, and I don't really want to slow it down)
>
> 2. incoming addresses and their corresponding VRFs are not controlled by
> "us", so this is a random set. Regrouping effectively is sorting. I'm
> not really happy to have nlogn complexity on a fast path :)
Without benchmarks, we do not know whether the flat approach is
actually faster than regroup + per-VRF lookup.
>
> >
> > On the memory trade-off and VRF ID mapping: the API uses vrf_id as
> > a direct index (0 to max_vrfs-1). With 256 VRFs and 8B nexthops,
> > TBL24 alone costs 32 GB for IPv4 and 32 GB for IPv6 -- 64 GB total
> > at startup. In grout, VRF IDs are interface IDs that can be any
> > uint16_t, so we would also need to maintain a mapping between our
> > VRF IDs and FIB slot indices.
> of course, this is an application responsibility. In FIB VRFs are in
> continuous range.
> > We would need to introduce a max_vrfs
> > limit, which forces a bad trade-off: either set it low (e.g. 16)
> > and limit deployments, or set it high (e.g. 256) and pay 64 GB at
> > startup even with a single VRF. With separate FIB instances per VRF,
> > we only allocate what we use.
> Yes, I understand this. In the end, if the user wants to use 256 VRFs,
> the amount of memory footprint will be at least 64Gb anyway.
The difference is when the memory is committed. With separate FIB
instances per VRF, you allocate 128 MB only when a VRF is actually
created at runtime. With the flat multi-VRF approach, you pay
max_vrfs * 128 MB at startup, even if only one VRF is active.
On top of that, the API uses vrf_id as a direct index (0 to
max_vrfs-1). As Stephen noted, there are multiple ways to model
VRFs. Depending on the networking stack, VRFs are identified by
ifindex (Linux l3mdev), by name (Cisco, Juniper), or by some
other scheme. This means the application must maintain a mapping
between its own VRF representation and the FIB slot indices, and
choose max_vrfs upfront. What is the benefit of this flat
multi-VRF FIB if the application still needs to manage a
translation layer and pre-commit memory for VRFs that may never
exist?
> As a trade-off for a bad trade-off ;) I can suggest to allocate it in
> chunks. Let's say you are starting with 16 VRFs, and during runtime, if
> the user wants to increase the number of VRFs above this limit, you can
> allocate another 16xVRF FIB. Then, of course, you need to split
> addresses into 2 bursts each for each FIB handle.
But then we are back to regrouping packets -- just by chunk of
VRFs instead of by individual VRF. If we have to sort the burst
anyway, what does the flat multi-VRF table buy us?
> >
> >>> I am not too familiar with DPDK FIB internals, but would it be
> >>> possible to keep a separate TBL24 per VRF and only share the TBL8
> >>> pool?
> >> it is how it is implemented right now with one note - TBL24 are pre
> >> allocated.
> >>> Something like pre-allocating an array of max_vrfs TBL24
> >>> pointers, allocating each TBL24 on demand at VRF add time,
> >> and you suggesting to allocate TBL24 on demand by adding an extra
> >> indirection layer. Thiswill leadtolowerperformance,whichIwouldliketo avoid.
> >>> and
> >>> having them all point into a shared TBL8 pool. The TBL8 index in
> >>> TBL24 entries seems to already be global, so would that work without
> >>> encoding changes?
> >>>
> >>> Going further: could the same idea extend to IPv6? The dir24_8 and
> >>> trie seem to use the same TBL8 block format (256 entries, same
> >>> (nh << 1) | ext_bit encoding, same size). Would unifying the TBL8
> >>> allocator allow a single pool shared across IPv4, IPv6, and all
> >>> VRFs? That could be a bigger win for /32-heavy and /128-heavy tables
> >>> and maybe a good first step before multi-VRF.
> >> So, you are suggesting merging IPv4 and IPv6 into a single unified FIB?
> >> I'm not sure how this can be a bigger win, could you please elaborate
> >> more on this?
> > On the IPv4/IPv6 TBL8 pool: I was not suggesting merging FIBs, just
> > sharing the TBL8 block allocator between separate FIB instances.
> > This is possible since dir24_8 and trie use the same TBL8 block
> > format (256 entries, same encoding, same size).
> >
> > Would it be possible to pass a shared TBL8 pool at rte_fib_create()
> > time? Each FIB keeps its own TBL24 and RIB, but TBL8 is shared
> > across all FIBs and potentially across IPv4/IPv6. Users would no
> > longer have to guess num_tbl8 per FIB.
> Yes, this is possible. However, this will significantly complicate the
> work with the library, solving a not so big problem.
Your series already shares TBL8 across all VRFs within a single
FIB -- that part is useful, and it does not require the flat
multi-VRF TBL24.
In grout, routes arrive from FRR (BGP, OSPF, etc.) at runtime.
We cannot predict TBL8 usage per VRF in advance -- it depends on
prefix length distribution which varies per VRF and changes over
time. No production LPM (Linux kernel, JunOS, IOS) asks the
operator to size these structures per routing table upfront.
Today we do not even have TBL8 usage stats (Robin's series
addresses that), and there is no way to resize a FIB without
destroying and recreating it.
Could you share performance numbers comparing the flat multi-VRF
lookup against regroup + per-VRF lookup?
> >>> Regards,
> >>>
> >>> Maxime Leroy
> >>>
> >>>> Vladimir Medvedkin (4):
> >>>> fib: add multi-VRF support
> >>>> fib: add VRF functional and unit tests
> >>>> fib6: add multi-VRF support
> >>>> fib6: add VRF functional and unit tests
> >>>>
> >>>> app/test-fib/main.c | 257 ++++++++++++++++++++++--
> >>>> app/test/test_fib.c | 298 +++++++++++++++++++++++++++
> >>>> app/test/test_fib6.c | 319 ++++++++++++++++++++++++++++-
> >>>> lib/fib/dir24_8.c | 241 ++++++++++++++++------
> >>>> lib/fib/dir24_8.h | 255 ++++++++++++++++--------
> >>>> lib/fib/dir24_8_avx512.c | 420 +++++++++++++++++++++++++++++++--------
> >>>> lib/fib/dir24_8_avx512.h | 80 +++++++-
> >>>> lib/fib/rte_fib.c | 158 ++++++++++++---
> >>>> lib/fib/rte_fib.h | 94 ++++++++-
> >>>> lib/fib/rte_fib6.c | 166 +++++++++++++---
> >>>> lib/fib/rte_fib6.h | 88 +++++++-
> >>>> lib/fib/trie.c | 158 +++++++++++----
> >>>> lib/fib/trie.h | 51 +++--
> >>>> lib/fib/trie_avx512.c | 225 +++++++++++++++++++--
> >>>> lib/fib/trie_avx512.h | 39 +++-
> >>>> 15 files changed, 2453 insertions(+), 396 deletions(-)
> >>>>
> >>>> --
> >>>> 2.43.0
> >>>>
> >> --
> >> Regards,
> >> Vladimir
> >>
> >
> --
> Regards,
> Vladimir
>
Regards,
Maxime
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH 1/4] fib: add multi-VRF support
2026-03-23 22:22 ` Konstantin Ananyev
@ 2026-03-25 14:09 ` Medvedkin, Vladimir
2026-03-26 10:13 ` Konstantin Ananyev
0 siblings, 1 reply; 45+ messages in thread
From: Medvedkin, Vladimir @ 2026-03-25 14:09 UTC (permalink / raw)
To: Konstantin Ananyev, dev@dpdk.org
Cc: rjarry@redhat.com, nsaxena16@gmail.com, mb@smartsharesystems.com,
adwivedi@marvell.com, jerinjacobk@gmail.com, Maxime Leroy
On 3/23/2026 10:22 PM, Konstantin Ananyev wrote:
>>>> Add VRF (Virtual Routing and Forwarding) support to the IPv4
>>>> FIB library, allowing multiple independent routing tables
>>>> within a single FIB instance.
>>>>
>>>> Introduce max_vrfs and vrf_default_nh fields in rte_fib_conf
>>>> to configure the number of VRFs and per-VRF default nexthops.
>>> Thanks Vladimir, allowing multiple VRFs per same LPM table will
>>> definitely be a useful thing to have.
>>> Though, I have the same concern as Maxime:
>>> memory requirements are just overwhelming.
>>> Stupid q - why just not to store a pointer to a vector of next-hops
>>> within the table entry?
>> Am I understand correctly, a vector with max_number_of_vrfs entries and
>> use vrf id to address a nexthop?
> Yes.
Here I can see 2 problems:
1. tbl entries must be the size of a pointer, so no way to use smaller sizes
2. those vectors will be sparsely populated and, depending on the
runtime configuration, may consume a lot of memory too (as Robin
mentioned they may have 1024 VRFs)
>
>> Yes, this may work.
>> But, if we are going to do an extra memory access, I'd better to
>> maintain an internal hash table with 5 byte keys {24_bits_from_LPM,
>> 16_bits_vrf_id} to retrieve a nexthop.
> Hmm... and what to do with entries in tbl8, I mean what will be the key for them?
> Or you don't plan to put entries from tbl8 to that hash table?
The idea is to have a single LPM struct with a join superset of all
prefixes existing in all VRFs. Each prefix in this LPM struct has its
own unique "nexthop", which is not the final next hop, but an
intermediate metadata defining this unique prefix. Then, the following
search is performed with the key containing this intermediate metadata +
vrf_id in some exact match database like hash table. This approach is
the most memory friendly, since there is only one LPM data struct (which
scales well with number of prefixes it has) with intermediate entries
only 4b long.
On the other hand it requires an extra search, so lookup will be slower.
Also, some current LPM optimizations, like tbl8 collapsing if all tbl8
entries have a similar value, will be gone.
>
>>> And we can provide to the user with ability to specify custom
>>> alloc/free function for these vectors.
>>> That would help to avoid allocating huge chunks of memory at startup.
>>> I understand that it will be one extra memory dereference,
>>> but probably it will be not that critical in terms of performance .
>>> Again for bulk function we might be able to pipeline lookups and
>>> de-references and hide that extra load latency.
>>>
>>>> Add four new experimental APIs:
>>>> - rte_fib_vrf_add() and rte_fib_vrf_delete() to manage routes
>>>> per VRF
>>>> - rte_fib_vrf_lookup_bulk() for multi-VRF bulk lookups
>>>> - rte_fib_vrf_get_rib() to retrieve a per-VRF RIB handle
>>>>
>>>> Signed-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
>>>> ---
>>>> lib/fib/dir24_8.c | 241 ++++++++++++++++------
>>>> lib/fib/dir24_8.h | 255 ++++++++++++++++--------
>>>> lib/fib/dir24_8_avx512.c | 420 +++++++++++++++++++++++++++++++--------
>>>> lib/fib/dir24_8_avx512.h | 80 +++++++-
>>>> lib/fib/rte_fib.c | 158 ++++++++++++---
>>>> lib/fib/rte_fib.h | 94 ++++++++-
>>>> 6 files changed, 988 insertions(+), 260 deletions(-)
>>>>
>> <snip>
>>
>> --
>> Regards,
>> Vladimir
>>
--
Regards,
Vladimir
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH 0/4] VRF support in FIB library
2026-03-24 9:19 ` Maxime Leroy
@ 2026-03-25 15:56 ` Medvedkin, Vladimir
2026-03-25 21:43 ` Maxime Leroy
0 siblings, 1 reply; 45+ messages in thread
From: Medvedkin, Vladimir @ 2026-03-25 15:56 UTC (permalink / raw)
To: Maxime Leroy; +Cc: dev, rjarry, nsaxena16, mb, adwivedi, jerinjacobk
On 3/24/2026 9:19 AM, Maxime Leroy wrote:
> Hi Vladimir,
>
> On Mon, Mar 23, 2026 at 7:46 PM Medvedkin, Vladimir
> <vladimir.medvedkin@intel.com> wrote:
>>
>> On 3/23/2026 2:53 PM, Maxime Leroy wrote:
>>> On Mon, Mar 23, 2026 at 1:49 PM Medvedkin, Vladimir
>>> <vladimir.medvedkin@intel.com> wrote:
>>>> Hi Maxime,
>>>>
>>>> On 3/23/2026 11:27 AM, Maxime Leroy wrote:
>>>>> Hi Vladimir,
>>>>>
>>>>>
>>>>> On Sun, Mar 22, 2026 at 4:42 PM Vladimir Medvedkin
>>>>> <vladimir.medvedkin@intel.com> wrote:
>>>>>> This series adds multi-VRF support to both IPv4 and IPv6 FIB paths by
>>>>>> allowing a single FIB instance to host multiple isolated routing domains.
>>>>>>
>>>>>> Currently FIB instance represents one routing instance. For workloads that
>>>>>> need multiple VRFs, the only option is to create multiple FIB objects. In a
>>>>>> burst oriented datapath, packets in the same batch can belong to different VRFs, so
>>>>>> the application either does per-packet lookup in different FIB instances or
>>>>>> regroups packets by VRF before lookup. Both approaches are expensive.
>>>>>>
>>>>>> To remove that cost, this series keeps all VRFs inside one FIB instance and
>>>>>> extends lookup input with per-packet VRF IDs.
>>>>>>
>>>>>> The design follows the existing fast-path structure for both families. IPv4 and
>>>>>> IPv6 use multi-ary trees with a 2^24 associativity on a first level (tbl24). The
>>>>>> first-level table scales per configured VRF. This increases memory usage, but
>>>>>> keeps performance and lookup complexity on par with non-VRF implementation.
>>>>>>
>>>>> Thanks for the RFC. Some thoughts below.
>>>>>
>>>>> Memory cost: the flat TBL24 replicates the entire table for every VRF
>>>>> (num_vrfs * 2^24 * nh_size). With 256 VRFs and 8B nexthops that is
>>>>> 32 GB for TBL24 alone. In grout we support up to 256 VRFs allocated
>>>>> on demand -- this approach forces the full cost upfront even if most
>>>>> VRFs are empty.
>>>> Yes, increased memory consumption is the
>>>> trade-off.WemakethischoiceinDPDKquite often,such as pre-allocatedmbufs,
>>>> mempoolsand many other stuff allocated in advance to gain performance.
>>>> For FIB, I chose to replicate TBL24 per VRF for this same reason.
>>>>
>>>> And, as Morten mentioned earlier, if memory is the priority, a table
>>>> instance per VRF allocated on-demand is still supported.
>>>>
>>>> The high memory cost stems from TBL24's design: for IPv4, it was
>>>> justified by the BGP filtering convention (no prefixes more specific
>>>> than /24 in BGPv4 full view), ensuring most lookups hit with just one
>>>> random memory access. For IPv6, we should likely switch to a 16-bit TRIE
>>>> scheme on all layers. For IPv4, alternative algorithms with smaller
>>>> footprints (like DXR or DIR16-8-8, as used in VPP) may be worth
>>>> exploring if BGP full view is not required for those VRFs.
>>>>
>>>>> Per-packet VRF lookup: Rx bursts come from one port, thus one VRF.
>>>>> Mixed-VRF bulk lookups do not occur in practice. The three AVX512
>>>>> code paths add complexity for a scenario that does not exist, at
>>>>> least for a classic router. Am I missing a use-case?
>>>> That's not true, you're missing out on a lot of established core use
>>>> cases that are at least 2 decades old:
>>>>
>>>> - VLAN subinterface abstraction. Each subinterface may belong to a
>>>> separate VRF
>>>>
>>>> - MPLS VPN
>>>>
>>>> - Policy based routing
>>>>
>>> Fair point on VLAN subinterfaces and MPLS VPN. SRv6 L3VPN (End.DT4/
>>> End.DT6) also fits that pattern after decap.
>>>
>>> I agree DPDK often pre-allocates for performance, but I wonder if the
>>> flat TBL24 actually helps here. Each VRF's working set is spread
>>> 128 MB apart in the flat table. Would regrouping packets by VRF and
>>> doing one bulk lookup per VRF with separate contiguous TBL24s be
>>> more cache-friendly than a single mixed-VRF gather? Do you have
>>> benchmarks comparing the two approaches?
>> It depends. Generally, if we assume that we are working with wide
>> internet traffic, then even for a single VRF we most likely will miss
>> the cache for TLB24, thus, regardless of the size of the tbl24, each
>> memory access will be performed directly to DRAM.
> If the lookup is DRAM-bound anyway, then the 10 cycles/addr cost
> is dominated by memory latency, not CPU. The CPU cost of a bucket
> sort on 32-64 packets is negligible next to a DRAM access (~80-100
> ns per cache miss).
memory accesses are independent and executed in parallel in the CPU
pipeline
> That actually makes the case for regroup +
> per-VRF lookup: the regrouping is pure CPU work hidden behind
> memory stalls,
regrouping must be performed before memory accesses, so it cannot be
amortized in between of memory reads
> and each per-VRF bulk lookup hits a contiguous
> TBL24 instead of scattering across 128 MB-apart VRF regions.
why is a contiguous 128Mb single-VRF TBL24 OK for you, but bigger
contiguous multi-VRF TBL24 is not OK in the context of lookup (here we
are talking about lookup, omitting the problem of memory consumption on
init)?
In both of these cases, memory access behaves the same way within a
single batch of packets during lookup, i.e. the first hit is likely a
cache miss, regardless of whether we are dealing with one or more VRFs,
it will not maintain TBL24 in L3$ in any way in a real dataplane app.
>
>> And if the addresses are localized (i.e. most traffic is internal), then
>> having multiple TBL24 won'tmake the situationmuchworse.
>>
> With localized traffic, regrouping by VRF + per-VRF lookup on
> contiguous TBL24s would benefit from cache locality,
why so? There will be no differences within a single batch with a
reasonable size (for example 64), because within the lookup session, no
matter with or without regrouping, temporal cache locality will be the same.
Let't look at it from a different angle. Is it
worthregroupingipaddressesby/8(i.e. 8 MSBs)withthe
currentimplementationof a singleVRFFIB?
> while the
> flat multi-VRF table spreads hot entries 128 MB apart. The flat
> approach may actually be worse in that scenario
>
>> I don't have any benchmarks for regrouping, however I have 2 things to
>> consider:
>>
>> 1. lookup is relatively fast (for IPv4 it is about 10 cycles per
>> address, and I don't really want to slow it down)
>>
>> 2. incoming addresses and their corresponding VRFs are not controlled by
>> "us", so this is a random set. Regrouping effectively is sorting. I'm
>> not really happy to have nlogn complexity on a fast path :)
> Without benchmarks, we do not know whether the flat approach is
> actually faster than regroup + per-VRF lookup.
feel free to share benchmark results. The only thing you need to add is
the packets regrouping logic, and then use separate single-VRF FIB
instances.
>>> On the memory trade-off and VRF ID mapping: the API uses vrf_id as
>>> a direct index (0 to max_vrfs-1). With 256 VRFs and 8B nexthops,
>>> TBL24 alone costs 32 GB for IPv4 and 32 GB for IPv6 -- 64 GB total
>>> at startup. In grout, VRF IDs are interface IDs that can be any
>>> uint16_t, so we would also need to maintain a mapping between our
>>> VRF IDs and FIB slot indices.
>> of course, this is an application responsibility. In FIB VRFs are in
>> continuous range.
>>> We would need to introduce a max_vrfs
>>> limit, which forces a bad trade-off: either set it low (e.g. 16)
>>> and limit deployments, or set it high (e.g. 256) and pay 64 GB at
>>> startup even with a single VRF. With separate FIB instances per VRF,
>>> we only allocate what we use.
>> Yes, I understand this. In the end, if the user wants to use 256 VRFs,
>> the amount of memory footprint will be at least 64Gb anyway.
> The difference is when the memory is committed.
yes, this's the only difference. It all comes down to the static vs
dynamic memory allocation problem. And each of these approaches is good
for solving a specific task. For the task of creating a new VRF, what is
more preferable - to fail on init or runtime?
> With separate FIB
> instances per VRF, you allocate 128 MB only when a VRF is actually
> created at runtime. With the flat multi-VRF approach, you pay
> max_vrfs * 128 MB at startup, even if only one VRF is active.
>
> On top of that, the API uses vrf_id as a direct index (0 to
> max_vrfs-1). As Stephen noted, there are multiple ways to model
> VRFs. Depending on the networking stack, VRFs are identified by
> ifindex (Linux l3mdev), by name (Cisco, Juniper), or by some
> other scheme. This means the application must maintain a mapping
> between its own VRF representation and the FIB slot indices, and
> choose max_vrfs upfront. What is the benefit of this flat
> multi-VRF FIB if the application still needs to manage a
> translation layer and pre-commit memory for VRFs that may never
> exist?
This is the control plane task.
>
>> As a trade-off for a bad trade-off ;) I can suggest to allocate it in
>> chunks. Let's say you are starting with 16 VRFs, and during runtime, if
>> the user wants to increase the number of VRFs above this limit, you can
>> allocate another 16xVRF FIB. Then, of course, you need to split
>> addresses into 2 bursts each for each FIB handle.
> But then we are back to regrouping packets -- just by chunk of
> VRFs instead of by individual VRF. If we have to sort the burst
> anyway, what does the flat multi-VRF table buy us?
>
>>>>> I am not too familiar with DPDK FIB internals, but would it be
>>>>> possible to keep a separate TBL24 per VRF and only share the TBL8
>>>>> pool?
>>>> it is how it is implemented right now with one note - TBL24 are pre
>>>> allocated.
>>>>> Something like pre-allocating an array of max_vrfs TBL24
>>>>> pointers, allocating each TBL24 on demand at VRF add time,
>>>> and you suggesting to allocate TBL24 on demand by adding an extra
>>>> indirection layer. Thiswill leadtolowerperformance,whichIwouldliketo avoid.
>>>>> and
>>>>> having them all point into a shared TBL8 pool. The TBL8 index in
>>>>> TBL24 entries seems to already be global, so would that work without
>>>>> encoding changes?
>>>>>
>>>>> Going further: could the same idea extend to IPv6? The dir24_8 and
>>>>> trie seem to use the same TBL8 block format (256 entries, same
>>>>> (nh << 1) | ext_bit encoding, same size). Would unifying the TBL8
>>>>> allocator allow a single pool shared across IPv4, IPv6, and all
>>>>> VRFs? That could be a bigger win for /32-heavy and /128-heavy tables
>>>>> and maybe a good first step before multi-VRF.
>>>> So, you are suggesting merging IPv4 and IPv6 into a single unified FIB?
>>>> I'm not sure how this can be a bigger win, could you please elaborate
>>>> more on this?
>>> On the IPv4/IPv6 TBL8 pool: I was not suggesting merging FIBs, just
>>> sharing the TBL8 block allocator between separate FIB instances.
>>> This is possible since dir24_8 and trie use the same TBL8 block
>>> format (256 entries, same encoding, same size).
>>>
>>> Would it be possible to pass a shared TBL8 pool at rte_fib_create()
>>> time? Each FIB keeps its own TBL24 and RIB, but TBL8 is shared
>>> across all FIBs and potentially across IPv4/IPv6. Users would no
>>> longer have to guess num_tbl8 per FIB.
>> Yes, this is possible. However, this will significantly complicate the
>> work with the library, solving a not so big problem.
> Your series already shares TBL8 across all VRFs within a single
> FIB -- that part is useful, and it does not require the flat
> multi-VRF TBL24.
>
> In grout, routes arrive from FRR (BGP, OSPF, etc.) at runtime.
> We cannot predict TBL8 usage per VRF in advance
and you don't need it (knowing per-VRF consumption) now. If I understood
your request here properly, do you want to share TBL8 between ipv4 and
ipv6 FIBs? I don't think this is a good idea at all. At least because
this is a good idea to split them in case if one AF consumes (because of
attack/bogus CP) all TBL8, another AF remains intact.
> -- it depends on
> prefix length distribution which varies per VRF and changes over
> time. No production LPM (Linux kernel, JunOS, IOS) asks the
> operator to size these structures per routing table upfront.
- they are using different lpm algorithms
- you use these facilities, developers properly tuned them. FIB is a low
level library, it cannot be used without any knowledge, it will not
solve all the problems with a single red button "make it work, don't do
any bugs"
P.S. how do you know how JunOS /IOS implements their LPMs? ;)
> Today we do not even have TBL8 usage stats (Robin's series
> addresses that)
I will tryto findtimetoreviewthispatchinthe nearfuture
> , and there is no way to resize a FIB without
> destroying and recreating it.
>
> Could you share performance numbers comparing the flat multi-VRF
> lookup against regroup + per-VRF lookup?
>
>>>>> Regards,
>>>>>
>>>>> Maxime Leroy
>>>>>
>>>>>> Vladimir Medvedkin (4):
>>>>>> fib: add multi-VRF support
>>>>>> fib: add VRF functional and unit tests
>>>>>> fib6: add multi-VRF support
>>>>>> fib6: add VRF functional and unit tests
>>>>>>
>>>>>> app/test-fib/main.c | 257 ++++++++++++++++++++++--
>>>>>> app/test/test_fib.c | 298 +++++++++++++++++++++++++++
>>>>>> app/test/test_fib6.c | 319 ++++++++++++++++++++++++++++-
>>>>>> lib/fib/dir24_8.c | 241 ++++++++++++++++------
>>>>>> lib/fib/dir24_8.h | 255 ++++++++++++++++--------
>>>>>> lib/fib/dir24_8_avx512.c | 420 +++++++++++++++++++++++++++++++--------
>>>>>> lib/fib/dir24_8_avx512.h | 80 +++++++-
>>>>>> lib/fib/rte_fib.c | 158 ++++++++++++---
>>>>>> lib/fib/rte_fib.h | 94 ++++++++-
>>>>>> lib/fib/rte_fib6.c | 166 +++++++++++++---
>>>>>> lib/fib/rte_fib6.h | 88 +++++++-
>>>>>> lib/fib/trie.c | 158 +++++++++++----
>>>>>> lib/fib/trie.h | 51 +++--
>>>>>> lib/fib/trie_avx512.c | 225 +++++++++++++++++++--
>>>>>> lib/fib/trie_avx512.h | 39 +++-
>>>>>> 15 files changed, 2453 insertions(+), 396 deletions(-)
>>>>>>
>>>>>> --
>>>>>> 2.43.0
>>>>>>
>>>> --
>>>> Regards,
>>>> Vladimir
>>>>
>> --
>> Regards,
>> Vladimir
>>
> Regards,
>
> Maxime
--
Regards,
Vladimir
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH 0/4] VRF support in FIB library
2026-03-25 15:56 ` Medvedkin, Vladimir
@ 2026-03-25 21:43 ` Maxime Leroy
2026-03-27 18:27 ` Medvedkin, Vladimir
0 siblings, 1 reply; 45+ messages in thread
From: Maxime Leroy @ 2026-03-25 21:43 UTC (permalink / raw)
To: Medvedkin, Vladimir; +Cc: dev, rjarry, nsaxena16, mb, adwivedi, jerinjacobk
Hi Vladimir,
On Wed, Mar 25, 2026 at 4:56 PM Medvedkin, Vladimir
<vladimir.medvedkin@intel.com> wrote:
>
>
> On 3/24/2026 9:19 AM, Maxime Leroy wrote:
> > Hi Vladimir,
> >
> > On Mon, Mar 23, 2026 at 7:46 PM Medvedkin, Vladimir
> > <vladimir.medvedkin@intel.com> wrote:
> >>
> >> On 3/23/2026 2:53 PM, Maxime Leroy wrote:
> >>> On Mon, Mar 23, 2026 at 1:49 PM Medvedkin, Vladimir
> >>> <vladimir.medvedkin@intel.com> wrote:
> >>>> Hi Maxime,
> >>>>
> >>>> On 3/23/2026 11:27 AM, Maxime Leroy wrote:
> >>>>> Hi Vladimir,
> >>>>>
> >>>>>
> >>>>> On Sun, Mar 22, 2026 at 4:42 PM Vladimir Medvedkin
> >>>>> <vladimir.medvedkin@intel.com> wrote:
> >>>>>> This series adds multi-VRF support to both IPv4 and IPv6 FIB paths by
> >>>>>> allowing a single FIB instance to host multiple isolated routing domains.
> >>>>>>
> >>>>>> Currently FIB instance represents one routing instance. For workloads that
> >>>>>> need multiple VRFs, the only option is to create multiple FIB objects. In a
> >>>>>> burst oriented datapath, packets in the same batch can belong to different VRFs, so
> >>>>>> the application either does per-packet lookup in different FIB instances or
> >>>>>> regroups packets by VRF before lookup. Both approaches are expensive.
> >>>>>>
> >>>>>> To remove that cost, this series keeps all VRFs inside one FIB instance and
> >>>>>> extends lookup input with per-packet VRF IDs.
> >>>>>>
> >>>>>> The design follows the existing fast-path structure for both families. IPv4 and
> >>>>>> IPv6 use multi-ary trees with a 2^24 associativity on a first level (tbl24). The
> >>>>>> first-level table scales per configured VRF. This increases memory usage, but
> >>>>>> keeps performance and lookup complexity on par with non-VRF implementation.
> >>>>>>
> >>>>> Thanks for the RFC. Some thoughts below.
> >>>>>
> >>>>> Memory cost: the flat TBL24 replicates the entire table for every VRF
> >>>>> (num_vrfs * 2^24 * nh_size). With 256 VRFs and 8B nexthops that is
> >>>>> 32 GB for TBL24 alone. In grout we support up to 256 VRFs allocated
> >>>>> on demand -- this approach forces the full cost upfront even if most
> >>>>> VRFs are empty.
> >>>> Yes, increased memory consumption is the
> >>>> trade-off.WemakethischoiceinDPDKquite often,such as pre-allocatedmbufs,
> >>>> mempoolsand many other stuff allocated in advance to gain performance.
> >>>> For FIB, I chose to replicate TBL24 per VRF for this same reason.
> >>>>
> >>>> And, as Morten mentioned earlier, if memory is the priority, a table
> >>>> instance per VRF allocated on-demand is still supported.
> >>>>
> >>>> The high memory cost stems from TBL24's design: for IPv4, it was
> >>>> justified by the BGP filtering convention (no prefixes more specific
> >>>> than /24 in BGPv4 full view), ensuring most lookups hit with just one
> >>>> random memory access. For IPv6, we should likely switch to a 16-bit TRIE
> >>>> scheme on all layers. For IPv4, alternative algorithms with smaller
> >>>> footprints (like DXR or DIR16-8-8, as used in VPP) may be worth
> >>>> exploring if BGP full view is not required for those VRFs.
> >>>>
> >>>>> Per-packet VRF lookup: Rx bursts come from one port, thus one VRF.
> >>>>> Mixed-VRF bulk lookups do not occur in practice. The three AVX512
> >>>>> code paths add complexity for a scenario that does not exist, at
> >>>>> least for a classic router. Am I missing a use-case?
> >>>> That's not true, you're missing out on a lot of established core use
> >>>> cases that are at least 2 decades old:
> >>>>
> >>>> - VLAN subinterface abstraction. Each subinterface may belong to a
> >>>> separate VRF
> >>>>
> >>>> - MPLS VPN
> >>>>
> >>>> - Policy based routing
> >>>>
> >>> Fair point on VLAN subinterfaces and MPLS VPN. SRv6 L3VPN (End.DT4/
> >>> End.DT6) also fits that pattern after decap.
> >>>
> >>> I agree DPDK often pre-allocates for performance, but I wonder if the
> >>> flat TBL24 actually helps here. Each VRF's working set is spread
> >>> 128 MB apart in the flat table. Would regrouping packets by VRF and
> >>> doing one bulk lookup per VRF with separate contiguous TBL24s be
> >>> more cache-friendly than a single mixed-VRF gather? Do you have
> >>> benchmarks comparing the two approaches?
> >> It depends. Generally, if we assume that we are working with wide
> >> internet traffic, then even for a single VRF we most likely will miss
> >> the cache for TLB24, thus, regardless of the size of the tbl24, each
> >> memory access will be performed directly to DRAM.
> > If the lookup is DRAM-bound anyway, then the 10 cycles/addr cost
> > is dominated by memory latency, not CPU. The CPU cost of a bucket
> > sort on 32-64 packets is negligible next to a DRAM access (~80-100
> > ns per cache miss).
> memory accesses are independent and executed in parallel in the CPU
> pipeline
> > That actually makes the case for regroup +
> > per-VRF lookup: the regrouping is pure CPU work hidden behind
> > memory stalls,
> regrouping must be performed before memory accesses, so it cannot be
> amortized in between of memory reads
With internet traffic, TBL24 lookups quickly become limited by
cache misses, not CPU cycles. Even if some bursts hit the same
routes and benefit from cache locality, the CPU has a limited
number of outstanding misses (load buffer entries, MSHRs) --
out-of-order execution helps, but it is not magic.
The whole point of vector/graph processing (VPP, DPDK graph, etc.)
is to amortize that memory latency: prefetch for packet N+1 while
processing packet N. This works because all packets in a batch
hit the same data structure in a tight loop.
With separate per-VRF TBL24s, a bucket sort by VRF -- a few
dozen cycles, all in L1 -- gives you clean batches where
prefetching works as designed. This is exactly what graph nodes
already do: classify, then process per-class in a tight loop.
> > and each per-VRF bulk lookup hits a contiguous
> > TBL24 instead of scattering across 128 MB-apart VRF regions.
>
> why is a contiguous 128Mb single-VRF TBL24 OK for you, but bigger
> contiguous multi-VRF TBL24 is not OK in the context of lookup (here we
> are talking about lookup, omitting the problem of memory consumption on
> init)?
The performance difference may be small, but the flat approach
is not faster either -- while costing 64 GB upfront.
> In both of these cases, memory access behaves the same way within a
> single batch of packets during lookup, i.e. the first hit is likely a
> cache miss, regardless of whether we are dealing with one or more VRFs,
> it will not maintain TBL24 in L3$ in any way in a real dataplane app.
>
> >
> >> And if the addresses are localized (i.e. most traffic is internal), then
> >> having multiple TBL24 won'tmake the situationmuchworse.
> >>
> > With localized traffic, regrouping by VRF + per-VRF lookup on
> > contiguous TBL24s would benefit from cache locality,
>
> why so? There will be no differences within a single batch with a
> reasonable size (for example 64), because within the lookup session, no
> matter with or without regrouping, temporal cache locality will be the same.
>
> Let't look at it from a different angle. Is it
> worthregroupingipaddressesby/8(i.e. 8 MSBs)withthe
> currentimplementationof a singleVRFFIB?
>
> > while the
> > flat multi-VRF table spreads hot entries 128 MB apart. The flat
> > approach may actually be worse in that scenario
> >
> >> I don't have any benchmarks for regrouping, however I have 2 things to
> >> consider:
> >>
> >> 1. lookup is relatively fast (for IPv4 it is about 10 cycles per
> >> address, and I don't really want to slow it down)
> >>
> >> 2. incoming addresses and their corresponding VRFs are not controlled by
> >> "us", so this is a random set. Regrouping effectively is sorting. I'm
> >> not really happy to have nlogn complexity on a fast path :)
> > Without benchmarks, we do not know whether the flat approach is
> > actually faster than regroup + per-VRF lookup.
> feel free to share benchmark results. The only thing you need to add is
> the packets regrouping logic, and then use separate single-VRF FIB
> instances.
Your series introduces a new API that optimizes multi-VRF lookup.
The performance numbers should come with the proposal.
> >>> On the memory trade-off and VRF ID mapping: the API uses vrf_id as
> >>> a direct index (0 to max_vrfs-1). With 256 VRFs and 8B nexthops,
> >>> TBL24 alone costs 32 GB for IPv4 and 32 GB for IPv6 -- 64 GB total
> >>> at startup. In grout, VRF IDs are interface IDs that can be any
> >>> uint16_t, so we would also need to maintain a mapping between our
> >>> VRF IDs and FIB slot indices.
> >> of course, this is an application responsibility. In FIB VRFs are in
> >> continuous range.
> >>> We would need to introduce a max_vrfs
> >>> limit, which forces a bad trade-off: either set it low (e.g. 16)
> >>> and limit deployments, or set it high (e.g. 256) and pay 64 GB at
> >>> startup even with a single VRF. With separate FIB instances per VRF,
> >>> we only allocate what we use.
> >> Yes, I understand this. In the end, if the user wants to use 256 VRFs,
> >> the amount of memory footprint will be at least 64Gb anyway.
> > The difference is when the memory is committed.
> yes, this's the only difference. It all comes down to the static vs
> dynamic memory allocation problem. And each of these approaches is good
> for solving a specific task. For the task of creating a new VRF, what is
> more preferable - to fail on init or runtime?
The main problem is that your series imposes contiguous VRF IDs
(0 to max_vrfs-1). How a VRF is represented is a network stack
design decision -- in Linux it is an ifindex, in Cisco a name,
in grout an interface ID. Any application using this API needs
a mapping layer on top.
In grout, everything is allocated dynamically: mempools, FIBs,
conntrack tables. Pre-allocating everything at init forces
hardcoded arbitrary limits and prevents memory reuse between
subsystems -- memory reserved for FIB TBL24 cannot be used for
conntrack when the VRF has no routes, and vice versa. We prefer
to allocate resources only when needed. It is simpler for users
and more efficient for memory.
> > With separate FIB
> > instances per VRF, you allocate 128 MB only when a VRF is actually
> > created at runtime. With the flat multi-VRF approach, you pay
> > max_vrfs * 128 MB at startup, even if only one VRF is active.
> >
> > On top of that, the API uses vrf_id as a direct index (0 to
> > max_vrfs-1). As Stephen noted, there are multiple ways to model
> > VRFs. Depending on the networking stack, VRFs are identified by
> > ifindex (Linux l3mdev), by name (Cisco, Juniper), or by some
> > other scheme. This means the application must maintain a mapping
> > between its own VRF representation and the FIB slot indices, and
> > choose max_vrfs upfront. What is the benefit of this flat
> > multi-VRF FIB if the application still needs to manage a
> > translation layer and pre-commit memory for VRFs that may never
> > exist?
> This is the control plane task.
> >
> >> As a trade-off for a bad trade-off ;) I can suggest to allocate it in
> >> chunks. Let's say you are starting with 16 VRFs, and during runtime, if
> >> the user wants to increase the number of VRFs above this limit, you can
> >> allocate another 16xVRF FIB. Then, of course, you need to split
> >> addresses into 2 bursts each for each FIB handle.
> > But then we are back to regrouping packets -- just by chunk of
> > VRFs instead of by individual VRF. If we have to sort the burst
> > anyway, what does the flat multi-VRF table buy us?
> >
> >>>>> I am not too familiar with DPDK FIB internals, but would it be
> >>>>> possible to keep a separate TBL24 per VRF and only share the TBL8
> >>>>> pool?
> >>>> it is how it is implemented right now with one note - TBL24 are pre
> >>>> allocated.
> >>>>> Something like pre-allocating an array of max_vrfs TBL24
> >>>>> pointers, allocating each TBL24 on demand at VRF add time,
> >>>> and you suggesting to allocate TBL24 on demand by adding an extra
> >>>> indirection layer. Thiswill leadtolowerperformance,whichIwouldliketo avoid.
> >>>>> and
> >>>>> having them all point into a shared TBL8 pool. The TBL8 index in
> >>>>> TBL24 entries seems to already be global, so would that work without
> >>>>> encoding changes?
> >>>>>
> >>>>> Going further: could the same idea extend to IPv6? The dir24_8 and
> >>>>> trie seem to use the same TBL8 block format (256 entries, same
> >>>>> (nh << 1) | ext_bit encoding, same size). Would unifying the TBL8
> >>>>> allocator allow a single pool shared across IPv4, IPv6, and all
> >>>>> VRFs? That could be a bigger win for /32-heavy and /128-heavy tables
> >>>>> and maybe a good first step before multi-VRF.
> >>>> So, you are suggesting merging IPv4 and IPv6 into a single unified FIB?
> >>>> I'm not sure how this can be a bigger win, could you please elaborate
> >>>> more on this?
> >>> On the IPv4/IPv6 TBL8 pool: I was not suggesting merging FIBs, just
> >>> sharing the TBL8 block allocator between separate FIB instances.
> >>> This is possible since dir24_8 and trie use the same TBL8 block
> >>> format (256 entries, same encoding, same size).
> >>>
> >>> Would it be possible to pass a shared TBL8 pool at rte_fib_create()
> >>> time? Each FIB keeps its own TBL24 and RIB, but TBL8 is shared
> >>> across all FIBs and potentially across IPv4/IPv6. Users would no
> >>> longer have to guess num_tbl8 per FIB.
> >> Yes, this is possible. However, this will significantly complicate the
> >> work with the library, solving a not so big problem.
> > Your series already shares TBL8 across all VRFs within a single
> > FIB -- that part is useful, and it does not require the flat
> > multi-VRF TBL24.
> >
> > In grout, routes arrive from FRR (BGP, OSPF, etc.) at runtime.
> > We cannot predict TBL8 usage per VRF in advance
> and you don't need it (knowing per-VRF consumption) now. If I understood
> your request here properly, do you want to share TBL8 between ipv4 and
> ipv6 FIBs? I don't think this is a good idea at all. At least because
> this is a good idea to split them in case if one AF consumes (because of
> attack/bogus CP) all TBL8, another AF remains intact.
If TBL8 isolation per AF is meant as a protection against route
floods, then the same argument applies between VRFs: your series
shares TBL8 across all VRFs within a single FIB, so a bogus
control plane in one VRF exhausts TBL8 for all other VRFs.
But more fundamentally, this is not how route flood protection
works. It is handled in the control plane: the routing daemon
limits the number of prefixes accepted per BGP session
(max-prefix) and selects which routes are installed via prefix
filters -- before those routes ever reach the forwarding table.
The Linux kernel is a good reference here. IPv6 used to enforce
a max_size limit on FIB + cache entries (net.ipv6.route.max_size,
defaulting to 4096). It caused real production issues and was
removed in kernel 6.3. IPv4 never had a FIB route limit. There
is no per-VRF route limit either. The kernel relies entirely on
the control plane for route flood protection.
> > -- it depends on
> > prefix length distribution which varies per VRF and changes over
> > time. No production LPM (Linux kernel, JunOS, IOS) asks the
> > operator to size these structures per routing table upfront.
>
> - they are using different lpm algorithms
> - you use these facilities, developers properly tuned them. FIB is a low
> level library, it cannot be used without any knowledge, it will not
> solve all the problems with a single red button "make it work, don't do
> any bugs"
> P.S. how do you know how JunOS /IOS implements their LPMs? ;)
>
I do not need to know their LPM implementation -- I only need
to know how they are configured. No production router requires
the operator to size internal LPM structures.
We can impose a maximum number of IPv4/IPv6 routes on the user
-- even though the kernel does not need this either. But TBL8
is a different problem: the application cannot predict TBL8
consumption because it depends on prefix length distribution,
which varies per VRF and changes over time with dynamic routing.
Today there is no API to query TBL8 usage, and no API to resize
a FIB without destroying it.
This is exactly why a shared TBL8 pool across VRFs is useful:
VRFs with few long prefixes naturally leave room for VRFs that
need more. This is the valuable part of your series. But it
does not require a flat multi-VRF TBL24 -- separate per-VRF
TBL24s sharing a common TBL8 pool would give the same benefit
without the 64 GB upfront cost.
> > Today we do not even have TBL8 usage stats (Robin's series
> > addresses that)
> I will tryto findtimetoreviewthispatchinthe nearfuture
Thanks, Robin's TBL8 stats series would help users understand
their TBL8 consumption -- a more practical improvement for
current users.
Regards,
Maxime
^ permalink raw reply [flat|nested] 45+ messages in thread
* RE: [RFC PATCH 1/4] fib: add multi-VRF support
2026-03-25 14:09 ` Medvedkin, Vladimir
@ 2026-03-26 10:13 ` Konstantin Ananyev
2026-03-27 18:32 ` Medvedkin, Vladimir
0 siblings, 1 reply; 45+ messages in thread
From: Konstantin Ananyev @ 2026-03-26 10:13 UTC (permalink / raw)
To: Medvedkin, Vladimir, dev@dpdk.org
Cc: rjarry@redhat.com, nsaxena16@gmail.com, mb@smartsharesystems.com,
adwivedi@marvell.com, jerinjacobk@gmail.com, Maxime Leroy
> >>>> Add VRF (Virtual Routing and Forwarding) support to the IPv4
> >>>> FIB library, allowing multiple independent routing tables
> >>>> within a single FIB instance.
> >>>>
> >>>> Introduce max_vrfs and vrf_default_nh fields in rte_fib_conf
> >>>> to configure the number of VRFs and per-VRF default nexthops.
> >>> Thanks Vladimir, allowing multiple VRFs per same LPM table will
> >>> definitely be a useful thing to have.
> >>> Though, I have the same concern as Maxime:
> >>> memory requirements are just overwhelming.
> >>> Stupid q - why just not to store a pointer to a vector of next-hops
> >>> within the table entry?
> >> Am I understand correctly, a vector with max_number_of_vrfs entries and
> >> use vrf id to address a nexthop?
> > Yes.
>
> Here I can see 2 problems:
>
> 1. tbl entries must be the size of a pointer, so no way to use smaller sizes
Yes, but as we are talking about storing nexthops for multiple VRFs anyway,
I don't think it is a big deal.
> 2. those vectors will be sparsely populated and, depending on the
> runtime configuration, may consume a lot of memory too (as Robin
> mentioned they may have 1024 VRFs)
Yeas, each VRF vector can become really sparse and we waste a lot of memory.
If that's an issue, we probably can think about something smarter
then simple flat array indexed by vrf-id: something like 2-level B-tree or so.
The main positives that I see in that approach:
- low extra overhead at lookup - one/two extra pointer de-refernces.
- it allows CP to allocate/free space for each such vecto separately,
so we don't need to pre-allocate memory for max possible entries at startup.
> >
> >> Yes, this may work.
> >> But, if we are going to do an extra memory access, I'd better to
> >> maintain an internal hash table with 5 byte keys {24_bits_from_LPM,
> >> 16_bits_vrf_id} to retrieve a nexthop.
> > Hmm... and what to do with entries in tbl8, I mean what will be the key for
> them?
> > Or you don't plan to put entries from tbl8 to that hash table?
>
> The idea is to have a single LPM struct with a join superset of all
> prefixes existing in all VRFs. Each prefix in this LPM struct has its
> own unique "nexthop", which is not the final next hop, but an
> intermediate metadata defining this unique prefix. Then, the following
> search is performed with the key containing this intermediate metadata +
> vrf_id in some exact match database like hash table. This approach is
> the most memory friendly, since there is only one LPM data struct (which
> scales well with number of prefixes it has) with intermediate entries
> only 4b long.
> On the other hand it requires an extra search, so lookup will be slower.
> Also, some current LPM optimizations, like tbl8 collapsing if all tbl8
> entries have a similar value, will be gone.
Yes, and yes :)
Yes it would help to save memory, and yes lookup will most likely be slower.
The other thing that I consider as a possible drawback here - with current rte_hash
implementation we still need to allocate space for all possible max entries at startup.
But that's not new in DPDK, and for most cases it is considered as acceptable trade-off.
Overall, it seems like a possible approach to me, I suppose the main question is:
what will be the price of that extra hash-lookup here.
Again there is a bulk version of hash lookup and in theory it might be it can be
improved further (avx512 version on x86?).
>
> >
> >>> And we can provide to the user with ability to specify custom
> >>> alloc/free function for these vectors.
> >>> That would help to avoid allocating huge chunks of memory at startup.
> >>> I understand that it will be one extra memory dereference,
> >>> but probably it will be not that critical in terms of performance .
> >>> Again for bulk function we might be able to pipeline lookups and
> >>> de-references and hide that extra load latency.
> >>>
> >>>> Add four new experimental APIs:
> >>>> - rte_fib_vrf_add() and rte_fib_vrf_delete() to manage routes
> >>>> per VRF
> >>>> - rte_fib_vrf_lookup_bulk() for multi-VRF bulk lookups
> >>>> - rte_fib_vrf_get_rib() to retrieve a per-VRF RIB handle
> >>>>
> >>>> Signed-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
> >>>> ---
> >>>> lib/fib/dir24_8.c | 241 ++++++++++++++++------
> >>>> lib/fib/dir24_8.h | 255 ++++++++++++++++--------
> >>>> lib/fib/dir24_8_avx512.c | 420 +++++++++++++++++++++++++++++++--------
> >>>> lib/fib/dir24_8_avx512.h | 80 +++++++-
> >>>> lib/fib/rte_fib.c | 158 ++++++++++++---
> >>>> lib/fib/rte_fib.h | 94 ++++++++-
> >>>> 6 files changed, 988 insertions(+), 260 deletions(-)
> >>>>
> >> <snip>
> >>
> >> --
> >> Regards,
> >> Vladimir
> >>
> --
> Regards,
> Vladimir
>
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH 0/4] VRF support in FIB library
2026-03-25 21:43 ` Maxime Leroy
@ 2026-03-27 18:27 ` Medvedkin, Vladimir
2026-04-02 16:51 ` Maxime Leroy
0 siblings, 1 reply; 45+ messages in thread
From: Medvedkin, Vladimir @ 2026-03-27 18:27 UTC (permalink / raw)
To: Maxime Leroy
Cc: dev, rjarry, nsaxena16, mb, adwivedi, jerinjacobk,
Vladimir Medvedkin
Hi Maxime,
On 3/25/2026 9:43 PM, Maxime Leroy wrote:
> Hi Vladimir,
>
> On Wed, Mar 25, 2026 at 4:56 PM Medvedkin, Vladimir
> <vladimir.medvedkin@intel.com> wrote:
>>
>> On 3/24/2026 9:19 AM, Maxime Leroy wrote:
>>> Hi Vladimir,
>>>
>>> On Mon, Mar 23, 2026 at 7:46 PM Medvedkin, Vladimir
>>> <vladimir.medvedkin@intel.com> wrote:
>>>> On 3/23/2026 2:53 PM, Maxime Leroy wrote:
>>>>> On Mon, Mar 23, 2026 at 1:49 PM Medvedkin, Vladimir
>>>>> <vladimir.medvedkin@intel.com> wrote:
>>>>>> Hi Maxime,
>>>>>>
>>>>>> On 3/23/2026 11:27 AM, Maxime Leroy wrote:
>>>>>>> Hi Vladimir,
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Mar 22, 2026 at 4:42 PM Vladimir Medvedkin
>>>>>>> <vladimir.medvedkin@intel.com> wrote:
>>>>>>>> This series adds multi-VRF support to both IPv4 and IPv6 FIB paths by
>>>>>>>> allowing a single FIB instance to host multiple isolated routing domains.
>>>>>>>>
>>>>>>>> Currently FIB instance represents one routing instance. For workloads that
>>>>>>>> need multiple VRFs, the only option is to create multiple FIB objects. In a
>>>>>>>> burst oriented datapath, packets in the same batch can belong to different VRFs, so
>>>>>>>> the application either does per-packet lookup in different FIB instances or
>>>>>>>> regroups packets by VRF before lookup. Both approaches are expensive.
>>>>>>>>
>>>>>>>> To remove that cost, this series keeps all VRFs inside one FIB instance and
>>>>>>>> extends lookup input with per-packet VRF IDs.
>>>>>>>>
>>>>>>>> The design follows the existing fast-path structure for both families. IPv4 and
>>>>>>>> IPv6 use multi-ary trees with a 2^24 associativity on a first level (tbl24). The
>>>>>>>> first-level table scales per configured VRF. This increases memory usage, but
>>>>>>>> keeps performance and lookup complexity on par with non-VRF implementation.
>>>>>>>>
>>>>>>> Thanks for the RFC. Some thoughts below.
>>>>>>>
>>>>>>> Memory cost: the flat TBL24 replicates the entire table for every VRF
>>>>>>> (num_vrfs * 2^24 * nh_size). With 256 VRFs and 8B nexthops that is
>>>>>>> 32 GB for TBL24 alone. In grout we support up to 256 VRFs allocated
>>>>>>> on demand -- this approach forces the full cost upfront even if most
>>>>>>> VRFs are empty.
>>>>>> Yes, increased memory consumption is the
>>>>>> trade-off.WemakethischoiceinDPDKquite often,such as pre-allocatedmbufs,
>>>>>> mempoolsand many other stuff allocated in advance to gain performance.
>>>>>> For FIB, I chose to replicate TBL24 per VRF for this same reason.
>>>>>>
>>>>>> And, as Morten mentioned earlier, if memory is the priority, a table
>>>>>> instance per VRF allocated on-demand is still supported.
>>>>>>
>>>>>> The high memory cost stems from TBL24's design: for IPv4, it was
>>>>>> justified by the BGP filtering convention (no prefixes more specific
>>>>>> than /24 in BGPv4 full view), ensuring most lookups hit with just one
>>>>>> random memory access. For IPv6, we should likely switch to a 16-bit TRIE
>>>>>> scheme on all layers. For IPv4, alternative algorithms with smaller
>>>>>> footprints (like DXR or DIR16-8-8, as used in VPP) may be worth
>>>>>> exploring if BGP full view is not required for those VRFs.
>>>>>>
>>>>>>> Per-packet VRF lookup: Rx bursts come from one port, thus one VRF.
>>>>>>> Mixed-VRF bulk lookups do not occur in practice. The three AVX512
>>>>>>> code paths add complexity for a scenario that does not exist, at
>>>>>>> least for a classic router. Am I missing a use-case?
>>>>>> That's not true, you're missing out on a lot of established core use
>>>>>> cases that are at least 2 decades old:
>>>>>>
>>>>>> - VLAN subinterface abstraction. Each subinterface may belong to a
>>>>>> separate VRF
>>>>>>
>>>>>> - MPLS VPN
>>>>>>
>>>>>> - Policy based routing
>>>>>>
>>>>> Fair point on VLAN subinterfaces and MPLS VPN. SRv6 L3VPN (End.DT4/
>>>>> End.DT6) also fits that pattern after decap.
>>>>>
>>>>> I agree DPDK often pre-allocates for performance, but I wonder if the
>>>>> flat TBL24 actually helps here. Each VRF's working set is spread
>>>>> 128 MB apart in the flat table. Would regrouping packets by VRF and
>>>>> doing one bulk lookup per VRF with separate contiguous TBL24s be
>>>>> more cache-friendly than a single mixed-VRF gather? Do you have
>>>>> benchmarks comparing the two approaches?
>>>> It depends. Generally, if we assume that we are working with wide
>>>> internet traffic, then even for a single VRF we most likely will miss
>>>> the cache for TLB24, thus, regardless of the size of the tbl24, each
>>>> memory access will be performed directly to DRAM.
>>> If the lookup is DRAM-bound anyway, then the 10 cycles/addr cost
>>> is dominated by memory latency, not CPU. The CPU cost of a bucket
>>> sort on 32-64 packets is negligible next to a DRAM access (~80-100
>>> ns per cache miss).
>> memory accesses are independent and executed in parallel in the CPU
>> pipeline
>>> That actually makes the case for regroup +
>>> per-VRF lookup: the regrouping is pure CPU work hidden behind
>>> memory stalls,
>> regrouping must be performed before memory accesses, so it cannot be
>> amortized in between of memory reads
> With internet traffic, TBL24 lookups quickly become limited by
> cache misses, not CPU cycles. Even if some bursts hit the same
> routes and benefit from cache locality, the CPU has a limited
> number of outstanding misses (load buffer entries, MSHRs) --
> out-of-order execution helps, but it is not magic.
Correct, but this does not contradict to what I'm saying
>
> The whole point of vector/graph processing (VPP, DPDK graph, etc.)
> is to amortize that memory latency: prefetch for packet N+1 while
> processing packet N. This works because all packets in a batch
> hit the same data structure in a tight loop.
https://github.com/DPDK/dpdk/blob/626d4e39327333cd5508885162e45ca7fb94ef7f/lib/fib/dir24_8.h#L161
>
> With separate per-VRF TBL24s, a bucket sort by VRF -- a few
> dozen cycles, all in L1 -- gives you clean batches where
> prefetching works as designed. This is exactly what graph nodes
> already do: classify, then process per-class in a tight loop.
How lookup is performed in this design? Am I understand it right:
1. sort the batch by VRF ids, splitting IPs from the batch with IP
sub-batches belonging to the same VRF id
2. for each subset of IPs perform lookup in tbl24[batch_common_vrf_id]
3. unsort nexthops
Correct?
>
>>> and each per-VRF bulk lookup hits a contiguous
>>> TBL24 instead of scattering across 128 MB-apart VRF regions.
>> why is a contiguous 128Mb single-VRF TBL24 OK for you, but bigger
>> contiguous multi-VRF TBL24 is not OK in the context of lookup (here we
>> are talking about lookup, omitting the problem of memory consumption on
>> init)?
> The performance difference may be small, but the flat approach
> is not faster either -- while costing 64 GB upfront.
it seems you are implicitly take an assumption of 256 VRFs. Is my
usecase with a few VRFs have a right to exist?
>
>> In both of these cases, memory access behaves the same way within a
>> single batch of packets during lookup, i.e. the first hit is likely a
>> cache miss, regardless of whether we are dealing with one or more VRFs,
>> it will not maintain TBL24 in L3$ in any way in a real dataplane app.
>>
>>>> And if the addresses are localized (i.e. most traffic is internal), then
>>>> having multiple TBL24 won'tmake the situationmuchworse.
>>>>
>>> With localized traffic, regrouping by VRF + per-VRF lookup on
>>> contiguous TBL24s would benefit from cache locality,
>> why so? There will be no differences within a single batch with a
>> reasonable size (for example 64), because within the lookup session, no
>> matter with or without regrouping, temporal cache locality will be the same.
>>
>> Let't look at it from a different angle. Is it
>> worthregroupingipaddressesby/8(i.e. 8 MSBs)withthe
>> currentimplementationof a singleVRFFIB?
>>
>>> while the
>>> flat multi-VRF table spreads hot entries 128 MB apart. The flat
>>> approach may actually be worse in that scenario
>>>
>>>> I don't have any benchmarks for regrouping, however I have 2 things to
>>>> consider:
>>>>
>>>> 1. lookup is relatively fast (for IPv4 it is about 10 cycles per
>>>> address, and I don't really want to slow it down)
>>>>
>>>> 2. incoming addresses and their corresponding VRFs are not controlled by
>>>> "us", so this is a random set. Regrouping effectively is sorting. I'm
>>>> not really happy to have nlogn complexity on a fast path :)
>>> Without benchmarks, we do not know whether the flat approach is
>>> actually faster than regroup + per-VRF lookup.
>> feel free to share benchmark results. The only thing you need to add is
>> the packets regrouping logic, and then use separate single-VRF FIB
>> instances.
> Your series introduces a new API that optimizes multi-VRF lookup.
> The performance numbers should come with the proposal.
By the policy we can not share raw performance numbers, and I think this
is unnecessary, because performance depends on the testing environment
(content of the routing table, CPU model, etc).
Tests I've done on my board with ipv4 full-view (782940 routes) with 4
VRFs performing random lookup in all of them was 180% in cost comparing
to a single VRF with the same RT content.
You can test it in your environment with
dpdk-test-fib -l 1,2 --no-pci -- -f <path to your routes> -e 4 -l
100000000 -V <number of VRFs>
>
>>>>> On the memory trade-off and VRF ID mapping: the API uses vrf_id as
>>>>> a direct index (0 to max_vrfs-1). With 256 VRFs and 8B nexthops,
>>>>> TBL24 alone costs 32 GB for IPv4 and 32 GB for IPv6 -- 64 GB total
>>>>> at startup. In grout, VRF IDs are interface IDs that can be any
>>>>> uint16_t, so we would also need to maintain a mapping between our
>>>>> VRF IDs and FIB slot indices.
>>>> of course, this is an application responsibility. In FIB VRFs are in
>>>> continuous range.
>>>>> We would need to introduce a max_vrfs
>>>>> limit, which forces a bad trade-off: either set it low (e.g. 16)
>>>>> and limit deployments, or set it high (e.g. 256) and pay 64 GB at
>>>>> startup even with a single VRF. With separate FIB instances per VRF,
>>>>> we only allocate what we use.
>>>> Yes, I understand this. In the end, if the user wants to use 256 VRFs,
>>>> the amount of memory footprint will be at least 64Gb anyway.
>>> The difference is when the memory is committed.
>> yes, this's the only difference. It all comes down to the static vs
>> dynamic memory allocation problem. And each of these approaches is good
>> for solving a specific task. For the task of creating a new VRF, what is
>> more preferable - to fail on init or runtime?
> The main problem is that your series imposes contiguous VRF IDs
> (0 to max_vrfs-1). How a VRF is represented is a network stack
> design decision
exactly - a network stack decision. FIB is not a network stack.
> -- in Linux it is an ifindex,
so every interface lives in it's own private VRF?
> in Cisco a name,
are you going to pass an array of strings on lookup?
> in grout an interface ID.
haven't we decided this is a problematic design(VLANs, L3VPN, etc)?
> Any application using this API needs
> a mapping layer on top.
I think from my rhetoric questions this should be obvious
>
> In grout, everything is allocated dynamically: mempools, FIBs,
> conntrack tables. Pre-allocating everything at init forces
> hardcoded arbitrary limits and prevents memory reuse between
> subsystems -- memory reserved for FIB TBL24 cannot be used for
> conntrack when the VRF has no routes, and vice versa. We prefer
> to allocate resources only when needed. It is simpler for users
> and more efficient for memory.
>
>>> With separate FIB
>>> instances per VRF, you allocate 128 MB only when a VRF is actually
>>> created at runtime. With the flat multi-VRF approach, you pay
>>> max_vrfs * 128 MB at startup, even if only one VRF is active.
>>>
>>> On top of that, the API uses vrf_id as a direct index (0 to
>>> max_vrfs-1). As Stephen noted, there are multiple ways to model
>>> VRFs. Depending on the networking stack, VRFs are identified by
>>> ifindex (Linux l3mdev), by name (Cisco, Juniper), or by some
>>> other scheme. This means the application must maintain a mapping
>>> between its own VRF representation and the FIB slot indices, and
>>> choose max_vrfs upfront. What is the benefit of this flat
>>> multi-VRF FIB if the application still needs to manage a
>>> translation layer and pre-commit memory for VRFs that may never
>>> exist?
>> This is the control plane task.
>>>> As a trade-off for a bad trade-off ;) I can suggest to allocate it in
>>>> chunks. Let's say you are starting with 16 VRFs, and during runtime, if
>>>> the user wants to increase the number of VRFs above this limit, you can
>>>> allocate another 16xVRF FIB. Then, of course, you need to split
>>>> addresses into 2 bursts each for each FIB handle.
>>> But then we are back to regrouping packets -- just by chunk of
>>> VRFs instead of by individual VRF. If we have to sort the burst
>>> anyway, what does the flat multi-VRF table buy us?
>>>
>>>>>>> I am not too familiar with DPDK FIB internals, but would it be
>>>>>>> possible to keep a separate TBL24 per VRF and only share the TBL8
>>>>>>> pool?
>>>>>> it is how it is implemented right now with one note - TBL24 are pre
>>>>>> allocated.
>>>>>>> Something like pre-allocating an array of max_vrfs TBL24
>>>>>>> pointers, allocating each TBL24 on demand at VRF add time,
>>>>>> and you suggesting to allocate TBL24 on demand by adding an extra
>>>>>> indirection layer. Thiswill leadtolowerperformance,whichIwouldliketo avoid.
>>>>>>> and
>>>>>>> having them all point into a shared TBL8 pool. The TBL8 index in
>>>>>>> TBL24 entries seems to already be global, so would that work without
>>>>>>> encoding changes?
>>>>>>>
>>>>>>> Going further: could the same idea extend to IPv6? The dir24_8 and
>>>>>>> trie seem to use the same TBL8 block format (256 entries, same
>>>>>>> (nh << 1) | ext_bit encoding, same size). Would unifying the TBL8
>>>>>>> allocator allow a single pool shared across IPv4, IPv6, and all
>>>>>>> VRFs? That could be a bigger win for /32-heavy and /128-heavy tables
>>>>>>> and maybe a good first step before multi-VRF.
>>>>>> So, you are suggesting merging IPv4 and IPv6 into a single unified FIB?
>>>>>> I'm not sure how this can be a bigger win, could you please elaborate
>>>>>> more on this?
>>>>> On the IPv4/IPv6 TBL8 pool: I was not suggesting merging FIBs, just
>>>>> sharing the TBL8 block allocator between separate FIB instances.
>>>>> This is possible since dir24_8 and trie use the same TBL8 block
>>>>> format (256 entries, same encoding, same size).
>>>>>
>>>>> Would it be possible to pass a shared TBL8 pool at rte_fib_create()
>>>>> time? Each FIB keeps its own TBL24 and RIB, but TBL8 is shared
>>>>> across all FIBs and potentially across IPv4/IPv6. Users would no
>>>>> longer have to guess num_tbl8 per FIB.
>>>> Yes, this is possible. However, this will significantly complicate the
>>>> work with the library, solving a not so big problem.
>>> Your series already shares TBL8 across all VRFs within a single
>>> FIB -- that part is useful, and it does not require the flat
>>> multi-VRF TBL24.
>>>
>>> In grout, routes arrive from FRR (BGP, OSPF, etc.) at runtime.
>>> We cannot predict TBL8 usage per VRF in advance
>> and you don't need it (knowing per-VRF consumption) now. If I understood
>> your request here properly, do you want to share TBL8 between ipv4 and
>> ipv6 FIBs? I don't think this is a good idea at all. At least because
>> this is a good idea to split them in case if one AF consumes (because of
>> attack/bogus CP) all TBL8, another AF remains intact.
> If TBL8 isolation per AF is meant as a protection against route
> floods, then the same argument applies between VRFs: your series
> shares TBL8 across all VRFs within a single FIB, so a bogus
> control plane in one VRF exhausts TBL8 for all other VRFs.
>
> But more fundamentally, this is not how route flood protection
> works. It is handled in the control plane: the routing daemon
> limits the number of prefixes accepted per BGP session
> (max-prefix) and selects which routes are installed via prefix
> filters -- before those routes ever reach the forwarding table.
>
> The Linux kernel is a good reference here. IPv6 used to enforce
> a max_size limit on FIB + cache entries (net.ipv6.route.max_size,
> defaulting to 4096). It caused real production issues and was
> removed in kernel 6.3. IPv4 never had a FIB route limit. There
> is no per-VRF route limit either. The kernel relies entirely on
> the control plane for route flood protection.
FIB is not the Linux kernel, as well as not a network stack. We can not
rely on a control plane protection, since control plane is a 3rd party
software.
Also, I think allocating a very algorithm-specific entity such as pool
of TBL8 prior to calling rte_fib_create and passing a pointer on it
could be confusing for many users and bloating API.
FIB supports pluggable lookup algorithms, you can write your own and
specify a pointer to the tbl8_pool in an algorithm-specific
configuration defined for your algorithm, where you may also create a
dynamic table of TBL24 pointers per each VRF. If you need any help with
this task - I would be happy to help.
>
>>> -- it depends on
>>> prefix length distribution which varies per VRF and changes over
>>> time. No production LPM (Linux kernel, JunOS, IOS) asks the
>>> operator to size these structures per routing table upfront.
>> - they are using different lpm algorithms
>> - you use these facilities, developers properly tuned them. FIB is a low
>> level library, it cannot be used without any knowledge, it will not
>> solve all the problems with a single red button "make it work, don't do
>> any bugs"
>> P.S. how do you know how JunOS /IOS implements their LPMs? ;)
>>
> I do not need to know their LPM implementation -- I only need
> to know how they are configured. No production router requires
> the operator to size internal LPM structures.
>
> We can impose a maximum number of IPv4/IPv6 routes on the user
> -- even though the kernel does not need this either. But TBL8
> is a different problem: the application cannot predict TBL8
> consumption because it depends on prefix length distribution,
> which varies per VRF and changes over time with dynamic routing.
> Today there is no API to query TBL8 usage, and no API to resize
> a FIB without destroying it.
>
> This is exactly why a shared TBL8 pool across VRFs is useful:
> VRFs with few long prefixes naturally leave room for VRFs that
> need more.
but this is already implemented. I don't know why you repeatedly
concerning about this. We are aligned about this and this feature is
already there - in the patch
On the other hand what we disagreed on is the sharing not only across
VRFs, but also across address families. If you don't understand the
amount of TBL8 per AF, how would you magically understand the number of
TBL8s for a merged pool?
> This is the valuable part of your series. But it
> does not require a flat multi-VRF TBL24 -- separate per-VRF
> TBL24s sharing a common TBL8 pool would give the same benefit
> without the 64 GB upfront cost.
andthat'sa completelydifferentproblem.Please,let's separatethe
problemsandnotmixthemup.
I understand your concern about memory consumption. Ihavesomeideason
howto solvethisproblem as a parallel to proposed solution.
>
>>> Today we do not even have TBL8 usage stats (Robin's series
>>> addresses that)
>> I will tryto findtimetoreviewthispatchinthe nearfuture
> Thanks, Robin's TBL8 stats series would help users understand
> their TBL8 consumption -- a more practical improvement for
> current users.
>
> Regards,
> Maxime
--
Regards,
Vladimir
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH 1/4] fib: add multi-VRF support
2026-03-26 10:13 ` Konstantin Ananyev
@ 2026-03-27 18:32 ` Medvedkin, Vladimir
0 siblings, 0 replies; 45+ messages in thread
From: Medvedkin, Vladimir @ 2026-03-27 18:32 UTC (permalink / raw)
To: Konstantin Ananyev, dev@dpdk.org
Cc: rjarry@redhat.com, nsaxena16@gmail.com, mb@smartsharesystems.com,
adwivedi@marvell.com, jerinjacobk@gmail.com, Maxime Leroy,
Vladimir Medvedkin
On 3/26/2026 10:13 AM, Konstantin Ananyev wrote:
>
>>>>>> Add VRF (Virtual Routing and Forwarding) support to the IPv4
>>>>>> FIB library, allowing multiple independent routing tables
>>>>>> within a single FIB instance.
>>>>>>
>>>>>> Introduce max_vrfs and vrf_default_nh fields in rte_fib_conf
>>>>>> to configure the number of VRFs and per-VRF default nexthops.
>>>>> Thanks Vladimir, allowing multiple VRFs per same LPM table will
>>>>> definitely be a useful thing to have.
>>>>> Though, I have the same concern as Maxime:
>>>>> memory requirements are just overwhelming.
>>>>> Stupid q - why just not to store a pointer to a vector of next-hops
>>>>> within the table entry?
>>>> Am I understand correctly, a vector with max_number_of_vrfs entries and
>>>> use vrf id to address a nexthop?
>>> Yes.
>> Here I can see 2 problems:
>>
>> 1. tbl entries must be the size of a pointer, so no way to use smaller sizes
> Yes, but as we are talking about storing nexthops for multiple VRFs anyway,
> I don't think it is a big deal.
>
>> 2. those vectors will be sparsely populated and, depending on the
>> runtime configuration, may consume a lot of memory too (as Robin
>> mentioned they may have 1024 VRFs)
> Yeas, each VRF vector can become really sparse and we waste a lot of memory.
> If that's an issue, we probably can think about something smarter
> then simple flat array indexed by vrf-id: something like 2-level B-tree or so.
> The main positives that I see in that approach:
> - low extra overhead at lookup - one/two extra pointer de-refernces.
I'm afraidtheoverheadwillbe
comparativelylargejustbecausethecurrentimplementationis fastandmost
likely hit with a single memory access. However, for a low number of
VRFs, B-tree may be a good solution
> - it allows CP to allocate/free space for each such vecto separately,
> so we don't need to pre-allocate memory for max possible entries at startup.
>
>>>> Yes, this may work.
>>>> But, if we are going to do an extra memory access, I'd better to
>>>> maintain an internal hash table with 5 byte keys {24_bits_from_LPM,
>>>> 16_bits_vrf_id} to retrieve a nexthop.
>>> Hmm... and what to do with entries in tbl8, I mean what will be the key for
>> them?
>>> Or you don't plan to put entries from tbl8 to that hash table?
>> The idea is to have a single LPM struct with a join superset of all
>> prefixes existing in all VRFs. Each prefix in this LPM struct has its
>> own unique "nexthop", which is not the final next hop, but an
>> intermediate metadata defining this unique prefix. Then, the following
>> search is performed with the key containing this intermediate metadata +
>> vrf_id in some exact match database like hash table. This approach is
>> the most memory friendly, since there is only one LPM data struct (which
>> scales well with number of prefixes it has) with intermediate entries
>> only 4b long.
>> On the other hand it requires an extra search, so lookup will be slower.
>> Also, some current LPM optimizations, like tbl8 collapsing if all tbl8
>> entries have a similar value, will be gone.
> Yes, and yes :)
> Yes it would help to save memory, and yes lookup will most likely be slower.
> The other thing that I consider as a possible drawback here - with current rte_hash
> implementation we still need to allocate space for all possible max entries at startup.
I don't think this is a big problem, since the size of this memory will
be reasonable and will not grow linearly with the number of VRFs. So I
agree it is an acceptable trade-off
> But that's not new in DPDK, and for most cases it is considered as acceptable trade-off.
> Overall, it seems like a possible approach to me, I suppose the main question is:
> what will be the price of that extra hash-lookup here.
And this is the key problem. I don't think rte_hash is well suitable
here, at best we need some kind of a perfect hash. I have a few ideas on
this, stay tuned :)
>
> Again there is a bulk version of hash lookup and in theory it might be it can be
> improved further (avx512 version on x86?).
>
>>>>> And we can provide to the user with ability to specify custom
>>>>> alloc/free function for these vectors.
>>>>> That would help to avoid allocating huge chunks of memory at startup.
>>>>> I understand that it will be one extra memory dereference,
>>>>> but probably it will be not that critical in terms of performance .
>>>>> Again for bulk function we might be able to pipeline lookups and
>>>>> de-references and hide that extra load latency.
>>>>>
>>>>>> Add four new experimental APIs:
>>>>>> - rte_fib_vrf_add() and rte_fib_vrf_delete() to manage routes
>>>>>> per VRF
>>>>>> - rte_fib_vrf_lookup_bulk() for multi-VRF bulk lookups
>>>>>> - rte_fib_vrf_get_rib() to retrieve a per-VRF RIB handle
>>>>>>
>>>>>> Signed-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
>>>>>> ---
>>>>>> lib/fib/dir24_8.c | 241 ++++++++++++++++------
>>>>>> lib/fib/dir24_8.h | 255 ++++++++++++++++--------
>>>>>> lib/fib/dir24_8_avx512.c | 420 +++++++++++++++++++++++++++++++--------
>>>>>> lib/fib/dir24_8_avx512.h | 80 +++++++-
>>>>>> lib/fib/rte_fib.c | 158 ++++++++++++---
>>>>>> lib/fib/rte_fib.h | 94 ++++++++-
>>>>>> 6 files changed, 988 insertions(+), 260 deletions(-)
>>>>>>
>>>> <snip>
>>>>
>>>> --
>>>> Regards,
>>>> Vladimir
>>>>
>> --
>> Regards,
>> Vladimir
>>
--
Regards,
Vladimir
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH 0/4] VRF support in FIB library
2026-03-27 18:27 ` Medvedkin, Vladimir
@ 2026-04-02 16:51 ` Maxime Leroy
0 siblings, 0 replies; 45+ messages in thread
From: Maxime Leroy @ 2026-04-02 16:51 UTC (permalink / raw)
To: Medvedkin, Vladimir
Cc: dev, rjarry, nsaxena16, mb, adwivedi, jerinjacobk,
Vladimir Medvedkin
Hi Vladimir,
On Fri, Mar 27, 2026 at 7:28 PM Medvedkin, Vladimir
<vladimir.medvedkin@intel.com> wrote:
>
> Hi Maxime,
>
> On 3/25/2026 9:43 PM, Maxime Leroy wrote:
> > Hi Vladimir,
> >
> > On Wed, Mar 25, 2026 at 4:56 PM Medvedkin, Vladimir
> > <vladimir.medvedkin@intel.com> wrote:
> >>
> >> On 3/24/2026 9:19 AM, Maxime Leroy wrote:
> >>> Hi Vladimir,
> >>>
> >>> On Mon, Mar 23, 2026 at 7:46 PM Medvedkin, Vladimir
> >>> <vladimir.medvedkin@intel.com> wrote:
> >>>> On 3/23/2026 2:53 PM, Maxime Leroy wrote:
> >>>>> On Mon, Mar 23, 2026 at 1:49 PM Medvedkin, Vladimir
> >>>>> <vladimir.medvedkin@intel.com> wrote:
> >>>>>> Hi Maxime,
> >>>>>>
> >>>>>> On 3/23/2026 11:27 AM, Maxime Leroy wrote:
> >>>>>>> Hi Vladimir,
> >>>>>>>
> >>>>>>>
> >>>>>>> On Sun, Mar 22, 2026 at 4:42 PM Vladimir Medvedkin
> >>>>>>> <vladimir.medvedkin@intel.com> wrote:
> >>>>>>>> This series adds multi-VRF support to both IPv4 and IPv6 FIB paths by
> >>>>>>>> allowing a single FIB instance to host multiple isolated routing domains.
> >>>>>>>>
> >>>>>>>> Currently FIB instance represents one routing instance. For workloads that
> >>>>>>>> need multiple VRFs, the only option is to create multiple FIB objects. In a
> >>>>>>>> burst oriented datapath, packets in the same batch can belong to different VRFs, so
> >>>>>>>> the application either does per-packet lookup in different FIB instances or
> >>>>>>>> regroups packets by VRF before lookup. Both approaches are expensive.
> >>>>>>>>
> >>>>>>>> To remove that cost, this series keeps all VRFs inside one FIB instance and
> >>>>>>>> extends lookup input with per-packet VRF IDs.
> >>>>>>>>
> >>>>>>>> The design follows the existing fast-path structure for both families. IPv4 and
> >>>>>>>> IPv6 use multi-ary trees with a 2^24 associativity on a first level (tbl24). The
> >>>>>>>> first-level table scales per configured VRF. This increases memory usage, but
> >>>>>>>> keeps performance and lookup complexity on par with non-VRF implementation.
> >>>>>>>>
> >>>>>>> Thanks for the RFC. Some thoughts below.
> >>>>>>>
> >>>>>>> Memory cost: the flat TBL24 replicates the entire table for every VRF
> >>>>>>> (num_vrfs * 2^24 * nh_size). With 256 VRFs and 8B nexthops that is
> >>>>>>> 32 GB for TBL24 alone. In grout we support up to 256 VRFs allocated
> >>>>>>> on demand -- this approach forces the full cost upfront even if most
> >>>>>>> VRFs are empty.
> >>>>>> Yes, increased memory consumption is the
> >>>>>> trade-off.WemakethischoiceinDPDKquite often,such as pre-allocatedmbufs,
> >>>>>> mempoolsand many other stuff allocated in advance to gain performance.
> >>>>>> For FIB, I chose to replicate TBL24 per VRF for this same reason.
> >>>>>>
> >>>>>> And, as Morten mentioned earlier, if memory is the priority, a table
> >>>>>> instance per VRF allocated on-demand is still supported.
> >>>>>>
> >>>>>> The high memory cost stems from TBL24's design: for IPv4, it was
> >>>>>> justified by the BGP filtering convention (no prefixes more specific
> >>>>>> than /24 in BGPv4 full view), ensuring most lookups hit with just one
> >>>>>> random memory access. For IPv6, we should likely switch to a 16-bit TRIE
> >>>>>> scheme on all layers. For IPv4, alternative algorithms with smaller
> >>>>>> footprints (like DXR or DIR16-8-8, as used in VPP) may be worth
> >>>>>> exploring if BGP full view is not required for those VRFs.
> >>>>>>
> >>>>>>> Per-packet VRF lookup: Rx bursts come from one port, thus one VRF.
> >>>>>>> Mixed-VRF bulk lookups do not occur in practice. The three AVX512
> >>>>>>> code paths add complexity for a scenario that does not exist, at
> >>>>>>> least for a classic router. Am I missing a use-case?
> >>>>>> That's not true, you're missing out on a lot of established core use
> >>>>>> cases that are at least 2 decades old:
> >>>>>>
> >>>>>> - VLAN subinterface abstraction. Each subinterface may belong to a
> >>>>>> separate VRF
> >>>>>>
> >>>>>> - MPLS VPN
> >>>>>>
> >>>>>> - Policy based routing
> >>>>>>
> >>>>> Fair point on VLAN subinterfaces and MPLS VPN. SRv6 L3VPN (End.DT4/
> >>>>> End.DT6) also fits that pattern after decap.
> >>>>>
> >>>>> I agree DPDK often pre-allocates for performance, but I wonder if the
> >>>>> flat TBL24 actually helps here. Each VRF's working set is spread
> >>>>> 128 MB apart in the flat table. Would regrouping packets by VRF and
> >>>>> doing one bulk lookup per VRF with separate contiguous TBL24s be
> >>>>> more cache-friendly than a single mixed-VRF gather? Do you have
> >>>>> benchmarks comparing the two approaches?
> >>>> It depends. Generally, if we assume that we are working with wide
> >>>> internet traffic, then even for a single VRF we most likely will miss
> >>>> the cache for TLB24, thus, regardless of the size of the tbl24, each
> >>>> memory access will be performed directly to DRAM.
> >>> If the lookup is DRAM-bound anyway, then the 10 cycles/addr cost
> >>> is dominated by memory latency, not CPU. The CPU cost of a bucket
> >>> sort on 32-64 packets is negligible next to a DRAM access (~80-100
> >>> ns per cache miss).
> >> memory accesses are independent and executed in parallel in the CPU
> >> pipeline
> >>> That actually makes the case for regroup +
> >>> per-VRF lookup: the regrouping is pure CPU work hidden behind
> >>> memory stalls,
> >> regrouping must be performed before memory accesses, so it cannot be
> >> amortized in between of memory reads
> > With internet traffic, TBL24 lookups quickly become limited by
> > cache misses, not CPU cycles. Even if some bursts hit the same
> > routes and benefit from cache locality, the CPU has a limited
> > number of outstanding misses (load buffer entries, MSHRs) --
> > out-of-order execution helps, but it is not magic.
> Correct, but this does not contradict to what I'm saying
> >
> > The whole point of vector/graph processing (VPP, DPDK graph, etc.)
> > is to amortize that memory latency: prefetch for packet N+1 while
> > processing packet N. This works because all packets in a batch
> > hit the same data structure in a tight loop.
> https://github.com/DPDK/dpdk/blob/626d4e39327333cd5508885162e45ca7fb94ef7f/lib/fib/dir24_8.h#L161
>
> >
> > With separate per-VRF TBL24s, a bucket sort by VRF -- a few
> > dozen cycles, all in L1 -- gives you clean batches where
> > prefetching works as designed. This is exactly what graph nodes
> > already do: classify, then process per-class in a tight loop.
> How lookup is performed in this design? Am I understand it right:
> 1. sort the batch by VRF ids, splitting IPs from the batch with IP
> sub-batches belonging to the same VRF id
> 2. for each subset of IPs perform lookup in tbl24[batch_common_vrf_id]
> 3. unsort nexthops
>
> Correct?
No sort/unsort. This is how rte_graph classification works:
ip_input (validation)
-> ip_lookup-v0 (bulk fib4_lookup on homogeneous VRF 0 burst)
-> ip_lookup-v1 (bulk fib4_lookup on homogeneous VRF 1 burst)
-> ip_forward / ip_input_local / ...
ip_input already iterates over packets for header validation and
enqueues them to different next nodes. Adding a per-VRF edge costs
one iface->vrf_id load (already in L1) and one rte_node_enqueue_x1()
(already done today). Each ip_lookup-vN clone holds its VRF's
rte_fib in node context and calls rte_fib_lookup_bulk() on the
whole burst at once.
We do not use bulk lookups yet in grout (each packet does its own
rte_fib_lookup_bulk(..., 1) today), but this is how we would
implement it.
The tradeoff is batch fragmentation: with traffic spread across K
active VRFs, each sub-batch is ~N/K packets. But in practice, most
deployments have 1-3 hot VRFs, so batches stay large. And even
fragmented batches benefit from the vectorized lookup -- 8 packets
is still one AVX512 iteration, vs. 8 scalar lookups today.
> >
> >>> and each per-VRF bulk lookup hits a contiguous
> >>> TBL24 instead of scattering across 128 MB-apart VRF regions.
> >> why is a contiguous 128Mb single-VRF TBL24 OK for you, but bigger
> >> contiguous multi-VRF TBL24 is not OK in the context of lookup (here we
> >> are talking about lookup, omitting the problem of memory consumption on
> >> init)?
> > The performance difference may be small, but the flat approach
> > is not faster either -- while costing 64 GB upfront.
> it seems you are implicitly take an assumption of 256 VRFs. Is my
> usecase with a few VRFs have a right to exist?
> >
> >> In both of these cases, memory access behaves the same way within a
> >> single batch of packets during lookup, i.e. the first hit is likely a
> >> cache miss, regardless of whether we are dealing with one or more VRFs,
> >> it will not maintain TBL24 in L3$ in any way in a real dataplane app.
> >>
> >>>> And if the addresses are localized (i.e. most traffic is internal), then
> >>>> having multiple TBL24 won'tmake the situationmuchworse.
> >>>>
> >>> With localized traffic, regrouping by VRF + per-VRF lookup on
> >>> contiguous TBL24s would benefit from cache locality,
> >> why so? There will be no differences within a single batch with a
> >> reasonable size (for example 64), because within the lookup session, no
> >> matter with or without regrouping, temporal cache locality will be the same.
> >>
> >> Let't look at it from a different angle. Is it
> >> worthregroupingipaddressesby/8(i.e. 8 MSBs)withthe
> >> currentimplementationof a singleVRFFIB?
> >>
> >>> while the
> >>> flat multi-VRF table spreads hot entries 128 MB apart. The flat
> >>> approach may actually be worse in that scenario
> >>>
> >>>> I don't have any benchmarks for regrouping, however I have 2 things to
> >>>> consider:
> >>>>
> >>>> 1. lookup is relatively fast (for IPv4 it is about 10 cycles per
> >>>> address, and I don't really want to slow it down)
> >>>>
> >>>> 2. incoming addresses and their corresponding VRFs are not controlled by
> >>>> "us", so this is a random set. Regrouping effectively is sorting. I'm
> >>>> not really happy to have nlogn complexity on a fast path :)
> >>> Without benchmarks, we do not know whether the flat approach is
> >>> actually faster than regroup + per-VRF lookup.
> >> feel free to share benchmark results. The only thing you need to add is
> >> the packets regrouping logic, and then use separate single-VRF FIB
> >> instances.
> > Your series introduces a new API that optimizes multi-VRF lookup.
> > The performance numbers should come with the proposal.
> By the policy we can not share raw performance numbers, and I think this
> is unnecessary, because performance depends on the testing environment
> (content of the routing table, CPU model, etc).
>
> Tests I've done on my board with ipv4 full-view (782940 routes) with 4
> VRFs performing random lookup in all of them was 180% in cost comparing
> to a single VRF with the same RT content.
>
> You can test it in your environment with
> dpdk-test-fib -l 1,2 --no-pci -- -f <path to your routes> -e 4 -l
> 100000000 -V <number of VRFs>
The dpdk-test-fib benchmark is useful for measuring raw lookup
throughput, but it does not capture the full picture. In a real
router stack with rte_graph, the classification by VRF happens
naturally as part of packet processing -- it is not an extra
sorting step. The only way to compare both approaches fairly is
to measure end-to-end forwarding performance in a real datapath.
grout is an open source DPDK-based router built on rte_graph,
designed to exercise and validate DPDK APIs in realistic
conditions. I would be happy to help benchmark both approaches
there.
>
> >
> >>>>> On the memory trade-off and VRF ID mapping: the API uses vrf_id as
> >>>>> a direct index (0 to max_vrfs-1). With 256 VRFs and 8B nexthops,
> >>>>> TBL24 alone costs 32 GB for IPv4 and 32 GB for IPv6 -- 64 GB total
> >>>>> at startup. In grout, VRF IDs are interface IDs that can be any
> >>>>> uint16_t, so we would also need to maintain a mapping between our
> >>>>> VRF IDs and FIB slot indices.
> >>>> of course, this is an application responsibility. In FIB VRFs are in
> >>>> continuous range.
> >>>>> We would need to introduce a max_vrfs
> >>>>> limit, which forces a bad trade-off: either set it low (e.g. 16)
> >>>>> and limit deployments, or set it high (e.g. 256) and pay 64 GB at
> >>>>> startup even with a single VRF. With separate FIB instances per VRF,
> >>>>> we only allocate what we use.
> >>>> Yes, I understand this. In the end, if the user wants to use 256 VRFs,
> >>>> the amount of memory footprint will be at least 64Gb anyway.
> >>> The difference is when the memory is committed.
> >> yes, this's the only difference. It all comes down to the static vs
> >> dynamic memory allocation problem. And each of these approaches is good
> >> for solving a specific task. For the task of creating a new VRF, what is
> >> more preferable - to fail on init or runtime?
> > The main problem is that your series imposes contiguous VRF IDs
> > (0 to max_vrfs-1). How a VRF is represented is a network stack
> > design decision
> exactly - a network stack decision. FIB is not a network stack.
> > -- in Linux it is an ifindex,
> so every interface lives in it's own private VRF?
> > in Cisco a name,
> are you going to pass an array of strings on lookup?
> > in grout an interface ID.
> haven't we decided this is a problematic design(VLANs, L3VPN, etc)?
> > Any application using this API needs
> > a mapping layer on top.
> I think from my rhetoric questions this should be obvious
> >
> > In grout, everything is allocated dynamically: mempools, FIBs,
> > conntrack tables. Pre-allocating everything at init forces
> > hardcoded arbitrary limits and prevents memory reuse between
> > subsystems -- memory reserved for FIB TBL24 cannot be used for
> > conntrack when the VRF has no routes, and vice versa. We prefer
> > to allocate resources only when needed. It is simpler for users
> > and more efficient for memory.
> >
> >>> With separate FIB
> >>> instances per VRF, you allocate 128 MB only when a VRF is actually
> >>> created at runtime. With the flat multi-VRF approach, you pay
> >>> max_vrfs * 128 MB at startup, even if only one VRF is active.
> >>>
> >>> On top of that, the API uses vrf_id as a direct index (0 to
> >>> max_vrfs-1). As Stephen noted, there are multiple ways to model
> >>> VRFs. Depending on the networking stack, VRFs are identified by
> >>> ifindex (Linux l3mdev), by name (Cisco, Juniper), or by some
> >>> other scheme. This means the application must maintain a mapping
> >>> between its own VRF representation and the FIB slot indices, and
> >>> choose max_vrfs upfront. What is the benefit of this flat
> >>> multi-VRF FIB if the application still needs to manage a
> >>> translation layer and pre-commit memory for VRFs that may never
> >>> exist?
> >> This is the control plane task.
> >>>> As a trade-off for a bad trade-off ;) I can suggest to allocate it in
> >>>> chunks. Let's say you are starting with 16 VRFs, and during runtime, if
> >>>> the user wants to increase the number of VRFs above this limit, you can
> >>>> allocate another 16xVRF FIB. Then, of course, you need to split
> >>>> addresses into 2 bursts each for each FIB handle.
> >>> But then we are back to regrouping packets -- just by chunk of
> >>> VRFs instead of by individual VRF. If we have to sort the burst
> >>> anyway, what does the flat multi-VRF table buy us?
> >>>
> >>>>>>> I am not too familiar with DPDK FIB internals, but would it be
> >>>>>>> possible to keep a separate TBL24 per VRF and only share the TBL8
> >>>>>>> pool?
> >>>>>> it is how it is implemented right now with one note - TBL24 are pre
> >>>>>> allocated.
> >>>>>>> Something like pre-allocating an array of max_vrfs TBL24
> >>>>>>> pointers, allocating each TBL24 on demand at VRF add time,
> >>>>>> and you suggesting to allocate TBL24 on demand by adding an extra
> >>>>>> indirection layer. Thiswill leadtolowerperformance,whichIwouldliketo avoid.
> >>>>>>> and
> >>>>>>> having them all point into a shared TBL8 pool. The TBL8 index in
> >>>>>>> TBL24 entries seems to already be global, so would that work without
> >>>>>>> encoding changes?
> >>>>>>>
> >>>>>>> Going further: could the same idea extend to IPv6? The dir24_8 and
> >>>>>>> trie seem to use the same TBL8 block format (256 entries, same
> >>>>>>> (nh << 1) | ext_bit encoding, same size). Would unifying the TBL8
> >>>>>>> allocator allow a single pool shared across IPv4, IPv6, and all
> >>>>>>> VRFs? That could be a bigger win for /32-heavy and /128-heavy tables
> >>>>>>> and maybe a good first step before multi-VRF.
> >>>>>> So, you are suggesting merging IPv4 and IPv6 into a single unified FIB?
> >>>>>> I'm not sure how this can be a bigger win, could you please elaborate
> >>>>>> more on this?
> >>>>> On the IPv4/IPv6 TBL8 pool: I was not suggesting merging FIBs, just
> >>>>> sharing the TBL8 block allocator between separate FIB instances.
> >>>>> This is possible since dir24_8 and trie use the same TBL8 block
> >>>>> format (256 entries, same encoding, same size).
> >>>>>
> >>>>> Would it be possible to pass a shared TBL8 pool at rte_fib_create()
> >>>>> time? Each FIB keeps its own TBL24 and RIB, but TBL8 is shared
> >>>>> across all FIBs and potentially across IPv4/IPv6. Users would no
> >>>>> longer have to guess num_tbl8 per FIB.
> >>>> Yes, this is possible. However, this will significantly complicate the
> >>>> work with the library, solving a not so big problem.
> >>> Your series already shares TBL8 across all VRFs within a single
> >>> FIB -- that part is useful, and it does not require the flat
> >>> multi-VRF TBL24.
> >>>
> >>> In grout, routes arrive from FRR (BGP, OSPF, etc.) at runtime.
> >>> We cannot predict TBL8 usage per VRF in advance
> >> and you don't need it (knowing per-VRF consumption) now. If I understood
> >> your request here properly, do you want to share TBL8 between ipv4 and
> >> ipv6 FIBs? I don't think this is a good idea at all. At least because
> >> this is a good idea to split them in case if one AF consumes (because of
> >> attack/bogus CP) all TBL8, another AF remains intact.
> > If TBL8 isolation per AF is meant as a protection against route
> > floods, then the same argument applies between VRFs: your series
> > shares TBL8 across all VRFs within a single FIB, so a bogus
> > control plane in one VRF exhausts TBL8 for all other VRFs.
> >
> > But more fundamentally, this is not how route flood protection
> > works. It is handled in the control plane: the routing daemon
> > limits the number of prefixes accepted per BGP session
> > (max-prefix) and selects which routes are installed via prefix
> > filters -- before those routes ever reach the forwarding table.
> >
> > The Linux kernel is a good reference here. IPv6 used to enforce
> > a max_size limit on FIB + cache entries (net.ipv6.route.max_size,
> > defaulting to 4096). It caused real production issues and was
> > removed in kernel 6.3. IPv4 never had a FIB route limit. There
> > is no per-VRF route limit either. The kernel relies entirely on
> > the control plane for route flood protection.
>
> FIB is not the Linux kernel, as well as not a network stack. We can not
> rely on a control plane protection, since control plane is a 3rd party
> software.
>
> Also, I think allocating a very algorithm-specific entity such as pool
> of TBL8 prior to calling rte_fib_create and passing a pointer on it
> could be confusing for many users and bloating API.
>
> FIB supports pluggable lookup algorithms, you can write your own and
> specify a pointer to the tbl8_pool in an algorithm-specific
> configuration defined for your algorithm, where you may also create a
> dynamic table of TBL24 pointers per each VRF. If you need any help with
> this task - I would be happy to help.
I have sent this RFC:
https://mails.dpdk.org/archives/dev/2026-March/335512.html
Thanks in advance for your help.
>
> >
> >>> -- it depends on
> >>> prefix length distribution which varies per VRF and changes over
> >>> time. No production LPM (Linux kernel, JunOS, IOS) asks the
> >>> operator to size these structures per routing table upfront.
> >> - they are using different lpm algorithms
> >> - you use these facilities, developers properly tuned them. FIB is a low
> >> level library, it cannot be used without any knowledge, it will not
> >> solve all the problems with a single red button "make it work, don't do
> >> any bugs"
> >> P.S. how do you know how JunOS /IOS implements their LPMs? ;)
> >>
> > I do not need to know their LPM implementation -- I only need
> > to know how they are configured. No production router requires
> > the operator to size internal LPM structures.
> >
> > We can impose a maximum number of IPv4/IPv6 routes on the user
> > -- even though the kernel does not need this either. But TBL8
> > is a different problem: the application cannot predict TBL8
> > consumption because it depends on prefix length distribution,
> > which varies per VRF and changes over time with dynamic routing.
> > Today there is no API to query TBL8 usage, and no API to resize
> > a FIB without destroying it.
> >
> > This is exactly why a shared TBL8 pool across VRFs is useful:
> > VRFs with few long prefixes naturally leave room for VRFs that
> > need more.
>
> but this is already implemented. I don't know why you repeatedly
> concerning about this. We are aligned about this and this feature is
> already there - in the patch
> On the other hand what we disagreed on is the sharing not only across
> VRFs, but also across address families. If you don't understand the
> amount of TBL8 per AF, how would you magically understand the number of
> TBL8s for a merged pool?
>
> > This is the valuable part of your series. But it
> > does not require a flat multi-VRF TBL24 -- separate per-VRF
> > TBL24s sharing a common TBL8 pool would give the same benefit
> > without the 64 GB upfront cost.
>
> andthat'sa completelydifferentproblem.Please,let's separatethe
> problemsandnotmixthemup.
>
> I understand your concern about memory consumption. Ihavesomeideason
> howto solvethisproblem as a parallel to proposed solution.
>
> >
> >>> Today we do not even have TBL8 usage stats (Robin's series
> >>> addresses that)
> >> I will tryto findtimetoreviewthispatchinthe nearfuture
> > Thanks, Robin's TBL8 stats series would help users understand
> > their TBL8 consumption -- a more practical improvement for
> > current users.
> >
> > Regards,
> > Maxime
>
> --
> Regards,
> Vladimir
>
--
Regards,
Maxime
^ permalink raw reply [flat|nested] 45+ messages in thread
* [RFC PATCH v2 0/7] VRF support in FIB library
2026-03-22 15:42 [RFC PATCH 0/4] VRF support in FIB library Vladimir Medvedkin
` (7 preceding siblings ...)
2026-03-23 19:05 ` Stephen Hemminger
@ 2026-10-14 18:39 ` Vladimir Medvedkin
2026-10-14 18:39 ` [RFC PATCH v2 1/7] fib: fix return value of a nexthop update Vladimir Medvedkin
` (7 more replies)
8 siblings, 8 replies; 45+ messages in thread
From: Vladimir Medvedkin @ 2026-10-14 18:39 UTC (permalink / raw)
To: dev; +Cc: rjarry, mb, maxime, konstantin.ananyev
This series adds multi-VRF support to the IPv4 FIB, allowing a single FIB
instance to host multiple isolated routing tables.
Today a FIB instance is a single routing table, so an application serving
several VRFs needs one FIB per VRF. In a burst oriented datapath packets of
one burst can belong to different VRFs, and the application either does
per-packet lookups in different FIBs or regroups the burst by VRF before
the lookup. Both are expensive. With this series all the VRFs live in one
FIB, the bulk lookup takes a VRF ID per address, and the existing API
operates on VRF 0.
v1 replicated the 16M entry tbl24 per VRF. That kept the lookup on par with
the single VRF case, but memory grew linearly with the number of VRFs,
which was the main concern raised on v1. v2 resolves the issue by adding
of two dataplane algorithms with different memory/lookup trade-offs.
DIR8-16-8 is a new three level variant of DIR-24-8 with 8, 16 and 8 bit
strides. Only the 256 entry root table (2 KB) is replicated per VRF; the
tbl16 and tbl8 groups come from pools shared by all the VRFs, so memory
follows the routes actually installed rather than the number of VRFs. The
cost is an extra memory access for prefixes longer than /8.
DIR-24-8 gets multi-VRF support by resolving the VRF after the longest
prefix match, along the lines Konstantin suggested on v1. Instead of a
vector of next hops per prefix, a single tbl24_8 shared by all the VRFs
returns an internal prefix ID, and a hash table keyed by {VRF, prefix ID}
returns the next hop, a miss meaning the VRF default next hop. The aim is
to resolve the VRF with a single memory access, so buckets hold 8 records
and there is no probing beyond the bucket.
Such a table only works if the control path can keep buckets from
overflowing while the hash stays cheap on the datapath. Since the prefix
ID is internal, the library can pick it so that the records of a prefix
land in buckets with free room. Collisions are chosen deliberately, so they
are not a security concern, but finding them has to be cheap. Toeplitz was
the first candidate: its structure is well studied and the thash library
already knows how to find collisions, but it is only fast with GFNI. CRC
has the same linear properties, is trivially inverted and is hardware
accelerated on x86 and Arm, so CRC32C is used. Linearity also places the
buckets of one prefix in different VRFs at fixed XOR offsets from each
other, so a candidate prefix ID is checked for all the VRFs referencing
the prefix at once, and these records never share a bucket. If a bucket
fills up anyway, a prefix already in it is renumbered to make room.
The price of the shared table is paid on the control path: a route in one
VRF needs a record for every more specific prefix installed below it by
other VRFs. Default routes are excluded, they map onto the per-VRF default
next hop.
The two algorithms target different deployments. DIR8-16-8 fits a limited
number of VRFs (tens to hundreds) carrying large tables, up to a BGP full
view each. The hash based DIR-24-8 fits one large VRF alongside hundreds
or thousands of small ones.
test-fib gets a VRF mode, including a profile with a full view in VRF 0 and
smaller VRFs made of random subsets of it. The first three patches are fixes
found along the way and are independent of the rest.
Not done yet:
- IPv6. The VRF bookkeeping of the hash based scheme is address family
agnostic and is meant to be reused by TRIE.
- RCU support for multi-VRF DIR-24-8.
Vladimir Medvedkin (7):
fib: fix return value of a nexthop update
app/test-fib: fix lookup of the last burst
app/test-fib: fix size of the tables for large counts
fib: add DIR8-16-8 algorithm
fib: add multi-VRF support
fib: split DIR24-8 table update from the FIB wrapper
fib: add hash based VRF support for DIR24-8
app/test-fib/main.c | 673 ++++++++++---
app/test/test_fib.c | 1001 ++++++++++++++++++-
app/test/test_fib_perf.c | 440 +++------
app/test/test_fib_routes.h | 282 ++++++
doc/guides/prog_guide/fib_lib.rst | 66 ++
doc/guides/rel_notes/release_26_11.rst | 21 +
lib/fib/dir24_8.c | 34 +-
lib/fib/dir24_8.h | 43 +-
lib/fib/dir24_8_avx512.c | 18 +-
lib/fib/dir24_8_avx512.h | 32 +-
lib/fib/dir24_8_rvv.c | 3 +-
lib/fib/dir24_8_rvv.h | 16 +-
lib/fib/dir24_8_vrf.c | 290 ++++++
lib/fib/dir24_8_vrf.h | 60 ++
lib/fib/dir8_16_8.c | 665 +++++++++++++
lib/fib/dir8_16_8.h | 192 ++++
lib/fib/dir8_16_8_avx512.c | 183 ++++
lib/fib/dir8_16_8_avx512.h | 24 +
lib/fib/fib_vrf.c | 1256 ++++++++++++++++++++++++
lib/fib/fib_vrf.h | 277 ++++++
lib/fib/fib_vrf_avx512.c | 142 +++
lib/fib/meson.build | 22 +-
lib/fib/rte_fib.c | 249 ++++-
lib/fib/rte_fib.h | 127 ++-
24 files changed, 5593 insertions(+), 523 deletions(-)
create mode 100644 app/test/test_fib_routes.h
create mode 100644 lib/fib/dir24_8_vrf.c
create mode 100644 lib/fib/dir24_8_vrf.h
create mode 100644 lib/fib/dir8_16_8.c
create mode 100644 lib/fib/dir8_16_8.h
create mode 100644 lib/fib/dir8_16_8_avx512.c
create mode 100644 lib/fib/dir8_16_8_avx512.h
create mode 100644 lib/fib/fib_vrf.c
create mode 100644 lib/fib/fib_vrf.h
create mode 100644 lib/fib/fib_vrf_avx512.c
--
2.43.0
^ permalink raw reply [flat|nested] 45+ messages in thread
* [RFC PATCH v2 1/7] fib: fix return value of a nexthop update
2026-10-14 18:39 ` [RFC PATCH v2 0/7] " Vladimir Medvedkin
@ 2026-10-14 18:39 ` Vladimir Medvedkin
2026-10-14 18:39 ` [RFC PATCH v2 2/7] app/test-fib: fix lookup of the last burst Vladimir Medvedkin
` (6 subsequent siblings)
7 siblings, 0 replies; 45+ messages in thread
From: Vladimir Medvedkin @ 2026-10-14 18:39 UTC (permalink / raw)
To: dev; +Cc: rjarry, mb, maxime, konstantin.ananyev, stable
When the nexthop of an existing route is changed, the error returned by
the dataplane update was discarded and success was reported even if the
DIR24_8 table could not be updated.
Return the actual status, as the IPv6 TRIE algorithm does.
Fixes: 7dc7868b200d ("fib: add DIR24-8 dataplane algorithm")
Cc: stable@dpdk.org
Signed-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
---
lib/fib/dir24_8.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/fib/dir24_8.c b/lib/fib/dir24_8.c
index 489d2ef427..bb7975eca1 100644
--- a/lib/fib/dir24_8.c
+++ b/lib/fib/dir24_8.c
@@ -498,7 +498,7 @@ dir24_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t depth,
ret = modify_fib(dp, rib, ip, depth, next_hop);
if (ret == 0)
rte_rib_set_nh(node, next_hop);
- return 0;
+ return ret;
}
if (depth > 24) {
tmp = rte_rib_get_nxt(rib, ip, 24, NULL,
--
2.43.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [RFC PATCH v2 2/7] app/test-fib: fix lookup of the last burst
2026-10-14 18:39 ` [RFC PATCH v2 0/7] " Vladimir Medvedkin
2026-10-14 18:39 ` [RFC PATCH v2 1/7] fib: fix return value of a nexthop update Vladimir Medvedkin
@ 2026-10-14 18:39 ` Vladimir Medvedkin
2026-10-14 18:39 ` [RFC PATCH v2 3/7] app/test-fib: fix size of the tables for large counts Vladimir Medvedkin
` (5 subsequent siblings)
7 siblings, 0 replies; 45+ messages in thread
From: Vladimir Medvedkin @ 2026-10-14 18:39 UTC (permalink / raw)
To: dev; +Cc: rjarry, mb, maxime, konstantin.ananyev, stable
The lookup loops process the lookup table in bursts of 64 addresses and
always pass a full burst, reading and comparing past the end of the table
when the number of addresses is not a multiple of the burst size.
Round the number of addresses down to a multiple of the burst size once
they are counted, so that the timed loops keep passing full bursts, and
read only that many addresses from the lookup file.
Fixes: 103809d032cd ("app/test-fib: add test application for FIB")
Cc: stable@dpdk.org
Signed-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
---
app/test-fib/main.c | 16 ++++++++++++----
1 file changed, 12 insertions(+), 4 deletions(-)
diff --git a/app/test-fib/main.c b/app/test-fib/main.c
index dd1a6d7297..d85d94fdd1 100644
--- a/app/test-fib/main.c
+++ b/app/test-fib/main.c
@@ -523,19 +523,21 @@ parse_rt_6(FILE *f)
static int
parse_lookup(FILE *f, int af)
{
- int ret, i = 0;
+ int ret;
+ uint32_t i;
uint8_t *tbl = (uint8_t *)config.lookup_tbl;
int step = (af == AF_INET) ? 4 : 16;
char *s;
- while (fgets(line, sizeof(line), f) != NULL) {
+ /* Only the addresses of the full bursts are read. */
+ for (i = 0; i < config.nb_lookup_ips &&
+ fgets(line, sizeof(line), f) != NULL; i++) {
s = strtok(line, " \t\n");
if (s == NULL)
return -EINVAL;
- ret = inet_pton(af, s, &tbl[i]);
+ ret = inet_pton(af, s, &tbl[(size_t)i * step]);
if (ret != 1)
return -EINVAL;
- i += step;
}
return 0;
}
@@ -1251,6 +1253,12 @@ main(int argc, char **argv)
rewind(fl);
}
+ /* Lookups are done in full bursts, the leftovers are dropped. */
+ config.nb_lookup_ips = RTE_ALIGN_FLOOR(config.nb_lookup_ips, BURST_SZ);
+ if (config.nb_lookup_ips == 0)
+ rte_exit(-EINVAL, "At least %u ip's for lookup are needed\n",
+ BURST_SZ);
+
/* Alloc routes table*/
config.rt = rte_malloc(NULL, rt_ent_sz * config.nb_routes, 0);
if (config.rt == NULL)
--
2.43.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [RFC PATCH v2 3/7] app/test-fib: fix size of the tables for large counts
2026-10-14 18:39 ` [RFC PATCH v2 0/7] " Vladimir Medvedkin
2026-10-14 18:39 ` [RFC PATCH v2 1/7] fib: fix return value of a nexthop update Vladimir Medvedkin
2026-10-14 18:39 ` [RFC PATCH v2 2/7] app/test-fib: fix lookup of the last burst Vladimir Medvedkin
@ 2026-10-14 18:39 ` Vladimir Medvedkin
2026-10-14 22:52 ` Stephen Hemminger
2026-10-14 18:39 ` [RFC PATCH v2 4/7] fib: add DIR8-16-8 algorithm Vladimir Medvedkin
` (4 subsequent siblings)
7 siblings, 1 reply; 45+ messages in thread
From: Vladimir Medvedkin @ 2026-10-14 18:39 UTC (permalink / raw)
To: dev; +Cc: rjarry, mb, maxime, konstantin.ananyev, stable
The sizes of the routes and lookup tables are computed in 32 bits, so
more than 268M IPv6 addresses for lookup overflow the size passed to
rte_malloc() and the tables are written past their end.
Widen the multiplications to size_t.
Fixes: 103809d032cd ("app/test-fib: add test application for FIB")
Cc: stable@dpdk.org
Signed-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
---
app/test-fib/main.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/app/test-fib/main.c b/app/test-fib/main.c
index d85d94fdd1..0bc8dd2aa4 100644
--- a/app/test-fib/main.c
+++ b/app/test-fib/main.c
@@ -1260,12 +1260,12 @@ main(int argc, char **argv)
BURST_SZ);
/* Alloc routes table*/
- config.rt = rte_malloc(NULL, rt_ent_sz * config.nb_routes, 0);
+ config.rt = rte_malloc(NULL, (size_t)rt_ent_sz * config.nb_routes, 0);
if (config.rt == NULL)
rte_exit(-ENOMEM, "Can not alloc rt\n");
/* Alloc table with ip's for lookup*/
- config.lookup_tbl = rte_malloc(NULL, lookup_ent_sz *
+ config.lookup_tbl = rte_malloc(NULL, (size_t)lookup_ent_sz *
config.nb_lookup_ips, 0);
if (config.lookup_tbl == NULL)
rte_exit(-ENOMEM, "Can not alloc lookup table\n");
--
2.43.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [RFC PATCH v2 4/7] fib: add DIR8-16-8 algorithm
2026-10-14 18:39 ` [RFC PATCH v2 0/7] " Vladimir Medvedkin
` (2 preceding siblings ...)
2026-10-14 18:39 ` [RFC PATCH v2 3/7] app/test-fib: fix size of the tables for large counts Vladimir Medvedkin
@ 2026-10-14 18:39 ` Vladimir Medvedkin
2026-10-14 18:39 ` [RFC PATCH v2 5/7] fib: add multi-VRF support Vladimir Medvedkin
` (3 subsequent siblings)
7 siblings, 0 replies; 45+ messages in thread
From: Vladimir Medvedkin @ 2026-10-14 18:39 UTC (permalink / raw)
To: dev; +Cc: rjarry, mb, maxime, konstantin.ananyev
Add an IPv4 dataplane algorithm splitting the address in three strides
of 8, 16 and 8 bits: a 256 entry root table resolves the first stride,
then 65536 entry groups (tbl16) and 256 entry groups (tbl8) allocated
from pools sized on FIB creation resolve the next ones. A lookup needs
one to three memory accesses.
Signed-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
---
app/test-fib/main.c | 86 +++-
app/test/test_fib.c | 101 +++-
app/test/test_fib_perf.c | 65 ++-
doc/guides/prog_guide/fib_lib.rst | 26 ++
doc/guides/rel_notes/release_26_11.rst | 5 +
lib/fib/dir8_16_8.c | 614 +++++++++++++++++++++++++
lib/fib/dir8_16_8.h | 143 ++++++
lib/fib/dir8_16_8_avx512.c | 142 ++++++
lib/fib/dir8_16_8_avx512.h | 16 +
lib/fib/meson.build | 4 +-
lib/fib/rte_fib.c | 24 +-
lib/fib/rte_fib.h | 17 +-
12 files changed, 1204 insertions(+), 39 deletions(-)
create mode 100644 lib/fib/dir8_16_8.c
create mode 100644 lib/fib/dir8_16_8.h
create mode 100644 lib/fib/dir8_16_8_avx512.c
create mode 100644 lib/fib/dir8_16_8_avx512.h
diff --git a/app/test-fib/main.c b/app/test-fib/main.c
index 0bc8dd2aa4..be1ba20277 100644
--- a/app/test-fib/main.c
+++ b/app/test-fib/main.c
@@ -39,6 +39,8 @@
#define DEF_LOOKUP_IPS_NUM 0x100000
#define BURST_SZ 64
#define DEFAULT_LPM_TBL8 100000U
+/* One second level (16 bit stride) group per /8 covers the whole space. */
+#define DIR8_16_8_MAX_TBL16 256U
#define CMP_FLAG (1 << 0)
#define CMP_ALL_FLAG (1 << 1)
@@ -46,9 +48,12 @@
#define FIB_RIB_TYPE (1 << 3)
#define FIB_V4_DIR_TYPE (1 << 4)
#define FIB_V6_TRIE_TYPE (1 << 4)
-#define FIB_TYPE_MASK (FIB_RIB_TYPE|FIB_V4_DIR_TYPE|FIB_V6_TRIE_TYPE)
+#define FIB_V4_DIR_8_16_8_TYPE (1 << 5)
+#define FIB_TYPE_MASK (FIB_RIB_TYPE|FIB_V4_DIR_TYPE|FIB_V6_TRIE_TYPE| \
+ FIB_V4_DIR_8_16_8_TYPE)
#define SHUFFLE_FLAG (1 << 7)
#define DRY_RUN_FLAG (1 << 8)
+#define ENT_SZ_FLAG (1 << 9)
static char *distrib_string;
static char line[LINE_MAX];
@@ -147,6 +152,8 @@ get_fib_type(void)
} else {
if ((config.flags & FIB_TYPE_MASK) == FIB_V4_DIR_TYPE)
return RTE_FIB_DIR24_8;
+ if ((config.flags & FIB_TYPE_MASK) == FIB_V4_DIR_8_16_8_TYPE)
+ return RTE_FIB_DIR8_16_8;
if ((config.flags & FIB_TYPE_MASK) == FIB_RIB_TYPE)
return RTE_FIB_DUMMY;
}
@@ -619,18 +626,20 @@ print_usage(void)
"[-b <fib algorithm>]\n\tavailable options for ipv4\n"
"\t\trib - RIB based FIB\n"
"\t\tdir - DIR24_8 based FIB\n"
+ "\t\tdir8_16_8 - DIR8_16_8 based FIB (8 byte next hop)\n"
"\tavailable options for ipv6:\n"
"\t\trib - RIB based FIB\n"
"\t\ttrie - TRIE based FIB\n"
"defaults are: dir for ipv4 and trie for ipv6\n"
"[-e <entry size (valid only for dir and trie fib types): "
"1/2/4/8 (default 4)>]\n"
- "[-g <number of tbl8's for dir24_8 or trie FIBs>]\n"
+ "[-g <number of tbl8's for dir24_8, dir8_16_8 or trie FIBs>]\n"
"[-w <path to the file to dump routing table>]\n"
"[-u <path to the file to dump ip's for lookup>]\n"
"[-v <type of lookup function:"
"\ts1, s2, s3 (3 types of scalar), v (vector) -"
" for DIR24_8 based FIB\n"
+ "\ts, v - for DIR8_16_8 based FIB\n"
"\ts, v - for TRIE based ipv6 FIB>]\n",
config.prgname);
}
@@ -665,6 +674,22 @@ check_config(void)
printf("-e 1 is valid only for ipv4\n");
return -1;
}
+
+ if ((config.flags & FIB_V4_DIR_8_16_8_TYPE) &&
+ (config.flags & IPV6_FLAG)) {
+ printf("dir8_16_8 fib type is only valid for ipv4\n");
+ return -1;
+ }
+ /* The dir8_16_8 fib type only supports 8 byte next hops. */
+ if ((config.flags & FIB_TYPE_MASK) == FIB_V4_DIR_8_16_8_TYPE) {
+ if ((config.flags & ENT_SZ_FLAG) && config.ent_sz != 8) {
+ printf("-e %u is not valid for the dir8_16_8 fib type, "
+ "only 8 byte next hops are supported\n",
+ config.ent_sz);
+ return -1;
+ }
+ config.ent_sz = 8;
+ }
return 0;
}
@@ -744,6 +769,9 @@ parse_opts(int argc, char **argv)
} else if (strcmp(optarg, "dir") == 0) {
config.flags &= ~FIB_TYPE_MASK;
config.flags |= FIB_V4_DIR_TYPE;
+ } else if (strcmp(optarg, "dir8_16_8") == 0) {
+ config.flags &= ~FIB_TYPE_MASK;
+ config.flags |= FIB_V4_DIR_8_16_8_TYPE;
} else if (strcmp(optarg, "trie") == 0) {
config.flags &= ~FIB_TYPE_MASK;
config.flags |= FIB_V6_TRIE_TYPE;
@@ -757,6 +785,7 @@ parse_opts(int argc, char **argv)
print_usage();
rte_exit(-EINVAL, "Invalid option -e\n");
}
+ config.flags |= ENT_SZ_FLAG;
break;
case 'g':
errno = 0;
@@ -817,6 +846,35 @@ print_depth_err(void)
"-d 0:0 option or remove /0 prefix from routes file\n");
}
+static int
+select_lookup_fn_v4(struct rte_fib *fib, enum rte_fib_type type)
+{
+ enum rte_fib_lookup_type lookup_type;
+
+ if (type == RTE_FIB_DIR8_16_8) {
+ if (config.lookup_fn == 1)
+ lookup_type = RTE_FIB_LOOKUP_DIR8_16_8_SCALAR;
+ else if (config.lookup_fn == 2)
+ lookup_type = RTE_FIB_LOOKUP_DIR8_16_8_VECTOR_AVX512;
+ else
+ return -EINVAL;
+ } else if (type == RTE_FIB_DIR24_8) {
+ if (config.lookup_fn == 1)
+ lookup_type = RTE_FIB_LOOKUP_DIR24_8_SCALAR_MACRO;
+ else if (config.lookup_fn == 2)
+ lookup_type = RTE_FIB_LOOKUP_DIR24_8_VECTOR_AVX512;
+ else if (config.lookup_fn == 3)
+ lookup_type = RTE_FIB_LOOKUP_DIR24_8_SCALAR_INLINE;
+ else if (config.lookup_fn == 4)
+ lookup_type = RTE_FIB_LOOKUP_DIR24_8_SCALAR_UNI;
+ else
+ return -EINVAL;
+ } else {
+ return -EINVAL;
+ }
+ return rte_fib_select_lookup(fib, lookup_type);
+}
+
static int
run_v4(void)
{
@@ -853,6 +911,15 @@ run_v4(void)
conf.dir24_8.nh_sz = rte_ctz32(config.ent_sz);
conf.dir24_8.num_tbl8 = RTE_MIN(config.tbl8,
get_max_nh(conf.dir24_8.nh_sz));
+ } else if (conf.type == RTE_FIB_DIR8_16_8) {
+ /*
+ * DIR8_16_8 only supports 8 byte next hops. The first stride
+ * (8 bits) needs at most one second level group per /8, so 256
+ * groups always suffice. The third level (last 8 bits) groups
+ * are sized like the DIR24_8 tbl8s.
+ */
+ conf.dir8_16_8.num_tbl16 = DIR8_16_8_MAX_TBL16;
+ conf.dir8_16_8.num_tbl8 = config.tbl8;
}
fib = rte_fib_create("test", -1, &conf);
@@ -862,20 +929,7 @@ run_v4(void)
}
if (config.lookup_fn != 0) {
- if (config.lookup_fn == 1)
- ret = rte_fib_select_lookup(fib,
- RTE_FIB_LOOKUP_DIR24_8_SCALAR_MACRO);
- else if (config.lookup_fn == 2)
- ret = rte_fib_select_lookup(fib,
- RTE_FIB_LOOKUP_DIR24_8_VECTOR_AVX512);
- else if (config.lookup_fn == 3)
- ret = rte_fib_select_lookup(fib,
- RTE_FIB_LOOKUP_DIR24_8_SCALAR_INLINE);
- else if (config.lookup_fn == 4)
- ret = rte_fib_select_lookup(fib,
- RTE_FIB_LOOKUP_DIR24_8_SCALAR_UNI);
- else
- ret = -EINVAL;
+ ret = select_lookup_fn_v4(fib, conf.type);
if (ret != 0) {
printf("Can not init lookup function\n");
return ret;
diff --git a/app/test/test_fib.c b/app/test/test_fib.c
index bd73399d56..bb8021c818 100644
--- a/app/test/test_fib.c
+++ b/app/test/test_fib.c
@@ -6,9 +6,12 @@
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
+#include <string.h>
+#include <inttypes.h>
#include <rte_ip.h>
#include <rte_log.h>
+#include <rte_byteorder.h>
#include <rte_fib.h>
#include <rte_malloc.h>
@@ -65,7 +68,7 @@ test_create_invalid(void)
"Call succeeded with invalid parameters\n");
config.max_routes = MAX_ROUTES;
- config.type = RTE_FIB_DIR24_8 + 1;
+ config.type = RTE_FIB_DIR8_16_8 + 1;
fib = rte_fib_create(__func__, SOCKET_ID_ANY, &config);
RTE_TEST_ASSERT(fib == NULL,
"Call succeeded with invalid parameters\n");
@@ -84,6 +87,19 @@ test_create_invalid(void)
RTE_TEST_ASSERT(fib == NULL,
"Call succeeded with invalid parameters\n");
+ config.type = RTE_FIB_DIR8_16_8;
+ config.dir8_16_8.num_tbl16 = MAX_TBL8;
+ config.dir8_16_8.num_tbl8 = 0;
+ fib = rte_fib_create(__func__, SOCKET_ID_ANY, &config);
+ RTE_TEST_ASSERT(fib == NULL,
+ "Call succeeded with invalid parameters\n");
+
+ config.dir8_16_8.num_tbl16 = 0;
+ config.dir8_16_8.num_tbl8 = MAX_TBL8;
+ fib = rte_fib_create(__func__, SOCKET_ID_ANY, &config);
+ RTE_TEST_ASSERT(fib == NULL,
+ "Call succeeded with invalid parameters\n");
+
return TEST_SUCCESS;
}
@@ -376,6 +392,23 @@ test_lookup(void)
"Check_fib fails for DIR24_8_8B type\n");
rte_fib_free(fib);
+ config.type = RTE_FIB_DIR8_16_8;
+ config.dir8_16_8.num_tbl16 = 16;
+ config.dir8_16_8.num_tbl8 = MAX_TBL8;
+ fib = rte_fib_create(__func__, SOCKET_ID_ANY, &config);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create FIB\n");
+ ret = check_fib(fib);
+ RTE_TEST_ASSERT(ret == TEST_SUCCESS,
+ "Check_fib fails for DIR8_16_8 type\n");
+ /* The vector lookup is only available with a 512 bit SIMD width. */
+ if (rte_fib_select_lookup(fib,
+ RTE_FIB_LOOKUP_DIR8_16_8_VECTOR_AVX512) == 0) {
+ ret = check_fib(fib);
+ RTE_TEST_ASSERT(ret == TEST_SUCCESS,
+ "Check_fib fails for DIR8_16_8 type, AVX512 lookup\n");
+ }
+ rte_fib_free(fib);
+
return TEST_SUCCESS;
}
@@ -588,6 +621,71 @@ test_fib_rcu_sync_rw(void)
return status == 0 ? TEST_SUCCESS : TEST_FAILED;
}
+#define TEST_NUM_TBL16 16
+#define TEST_NUM_TBL8 64
+#define DEF_NH 77
+
+/*
+ * Network order lookups with every lookup implementation of DIR8_16_8, with a
+ * burst of two vector groups and a leftover.
+ */
+static int32_t
+test_lookup_network_order(void)
+{
+ const enum rte_fib_lookup_type types[] = {
+ RTE_FIB_LOOKUP_DIR8_16_8_SCALAR,
+ RTE_FIB_LOOKUP_DIR8_16_8_VECTOR_AVX512,
+ };
+ const uint32_t probes[] = {
+ RTE_IPV4(10, 2, 3, 4), /* /8 */
+ RTE_IPV4(10, 1, 1, 5), /* /24 */
+ RTE_IPV4(10, 1, 1, 7), /* /32 */
+ RTE_IPV4(8, 8, 8, 8), /* miss */
+ };
+ const uint64_t probe_nhs[] = { 100, 300, 400, DEF_NH };
+ struct rte_fib_conf cfg = { 0 };
+ struct rte_fib *fib;
+ uint32_t ips[17];
+ uint64_t nhs[17];
+ unsigned int i, t;
+
+ cfg.type = RTE_FIB_DIR8_16_8;
+ cfg.max_routes = 100;
+ cfg.default_nh = DEF_NH;
+ cfg.flags = RTE_FIB_F_LOOKUP_NETWORK_ORDER;
+ cfg.dir8_16_8.num_tbl16 = TEST_NUM_TBL16;
+ cfg.dir8_16_8.num_tbl8 = TEST_NUM_TBL8;
+
+ fib = rte_fib_create("fib_be", SOCKET_ID_ANY, &cfg);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create FIB\n");
+
+ RTE_TEST_ASSERT(rte_fib_add(fib, RTE_IPV4(10, 0, 0, 0), 8, 100) == 0,
+ "add /8\n");
+ RTE_TEST_ASSERT(rte_fib_add(fib, RTE_IPV4(10, 1, 1, 0), 24, 300) == 0,
+ "add /24\n");
+ RTE_TEST_ASSERT(rte_fib_add(fib, RTE_IPV4(10, 1, 1, 7), 32, 400) == 0,
+ "add /32\n");
+ for (i = 0; i < RTE_DIM(ips); i++)
+ ips[i] = rte_cpu_to_be_32(probes[i % RTE_DIM(probes)]);
+
+ for (t = 0; t < RTE_DIM(types); t++) {
+ /* The vector lookup needs a 512 bit SIMD width. */
+ if (rte_fib_select_lookup(fib, types[t]) != 0)
+ continue;
+
+ memset(nhs, 0, sizeof(nhs));
+ RTE_TEST_ASSERT(rte_fib_lookup_bulk(fib, ips, nhs,
+ RTE_DIM(ips)) == 0, "lookup failed\n");
+ for (i = 0; i < RTE_DIM(ips); i++)
+ RTE_TEST_ASSERT(nhs[i] == probe_nhs[i % RTE_DIM(probes)],
+ "lookup type %u: ip %u got %"PRIu64"\n",
+ types[t], i, nhs[i]);
+ }
+ rte_fib_free(fib);
+
+ return TEST_SUCCESS;
+}
+
static struct unit_test_suite fib_fast_tests = {
.suite_name = "fib autotest",
.setup = NULL,
@@ -598,6 +696,7 @@ static struct unit_test_suite fib_fast_tests = {
TEST_CASE(test_add_del_invalid),
TEST_CASE(test_get_invalid),
TEST_CASE(test_lookup),
+ TEST_CASE(test_lookup_network_order),
TEST_CASE(test_invalid_rcu),
TEST_CASE(test_fib_rcu_sync_rw),
TEST_CASES_END()
diff --git a/app/test/test_fib_perf.c b/app/test/test_fib_perf.c
index 1b2ad04854..0967644316 100644
--- a/app/test/test_fib_perf.c
+++ b/app/test/test_fib_perf.c
@@ -7,6 +7,7 @@
#include <stdint.h>
#include <stdlib.h>
#include <math.h>
+#include <inttypes.h>
#include <rte_cycles.h>
#include <rte_random.h>
@@ -317,31 +318,16 @@ print_route_distribution(const struct route_rule *table, uint32_t n)
}
static int
-test_fib_perf(void)
+measure_fib_perf(const char *name, struct rte_fib_conf *config)
{
struct rte_fib *fib = NULL;
- struct rte_fib_conf config = { 0 };
-
- config.max_routes = 2000000;
- config.rib_ext_sz = 0;
- config.type = RTE_FIB_DIR24_8;
- config.default_nh = 0;
- config.dir24_8.nh_sz = RTE_FIB_DIR24_8_4B;
- config.dir24_8.num_tbl8 = 65535;
uint64_t begin, total_time;
unsigned int i, j;
uint32_t next_hop_add = 0xAA;
int status = 0;
int64_t count = 0;
- generate_large_route_rule_table();
-
- printf("No. routes = %u\n", (unsigned int) NUM_ROUTE_ENTRIES);
-
- print_route_distribution(large_route_table,
- (uint32_t) NUM_ROUTE_ENTRIES);
-
- fib = rte_fib_create(__func__, SOCKET_ID_ANY, &config);
+ fib = rte_fib_create(name, SOCKET_ID_ANY, config);
TEST_FIB_ASSERT(fib != NULL);
/* Measure add. */
@@ -407,4 +393,49 @@ test_fib_perf(void)
return 0;
}
+static int
+test_fib_perf(void)
+{
+ struct rte_fib_conf config = { 0 };
+
+ config.max_routes = 2000000;
+ config.rib_ext_sz = 0;
+ config.type = RTE_FIB_DIR24_8;
+ config.default_nh = 0;
+ config.dir24_8.nh_sz = RTE_FIB_DIR24_8_4B;
+ config.dir24_8.num_tbl8 = 65535;
+
+ generate_large_route_rule_table();
+
+ printf("No. routes = %u\n", (unsigned int) NUM_ROUTE_ENTRIES);
+
+ print_route_distribution(large_route_table,
+ (uint32_t) NUM_ROUTE_ENTRIES);
+
+ return measure_fib_perf("fib_perf_dir24_8", &config);
+}
+
+static int
+test_fib_perf_dir8_16_8(void)
+{
+ struct rte_fib_conf config = { 0 };
+
+ config.max_routes = 2000000;
+ config.rib_ext_sz = 0;
+ config.type = RTE_FIB_DIR8_16_8;
+ config.default_nh = 0;
+ config.dir8_16_8.num_tbl16 = 256;
+ config.dir8_16_8.num_tbl8 = 1 << 16;
+
+ generate_large_route_rule_table();
+
+ printf("No. routes = %u\n", (unsigned int) NUM_ROUTE_ENTRIES);
+
+ print_route_distribution(large_route_table,
+ (uint32_t) NUM_ROUTE_ENTRIES);
+
+ return measure_fib_perf("fib_perf_dir8_16_8", &config);
+}
+
REGISTER_PERF_TEST(fib_perf_autotest, test_fib_perf);
+REGISTER_PERF_TEST(fib_perf_dir8_16_8_autotest, test_fib_perf_dir8_16_8);
diff --git a/doc/guides/prog_guide/fib_lib.rst b/doc/guides/prog_guide/fib_lib.rst
index 41ab5764b6..c20759faaf 100644
--- a/doc/guides/prog_guide/fib_lib.rst
+++ b/doc/guides/prog_guide/fib_lib.rst
@@ -127,6 +127,32 @@ An entry in a tbl8 contains the following fields:
* 1 bit indicating if the lookup should proceed inside the tbl8.
+DIR-8-16-8
+~~~~~~~~~~
+
+This algorithm is a three level variant of DIR-24-8: the IPv4 address is split
+in strides of 8, 16 and 8 bits. The first level is a 256 entry root table, the
+second level is made of 65536 entry groups (``tbl16``) and the third level of
+256 entry groups (``tbl8``), both allocated from pools sized on FIB creation.
+A lookup needs one to three memory accesses. Only 8 byte next hop IDs are
+supported.
+
+This algorithm will be used if the ``RTE_FIB_DIR8_16_8`` type is configured as the
+dataplane algorithm on FIB creation.
+
+The main FIB configuration struct stores the dataplane parameters inside ``dir8_16_8``
+within the ``rte_fib_conf`` and it consists of:
+
+* ``num_tbl16``: The number of second level groups, at most one per /8 is ever
+ needed.
+
+* ``num_tbl8``: The number of third level groups.
+
+Compared to DIR-24-8, the root table is small (2 KB), which makes it suitable
+when many tables must be kept, at the price of an extra memory access for
+prefixes longer than 8 bits.
+
+
Use cases
---------
diff --git a/doc/guides/rel_notes/release_26_11.rst b/doc/guides/rel_notes/release_26_11.rst
index 03b4feae0f..04781a2761 100644
--- a/doc/guides/rel_notes/release_26_11.rst
+++ b/doc/guides/rel_notes/release_26_11.rst
@@ -206,6 +206,11 @@ New Features
when using ``RTE_UNCORE_PM_ENV_AUTO_DETECT``,
instead of defaulting only to the Intel uncore driver.
+* **Added new FIB dataplane algorithms.**
+
+ * Added the ``RTE_FIB_DIR8_16_8`` IPv4 algorithm, a three level variant of
+ DIR-24-8 with a small root table, with scalar and AVX512 lookup functions.
+
Removed Items
-------------
diff --git a/lib/fib/dir8_16_8.c b/lib/fib/dir8_16_8.c
new file mode 100644
index 0000000000..7207cd4748
--- /dev/null
+++ b/lib/fib/dir8_16_8.c
@@ -0,0 +1,614 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Intel Corporation
+ */
+
+#include <stdint.h>
+#include <stdio.h>
+
+#include <rte_debug.h>
+#include <rte_malloc.h>
+#include <rte_errno.h>
+#include <rte_vect.h>
+#include <rte_cpuflags.h>
+
+#include <rte_rib.h>
+#include <rte_fib.h>
+#include "dir8_16_8.h"
+#include "fib_log.h"
+
+#ifdef CC_AVX512_SUPPORT
+#include "dir8_16_8_avx512.h"
+#endif
+
+#define DIR8_16_8_NAMESIZE 64
+
+/*
+ * The IPv4 address is split in three strides handled by three table levels:
+ * level 0: bits [31..24], 8 bits, fixed 256 entry root table (dp->tbl0)
+ * level 1: bits [23..8], 16 bits, dynamically allocated 65536 entry groups
+ * level 2: bits [7..0], 8 bits, dynamically allocated 256 entry groups
+ */
+
+struct dir8_16_8_level {
+ uint32_t num_ent; /**< Number of entries per group. */
+ uint8_t shift; /**< IP bit offset of this level. */
+ uint8_t mask_bits; /**< Number of IP bits at this level. */
+};
+
+static const struct dir8_16_8_level dir8_16_8_levels[] = {
+ { DIR8_16_8_TBL0_NUM_ENT, 24, 8 },
+ { DIR8_16_8_TBL16_GRP_NUM_ENT, 8, 16 },
+ { DIR8_16_8_TBL8_GRP_NUM_ENT, 0, 8 },
+};
+
+#define DIR8_16_8_NUM_LEVELS RTE_DIM(dir8_16_8_levels)
+
+static inline uint32_t
+dir8_16_8_level_idx(uint32_t ip, int level)
+{
+ const struct dir8_16_8_level *l = &dir8_16_8_levels[level];
+
+ return (ip >> l->shift) & ((1U << l->mask_bits) - 1);
+}
+
+struct dir8_16_8_rcu_dq_entry {
+ uint32_t tbl_idx;
+ uint32_t level;
+};
+
+static rte_fib_lookup_fn_t
+get_scalar_fn(bool be_addr)
+{
+ return be_addr ? dir8_16_8_lookup_bulk_be : dir8_16_8_lookup_bulk;
+}
+
+static rte_fib_lookup_fn_t
+get_vector_fn(bool be_addr)
+{
+#ifdef CC_AVX512_SUPPORT
+ if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512F) <= 0 ||
+ rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512DQ) <= 0 ||
+ rte_vect_get_max_simd_bitwidth() < RTE_VECT_SIMD_512)
+ return NULL;
+
+ if (be_addr && rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512BW) <= 0)
+ return NULL;
+
+ return be_addr ? rte_dir8_16_8_vec_lookup_bulk_be :
+ rte_dir8_16_8_vec_lookup_bulk;
+#else
+ RTE_SET_USED(be_addr);
+ return NULL;
+#endif
+}
+
+rte_fib_lookup_fn_t
+dir8_16_8_get_lookup_fn(void *p, enum rte_fib_lookup_type type, bool be_addr)
+{
+ struct dir8_16_8_tbl *dp = p;
+ rte_fib_lookup_fn_t fn;
+
+ if (dp == NULL)
+ return NULL;
+
+ switch (type) {
+ case RTE_FIB_LOOKUP_DIR8_16_8_SCALAR:
+ return get_scalar_fn(be_addr);
+ case RTE_FIB_LOOKUP_DIR8_16_8_VECTOR_AVX512:
+ return get_vector_fn(be_addr);
+ case RTE_FIB_LOOKUP_DEFAULT:
+ fn = get_vector_fn(be_addr);
+ return (fn != NULL) ? fn : get_scalar_fn(be_addr);
+ default:
+ return NULL;
+ }
+}
+
+static inline uint64_t *
+get_tbl_by_level(struct dir8_16_8_tbl *dp, int level)
+{
+ switch (level) {
+ case 1:
+ return dp->tbl16;
+ case 2:
+ return dp->tbl8;
+ default:
+ return dp->tbl0;
+ }
+}
+
+static inline int32_t
+tbl_pool_get(struct dir8_16_8_tbl *dp, int level)
+{
+ uint32_t *pool = (level == 1) ? dp->tbl16_pool : dp->tbl8_pool;
+ uint32_t *pos = (level == 1) ? &dp->tbl16_pool_pos : &dp->tbl8_pool_pos;
+ uint32_t num = (level == 1) ? dp->number_tbl16s : dp->number_tbl8s;
+
+ if (*pos == num)
+ return -ENOSPC;
+
+ return pool[(*pos)++];
+}
+
+static inline void
+tbl_pool_put(struct dir8_16_8_tbl *dp, int level, uint32_t idx)
+{
+ uint32_t *pool = (level == 1) ? dp->tbl16_pool : dp->tbl8_pool;
+ uint32_t *pos = (level == 1) ? &dp->tbl16_pool_pos : &dp->tbl8_pool_pos;
+
+ pool[--(*pos)] = idx;
+}
+
+static inline uint64_t *
+group_ptr(struct dir8_16_8_tbl *dp, int level, uint32_t grp_idx)
+{
+ uint64_t *tbl = get_tbl_by_level(dp, level);
+
+ return &tbl[(uint64_t)grp_idx * dir8_16_8_levels[level].num_ent];
+}
+
+static int
+tbl_group_alloc(struct dir8_16_8_tbl *dp, int level, uint64_t nh)
+{
+ uint32_t i, ent_num = dir8_16_8_levels[level].num_ent;
+ uint64_t *grp;
+ int idx;
+
+ idx = tbl_pool_get(dp, level);
+
+ /* If there are no free groups try to reclaim one. */
+ if (unlikely(idx == -ENOSPC && dp->dq)) {
+ /*
+ * Deferred queue contains tbl8 mixed with tbl16 groups.
+ * Reclaim until required group is reclaimed
+ */
+ unsigned int freed = 0;
+ int ret;
+ do {
+ ret = rte_rcu_qsbr_dq_reclaim(dp->dq, 1, &freed, NULL, NULL);
+ idx = tbl_pool_get(dp, level);
+ } while (idx < 0 && ret == 0 && freed != 0);
+ }
+
+ if (idx < 0)
+ return idx;
+
+ grp = group_ptr(dp, level, idx);
+ for (i = 0; i < ent_num; i++)
+ grp[i] = nh & ~(uint64_t)DIR8_16_8_EXT_ENT;
+
+ if (level == 1)
+ dp->cur_tbl16s++;
+ else
+ dp->cur_tbl8s++;
+
+ return idx;
+}
+
+static void
+tbl_group_cleanup_and_free(struct dir8_16_8_tbl *dp, int level, uint32_t idx)
+{
+ uint64_t *grp = group_ptr(dp, level, idx);
+
+ memset(grp, 0, dir8_16_8_levels[level].num_ent * sizeof(uint64_t));
+ tbl_pool_put(dp, level, idx);
+ if (level == 1)
+ dp->cur_tbl16s--;
+ else
+ dp->cur_tbl8s--;
+}
+
+static void
+__rcu_qsbr_free_resource(void *p, void *data, unsigned int n __rte_unused)
+{
+ struct dir8_16_8_tbl *dp = p;
+ struct dir8_16_8_rcu_dq_entry *e = data;
+
+ tbl_group_cleanup_and_free(dp, e->level, e->tbl_idx);
+}
+
+/*
+ * If every entry of a group holds the same final nexthop, collapse the group:
+ * write that nexthop into the parent entry and free the group.
+ */
+static void
+tbl_group_recycle(struct dir8_16_8_tbl *dp, int level, uint64_t *par_ent,
+ uint32_t grp_idx)
+{
+ struct dir8_16_8_rcu_dq_entry dq_entry;
+ uint64_t *grp = group_ptr(dp, level, grp_idx);
+ uint64_t nh = grp[0];
+ uint32_t i;
+
+ if (dir8_16_8_is_entry_extended(nh))
+ return;
+ for (i = 1; i < dir8_16_8_levels[level].num_ent; i++) {
+ if (nh != grp[i])
+ return;
+ }
+
+ *par_ent = nh & ~DIR8_16_8_EXT_ENT;
+
+ if (dp->v == NULL) {
+ tbl_group_cleanup_and_free(dp, level, grp_idx);
+ } else if (dp->rcu_mode == RTE_FIB_QSBR_MODE_SYNC) {
+ rte_rcu_qsbr_synchronize(dp->v, RTE_QSBR_THRID_INVALID);
+ tbl_group_cleanup_and_free(dp, level, grp_idx);
+ } else { /* RTE_FIB_QSBR_MODE_DQ */
+ dq_entry.tbl_idx = grp_idx;
+ dq_entry.level = level;
+ if (rte_rcu_qsbr_dq_enqueue(dp->dq, &dq_entry))
+ FIB_LOG(ERR, "Failed to push QSBR FIFO");
+ }
+}
+
+/*
+ * Make sure the entry "ent" at "level" points to a child group, allocating a
+ * new group (initialized with the entry's current nexthop) if necessary.
+ * On success returns the child group index, otherwise a negative errno.
+ */
+static int
+tbl_get_child(struct dir8_16_8_tbl *dp, int level, uint64_t *ent)
+{
+ int idx;
+
+ if (dir8_16_8_is_entry_extended(*ent))
+ return *ent >> 1;
+
+ idx = tbl_group_alloc(dp, level + 1, *ent);
+ if (idx < 0)
+ return idx;
+ *ent = ((uint64_t)idx << 1) | DIR8_16_8_EXT_ENT;
+
+ return idx;
+}
+
+static int
+install_to_fib(struct dir8_16_8_tbl *dp, int level, uint64_t base,
+ uint64_t ledge, uint64_t redge, uint64_t next_hop)
+{
+ const struct dir8_16_8_level *l = &dir8_16_8_levels[level];
+ uint64_t *tbl = get_tbl_by_level(dp, level);
+ uint64_t step = 1ULL << l->shift;
+ uint64_t lalign, ralign;
+ uint32_t lidx, ridx, idx;
+ int child, ret;
+
+ if (level == (int)DIR8_16_8_NUM_LEVELS - 1) {
+ /* Leaf level: every entry is a single address, plain fill. */
+ lidx = dir8_16_8_level_idx((uint32_t)ledge, level);
+ ridx = lidx + (uint32_t)(redge - ledge);
+ for (idx = lidx; idx < ridx; idx++)
+ tbl[base + idx] = (next_hop << 1);
+
+ return 0;
+ }
+
+ lalign = RTE_ALIGN_CEIL(ledge, step);
+ ralign = RTE_ALIGN_FLOOR(redge, step);
+
+ /* Left partial sub-range that does not start on a child boundary. */
+ if (ledge < lalign) {
+ uint64_t sub_right_edge = RTE_MIN(lalign, redge);
+
+ lidx = dir8_16_8_level_idx((uint32_t)ledge, level);
+ child = tbl_get_child(dp, level, &tbl[base + lidx]);
+ if (child < 0)
+ return child;
+
+ ret = install_to_fib(dp, level + 1,
+ (uint64_t)child * dir8_16_8_levels[level + 1].num_ent,
+ ledge, sub_right_edge, next_hop);
+ if (ret != 0)
+ return ret;
+
+ tbl_group_recycle(dp, level + 1, &tbl[base + lidx], child);
+ ledge = sub_right_edge;
+ }
+
+ /* Whole child sub-ranges filled directly at this level. */
+ if (ledge < ralign) {
+ lidx = dir8_16_8_level_idx((uint32_t)ledge, level);
+ ridx = lidx + (uint32_t)((ralign - ledge) >> l->shift);
+ for (idx = lidx; idx < ridx; idx++)
+ tbl[base + idx] = (next_hop << 1);
+
+ ledge = ralign;
+ }
+
+ /* Right partial sub-range that does not end on a child boundary. */
+ if (ledge < redge) {
+ lidx = dir8_16_8_level_idx((uint32_t)ledge, level);
+ child = tbl_get_child(dp, level, &tbl[base + lidx]);
+ if (child < 0)
+ return child;
+
+ ret = install_to_fib(dp, level + 1,
+ (uint64_t)child * dir8_16_8_levels[level + 1].num_ent,
+ ledge, redge, next_hop);
+ if (ret != 0)
+ return ret;
+
+ tbl_group_recycle(dp, level + 1, &tbl[base + lidx], child);
+ }
+
+ return 0;
+}
+
+static int
+modify_fib(struct dir8_16_8_tbl *dp, struct rte_rib *rib, uint32_t ip,
+ uint8_t depth, uint64_t next_hop)
+{
+ struct rte_rib_node *tmp = NULL;
+ uint64_t ledge, redge;
+ uint32_t tmp_ip;
+ int ret;
+ uint8_t tmp_depth;
+
+ ledge = ip;
+ do {
+ tmp = rte_rib_get_nxt(rib, ip, depth, tmp,
+ RTE_RIB_GET_NXT_COVER);
+ if (tmp != NULL) {
+ rte_rib_get_depth(tmp, &tmp_depth);
+ if (tmp_depth == depth)
+ continue;
+
+ rte_rib_get_ip(tmp, &tmp_ip);
+ redge = tmp_ip & rte_rib_depth_to_mask(tmp_depth);
+ if (ledge == redge) {
+ ledge = redge + (1ULL << (32 - tmp_depth));
+ continue;
+ }
+
+ ret = install_to_fib(dp, 0, 0, ledge, redge, next_hop);
+ if (ret != 0)
+ return ret;
+
+ ledge = redge + (1ULL << (32 - tmp_depth));
+ /*
+ * we got to the end of address space
+ * and wrapped around
+ */
+ if (ledge >= (1ULL << 32))
+ break;
+ } else {
+ redge = (uint64_t)ip + (1ULL << (32 - depth));
+ if (ledge == redge && ledge != 0)
+ break;
+
+ ret = install_to_fib(dp, 0, 0, ledge, redge, next_hop);
+ if (ret != 0)
+ return ret;
+ }
+ } while (tmp);
+
+ return 0;
+}
+
+int
+dir8_16_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t depth,
+ uint64_t next_hop, int op)
+{
+ struct dir8_16_8_tbl *dp;
+ struct rte_rib *rib;
+ struct rte_rib_node *node;
+ struct rte_rib_node *parent;
+ int ret = 0;
+ uint64_t par_nh, node_nh;
+
+ if ((fib == NULL) || (depth > RTE_FIB_MAXDEPTH))
+ return -EINVAL;
+
+ dp = rte_fib_get_dp(fib);
+ rib = rte_fib_get_rib(fib);
+ RTE_ASSERT((dp != NULL) && (rib != NULL));
+
+ if (next_hop > dir8_16_8_get_max_nh())
+ return -EINVAL;
+
+ ip &= rte_rib_depth_to_mask(depth);
+
+ node = rte_rib_lookup_exact(rib, ip, depth);
+ switch (op) {
+ case RTE_FIB_ADD:
+ if (node != NULL) {
+ rte_rib_get_nh(node, &node_nh);
+ if (node_nh == next_hop)
+ return 0;
+
+ ret = modify_fib(dp, rib, ip, depth, next_hop);
+ if (ret == 0)
+ rte_rib_set_nh(node, next_hop);
+
+ return ret;
+ }
+ node = rte_rib_insert(rib, ip, depth);
+ if (node == NULL)
+ return -rte_errno;
+
+ rte_rib_set_nh(node, next_hop);
+ parent = rte_rib_lookup_parent(node);
+ if (parent != NULL) {
+ rte_rib_get_nh(parent, &par_nh);
+ if (par_nh == next_hop)
+ return 0;
+ }
+ ret = modify_fib(dp, rib, ip, depth, next_hop);
+ if (ret != 0) {
+ rte_rib_remove(rib, ip, depth);
+ return ret;
+ }
+
+ return 0;
+ case RTE_FIB_DEL:
+ if (node == NULL)
+ return -ENOENT;
+
+ parent = rte_rib_lookup_parent(node);
+ if (parent != NULL) {
+ rte_rib_get_nh(parent, &par_nh);
+ rte_rib_get_nh(node, &node_nh);
+ if (par_nh != node_nh)
+ ret = modify_fib(dp, rib, ip, depth, par_nh);
+ } else
+ ret = modify_fib(dp, rib, ip, depth, dp->def_nh);
+ if (ret == 0)
+ rte_rib_remove(rib, ip, depth);
+ return ret;
+ default:
+ break;
+ }
+
+ return -EINVAL;
+}
+
+void *
+dir8_16_8_create(const char *name, int socket_id, struct rte_fib_conf *fib_conf)
+{
+ char mem_name[DIR8_16_8_NAMESIZE];
+ struct dir8_16_8_tbl *dp;
+ uint64_t def_nh;
+ uint32_t num_tbl16, num_tbl8;
+ uint32_t i;
+ uint64_t tbl16_sz, tbl8_sz;
+
+ if ((name == NULL) || (fib_conf == NULL) || (fib_conf->dir8_16_8.num_tbl16 == 0) ||
+ (fib_conf->dir8_16_8.num_tbl8 == 0) ||
+ (fib_conf->default_nh > dir8_16_8_get_max_nh())) {
+ rte_errno = EINVAL;
+ return NULL;
+ }
+
+ def_nh = fib_conf->default_nh;
+ num_tbl16 = fib_conf->dir8_16_8.num_tbl16;
+ num_tbl8 = fib_conf->dir8_16_8.num_tbl8;
+
+ dp = rte_zmalloc_socket(name, sizeof(struct dir8_16_8_tbl) +
+ DIR8_16_8_TBL0_NUM_ENT * sizeof(uint64_t), RTE_CACHE_LINE_SIZE, socket_id);
+ if (dp == NULL) {
+ rte_errno = ENOMEM;
+ return NULL;
+ }
+
+ /* Init first level table with the default nexthop. */
+ for (i = 0; i < DIR8_16_8_TBL0_NUM_ENT; i++)
+ dp->tbl0[i] = def_nh << 1;
+
+ snprintf(mem_name, sizeof(mem_name), "TBL16_%p", dp);
+ tbl16_sz = (uint64_t)DIR8_16_8_TBL16_GRP_NUM_ENT * sizeof(uint64_t) * num_tbl16;
+ dp->tbl16 = rte_zmalloc_socket(mem_name, tbl16_sz, RTE_CACHE_LINE_SIZE, socket_id);
+ if (dp->tbl16 == NULL) {
+ rte_errno = ENOMEM;
+ goto free_dp;
+ }
+
+ snprintf(mem_name, sizeof(mem_name), "TBL8_%p", dp);
+ tbl8_sz = (uint64_t)DIR8_16_8_TBL8_GRP_NUM_ENT * sizeof(uint64_t) * num_tbl8;
+ dp->tbl8 = rte_zmalloc_socket(mem_name, tbl8_sz, RTE_CACHE_LINE_SIZE, socket_id);
+ if (dp->tbl8 == NULL) {
+ rte_errno = ENOMEM;
+ goto free_tbl16;
+ }
+
+ snprintf(mem_name, sizeof(mem_name), "TBL16_pool_%p", dp);
+ dp->tbl16_pool = rte_zmalloc_socket(mem_name, sizeof(uint32_t) * num_tbl16,
+ RTE_CACHE_LINE_SIZE, socket_id);
+ if (dp->tbl16_pool == NULL) {
+ rte_errno = ENOMEM;
+ goto free_tbl8;
+ }
+ for (i = 0; i < num_tbl16; i++)
+ dp->tbl16_pool[i] = i;
+
+ snprintf(mem_name, sizeof(mem_name), "TBL8_pool_%p", dp);
+ dp->tbl8_pool = rte_zmalloc_socket(mem_name, sizeof(uint32_t) * num_tbl8,
+ RTE_CACHE_LINE_SIZE, socket_id);
+ if (dp->tbl8_pool == NULL) {
+ rte_errno = ENOMEM;
+ goto free_tbl16_pool;
+ }
+
+ for (i = 0; i < num_tbl8; i++)
+ dp->tbl8_pool[i] = i;
+
+ dp->def_nh = def_nh;
+ dp->number_tbl16s = num_tbl16;
+ dp->number_tbl8s = num_tbl8;
+
+ return dp;
+
+free_tbl16_pool:
+ rte_free(dp->tbl16_pool);
+free_tbl8:
+ rte_free(dp->tbl8);
+free_tbl16:
+ rte_free(dp->tbl16);
+free_dp:
+ rte_free(dp);
+ return NULL;
+}
+
+void
+dir8_16_8_free(void *p)
+{
+ struct dir8_16_8_tbl *dp = (struct dir8_16_8_tbl *)p;
+
+ if (dp == NULL)
+ return;
+
+ rte_rcu_qsbr_dq_delete(dp->dq);
+ rte_free(dp->tbl8_pool);
+ rte_free(dp->tbl16_pool);
+ rte_free(dp->tbl8);
+ rte_free(dp->tbl16);
+ rte_free(dp);
+}
+
+int
+dir8_16_8_rcu_qsbr_add(struct dir8_16_8_tbl *dp, struct rte_fib_rcu_config *cfg,
+ const char *name)
+{
+ struct rte_rcu_qsbr_dq_parameters params = {0};
+ char rcu_dq_name[RTE_RCU_QSBR_DQ_NAMESIZE];
+
+ if (dp == NULL || cfg == NULL)
+ return -EINVAL;
+
+ if (dp->v != NULL)
+ return -EEXIST;
+
+ if (cfg->mode == RTE_FIB_QSBR_MODE_SYNC) {
+ /* No other things to do. */
+ } else if (cfg->mode == RTE_FIB_QSBR_MODE_DQ) {
+ /* Init QSBR defer queue. */
+ snprintf(rcu_dq_name, sizeof(rcu_dq_name),
+ "FIB_RCU_%s", name);
+ params.name = rcu_dq_name;
+ params.size = cfg->dq_size;
+ if (params.size == 0)
+ params.size = RTE_FIB_RCU_DQ_RECLAIM_SZ;
+
+ params.trigger_reclaim_limit = cfg->reclaim_thd;
+ params.max_reclaim_size = cfg->reclaim_max;
+ if (params.max_reclaim_size == 0)
+ params.max_reclaim_size = RTE_FIB_RCU_DQ_RECLAIM_MAX;
+
+ params.esize = sizeof(struct dir8_16_8_rcu_dq_entry);
+ params.free_fn = __rcu_qsbr_free_resource;
+ params.p = dp;
+ params.v = cfg->v;
+ dp->dq = rte_rcu_qsbr_dq_create(¶ms);
+ if (dp->dq == NULL) {
+ FIB_LOG(ERR, "LPM defer queue creation failed");
+ return -rte_errno;
+ }
+ } else {
+ return -EINVAL;
+ }
+
+ dp->rcu_mode = cfg->mode;
+ dp->v = cfg->v;
+
+ return 0;
+}
diff --git a/lib/fib/dir8_16_8.h b/lib/fib/dir8_16_8.h
new file mode 100644
index 0000000000..31ca5162de
--- /dev/null
+++ b/lib/fib/dir8_16_8.h
@@ -0,0 +1,143 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Intel Corporation
+ */
+
+#ifndef _DIR8_16_8_H_
+#define _DIR8_16_8_H_
+
+#include <stdalign.h>
+#include <stdbool.h>
+
+#include <rte_byteorder.h>
+#include <rte_prefetch.h>
+#include <rte_branch_prediction.h>
+#include <rte_rcu_qsbr.h>
+
+/**
+ * @file
+ * DIR8_16_8 algorithm
+ *
+ * Three level variant of the DIR24_8 algorithm.
+ * The 32 bit IPv4 address is split in three strides of 8, 16 and 8 bits.
+ * Only a single nexthop size of 8 bytes is supported.
+ */
+
+/** Number of entries in the first level table (top 8 bits). */
+#define DIR8_16_8_TBL0_NUM_ENT 256U
+/** Number of entries in a second level group (middle 16 bits). */
+#define DIR8_16_8_TBL16_GRP_NUM_ENT (1U << 16)
+/** Number of entries in a third level group (last 8 bits). */
+#define DIR8_16_8_TBL8_GRP_NUM_ENT 256U
+/** Flag set in an entry pointing to the next level table. */
+#define DIR8_16_8_EXT_ENT 1
+
+struct dir8_16_8_tbl {
+ uint32_t number_tbl16s; /**< Total number of tbl16 groups. */
+ uint32_t cur_tbl16s; /**< Current number of tbl16 groups. */
+ uint32_t number_tbl8s; /**< Total number of tbl8 groups. */
+ uint32_t cur_tbl8s; /**< Current number of tbl8 groups. */
+ uint64_t def_nh; /**< Default next hop. */
+ uint64_t *tbl16; /**< Second level table. */
+ uint64_t *tbl8; /**< Third level table. */
+ uint32_t *tbl16_pool; /**< Stack of free tbl16 group idxes. */
+ uint32_t *tbl8_pool; /**< Stack of free tbl8 group idxes. */
+ uint32_t tbl16_pool_pos; /**< Allocation pos in tbl16 pool. */
+ uint32_t tbl8_pool_pos; /**< Allocation pos in tbl8 pool. */
+ /* RCU config. */
+ enum rte_fib_qsbr_mode rcu_mode;/* Blocking, defer queue. */
+ struct rte_rcu_qsbr *v; /* RCU QSBR variable. */
+ struct rte_rcu_qsbr_dq *dq; /* RCU QSBR defer queue. */
+ /* First level table (top 8 bits). */
+ alignas(RTE_CACHE_LINE_SIZE) uint64_t tbl0[];
+};
+
+static inline uint32_t
+dir8_16_8_get_tbl0_idx(uint32_t ip)
+{
+ return ip >> 24;
+}
+
+static inline size_t
+dir8_16_8_get_tbl16_idx(uint64_t ent, uint32_t ip)
+{
+ return (ent >> 1) * DIR8_16_8_TBL16_GRP_NUM_ENT +
+ ((ip >> 8) & 0xffff);
+}
+
+static inline size_t
+dir8_16_8_get_tbl8_idx(uint64_t ent, uint32_t ip)
+{
+ return (ent >> 1) * DIR8_16_8_TBL8_GRP_NUM_ENT + (uint8_t)ip;
+}
+
+static inline uint64_t
+dir8_16_8_get_max_nh(void)
+{
+ return (UINT64_MAX >> 1);
+}
+
+static inline int
+dir8_16_8_is_entry_extended(uint64_t ent)
+{
+ return (ent & DIR8_16_8_EXT_ENT) == DIR8_16_8_EXT_ENT;
+}
+
+static inline void
+dir8_16_8_lookup_bulk(void *p, const uint32_t *ips,
+ uint64_t *next_hops, const unsigned int n)
+{
+ struct dir8_16_8_tbl *dp = (struct dir8_16_8_tbl *)p;
+ uint64_t tmp;
+ uint32_t i;
+
+ for (i = 0; i < n; i++) {
+ tmp = dp->tbl0[dir8_16_8_get_tbl0_idx(ips[i])];
+ /* Prefixes are usually longer than /8. */
+ if (dir8_16_8_is_entry_extended(tmp))
+ tmp = dp->tbl16[dir8_16_8_get_tbl16_idx(tmp, ips[i])];
+ if (unlikely(dir8_16_8_is_entry_extended(tmp)))
+ tmp = dp->tbl8[dir8_16_8_get_tbl8_idx(tmp, ips[i])];
+ next_hops[i] = tmp >> 1;
+ }
+}
+
+#define DIR8_16_8_BSWAP_MAX_LENGTH 64
+
+static inline void
+dir8_16_8_lookup_bulk_be(void *p, const uint32_t *ips,
+ uint64_t *next_hops, const unsigned int n)
+{
+#if RTE_BYTE_ORDER == RTE_BIG_ENDIAN
+ dir8_16_8_lookup_bulk(p, ips, next_hops, n);
+#else
+ uint32_t le_ips[DIR8_16_8_BSWAP_MAX_LENGTH];
+ unsigned int i;
+
+ for (i = 0; i < n; i += DIR8_16_8_BSWAP_MAX_LENGTH) {
+ unsigned int j;
+ for (j = 0; j < DIR8_16_8_BSWAP_MAX_LENGTH && i + j < n; j++)
+ le_ips[j] = rte_be_to_cpu_32(ips[i + j]);
+
+ dir8_16_8_lookup_bulk(p, le_ips, next_hops + i, j);
+ }
+#endif
+}
+
+void *
+dir8_16_8_create(const char *name, int socket_id, struct rte_fib_conf *conf);
+
+void
+dir8_16_8_free(void *p);
+
+rte_fib_lookup_fn_t
+dir8_16_8_get_lookup_fn(void *p, enum rte_fib_lookup_type type, bool be_addr);
+
+int
+dir8_16_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t depth,
+ uint64_t next_hop, int op);
+
+int
+dir8_16_8_rcu_qsbr_add(struct dir8_16_8_tbl *dp, struct rte_fib_rcu_config *cfg,
+ const char *name);
+
+#endif /* _DIR8_16_8_H_ */
diff --git a/lib/fib/dir8_16_8_avx512.c b/lib/fib/dir8_16_8_avx512.c
new file mode 100644
index 0000000000..c147451252
--- /dev/null
+++ b/lib/fib/dir8_16_8_avx512.c
@@ -0,0 +1,142 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Intel Corporation
+ */
+
+#include <rte_vect.h>
+#include <rte_fib.h>
+
+#include "dir8_16_8.h"
+#include "dir8_16_8_avx512.h"
+
+static inline __m256i
+dir8_16_8_vec_load_ips(const uint32_t *ips, bool be_addr)
+{
+ const __m256i bswap32 = _mm256_set_epi8(
+ 12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1, 2, 3,
+ 12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1, 2, 3
+ );
+ __m256i ip_vec = _mm256_loadu_si256((const void *)ips);
+
+ return be_addr ? _mm256_shuffle_epi8(ip_vec, bswap32) : ip_vec;
+}
+
+static inline __m256i
+dir8_16_8_vec_get_tbl0_idx(__m256i ip_vec)
+{
+ return _mm256_srli_epi32(ip_vec, 24);
+}
+
+/* Second lookup stage input for 8 IPs, filled by the first stage. */
+struct dir8_16_8_vec_stage {
+ /* Address of the entry to read for every IP. */
+ const uint64_t *ent[8];
+ /*
+ * Copy of the tbl0 entries. A not extended entry is read from here,
+ * reading it again from tbl0 could see a concurrent update.
+ */
+ uint64_t tbl0_ent[8];
+};
+
+/*
+ * First lookup stage of 8 IPs: read their tbl0 entries and prefetch the
+ * tbl16 entries the extended ones point to. Scalar loads are used, as on
+ * some CPUs a gather costs much more than the 8 loads it replaces.
+ */
+static inline void
+dir8_16_8_vec_prefetch_x8(const struct dir8_16_8_tbl *dp, const uint32_t *ips, bool be_addr,
+ struct dir8_16_8_vec_stage *st)
+{
+ const __m256i ip_vec = dir8_16_8_vec_load_ips(ips, be_addr);
+ const rte_ymm_t idx0 = { .y = dir8_16_8_vec_get_tbl0_idx(ip_vec) };
+ const rte_ymm_t ip = { .y = ip_vec };
+ const uint64_t *tbl0 = dp->tbl0;
+ const uint64_t *tbl16 = dp->tbl16;
+ uint64_t ent;
+ uint32_t i;
+
+ for (i = 0; i < 8; i++) {
+ ent = tbl0[idx0.u32[i]];
+ st->tbl0_ent[i] = ent;
+ st->ent[i] = &st->tbl0_ent[i];
+ if (dir8_16_8_is_entry_extended(ent)) {
+ st->ent[i] = &tbl16[dir8_16_8_get_tbl16_idx(ent, ip.u32[i])];
+ rte_prefetch0(st->ent[i]);
+ }
+ }
+}
+
+/* Second lookup stage of 8 IPs: read the entries found by the first stage. */
+static inline void
+dir8_16_8_vec_lookup_x8(const struct dir8_16_8_tbl *dp, const struct dir8_16_8_vec_stage *st,
+ const uint32_t *ips, bool be_addr, uint64_t *next_hops)
+{
+ const uint64_t * const *ent = st->ent;
+ const __m512i lsb = _mm512_set1_epi64(1);
+ __m512i ip64, res, idxes;
+ __mmask8 msk_ext;
+
+ res = _mm512_set_epi64(*ent[7], *ent[6], *ent[5], *ent[4],
+ *ent[3], *ent[2], *ent[1], *ent[0]);
+ ip64 = _mm512_cvtepu32_epi64(dir8_16_8_vec_load_ips(ips, be_addr));
+
+ msk_ext = _mm512_test_epi64_mask(res, lsb);
+ if (msk_ext != 0) {
+ idxes = _mm512_slli_epi64(_mm512_srli_epi64(res, 1), 8);
+ idxes = _mm512_add_epi64(idxes, _mm512_and_epi64(ip64, _mm512_set1_epi64(0xff)));
+ res = _mm512_mask_i64gather_epi64(res, msk_ext, idxes, (const void *)dp->tbl8, 8);
+ }
+
+ _mm512_storeu_si512(next_hops, _mm512_srli_epi64(res, 1));
+}
+
+static inline void
+vec_lookup_bulk(void *p, const uint32_t *ips, uint64_t *next_hops, unsigned int n, bool be_addr)
+{
+ const struct dir8_16_8_tbl *dp = p;
+ struct dir8_16_8_vec_stage stage[3];
+ struct dir8_16_8_vec_stage *st0 = &stage[0];
+ struct dir8_16_8_vec_stage *st1 = &stage[1];
+ struct dir8_16_8_vec_stage *st2 = &stage[2];
+ struct dir8_16_8_vec_stage *tmp;
+ unsigned int i = 0;
+
+ if (n >= 16) {
+ dir8_16_8_vec_prefetch_x8(dp, ips, be_addr, st0);
+ dir8_16_8_vec_prefetch_x8(dp, &ips[8], be_addr, st1);
+ for (; i + 24 <= n; i += 8) {
+ dir8_16_8_vec_prefetch_x8(dp, &ips[i + 16], be_addr, st2);
+ dir8_16_8_vec_lookup_x8(dp, st0, &ips[i], be_addr, &next_hops[i]);
+ tmp = st0;
+ st0 = st1;
+ st1 = st2;
+ st2 = tmp;
+ }
+ dir8_16_8_vec_lookup_x8(dp, st0, &ips[i], be_addr, &next_hops[i]);
+ dir8_16_8_vec_lookup_x8(dp, st1, &ips[i + 8], be_addr, &next_hops[i + 8]);
+ i += 16;
+ } else if (n >= 8) {
+ dir8_16_8_vec_prefetch_x8(dp, ips, be_addr, st0);
+ dir8_16_8_vec_lookup_x8(dp, st0, ips, be_addr, next_hops);
+ i = 8;
+ }
+
+ /* Scalar lookup of the last n % 8 IPs. */
+ if (be_addr)
+ dir8_16_8_lookup_bulk_be(p, &ips[i], &next_hops[i], n - i);
+ else
+ dir8_16_8_lookup_bulk(p, &ips[i], &next_hops[i], n - i);
+}
+
+void
+rte_dir8_16_8_vec_lookup_bulk(void *p, const uint32_t *ips,
+ uint64_t *next_hops, const unsigned int n)
+{
+ vec_lookup_bulk(p, ips, next_hops, n, false);
+}
+
+void
+rte_dir8_16_8_vec_lookup_bulk_be(void *p, const uint32_t *ips,
+ uint64_t *next_hops, const unsigned int n)
+{
+ vec_lookup_bulk(p, ips, next_hops, n, true);
+}
diff --git a/lib/fib/dir8_16_8_avx512.h b/lib/fib/dir8_16_8_avx512.h
new file mode 100644
index 0000000000..d468481029
--- /dev/null
+++ b/lib/fib/dir8_16_8_avx512.h
@@ -0,0 +1,16 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Intel Corporation
+ */
+
+#ifndef _DIR8_16_8_AVX512_H_
+#define _DIR8_16_8_AVX512_H_
+
+void
+rte_dir8_16_8_vec_lookup_bulk(void *p, const uint32_t *ips,
+ uint64_t *next_hops, const unsigned int n);
+
+void
+rte_dir8_16_8_vec_lookup_bulk_be(void *p, const uint32_t *ips,
+ uint64_t *next_hops, const unsigned int n);
+
+#endif /* _DIR8_16_8_AVX512_H_ */
diff --git a/lib/fib/meson.build b/lib/fib/meson.build
index 573fc50ff1..f24f679039 100644
--- a/lib/fib/meson.build
+++ b/lib/fib/meson.build
@@ -2,14 +2,14 @@
# Copyright(c) 2018 Vladimir Medvedkin <medvedkinv@gmail.com>
# Copyright(c) 2019 Intel Corporation
-sources = files('rte_fib.c', 'rte_fib6.c', 'dir24_8.c', 'trie.c')
+sources = files('rte_fib.c', 'rte_fib6.c', 'dir24_8.c', 'dir8_16_8.c', 'trie.c')
headers = files('rte_fib.h', 'rte_fib6.h')
deps += ['rib']
deps += ['rcu']
deps += ['net']
if dpdk_conf.has('RTE_ARCH_X86_64')
- sources_avx512 += files('dir24_8_avx512.c', 'trie_avx512.c')
+ sources_avx512 += files('dir24_8_avx512.c', 'dir8_16_8_avx512.c', 'trie_avx512.c')
elif dpdk_conf.has('RTE_ARCH_RISCV')
sources += files('dir24_8_rvv.c')
endif
diff --git a/lib/fib/rte_fib.c b/lib/fib/rte_fib.c
index 184210f380..fcc28899cb 100644
--- a/lib/fib/rte_fib.c
+++ b/lib/fib/rte_fib.c
@@ -18,6 +18,7 @@
#include <rte_fib.h>
#include "dir24_8.h"
+#include "dir8_16_8.h"
#include "fib_log.h"
RTE_LOG_REGISTER_DEFAULT(fib_logtype, INFO);
@@ -112,6 +113,14 @@ init_dataplane(struct rte_fib *fib, __rte_unused int socket_id,
RTE_FIB_LOOKUP_DEFAULT, !!(fib->flags & RTE_FIB_F_LOOKUP_NETWORK_ORDER));
fib->modify = dir24_8_modify;
return 0;
+ case RTE_FIB_DIR8_16_8:
+ fib->dp = dir8_16_8_create(dp_name, socket_id, conf);
+ if (fib->dp == NULL)
+ return -rte_errno;
+ fib->lookup = dir8_16_8_get_lookup_fn(fib->dp,
+ RTE_FIB_LOOKUP_DEFAULT, !!(fib->flags & RTE_FIB_F_LOOKUP_NETWORK_ORDER));
+ fib->modify = dir8_16_8_modify;
+ return 0;
default:
return -EINVAL;
}
@@ -165,7 +174,7 @@ rte_fib_create(const char *name, int socket_id, struct rte_fib_conf *conf)
/* Check user arguments. */
if ((name == NULL) || (conf == NULL) || (conf->max_routes < 0) ||
(conf->flags & ~RTE_FIB_ALLOWED_FLAGS) ||
- (conf->type > RTE_FIB_DIR24_8)) {
+ (conf->type > RTE_FIB_DIR8_16_8)) {
rte_errno = EINVAL;
return NULL;
}
@@ -281,6 +290,10 @@ free_dataplane(struct rte_fib *fib)
return;
case RTE_FIB_DIR24_8:
dir24_8_free(fib->dp);
+ return;
+ case RTE_FIB_DIR8_16_8:
+ dir8_16_8_free(fib->dp);
+ return;
default:
return;
}
@@ -345,6 +358,13 @@ rte_fib_select_lookup(struct rte_fib *fib,
return -EINVAL;
fib->lookup = fn;
return 0;
+ case RTE_FIB_DIR8_16_8:
+ fn = dir8_16_8_get_lookup_fn(fib->dp, type,
+ !!(fib->flags & RTE_FIB_F_LOOKUP_NETWORK_ORDER));
+ if (fn == NULL)
+ return -EINVAL;
+ fib->lookup = fn;
+ return 0;
default:
return -EINVAL;
}
@@ -360,6 +380,8 @@ rte_fib_rcu_qsbr_add(struct rte_fib *fib, struct rte_fib_rcu_config *cfg)
switch (fib->type) {
case RTE_FIB_DIR24_8:
return dir24_8_rcu_qsbr_add(fib->dp, cfg, fib->name);
+ case RTE_FIB_DIR8_16_8:
+ return dir8_16_8_rcu_qsbr_add(fib->dp, cfg, fib->name);
default:
return -ENOTSUP;
}
diff --git a/lib/fib/rte_fib.h b/lib/fib/rte_fib.h
index b16a653535..68ee6f0652 100644
--- a/lib/fib/rte_fib.h
+++ b/lib/fib/rte_fib.h
@@ -49,7 +49,8 @@ enum rte_fib_qsbr_mode {
/** Type of FIB struct */
enum rte_fib_type {
RTE_FIB_DUMMY, /**< RIB tree based FIB */
- RTE_FIB_DIR24_8 /**< DIR24_8 based FIB */
+ RTE_FIB_DIR24_8, /**< DIR24_8 based FIB */
+ RTE_FIB_DIR8_16_8 /**< DIR8_16_8 based FIB */
};
/** Modify FIB function */
@@ -87,8 +88,14 @@ enum rte_fib_lookup_type {
/**<
* Unified lookup function for all next hop sizes
*/
- RTE_FIB_LOOKUP_DIR24_8_VECTOR_AVX512
+ RTE_FIB_LOOKUP_DIR24_8_VECTOR_AVX512,
/**< Vector implementation using AVX512 */
+ RTE_FIB_LOOKUP_DIR8_16_8_SCALAR,
+ /**<
+ * Scalar implementation for the DIR8_16_8 algorithm
+ */
+ RTE_FIB_LOOKUP_DIR8_16_8_VECTOR_AVX512
+ /**< Vector (AVX512) lookup for the DIR8_16_8 algorithm */
};
/** If set, fib lookup is expecting IPv4 address in network byte order */
@@ -108,6 +115,12 @@ struct rte_fib_conf {
enum rte_fib_dir24_8_nh_sz nh_sz;
uint32_t num_tbl8;
} dir24_8;
+ struct {
+ /** Number of second level (16 bit stride) groups. */
+ uint32_t num_tbl16;
+ /** Number of third level (8 bit stride) groups. */
+ uint32_t num_tbl8;
+ } dir8_16_8;
};
unsigned int flags; /**< Optional feature flags from RTE_FIB_F_* */
};
--
2.43.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [RFC PATCH v2 5/7] fib: add multi-VRF support
2026-10-14 18:39 ` [RFC PATCH v2 0/7] " Vladimir Medvedkin
` (3 preceding siblings ...)
2026-10-14 18:39 ` [RFC PATCH v2 4/7] fib: add DIR8-16-8 algorithm Vladimir Medvedkin
@ 2026-10-14 18:39 ` Vladimir Medvedkin
2026-10-14 18:39 ` [RFC PATCH v2 6/7] fib: split DIR24-8 table update from the FIB wrapper Vladimir Medvedkin
` (2 subsequent siblings)
7 siblings, 0 replies; 45+ messages in thread
From: Vladimir Medvedkin @ 2026-10-14 18:39 UTC (permalink / raw)
To: dev; +Cc: rjarry, mb, maxime, konstantin.ananyev
Allow a FIB to hold several independent routing tables (VRFs):
- rte_fib_conf gains the number of VRFs (max_vrfs, 0 or 1 meaning a
single VRF as before) and optional per-VRF default nexthops;
- rte_fib_vrf_add(), rte_fib_vrf_delete() and rte_fib_vrf_lookup_bulk()
take a VRF ID (one per address for the lookup), rte_fib_vrf_get_rib()
returns the RIB of a VRF. The legacy functions operate on VRF 0;
- the dataplane modify and lookup callbacks gain a VRF argument, a NULL
VRF array meaning VRF 0 for every address.
The FIB keeps one RIB per VRF. The DIR8_16_8 algorithm supports
multiple VRFs natively: its 2 KB root table is replicated per VRF while
the tbl16 and tbl8 pools are shared, and the scalar and AVX512 lookup
functions index the root table with the VRF ID when more than one VRF
is configured. The other algorithms reject max_vrfs >= 2.
Add a VRF mode to test-fib, enabled by the new "-V <number of VRFs>"
option or by a routes or lookup file with a leading VRF field
("<vrf> <prefix>/<depth> <nexthop>", "<vrf> <address>"), the format
being detected from the number of fields. The FIB is created with that
many VRFs and exercised through the VRF API: generated routes are
spread randomly over the VRFs and each generated lookup address is
looked up in the VRF of the route it is derived from (or in a random
VRF for the random ones), -w and -u dump the VRF formats, and the
comparison with LPM (-c) uses one LPM table per VRF, demultiplexing
every burst by VRF as an application would do.
A multi-VRF deployment usually has a full view in VRF 0 and much smaller
tables in the other VRFs, which a random VRF per route does not
reproduce. The "--vrfconf" option, a list of "<vrf> <n>%" items, gives
every listed VRF a random n% of the VRF 0 routes (read from the file or
generated), with their next hops. The routes of the other VRFs are
appended to the table, so that the lookup addresses derived from the
routes are spread over the VRFs like the routes are.
Signed-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
---
app/test-fib/main.c | 557 +++++++++++++++----
app/test/test_fib.c | 714 ++++++++++++++++++++++++-
app/test/test_fib_perf.c | 350 +++---------
app/test/test_fib_routes.h | 282 ++++++++++
doc/guides/prog_guide/fib_lib.rst | 33 +-
doc/guides/rel_notes/release_26_11.rst | 15 +
lib/fib/dir24_8.c | 4 +-
lib/fib/dir24_8.h | 39 +-
lib/fib/dir24_8_avx512.c | 18 +-
lib/fib/dir24_8_avx512.h | 32 +-
lib/fib/dir24_8_rvv.c | 3 +-
lib/fib/dir24_8_rvv.h | 16 +-
lib/fib/dir8_16_8.c | 127 +++--
lib/fib/dir8_16_8.h | 63 ++-
lib/fib/dir8_16_8_avx512.c | 79 ++-
lib/fib/dir8_16_8_avx512.h | 16 +-
lib/fib/rte_fib.c | 192 +++++--
lib/fib/rte_fib.h | 98 +++-
18 files changed, 2077 insertions(+), 561 deletions(-)
create mode 100644 app/test/test_fib_routes.h
diff --git a/app/test-fib/main.c b/app/test-fib/main.c
index be1ba20277..96efe0d618 100644
--- a/app/test-fib/main.c
+++ b/app/test-fib/main.c
@@ -2,7 +2,9 @@
* Copyright(c) 2019 Intel Corporation
*/
+#include <ctype.h>
#include <getopt.h>
+#include <stdbool.h>
#include <stdlib.h>
#include <string.h>
@@ -54,16 +56,33 @@
#define SHUFFLE_FLAG (1 << 7)
#define DRY_RUN_FLAG (1 << 8)
#define ENT_SZ_FLAG (1 << 9)
+#define VRF_FLAG (1 << 10)
+
+/* Maximum number of VRFs derived from VRF 0 with --vrfconf. */
+#define MAX_VRF_CONF 64
static char *distrib_string;
static char line[LINE_MAX];
enum {
+ RT_VRF,
RT_PREFIX,
RT_NEXTHOP,
RT_NUM
};
+enum {
+ LK_VRF,
+ LK_IP,
+ LK_NUM
+};
+
+/* A VRF holding a random share of the VRF 0 routes. */
+struct vrf_conf {
+ uint16_t vrf;
+ uint8_t pct;
+};
+
#ifndef NIPQUAD
#define NIPQUAD_FMT "%u.%u.%u.%u"
#define NIPQUAD(addr) \
@@ -81,12 +100,20 @@ static struct {
const char *lookup_ips_file_s;
void *rt;
void *lookup_tbl;
- uint32_t nb_routes;
+ uint16_t *lookup_vrfs; /**< Per lookup address VRF (VRF mode). */
+ struct vrf_conf vrf_conf[MAX_VRF_CONF];
+ unsigned int nb_vrf_conf;
+ uint32_t nb_routes; /**< All the routes, VRF copies included. */
+ uint32_t nb_base_routes; /**< Routes read from the file or generated. */
uint32_t nb_lookup_ips;
uint32_t nb_lookup_ips_rnd;
uint32_t nb_routes_per_depth[128 + 1];
uint32_t flags;
uint32_t tbl8;
+ uint16_t nb_vrfs; /**< Number of VRFs, 0 = VRF mode off. */
+ uint32_t *nb_vrf_routes; /**< Number of routes of every VRF. */
+ bool routes_file_vrf; /**< Routes file has a VRF field. */
+ bool lookup_file_vrf; /**< Lookup file has a VRF field. */
uint8_t ent_sz;
uint8_t rnd_lookup_ips_ratio;
uint8_t print_fract;
@@ -100,6 +127,7 @@ static struct {
.nb_routes_per_depth = {0},
.flags = FIB_V4_DIR_TYPE,
.tbl8 = DEFAULT_LPM_TBL8,
+ .nb_vrfs = 0,
.ent_sz = 4,
.rnd_lookup_ips_ratio = 0,
.print_fract = 10,
@@ -109,15 +137,39 @@ static struct {
struct rt_rule_4 {
uint32_t addr;
uint8_t depth;
+ uint16_t vrf;
uint64_t nh;
};
struct rt_rule_6 {
struct rte_ipv6_addr addr;
uint8_t depth;
+ uint16_t vrf;
uint64_t nh;
};
+static inline bool
+vrf_enabled(void)
+{
+ return (config.flags & VRF_FLAG) != 0;
+}
+
+static inline uint16_t
+rnd_vrf(void)
+{
+ return vrf_enabled() ? rte_rand() % config.nb_vrfs : 0;
+}
+
+/* Generated routes go to VRF 0 when the other VRFs are derived from it. */
+static inline uint16_t
+gen_vrf(void)
+{
+ uint16_t vrf = (config.nb_vrf_conf != 0) ? 0 : rnd_vrf();
+
+ config.nb_vrf_routes[vrf]++;
+ return vrf;
+}
+
static uint64_t
get_rnd_rng(uint64_t l, uint64_t u)
{
@@ -276,17 +328,9 @@ shuffle_rt_4(struct rt_rule_4 *rt, int n)
for (i = 0; i < n; i++) {
j = rte_rand() % n;
- tmp.addr = rt[i].addr;
- tmp.depth = rt[i].depth;
- tmp.nh = rt[i].nh;
-
- rt[i].addr = rt[j].addr;
- rt[i].depth = rt[j].depth;
- rt[i].nh = rt[j].nh;
-
- rt[j].addr = tmp.addr;
- rt[j].depth = tmp.depth;
- rt[j].nh = tmp.nh;
+ tmp = rt[i];
+ rt[i] = rt[j];
+ rt[j] = tmp;
}
}
@@ -298,17 +342,9 @@ shuffle_rt_6(struct rt_rule_6 *rt, int n)
for (i = 0; i < n; i++) {
j = rte_rand() % n;
- tmp.addr = rt[i].addr;
- tmp.depth = rt[i].depth;
- tmp.nh = rt[i].nh;
-
- rt[i].addr = rt[j].addr;
- rt[i].depth = rt[j].depth;
- rt[i].nh = rt[j].nh;
-
- rt[j].addr = tmp.addr;
- rt[j].depth = tmp.depth;
- rt[j].nh = tmp.nh;
+ tmp = rt[i];
+ rt[i] = rt[j];
+ rt[j] = tmp;
}
}
@@ -320,6 +356,7 @@ gen_random_rt_4(struct rt_rule_4 *rt, int nh_sz)
if (config.nb_routes_per_depth[0] != 0) {
rt[k].addr = 0;
rt[k].depth = 0;
+ rt[k].vrf = gen_vrf();
rt[k++].nh = rte_rand() & get_max_nh(nh_sz);
}
@@ -333,6 +370,7 @@ gen_random_rt_4(struct rt_rule_4 *rt, int nh_sz)
(uint64_t)(edge + step));
rt[k].addr = rnd_val << (32 - i);
rt[k].depth = i;
+ rt[k].vrf = gen_vrf();
rt[k].nh = rte_rand() & get_max_nh(nh_sz);
}
}
@@ -358,6 +396,7 @@ gen_random_rt_6(struct rt_rule_6 *rt, int nh_sz)
if (config.nb_routes_per_depth[0] != 0) {
memset(&rt[k].addr, 0, 16);
rt[k].depth = 0;
+ rt[k].vrf = gen_vrf();
rt[k++].nh = rte_rand() & get_max_nh(nh_sz);
}
@@ -375,6 +414,7 @@ gen_random_rt_6(struct rt_rule_6 *rt, int nh_sz)
complete_v6_addr((uint32_t *)&rt[k].addr,
rnd, a);
rt[k].depth = (a * 32) + i;
+ rt[k].vrf = gen_vrf();
rt[k].nh = rte_rand() & get_max_nh(nh_sz);
}
}
@@ -415,19 +455,24 @@ gen_rnd_lookup_tbl(int af)
i++, j = (j + 1) % config.nb_routes) {
if ((rte_rand() % 100) < config.rnd_lookup_ips_ratio) {
tbl4[i] = rte_rand();
+ config.lookup_vrfs[i] = rnd_vrf();
config.nb_lookup_ips_rnd++;
- } else
+ } else {
tbl4[i] = rt4[j].addr | (rte_rand() &
((1ULL << (32 - rt4[j].depth)) - 1));
+ config.lookup_vrfs[i] = rt4[j].vrf;
+ }
}
} else {
for (i = 0, j = 0; i < config.nb_lookup_ips;
i++, j = (j + 1) % config.nb_routes) {
if ((rte_rand() % 100) < config.rnd_lookup_ips_ratio) {
set_rnd_ipv6(&tbl6[i], &rt6[j].addr, 0);
+ config.lookup_vrfs[i] = rnd_vrf();
config.nb_lookup_ips_rnd++;
} else {
set_rnd_ipv6(&tbl6[i], &rt6[j].addr, rt6[j].depth);
+ config.lookup_vrfs[i] = rt6[j].vrf;
}
}
}
@@ -459,33 +504,159 @@ _inet_net_pton(int af, char *prefix, void *addr)
return depth;
}
+/*
+ * Split a line into its fields: "[<vrf>] <prefix>/<depth> <nexthop>" for
+ * routes, "[<vrf>] <address>" for lookups. The VRF field is validated
+ * against the number of VRFs.
+ */
static int
-parse_rt_4(FILE *f)
+parse_fields(char *s, char **in, int nb_fields, bool has_vrf, uint16_t *vrf)
{
- int ret, i, j = 0;
- char *s, *sp, *in[RT_NUM];
static const char *dlm = " \t\n";
- int string_tok_nb = RTE_DIM(in);
+ char *sp, *tok;
+ int i, first = has_vrf ? 0 : 1;
+
+ for (i = first; i < nb_fields; i++) {
+ tok = strtok_r(s, dlm, &sp);
+ if (tok == NULL)
+ return -EINVAL;
+ in[i] = tok;
+ s = NULL;
+ }
+ if (strtok_r(NULL, dlm, &sp) != NULL)
+ return -EINVAL;
+
+ *vrf = 0;
+ if (has_vrf) {
+ tok = in[0];
+ GET_CB_FIELD(tok, *vrf, 0,
+ (unsigned long)RTE_MAX(config.nb_vrfs, 1) - 1, 0);
+ }
+ return 0;
+}
+
+/*
+ * Count the lines of a file and detect its format: a line with one more
+ * field than the plain format starts with a VRF, the biggest one is returned.
+ */
+static uint32_t
+scan_file(FILE *f, int nb_plain_fields, bool *has_vrf, uint16_t *max_vrf)
+{
+ uint32_t nb_lines = 0;
+ int nb_fields;
+ char *s;
+
+ *has_vrf = false;
+ *max_vrf = 0;
+ while (fgets(line, sizeof(line), f) != NULL) {
+ nb_fields = 0;
+ for (s = line; *s != '\0'; s++) {
+ if (!isspace((unsigned char)*s) &&
+ (s == line || isspace((unsigned char)s[-1])))
+ nb_fields++;
+ }
+ if (nb_fields == 0)
+ continue;
+ if (nb_lines == 0)
+ *has_vrf = (nb_fields == nb_plain_fields + 1);
+ if (*has_vrf) {
+ unsigned long vrf;
+ char *end;
+
+ errno = 0;
+ vrf = strtoul(line, &end, 0);
+ if (errno != 0 || end == line || vrf >= UINT16_MAX)
+ rte_exit(-EINVAL, "Bad VRF at line %u\n",
+ nb_lines + 1);
+ *max_vrf = RTE_MAX(*max_vrf, (uint16_t)vrf);
+ }
+ nb_lines++;
+ }
+ rewind(f);
+ return nb_lines;
+}
+
+/* Number of routes of a VRF derived from VRF 0 with --vrfconf. */
+static uint32_t
+nb_vrf_conf_routes(const struct vrf_conf *vc)
+{
+ return RTE_MAX((uint64_t)config.nb_base_routes * vc->pct / 100, 1U);
+}
+
+/*
+ * Every VRF of --vrfconf gets a random subset of the VRF 0 routes, with their
+ * next hops, appended after the base routes.
+ */
+static void
+gen_vrf_copies(int af)
+{
+ struct rt_rule_4 *rt4 = config.rt;
+ struct rt_rule_6 *rt6 = config.rt;
+ uint64_t *copied;
+ uint32_t i, j, k, n;
+ unsigned int c;
+
+ if (config.nb_vrf_conf == 0)
+ return;
+
+ /* Bitmap of the base routes already copied to the current VRF. */
+ copied = malloc(RTE_ALIGN_CEIL(config.nb_base_routes, 64) / 8);
+ if (copied == NULL)
+ rte_exit(-ENOMEM, "Can not alloc the VRF copies bitmap\n");
+
+ k = config.nb_base_routes;
+ for (c = 0; c < config.nb_vrf_conf; c++) {
+ uint16_t vrf = config.vrf_conf[c].vrf;
+
+ n = RTE_MIN(nb_vrf_conf_routes(&config.vrf_conf[c]),
+ config.nb_vrf_routes[0]);
+ memset(copied, 0, RTE_ALIGN_CEIL(config.nb_base_routes, 64) / 8);
+ for (i = 0; i < n; i++, k++) {
+ /* A random VRF 0 route not copied to this VRF yet. */
+ do {
+ j = rte_rand() % config.nb_base_routes;
+ } while ((copied[j / 64] & RTE_BIT64(j % 64)) != 0 ||
+ ((af == AF_INET) ? rt4[j].vrf : rt6[j].vrf) != 0);
+ copied[j / 64] |= RTE_BIT64(j % 64);
+ if (af == AF_INET) {
+ rt4[k] = rt4[j];
+ rt4[k].vrf = vrf;
+ config.nb_routes_per_depth[rt4[k].depth]++;
+ } else {
+ rt6[k] = rt6[j];
+ rt6[k].vrf = vrf;
+ config.nb_routes_per_depth[rt6[k].depth]++;
+ }
+ }
+ config.nb_vrf_routes[vrf] += n;
+ }
+ config.nb_routes = k;
+ free(copied);
+}
+
+static int
+parse_rt_4(FILE *f)
+{
+ int ret, j = 0;
+ char *in[RT_NUM];
struct rt_rule_4 *rt;
rt = (struct rt_rule_4 *)config.rt;
while (fgets(line, sizeof(line), f) != NULL) {
- s = line;
- for (i = 0; i != string_tok_nb; i++) {
- in[i] = strtok_r(s, dlm, &sp);
- if (in[i] == NULL)
- return -EINVAL;
- s = NULL;
- }
+ ret = parse_fields(line, in, RT_NUM, config.routes_file_vrf,
+ &rt[j].vrf);
+ if (ret != 0)
+ return ret;
ret = _inet_net_pton(AF_INET, in[RT_PREFIX], &rt[j].addr);
- if (ret == -1)
- return -errno;
+ if (ret < 0)
+ return ret;
rt[j].addr = rte_be_to_cpu_32(rt[j].addr);
rt[j].depth = ret;
config.nb_routes_per_depth[ret]++;
+ config.nb_vrf_routes[rt[j].vrf]++;
GET_CB_FIELD(in[RT_NEXTHOP], rt[j].nh, 0,
UINT32_MAX, 0);
j++;
@@ -496,22 +667,17 @@ parse_rt_4(FILE *f)
static int
parse_rt_6(FILE *f)
{
- int ret, i, j = 0;
- char *s, *sp, *in[RT_NUM];
- static const char *dlm = " \t\n";
- int string_tok_nb = RTE_DIM(in);
+ int ret, j = 0;
+ char *in[RT_NUM];
struct rt_rule_6 *rt;
rt = (struct rt_rule_6 *)config.rt;
while (fgets(line, sizeof(line), f) != NULL) {
- s = line;
- for (i = 0; i != string_tok_nb; i++) {
- in[i] = strtok_r(s, dlm, &sp);
- if (in[i] == NULL)
- return -EINVAL;
- s = NULL;
- }
+ ret = parse_fields(line, in, RT_NUM, config.routes_file_vrf,
+ &rt[j].vrf);
+ if (ret != 0)
+ return ret;
ret = _inet_net_pton(AF_INET6, in[RT_PREFIX], &rt[j].addr);
if (ret < 0)
@@ -519,6 +685,7 @@ parse_rt_6(FILE *f)
rt[j].depth = ret;
config.nb_routes_per_depth[ret]++;
+ config.nb_vrf_routes[rt[j].vrf]++;
GET_CB_FIELD(in[RT_NEXTHOP], rt[j].nh, 0,
UINT32_MAX, 0);
j++;
@@ -534,21 +701,29 @@ parse_lookup(FILE *f, int af)
uint32_t i;
uint8_t *tbl = (uint8_t *)config.lookup_tbl;
int step = (af == AF_INET) ? 4 : 16;
- char *s;
+ char *in[LK_NUM];
/* Only the addresses of the full bursts are read. */
for (i = 0; i < config.nb_lookup_ips &&
fgets(line, sizeof(line), f) != NULL; i++) {
- s = strtok(line, " \t\n");
- if (s == NULL)
- return -EINVAL;
- ret = inet_pton(af, s, &tbl[(size_t)i * step]);
+ ret = parse_fields(line, in, LK_NUM, config.lookup_file_vrf,
+ &config.lookup_vrfs[i]);
+ if (ret != 0)
+ return ret;
+ ret = inet_pton(af, in[LK_IP], &tbl[(size_t)i * step]);
if (ret != 1)
return -EINVAL;
}
return 0;
}
+static void
+fprint_vrf(FILE *f, uint16_t vrf)
+{
+ if (vrf_enabled())
+ fprintf(f, "%u ", vrf);
+}
+
static int
dump_lookup(int af)
{
@@ -564,11 +739,16 @@ dump_lookup(int af)
}
if (af == AF_INET) {
- for (i = 0; i < config.nb_lookup_ips; i++)
+ for (i = 0; i < config.nb_lookup_ips; i++) {
+ fprint_vrf(f, config.lookup_vrfs[i]);
fprintf(f, NIPQUAD_FMT"\n", NIPQUAD(tbl4[i]));
+ }
} else {
- for (i = 0; i < config.nb_lookup_ips; i++)
- fprintf(f, RTE_IPV6_ADDR_FMT"\n", RTE_IPV6_ADDR_SPLIT(&tbl6[i * 16]));
+ for (i = 0; i < config.nb_lookup_ips; i++) {
+ fprint_vrf(f, config.lookup_vrfs[i]);
+ fprintf(f, RTE_IPV6_ADDR_FMT"\n",
+ RTE_IPV6_ADDR_SPLIT(&tbl6[i]));
+ }
}
fclose(f);
return 0;
@@ -603,6 +783,14 @@ print_config(void)
"Random lookup ips: %u\n",
config.nb_lookup_ips, config.rnd_lookup_ips_ratio,
config.nb_lookup_ips_rnd);
+ if (vrf_enabled()) {
+ uint32_t j;
+
+ fprintf(stdout, "VRFs: %u, routes per VRF:", config.nb_vrfs);
+ for (j = 0; j < RTE_MIN(config.nb_vrfs, 16U); j++)
+ fprintf(stdout, " %u", config.nb_vrf_routes[j]);
+ fprintf(stdout, (config.nb_vrfs > 16) ? " ...\n" : "\n");
+ }
}
static void
@@ -634,6 +822,13 @@ print_usage(void)
"[-e <entry size (valid only for dir and trie fib types): "
"1/2/4/8 (default 4)>]\n"
"[-g <number of tbl8's for dir24_8, dir8_16_8 or trie FIBs>]\n"
+ "[-V <number of VRFs (2 to 65535)>]\n"
+ "[--vrfconf <\",\" separated \"<vrf> <n>%%\" list: every listed "
+ "VRF gets a random n%% of the VRF 0 routes>]\n"
+ "\tVRF mode (valid for the dir8_16_8 FIB) is "
+ "enabled by -V, by --vrfconf or by a routes or lookup file with "
+ "a leading VRF field (\"<vrf> <prefix>/<depth> <nexthop>\", "
+ "\"<vrf> <address>\"), the other files put everything in VRF 0\n"
"[-w <path to the file to dump routing table>]\n"
"[-u <path to the file to dump ip's for lookup>]\n"
"[-v <type of lookup function:"
@@ -693,15 +888,89 @@ check_config(void)
return 0;
}
+/* "<vrf> <percent>%[, ...]": VRFs holding a random share of VRF 0. */
+static int
+parse_vrf_conf(char *s)
+{
+ char *item, *sp, *end;
+ unsigned long vrf, pct;
+ unsigned int i;
+
+ for (item = strtok_r(s, ",", &sp); item != NULL;
+ item = strtok_r(NULL, ",", &sp)) {
+ if (config.nb_vrf_conf == MAX_VRF_CONF)
+ return -EINVAL;
+ errno = 0;
+ vrf = strtoul(item, &end, 0);
+ if (errno != 0 || end == item || vrf == 0 || vrf >= UINT16_MAX)
+ return -EINVAL;
+ item = end;
+ pct = strtoul(item, &end, 0);
+ if (errno != 0 || end == item || pct == 0 || pct > 100)
+ return -EINVAL;
+ if (*end == '%')
+ end++;
+ while (isspace((unsigned char)*end))
+ end++;
+ if (*end != '\0')
+ return -EINVAL;
+ for (i = 0; i < config.nb_vrf_conf; i++) {
+ if (config.vrf_conf[i].vrf == vrf)
+ return -EINVAL;
+ }
+ config.vrf_conf[config.nb_vrf_conf].vrf = vrf;
+ config.vrf_conf[config.nb_vrf_conf].pct = pct;
+ config.nb_vrf_conf++;
+ }
+ return (config.nb_vrf_conf != 0) ? 0 : -EINVAL;
+}
+
+/*
+ * VRF mode is enabled by the files with a VRF field, by --vrfconf and by -V,
+ * which can only raise the number of VRFs they need.
+ */
+static void
+setup_vrfs(uint16_t max_file_vrf)
+{
+ uint32_t nb_vrfs = (uint32_t)max_file_vrf + 1;
+ unsigned int i;
+
+ for (i = 0; i < config.nb_vrf_conf; i++)
+ nb_vrfs = RTE_MAX(nb_vrfs, (uint32_t)config.vrf_conf[i].vrf + 1);
+ if (config.nb_vrfs != 0 && config.nb_vrfs < nb_vrfs)
+ rte_exit(-EINVAL, "-V %u is less than the %u VRFs used\n",
+ config.nb_vrfs, nb_vrfs);
+ if (config.nb_vrfs == 0 && nb_vrfs > 1)
+ config.nb_vrfs = nb_vrfs;
+ if (config.nb_vrfs == 0)
+ return;
+ if ((config.flags & IPV6_FLAG) ||
+ (config.flags & FIB_TYPE_MASK) != FIB_V4_DIR_8_16_8_TYPE)
+ rte_exit(-EINVAL, "VRFs are only supported by the dir8_16_8 FIB "
+ "type\n");
+ config.flags |= VRF_FLAG;
+}
+
static void
parse_opts(int argc, char **argv)
{
- int opt;
+ static const struct option long_opts[] = {
+ { "vrfconf", required_argument, NULL, 0 },
+ { NULL, 0, NULL, 0 }
+ };
+ int opt, opt_idx;
char *endptr;
+ unsigned long val;
- while ((opt = getopt(argc, argv, "f:t:n:d:l:r:c6ab:e:g:w:u:sv:")) !=
- -1) {
+ while ((opt = getopt_long(argc, argv, "f:t:n:d:l:r:c6ab:e:g:w:u:sv:V:",
+ long_opts, &opt_idx)) != -1) {
switch (opt) {
+ case 0:
+ if (parse_vrf_conf(optarg) != 0) {
+ print_usage();
+ rte_exit(-EINVAL, "Invalid option --vrfconf\n");
+ }
+ break;
case 'f':
config.routes_file = optarg;
break;
@@ -795,6 +1064,16 @@ parse_opts(int argc, char **argv)
rte_exit(-EINVAL, "Invalid option -g\n");
}
break;
+ case 'V':
+ errno = 0;
+ val = strtoul(optarg, &endptr, 10);
+ if ((errno != 0) || (*endptr != '\0') || (val < 2) ||
+ (val > UINT16_MAX)) {
+ print_usage();
+ rte_exit(-EINVAL, "Invalid option -V\n");
+ }
+ config.nb_vrfs = val;
+ break;
case 'v':
if ((strcmp(optarg, "s1") == 0) ||
(strcmp(optarg, "s") == 0)) {
@@ -831,9 +1110,11 @@ dump_rt_4(struct rt_rule_4 *rt)
return -1;
}
- for (i = 0; i < config.nb_routes; i++)
+ for (i = 0; i < config.nb_routes; i++) {
+ fprint_vrf(f, rt[i].vrf);
fprintf(f, NIPQUAD_FMT"/%d %"PRIu64"\n", NIPQUAD(rt[i].addr),
rt[i].depth, rt[i].nh);
+ }
fclose(f);
return 0;
@@ -846,6 +1127,43 @@ print_depth_err(void)
"-d 0:0 option or remove /0 prefix from routes file\n");
}
+/* Number of LPM tables needed for the comparison: one per VRF. */
+static inline unsigned int
+nb_lpm_tables(void)
+{
+ return vrf_enabled() ? config.nb_vrfs : 1;
+}
+
+/*
+ * Per-VRF LPM lookup of a burst: the addresses are demultiplexed by VRF,
+ * looked up in bulk in the matching LPM table and the results put back in
+ * place. This is what an application would do with one LPM table per VRF.
+ */
+static void
+lpm_vrf_lookup_burst(struct rte_lpm **lpm, const uint16_t *vrfs,
+ const uint32_t *ips, uint32_t *nhs, unsigned int n)
+{
+ uint32_t tmp_ips[BURST_SZ], tmp_nhs[BURST_SZ];
+ unsigned int idx[BURST_SZ];
+ uint64_t done = 0;
+ unsigned int i, j, k;
+
+ for (i = 0; i < n; i++) {
+ if (done & (1ULL << i))
+ continue;
+ for (j = i, k = 0; j < n; j++) {
+ if (vrfs[j] != vrfs[i])
+ continue;
+ done |= 1ULL << j;
+ idx[k] = j;
+ tmp_ips[k++] = ips[j];
+ }
+ rte_lpm_lookup_bulk(lpm[vrfs[i]], tmp_ips, tmp_nhs, k);
+ for (j = 0; j < k; j++)
+ nhs[idx[j]] = tmp_nhs[j];
+ }
+}
+
static int
select_lookup_fn_v4(struct rte_fib *fib, enum rte_fib_type type)
{
@@ -885,11 +1203,13 @@ run_v4(void)
struct rt_rule_4 *rt;
uint32_t i, j, k;
int ret = 0;
- struct rte_lpm *lpm = NULL;
+ struct rte_lpm **lpm = NULL;
struct rte_lpm_config lpm_conf;
uint32_t *tbl4 = config.lookup_tbl;
+ const uint16_t *vrfs = vrf_enabled() ? config.lookup_vrfs : NULL;
uint64_t fib_nh[BURST_SZ];
uint32_t lpm_nh[BURST_SZ];
+ char name[32];
rt = (struct rt_rule_4 *)config.rt;
@@ -905,8 +1225,9 @@ run_v4(void)
conf.type = get_fib_type();
conf.default_nh = def_nh;
- conf.max_routes = config.nb_routes * 2;
+ conf.max_routes = config.nb_base_routes * 2;
conf.rib_ext_sz = 0;
+ conf.max_vrfs = config.nb_vrfs;
if (conf.type == RTE_FIB_DIR24_8) {
conf.dir24_8.nh_sz = rte_ctz32(config.ent_sz);
conf.dir24_8.num_tbl8 = RTE_MIN(config.tbl8,
@@ -914,11 +1235,12 @@ run_v4(void)
} else if (conf.type == RTE_FIB_DIR8_16_8) {
/*
* DIR8_16_8 only supports 8 byte next hops. The first stride
- * (8 bits) needs at most one second level group per /8, so 256
- * groups always suffice. The third level (last 8 bits) groups
- * are sized like the DIR24_8 tbl8s.
+ * (8 bits) needs at most one second level group per /8 and
+ * VRF, so 256 groups per VRF always suffice. The third level
+ * (last 8 bits) groups are sized like the DIR24_8 tbl8s.
*/
- conf.dir8_16_8.num_tbl16 = DIR8_16_8_MAX_TBL16;
+ conf.dir8_16_8.num_tbl16 = DIR8_16_8_MAX_TBL16 *
+ RTE_MAX(config.nb_vrfs, 1);
conf.dir8_16_8.num_tbl8 = config.tbl8;
}
@@ -939,8 +1261,8 @@ run_v4(void)
for (k = config.print_fract, i = 0; k > 0; k--) {
start = rte_rdtsc_precise();
for (j = 0; j < (config.nb_routes - i) / k; j++) {
- ret = rte_fib_add(fib, rt[i + j].addr, rt[i + j].depth,
- rt[i + j].nh);
+ ret = rte_fib_vrf_add(fib, rt[i + j].vrf, rt[i + j].addr,
+ rt[i + j].depth, rt[i + j].nh);
if (unlikely(ret != 0)) {
printf("Can not add a route to FIB, err %d\n",
ret);
@@ -957,16 +1279,23 @@ run_v4(void)
lpm_conf.number_tbl8s = RTE_MAX(conf.dir24_8.num_tbl8,
config.tbl8);
- lpm = rte_lpm_create("test_lpm", -1, &lpm_conf);
- if (lpm == NULL) {
- printf("Can not alloc LPM, err %d\n", rte_errno);
- return -rte_errno;
+ lpm = calloc(nb_lpm_tables(), sizeof(*lpm));
+ if (lpm == NULL)
+ return -ENOMEM;
+ for (i = 0; i < nb_lpm_tables(); i++) {
+ snprintf(name, sizeof(name), "test_lpm%u", i);
+ lpm[i] = rte_lpm_create(name, -1, &lpm_conf);
+ if (lpm[i] == NULL) {
+ printf("Can not alloc LPM, err %d\n", rte_errno);
+ return -rte_errno;
+ }
}
for (k = config.print_fract, i = 0; k > 0; k--) {
start = rte_rdtsc_precise();
for (j = 0; j < (config.nb_routes - i) / k; j++) {
- ret = rte_lpm_add(lpm, rt[i + j].addr,
- rt[i + j].depth, rt[i + j].nh);
+ ret = rte_lpm_add(lpm[rt[i + j].vrf],
+ rt[i + j].addr, rt[i + j].depth,
+ rt[i + j].nh);
if (ret != 0) {
if (rt[i + j].depth == 0)
print_depth_err();
@@ -984,7 +1313,8 @@ run_v4(void)
acc = 0;
for (i = 0; i < config.nb_lookup_ips; i += BURST_SZ) {
start = rte_rdtsc_precise();
- ret = rte_fib_lookup_bulk(fib, tbl4 + i, fib_nh, BURST_SZ);
+ ret = rte_fib_vrf_lookup_bulk(fib, vrfs ? vrfs + i : NULL,
+ tbl4 + i, fib_nh, BURST_SZ);
acc += rte_rdtsc_precise() - start;
if (ret != 0) {
printf("FIB lookup fails, err %d\n", ret);
@@ -997,8 +1327,12 @@ run_v4(void)
acc = 0;
for (i = 0; i < config.nb_lookup_ips; i += BURST_SZ) {
start = rte_rdtsc_precise();
- ret = rte_lpm_lookup_bulk(lpm, tbl4 + i, lpm_nh,
- BURST_SZ);
+ if (vrfs != NULL)
+ lpm_vrf_lookup_burst(lpm, vrfs + i, tbl4 + i,
+ lpm_nh, BURST_SZ);
+ else
+ ret = rte_lpm_lookup_bulk(lpm[0], tbl4 + i,
+ lpm_nh, BURST_SZ);
acc += rte_rdtsc_precise() - start;
if (ret != 0) {
printf("LPM lookup fails, err %d\n", ret);
@@ -1008,8 +1342,14 @@ run_v4(void)
printf("AVG LPM lookup %.1f\n", (double)acc / (double)i);
for (i = 0; i < config.nb_lookup_ips; i += BURST_SZ) {
- rte_fib_lookup_bulk(fib, tbl4 + i, fib_nh, BURST_SZ);
- rte_lpm_lookup_bulk(lpm, tbl4 + i, lpm_nh, BURST_SZ);
+ rte_fib_vrf_lookup_bulk(fib, vrfs ? vrfs + i : NULL,
+ tbl4 + i, fib_nh, BURST_SZ);
+ if (vrfs != NULL)
+ lpm_vrf_lookup_burst(lpm, vrfs + i, tbl4 + i,
+ lpm_nh, BURST_SZ);
+ else
+ rte_lpm_lookup_bulk(lpm[0], tbl4 + i, lpm_nh,
+ BURST_SZ);
for (j = 0; j < BURST_SZ; j++) {
struct rte_lpm_tbl_entry *tbl;
tbl = (struct rte_lpm_tbl_entry *)&lpm_nh[j];
@@ -1027,7 +1367,8 @@ run_v4(void)
for (k = config.print_fract, i = 0; k > 0; k--) {
start = rte_rdtsc_precise();
for (j = 0; j < (config.nb_routes - i) / k; j++)
- rte_fib_delete(fib, rt[i + j].addr, rt[i + j].depth);
+ rte_fib_vrf_delete(fib, rt[i + j].vrf, rt[i + j].addr,
+ rt[i + j].depth);
printf("AVG FIB delete %"PRIu64"\n",
(rte_rdtsc_precise() - start) / j);
@@ -1038,7 +1379,7 @@ run_v4(void)
for (k = config.print_fract, i = 0; k > 0; k--) {
start = rte_rdtsc_precise();
for (j = 0; j < (config.nb_routes - i) / k; j++)
- rte_lpm_delete(lpm, rt[i + j].addr,
+ rte_lpm_delete(lpm[rt[i + j].vrf], rt[i + j].addr,
rt[i + j].depth);
printf("AVG LPM delete %"PRIu64"\n",
@@ -1063,9 +1404,9 @@ dump_rt_6(struct rt_rule_6 *rt)
}
for (i = 0; i < config.nb_routes; i++) {
- fprintf(f, RTE_IPV6_ADDR_FMT"/%d %"PRIu64"\n", RTE_IPV6_ADDR_SPLIT(&rt[i].addr),
- rt[i].depth, rt[i].nh);
-
+ fprint_vrf(f, rt[i].vrf);
+ fprintf(f, RTE_IPV6_ADDR_FMT"/%d %"PRIu64"\n",
+ RTE_IPV6_ADDR_SPLIT(&rt[i].addr), rt[i].depth, rt[i].nh);
}
fclose(f);
return 0;
@@ -1254,6 +1595,8 @@ main(int argc, char **argv)
int ret, af, rt_ent_sz, lookup_ent_sz;
FILE *fr = NULL;
FILE *fl = NULL;
+ uint16_t max_vrf, max_file_vrf = 0;
+ unsigned int i;
uint8_t depth_lim;
ret = rte_eal_init(argc, argv);
@@ -1284,14 +1627,9 @@ main(int argc, char **argv)
rte_exit(-errno, "Can not open file with routes %s\n",
config.routes_file);
- config.nb_routes = 0;
- while (fgets(line, sizeof(line), fr) != NULL)
- config.nb_routes++;
-
- if (config.nb_routes < config.print_fract)
- config.print_fract = config.nb_routes;
-
- rewind(fr);
+ config.nb_routes = scan_file(fr, RT_NUM - 1,
+ &config.routes_file_vrf, &max_vrf);
+ max_file_vrf = RTE_MAX(max_file_vrf, max_vrf);
}
/* Count number of ip's in file*/
@@ -1301,12 +1639,24 @@ main(int argc, char **argv)
rte_exit(-errno, "Can not open file with ip's %s\n",
config.lookup_ips_file);
- config.nb_lookup_ips = 0;
- while (fgets(line, sizeof(line), fl) != NULL)
- config.nb_lookup_ips++;
- rewind(fl);
+ config.nb_lookup_ips = scan_file(fl, LK_NUM - 1,
+ &config.lookup_file_vrf, &max_vrf);
+ max_file_vrf = RTE_MAX(max_file_vrf, max_vrf);
}
+ setup_vrfs(max_file_vrf);
+ config.nb_vrf_routes = calloc(RTE_MAX(config.nb_vrfs, 1),
+ sizeof(*config.nb_vrf_routes));
+ if (config.nb_vrf_routes == NULL)
+ rte_exit(-ENOMEM, "Can not alloc the per VRF route counts\n");
+
+ /* The VRFs of --vrfconf hold copies of the VRF 0 routes. */
+ config.nb_base_routes = config.nb_routes;
+ for (i = 0; i < config.nb_vrf_conf; i++)
+ config.nb_routes += nb_vrf_conf_routes(&config.vrf_conf[i]);
+ if (config.nb_routes < config.print_fract)
+ config.print_fract = config.nb_routes;
+
/* Lookups are done in full bursts, the leftovers are dropped. */
config.nb_lookup_ips = RTE_ALIGN_FLOOR(config.nb_lookup_ips, BURST_SZ);
if (config.nb_lookup_ips == 0)
@@ -1324,14 +1674,20 @@ main(int argc, char **argv)
if (config.lookup_tbl == NULL)
rte_exit(-ENOMEM, "Can not alloc lookup table\n");
+ /* Alloc table with the VRF of every ip for lookup (VRF 0 otherwise) */
+ config.lookup_vrfs = rte_zmalloc(NULL, sizeof(uint16_t) *
+ config.nb_lookup_ips, 0);
+ if (config.lookup_vrfs == NULL)
+ rte_exit(-ENOMEM, "Can not alloc lookup VRF table\n");
+
/* Fill routes table */
if (fr == NULL) {
if (distrib_string != NULL)
- ret = parse_distrib(depth_lim, config.nb_routes);
+ ret = parse_distrib(depth_lim, config.nb_base_routes);
else {
uint8_t rpd[129] = {0};
uint32_t nrpd[129] = {0};
- ret = complete_distrib(depth_lim, config.nb_routes,
+ ret = complete_distrib(depth_lim, config.nb_base_routes,
rpd, nrpd);
}
if (ret != 0)
@@ -1341,12 +1697,12 @@ main(int argc, char **argv)
gen_random_rt_4(config.rt,
rte_ctz32(config.ent_sz));
if (config.flags & SHUFFLE_FLAG)
- shuffle_rt_4(config.rt, config.nb_routes);
+ shuffle_rt_4(config.rt, config.nb_base_routes);
} else {
gen_random_rt_6(config.rt,
rte_ctz32(config.ent_sz));
if (config.flags & SHUFFLE_FLAG)
- shuffle_rt_6(config.rt, config.nb_routes);
+ shuffle_rt_6(config.rt, config.nb_base_routes);
}
} else {
if (af == AF_INET)
@@ -1359,6 +1715,7 @@ main(int argc, char **argv)
config.routes_file);
}
}
+ gen_vrf_copies(af);
/* Fill lookup table with ip's*/
if (fl == NULL)
diff --git a/app/test/test_fib.c b/app/test/test_fib.c
index bb8021c818..ce5e48264a 100644
--- a/app/test/test_fib.c
+++ b/app/test/test_fib.c
@@ -8,6 +8,7 @@
#include <stdlib.h>
#include <string.h>
#include <inttypes.h>
+#include <errno.h>
#include <rte_ip.h>
#include <rte_log.h>
@@ -16,6 +17,7 @@
#include <rte_malloc.h>
#include "test.h"
+#include "test_fib_routes.h"
typedef int32_t (*rte_fib_test)(void);
@@ -621,13 +623,479 @@ test_fib_rcu_sync_rw(void)
return status == 0 ? TEST_SUCCESS : TEST_FAILED;
}
+/*
+ * VRF (Virtual Routing and Forwarding) tests for the DIR8_16_8 FIB.
+ * A VRF-capable FIB is created with max_vrfs >= 2; every VRF holds an
+ * independent routing table while sharing the underlying table pools.
+ */
+#define TEST_MAX_VRFS 4
#define TEST_NUM_TBL16 16
#define TEST_NUM_TBL8 64
+#define VRF_BURST_SZ 16
#define DEF_NH 77
+static struct rte_fib *
+vrf_fib_create(const char *name, uint64_t *vrf_default_nh)
+{
+ struct rte_fib_conf cfg = {0};
+ char fib_name[24];
+
+ cfg.type = RTE_FIB_DIR8_16_8;
+ cfg.max_routes = 1000;
+ cfg.default_nh = DEF_NH;
+ cfg.max_vrfs = TEST_MAX_VRFS;
+ cfg.vrf_default_nh = vrf_default_nh;
+ cfg.dir8_16_8.num_tbl16 = TEST_NUM_TBL16;
+ cfg.dir8_16_8.num_tbl8 = TEST_NUM_TBL8;
+
+ snprintf(fib_name, sizeof(fib_name), "%.20s", name);
+ return rte_fib_create(fib_name, SOCKET_ID_ANY, &cfg);
+}
+
+static uint64_t
+vrf_lookup1(struct rte_fib *fib, uint16_t vrf, uint32_t ip)
+{
+ uint32_t ips[1] = { ip };
+ uint16_t vrfs[1] = { vrf };
+ uint64_t nhs[1] = { 0 };
+
+ if (rte_fib_vrf_lookup_bulk(fib, vrfs, ips, nhs, 1) != 0)
+ return UINT64_MAX;
+ return nhs[0];
+}
+
+static int32_t
+test_vrf_create_invalid(void)
+{
+ struct rte_fib *fib;
+ struct rte_fib_conf cfg = {0};
+ const char *name = "vrf_bad";
+
+ cfg.max_routes = 100;
+ cfg.default_nh = 0;
+ cfg.max_vrfs = TEST_MAX_VRFS;
+
+ cfg.type = RTE_FIB_DIR24_8;
+ cfg.dir24_8.nh_sz = RTE_FIB_DIR24_8_4B;
+ cfg.dir24_8.num_tbl8 = 128;
+ fib = rte_fib_create(name, SOCKET_ID_ANY, &cfg);
+ RTE_TEST_ASSERT(fib == NULL, "DIR24_8 must reject max_vrfs >= 2\n");
+
+ cfg.type = RTE_FIB_DUMMY;
+ fib = rte_fib_create(name, SOCKET_ID_ANY, &cfg);
+ RTE_TEST_ASSERT(fib == NULL, "DUMMY must reject max_vrfs >= 2\n");
+
+ cfg.type = RTE_FIB_DIR8_16_8;
+ cfg.dir8_16_8.num_tbl16 = 0;
+ cfg.dir8_16_8.num_tbl8 = TEST_NUM_TBL8;
+ fib = rte_fib_create(name, SOCKET_ID_ANY, &cfg);
+ RTE_TEST_ASSERT(fib == NULL, "num_tbl16 == 0 must fail\n");
+
+ return TEST_SUCCESS;
+}
+
+/*
+ * Same prefixes in each VRF resolve to that VRF's own nexthop, and a delete
+ * in one VRF does not affect the others.
+ */
+static int32_t
+test_vrf_isolation(void)
+{
+ struct rte_fib *fib;
+ uint16_t v;
+ uint64_t nh;
+
+ fib = vrf_fib_create(__func__, NULL);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create VRF FIB\n");
+
+ for (v = 0; v < TEST_MAX_VRFS; v++) {
+ RTE_TEST_ASSERT(rte_fib_vrf_add(fib, v, RTE_IPV4(10, 0, 0, 0),
+ 8, 100 + v) == 0, "add /8 vrf %u\n", v);
+ RTE_TEST_ASSERT(rte_fib_vrf_add(fib, v, RTE_IPV4(192, 168, 0, 0),
+ 16, 200 + v) == 0, "add /16 vrf %u\n", v);
+ RTE_TEST_ASSERT(rte_fib_vrf_add(fib, v, RTE_IPV4(10, 1, 1, 0),
+ 24, 300 + v) == 0, "add /24 vrf %u\n", v);
+ }
+
+ for (v = 0; v < TEST_MAX_VRFS; v++) {
+ nh = vrf_lookup1(fib, v, RTE_IPV4(10, 2, 3, 4));
+ RTE_TEST_ASSERT(nh == (uint64_t)(100 + v),
+ "vrf %u /8 lookup got %"PRIu64"\n", v, nh);
+ nh = vrf_lookup1(fib, v, RTE_IPV4(192, 168, 5, 5));
+ RTE_TEST_ASSERT(nh == (uint64_t)(200 + v),
+ "vrf %u /16 lookup got %"PRIu64"\n", v, nh);
+ nh = vrf_lookup1(fib, v, RTE_IPV4(10, 1, 1, 7));
+ RTE_TEST_ASSERT(nh == (uint64_t)(300 + v),
+ "vrf %u /24 lookup got %"PRIu64"\n", v, nh);
+ nh = vrf_lookup1(fib, v, RTE_IPV4(8, 8, 8, 8));
+ RTE_TEST_ASSERT(nh == DEF_NH,
+ "vrf %u default lookup got %"PRIu64"\n", v, nh);
+ }
+
+ /* Delete the /8 from VRF 1 only. */
+ RTE_TEST_ASSERT(rte_fib_vrf_delete(fib, 1, RTE_IPV4(10, 0, 0, 0),
+ 8) == 0, "del /8 vrf 1\n");
+ nh = vrf_lookup1(fib, 1, RTE_IPV4(10, 2, 3, 4));
+ RTE_TEST_ASSERT(nh == DEF_NH, "vrf 1 /8 after delete got %"PRIu64"\n",
+ nh);
+ /* The more specific /24 in VRF 1 still matches. */
+ nh = vrf_lookup1(fib, 1, RTE_IPV4(10, 1, 1, 7));
+ RTE_TEST_ASSERT(nh == 301, "vrf 1 /24 after /8 delete got %"PRIu64"\n",
+ nh);
+ /* Other VRFs are unaffected. */
+ for (v = 0; v < TEST_MAX_VRFS; v++) {
+ if (v == 1)
+ continue;
+ nh = vrf_lookup1(fib, v, RTE_IPV4(10, 2, 3, 4));
+ RTE_TEST_ASSERT(nh == (uint64_t)(100 + v),
+ "vrf %u /8 unexpectedly changed to %"PRIu64"\n", v, nh);
+ }
+
+ rte_fib_free(fib);
+ return TEST_SUCCESS;
+}
+
+/*
+ * The same network owned by a different prefix length in each VRF resolves
+ * independently; a VRF with no matching route returns its default.
+ */
+static int32_t
+test_vrf_overlapping_prefixes(void)
+{
+ struct rte_fib *fib;
+ uint64_t nh;
+
+ fib = vrf_fib_create(__func__, NULL);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create VRF FIB\n");
+
+ RTE_TEST_ASSERT(rte_fib_vrf_add(fib, 0, RTE_IPV4(10, 0, 0, 0), 8, 10)
+ == 0, "add /8 vrf 0\n");
+ RTE_TEST_ASSERT(rte_fib_vrf_add(fib, 1, RTE_IPV4(10, 1, 0, 0), 16, 20)
+ == 0, "add /16 vrf 1\n");
+ RTE_TEST_ASSERT(rte_fib_vrf_add(fib, 2, RTE_IPV4(10, 1, 1, 0), 24, 30)
+ == 0, "add /24 vrf 2\n");
+ /* VRF 3 has no matching route. */
+ nh = vrf_lookup1(fib, 0, RTE_IPV4(10, 1, 1, 5));
+ RTE_TEST_ASSERT(nh == 10, "vrf 0 /8 got %"PRIu64"\n", nh);
+ nh = vrf_lookup1(fib, 1, RTE_IPV4(10, 1, 1, 5));
+ RTE_TEST_ASSERT(nh == 20, "vrf 1 /16 got %"PRIu64"\n", nh);
+ nh = vrf_lookup1(fib, 2, RTE_IPV4(10, 1, 1, 5));
+ RTE_TEST_ASSERT(nh == 30, "vrf 2 /24 got %"PRIu64"\n", nh);
+ nh = vrf_lookup1(fib, 3, RTE_IPV4(10, 1, 1, 5));
+ RTE_TEST_ASSERT(nh == DEF_NH, "vrf 3 miss got %"PRIu64"\n", nh);
+
+ rte_fib_free(fib);
+ return TEST_SUCCESS;
+}
+
+/*
+ * A single bulk lookup carrying an interleaved per-IP VRF array resolves
+ * each address in its own VRF, with every available lookup implementation.
+ */
+static int32_t
+test_vrf_mixed_lookup(void)
+{
+ const enum rte_fib_lookup_type types[] = {
+ RTE_FIB_LOOKUP_DIR8_16_8_SCALAR,
+ RTE_FIB_LOOKUP_DIR8_16_8_VECTOR_AVX512,
+ };
+ struct rte_fib *fib;
+ uint32_t ips[VRF_BURST_SZ];
+ uint16_t vrfs[VRF_BURST_SZ];
+ uint64_t nhs[VRF_BURST_SZ];
+ unsigned int i, t;
+
+ fib = vrf_fib_create(__func__, NULL);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create VRF FIB\n");
+
+ /* Same prefix in every VRF, a distinct nexthop per VRF. */
+ for (i = 0; i < TEST_MAX_VRFS; i++)
+ RTE_TEST_ASSERT(rte_fib_vrf_add(fib, i, RTE_IPV4(20, 0, 0, 0),
+ 8, i + 1) == 0, "add vrf %u\n", i);
+
+ for (i = 0; i < VRF_BURST_SZ; i++) {
+ ips[i] = RTE_IPV4(20, 0, 0, i + 1);
+ vrfs[i] = i % TEST_MAX_VRFS;
+ }
+
+ for (t = 0; t < RTE_DIM(types); t++) {
+ /* The vector lookup needs a 512 bit SIMD width. */
+ if (rte_fib_select_lookup(fib, types[t]) != 0)
+ continue;
+ memset(nhs, 0, sizeof(nhs));
+ RTE_TEST_ASSERT(rte_fib_vrf_lookup_bulk(fib, vrfs, ips, nhs,
+ VRF_BURST_SZ) == 0, "mixed lookup failed\n");
+
+ for (i = 0; i < VRF_BURST_SZ; i++)
+ RTE_TEST_ASSERT(nhs[i] == (i % TEST_MAX_VRFS) + 1,
+ "lookup type %u: pkt %u vrf %u got %"PRIu64"\n",
+ types[t], i, vrfs[i], nhs[i]);
+ }
+
+ rte_fib_free(fib);
+ return TEST_SUCCESS;
+}
+
+/* A lookup miss returns that VRF's configured default nexthop. */
+static int32_t
+test_vrf_default_nh(void)
+{
+ struct rte_fib *fib;
+ uint64_t vrf_def[TEST_MAX_VRFS];
+ uint16_t v;
+ uint64_t nh;
+
+ for (v = 0; v < TEST_MAX_VRFS; v++)
+ vrf_def[v] = 1000 + v;
+
+ fib = vrf_fib_create(__func__, vrf_def);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create VRF FIB\n");
+
+ for (v = 0; v < TEST_MAX_VRFS; v++) {
+ nh = vrf_lookup1(fib, v, RTE_IPV4(8, 8, 8, 8));
+ RTE_TEST_ASSERT(nh == (uint64_t)(1000 + v),
+ "vrf %u default nh got %"PRIu64"\n", v, nh);
+ }
+
+ rte_fib_free(fib);
+ return TEST_SUCCESS;
+}
+
+/* Re-adding an existing prefix updates its nexthop in place, per VRF. */
+static int32_t
+test_vrf_update_route(void)
+{
+ struct rte_fib *fib;
+ uint64_t nh;
+
+ fib = vrf_fib_create(__func__, NULL);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create VRF FIB\n");
+
+ RTE_TEST_ASSERT(rte_fib_vrf_add(fib, 2, RTE_IPV4(40, 0, 0, 0), 8, 5)
+ == 0, "initial add\n");
+ nh = vrf_lookup1(fib, 2, RTE_IPV4(40, 1, 1, 1));
+ RTE_TEST_ASSERT(nh == 5, "initial nh got %"PRIu64"\n", nh);
+
+ /* Re-add the same prefix with a new nexthop. */
+ RTE_TEST_ASSERT(rte_fib_vrf_add(fib, 2, RTE_IPV4(40, 0, 0, 0), 8, 99)
+ == 0, "update add\n");
+ nh = vrf_lookup1(fib, 2, RTE_IPV4(40, 1, 1, 1));
+ RTE_TEST_ASSERT(nh == 99, "updated nh got %"PRIu64"\n", nh);
+
+ /* The update stays within its VRF. */
+ nh = vrf_lookup1(fib, 0, RTE_IPV4(40, 1, 1, 1));
+ RTE_TEST_ASSERT(nh == DEF_NH, "vrf 0 leaked update %"PRIu64"\n", nh);
+
+ rte_fib_free(fib);
+ return TEST_SUCCESS;
+}
+
+/* Out-of-range VRF IDs are rejected by the VRF API. */
+static int32_t
+test_vrf_bounds(void)
+{
+ struct rte_fib *fib;
+ uint32_t ips[1] = { RTE_IPV4(1, 2, 3, 4) };
+ uint16_t vrfs[1];
+ uint64_t nhs[1];
+
+ fib = vrf_fib_create(__func__, NULL);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create VRF FIB\n");
+
+ RTE_TEST_ASSERT(rte_fib_vrf_add(fib, TEST_MAX_VRFS,
+ RTE_IPV4(1, 0, 0, 0), 8, 1) == -EINVAL,
+ "add with out-of-range vrf must fail\n");
+ RTE_TEST_ASSERT(rte_fib_vrf_delete(fib, TEST_MAX_VRFS,
+ RTE_IPV4(1, 0, 0, 0), 8) == -EINVAL,
+ "delete with out-of-range vrf must fail\n");
+ RTE_TEST_ASSERT(rte_fib_vrf_get_rib(fib, TEST_MAX_VRFS) == NULL,
+ "get_rib with out-of-range vrf must return NULL\n");
+
+ RTE_TEST_ASSERT(rte_fib_vrf_add(fib, TEST_MAX_VRFS - 1,
+ RTE_IPV4(1, 0, 0, 0), 8, 1) == 0,
+ "add to the last valid vrf must succeed\n");
+
+ /* A valid per-IP VRF lookup in the last VRF returns the new nexthop. */
+ vrfs[0] = TEST_MAX_VRFS - 1;
+ RTE_TEST_ASSERT(rte_fib_vrf_lookup_bulk(fib, vrfs, ips, nhs, 1) == 0,
+ "valid vrf lookup must succeed\n");
+ RTE_TEST_ASSERT(nhs[0] == 1, "lookup must return the added nexthop\n");
+
+ rte_fib_free(fib);
+ return TEST_SUCCESS;
+}
+
+/* Each VRF exposes a distinct, non-NULL RIB; VRF 0 is the main RIB. */
+static int32_t
+test_vrf_get_rib(void)
+{
+ struct rte_fib *fib;
+ struct rte_rib *ribs[TEST_MAX_VRFS];
+ uint16_t v, w;
+
+ fib = vrf_fib_create(__func__, NULL);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create VRF FIB\n");
+
+ for (v = 0; v < TEST_MAX_VRFS; v++) {
+ ribs[v] = rte_fib_vrf_get_rib(fib, v);
+ RTE_TEST_ASSERT(ribs[v] != NULL, "vrf %u rib is NULL\n", v);
+ }
+ RTE_TEST_ASSERT(rte_fib_vrf_get_rib(fib, 0) == rte_fib_get_rib(fib),
+ "vrf 0 rib must be the main rib\n");
+ for (v = 0; v < TEST_MAX_VRFS; v++)
+ for (w = v + 1; w < TEST_MAX_VRFS; w++)
+ RTE_TEST_ASSERT(ribs[v] != ribs[w],
+ "vrf %u and %u share a rib\n", v, w);
+
+ rte_fib_free(fib);
+ return TEST_SUCCESS;
+}
+
/*
- * Network order lookups with every lookup implementation of DIR8_16_8, with a
- * burst of two vector groups and a leftover.
+ * RCU QSBR can be attached to a VRF FIB in both SYNC and DQ modes, and a
+ * second attach is rejected.
+ */
+static int32_t
+test_vrf_rcu(void)
+{
+ struct rte_fib *fib;
+ struct rte_rcu_qsbr *qsv, *qsv2;
+ struct rte_fib_rcu_config rcu_cfg = {0};
+ size_t sz;
+ int32_t status;
+
+ fib = vrf_fib_create(__func__, NULL);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create VRF FIB\n");
+
+ sz = rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE);
+ qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+ RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+ RTE_TEST_ASSERT(qsv != NULL, "Can not allocate memory for RCU\n");
+ RTE_TEST_ASSERT(rte_rcu_qsbr_init(qsv, RTE_MAX_LCORE) == 0,
+ "Can not initialize RCU\n");
+
+ rcu_cfg.v = qsv;
+ rcu_cfg.mode = RTE_FIB_QSBR_MODE_SYNC;
+ status = rte_fib_rcu_qsbr_add(fib, &rcu_cfg);
+ RTE_TEST_ASSERT(status == 0, "Can not attach RCU to VRF FIB\n");
+
+ /* A second attach must be rejected. */
+ qsv2 = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+ RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+ RTE_TEST_ASSERT(qsv2 != NULL, "Can not allocate memory for RCU\n");
+ rcu_cfg.v = qsv2;
+ status = rte_fib_rcu_qsbr_add(fib, &rcu_cfg);
+ RTE_TEST_ASSERT(status == -EEXIST,
+ "Secondary RCU was mistakenly attached\n");
+
+ rte_fib_free(fib);
+ rte_free(qsv);
+ rte_free(qsv2);
+
+ /* DQ (defer queue) mode must also attach to a VRF FIB. */
+ fib = vrf_fib_create("vrf_rcu_dq", NULL);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create VRF FIB\n");
+ qsv = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+ RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+ RTE_TEST_ASSERT(qsv != NULL, "Can not allocate memory for RCU\n");
+ RTE_TEST_ASSERT(rte_rcu_qsbr_init(qsv, RTE_MAX_LCORE) == 0,
+ "Can not initialize RCU\n");
+ rcu_cfg.v = qsv;
+ rcu_cfg.mode = RTE_FIB_QSBR_MODE_DQ;
+ status = rte_fib_rcu_qsbr_add(fib, &rcu_cfg);
+ RTE_TEST_ASSERT(status == 0, "Can not attach DQ-mode RCU to VRF FIB\n");
+
+ rte_fib_free(fib);
+ rte_free(qsv);
+
+ return TEST_SUCCESS;
+}
+
+/*
+ * RCU sync reader/writer test for a VRF FIB. The reader looks up g_ip via
+ * the legacy bulk API (which targets VRF 0) while the writer churns the
+ * same route in VRF 0 with depth > 24 to force tbl8 group reclamation.
+ */
+static int32_t
+test_vrf_rcu_sync_rw(void)
+{
+ struct rte_fib_conf config = { 0 };
+ struct rte_fib_rcu_config rcu_cfg = {0};
+ size_t sz;
+ int32_t status;
+ uint32_t i, next_hop;
+ uint8_t depth;
+
+ if (rte_lcore_count() < 2) {
+ printf("Not enough cores for %s, expecting at least 2\n",
+ __func__);
+ return TEST_SKIPPED;
+ }
+
+ config.max_routes = MAX_ROUTES;
+ config.default_nh = DEF_NH;
+ config.type = RTE_FIB_DIR8_16_8;
+ config.max_vrfs = TEST_MAX_VRFS;
+ config.dir8_16_8.num_tbl16 = TEST_NUM_TBL16;
+ config.dir8_16_8.num_tbl8 = 1;
+
+ g_fib = rte_fib_create(__func__, SOCKET_ID_ANY, &config);
+ RTE_TEST_ASSERT(g_fib != NULL, "Failed to create VRF FIB\n");
+
+ sz = rte_rcu_qsbr_get_memsize(1);
+ g_v = (struct rte_rcu_qsbr *)rte_zmalloc_socket(NULL, sz,
+ RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+ RTE_TEST_ASSERT(g_v != NULL, "Can not allocate memory for RCU\n");
+
+ status = rte_rcu_qsbr_init(g_v, 1);
+ RTE_TEST_ASSERT(status == 0, "Can not initialize RCU\n");
+
+ rcu_cfg.v = g_v;
+ rcu_cfg.mode = RTE_FIB_QSBR_MODE_SYNC;
+ status = rte_fib_rcu_qsbr_add(g_fib, &rcu_cfg);
+ RTE_TEST_ASSERT(status == 0, "Can not attach RCU to VRF FIB\n");
+
+ writer_done = 0;
+ rte_eal_remote_launch(test_fib_rcu_qsbr_reader, NULL,
+ rte_get_next_lcore(-1, 1, 0));
+
+ depth = 28;
+ next_hop = 1;
+ status = rte_fib_vrf_add(g_fib, 0, g_ip, depth, next_hop);
+ if (status != 0) {
+ printf("%s: Failed to add rule\n", __func__);
+ goto error;
+ }
+
+ for (i = 0; i < WRITER_ITERATIONS; i++) {
+ status = rte_fib_vrf_delete(g_fib, 0, g_ip, depth);
+ if (status != 0) {
+ printf("%s: Failed to delete rule at iteration %d\n",
+ __func__, i);
+ goto error;
+ }
+
+ status = rte_fib_vrf_add(g_fib, 0, g_ip, depth, next_hop);
+ if (status != 0) {
+ printf("%s: Failed to add rule at iteration %d\n",
+ __func__, i);
+ goto error;
+ }
+ }
+
+error:
+ writer_done = 1;
+ rte_eal_mp_wait_lcore();
+
+ rte_fib_free(g_fib);
+ rte_free(g_v);
+
+ return status == 0 ? TEST_SUCCESS : TEST_FAILED;
+}
+
+/*
+ * Network order lookups with every lookup implementation of DIR8_16_8, in
+ * single and multi VRF FIBs, with a burst of two vector groups and a leftover.
*/
static int32_t
test_lookup_network_order(void)
@@ -645,9 +1113,11 @@ test_lookup_network_order(void)
const uint64_t probe_nhs[] = { 100, 300, 400, DEF_NH };
struct rte_fib_conf cfg = { 0 };
struct rte_fib *fib;
- uint32_t ips[17];
- uint64_t nhs[17];
- unsigned int i, t;
+ uint32_t ips[VRF_BURST_SZ + 1];
+ uint16_t vrfs[VRF_BURST_SZ + 1];
+ uint64_t nhs[VRF_BURST_SZ + 1];
+ unsigned int i, t, nb_vrfs;
+ uint16_t v;
cfg.type = RTE_FIB_DIR8_16_8;
cfg.max_routes = 100;
@@ -656,34 +1126,217 @@ test_lookup_network_order(void)
cfg.dir8_16_8.num_tbl16 = TEST_NUM_TBL16;
cfg.dir8_16_8.num_tbl8 = TEST_NUM_TBL8;
- fib = rte_fib_create("fib_be", SOCKET_ID_ANY, &cfg);
- RTE_TEST_ASSERT(fib != NULL, "Failed to create FIB\n");
+ for (nb_vrfs = 1; nb_vrfs <= TEST_MAX_VRFS; nb_vrfs += TEST_MAX_VRFS - 1) {
+ cfg.max_vrfs = nb_vrfs;
+ fib = rte_fib_create("fib_be", SOCKET_ID_ANY, &cfg);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create FIB\n");
- RTE_TEST_ASSERT(rte_fib_add(fib, RTE_IPV4(10, 0, 0, 0), 8, 100) == 0,
- "add /8\n");
- RTE_TEST_ASSERT(rte_fib_add(fib, RTE_IPV4(10, 1, 1, 0), 24, 300) == 0,
- "add /24\n");
- RTE_TEST_ASSERT(rte_fib_add(fib, RTE_IPV4(10, 1, 1, 7), 32, 400) == 0,
- "add /32\n");
- for (i = 0; i < RTE_DIM(ips); i++)
- ips[i] = rte_cpu_to_be_32(probes[i % RTE_DIM(probes)]);
+ for (v = 0; v < nb_vrfs; v++) {
+ RTE_TEST_ASSERT(rte_fib_vrf_add(fib, v,
+ RTE_IPV4(10, 0, 0, 0), 8, 100 + v) == 0,
+ "add /8 vrf %u\n", v);
+ RTE_TEST_ASSERT(rte_fib_vrf_add(fib, v,
+ RTE_IPV4(10, 1, 1, 0), 24, 300 + v) == 0,
+ "add /24 vrf %u\n", v);
+ RTE_TEST_ASSERT(rte_fib_vrf_add(fib, v,
+ RTE_IPV4(10, 1, 1, 7), 32, 400 + v) == 0,
+ "add /32 vrf %u\n", v);
+ }
+ for (i = 0; i < RTE_DIM(ips); i++) {
+ ips[i] = rte_cpu_to_be_32(probes[i % RTE_DIM(probes)]);
+ vrfs[i] = i % nb_vrfs;
+ }
+
+ for (t = 0; t < RTE_DIM(types); t++) {
+ /* The vector lookup needs a 512 bit SIMD width. */
+ if (rte_fib_select_lookup(fib, types[t]) != 0)
+ continue;
+
+ memset(nhs, 0, sizeof(nhs));
+ RTE_TEST_ASSERT(rte_fib_vrf_lookup_bulk(fib, vrfs, ips,
+ nhs, RTE_DIM(ips)) == 0, "lookup failed\n");
+ for (i = 0; i < RTE_DIM(ips); i++) {
+ uint64_t exp = probe_nhs[i % RTE_DIM(probes)];
+
+ if (exp != DEF_NH)
+ exp += vrfs[i];
+ RTE_TEST_ASSERT(nhs[i] == exp, "%u vrfs, lookup "
+ "type %u: ip %u vrf %u got %"PRIu64
+ " expected %"PRIu64"\n", nb_vrfs,
+ types[t], i, vrfs[i], nhs[i], exp);
+ }
+
+ /* A NULL VRF array (and the legacy lookup) select VRF 0. */
+ memset(nhs, 0, sizeof(nhs));
+ RTE_TEST_ASSERT(rte_fib_vrf_lookup_bulk(fib, NULL, ips,
+ nhs, RTE_DIM(ips)) == 0, "lookup failed\n");
+ for (i = 0; i < RTE_DIM(ips); i++)
+ RTE_TEST_ASSERT(nhs[i] ==
+ probe_nhs[i % RTE_DIM(probes)],
+ "%u vrfs, lookup type %u: ip %u vrf 0 "
+ "got %"PRIu64"\n", nb_vrfs, types[t], i,
+ nhs[i]);
+ }
+ rte_fib_free(fib);
+ }
+
+ return TEST_SUCCESS;
+}
+
+#define ITERATIONS (1 << 10)
+#define BATCH_SIZE (1 << 12)
+#define BULK_SIZE 32
+
+/*
+ * VRF cross-validation.
+ *
+ * Build one VRF-capable DIR8_16_8 FIB with N VRFs and N independent reference
+ * DIR8_16_8 FIBs. The same large route table is added to each VRF and its
+ * matching reference, using a per-VRF distinct unique next hop. Each VRF lookup
+ * must match its reference exactly, which proves both per-VRF correctness and
+ * isolation (no VRF bleeds into another), with every available lookup
+ * implementation.
+ */
+static int
+vrf_validate_lookups_type(struct rte_fib *dut, struct rte_fib **ref,
+ enum rte_fib_lookup_type type)
+{
+ uint64_t mism = 0, total = 0;
+ unsigned int i, j;
+
+ /* Compare mixed per-IP VRF lookups against the matching references. */
+ for (i = 0; i < ITERATIONS; i++) {
+ static uint32_t ip_batch[BATCH_SIZE];
+ static uint16_t vrf_batch[BATCH_SIZE];
+ uint64_t nh_dut[BULK_SIZE];
+ uint64_t nh_ref[BULK_SIZE];
+
+ for (j = 0; j < BATCH_SIZE; j++) {
+ ip_batch[j] = rte_rand();
+ vrf_batch[j] = rte_rand() % TEST_MAX_VRFS;
+ }
+
+ for (j = 0; j < BATCH_SIZE; j += BULK_SIZE) {
+ uint32_t k;
+
+ rte_fib_vrf_lookup_bulk(dut, &vrf_batch[j],
+ &ip_batch[j], nh_dut, BULK_SIZE);
+ /* Reference: each IP resolved in its own VRF's FIB. */
+ for (k = 0; k < BULK_SIZE; k++)
+ rte_fib_lookup_bulk(ref[vrf_batch[j + k]],
+ &ip_batch[j + k], &nh_ref[k], 1);
+ for (k = 0; k < BULK_SIZE; k++) {
+ total++;
+ if (nh_dut[k] != nh_ref[k]) {
+ if (mism < 20)
+ printf("Mismatch vrf=%u "
+ "ip=0x%08x dut=%" PRIu64
+ " ref=%" PRIu64 "\n",
+ vrf_batch[j + k],
+ ip_batch[j + k],
+ nh_dut[k], nh_ref[k]);
+ mism++;
+ }
+ }
+ }
+ }
+
+ printf("Lookup type %u: validated %" PRIu64 " lookups across %u VRFs, "
+ "%" PRIu64 " mismatches\n", type, total, TEST_MAX_VRFS, mism);
+ return (mism != 0) ? -1 : 0;
+}
+
+/* Validate the lookups with every lookup implementation of the algorithm. */
+static int
+vrf_validate_lookups(struct rte_fib *dut, struct rte_fib **ref)
+{
+ const enum rte_fib_lookup_type types[] = {
+ RTE_FIB_LOOKUP_DIR8_16_8_SCALAR,
+ RTE_FIB_LOOKUP_DIR8_16_8_VECTOR_AVX512,
+ };
+ unsigned int t;
+ int ret;
for (t = 0; t < RTE_DIM(types); t++) {
/* The vector lookup needs a 512 bit SIMD width. */
- if (rte_fib_select_lookup(fib, types[t]) != 0)
+ if (rte_fib_select_lookup(dut, types[t]) != 0)
continue;
+ ret = vrf_validate_lookups_type(dut, ref, types[t]);
+ if (ret != 0)
+ return ret;
+ }
+ return 0;
+}
- memset(nhs, 0, sizeof(nhs));
- RTE_TEST_ASSERT(rte_fib_lookup_bulk(fib, ips, nhs,
- RTE_DIM(ips)) == 0, "lookup failed\n");
- for (i = 0; i < RTE_DIM(ips); i++)
- RTE_TEST_ASSERT(nhs[i] == probe_nhs[i % RTE_DIM(probes)],
- "lookup type %u: ip %u got %"PRIu64"\n",
- types[t], i, nhs[i]);
+static int
+test_fib_vrf_validate_dir8_16_8(void)
+{
+ struct rte_fib *dut = NULL;
+ struct rte_fib *ref[TEST_MAX_VRFS] = { NULL };
+ struct rte_fib_conf cfg = { 0 };
+ struct rte_fib_conf cfg_ref = { 0 };
+ unsigned int i;
+ uint16_t v;
+ int ret = 0;
+ char name[64];
+
+ generate_large_route_rule_table();
+
+ printf("No. routes = %u, VRFs = %u\n",
+ (unsigned int) NUM_ROUTE_ENTRIES, TEST_MAX_VRFS);
+
+ cfg.max_routes = 2000000;
+ cfg.default_nh = 0;
+ cfg.type = RTE_FIB_DIR8_16_8;
+ cfg.max_vrfs = TEST_MAX_VRFS;
+ cfg.dir8_16_8.num_tbl16 = 256 * TEST_MAX_VRFS;
+ cfg.dir8_16_8.num_tbl8 = 1 << 18;
+
+ dut = rte_fib_create("vrf_validate_dut", SOCKET_ID_ANY, &cfg);
+ RTE_TEST_ASSERT(dut != NULL, "Failed to create VRF FIB\n");
+
+ cfg_ref.max_routes = 2000000;
+ cfg_ref.default_nh = 0;
+ cfg_ref.type = RTE_FIB_DIR8_16_8;
+ cfg_ref.dir8_16_8.num_tbl16 = 256;
+ cfg_ref.dir8_16_8.num_tbl8 = 1 << 16;
+ for (v = 0; v < TEST_MAX_VRFS; v++) {
+ snprintf(name, sizeof(name), "vrf_validate_ref%u", v);
+ ref[v] = rte_fib_create(name, SOCKET_ID_ANY, &cfg_ref);
+ if (ref[v] == NULL) {
+ printf("Failed to create reference FIB %u\n", v);
+ ret = -1;
+ goto out;
+ }
}
- rte_fib_free(fib);
- return TEST_SUCCESS;
+ /* Add the same routes to each VRF and its reference, per-VRF unique nh. */
+ for (i = 0; i < NUM_ROUTE_ENTRIES; i++) {
+ for (v = 0; v < TEST_MAX_VRFS; v++) {
+ uint64_t nh = (uint64_t)i + 1 +
+ (uint64_t)v * ((uint64_t)MAX_RULE_NUM + 1);
+ int r_dut = rte_fib_vrf_add(dut, v,
+ large_route_table[i].ip,
+ large_route_table[i].depth, nh);
+ int r_ref = rte_fib_add(ref[v],
+ large_route_table[i].ip,
+ large_route_table[i].depth, nh);
+ if (r_dut != r_ref) {
+ printf("Add status mismatch i=%u vrf=%u "
+ "dut=%d ref=%d\n", i, v, r_dut, r_ref);
+ ret = -1;
+ goto out;
+ }
+ }
+ }
+
+ ret = vrf_validate_lookups(dut, ref);
+out:
+ rte_fib_free(dut);
+ for (v = 0; v < TEST_MAX_VRFS; v++)
+ rte_fib_free(ref[v]);
+
+ return ret;
}
static struct unit_test_suite fib_fast_tests = {
@@ -699,6 +1352,16 @@ static struct unit_test_suite fib_fast_tests = {
TEST_CASE(test_lookup_network_order),
TEST_CASE(test_invalid_rcu),
TEST_CASE(test_fib_rcu_sync_rw),
+ TEST_CASE(test_vrf_create_invalid),
+ TEST_CASE(test_vrf_isolation),
+ TEST_CASE(test_vrf_overlapping_prefixes),
+ TEST_CASE(test_vrf_mixed_lookup),
+ TEST_CASE(test_vrf_default_nh),
+ TEST_CASE(test_vrf_update_route),
+ TEST_CASE(test_vrf_bounds),
+ TEST_CASE(test_vrf_get_rib),
+ TEST_CASE(test_vrf_rcu),
+ TEST_CASE(test_vrf_rcu_sync_rw),
TEST_CASES_END()
}
};
@@ -709,6 +1372,7 @@ static struct unit_test_suite fib_slow_tests = {
.teardown = NULL,
.unit_test_cases = {
TEST_CASE(test_multiple_create),
+ TEST_CASE(test_fib_vrf_validate_dir8_16_8),
TEST_CASES_END()
}
};
diff --git a/app/test/test_fib_perf.c b/app/test/test_fib_perf.c
index 0967644316..ff8240aabf 100644
--- a/app/test/test_fib_perf.c
+++ b/app/test/test_fib_perf.c
@@ -6,17 +6,16 @@
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
-#include <math.h>
#include <inttypes.h>
#include <rte_cycles.h>
#include <rte_random.h>
#include <rte_branch_prediction.h>
-#include <rte_ip.h>
#include <rte_fib.h>
#include "test.h"
#include "test_xmmt_ops.h"
+#include "test_fib_routes.h"
#define TEST_FIB_ASSERT(cond) do { \
if (!(cond)) { \
@@ -29,270 +28,6 @@
#define BATCH_SIZE (1 << 12)
#define BULK_SIZE 32
-#define MAX_RULE_NUM (1200000)
-
-struct route_rule {
- uint32_t ip;
- uint8_t depth;
-};
-
-static struct route_rule large_route_table[MAX_RULE_NUM];
-
-static uint32_t num_route_entries;
-#define NUM_ROUTE_ENTRIES num_route_entries
-
-enum {
- IP_CLASS_A,
- IP_CLASS_B,
- IP_CLASS_C
-};
-#define RTE_FIB_MAX_DEPTH 32
-/* struct route_rule_count defines the total number of rules in following a/b/c
- * each item in a[]/b[]/c[] is the number of common IP address class A/B/C, not
- * including the ones for private local network.
- */
-struct route_rule_count {
- uint32_t a[RTE_FIB_MAX_DEPTH];
- uint32_t b[RTE_FIB_MAX_DEPTH];
- uint32_t c[RTE_FIB_MAX_DEPTH];
-};
-
-/* All following numbers of each depth of each common IP class are just
- * got from previous large constant table in app/test/test_lpm_routes.h .
- * In order to match similar performance, they keep same depth and IP
- * address coverage as previous constant table. These numbers don't
- * include any private local IP address. As previous large const rule
- * table was just dumped from a real router, there are no any IP address
- * in class C or D.
- */
-static struct route_rule_count rule_count = {
- .a = { /* IP class A in which the most significant bit is 0 */
- 0, /* depth = 1 */
- 0, /* depth = 2 */
- 1, /* depth = 3 */
- 0, /* depth = 4 */
- 2, /* depth = 5 */
- 1, /* depth = 6 */
- 3, /* depth = 7 */
- 185, /* depth = 8 */
- 26, /* depth = 9 */
- 16, /* depth = 10 */
- 39, /* depth = 11 */
- 144, /* depth = 12 */
- 233, /* depth = 13 */
- 528, /* depth = 14 */
- 866, /* depth = 15 */
- 3856, /* depth = 16 */
- 3268, /* depth = 17 */
- 5662, /* depth = 18 */
- 17301, /* depth = 19 */
- 22226, /* depth = 20 */
- 11147, /* depth = 21 */
- 16746, /* depth = 22 */
- 17120, /* depth = 23 */
- 77578, /* depth = 24 */
- 401, /* depth = 25 */
- 656, /* depth = 26 */
- 1107, /* depth = 27 */
- 1121, /* depth = 28 */
- 2316, /* depth = 29 */
- 717, /* depth = 30 */
- 10, /* depth = 31 */
- 66 /* depth = 32 */
- },
- .b = { /* IP class A in which the most 2 significant bits are 10 */
- 0, /* depth = 1 */
- 0, /* depth = 2 */
- 0, /* depth = 3 */
- 0, /* depth = 4 */
- 1, /* depth = 5 */
- 1, /* depth = 6 */
- 1, /* depth = 7 */
- 3, /* depth = 8 */
- 3, /* depth = 9 */
- 30, /* depth = 10 */
- 25, /* depth = 11 */
- 168, /* depth = 12 */
- 305, /* depth = 13 */
- 569, /* depth = 14 */
- 1129, /* depth = 15 */
- 50800, /* depth = 16 */
- 1645, /* depth = 17 */
- 1820, /* depth = 18 */
- 3506, /* depth = 19 */
- 3258, /* depth = 20 */
- 3424, /* depth = 21 */
- 4971, /* depth = 22 */
- 6885, /* depth = 23 */
- 39771, /* depth = 24 */
- 424, /* depth = 25 */
- 170, /* depth = 26 */
- 433, /* depth = 27 */
- 92, /* depth = 28 */
- 366, /* depth = 29 */
- 377, /* depth = 30 */
- 2, /* depth = 31 */
- 200 /* depth = 32 */
- },
- .c = { /* IP class A in which the most 3 significant bits are 110 */
- 0, /* depth = 1 */
- 0, /* depth = 2 */
- 0, /* depth = 3 */
- 0, /* depth = 4 */
- 0, /* depth = 5 */
- 0, /* depth = 6 */
- 0, /* depth = 7 */
- 12, /* depth = 8 */
- 8, /* depth = 9 */
- 9, /* depth = 10 */
- 33, /* depth = 11 */
- 69, /* depth = 12 */
- 237, /* depth = 13 */
- 1007, /* depth = 14 */
- 1717, /* depth = 15 */
- 14663, /* depth = 16 */
- 8070, /* depth = 17 */
- 16185, /* depth = 18 */
- 48261, /* depth = 19 */
- 36870, /* depth = 20 */
- 33960, /* depth = 21 */
- 50638, /* depth = 22 */
- 61422, /* depth = 23 */
- 466549, /* depth = 24 */
- 1829, /* depth = 25 */
- 4824, /* depth = 26 */
- 4927, /* depth = 27 */
- 5914, /* depth = 28 */
- 10254, /* depth = 29 */
- 4905, /* depth = 30 */
- 1, /* depth = 31 */
- 716 /* depth = 32 */
- }
-};
-
-static void generate_random_rule_prefix(uint32_t ip_class, uint8_t depth)
-{
-/* IP address class A, the most significant bit is 0 */
-#define IP_HEAD_MASK_A 0x00000000
-#define IP_HEAD_BIT_NUM_A 1
-
-/* IP address class B, the most significant 2 bits are 10 */
-#define IP_HEAD_MASK_B 0x80000000
-#define IP_HEAD_BIT_NUM_B 2
-
-/* IP address class C, the most significant 3 bits are 110 */
-#define IP_HEAD_MASK_C 0xC0000000
-#define IP_HEAD_BIT_NUM_C 3
-
- uint32_t class_depth;
- uint32_t range;
- uint32_t mask;
- uint32_t step;
- uint32_t start;
- uint32_t fixed_bit_num;
- uint32_t ip_head_mask;
- uint32_t rule_num;
- uint32_t k;
- struct route_rule *ptr_rule;
-
- if (ip_class == IP_CLASS_A) { /* IP Address class A */
- fixed_bit_num = IP_HEAD_BIT_NUM_A;
- ip_head_mask = IP_HEAD_MASK_A;
- rule_num = rule_count.a[depth - 1];
- } else if (ip_class == IP_CLASS_B) { /* IP Address class B */
- fixed_bit_num = IP_HEAD_BIT_NUM_B;
- ip_head_mask = IP_HEAD_MASK_B;
- rule_num = rule_count.b[depth - 1];
- } else { /* IP Address class C */
- fixed_bit_num = IP_HEAD_BIT_NUM_C;
- ip_head_mask = IP_HEAD_MASK_C;
- rule_num = rule_count.c[depth - 1];
- }
-
- if (rule_num == 0)
- return;
-
- /* the number of rest bits which don't include the most significant
- * fixed bits for this IP address class
- */
- class_depth = depth - fixed_bit_num;
-
- /* range is the maximum number of rules for this depth and
- * this IP address class
- */
- range = 1 << class_depth;
-
- /* only mask the most depth significant generated bits
- * except fixed bits for IP address class
- */
- mask = range - 1;
-
- /* Widen coverage of IP address in generated rules */
- if (range <= rule_num)
- step = 1;
- else
- step = round((double)range / rule_num);
-
- /* Only generate rest bits except the most significant
- * fixed bits for IP address class
- */
- start = rte_rand() & mask;
- ptr_rule = &large_route_table[num_route_entries];
- for (k = 0; k < rule_num; k++) {
- ptr_rule->ip = (start << (RTE_FIB_MAX_DEPTH - depth))
- | ip_head_mask;
- ptr_rule->depth = depth;
- ptr_rule++;
- start = (start + step) & mask;
- }
- num_route_entries += rule_num;
-}
-
-static void insert_rule_in_random_pos(uint32_t ip, uint8_t depth)
-{
- uint32_t pos;
- int try_count = 0;
- struct route_rule tmp;
-
- do {
- pos = rte_rand();
- try_count++;
- } while ((try_count < 10) && (pos > num_route_entries));
-
- if ((pos > num_route_entries) || (pos >= MAX_RULE_NUM))
- pos = num_route_entries >> 1;
-
- tmp = large_route_table[pos];
- large_route_table[pos].ip = ip;
- large_route_table[pos].depth = depth;
- if (num_route_entries < MAX_RULE_NUM)
- large_route_table[num_route_entries++] = tmp;
-}
-
-static void generate_large_route_rule_table(void)
-{
- uint32_t ip_class;
- uint8_t depth;
-
- num_route_entries = 0;
- memset(large_route_table, 0, sizeof(large_route_table));
-
- for (ip_class = IP_CLASS_A; ip_class <= IP_CLASS_C; ip_class++) {
- for (depth = 1; depth <= RTE_FIB_MAX_DEPTH; depth++)
- generate_random_rule_prefix(ip_class, depth);
- }
-
- /* Add following rules to keep same as previous large constant table,
- * they are 4 rules with private local IP address and 1 all-zeros prefix
- * with depth = 8.
- */
- insert_rule_in_random_pos(RTE_IPV4(0, 0, 0, 0), 8);
- insert_rule_in_random_pos(RTE_IPV4(10, 2, 23, 147), 32);
- insert_rule_in_random_pos(RTE_IPV4(192, 168, 100, 10), 24);
- insert_rule_in_random_pos(RTE_IPV4(192, 168, 25, 100), 24);
- insert_rule_in_random_pos(RTE_IPV4(192, 168, 129, 124), 32);
-}
-
static void
print_route_distribution(const struct route_rule *table, uint32_t n)
{
@@ -437,5 +172,88 @@ test_fib_perf_dir8_16_8(void)
return measure_fib_perf("fib_perf_dir8_16_8", &config);
}
+#define VRF_PERF_NB_VRFS 4
+
+static int
+test_fib_perf_vrf_dir8_16_8(void)
+{
+ struct rte_fib *fib = NULL;
+ struct rte_fib_conf config = { 0 };
+ uint64_t begin, total_time;
+ unsigned int i, j;
+ uint16_t v;
+ int status = 0;
+ int64_t count = 0;
+
+ config.max_routes = 2000000;
+ config.rib_ext_sz = 0;
+ config.type = RTE_FIB_DIR8_16_8;
+ config.default_nh = 0;
+ config.max_vrfs = VRF_PERF_NB_VRFS;
+ config.dir8_16_8.num_tbl16 = 256 * VRF_PERF_NB_VRFS;
+ config.dir8_16_8.num_tbl8 = 1 << 18;
+
+ generate_large_route_rule_table();
+
+ printf("No. routes = %u, VRFs = %u\n",
+ (unsigned int) NUM_ROUTE_ENTRIES, VRF_PERF_NB_VRFS);
+
+ fib = rte_fib_create("vrf_perf_d8", SOCKET_ID_ANY, &config);
+ TEST_FIB_ASSERT(fib != NULL);
+
+ begin = rte_rdtsc();
+ for (i = 0; i < NUM_ROUTE_ENTRIES; i++)
+ for (v = 0; v < VRF_PERF_NB_VRFS; v++)
+ if (rte_fib_vrf_add(fib, v, large_route_table[i].ip,
+ large_route_table[i].depth,
+ (uint64_t)i | ((uint64_t)v << 32)) == 0)
+ status++;
+ total_time = rte_rdtsc() - begin;
+ printf("Unique added entries = %d\n", status);
+ printf("Average VRF FIB Add: %g cycles\n", (double)total_time /
+ ((double)NUM_ROUTE_ENTRIES * VRF_PERF_NB_VRFS));
+
+ total_time = 0;
+ count = 0;
+ for (i = 0; i < ITERATIONS; i++) {
+ static uint32_t ip_batch[BATCH_SIZE];
+ static uint16_t vrf_batch[BATCH_SIZE];
+ uint64_t next_hops[BULK_SIZE];
+
+ for (j = 0; j < BATCH_SIZE; j++) {
+ ip_batch[j] = rte_rand();
+ vrf_batch[j] = rte_rand() % VRF_PERF_NB_VRFS;
+ }
+
+ begin = rte_rdtsc();
+ for (j = 0; j < BATCH_SIZE; j += BULK_SIZE) {
+ uint32_t k;
+ rte_fib_vrf_lookup_bulk(fib, &vrf_batch[j], &ip_batch[j],
+ next_hops, BULK_SIZE);
+ for (k = 0; k < BULK_SIZE; k++)
+ if (unlikely(!(next_hops[k] != 0)))
+ count++;
+ }
+ total_time += rte_rdtsc() - begin;
+ }
+ printf("BULK VRF FIB Lookup: %.1f cycles (fails = %.1f%%)\n",
+ (double)total_time / ((double)ITERATIONS * BATCH_SIZE),
+ (count * 100.0) / (double)(ITERATIONS * BATCH_SIZE));
+
+ begin = rte_rdtsc();
+ for (i = 0; i < NUM_ROUTE_ENTRIES; i++)
+ for (v = 0; v < VRF_PERF_NB_VRFS; v++)
+ rte_fib_vrf_delete(fib, v, large_route_table[i].ip,
+ large_route_table[i].depth);
+ total_time = rte_rdtsc() - begin;
+ printf("Average VRF FIB Delete: %g cycles\n", (double)total_time /
+ ((double)NUM_ROUTE_ENTRIES * VRF_PERF_NB_VRFS));
+
+ rte_fib_free(fib);
+
+ return 0;
+}
+
REGISTER_PERF_TEST(fib_perf_autotest, test_fib_perf);
REGISTER_PERF_TEST(fib_perf_dir8_16_8_autotest, test_fib_perf_dir8_16_8);
+REGISTER_PERF_TEST(fib_perf_vrf_dir8_16_8_autotest, test_fib_perf_vrf_dir8_16_8);
diff --git a/app/test/test_fib_routes.h b/app/test/test_fib_routes.h
new file mode 100644
index 0000000000..c03eaede05
--- /dev/null
+++ b/app/test/test_fib_routes.h
@@ -0,0 +1,282 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Vladimir Medvedkin <medvedkinv@gmail.com>
+ * Copyright(c) 2019 Intel Corporation
+ */
+
+#ifndef _TEST_FIB_ROUTES_H_
+#define _TEST_FIB_ROUTES_H_
+
+#include <math.h>
+#include <stdint.h>
+#include <string.h>
+
+#include <rte_ip.h>
+#include <rte_random.h>
+
+/* Large IPv4 route table generator shared by the FIB tests. */
+
+#define MAX_RULE_NUM (1200000)
+
+struct route_rule {
+ uint32_t ip;
+ uint8_t depth;
+};
+
+static struct route_rule large_route_table[MAX_RULE_NUM];
+
+static uint32_t num_route_entries;
+#define NUM_ROUTE_ENTRIES num_route_entries
+
+enum {
+ IP_CLASS_A,
+ IP_CLASS_B,
+ IP_CLASS_C
+};
+#define RTE_FIB_MAX_DEPTH 32
+/* struct route_rule_count defines the total number of rules in following a/b/c
+ * each item in a[]/b[]/c[] is the number of common IP address class A/B/C, not
+ * including the ones for private local network.
+ */
+struct route_rule_count {
+ uint32_t a[RTE_FIB_MAX_DEPTH];
+ uint32_t b[RTE_FIB_MAX_DEPTH];
+ uint32_t c[RTE_FIB_MAX_DEPTH];
+};
+
+/* All following numbers of each depth of each common IP class are just
+ * got from previous large constant table in app/test/test_lpm_routes.h .
+ * In order to match similar performance, they keep same depth and IP
+ * address coverage as previous constant table. These numbers don't
+ * include any private local IP address. As previous large const rule
+ * table was just dumped from a real router, there are no any IP address
+ * in class C or D.
+ */
+static struct route_rule_count rule_count = {
+ .a = { /* IP class A in which the most significant bit is 0 */
+ 0, /* depth = 1 */
+ 0, /* depth = 2 */
+ 1, /* depth = 3 */
+ 0, /* depth = 4 */
+ 2, /* depth = 5 */
+ 1, /* depth = 6 */
+ 3, /* depth = 7 */
+ 185, /* depth = 8 */
+ 26, /* depth = 9 */
+ 16, /* depth = 10 */
+ 39, /* depth = 11 */
+ 144, /* depth = 12 */
+ 233, /* depth = 13 */
+ 528, /* depth = 14 */
+ 866, /* depth = 15 */
+ 3856, /* depth = 16 */
+ 3268, /* depth = 17 */
+ 5662, /* depth = 18 */
+ 17301, /* depth = 19 */
+ 22226, /* depth = 20 */
+ 11147, /* depth = 21 */
+ 16746, /* depth = 22 */
+ 17120, /* depth = 23 */
+ 77578, /* depth = 24 */
+ 401, /* depth = 25 */
+ 656, /* depth = 26 */
+ 1107, /* depth = 27 */
+ 1121, /* depth = 28 */
+ 2316, /* depth = 29 */
+ 717, /* depth = 30 */
+ 10, /* depth = 31 */
+ 66 /* depth = 32 */
+ },
+ .b = { /* IP class A in which the most 2 significant bits are 10 */
+ 0, /* depth = 1 */
+ 0, /* depth = 2 */
+ 0, /* depth = 3 */
+ 0, /* depth = 4 */
+ 1, /* depth = 5 */
+ 1, /* depth = 6 */
+ 1, /* depth = 7 */
+ 3, /* depth = 8 */
+ 3, /* depth = 9 */
+ 30, /* depth = 10 */
+ 25, /* depth = 11 */
+ 168, /* depth = 12 */
+ 305, /* depth = 13 */
+ 569, /* depth = 14 */
+ 1129, /* depth = 15 */
+ 50800, /* depth = 16 */
+ 1645, /* depth = 17 */
+ 1820, /* depth = 18 */
+ 3506, /* depth = 19 */
+ 3258, /* depth = 20 */
+ 3424, /* depth = 21 */
+ 4971, /* depth = 22 */
+ 6885, /* depth = 23 */
+ 39771, /* depth = 24 */
+ 424, /* depth = 25 */
+ 170, /* depth = 26 */
+ 433, /* depth = 27 */
+ 92, /* depth = 28 */
+ 366, /* depth = 29 */
+ 377, /* depth = 30 */
+ 2, /* depth = 31 */
+ 200 /* depth = 32 */
+ },
+ .c = { /* IP class A in which the most 3 significant bits are 110 */
+ 0, /* depth = 1 */
+ 0, /* depth = 2 */
+ 0, /* depth = 3 */
+ 0, /* depth = 4 */
+ 0, /* depth = 5 */
+ 0, /* depth = 6 */
+ 0, /* depth = 7 */
+ 12, /* depth = 8 */
+ 8, /* depth = 9 */
+ 9, /* depth = 10 */
+ 33, /* depth = 11 */
+ 69, /* depth = 12 */
+ 237, /* depth = 13 */
+ 1007, /* depth = 14 */
+ 1717, /* depth = 15 */
+ 14663, /* depth = 16 */
+ 8070, /* depth = 17 */
+ 16185, /* depth = 18 */
+ 48261, /* depth = 19 */
+ 36870, /* depth = 20 */
+ 33960, /* depth = 21 */
+ 50638, /* depth = 22 */
+ 61422, /* depth = 23 */
+ 466549, /* depth = 24 */
+ 1829, /* depth = 25 */
+ 4824, /* depth = 26 */
+ 4927, /* depth = 27 */
+ 5914, /* depth = 28 */
+ 10254, /* depth = 29 */
+ 4905, /* depth = 30 */
+ 1, /* depth = 31 */
+ 716 /* depth = 32 */
+ }
+};
+
+static void generate_random_rule_prefix(uint32_t ip_class, uint8_t depth)
+{
+/* IP address class A, the most significant bit is 0 */
+#define IP_HEAD_MASK_A 0x00000000
+#define IP_HEAD_BIT_NUM_A 1
+
+/* IP address class B, the most significant 2 bits are 10 */
+#define IP_HEAD_MASK_B 0x80000000
+#define IP_HEAD_BIT_NUM_B 2
+
+/* IP address class C, the most significant 3 bits are 110 */
+#define IP_HEAD_MASK_C 0xC0000000
+#define IP_HEAD_BIT_NUM_C 3
+
+ uint32_t class_depth;
+ uint32_t range;
+ uint32_t mask;
+ uint32_t step;
+ uint32_t start;
+ uint32_t fixed_bit_num;
+ uint32_t ip_head_mask;
+ uint32_t rule_num;
+ uint32_t k;
+ struct route_rule *ptr_rule;
+
+ if (ip_class == IP_CLASS_A) { /* IP Address class A */
+ fixed_bit_num = IP_HEAD_BIT_NUM_A;
+ ip_head_mask = IP_HEAD_MASK_A;
+ rule_num = rule_count.a[depth - 1];
+ } else if (ip_class == IP_CLASS_B) { /* IP Address class B */
+ fixed_bit_num = IP_HEAD_BIT_NUM_B;
+ ip_head_mask = IP_HEAD_MASK_B;
+ rule_num = rule_count.b[depth - 1];
+ } else { /* IP Address class C */
+ fixed_bit_num = IP_HEAD_BIT_NUM_C;
+ ip_head_mask = IP_HEAD_MASK_C;
+ rule_num = rule_count.c[depth - 1];
+ }
+
+ if (rule_num == 0)
+ return;
+
+ /* the number of rest bits which don't include the most significant
+ * fixed bits for this IP address class
+ */
+ class_depth = depth - fixed_bit_num;
+
+ /* range is the maximum number of rules for this depth and
+ * this IP address class
+ */
+ range = 1 << class_depth;
+
+ /* only mask the most depth significant generated bits
+ * except fixed bits for IP address class
+ */
+ mask = range - 1;
+
+ /* Widen coverage of IP address in generated rules */
+ if (range <= rule_num)
+ step = 1;
+ else
+ step = round((double)range / rule_num);
+
+ /* Only generate rest bits except the most significant
+ * fixed bits for IP address class
+ */
+ start = rte_rand() & mask;
+ ptr_rule = &large_route_table[num_route_entries];
+ for (k = 0; k < rule_num; k++) {
+ ptr_rule->ip = (start << (RTE_FIB_MAX_DEPTH - depth))
+ | ip_head_mask;
+ ptr_rule->depth = depth;
+ ptr_rule++;
+ start = (start + step) & mask;
+ }
+ num_route_entries += rule_num;
+}
+
+static void insert_rule_in_random_pos(uint32_t ip, uint8_t depth)
+{
+ uint32_t pos;
+ int try_count = 0;
+ struct route_rule tmp;
+
+ do {
+ pos = rte_rand();
+ try_count++;
+ } while ((try_count < 10) && (pos > num_route_entries));
+
+ if ((pos > num_route_entries) || (pos >= MAX_RULE_NUM))
+ pos = num_route_entries >> 1;
+
+ tmp = large_route_table[pos];
+ large_route_table[pos].ip = ip;
+ large_route_table[pos].depth = depth;
+ if (num_route_entries < MAX_RULE_NUM)
+ large_route_table[num_route_entries++] = tmp;
+}
+
+static void generate_large_route_rule_table(void)
+{
+ uint32_t ip_class;
+ uint8_t depth;
+
+ num_route_entries = 0;
+ memset(large_route_table, 0, sizeof(large_route_table));
+
+ for (ip_class = IP_CLASS_A; ip_class <= IP_CLASS_C; ip_class++) {
+ for (depth = 1; depth <= RTE_FIB_MAX_DEPTH; depth++)
+ generate_random_rule_prefix(ip_class, depth);
+ }
+
+ /* Add following rules to keep same as previous large constant table,
+ * they are 4 rules with private local IP address and 1 all-zeros prefix
+ * with depth = 8.
+ */
+ insert_rule_in_random_pos(RTE_IPV4(0, 0, 0, 0), 8);
+ insert_rule_in_random_pos(RTE_IPV4(10, 2, 23, 147), 32);
+ insert_rule_in_random_pos(RTE_IPV4(192, 168, 100, 10), 24);
+ insert_rule_in_random_pos(RTE_IPV4(192, 168, 25, 100), 24);
+ insert_rule_in_random_pos(RTE_IPV4(192, 168, 129, 124), 32);
+}
+
+#endif /* _TEST_FIB_ROUTES_H_ */
diff --git a/doc/guides/prog_guide/fib_lib.rst b/doc/guides/prog_guide/fib_lib.rst
index c20759faaf..888c1acdfa 100644
--- a/doc/guides/prog_guide/fib_lib.rst
+++ b/doc/guides/prog_guide/fib_lib.rst
@@ -46,6 +46,31 @@ The main methods within the ``rte_fib`` API are:
for a set of IP addresses, it will return a set of corresponding next hop IDs.
+Virtual Routing and Forwarding (VRF)
+------------------------------------
+
+A FIB can hold several independent routing tables, one per VRF, selected by
+``max_vrfs`` in the configuration struct (0 or 1 means a single VRF).
+Every VRF has its own default next hop ID (``vrf_default_nh``, or ``default_nh``
+for all of them when it is NULL) and is addressed by a VRF ID from 0 to
+``max_vrfs - 1``.
+
+The VRF aware methods are:
+
+* ``rte_fib_vrf_add()``, ``rte_fib_vrf_delete()``: Same as ``rte_fib_add()``
+ and ``rte_fib_delete()`` for a given VRF.
+
+* ``rte_fib_vrf_lookup_bulk()``: Bulk LPM lookup with a VRF ID per IP address.
+
+* ``rte_fib_vrf_get_rib()``: Get the ``rte_rib`` of a given VRF.
+
+The legacy methods operate on VRF 0.
+Multi-VRF FIBs are supported by the ``RTE_FIB_DIR8_16_8`` algorithm, which keeps
+an independent ``rte_rib`` and an independent root table per VRF, while the lower
+level tables are allocated from pools shared by all the VRFs.
+The VRF ID selects the root table, then the lookup proceeds as for a single VRF.
+
+
Implementation details
----------------------
@@ -143,13 +168,13 @@ dataplane algorithm on FIB creation.
The main FIB configuration struct stores the dataplane parameters inside ``dir8_16_8``
within the ``rte_fib_conf`` and it consists of:
-* ``num_tbl16``: The number of second level groups, at most one per /8 is ever
- needed.
+* ``num_tbl16``: The number of second level groups, at most one per /8 and VRF is
+ ever needed.
* ``num_tbl8``: The number of third level groups.
-Compared to DIR-24-8, the root table is small (2 KB), which makes it suitable
-when many tables must be kept, at the price of an extra memory access for
+Compared to DIR-24-8, the root table is small (2 KB per VRF), which makes it
+suitable for a large number of VRFs, at the price of an extra memory access for
prefixes longer than 8 bits.
diff --git a/doc/guides/rel_notes/release_26_11.rst b/doc/guides/rel_notes/release_26_11.rst
index 04781a2761..5f77116d8c 100644
--- a/doc/guides/rel_notes/release_26_11.rst
+++ b/doc/guides/rel_notes/release_26_11.rst
@@ -211,6 +211,13 @@ New Features
* Added the ``RTE_FIB_DIR8_16_8`` IPv4 algorithm, a three level variant of
DIR-24-8 with a small root table, with scalar and AVX512 lookup functions.
+* **Added VRF support to the FIB library.**
+
+ A FIB can now hold several independent routing tables (VRFs), configured with
+ the ``max_vrfs`` and ``vrf_default_nh`` fields of ``rte_fib_conf`` and
+ accessed with the ``rte_fib_vrf_*`` functions. ``RTE_FIB_DIR8_16_8`` keeps a
+ root table per VRF.
+
Removed Items
-------------
@@ -466,6 +473,11 @@ API Changes
* ``rte_tel_data_add_dict_uint_hex``
* ``rte_telemetry_register_cmd_arg``
+* fib: Added a VRF ID argument to the dataplane callbacks.
+
+ The ``rte_fib_modify_fn_t`` and ``rte_fib_lookup_fn_t`` function types gained
+ a VRF argument.
+
ABI Changes
-----------
@@ -488,6 +500,9 @@ ABI Changes
* cryptodev: The struct ``rte_crypto_rsa_op_param``
was updated to include ``pss_salt``.
+* fib: ``struct rte_fib_conf`` was extended with the ``max_vrfs`` and
+ ``vrf_default_nh`` fields.
+
Known Issues
------------
diff --git a/lib/fib/dir24_8.c b/lib/fib/dir24_8.c
index bb7975eca1..9b14693320 100644
--- a/lib/fib/dir24_8.c
+++ b/lib/fib/dir24_8.c
@@ -465,7 +465,7 @@ modify_fib(struct dir24_8_tbl *dp, struct rte_rib *rib, uint32_t ip,
}
int
-dir24_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t depth,
+dir24_8_modify(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip, uint8_t depth,
uint64_t next_hop, int op)
{
struct dir24_8_tbl *dp;
@@ -476,6 +476,8 @@ dir24_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t depth,
int ret = 0;
uint64_t par_nh, node_nh;
+ RTE_SET_USED(vrf_id);
+
if ((fib == NULL) || (depth > RTE_FIB_MAXDEPTH))
return -EINVAL;
diff --git a/lib/fib/dir24_8.h b/lib/fib/dir24_8.h
index b343b5d686..88c47072d9 100644
--- a/lib/fib/dir24_8.h
+++ b/lib/fib/dir24_8.h
@@ -116,7 +116,8 @@ is_entry_extended(uint64_t ent)
}
#define LOOKUP_FUNC(suffix, type, bulk_prefetch, nh_sz) \
-static inline void dir24_8_lookup_bulk_##suffix(void *p, const uint32_t *ips, \
+static inline void dir24_8_lookup_bulk_##suffix(void *p, \
+ const uint16_t *vrf_ids, const uint32_t *ips, \
uint64_t *next_hops, const unsigned int n) \
{ \
struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p; \
@@ -125,6 +126,7 @@ static inline void dir24_8_lookup_bulk_##suffix(void *p, const uint32_t *ips, \
uint32_t prefetch_offset = \
RTE_MIN((unsigned int)bulk_prefetch, n); \
\
+ RTE_SET_USED(vrf_ids); \
for (i = 0; i < prefetch_offset; i++) \
rte_prefetch0(get_tbl24_p(dp, ips[i], nh_sz)); \
for (i = 0; i < (n - prefetch_offset); i++) { \
@@ -179,43 +181,47 @@ dir24_8_lookup_bulk(struct dir24_8_tbl *dp, const uint32_t *ips,
}
static inline void
-dir24_8_lookup_bulk_0(void *p, const uint32_t *ips,
+dir24_8_lookup_bulk_0(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
uint64_t *next_hops, const unsigned int n)
{
struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
+ RTE_SET_USED(vrf_ids);
dir24_8_lookup_bulk(dp, ips, next_hops, n, 0);
}
static inline void
-dir24_8_lookup_bulk_1(void *p, const uint32_t *ips,
+dir24_8_lookup_bulk_1(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
uint64_t *next_hops, const unsigned int n)
{
struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
+ RTE_SET_USED(vrf_ids);
dir24_8_lookup_bulk(dp, ips, next_hops, n, 1);
}
static inline void
-dir24_8_lookup_bulk_2(void *p, const uint32_t *ips,
+dir24_8_lookup_bulk_2(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
uint64_t *next_hops, const unsigned int n)
{
struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
+ RTE_SET_USED(vrf_ids);
dir24_8_lookup_bulk(dp, ips, next_hops, n, 2);
}
static inline void
-dir24_8_lookup_bulk_3(void *p, const uint32_t *ips,
+dir24_8_lookup_bulk_3(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
uint64_t *next_hops, const unsigned int n)
{
struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
+ RTE_SET_USED(vrf_ids);
dir24_8_lookup_bulk(dp, ips, next_hops, n, 3);
}
static inline void
-dir24_8_lookup_bulk_uni(void *p, const uint32_t *ips,
+dir24_8_lookup_bulk_uni(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
uint64_t *next_hops, const unsigned int n)
{
struct dir24_8_tbl *dp = (struct dir24_8_tbl *)p;
@@ -224,6 +230,8 @@ dir24_8_lookup_bulk_uni(void *p, const uint32_t *ips,
uint32_t prefetch_offset = RTE_MIN(15U, n);
uint8_t nh_sz = dp->nh_sz;
+ RTE_SET_USED(vrf_ids);
+
for (i = 0; i < prefetch_offset; i++)
rte_prefetch0(get_tbl24_p(dp, ips[i], nh_sz));
for (i = 0; i < (n - prefetch_offset); i++) {
@@ -246,34 +254,35 @@ dir24_8_lookup_bulk_uni(void *p, const uint32_t *ips,
#define BSWAP_MAX_LENGTH 64
-typedef void (*dir24_8_lookup_bulk_be_cb)(void *p, const uint32_t *ips, uint64_t *next_hops,
- const unsigned int n);
+typedef void (*dir24_8_lookup_bulk_be_cb)(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
static inline void
-dir24_8_lookup_bulk_be(void *p, const uint32_t *ips, uint64_t *next_hops, const unsigned int n,
- dir24_8_lookup_bulk_be_cb cb)
+dir24_8_lookup_bulk_be(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
+ uint64_t *next_hops, const unsigned int n, dir24_8_lookup_bulk_be_cb cb)
{
uint32_t le_ips[BSWAP_MAX_LENGTH];
unsigned int i;
#if RTE_BYTE_ORDER == RTE_BIG_ENDIAN
- cb(p, ips, next_hops, n);
+ cb(p, vrf_ids, ips, next_hops, n);
#else
for (i = 0; i < n; i += BSWAP_MAX_LENGTH) {
int j;
for (j = 0; j < BSWAP_MAX_LENGTH && i + j < n; j++)
le_ips[j] = rte_be_to_cpu_32(ips[i + j]);
- cb(p, le_ips, next_hops + i, j);
+ cb(p, vrf_ids, le_ips, next_hops + i, j);
}
#endif
}
#define DECLARE_BE_LOOKUP_FN(name) \
static inline void \
-name##_be(void *p, const uint32_t *ips, uint64_t *next_hops, const unsigned int n) \
+name##_be(void *p, const uint16_t *vrf_ids, const uint32_t *ips, \
+ uint64_t *next_hops, const unsigned int n) \
{ \
- dir24_8_lookup_bulk_be(p, ips, next_hops, n, name); \
+ dir24_8_lookup_bulk_be(p, vrf_ids, ips, next_hops, n, name); \
}
DECLARE_BE_LOOKUP_FN(dir24_8_lookup_bulk_1b)
@@ -296,7 +305,7 @@ rte_fib_lookup_fn_t
dir24_8_get_lookup_fn(void *p, enum rte_fib_lookup_type type, bool be_addr);
int
-dir24_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t depth,
+dir24_8_modify(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip, uint8_t depth,
uint64_t next_hop, int op);
int
diff --git a/lib/fib/dir24_8_avx512.c b/lib/fib/dir24_8_avx512.c
index 89b43583c7..77b7a8b3de 100644
--- a/lib/fib/dir24_8_avx512.c
+++ b/lib/fib/dir24_8_avx512.c
@@ -133,14 +133,14 @@ dir24_8_vec_lookup_x8_8b(void *p, const uint32_t *ips,
#define DECLARE_VECTOR_FN(suffix, nh_type, be_addr) \
void \
-rte_dir24_8_vec_lookup_bulk_##suffix(void *p, const uint32_t *ips, uint64_t *next_hops, \
- const unsigned int n) \
+rte_dir24_8_vec_lookup_bulk_##suffix(void *p, const uint16_t *vrf_ids, \
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n) \
{ \
uint32_t i; \
for (i = 0; i < (n / 16); i++) \
dir24_8_vec_lookup_x16(p, ips + i * 16, next_hops + i * 16, sizeof(nh_type), \
be_addr); \
- dir24_8_lookup_bulk_##suffix(p, ips + i * 16, next_hops + i * 16, n - i * 16); \
+ dir24_8_lookup_bulk_##suffix(p, vrf_ids, ips + i * 16, next_hops + i * 16, n - i * 16); \
}
DECLARE_VECTOR_FN(1b, uint8_t, false)
@@ -151,21 +151,21 @@ DECLARE_VECTOR_FN(4b, uint32_t, false)
DECLARE_VECTOR_FN(4b_be, uint32_t, true)
void
-rte_dir24_8_vec_lookup_bulk_8b(void *p, const uint32_t *ips,
- uint64_t *next_hops, const unsigned int n)
+rte_dir24_8_vec_lookup_bulk_8b(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
{
uint32_t i;
for (i = 0; i < (n / 8); i++)
dir24_8_vec_lookup_x8_8b(p, ips + i * 8, next_hops + i * 8, false);
- dir24_8_lookup_bulk_8b(p, ips + i * 8, next_hops + i * 8, n - i * 8);
+ dir24_8_lookup_bulk_8b(p, vrf_ids, ips + i * 8, next_hops + i * 8, n - i * 8);
}
void
-rte_dir24_8_vec_lookup_bulk_8b_be(void *p, const uint32_t *ips,
- uint64_t *next_hops, const unsigned int n)
+rte_dir24_8_vec_lookup_bulk_8b_be(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
{
uint32_t i;
for (i = 0; i < (n / 8); i++)
dir24_8_vec_lookup_x8_8b(p, ips + i * 8, next_hops + i * 8, true);
- dir24_8_lookup_bulk_8b_be(p, ips + i * 8, next_hops + i * 8, n - i * 8);
+ dir24_8_lookup_bulk_8b_be(p, vrf_ids, ips + i * 8, next_hops + i * 8, n - i * 8);
}
diff --git a/lib/fib/dir24_8_avx512.h b/lib/fib/dir24_8_avx512.h
index 3e2bbc2490..7681cac436 100644
--- a/lib/fib/dir24_8_avx512.h
+++ b/lib/fib/dir24_8_avx512.h
@@ -6,35 +6,35 @@
#define _DIR248_AVX512_H_
void
-rte_dir24_8_vec_lookup_bulk_1b(void *p, const uint32_t *ips,
- uint64_t *next_hops, const unsigned int n);
+rte_dir24_8_vec_lookup_bulk_1b(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
void
-rte_dir24_8_vec_lookup_bulk_2b(void *p, const uint32_t *ips,
- uint64_t *next_hops, const unsigned int n);
+rte_dir24_8_vec_lookup_bulk_2b(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
void
-rte_dir24_8_vec_lookup_bulk_4b(void *p, const uint32_t *ips,
- uint64_t *next_hops, const unsigned int n);
+rte_dir24_8_vec_lookup_bulk_4b(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
void
-rte_dir24_8_vec_lookup_bulk_8b(void *p, const uint32_t *ips,
- uint64_t *next_hops, const unsigned int n);
+rte_dir24_8_vec_lookup_bulk_8b(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
void
-rte_dir24_8_vec_lookup_bulk_1b_be(void *p, const uint32_t *ips,
- uint64_t *next_hops, const unsigned int n);
+rte_dir24_8_vec_lookup_bulk_1b_be(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
void
-rte_dir24_8_vec_lookup_bulk_2b_be(void *p, const uint32_t *ips,
- uint64_t *next_hops, const unsigned int n);
+rte_dir24_8_vec_lookup_bulk_2b_be(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
void
-rte_dir24_8_vec_lookup_bulk_4b_be(void *p, const uint32_t *ips,
- uint64_t *next_hops, const unsigned int n);
+rte_dir24_8_vec_lookup_bulk_4b_be(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
void
-rte_dir24_8_vec_lookup_bulk_8b_be(void *p, const uint32_t *ips,
- uint64_t *next_hops, const unsigned int n);
+rte_dir24_8_vec_lookup_bulk_8b_be(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
#endif /* _DIR248_AVX512_H_ */
diff --git a/lib/fib/dir24_8_rvv.c b/lib/fib/dir24_8_rvv.c
index 9c14ca0481..53158e42e8 100644
--- a/lib/fib/dir24_8_rvv.c
+++ b/lib/fib/dir24_8_rvv.c
@@ -13,7 +13,7 @@
#define DECLARE_VECTOR_FN(SFX, NH_SZ) \
void \
rte_dir24_8_vec_lookup_bulk_##SFX(void *p, \
- const uint32_t *ips, uint64_t *next_hops, unsigned int n) \
+ const uint16_t *vrf_ids, const uint32_t *ips, uint64_t *next_hops, unsigned int n) \
{ \
const uint8_t idx_bits = 3 - NH_SZ; \
const uint32_t idx_mask = (1u << (3 - NH_SZ)) - 1u; \
@@ -21,6 +21,7 @@ rte_dir24_8_vec_lookup_bulk_##SFX(void *p, \
struct dir24_8_tbl *tbl = (struct dir24_8_tbl *)p; \
const uint64_t *tbl24 = tbl->tbl24; \
size_t vl; \
+ RTE_SET_USED(vrf_ids); \
for (unsigned int i = 0; i < n; i += vl) { \
vl = __riscv_vsetvl_e32m4(n - i); \
vuint32m4_t v_ips = __riscv_vle32_v_u32m4(&ips[i], vl); \
diff --git a/lib/fib/dir24_8_rvv.h b/lib/fib/dir24_8_rvv.h
index 7be99f7882..7064665d60 100644
--- a/lib/fib/dir24_8_rvv.h
+++ b/lib/fib/dir24_8_rvv.h
@@ -6,19 +6,19 @@
#define _DIR248_RVV_H_
void
-rte_dir24_8_vec_lookup_bulk_1b(void *p, const uint32_t *ips,
- uint64_t *next_hops, const unsigned int n);
+rte_dir24_8_vec_lookup_bulk_1b(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
void
-rte_dir24_8_vec_lookup_bulk_2b(void *p, const uint32_t *ips,
- uint64_t *next_hops, const unsigned int n);
+rte_dir24_8_vec_lookup_bulk_2b(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
void
-rte_dir24_8_vec_lookup_bulk_4b(void *p, const uint32_t *ips,
- uint64_t *next_hops, const unsigned int n);
+rte_dir24_8_vec_lookup_bulk_4b(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
void
-rte_dir24_8_vec_lookup_bulk_8b(void *p, const uint32_t *ips,
- uint64_t *next_hops, const unsigned int n);
+rte_dir24_8_vec_lookup_bulk_8b(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
#endif /* _DIR248_RVV_H_ */
diff --git a/lib/fib/dir8_16_8.c b/lib/fib/dir8_16_8.c
index 7207cd4748..533aaf6cce 100644
--- a/lib/fib/dir8_16_8.c
+++ b/lib/fib/dir8_16_8.c
@@ -57,13 +57,16 @@ struct dir8_16_8_rcu_dq_entry {
};
static rte_fib_lookup_fn_t
-get_scalar_fn(bool be_addr)
+get_scalar_fn(const struct dir8_16_8_tbl *dp, bool be_addr)
{
+ if (dp->num_vrfs > 1)
+ return be_addr ? dir8_16_8_vrf_lookup_bulk_be :
+ dir8_16_8_vrf_lookup_bulk;
return be_addr ? dir8_16_8_lookup_bulk_be : dir8_16_8_lookup_bulk;
}
static rte_fib_lookup_fn_t
-get_vector_fn(bool be_addr)
+get_vector_fn(const struct dir8_16_8_tbl *dp, bool be_addr)
{
#ifdef CC_AVX512_SUPPORT
if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512F) <= 0 ||
@@ -74,9 +77,13 @@ get_vector_fn(bool be_addr)
if (be_addr && rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512BW) <= 0)
return NULL;
+ if (dp->num_vrfs > 1)
+ return be_addr ? rte_dir8_16_8_vrf_vec_lookup_bulk_be :
+ rte_dir8_16_8_vrf_vec_lookup_bulk;
return be_addr ? rte_dir8_16_8_vec_lookup_bulk_be :
rte_dir8_16_8_vec_lookup_bulk;
#else
+ RTE_SET_USED(dp);
RTE_SET_USED(be_addr);
return NULL;
#endif
@@ -93,12 +100,12 @@ dir8_16_8_get_lookup_fn(void *p, enum rte_fib_lookup_type type, bool be_addr)
switch (type) {
case RTE_FIB_LOOKUP_DIR8_16_8_SCALAR:
- return get_scalar_fn(be_addr);
+ return get_scalar_fn(dp, be_addr);
case RTE_FIB_LOOKUP_DIR8_16_8_VECTOR_AVX512:
- return get_vector_fn(be_addr);
+ return get_vector_fn(dp, be_addr);
case RTE_FIB_LOOKUP_DEFAULT:
- fn = get_vector_fn(be_addr);
- return (fn != NULL) ? fn : get_scalar_fn(be_addr);
+ fn = get_vector_fn(dp, be_addr);
+ return (fn != NULL) ? fn : get_scalar_fn(dp, be_addr);
default:
return NULL;
}
@@ -336,8 +343,8 @@ install_to_fib(struct dir8_16_8_tbl *dp, int level, uint64_t base,
}
static int
-modify_fib(struct dir8_16_8_tbl *dp, struct rte_rib *rib, uint32_t ip,
- uint8_t depth, uint64_t next_hop)
+modify_fib(struct dir8_16_8_tbl *dp, struct rte_rib *rib, uint32_t tbl0_base,
+ uint32_t ip, uint8_t depth, uint64_t next_hop)
{
struct rte_rib_node *tmp = NULL;
uint64_t ledge, redge;
@@ -360,8 +367,7 @@ modify_fib(struct dir8_16_8_tbl *dp, struct rte_rib *rib, uint32_t ip,
ledge = redge + (1ULL << (32 - tmp_depth));
continue;
}
-
- ret = install_to_fib(dp, 0, 0, ledge, redge, next_hop);
+ ret = install_to_fib(dp, 0, tbl0_base, ledge, redge, next_hop);
if (ret != 0)
return ret;
@@ -376,8 +382,7 @@ modify_fib(struct dir8_16_8_tbl *dp, struct rte_rib *rib, uint32_t ip,
redge = (uint64_t)ip + (1ULL << (32 - depth));
if (ledge == redge && ledge != 0)
break;
-
- ret = install_to_fib(dp, 0, 0, ledge, redge, next_hop);
+ ret = install_to_fib(dp, 0, tbl0_base, ledge, redge, next_hop);
if (ret != 0)
return ret;
}
@@ -386,24 +391,16 @@ modify_fib(struct dir8_16_8_tbl *dp, struct rte_rib *rib, uint32_t ip,
return 0;
}
-int
-dir8_16_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t depth,
+static int
+dir8_16_8_modify_core(struct dir8_16_8_tbl *dp, struct rte_rib *rib,
+ uint32_t tbl0_base, uint64_t def_nh, uint32_t ip, uint8_t depth,
uint64_t next_hop, int op)
{
- struct dir8_16_8_tbl *dp;
- struct rte_rib *rib;
struct rte_rib_node *node;
struct rte_rib_node *parent;
int ret = 0;
uint64_t par_nh, node_nh;
- if ((fib == NULL) || (depth > RTE_FIB_MAXDEPTH))
- return -EINVAL;
-
- dp = rte_fib_get_dp(fib);
- rib = rte_fib_get_rib(fib);
- RTE_ASSERT((dp != NULL) && (rib != NULL));
-
if (next_hop > dir8_16_8_get_max_nh())
return -EINVAL;
@@ -416,8 +413,7 @@ dir8_16_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t depth,
rte_rib_get_nh(node, &node_nh);
if (node_nh == next_hop)
return 0;
-
- ret = modify_fib(dp, rib, ip, depth, next_hop);
+ ret = modify_fib(dp, rib, tbl0_base, ip, depth, next_hop);
if (ret == 0)
rte_rib_set_nh(node, next_hop);
@@ -434,7 +430,7 @@ dir8_16_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t depth,
if (par_nh == next_hop)
return 0;
}
- ret = modify_fib(dp, rib, ip, depth, next_hop);
+ ret = modify_fib(dp, rib, tbl0_base, ip, depth, next_hop);
if (ret != 0) {
rte_rib_remove(rib, ip, depth);
return ret;
@@ -450,9 +446,10 @@ dir8_16_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t depth,
rte_rib_get_nh(parent, &par_nh);
rte_rib_get_nh(node, &node_nh);
if (par_nh != node_nh)
- ret = modify_fib(dp, rib, ip, depth, par_nh);
+ ret = modify_fib(dp, rib, tbl0_base, ip, depth,
+ par_nh);
} else
- ret = modify_fib(dp, rib, ip, depth, dp->def_nh);
+ ret = modify_fib(dp, rib, tbl0_base, ip, depth, def_nh);
if (ret == 0)
rte_rib_remove(rib, ip, depth);
return ret;
@@ -463,15 +460,38 @@ dir8_16_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t depth,
return -EINVAL;
}
+int
+dir8_16_8_modify(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip, uint8_t depth,
+ uint64_t next_hop, int op)
+{
+ struct dir8_16_8_tbl *dp;
+ struct rte_rib *rib;
+
+ if ((fib == NULL) || (depth > RTE_FIB_MAXDEPTH))
+ return -EINVAL;
+
+ dp = rte_fib_get_dp(fib);
+ RTE_ASSERT(dp != NULL);
+
+ if (vrf_id >= dp->num_vrfs)
+ return -EINVAL;
+
+ rib = rte_fib_vrf_get_rib(fib, vrf_id);
+ RTE_ASSERT(rib != NULL);
+
+ return dir8_16_8_modify_core(dp, rib,
+ (uint32_t)vrf_id * DIR8_16_8_TBL0_NUM_ENT,
+ dp->def_nh[vrf_id], ip, depth, next_hop, op);
+}
+
void *
dir8_16_8_create(const char *name, int socket_id, struct rte_fib_conf *fib_conf)
{
char mem_name[DIR8_16_8_NAMESIZE];
struct dir8_16_8_tbl *dp;
- uint64_t def_nh;
- uint32_t num_tbl16, num_tbl8;
- uint32_t i;
- uint64_t tbl16_sz, tbl8_sz;
+ uint64_t def_nh, tbl16_sz, tbl8_sz;
+ uint32_t num_tbl16, num_tbl8, i;
+ uint16_t num_vrfs, v;
if ((name == NULL) || (fib_conf == NULL) || (fib_conf->dir8_16_8.num_tbl16 == 0) ||
(fib_conf->dir8_16_8.num_tbl8 == 0) ||
@@ -480,27 +500,56 @@ dir8_16_8_create(const char *name, int socket_id, struct rte_fib_conf *fib_conf)
return NULL;
}
+ num_vrfs = RTE_MAX(fib_conf->max_vrfs, 1);
def_nh = fib_conf->default_nh;
num_tbl16 = fib_conf->dir8_16_8.num_tbl16;
num_tbl8 = fib_conf->dir8_16_8.num_tbl8;
+ /* Validate the per-VRF default nexthops, if provided. */
+ if (fib_conf->vrf_default_nh != NULL) {
+ for (v = 0; v < num_vrfs; v++) {
+ if (fib_conf->vrf_default_nh[v] > dir8_16_8_get_max_nh()) {
+ rte_errno = EINVAL;
+ return NULL;
+ }
+ }
+ }
+
dp = rte_zmalloc_socket(name, sizeof(struct dir8_16_8_tbl) +
- DIR8_16_8_TBL0_NUM_ENT * sizeof(uint64_t), RTE_CACHE_LINE_SIZE, socket_id);
+ (size_t)num_vrfs * DIR8_16_8_TBL0_NUM_ENT * sizeof(uint64_t),
+ RTE_CACHE_LINE_SIZE, socket_id);
if (dp == NULL) {
rte_errno = ENOMEM;
return NULL;
}
+ dp->num_vrfs = num_vrfs;
- /* Init first level table with the default nexthop. */
- for (i = 0; i < DIR8_16_8_TBL0_NUM_ENT; i++)
- dp->tbl0[i] = def_nh << 1;
+ /* Per-VRF default nexthop array. */
+ snprintf(mem_name, sizeof(mem_name), "DEFNH_%p", dp);
+ dp->def_nh = rte_zmalloc_socket(mem_name,
+ (size_t)num_vrfs * sizeof(uint64_t),
+ RTE_CACHE_LINE_SIZE, socket_id);
+ if (dp->def_nh == NULL) {
+ rte_errno = ENOMEM;
+ goto free_dp;
+ }
+ for (v = 0; v < num_vrfs; v++)
+ dp->def_nh[v] = (fib_conf->vrf_default_nh != NULL) ?
+ fib_conf->vrf_default_nh[v] : def_nh;
+
+ /* Init each VRF's tbl0 block with that VRF's default nexthop. */
+ for (v = 0; v < num_vrfs; v++) {
+ uint32_t base = (uint32_t)v * DIR8_16_8_TBL0_NUM_ENT;
+ for (i = 0; i < DIR8_16_8_TBL0_NUM_ENT; i++)
+ dp->tbl0[base + i] = dp->def_nh[v] << 1;
+ }
snprintf(mem_name, sizeof(mem_name), "TBL16_%p", dp);
tbl16_sz = (uint64_t)DIR8_16_8_TBL16_GRP_NUM_ENT * sizeof(uint64_t) * num_tbl16;
dp->tbl16 = rte_zmalloc_socket(mem_name, tbl16_sz, RTE_CACHE_LINE_SIZE, socket_id);
if (dp->tbl16 == NULL) {
rte_errno = ENOMEM;
- goto free_dp;
+ goto free_defnh;
}
snprintf(mem_name, sizeof(mem_name), "TBL8_%p", dp);
@@ -532,7 +581,6 @@ dir8_16_8_create(const char *name, int socket_id, struct rte_fib_conf *fib_conf)
for (i = 0; i < num_tbl8; i++)
dp->tbl8_pool[i] = i;
- dp->def_nh = def_nh;
dp->number_tbl16s = num_tbl16;
dp->number_tbl8s = num_tbl8;
@@ -544,6 +592,8 @@ dir8_16_8_create(const char *name, int socket_id, struct rte_fib_conf *fib_conf)
rte_free(dp->tbl8);
free_tbl16:
rte_free(dp->tbl16);
+free_defnh:
+ rte_free(dp->def_nh);
free_dp:
rte_free(dp);
return NULL;
@@ -558,6 +608,7 @@ dir8_16_8_free(void *p)
return;
rte_rcu_qsbr_dq_delete(dp->dq);
+ rte_free(dp->def_nh);
rte_free(dp->tbl8_pool);
rte_free(dp->tbl16_pool);
rte_free(dp->tbl8);
diff --git a/lib/fib/dir8_16_8.h b/lib/fib/dir8_16_8.h
index 31ca5162de..1ea867d4ba 100644
--- a/lib/fib/dir8_16_8.h
+++ b/lib/fib/dir8_16_8.h
@@ -36,7 +36,8 @@ struct dir8_16_8_tbl {
uint32_t cur_tbl16s; /**< Current number of tbl16 groups. */
uint32_t number_tbl8s; /**< Total number of tbl8 groups. */
uint32_t cur_tbl8s; /**< Current number of tbl8 groups. */
- uint64_t def_nh; /**< Default next hop. */
+ uint16_t num_vrfs; /**< Number of VRFs (1 = single VRF). */
+ uint64_t *def_nh; /**< Per-VRF default nexthops (num_vrfs). */
uint64_t *tbl16; /**< Second level table. */
uint64_t *tbl8; /**< Third level table. */
uint32_t *tbl16_pool; /**< Stack of free tbl16 group idxes. */
@@ -47,7 +48,7 @@ struct dir8_16_8_tbl {
enum rte_fib_qsbr_mode rcu_mode;/* Blocking, defer queue. */
struct rte_rcu_qsbr *v; /* RCU QSBR variable. */
struct rte_rcu_qsbr_dq *dq; /* RCU QSBR defer queue. */
- /* First level table (top 8 bits). */
+ /* First level table (top 8 bits), sized num_vrfs * 256. */
alignas(RTE_CACHE_LINE_SIZE) uint64_t tbl0[];
};
@@ -83,13 +84,14 @@ dir8_16_8_is_entry_extended(uint64_t ent)
}
static inline void
-dir8_16_8_lookup_bulk(void *p, const uint32_t *ips,
+dir8_16_8_lookup_bulk(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
uint64_t *next_hops, const unsigned int n)
{
struct dir8_16_8_tbl *dp = (struct dir8_16_8_tbl *)p;
uint64_t tmp;
uint32_t i;
+ RTE_SET_USED(vrf_ids);
for (i = 0; i < n; i++) {
tmp = dp->tbl0[dir8_16_8_get_tbl0_idx(ips[i])];
/* Prefixes are usually longer than /8. */
@@ -104,11 +106,11 @@ dir8_16_8_lookup_bulk(void *p, const uint32_t *ips,
#define DIR8_16_8_BSWAP_MAX_LENGTH 64
static inline void
-dir8_16_8_lookup_bulk_be(void *p, const uint32_t *ips,
+dir8_16_8_lookup_bulk_be(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
uint64_t *next_hops, const unsigned int n)
{
#if RTE_BYTE_ORDER == RTE_BIG_ENDIAN
- dir8_16_8_lookup_bulk(p, ips, next_hops, n);
+ dir8_16_8_lookup_bulk(p, vrf_ids, ips, next_hops, n);
#else
uint32_t le_ips[DIR8_16_8_BSWAP_MAX_LENGTH];
unsigned int i;
@@ -118,7 +120,54 @@ dir8_16_8_lookup_bulk_be(void *p, const uint32_t *ips,
for (j = 0; j < DIR8_16_8_BSWAP_MAX_LENGTH && i + j < n; j++)
le_ips[j] = rte_be_to_cpu_32(ips[i + j]);
- dir8_16_8_lookup_bulk(p, le_ips, next_hops + i, j);
+ dir8_16_8_lookup_bulk(p, vrf_ids, le_ips, next_hops + i, j);
+ }
+#endif
+}
+
+static inline void
+dir8_16_8_vrf_lookup_bulk(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
+ uint64_t *next_hops, const unsigned int n)
+{
+ struct dir8_16_8_tbl *dp = (struct dir8_16_8_tbl *)p;
+ uint64_t tmp;
+ uint32_t i;
+
+ /* A NULL vrf_ids array selects VRF 0 for every IP. */
+ if (unlikely(vrf_ids == NULL)) {
+ dir8_16_8_lookup_bulk(p, NULL, ips, next_hops, n);
+ return;
+ }
+
+ for (i = 0; i < n; i++) {
+ tmp = dp->tbl0[vrf_ids[i] * DIR8_16_8_TBL0_NUM_ENT +
+ dir8_16_8_get_tbl0_idx(ips[i])];
+ if (dir8_16_8_is_entry_extended(tmp))
+ tmp = dp->tbl16[dir8_16_8_get_tbl16_idx(tmp, ips[i])];
+ if (unlikely(dir8_16_8_is_entry_extended(tmp)))
+ tmp = dp->tbl8[dir8_16_8_get_tbl8_idx(tmp, ips[i])];
+ next_hops[i] = tmp >> 1;
+ }
+}
+
+static inline void
+dir8_16_8_vrf_lookup_bulk_be(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
+{
+#if RTE_BYTE_ORDER == RTE_BIG_ENDIAN
+ dir8_16_8_vrf_lookup_bulk(p, vrf_ids, ips, next_hops, n);
+#else
+ uint32_t le_ips[DIR8_16_8_BSWAP_MAX_LENGTH];
+ unsigned int i;
+
+ for (i = 0; i < n; i += DIR8_16_8_BSWAP_MAX_LENGTH) {
+ unsigned int j;
+ for (j = 0; j < DIR8_16_8_BSWAP_MAX_LENGTH && i + j < n; j++)
+ le_ips[j] = rte_be_to_cpu_32(ips[i + j]);
+
+ dir8_16_8_vrf_lookup_bulk(p,
+ (vrf_ids != NULL) ? vrf_ids + i : NULL,
+ le_ips, next_hops + i, j);
}
#endif
}
@@ -133,7 +182,7 @@ rte_fib_lookup_fn_t
dir8_16_8_get_lookup_fn(void *p, enum rte_fib_lookup_type type, bool be_addr);
int
-dir8_16_8_modify(struct rte_fib *fib, uint32_t ip, uint8_t depth,
+dir8_16_8_modify(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip, uint8_t depth,
uint64_t next_hop, int op);
int
diff --git a/lib/fib/dir8_16_8_avx512.c b/lib/fib/dir8_16_8_avx512.c
index c147451252..2a9c6b6a15 100644
--- a/lib/fib/dir8_16_8_avx512.c
+++ b/lib/fib/dir8_16_8_avx512.c
@@ -8,6 +8,9 @@
#include "dir8_16_8.h"
#include "dir8_16_8_avx512.h"
+/* VRF IDs used when the caller passes no vrf_ids array: VRF 0 for every IP. */
+static const rte_xmm_t dir8_16_8_def_vrf_ids;
+
static inline __m256i
dir8_16_8_vec_load_ips(const uint32_t *ips, bool be_addr)
{
@@ -21,9 +24,14 @@ dir8_16_8_vec_load_ips(const uint32_t *ips, bool be_addr)
}
static inline __m256i
-dir8_16_8_vec_get_tbl0_idx(__m256i ip_vec)
+dir8_16_8_vec_get_tbl0_idx(const uint16_t *vrf_ids, __m256i ip_vec)
{
- return _mm256_srli_epi32(ip_vec, 24);
+ __m256i vrf_vec = _mm256_cvtepu16_epi32(
+ _mm_loadu_si128((const void *)vrf_ids));
+
+ /* vrf_id * DIR8_16_8_TBL0_NUM_ENT + (ip >> 24) */
+ return _mm256_add_epi32(_mm256_slli_epi32(vrf_vec, 8),
+ _mm256_srli_epi32(ip_vec, 24));
}
/* Second lookup stage input for 8 IPs, filled by the first stage. */
@@ -43,11 +51,11 @@ struct dir8_16_8_vec_stage {
* some CPUs a gather costs much more than the 8 loads it replaces.
*/
static inline void
-dir8_16_8_vec_prefetch_x8(const struct dir8_16_8_tbl *dp, const uint32_t *ips, bool be_addr,
- struct dir8_16_8_vec_stage *st)
+dir8_16_8_vec_prefetch_x8(const struct dir8_16_8_tbl *dp, const uint16_t *vrf_ids,
+ const uint32_t *ips, bool be_addr, struct dir8_16_8_vec_stage *st)
{
const __m256i ip_vec = dir8_16_8_vec_load_ips(ips, be_addr);
- const rte_ymm_t idx0 = { .y = dir8_16_8_vec_get_tbl0_idx(ip_vec) };
+ const rte_ymm_t idx0 = { .y = dir8_16_8_vec_get_tbl0_idx(vrf_ids, ip_vec) };
const rte_ymm_t ip = { .y = ip_vec };
const uint64_t *tbl0 = dp->tbl0;
const uint64_t *tbl16 = dp->tbl16;
@@ -90,7 +98,8 @@ dir8_16_8_vec_lookup_x8(const struct dir8_16_8_tbl *dp, const struct dir8_16_8_v
}
static inline void
-vec_lookup_bulk(void *p, const uint32_t *ips, uint64_t *next_hops, unsigned int n, bool be_addr)
+vec_lookup_bulk(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
+ uint64_t *next_hops, unsigned int n, bool be_addr)
{
const struct dir8_16_8_tbl *dp = p;
struct dir8_16_8_vec_stage stage[3];
@@ -98,13 +107,26 @@ vec_lookup_bulk(void *p, const uint32_t *ips, uint64_t *next_hops, unsigned int
struct dir8_16_8_vec_stage *st1 = &stage[1];
struct dir8_16_8_vec_stage *st2 = &stage[2];
struct dir8_16_8_vec_stage *tmp;
+ const uint16_t *vrfs = vrf_ids;
+ unsigned int vrf_inc = 1;
unsigned int i = 0;
+ /*
+ * A NULL vrf_ids array selects VRF 0 for every IP: keep reading
+ * the zeroed default VRF vector instead of checking vrf_ids per batch.
+ */
+ if (vrf_ids == NULL) {
+ vrfs = dir8_16_8_def_vrf_ids.u16;
+ vrf_inc = 0;
+ }
+
if (n >= 16) {
- dir8_16_8_vec_prefetch_x8(dp, ips, be_addr, st0);
- dir8_16_8_vec_prefetch_x8(dp, &ips[8], be_addr, st1);
+ dir8_16_8_vec_prefetch_x8(dp, vrfs, ips, be_addr, st0);
+ dir8_16_8_vec_prefetch_x8(dp, &vrfs[8 * vrf_inc], &ips[8],
+ be_addr, st1);
for (; i + 24 <= n; i += 8) {
- dir8_16_8_vec_prefetch_x8(dp, &ips[i + 16], be_addr, st2);
+ dir8_16_8_vec_prefetch_x8(dp, &vrfs[(i + 16) * vrf_inc],
+ &ips[i + 16], be_addr, st2);
dir8_16_8_vec_lookup_x8(dp, st0, &ips[i], be_addr, &next_hops[i]);
tmp = st0;
st0 = st1;
@@ -115,28 +137,47 @@ vec_lookup_bulk(void *p, const uint32_t *ips, uint64_t *next_hops, unsigned int
dir8_16_8_vec_lookup_x8(dp, st1, &ips[i + 8], be_addr, &next_hops[i + 8]);
i += 16;
} else if (n >= 8) {
- dir8_16_8_vec_prefetch_x8(dp, ips, be_addr, st0);
+ dir8_16_8_vec_prefetch_x8(dp, vrfs, ips, be_addr, st0);
dir8_16_8_vec_lookup_x8(dp, st0, ips, be_addr, next_hops);
i = 8;
}
- /* Scalar lookup of the last n % 8 IPs. */
+ /* Scalar lookup of the last n % 8 IPs, the default VRF vector covers them. */
+ vrfs += i * vrf_inc;
if (be_addr)
- dir8_16_8_lookup_bulk_be(p, &ips[i], &next_hops[i], n - i);
+ dir8_16_8_vrf_lookup_bulk_be(p, vrfs, &ips[i], &next_hops[i],
+ n - i);
else
- dir8_16_8_lookup_bulk(p, &ips[i], &next_hops[i], n - i);
+ dir8_16_8_vrf_lookup_bulk(p, vrfs, &ips[i], &next_hops[i],
+ n - i);
+}
+
+void
+rte_dir8_16_8_vec_lookup_bulk(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
+{
+ RTE_SET_USED(vrf_ids);
+ vec_lookup_bulk(p, NULL, ips, next_hops, n, false);
+}
+
+void
+rte_dir8_16_8_vec_lookup_bulk_be(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
+{
+ RTE_SET_USED(vrf_ids);
+ vec_lookup_bulk(p, NULL, ips, next_hops, n, true);
}
void
-rte_dir8_16_8_vec_lookup_bulk(void *p, const uint32_t *ips,
- uint64_t *next_hops, const unsigned int n)
+rte_dir8_16_8_vrf_vec_lookup_bulk(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
{
- vec_lookup_bulk(p, ips, next_hops, n, false);
+ vec_lookup_bulk(p, vrf_ids, ips, next_hops, n, false);
}
void
-rte_dir8_16_8_vec_lookup_bulk_be(void *p, const uint32_t *ips,
- uint64_t *next_hops, const unsigned int n)
+rte_dir8_16_8_vrf_vec_lookup_bulk_be(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
{
- vec_lookup_bulk(p, ips, next_hops, n, true);
+ vec_lookup_bulk(p, vrf_ids, ips, next_hops, n, true);
}
diff --git a/lib/fib/dir8_16_8_avx512.h b/lib/fib/dir8_16_8_avx512.h
index d468481029..d1ab4ab128 100644
--- a/lib/fib/dir8_16_8_avx512.h
+++ b/lib/fib/dir8_16_8_avx512.h
@@ -6,11 +6,19 @@
#define _DIR8_16_8_AVX512_H_
void
-rte_dir8_16_8_vec_lookup_bulk(void *p, const uint32_t *ips,
- uint64_t *next_hops, const unsigned int n);
+rte_dir8_16_8_vec_lookup_bulk(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
void
-rte_dir8_16_8_vec_lookup_bulk_be(void *p, const uint32_t *ips,
- uint64_t *next_hops, const unsigned int n);
+rte_dir8_16_8_vec_lookup_bulk_be(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
+
+void
+rte_dir8_16_8_vrf_vec_lookup_bulk(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
+
+void
+rte_dir8_16_8_vrf_vec_lookup_bulk_be(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
#endif /* _DIR8_16_8_AVX512_H_ */
diff --git a/lib/fib/rte_fib.c b/lib/fib/rte_fib.c
index fcc28899cb..795031edcd 100644
--- a/lib/fib/rte_fib.c
+++ b/lib/fib/rte_fib.c
@@ -42,23 +42,24 @@ struct rte_fib {
char name[RTE_FIB_NAMESIZE];
enum rte_fib_type type; /**< Type of FIB struct */
unsigned int flags; /**< Flags */
- struct rte_rib *rib; /**< RIB helper datastructure */
+ struct rte_rib **ribs; /**< Per-VRF RIBs; ribs[0] is the main RIB. */
void *dp; /**< pointer to the dataplane struct*/
rte_fib_lookup_fn_t lookup; /**< FIB lookup function */
rte_fib_modify_fn_t modify; /**< modify FIB datastructure */
uint64_t def_nh;
+ uint16_t max_vrfs; /**< Number of VRFs (0/1 = single). */
};
static void
-dummy_lookup(void *fib_p, const uint32_t *ips, uint64_t *next_hops,
- const unsigned int n)
+dummy_lookup(void *fib_p, const uint16_t *vrf_ids __rte_unused,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
{
unsigned int i;
struct rte_fib *fib = fib_p;
struct rte_rib_node *node;
for (i = 0; i < n; i++) {
- node = rte_rib_lookup(fib->rib, ips[i]);
+ node = rte_rib_lookup(fib->ribs[0], ips[i]);
if (node != NULL)
rte_rib_get_nh(node, &next_hops[i]);
else
@@ -67,26 +68,26 @@ dummy_lookup(void *fib_p, const uint32_t *ips, uint64_t *next_hops,
}
static int
-dummy_modify(struct rte_fib *fib, uint32_t ip, uint8_t depth,
- uint64_t next_hop, int op)
+dummy_modify(struct rte_fib *fib, uint16_t vrf_id __rte_unused,
+ uint32_t ip, uint8_t depth, uint64_t next_hop, int op)
{
struct rte_rib_node *node;
if ((fib == NULL) || (depth > RTE_FIB_MAXDEPTH))
return -EINVAL;
- node = rte_rib_lookup_exact(fib->rib, ip, depth);
+ node = rte_rib_lookup_exact(fib->ribs[0], ip, depth);
switch (op) {
case RTE_FIB_ADD:
if (node == NULL)
- node = rte_rib_insert(fib->rib, ip, depth);
+ node = rte_rib_insert(fib->ribs[0], ip, depth);
if (node == NULL)
return -rte_errno;
return rte_rib_set_nh(node, next_hop);
case RTE_FIB_DEL:
if (node == NULL)
return -ENOENT;
- rte_rib_remove(fib->rib, ip, depth);
+ rte_rib_remove(fib->ribs[0], ip, depth);
return 0;
}
return -EINVAL;
@@ -118,7 +119,8 @@ init_dataplane(struct rte_fib *fib, __rte_unused int socket_id,
if (fib->dp == NULL)
return -rte_errno;
fib->lookup = dir8_16_8_get_lookup_fn(fib->dp,
- RTE_FIB_LOOKUP_DEFAULT, !!(fib->flags & RTE_FIB_F_LOOKUP_NETWORK_ORDER));
+ RTE_FIB_LOOKUP_DEFAULT,
+ !!(fib->flags & RTE_FIB_F_LOOKUP_NETWORK_ORDER));
fib->modify = dir8_16_8_modify;
return 0;
default:
@@ -134,7 +136,7 @@ rte_fib_add(struct rte_fib *fib, uint32_t ip, uint8_t depth, uint64_t next_hop)
if ((fib == NULL) || (fib->modify == NULL) ||
(depth > RTE_FIB_MAXDEPTH))
return -EINVAL;
- return fib->modify(fib, ip, depth, next_hop, RTE_FIB_ADD);
+ return fib->modify(fib, 0, ip, depth, next_hop, RTE_FIB_ADD);
}
RTE_EXPORT_SYMBOL(rte_fib_delete)
@@ -144,7 +146,7 @@ rte_fib_delete(struct rte_fib *fib, uint32_t ip, uint8_t depth)
if ((fib == NULL) || (fib->modify == NULL) ||
(depth > RTE_FIB_MAXDEPTH))
return -EINVAL;
- return fib->modify(fib, ip, depth, 0, RTE_FIB_DEL);
+ return fib->modify(fib, 0, ip, depth, 0, RTE_FIB_DEL);
}
RTE_EXPORT_SYMBOL(rte_fib_lookup_bulk)
@@ -155,10 +157,65 @@ rte_fib_lookup_bulk(struct rte_fib *fib, uint32_t *ips,
FIB_RETURN_IF_TRUE(((fib == NULL) || (ips == NULL) ||
(next_hops == NULL) || (fib->lookup == NULL)), -EINVAL);
- fib->lookup(fib->dp, ips, next_hops, n);
+ /* A NULL vrf_ids array selects VRF 0 for every IP. */
+ fib->lookup(fib->dp, NULL, ips, next_hops, n);
+ return 0;
+}
+
+static void
+free_dataplane(struct rte_fib *fib);
+
+/*
+ * Allocate the RIB array and create the FIB's RIBs: ribs[0] is the main RIB
+ * (named after the FIB), plus one independent RIB per extra VRF. A single-VRF
+ * FIB still keeps a one-element array so ribs[0] is always the main RIB.
+ */
+static int
+fib_create_ribs(struct rte_fib *fib, const char *name,
+ const struct rte_rib_conf *rib_conf, int socket_id)
+{
+ uint16_t v, num_vrfs = RTE_MAX(fib->max_vrfs, 1);
+
+ fib->ribs = rte_zmalloc_socket(NULL,
+ (size_t)num_vrfs * sizeof(struct rte_rib *),
+ RTE_CACHE_LINE_SIZE, socket_id);
+ if (fib->ribs == NULL)
+ return -ENOMEM;
+
+ fib->ribs[0] = rte_rib_create(name, socket_id, rib_conf);
+ if (fib->ribs[0] == NULL)
+ return -rte_errno;
+
+ for (v = 1; v < num_vrfs; v++) {
+ /*
+ * Derive a short unique RIB name from the FIB pointer: the
+ * user name plus a "_vN" suffix could overflow the RIB name
+ * length limit.
+ */
+ char rib_name[RTE_FIB_NAMESIZE];
+
+ snprintf(rib_name, sizeof(rib_name), "%p-v%u", (void *)fib, v);
+ fib->ribs[v] = rte_rib_create(rib_name, socket_id, rib_conf);
+ if (fib->ribs[v] == NULL)
+ return -rte_errno;
+ }
return 0;
}
+/* Free every RIB and the RIB array. */
+static void
+fib_free_ribs(struct rte_fib *fib)
+{
+ uint16_t v, num_vrfs = RTE_MAX(fib->max_vrfs, 1);
+
+ if (fib->ribs == NULL)
+ return;
+ for (v = 0; v < num_vrfs; v++)
+ rte_rib_free(fib->ribs[v]);
+ rte_free(fib->ribs);
+ fib->ribs = NULL;
+}
+
RTE_EXPORT_SYMBOL(rte_fib_create)
struct rte_fib *
rte_fib_create(const char *name, int socket_id, struct rte_fib_conf *conf)
@@ -166,15 +223,17 @@ rte_fib_create(const char *name, int socket_id, struct rte_fib_conf *conf)
char mem_name[RTE_FIB_NAMESIZE];
int ret;
struct rte_fib *fib = NULL;
- struct rte_rib *rib = NULL;
struct rte_tailq_entry *te;
+ struct rte_tailq_entry *tmp_te;
struct rte_fib_list *fib_list;
struct rte_rib_conf rib_conf;
/* Check user arguments. */
if ((name == NULL) || (conf == NULL) || (conf->max_routes < 0) ||
(conf->flags & ~RTE_FIB_ALLOWED_FLAGS) ||
- (conf->type > RTE_FIB_DIR8_16_8)) {
+ (conf->type > RTE_FIB_DIR8_16_8) ||
+ (conf->max_vrfs >= 2 &&
+ conf->type != RTE_FIB_DIR8_16_8)) {
rte_errno = EINVAL;
return NULL;
}
@@ -182,29 +241,7 @@ rte_fib_create(const char *name, int socket_id, struct rte_fib_conf *conf)
rib_conf.ext_sz = conf->rib_ext_sz;
rib_conf.max_nodes = conf->max_routes * 2;
- rib = rte_rib_create(name, socket_id, &rib_conf);
- if (rib == NULL) {
- FIB_LOG(ERR,
- "Can not allocate RIB %s", name);
- return NULL;
- }
-
snprintf(mem_name, sizeof(mem_name), "FIB_%s", name);
- fib_list = RTE_TAILQ_CAST(rte_fib_tailq.head, rte_fib_list);
-
- rte_mcfg_tailq_write_lock();
-
- /* guarantee there's no existing */
- TAILQ_FOREACH(te, fib_list, next) {
- fib = (struct rte_fib *)te->data;
- if (strncmp(name, fib->name, RTE_FIB_NAMESIZE) == 0)
- break;
- }
- fib = NULL;
- if (te != NULL) {
- rte_errno = EEXIST;
- goto exit;
- }
/* allocate tailq entry */
te = rte_zmalloc("FIB_TAILQ_ENTRY", sizeof(*te), 0);
@@ -212,7 +249,7 @@ rte_fib_create(const char *name, int socket_id, struct rte_fib_conf *conf)
FIB_LOG(ERR,
"Can not allocate tailq entry for FIB %s", name);
rte_errno = ENOMEM;
- goto exit;
+ return NULL;
}
/* Allocate memory to store the FIB data structures. */
@@ -225,10 +262,11 @@ rte_fib_create(const char *name, int socket_id, struct rte_fib_conf *conf)
}
rte_strlcpy(fib->name, name, sizeof(fib->name));
- fib->rib = rib;
fib->type = conf->type;
fib->flags = conf->flags;
fib->def_nh = conf->default_nh;
+ fib->max_vrfs = conf->max_vrfs;
+
ret = init_dataplane(fib, socket_id, conf);
if (ret < 0) {
FIB_LOG(ERR,
@@ -238,6 +276,27 @@ rte_fib_create(const char *name, int socket_id, struct rte_fib_conf *conf)
goto free_fib;
}
+ ret = fib_create_ribs(fib, name, &rib_conf, socket_id);
+ if (ret < 0) {
+ FIB_LOG(ERR, "Can not allocate RIBs for FIB %s", name);
+ rte_errno = -ret;
+ goto free_ribs;
+ }
+
+ fib_list = RTE_TAILQ_CAST(rte_fib_tailq.head, rte_fib_list);
+
+ rte_mcfg_tailq_write_lock();
+
+ /* guarantee there's no existing */
+ TAILQ_FOREACH(tmp_te, fib_list, next) {
+ if (strncmp(name, ((struct rte_fib *)tmp_te->data)->name,
+ RTE_FIB_NAMESIZE) == 0) {
+ rte_errno = EEXIST;
+ rte_mcfg_tailq_write_unlock();
+ goto free_ribs;
+ }
+ }
+
te->data = (void *)fib;
TAILQ_INSERT_TAIL(fib_list, te, next);
@@ -245,13 +304,13 @@ rte_fib_create(const char *name, int socket_id, struct rte_fib_conf *conf)
return fib;
+free_ribs:
+ fib_free_ribs(fib);
+ free_dataplane(fib);
free_fib:
rte_free(fib);
free_te:
rte_free(te);
-exit:
- rte_mcfg_tailq_write_unlock();
- rte_rib_free(rib);
return NULL;
}
@@ -324,7 +383,7 @@ rte_fib_free(struct rte_fib *fib)
rte_mcfg_tailq_write_unlock();
free_dataplane(fib);
- rte_rib_free(fib->rib);
+ fib_free_ribs(fib);
rte_free(fib);
rte_free(te);
}
@@ -340,7 +399,7 @@ RTE_EXPORT_SYMBOL(rte_fib_get_rib)
struct rte_rib *
rte_fib_get_rib(struct rte_fib *fib)
{
- return (fib == NULL) ? NULL : fib->rib;
+ return (fib == NULL) ? NULL : fib->ribs[0];
}
RTE_EXPORT_SYMBOL(rte_fib_select_lookup)
@@ -386,3 +445,48 @@ rte_fib_rcu_qsbr_add(struct rte_fib *fib, struct rte_fib_rcu_config *cfg)
return -ENOTSUP;
}
}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fib_vrf_add, 26.11)
+int
+rte_fib_vrf_add(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip,
+ uint8_t depth, uint64_t next_hop)
+{
+ if ((fib == NULL) || (fib->modify == NULL) ||
+ (depth > RTE_FIB_MAXDEPTH) ||
+ (vrf_id >= RTE_MAX(fib->max_vrfs, 1)))
+ return -EINVAL;
+ return fib->modify(fib, vrf_id, ip, depth, next_hop, RTE_FIB_ADD);
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fib_vrf_delete, 26.11)
+int
+rte_fib_vrf_delete(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip,
+ uint8_t depth)
+{
+ if ((fib == NULL) || (fib->modify == NULL) ||
+ (depth > RTE_FIB_MAXDEPTH) ||
+ (vrf_id >= RTE_MAX(fib->max_vrfs, 1)))
+ return -EINVAL;
+ return fib->modify(fib, vrf_id, ip, depth, 0, RTE_FIB_DEL);
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fib_vrf_lookup_bulk, 26.11)
+int
+rte_fib_vrf_lookup_bulk(struct rte_fib *fib, const uint16_t *vrf_ids,
+ uint32_t *ips, uint64_t *next_hops, int n)
+{
+ FIB_RETURN_IF_TRUE(((fib == NULL) || (ips == NULL) ||
+ (next_hops == NULL) || (fib->lookup == NULL)), -EINVAL);
+
+ fib->lookup(fib->dp, vrf_ids, ips, next_hops, n);
+ return 0;
+}
+
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_fib_vrf_get_rib, 26.11)
+struct rte_rib *
+rte_fib_vrf_get_rib(struct rte_fib *fib, uint16_t vrf_id)
+{
+ if ((fib == NULL) || (vrf_id >= RTE_MAX(fib->max_vrfs, 1)))
+ return NULL;
+ return fib->ribs[vrf_id];
+}
diff --git a/lib/fib/rte_fib.h b/lib/fib/rte_fib.h
index 68ee6f0652..948bef7600 100644
--- a/lib/fib/rte_fib.h
+++ b/lib/fib/rte_fib.h
@@ -54,11 +54,11 @@ enum rte_fib_type {
};
/** Modify FIB function */
-typedef int (*rte_fib_modify_fn_t)(struct rte_fib *fib, uint32_t ip,
- uint8_t depth, uint64_t next_hop, int op);
+typedef int (*rte_fib_modify_fn_t)(struct rte_fib *fib, uint16_t vrf_id,
+ uint32_t ip, uint8_t depth, uint64_t next_hop, int op);
/** FIB bulk lookup function */
-typedef void (*rte_fib_lookup_fn_t)(void *fib, const uint32_t *ips,
- uint64_t *next_hops, const unsigned int n);
+typedef void (*rte_fib_lookup_fn_t)(void *fib, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n);
enum rte_fib_op {
RTE_FIB_ADD,
@@ -123,6 +123,16 @@ struct rte_fib_conf {
} dir8_16_8;
};
unsigned int flags; /**< Optional feature flags from RTE_FIB_F_* */
+ /**
+ * Number of VRFs to support. 0 or 1 means a single (default) VRF.
+ * Values >= 2 require RTE_FIB_DIR8_16_8.
+ */
+ uint16_t max_vrfs;
+ /**
+ * Array of per-VRF default nexthops, indexed by VRF ID (max_vrfs
+ * entries). NULL means use default_nh for every VRF.
+ */
+ uint64_t *vrf_default_nh;
};
/** FIB RCU QSBR configuration structure. */
@@ -237,6 +247,86 @@ rte_fib_delete(struct rte_fib *fib, uint32_t ip, uint8_t depth);
int
rte_fib_lookup_bulk(struct rte_fib *fib, uint32_t *ips,
uint64_t *next_hops, int n);
+
+/**
+ * Add a route to a specific VRF of the FIB.
+ *
+ * @param fib
+ * FIB object handle
+ * @param vrf_id
+ * VRF ID (0 to max(max_vrfs, 1) - 1)
+ * @param ip
+ * IPv4 prefix address to be added to the FIB
+ * @param depth
+ * Prefix length
+ * @param next_hop
+ * Next hop to be added to the FIB
+ * @return
+ * 0 on success, negative value otherwise
+ */
+__rte_experimental
+int
+rte_fib_vrf_add(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip,
+ uint8_t depth, uint64_t next_hop);
+
+/**
+ * Delete a route from a specific VRF of the FIB.
+ *
+ * @param fib
+ * FIB object handle
+ * @param vrf_id
+ * VRF ID (0 to max(max_vrfs, 1) - 1)
+ * @param ip
+ * IPv4 prefix address to be deleted from the FIB
+ * @param depth
+ * Prefix length
+ * @return
+ * 0 on success, negative value otherwise
+ */
+__rte_experimental
+int
+rte_fib_vrf_delete(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip,
+ uint8_t depth);
+
+/**
+ * Lookup multiple IP addresses, each in its own VRF, in the FIB.
+ *
+ * @param fib
+ * FIB object handle
+ * @param vrf_ids
+ * Array of VRF IDs, one per IP: ips[i] is looked up in VRF vrf_ids[i].
+ * Each entry must be in the range 0 to max(max_vrfs, 1) - 1.
+ * A NULL pointer is treated as VRF 0 for every IP.
+ * @param ips
+ * Array of IPs to be looked up in the FIB
+ * @param next_hops
+ * Next hop of the most specific rule found for IP in its VRF.
+ * If the lookup fails, the element contains that VRF's default nexthop.
+ * @param n
+ * Number of elements in the vrf_ids, ips and next_hops arrays to lookup.
+ * @return
+ * -EINVAL for incorrect arguments, otherwise 0
+ */
+__rte_experimental
+int
+rte_fib_vrf_lookup_bulk(struct rte_fib *fib, const uint16_t *vrf_ids,
+ uint32_t *ips, uint64_t *next_hops, int n);
+
+/**
+ * Get pointer to the RIB of a specific VRF.
+ *
+ * @param fib
+ * FIB object handle
+ * @param vrf_id
+ * VRF ID (0 to max(max_vrfs, 1) - 1)
+ * @return
+ * Pointer to the VRF RIB on success
+ * NULL otherwise
+ */
+__rte_experimental
+struct rte_rib *
+rte_fib_vrf_get_rib(struct rte_fib *fib, uint16_t vrf_id);
+
/**
* Get pointer to the dataplane specific struct
*
--
2.43.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [RFC PATCH v2 6/7] fib: split DIR24-8 table update from the FIB wrapper
2026-10-14 18:39 ` [RFC PATCH v2 0/7] " Vladimir Medvedkin
` (4 preceding siblings ...)
2026-10-14 18:39 ` [RFC PATCH v2 5/7] fib: add multi-VRF support Vladimir Medvedkin
@ 2026-10-14 18:39 ` Vladimir Medvedkin
2026-10-14 18:39 ` [RFC PATCH v2 7/7] fib: add hash based VRF support for DIR24-8 Vladimir Medvedkin
2026-10-14 22:55 ` [RFC PATCH v2 0/7] VRF support in FIB library Stephen Hemminger
7 siblings, 0 replies; 45+ messages in thread
From: Vladimir Medvedkin @ 2026-10-14 18:39 UTC (permalink / raw)
To: dev; +Cc: rjarry, mb, maxime, konstantin.ananyev
Move the body of dir24_8_modify() into dir24_8_modify_dp(), taking the
dataplane table and the RIB directly, and expose it in the internal
header so that another dataplane can drive a DIR24_8 table.
Signed-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
---
lib/fib/dir24_8.c | 34 +++++++++++++++++++++-------------
lib/fib/dir24_8.h | 4 ++++
2 files changed, 25 insertions(+), 13 deletions(-)
diff --git a/lib/fib/dir24_8.c b/lib/fib/dir24_8.c
index 9b14693320..48be1c0617 100644
--- a/lib/fib/dir24_8.c
+++ b/lib/fib/dir24_8.c
@@ -465,26 +465,15 @@ modify_fib(struct dir24_8_tbl *dp, struct rte_rib *rib, uint32_t ip,
}
int
-dir24_8_modify(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip, uint8_t depth,
- uint64_t next_hop, int op)
+dir24_8_modify_dp(struct dir24_8_tbl *dp, struct rte_rib *rib, uint32_t ip,
+ uint8_t depth, uint64_t next_hop, int op)
{
- struct dir24_8_tbl *dp;
- struct rte_rib *rib;
struct rte_rib_node *tmp = NULL;
struct rte_rib_node *node;
struct rte_rib_node *parent;
int ret = 0;
uint64_t par_nh, node_nh;
- RTE_SET_USED(vrf_id);
-
- if ((fib == NULL) || (depth > RTE_FIB_MAXDEPTH))
- return -EINVAL;
-
- dp = rte_fib_get_dp(fib);
- rib = rte_fib_get_rib(fib);
- RTE_ASSERT((dp != NULL) && (rib != NULL));
-
if (next_hop > get_max_nh(dp->nh_sz))
return -EINVAL;
@@ -557,6 +546,25 @@ dir24_8_modify(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip, uint8_t depth,
return -EINVAL;
}
+int
+dir24_8_modify(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip, uint8_t depth,
+ uint64_t next_hop, int op)
+{
+ struct dir24_8_tbl *dp;
+ struct rte_rib *rib;
+
+ RTE_SET_USED(vrf_id);
+
+ if ((fib == NULL) || (depth > RTE_FIB_MAXDEPTH))
+ return -EINVAL;
+
+ dp = rte_fib_get_dp(fib);
+ rib = rte_fib_get_rib(fib);
+ RTE_ASSERT((dp != NULL) && (rib != NULL));
+
+ return dir24_8_modify_dp(dp, rib, ip, depth, next_hop, op);
+}
+
void *
dir24_8_create(const char *name, int socket_id, struct rte_fib_conf *fib_conf)
{
diff --git a/lib/fib/dir24_8.h b/lib/fib/dir24_8.h
index 88c47072d9..99e8be660e 100644
--- a/lib/fib/dir24_8.h
+++ b/lib/fib/dir24_8.h
@@ -308,6 +308,10 @@ int
dir24_8_modify(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip, uint8_t depth,
uint64_t next_hop, int op);
+int
+dir24_8_modify_dp(struct dir24_8_tbl *dp, struct rte_rib *rib, uint32_t ip,
+ uint8_t depth, uint64_t next_hop, int op);
+
int
dir24_8_rcu_qsbr_add(struct dir24_8_tbl *dp, struct rte_fib_rcu_config *cfg,
const char *name);
--
2.43.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* [RFC PATCH v2 7/7] fib: add hash based VRF support for DIR24-8
2026-10-14 18:39 ` [RFC PATCH v2 0/7] " Vladimir Medvedkin
` (5 preceding siblings ...)
2026-10-14 18:39 ` [RFC PATCH v2 6/7] fib: split DIR24-8 table update from the FIB wrapper Vladimir Medvedkin
@ 2026-10-14 18:39 ` Vladimir Medvedkin
2026-10-14 22:55 ` [RFC PATCH v2 0/7] VRF support in FIB library Stephen Hemminger
7 siblings, 0 replies; 45+ messages in thread
From: Vladimir Medvedkin @ 2026-10-14 18:39 UTC (permalink / raw)
To: dev; +Cc: rjarry, mb, maxime, konstantin.ananyev
Add multi-VRF support to the DIR24_8 algorithm without replicating its
16M entry tbl24 per VRF: all the VRFs share a single DIR24_8 table which
maps an address to the identifier of the longest prefix present in any
VRF. A hash table keyed by (VRF, prefix ID) then resolves the nexthop
of the VRF, a miss returning the per-VRF default.
The dataplane keeps an internal RIB of the prefixes of all the VRFs
holding the prefix IDs and the VRF records, while the RIB of every VRF
returned by rte_fib_vrf_get_rib() mirrors the routes of that VRF as
for the other algorithms.
The VRF specific bookkeeping is kept generic (fib_vrf.[ch]) and driven
by a small set of address family specific operations (RIB traversal and
LPM table update) so that the IPv6 TRIE algorithm can reuse it.
Signed-off-by: Vladimir Medvedkin <vladimir.medvedkin@intel.com>
---
app/test-fib/main.c | 28 +-
app/test/test_fib.c | 352 +++++--
app/test/test_fib_perf.c | 47 +-
doc/guides/prog_guide/fib_lib.rst | 25 +-
doc/guides/rel_notes/release_26_11.rst | 3 +-
lib/fib/dir24_8_vrf.c | 290 ++++++
lib/fib/dir24_8_vrf.h | 60 ++
lib/fib/fib_vrf.c | 1256 ++++++++++++++++++++++++
lib/fib/fib_vrf.h | 277 ++++++
lib/fib/fib_vrf_avx512.c | 142 +++
lib/fib/meson.build | 22 +-
lib/fib/rte_fib.c | 81 +-
lib/fib/rte_fib.h | 18 +-
13 files changed, 2485 insertions(+), 116 deletions(-)
create mode 100644 lib/fib/dir24_8_vrf.c
create mode 100644 lib/fib/dir24_8_vrf.h
create mode 100644 lib/fib/fib_vrf.c
create mode 100644 lib/fib/fib_vrf.h
create mode 100644 lib/fib/fib_vrf_avx512.c
diff --git a/app/test-fib/main.c b/app/test-fib/main.c
index 96efe0d618..3de2c062a2 100644
--- a/app/test-fib/main.c
+++ b/app/test-fib/main.c
@@ -820,12 +820,12 @@ print_usage(void)
"\t\ttrie - TRIE based FIB\n"
"defaults are: dir for ipv4 and trie for ipv6\n"
"[-e <entry size (valid only for dir and trie fib types): "
- "1/2/4/8 (default 4)>]\n"
+ "1/2/4/8 (default 4), only 8 for dir in VRF mode>]\n"
"[-g <number of tbl8's for dir24_8, dir8_16_8 or trie FIBs>]\n"
"[-V <number of VRFs (2 to 65535)>]\n"
"[--vrfconf <\",\" separated \"<vrf> <n>%%\" list: every listed "
"VRF gets a random n%% of the VRF 0 routes>]\n"
- "\tVRF mode (valid for the dir8_16_8 FIB) is "
+ "\tVRF mode (valid for dir and dir8_16_8 FIBs) is "
"enabled by -V, by --vrfconf or by a routes or lookup file with "
"a leading VRF field (\"<vrf> <prefix>/<depth> <nexthop>\", "
"\"<vrf> <address>\"), the other files put everything in VRF 0\n"
@@ -945,10 +945,20 @@ setup_vrfs(uint16_t max_file_vrf)
if (config.nb_vrfs == 0)
return;
if ((config.flags & IPV6_FLAG) ||
- (config.flags & FIB_TYPE_MASK) != FIB_V4_DIR_8_16_8_TYPE)
- rte_exit(-EINVAL, "VRFs are only supported by the dir8_16_8 FIB "
- "type\n");
+ (config.flags & FIB_TYPE_MASK) == FIB_RIB_TYPE)
+ rte_exit(-EINVAL, "VRFs are only supported by the dir and "
+ "dir8_16_8 FIB types\n");
config.flags |= VRF_FLAG;
+
+ /* The hash based VRF DIR24_8 FIB only supports 8 byte next hops. */
+ if (!(config.flags & IPV6_FLAG) &&
+ (config.flags & FIB_TYPE_MASK) == FIB_V4_DIR_TYPE) {
+ if ((config.flags & ENT_SZ_FLAG) && config.ent_sz != 8)
+ rte_exit(-EINVAL, "-e %u is not valid in VRF mode for "
+ "the dir FIB type, only 8 byte next hops are "
+ "supported\n", config.ent_sz);
+ config.ent_sz = 8;
+ }
}
static void
@@ -1168,6 +1178,7 @@ static int
select_lookup_fn_v4(struct rte_fib *fib, enum rte_fib_type type)
{
enum rte_fib_lookup_type lookup_type;
+ bool vrf = vrf_enabled();
if (type == RTE_FIB_DIR8_16_8) {
if (config.lookup_fn == 1)
@@ -1176,6 +1187,13 @@ select_lookup_fn_v4(struct rte_fib *fib, enum rte_fib_type type)
lookup_type = RTE_FIB_LOOKUP_DIR8_16_8_VECTOR_AVX512;
else
return -EINVAL;
+ } else if (type == RTE_FIB_DIR24_8 && vrf) {
+ if (config.lookup_fn == 1)
+ lookup_type = RTE_FIB_LOOKUP_DIR24_8_VRF_SCALAR;
+ else if (config.lookup_fn == 2)
+ lookup_type = RTE_FIB_LOOKUP_DIR24_8_VRF_VECTOR_AVX512;
+ else
+ return -EINVAL;
} else if (type == RTE_FIB_DIR24_8) {
if (config.lookup_fn == 1)
lookup_type = RTE_FIB_LOOKUP_DIR24_8_SCALAR_MACRO;
diff --git a/app/test/test_fib.c b/app/test/test_fib.c
index ce5e48264a..ffdbbadb0a 100644
--- a/app/test/test_fib.c
+++ b/app/test/test_fib.c
@@ -5,6 +5,7 @@
#include <stdio.h>
#include <stdint.h>
+#include <stdbool.h>
#include <stdlib.h>
#include <string.h>
#include <inttypes.h>
@@ -13,6 +14,7 @@
#include <rte_ip.h>
#include <rte_log.h>
#include <rte_byteorder.h>
+#include <rte_rib.h>
#include <rte_fib.h>
#include <rte_malloc.h>
@@ -635,23 +637,35 @@ test_fib_rcu_sync_rw(void)
#define DEF_NH 77
static struct rte_fib *
-vrf_fib_create(const char *name, uint64_t *vrf_default_nh)
+vrf_fib_create_type(const char *name, uint64_t *vrf_default_nh,
+ enum rte_fib_type type)
{
struct rte_fib_conf cfg = {0};
char fib_name[24];
- cfg.type = RTE_FIB_DIR8_16_8;
+ cfg.type = type;
cfg.max_routes = 1000;
cfg.default_nh = DEF_NH;
cfg.max_vrfs = TEST_MAX_VRFS;
cfg.vrf_default_nh = vrf_default_nh;
- cfg.dir8_16_8.num_tbl16 = TEST_NUM_TBL16;
- cfg.dir8_16_8.num_tbl8 = TEST_NUM_TBL8;
+ if (type == RTE_FIB_DIR24_8) {
+ cfg.dir24_8.nh_sz = RTE_FIB_DIR24_8_8B;
+ cfg.dir24_8.num_tbl8 = TEST_NUM_TBL8;
+ } else {
+ cfg.dir8_16_8.num_tbl16 = TEST_NUM_TBL16;
+ cfg.dir8_16_8.num_tbl8 = TEST_NUM_TBL8;
+ }
snprintf(fib_name, sizeof(fib_name), "%.20s", name);
return rte_fib_create(fib_name, SOCKET_ID_ANY, &cfg);
}
+static struct rte_fib *
+vrf_fib_create(const char *name, uint64_t *vrf_default_nh)
+{
+ return vrf_fib_create_type(name, vrf_default_nh, RTE_FIB_DIR8_16_8);
+}
+
static uint64_t
vrf_lookup1(struct rte_fib *fib, uint16_t vrf, uint32_t ip)
{
@@ -676,10 +690,16 @@ test_vrf_create_invalid(void)
cfg.max_vrfs = TEST_MAX_VRFS;
cfg.type = RTE_FIB_DIR24_8;
- cfg.dir24_8.nh_sz = RTE_FIB_DIR24_8_4B;
+ cfg.dir24_8.nh_sz = RTE_FIB_DIR24_8_8B;
cfg.dir24_8.num_tbl8 = 128;
fib = rte_fib_create(name, SOCKET_ID_ANY, &cfg);
- RTE_TEST_ASSERT(fib == NULL, "DIR24_8 must reject max_vrfs >= 2\n");
+ RTE_TEST_ASSERT(fib != NULL, "DIR24_8 must support max_vrfs >= 2\n");
+ rte_fib_free(fib);
+
+ cfg.dir24_8.nh_sz = RTE_FIB_DIR24_8_4B;
+ fib = rte_fib_create(name, SOCKET_ID_ANY, &cfg);
+ RTE_TEST_ASSERT(fib == NULL,
+ "DIR24_8 with max_vrfs >= 2 must reject nh_sz != 8B\n");
cfg.type = RTE_FIB_DUMMY;
fib = rte_fib_create(name, SOCKET_ID_ANY, &cfg);
@@ -1093,6 +1113,109 @@ test_vrf_rcu_sync_rw(void)
return status == 0 ? TEST_SUCCESS : TEST_FAILED;
}
+/*
+ * VRF behaviour on the hash-based DIR24_8 dataplane (a shared prefix trie
+ * with per-(vrf, prefix) nexthops in a hash table). Mirrors the DIR8_16_8 VRF
+ * checks: per-VRF isolation, longest-prefix match, defaults and deletes.
+ */
+static int32_t
+test_vrf_dir24_8(void)
+{
+ struct rte_fib *fib;
+ struct rte_rib_node *node;
+ uint32_t default_vrf_ip;
+ uint16_t v;
+ uint64_t default_vrf_nh, nh;
+
+ fib = vrf_fib_create_type(__func__, NULL, RTE_FIB_DIR24_8);
+ RTE_TEST_ASSERT(fib != NULL, "Failed to create DIR24_8 VRF FIB\n");
+
+ for (v = 0; v < TEST_MAX_VRFS; v++) {
+ RTE_TEST_ASSERT(rte_fib_vrf_add(fib, v, RTE_IPV4(10, 0, 0, 0),
+ 8, 100 + v) == 0, "add /8 vrf %u\n", v);
+ RTE_TEST_ASSERT(rte_fib_vrf_add(fib, v, RTE_IPV4(192, 168, 0, 0),
+ 16, 200 + v) == 0, "add /16 vrf %u\n", v);
+ RTE_TEST_ASSERT(rte_fib_vrf_add(fib, v, RTE_IPV4(10, 1, 1, 0),
+ 24, 300 + v) == 0, "add /24 vrf %u\n", v);
+ }
+
+ /* Every VRF has its own RIB mirroring its routes. */
+ for (v = 0; v < TEST_MAX_VRFS; v++) {
+ struct rte_rib *rib = rte_fib_vrf_get_rib(fib, v);
+
+ RTE_TEST_ASSERT(rib != NULL &&
+ (v == 0 || rib != rte_fib_get_rib(fib)),
+ "vrf %u must have its own RIB\n", v);
+ node = rte_rib_lookup_exact(rib, RTE_IPV4(10, 1, 1, 0), 24);
+ RTE_TEST_ASSERT(node != NULL, "vrf %u /24 missing from RIB\n", v);
+ rte_rib_get_nh(node, &nh);
+ RTE_TEST_ASSERT(nh == (uint64_t)(300 + v),
+ "vrf %u RIB /24 nexthop got %"PRIu64"\n", v, nh);
+ }
+
+ default_vrf_ip = RTE_IPV4(10, 2, 3, 4);
+ RTE_TEST_ASSERT(rte_fib_lookup_bulk(fib, &default_vrf_ip, &default_vrf_nh, 1) == 0,
+ "default VRF DIR24_8 lookup failed\n");
+ RTE_TEST_ASSERT(default_vrf_nh == 100,
+ "default VRF DIR24_8 lookup must use VRF 0 nexthop, got %"PRIu64"\n",
+ default_vrf_nh);
+
+ for (v = 0; v < TEST_MAX_VRFS; v++) {
+ nh = vrf_lookup1(fib, v, RTE_IPV4(10, 2, 3, 4));
+ RTE_TEST_ASSERT(nh == (uint64_t)(100 + v),
+ "vrf %u /8 lookup got %"PRIu64"\n", v, nh);
+ nh = vrf_lookup1(fib, v, RTE_IPV4(192, 168, 5, 5));
+ RTE_TEST_ASSERT(nh == (uint64_t)(200 + v),
+ "vrf %u /16 lookup got %"PRIu64"\n", v, nh);
+ nh = vrf_lookup1(fib, v, RTE_IPV4(10, 1, 1, 7));
+ RTE_TEST_ASSERT(nh == (uint64_t)(300 + v),
+ "vrf %u /24 lookup got %"PRIu64"\n", v, nh);
+ nh = vrf_lookup1(fib, v, RTE_IPV4(8, 8, 8, 8));
+ RTE_TEST_ASSERT(nh == DEF_NH,
+ "vrf %u default lookup got %"PRIu64"\n", v, nh);
+ }
+
+ /* Delete the /24 from VRF 1 only; its covering /8 then matches. */
+ RTE_TEST_ASSERT(rte_fib_vrf_delete(fib, 1, RTE_IPV4(10, 1, 1, 0),
+ 24) == 0, "del /24 vrf 1\n");
+ nh = vrf_lookup1(fib, 1, RTE_IPV4(10, 1, 1, 7));
+ RTE_TEST_ASSERT(nh == 101, "vrf 1 /24 after delete got %"PRIu64"\n",
+ nh);
+ RTE_TEST_ASSERT(rte_rib_lookup_exact(rte_fib_vrf_get_rib(fib, 1),
+ RTE_IPV4(10, 1, 1, 0), 24) == NULL,
+ "vrf 1 /24 still in its RIB after delete\n");
+ RTE_TEST_ASSERT(rte_rib_lookup_exact(rte_fib_vrf_get_rib(fib, 0),
+ RTE_IPV4(10, 1, 1, 0), 24) != NULL,
+ "vrf 0 /24 vanished from its RIB\n");
+ /* Other VRFs keep their own /24. */
+ for (v = 0; v < TEST_MAX_VRFS; v++) {
+ if (v == 1)
+ continue;
+ nh = vrf_lookup1(fib, v, RTE_IPV4(10, 1, 1, 7));
+ RTE_TEST_ASSERT(nh == (uint64_t)(300 + v),
+ "vrf %u /24 unexpectedly changed to %"PRIu64"\n",
+ v, nh);
+ }
+
+ /* A route owned by a single VRF exercises the full-removal path. */
+ RTE_TEST_ASSERT(rte_fib_vrf_add(fib, 3, RTE_IPV4(172, 16, 0, 0), 16,
+ 999) == 0, "add single-owner /16 vrf 3\n");
+ nh = vrf_lookup1(fib, 3, RTE_IPV4(172, 16, 5, 5));
+ RTE_TEST_ASSERT(nh == 999, "vrf 3 single-owner /16 got %"PRIu64"\n",
+ nh);
+ nh = vrf_lookup1(fib, 0, RTE_IPV4(172, 16, 5, 5));
+ RTE_TEST_ASSERT(nh == DEF_NH, "vrf 0 must miss 172.16/16 got %"PRIu64
+ "\n", nh);
+ RTE_TEST_ASSERT(rte_fib_vrf_delete(fib, 3, RTE_IPV4(172, 16, 0, 0),
+ 16) == 0, "del single-owner /16 vrf 3\n");
+ nh = vrf_lookup1(fib, 3, RTE_IPV4(172, 16, 5, 5));
+ RTE_TEST_ASSERT(nh == DEF_NH,
+ "vrf 3 after single-owner delete got %"PRIu64"\n", nh);
+
+ rte_fib_free(fib);
+ return TEST_SUCCESS;
+}
+
/*
* Network order lookups with every lookup implementation of DIR8_16_8, in
* single and multi VRF FIBs, with a burst of two vector groups and a leftover.
@@ -1190,21 +1313,28 @@ test_lookup_network_order(void)
/*
* VRF cross-validation.
*
- * Build one VRF-capable DIR8_16_8 FIB with N VRFs and N independent reference
- * DIR8_16_8 FIBs. The same large route table is added to each VRF and its
- * matching reference, using a per-VRF distinct unique next hop. Each VRF lookup
- * must match its reference exactly, which proves both per-VRF correctness and
- * isolation (no VRF bleeds into another), with every available lookup
- * implementation.
+ * Build one VRF-capable FIB with N VRFs and N independent single-VRF reference
+ * DIR24_8 FIBs. The large route table is added to every VRF and its reference
+ * with a per-VRF unique nexthop, then routes are deleted, updated and re-added
+ * in a pseudo-random pattern. After every phase, mixed per-IP VRF lookups must
+ * match the references exactly, which proves per-VRF correctness and isolation
+ * for all the add/delete code paths and every available lookup implementation.
*/
+
+static inline uint64_t
+vrf_validate_nh(uint32_t i, uint16_t v, uint32_t gen)
+{
+ return (uint64_t)i + 1 + (uint64_t)v * ((uint64_t)MAX_RULE_NUM + 1) +
+ ((uint64_t)gen << 40);
+}
+
static int
vrf_validate_lookups_type(struct rte_fib *dut, struct rte_fib **ref,
- enum rte_fib_lookup_type type)
+ const char *phase, enum rte_fib_lookup_type type)
{
uint64_t mism = 0, total = 0;
unsigned int i, j;
- /* Compare mixed per-IP VRF lookups against the matching references. */
for (i = 0; i < ITERATIONS; i++) {
static uint32_t ip_batch[BATCH_SIZE];
static uint16_t vrf_batch[BATCH_SIZE];
@@ -1212,7 +1342,17 @@ vrf_validate_lookups_type(struct rte_fib *dut, struct rte_fib **ref,
uint64_t nh_ref[BULK_SIZE];
for (j = 0; j < BATCH_SIZE; j++) {
- ip_batch[j] = rte_rand();
+ /* Half random, half inside a known prefix. */
+ if (j & 1) {
+ const struct route_rule *r =
+ &large_route_table[rte_rand() %
+ NUM_ROUTE_ENTRIES];
+
+ ip_batch[j] = r->ip | (rte_rand() &
+ (uint32_t)((1ULL << (32 - r->depth)) - 1));
+ } else {
+ ip_batch[j] = rte_rand();
+ }
vrf_batch[j] = rte_rand() % TEST_MAX_VRFS;
}
@@ -1221,17 +1361,16 @@ vrf_validate_lookups_type(struct rte_fib *dut, struct rte_fib **ref,
rte_fib_vrf_lookup_bulk(dut, &vrf_batch[j],
&ip_batch[j], nh_dut, BULK_SIZE);
- /* Reference: each IP resolved in its own VRF's FIB. */
for (k = 0; k < BULK_SIZE; k++)
rte_fib_lookup_bulk(ref[vrf_batch[j + k]],
&ip_batch[j + k], &nh_ref[k], 1);
for (k = 0; k < BULK_SIZE; k++) {
total++;
if (nh_dut[k] != nh_ref[k]) {
- if (mism < 20)
- printf("Mismatch vrf=%u "
+ if (mism < 10)
+ printf("%s: mismatch vrf=%u "
"ip=0x%08x dut=%" PRIu64
- " ref=%" PRIu64 "\n",
+ " ref=%" PRIu64 "\n", phase,
vrf_batch[j + k],
ip_batch[j + k],
nh_dut[k], nh_ref[k]);
@@ -1241,35 +1380,84 @@ vrf_validate_lookups_type(struct rte_fib *dut, struct rte_fib **ref,
}
}
- printf("Lookup type %u: validated %" PRIu64 " lookups across %u VRFs, "
- "%" PRIu64 " mismatches\n", type, total, TEST_MAX_VRFS, mism);
+ printf("%s: lookup type %u: validated %" PRIu64 " lookups across %u "
+ "VRFs, %" PRIu64 " mismatches\n", phase, type, total,
+ TEST_MAX_VRFS, mism);
return (mism != 0) ? -1 : 0;
}
/* Validate the lookups with every lookup implementation of the algorithm. */
static int
-vrf_validate_lookups(struct rte_fib *dut, struct rte_fib **ref)
+vrf_validate_lookups(struct rte_fib *dut, struct rte_fib **ref,
+ const char *phase)
{
const enum rte_fib_lookup_type types[] = {
RTE_FIB_LOOKUP_DIR8_16_8_SCALAR,
RTE_FIB_LOOKUP_DIR8_16_8_VECTOR_AVX512,
+ RTE_FIB_LOOKUP_DIR24_8_VRF_SCALAR,
+ RTE_FIB_LOOKUP_DIR24_8_VRF_VECTOR_AVX512,
};
unsigned int t;
int ret;
for (t = 0; t < RTE_DIM(types); t++) {
- /* The vector lookup needs a 512 bit SIMD width. */
+ /* Other algorithm, or vector lookup without 512 bit SIMD. */
if (rte_fib_select_lookup(dut, types[t]) != 0)
continue;
- ret = vrf_validate_lookups_type(dut, ref, types[t]);
+ ret = vrf_validate_lookups_type(dut, ref, phase, types[t]);
if (ret != 0)
return ret;
}
return 0;
}
+/*
+ * Apply one operation to every (route, VRF) pair, or to a random half of them
+ * if not all, checking that dut and ref agree on the return status.
+ */
static int
-test_fib_vrf_validate_dir8_16_8(void)
+vrf_validate_apply(struct rte_fib *dut, struct rte_fib **ref, bool all,
+ bool del, uint32_t gen, const char *phase)
+{
+ unsigned int i;
+ uint16_t v;
+
+ for (i = 0; i < NUM_ROUTE_ENTRIES; i++) {
+ for (v = 0; v < TEST_MAX_VRFS; v++) {
+ int r_dut, r_ref;
+
+ if (!all && (rte_rand() & 1) == 0)
+ continue;
+ if (del) {
+ r_dut = rte_fib_vrf_delete(dut, v,
+ large_route_table[i].ip,
+ large_route_table[i].depth);
+ r_ref = rte_fib_delete(ref[v],
+ large_route_table[i].ip,
+ large_route_table[i].depth);
+ } else {
+ uint64_t nh = vrf_validate_nh(i, v, gen);
+
+ r_dut = rte_fib_vrf_add(dut, v,
+ large_route_table[i].ip,
+ large_route_table[i].depth, nh);
+ r_ref = rte_fib_add(ref[v],
+ large_route_table[i].ip,
+ large_route_table[i].depth, nh);
+ }
+ if (r_dut != r_ref) {
+ printf("%s: status mismatch i=%u vrf=%u "
+ "dut=%d ref=%d\n", phase, i, v, r_dut,
+ r_ref);
+ return -1;
+ }
+ }
+ }
+ return 0;
+}
+
+static int
+vrf_validate(enum rte_fib_type type)
{
struct rte_fib *dut = NULL;
struct rte_fib *ref[TEST_MAX_VRFS] = { NULL };
@@ -1282,24 +1470,31 @@ test_fib_vrf_validate_dir8_16_8(void)
generate_large_route_rule_table();
- printf("No. routes = %u, VRFs = %u\n",
- (unsigned int) NUM_ROUTE_ENTRIES, TEST_MAX_VRFS);
+ printf("No. routes = %u, VRFs = %u, algorithm = %s\n",
+ (unsigned int) NUM_ROUTE_ENTRIES, TEST_MAX_VRFS,
+ (type == RTE_FIB_DIR24_8) ? "DIR24_8" : "DIR8_16_8");
cfg.max_routes = 2000000;
cfg.default_nh = 0;
- cfg.type = RTE_FIB_DIR8_16_8;
+ cfg.type = type;
cfg.max_vrfs = TEST_MAX_VRFS;
- cfg.dir8_16_8.num_tbl16 = 256 * TEST_MAX_VRFS;
- cfg.dir8_16_8.num_tbl8 = 1 << 18;
+ if (type == RTE_FIB_DIR24_8) {
+ cfg.dir24_8.nh_sz = RTE_FIB_DIR24_8_8B;
+ cfg.dir24_8.num_tbl8 = 1 << 16;
+ } else {
+ cfg.dir8_16_8.num_tbl16 = 256 * TEST_MAX_VRFS;
+ cfg.dir8_16_8.num_tbl8 = 1 << 18;
+ }
+ /* The references use 8 byte nexthops so results are comparable. */
+ cfg_ref.max_routes = 2000000;
+ cfg_ref.default_nh = 0;
+ cfg_ref.type = RTE_FIB_DIR24_8;
+ cfg_ref.dir24_8.nh_sz = RTE_FIB_DIR24_8_8B;
+ cfg_ref.dir24_8.num_tbl8 = 1 << 16;
dut = rte_fib_create("vrf_validate_dut", SOCKET_ID_ANY, &cfg);
RTE_TEST_ASSERT(dut != NULL, "Failed to create VRF FIB\n");
- cfg_ref.max_routes = 2000000;
- cfg_ref.default_nh = 0;
- cfg_ref.type = RTE_FIB_DIR8_16_8;
- cfg_ref.dir8_16_8.num_tbl16 = 256;
- cfg_ref.dir8_16_8.num_tbl8 = 1 << 16;
for (v = 0; v < TEST_MAX_VRFS; v++) {
snprintf(name, sizeof(name), "vrf_validate_ref%u", v);
ref[v] = rte_fib_create(name, SOCKET_ID_ANY, &cfg_ref);
@@ -1310,35 +1505,74 @@ test_fib_vrf_validate_dir8_16_8(void)
}
}
- /* Add the same routes to each VRF and its reference, per-VRF unique nh. */
- for (i = 0; i < NUM_ROUTE_ENTRIES; i++) {
- for (v = 0; v < TEST_MAX_VRFS; v++) {
- uint64_t nh = (uint64_t)i + 1 +
- (uint64_t)v * ((uint64_t)MAX_RULE_NUM + 1);
- int r_dut = rte_fib_vrf_add(dut, v,
- large_route_table[i].ip,
- large_route_table[i].depth, nh);
- int r_ref = rte_fib_add(ref[v],
- large_route_table[i].ip,
- large_route_table[i].depth, nh);
- if (r_dut != r_ref) {
- printf("Add status mismatch i=%u vrf=%u "
- "dut=%d ref=%d\n", i, v, r_dut, r_ref);
- ret = -1;
- goto out;
- }
- }
- }
-
- ret = vrf_validate_lookups(dut, ref);
+ /* 1: everything present in every VRF. */
+ ret = vrf_validate_apply(dut, ref, true, false, 0, "add all");
+ if (ret != 0)
+ goto out;
+ ret = vrf_validate_lookups(dut, ref, "add all");
+ if (ret != 0)
+ goto out;
+
+ /* 2: delete about half of the (route, VRF) pairs. */
+ ret = vrf_validate_apply(dut, ref, false, true, 0, "delete");
+ if (ret != 0)
+ goto out;
+ ret = vrf_validate_lookups(dut, ref, "delete");
+ if (ret != 0)
+ goto out;
+
+ /*
+ * 3: add about half of the (route, VRF) pairs with new nexthops, which
+ * updates the remaining routes and re-adds the deleted ones.
+ */
+ ret = vrf_validate_apply(dut, ref, false, false, 1, "update");
+ if (ret != 0)
+ goto out;
+ ret = vrf_validate_lookups(dut, ref, "update");
+ if (ret != 0)
+ goto out;
+
+ /* 4: put everything back with fresh nexthops. */
+ ret = vrf_validate_apply(dut, ref, true, false, 2, "re-add");
+ if (ret != 0)
+ goto out;
+ ret = vrf_validate_lookups(dut, ref, "re-add");
+ if (ret != 0)
+ goto out;
+
+ /* 5: delete everything, only default nexthops must remain. */
+ ret = vrf_validate_apply(dut, ref, true, true, 0, "delete all");
+ if (ret != 0)
+ goto out;
+ ret = vrf_validate_lookups(dut, ref, "delete all");
+ if (ret != 0)
+ goto out;
+
+ /* 6: once more from scratch, reusing the retired prefix IDs. */
+ ret = vrf_validate_apply(dut, ref, true, false, 3, "add again");
+ if (ret != 0)
+ goto out;
+ ret = vrf_validate_lookups(dut, ref, "add again");
out:
rte_fib_free(dut);
- for (v = 0; v < TEST_MAX_VRFS; v++)
- rte_fib_free(ref[v]);
+ for (i = 0; i < TEST_MAX_VRFS; i++)
+ rte_fib_free(ref[i]);
return ret;
}
+static int
+test_fib_vrf_validate_dir8_16_8(void)
+{
+ return vrf_validate(RTE_FIB_DIR8_16_8);
+}
+
+static int
+test_fib_vrf_validate_dir24_8(void)
+{
+ return vrf_validate(RTE_FIB_DIR24_8);
+}
+
static struct unit_test_suite fib_fast_tests = {
.suite_name = "fib autotest",
.setup = NULL,
@@ -1362,6 +1596,7 @@ static struct unit_test_suite fib_fast_tests = {
TEST_CASE(test_vrf_get_rib),
TEST_CASE(test_vrf_rcu),
TEST_CASE(test_vrf_rcu_sync_rw),
+ TEST_CASE(test_vrf_dir24_8),
TEST_CASES_END()
}
};
@@ -1373,6 +1608,7 @@ static struct unit_test_suite fib_slow_tests = {
.unit_test_cases = {
TEST_CASE(test_multiple_create),
TEST_CASE(test_fib_vrf_validate_dir8_16_8),
+ TEST_CASE(test_fib_vrf_validate_dir24_8),
TEST_CASES_END()
}
};
diff --git a/app/test/test_fib_perf.c b/app/test/test_fib_perf.c
index ff8240aabf..e8d44566bb 100644
--- a/app/test/test_fib_perf.c
+++ b/app/test/test_fib_perf.c
@@ -175,30 +175,21 @@ test_fib_perf_dir8_16_8(void)
#define VRF_PERF_NB_VRFS 4
static int
-test_fib_perf_vrf_dir8_16_8(void)
+measure_vrf_perf(const char *name, struct rte_fib_conf *config)
{
struct rte_fib *fib = NULL;
- struct rte_fib_conf config = { 0 };
uint64_t begin, total_time;
unsigned int i, j;
uint16_t v;
int status = 0;
int64_t count = 0;
- config.max_routes = 2000000;
- config.rib_ext_sz = 0;
- config.type = RTE_FIB_DIR8_16_8;
- config.default_nh = 0;
- config.max_vrfs = VRF_PERF_NB_VRFS;
- config.dir8_16_8.num_tbl16 = 256 * VRF_PERF_NB_VRFS;
- config.dir8_16_8.num_tbl8 = 1 << 18;
-
generate_large_route_rule_table();
printf("No. routes = %u, VRFs = %u\n",
(unsigned int) NUM_ROUTE_ENTRIES, VRF_PERF_NB_VRFS);
- fib = rte_fib_create("vrf_perf_d8", SOCKET_ID_ANY, &config);
+ fib = rte_fib_create(name, SOCKET_ID_ANY, config);
TEST_FIB_ASSERT(fib != NULL);
begin = rte_rdtsc();
@@ -254,6 +245,40 @@ test_fib_perf_vrf_dir8_16_8(void)
return 0;
}
+static int
+test_fib_perf_vrf_dir8_16_8(void)
+{
+ struct rte_fib_conf config = { 0 };
+
+ config.max_routes = 2000000;
+ config.rib_ext_sz = 0;
+ config.type = RTE_FIB_DIR8_16_8;
+ config.default_nh = 0;
+ config.max_vrfs = VRF_PERF_NB_VRFS;
+ config.dir8_16_8.num_tbl16 = 256 * VRF_PERF_NB_VRFS;
+ config.dir8_16_8.num_tbl8 = 1 << 18;
+
+ /* Keep the name short, the RIB derives memzone names from it. */
+ return measure_vrf_perf("vrf_perf_d8", &config);
+}
+
+static int
+test_fib_perf_vrf_dir24_8(void)
+{
+ struct rte_fib_conf config = { 0 };
+
+ config.max_routes = 2000000;
+ config.rib_ext_sz = 0;
+ config.type = RTE_FIB_DIR24_8;
+ config.default_nh = 0;
+ config.max_vrfs = VRF_PERF_NB_VRFS;
+ config.dir24_8.nh_sz = RTE_FIB_DIR24_8_8B;
+ config.dir24_8.num_tbl8 = 1 << 16;
+
+ return measure_vrf_perf("vrf_perf_d24", &config);
+}
+
REGISTER_PERF_TEST(fib_perf_autotest, test_fib_perf);
REGISTER_PERF_TEST(fib_perf_dir8_16_8_autotest, test_fib_perf_dir8_16_8);
REGISTER_PERF_TEST(fib_perf_vrf_dir8_16_8_autotest, test_fib_perf_vrf_dir8_16_8);
+REGISTER_PERF_TEST(fib_perf_vrf_dir24_8_autotest, test_fib_perf_vrf_dir24_8);
diff --git a/doc/guides/prog_guide/fib_lib.rst b/doc/guides/prog_guide/fib_lib.rst
index 888c1acdfa..64665cfc98 100644
--- a/doc/guides/prog_guide/fib_lib.rst
+++ b/doc/guides/prog_guide/fib_lib.rst
@@ -62,13 +62,28 @@ The VRF aware methods are:
* ``rte_fib_vrf_lookup_bulk()``: Bulk LPM lookup with a VRF ID per IP address.
-* ``rte_fib_vrf_get_rib()``: Get the ``rte_rib`` of a given VRF.
+* ``rte_fib_vrf_get_rib()``: Get the ``rte_rib`` holding the routes of a given VRF.
The legacy methods operate on VRF 0.
-Multi-VRF FIBs are supported by the ``RTE_FIB_DIR8_16_8`` algorithm, which keeps
-an independent ``rte_rib`` and an independent root table per VRF, while the lower
-level tables are allocated from pools shared by all the VRFs.
-The VRF ID selects the root table, then the lookup proceeds as for a single VRF.
+Multi-VRF FIBs are supported by the ``RTE_FIB_DIR24_8`` and ``RTE_FIB_DIR8_16_8``
+algorithms.
+Every VRF has its own ``rte_rib`` in both cases,
+but the dataplane tables are organized differently:
+
+Per-VRF root table
+ ``RTE_FIB_DIR8_16_8`` keeps an independent root table
+ per VRF, while the lower level tables are allocated from pools shared by all the VRFs.
+ The VRF ID selects the root table, then the lookup proceeds as for a single VRF.
+
+Shared table and hash resolution
+ ``RTE_FIB_DIR24_8`` keeps a single lookup table
+ holding the prefixes of all the VRFs.
+ The lookup table returns an internal prefix identifier, which is then combined
+ with the VRF ID to find the next hop ID in a hash table.
+ A miss in the hash table returns the VRF default next hop ID.
+ With this scheme the memory of the lookup table does not grow with the number of
+ VRFs, but ``max_routes`` is the number of distinct prefixes across all the VRFs
+ and ``nh_sz`` must select 8 byte next hop IDs.
Implementation details
diff --git a/doc/guides/rel_notes/release_26_11.rst b/doc/guides/rel_notes/release_26_11.rst
index 5f77116d8c..c42ca0830d 100644
--- a/doc/guides/rel_notes/release_26_11.rst
+++ b/doc/guides/rel_notes/release_26_11.rst
@@ -216,7 +216,8 @@ New Features
A FIB can now hold several independent routing tables (VRFs), configured with
the ``max_vrfs`` and ``vrf_default_nh`` fields of ``rte_fib_conf`` and
accessed with the ``rte_fib_vrf_*`` functions. ``RTE_FIB_DIR8_16_8`` keeps a
- root table per VRF.
+ root table per VRF, while ``RTE_FIB_DIR24_8`` shares its lookup table across
+ VRFs and resolves the per-VRF next hops through a hash table.
Removed Items
diff --git a/lib/fib/dir24_8_vrf.c b/lib/fib/dir24_8_vrf.c
new file mode 100644
index 0000000000..f2a5850758
--- /dev/null
+++ b/lib/fib/dir24_8_vrf.c
@@ -0,0 +1,290 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Intel Corporation
+ */
+
+#include <stdbool.h>
+#include <stdint.h>
+#include <string.h>
+
+#include <rte_cpuflags.h>
+#include <rte_debug.h>
+#include <rte_errno.h>
+#include <rte_malloc.h>
+#include <rte_vect.h>
+
+#include <rte_rib.h>
+#include <rte_fib.h>
+
+#include "dir24_8.h"
+#ifdef CC_AVX512_SUPPORT
+#include "dir24_8_avx512.h"
+#endif
+#include "dir24_8_vrf.h"
+#include "fib_log.h"
+#include "fib_vrf.h"
+
+/*
+ * IPv4 glue for the generic VRF code: RIB access and DIR24_8 table updates.
+ */
+
+static void *
+rib_lookup(void *rib, const void *ip)
+{
+ return rte_rib_lookup(rib, *(const uint32_t *)ip);
+}
+
+static void *
+rib_lookup_exact(void *rib, const void *ip, uint8_t depth)
+{
+ return rte_rib_lookup_exact(rib, *(const uint32_t *)ip, depth);
+}
+
+static void *
+rib_lookup_parent(void *node)
+{
+ return rte_rib_lookup_parent(node);
+}
+
+static void *
+rib_get_nxt_cover(void *rib, const void *ip, uint8_t depth, void *last)
+{
+ return rte_rib_get_nxt(rib, *(const uint32_t *)ip, depth, last,
+ RTE_RIB_GET_NXT_COVER);
+}
+
+static void *
+rib_walk(void *rib, void *last)
+{
+ return rte_rib_get_nxt(rib, 0, 0, last, RTE_RIB_GET_NXT_ALL);
+}
+
+static void *
+rib_insert(void *rib, const void *ip, uint8_t depth)
+{
+ return rte_rib_insert(rib, *(const uint32_t *)ip, depth);
+}
+
+static void
+rib_remove(void *rib, const void *ip, uint8_t depth)
+{
+ rte_rib_remove(rib, *(const uint32_t *)ip, depth);
+}
+
+static void
+rib_get_prefix(const void *node, void *ip, uint8_t *depth)
+{
+ rte_rib_get_ip(node, ip);
+ rte_rib_get_depth(node, depth);
+}
+
+static uint64_t
+rib_get_nh(const void *node)
+{
+ uint64_t nh;
+
+ rte_rib_get_nh(node, &nh);
+ return nh;
+}
+
+static void
+rib_set_nh(void *node, uint64_t nh)
+{
+ rte_rib_set_nh(node, nh);
+}
+
+static void *
+rib_get_ext(void *node)
+{
+ return rte_rib_get_ext(node);
+}
+
+static int
+lpm_modify(void *lpm, void *rib, const void *ip, uint8_t depth,
+ uint64_t prefix_id, enum fib_vrf_op op)
+{
+ return dir24_8_modify_dp(lpm, rib, *(const uint32_t *)ip, depth,
+ prefix_id, (op == FIB_VRF_ADD) ? RTE_FIB_ADD : RTE_FIB_DEL);
+}
+
+static const struct fib_vrf_ops dir24_8_vrf_ops = {
+ .rib_lookup = rib_lookup,
+ .rib_lookup_exact = rib_lookup_exact,
+ .rib_lookup_parent = rib_lookup_parent,
+ .rib_get_nxt_cover = rib_get_nxt_cover,
+ .rib_walk = rib_walk,
+ .rib_insert = rib_insert,
+ .rib_remove = rib_remove,
+ .rib_get_prefix = rib_get_prefix,
+ .rib_get_nh = rib_get_nh,
+ .rib_set_nh = rib_set_nh,
+ .rib_get_ext = rib_get_ext,
+ .lpm_modify = lpm_modify,
+};
+
+int
+dir24_8_vrf_modify(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip,
+ uint8_t depth, uint64_t next_hop, int op)
+{
+ struct dir24_8_vrf_tbl *dp;
+ struct rte_rib *vrf_rib;
+
+ if ((fib == NULL) || (depth > RTE_FIB_MAXDEPTH))
+ return -EINVAL;
+
+ dp = rte_fib_get_dp(fib);
+ vrf_rib = rte_fib_vrf_get_rib(fib, vrf_id);
+ RTE_ASSERT((dp != NULL) && (vrf_rib != NULL));
+
+ ip &= rte_rib_depth_to_mask(depth);
+ return fib_vrf_modify(&dp->ht, dp->lpm, dp->rib, vrf_rib, vrf_id, &ip,
+ depth, next_hop, (op == RTE_FIB_ADD) ? FIB_VRF_ADD : FIB_VRF_DEL);
+}
+
+void *
+dir24_8_vrf_create(const char *name, int socket_id,
+ struct rte_fib_conf *fib_conf)
+{
+ struct dir24_8_vrf_tbl *dp;
+ struct rte_fib_conf lpm_conf;
+ struct rte_rib_conf rib_conf;
+ char mem_name[64], dp_name[32];
+ int ret;
+
+ /* The hash table only stores 8 byte nexthops. */
+ if ((name == NULL) || (fib_conf == NULL) ||
+ (fib_conf->max_routes <= 0) ||
+ (fib_conf->dir24_8.nh_sz != RTE_FIB_DIR24_8_8B)) {
+ rte_errno = EINVAL;
+ return NULL;
+ }
+
+ snprintf(mem_name, sizeof(mem_name), "DP_%s", name);
+ dp = rte_zmalloc_socket(mem_name, sizeof(*dp), RTE_CACHE_LINE_SIZE,
+ socket_id);
+ if (dp == NULL) {
+ rte_errno = ENOMEM;
+ return NULL;
+ }
+ /* The RIB and hash table names must be unique. */
+ snprintf(dp_name, sizeof(dp_name), "%p", (void *)dp);
+
+ /*
+ * The DIR24_8 table stores 4 byte prefix IDs, 0 meaning no route,
+ * whatever default nexthop the user configured.
+ */
+ lpm_conf = *fib_conf;
+ lpm_conf.default_nh = 0;
+ lpm_conf.dir24_8.nh_sz = RTE_FIB_DIR24_8_4B;
+ snprintf(mem_name, sizeof(mem_name), "LPM_%s", name);
+ dp->lpm = dir24_8_create(mem_name, socket_id, &lpm_conf);
+ if (dp->lpm == NULL)
+ goto free_dp;
+
+ /* Internal RIB of all the prefixes, with the VRF records. */
+ rib_conf.ext_sz = fib_vrf_rib_ext_sz();
+ rib_conf.max_nodes = fib_conf->max_routes * 2;
+ dp->rib = rte_rib_create(dp_name, socket_id, &rib_conf);
+ if (dp->rib == NULL)
+ goto free_lpm;
+
+ ret = fib_vrf_ht_init(&dp->ht, dp_name, socket_id, &dir24_8_vrf_ops,
+ RTE_MAX(fib_conf->max_vrfs, 1), fib_conf->default_nh,
+ fib_conf->vrf_default_nh, fib_conf->max_routes);
+ if (ret != 0) {
+ rte_errno = -ret;
+ goto free_rib;
+ }
+
+ return dp;
+
+free_rib:
+ rte_rib_free(dp->rib);
+free_lpm:
+ dir24_8_free(dp->lpm);
+free_dp:
+ rte_free(dp);
+ return NULL;
+}
+
+void
+dir24_8_vrf_free(void *p)
+{
+ struct dir24_8_vrf_tbl *dp = p;
+
+ if (dp == NULL)
+ return;
+
+ fib_vrf_ht_free(&dp->ht, dp->rib);
+ rte_rib_free(dp->rib);
+ dir24_8_free(dp->lpm);
+ rte_free(dp);
+}
+
+#ifdef CC_AVX512_SUPPORT
+static void
+dir24_8_vrf_vec_lookup_bulk(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
+{
+ struct dir24_8_vrf_tbl *dp = p;
+
+ rte_dir24_8_vec_lookup_bulk_4b(dp->lpm, NULL, ips, next_hops, n);
+ fib_vrf_ht_resolve_avx512(&dp->ht, vrf_ids, next_hops, n);
+}
+
+static void
+dir24_8_vrf_vec_lookup_bulk_be(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
+{
+ struct dir24_8_vrf_tbl *dp = p;
+
+ rte_dir24_8_vec_lookup_bulk_4b_be(dp->lpm, NULL, ips, next_hops, n);
+ fib_vrf_ht_resolve_avx512(&dp->ht, vrf_ids, next_hops, n);
+}
+#endif
+
+static rte_fib_lookup_fn_t
+dir24_8_vrf_get_vector_fn(bool be_addr)
+{
+#ifdef CC_AVX512_SUPPORT
+ if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512F) <= 0 ||
+ rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512DQ) <= 0 ||
+ rte_cpu_get_flag_enabled(RTE_CPUFLAG_VPCLMULQDQ) <= 0 ||
+ rte_vect_get_max_simd_bitwidth() < RTE_VECT_SIMD_512)
+ return NULL;
+
+ if (be_addr && rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512BW) <= 0)
+ return NULL;
+
+ return be_addr ? dir24_8_vrf_vec_lookup_bulk_be :
+ dir24_8_vrf_vec_lookup_bulk;
+#else
+ RTE_SET_USED(be_addr);
+ return NULL;
+#endif
+}
+
+rte_fib_lookup_fn_t
+dir24_8_vrf_get_lookup_fn(void *p, enum rte_fib_lookup_type type,
+ bool be_addr)
+{
+ rte_fib_lookup_fn_t fn;
+
+ if (p == NULL)
+ return NULL;
+
+ switch (type) {
+ case RTE_FIB_LOOKUP_DIR24_8_VRF_SCALAR:
+ return be_addr ? dir24_8_vrf_lookup_bulk_be :
+ dir24_8_vrf_lookup_bulk;
+ case RTE_FIB_LOOKUP_DIR24_8_VRF_VECTOR_AVX512:
+ return dir24_8_vrf_get_vector_fn(be_addr);
+ case RTE_FIB_LOOKUP_DEFAULT:
+ fn = dir24_8_vrf_get_vector_fn(be_addr);
+ if (fn != NULL)
+ return fn;
+ return be_addr ? dir24_8_vrf_lookup_bulk_be :
+ dir24_8_vrf_lookup_bulk;
+ default:
+ return NULL;
+ }
+}
diff --git a/lib/fib/dir24_8_vrf.h b/lib/fib/dir24_8_vrf.h
new file mode 100644
index 0000000000..97ecbef7cc
--- /dev/null
+++ b/lib/fib/dir24_8_vrf.h
@@ -0,0 +1,60 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Intel Corporation
+ */
+
+#ifndef _DIR24_8_VRF_H_
+#define _DIR24_8_VRF_H_
+
+#include <rte_fib.h>
+
+#include "dir24_8.h"
+#include "fib_vrf.h"
+
+/**
+ * @file
+ * DIR24_8 algorithm with hash based VRF resolution: a single DIR24_8 table
+ * holding 4 byte prefix IDs, resolved to per-VRF nexthops through fib_vrf.
+ */
+
+struct dir24_8_vrf_tbl {
+ struct dir24_8_tbl *lpm; /**< Shared prefix ID table. */
+ struct rte_rib *rib; /**< Prefixes of all the VRFs, nh = prefix ID. */
+ struct fib_vrf_ht ht; /**< (VRF, prefix ID) -> nexthop. */
+};
+
+static inline void
+dir24_8_vrf_lookup_bulk(void *p, const uint16_t *vrf_ids, const uint32_t *ips,
+ uint64_t *next_hops, const unsigned int n)
+{
+ struct dir24_8_vrf_tbl *dp = p;
+
+ dir24_8_lookup_bulk(dp->lpm, ips, next_hops, n, RTE_FIB_DIR24_8_4B);
+ fib_vrf_ht_resolve(&dp->ht, vrf_ids, next_hops, n);
+}
+
+static inline void
+dir24_8_vrf_lookup_bulk_be(void *p, const uint16_t *vrf_ids,
+ const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
+{
+ struct dir24_8_vrf_tbl *dp = p;
+
+ dir24_8_lookup_bulk_4b_be(dp->lpm, NULL, ips, next_hops, n);
+ fib_vrf_ht_resolve(&dp->ht, vrf_ids, next_hops, n);
+}
+
+void *
+dir24_8_vrf_create(const char *name, int socket_id,
+ struct rte_fib_conf *fib_conf);
+
+void
+dir24_8_vrf_free(void *p);
+
+rte_fib_lookup_fn_t
+dir24_8_vrf_get_lookup_fn(void *p, enum rte_fib_lookup_type type,
+ bool be_addr);
+
+int
+dir24_8_vrf_modify(struct rte_fib *fib, uint16_t vrf_id, uint32_t ip,
+ uint8_t depth, uint64_t next_hop, int op);
+
+#endif /* _DIR24_8_VRF_H_ */
diff --git a/lib/fib/fib_vrf.c b/lib/fib/fib_vrf.c
new file mode 100644
index 0000000000..5213e3a64a
--- /dev/null
+++ b/lib/fib/fib_vrf.c
@@ -0,0 +1,1256 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Intel Corporation
+ */
+
+#include <stdalign.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <string.h>
+
+#include <rte_bitops.h>
+#include <rte_debug.h>
+#include <rte_errno.h>
+#include <rte_hash.h>
+#include <rte_hash_crc.h>
+#include <rte_malloc.h>
+#include <rte_random.h>
+
+#include "fib_log.h"
+#include "fib_vrf.h"
+
+/* VRF records of a RIB node: an inline block extended by chained chunks. */
+#define FIB_VRF_EXT_RECS 5
+#define FIB_VRF_CHUNK_RECS 24
+#define FIB_VRF_NO_VRF UINT16_MAX
+
+/*
+ * Smallest and largest hash table sizes (in buckets). With at least 2^17
+ * buckets, the records of a prefix in different VRFs never share a bucket.
+ */
+#define FIB_VRF_HT_MIN_LOG2 17
+#define FIB_VRF_HT_MAX_LOG2 24
+
+/* CRC32C polynomial and x^-64 mod P, used to invert the bucket hash. */
+#define FIB_VRF_CRC_POLY 0x1EDC6F41
+#define FIB_VRF_CRC_INV_X64 0xA6733E67
+
+struct fib_vrf_chunk {
+ struct fib_vrf_chunk *next;
+ uint32_t owner_mask;
+ uint16_t vrf[FIB_VRF_CHUNK_RECS];
+ uint64_t nh[FIB_VRF_CHUNK_RECS];
+};
+
+struct fib_vrf_node_ext {
+ struct fib_vrf_chunk *head;
+ uint32_t owner_mask;
+ uint16_t num_recs; /**< Records are stored contiguously. */
+ uint16_t vrf[FIB_VRF_EXT_RECS];
+ uint64_t nh[FIB_VRF_EXT_RECS];
+};
+
+/* Position of a record inside a node extension. */
+struct fib_vrf_slot {
+ struct fib_vrf_chunk **link; /**< Link pointing to chunk. */
+ struct fib_vrf_chunk *chunk; /**< NULL while in the inline block. */
+ unsigned int idx;
+};
+
+static inline uint16_t *
+slot_vrf(struct fib_vrf_node_ext *ext, const struct fib_vrf_slot *s)
+{
+ return (s->chunk != NULL) ? &s->chunk->vrf[s->idx] : &ext->vrf[s->idx];
+}
+
+static inline uint64_t *
+slot_nh(struct fib_vrf_node_ext *ext, const struct fib_vrf_slot *s)
+{
+ return (s->chunk != NULL) ? &s->chunk->nh[s->idx] : &ext->nh[s->idx];
+}
+
+static inline uint32_t *
+slot_mask(struct fib_vrf_node_ext *ext, const struct fib_vrf_slot *s)
+{
+ return (s->chunk != NULL) ? &s->chunk->owner_mask : &ext->owner_mask;
+}
+
+static inline bool
+slot_is_owner(struct fib_vrf_node_ext *ext, const struct fib_vrf_slot *s)
+{
+ return (*slot_mask(ext, s) & RTE_BIT32(s->idx)) != 0;
+}
+
+static inline void
+slot_set_owner(struct fib_vrf_node_ext *ext, const struct fib_vrf_slot *s, bool owner)
+{
+ if (owner)
+ *slot_mask(ext, s) |= RTE_BIT32(s->idx);
+ else
+ *slot_mask(ext, s) &= ~RTE_BIT32(s->idx);
+}
+
+static inline void
+slot_first(struct fib_vrf_slot *s)
+{
+ s->link = NULL;
+ s->chunk = NULL;
+ s->idx = 0;
+}
+
+static inline bool
+slot_next(struct fib_vrf_node_ext *ext, struct fib_vrf_slot *s)
+{
+ unsigned int cap = (s->chunk != NULL) ? FIB_VRF_CHUNK_RECS : FIB_VRF_EXT_RECS;
+
+ if (++s->idx < cap)
+ return true;
+
+ s->link = (s->chunk != NULL) ? &s->chunk->next : &ext->head;
+ s->chunk = *s->link;
+ s->idx = 0;
+
+ return s->chunk != NULL;
+}
+
+static struct fib_vrf_chunk *
+chunk_alloc(void)
+{
+ struct fib_vrf_chunk *chunk;
+ unsigned int i;
+
+ chunk = rte_zmalloc(NULL, sizeof(*chunk), 0);
+ if (chunk == NULL)
+ return NULL;
+
+ for (i = 0; i < FIB_VRF_CHUNK_RECS; i++)
+ chunk->vrf[i] = FIB_VRF_NO_VRF;
+
+ return chunk;
+}
+
+static int
+slot_at(struct fib_vrf_node_ext *ext, unsigned int n, struct fib_vrf_slot *s)
+{
+ unsigned int k;
+
+ slot_first(s);
+ for (k = 0; k < n; k++) {
+ if (slot_next(ext, s))
+ continue;
+
+ *s->link = chunk_alloc();
+ if (*s->link == NULL)
+ return -ENOMEM;
+
+ s->chunk = *s->link;
+ }
+
+ return 0;
+}
+
+static void
+ext_init(struct fib_vrf_node_ext *ext)
+{
+ unsigned int i;
+
+ ext->head = NULL;
+ ext->owner_mask = 0;
+ ext->num_recs = 0;
+ for (i = 0; i < FIB_VRF_EXT_RECS; i++)
+ ext->vrf[i] = FIB_VRF_NO_VRF;
+}
+
+static void
+ext_free_chunks(struct fib_vrf_node_ext *ext)
+{
+ struct fib_vrf_chunk *chunk, *next;
+
+ for (chunk = ext->head; chunk != NULL; chunk = next) {
+ next = chunk->next;
+ rte_free(chunk);
+ }
+ ext->head = NULL;
+}
+
+static int
+ext_reserve(struct fib_vrf_node_ext *ext, unsigned int n)
+{
+ struct fib_vrf_slot s;
+
+ return (n == 0) ? 0 : slot_at(ext, n - 1, &s);
+}
+
+static bool
+ext_find(struct fib_vrf_node_ext *ext, uint16_t vrf, struct fib_vrf_slot *s)
+{
+ unsigned int k;
+
+ slot_first(s);
+ for (k = 0; k < ext->num_recs; k++, slot_next(ext, s)) {
+ if (*slot_vrf(ext, s) == vrf)
+ return true;
+ }
+
+ return false;
+}
+
+static bool
+ext_get_nh(struct fib_vrf_node_ext *ext, uint16_t vrf, uint64_t *nh)
+{
+ struct fib_vrf_slot s;
+
+ if (!ext_find(ext, vrf, &s))
+ return false;
+
+ *nh = *slot_nh(ext, &s);
+
+ return true;
+}
+
+static bool
+ext_is_owner(struct fib_vrf_node_ext *ext, uint16_t vrf)
+{
+ struct fib_vrf_slot s;
+
+ return ext_find(ext, vrf, &s) && slot_is_owner(ext, &s);
+}
+
+static unsigned int
+ext_count_owners(const struct fib_vrf_node_ext *ext)
+{
+ const struct fib_vrf_chunk *chunk;
+ unsigned int n = rte_popcount32(ext->owner_mask);
+
+ for (chunk = ext->head; chunk != NULL; chunk = chunk->next)
+ n += rte_popcount32(chunk->owner_mask);
+
+ return n;
+}
+
+static int
+ext_set(struct fib_vrf_node_ext *ext, uint16_t vrf, uint64_t nh, bool owner)
+{
+ struct fib_vrf_slot s;
+ int ret;
+
+ if (!ext_find(ext, vrf, &s)) {
+ ret = slot_at(ext, ext->num_recs, &s);
+ if (ret != 0)
+ return ret;
+
+ ext->num_recs++;
+ }
+
+ *slot_vrf(ext, &s) = vrf;
+ *slot_nh(ext, &s) = nh;
+ slot_set_owner(ext, &s, owner);
+
+ return 0;
+}
+
+static int
+ext_del(struct fib_vrf_node_ext *ext, uint16_t vrf)
+{
+ struct fib_vrf_slot s, last;
+
+ if (!ext_find(ext, vrf, &s))
+ return -ENOENT;
+
+ /* Records are contiguous: move the last one into the hole. */
+ slot_at(ext, ext->num_recs - 1, &last);
+ *slot_vrf(ext, &s) = *slot_vrf(ext, &last);
+ *slot_nh(ext, &s) = *slot_nh(ext, &last);
+ slot_set_owner(ext, &s, slot_is_owner(ext, &last));
+ *slot_vrf(ext, &last) = FIB_VRF_NO_VRF;
+ *slot_nh(ext, &last) = 0;
+ slot_set_owner(ext, &last, false);
+ ext->num_recs--;
+
+ /* The last chunk became empty. */
+ if (last.chunk != NULL && last.idx == 0) {
+ *last.link = NULL;
+ rte_free(last.chunk);
+ }
+
+ return ext->num_recs;
+}
+
+static inline bool
+bmp_get(const uint64_t *bmp, uint32_t i)
+{
+ return (bmp[i / 64] & RTE_BIT64(i % 64)) != 0;
+}
+
+static inline void
+bmp_set(uint64_t *bmp, uint32_t i)
+{
+ bmp[i / 64] |= RTE_BIT64(i % 64);
+}
+
+static inline void
+bmp_clear(uint64_t *bmp, uint32_t i)
+{
+ bmp[i / 64] &= ~RTE_BIT64(i % 64);
+}
+
+static inline uint64_t
+bucket_key(const struct fib_vrf_bucket *b, unsigned int j)
+{
+ return rte_atomic_load_explicit(&b->key[j], rte_memory_order_relaxed);
+}
+
+static inline void
+bucket_set_key(struct fib_vrf_bucket *b, unsigned int j, uint64_t key)
+{
+ rte_atomic_store_explicit(&b->key[j], key, rte_memory_order_release);
+}
+
+static int
+bucket_find(const struct fib_vrf_bucket *b, uint64_t key)
+{
+ unsigned int j;
+
+ for (j = 0; j < FIB_VRF_HT_ASSOC; j++) {
+ if (bucket_key(b, j) == key)
+ return j;
+ }
+
+ return -1;
+}
+
+static void
+bucket_refresh_full(struct fib_vrf_ht *ht, uint32_t bidx)
+{
+ if (bucket_find(&ht->buckets[bidx], 0) < 0)
+ bmp_set(ht->full_bmp, bidx);
+ else
+ bmp_clear(ht->full_bmp, bidx);
+}
+
+static void *
+pid_map_node(const struct fib_vrf_ht *ht, uint32_t prefix_id)
+{
+ void *node;
+
+ if (rte_hash_lookup_data(ht->pid_map, &prefix_id, &node) < 0)
+ return NULL;
+
+ return node;
+}
+
+static inline struct fib_vrf_node_ext *
+node_ext(const struct fib_vrf_ht *ht, void *node)
+{
+ return ht->ops->rib_get_ext(node);
+}
+
+static inline uint32_t
+node_pid(const struct fib_vrf_ht *ht, const void *node)
+{
+ return (uint32_t)ht->ops->rib_get_nh(node);
+}
+
+/* A bucket is polarized when it holds a record of a prefix referenced by more than one VRF. */
+static void
+bucket_refresh_polarized(struct fib_vrf_ht *ht, uint32_t bidx)
+{
+ const struct fib_vrf_bucket *b = &ht->buckets[bidx];
+ unsigned int j;
+
+ for (j = 0; j < FIB_VRF_HT_ASSOC; j++) {
+ uint64_t key = bucket_key(b, j);
+ void *node;
+
+ if (key == 0)
+ continue;
+
+ node = pid_map_node(ht, (uint32_t)key);
+ if (node != NULL && node_ext(ht, node)->num_recs > 1) {
+ bmp_set(ht->polarized_bmp, bidx);
+ return;
+ }
+ }
+
+ bmp_clear(ht->polarized_bmp, bidx);
+}
+
+static int
+ht_delete(struct fib_vrf_ht *ht, uint16_t vrf, uint32_t prefix_id)
+{
+ uint64_t key = fib_vrf_key(vrf, prefix_id);
+ uint32_t bidx = fib_vrf_bucket_idx(ht, key);
+ struct fib_vrf_bucket *b = &ht->buckets[bidx];
+ int j = bucket_find(b, key);
+
+ if (j < 0)
+ return -ENOENT;
+
+ bucket_set_key(b, j, 0);
+ b->val[j] = 0;
+ bmp_clear(ht->full_bmp, bidx);
+ bucket_refresh_polarized(ht, bidx);
+
+ return 0;
+}
+
+/* Insert a record, or update it if it exists. */
+static int
+ht_upsert(struct fib_vrf_ht *ht, uint16_t vrf, uint32_t prefix_id, uint64_t next_hop)
+{
+ uint64_t key = fib_vrf_key(vrf, prefix_id);
+ uint32_t bidx = fib_vrf_bucket_idx(ht, key);
+ struct fib_vrf_bucket *b = &ht->buckets[bidx];
+ int j;
+
+ j = bucket_find(b, key);
+ if (j >= 0) {
+ b->val[j] = next_hop;
+ return 0;
+ }
+
+ j = bucket_find(b, 0);
+ if (j < 0)
+ return -ENOSPC;
+
+ /* Publish the value before the key. */
+ b->val[j] = next_hop;
+ bucket_set_key(b, j, key);
+ bucket_refresh_full(ht, bidx);
+
+ return 0;
+}
+
+static uint32_t
+invert_bits(uint32_t x)
+{
+ uint32_t y = 0;
+ int i;
+
+ for (i = 0; i < 32; i++) {
+ y = (y << 1) | (x & 1);
+ x >>= 1;
+ }
+
+ return y;
+}
+
+static uint32_t
+gf2_mul(uint32_t a, uint32_t b, uint32_t poly)
+{
+ uint64_t product = 0;
+ uint64_t r_poly = poly | RTE_BIT64(32);
+ int i;
+
+ for (; b != 0; b &= (b - 1))
+ product ^= (uint64_t)a << rte_ctz32(b);
+
+ for (i = 63; i >= 32; i--) {
+ if (product & RTE_BIT64(i))
+ product ^= r_poly << (i - 32);
+ }
+
+ return (uint32_t)product;
+}
+
+static uint32_t
+find_collision(uint16_t vrf, uint32_t desired_crc)
+{
+ uint32_t diff = rte_hash_crc_8byte(fib_vrf_key(vrf, 0), 0) ^ desired_crc;
+
+ return invert_bits(gf2_mul(invert_bits(diff), FIB_VRF_CRC_INV_X64, FIB_VRF_CRC_POLY));
+}
+
+static bool
+pid_is_usable(const struct fib_vrf_ht *ht, uint32_t prefix_id)
+{
+ return prefix_id != 0 && prefix_id <= FIB_VRF_PREFIX_ID_MAX &&
+ rte_hash_lookup(ht->pid_map, &prefix_id) < 0;
+}
+
+static int
+pid_for_bucket(const struct fib_vrf_ht *ht, uint16_t vrf, uint32_t bidx)
+{
+ uint32_t i, prefix_id;
+
+ for (i = 0; i < (1U << (32 - ht->bits)); i++) {
+ prefix_id = find_collision(vrf, bidx | i << ht->bits);
+ if (pid_is_usable(ht, prefix_id))
+ return (int)prefix_id;
+ }
+ return -ENOSPC;
+}
+
+/*
+ * First bucket of [from, to) with a free slot (and no record of a multi-VRF
+ * prefix when unpolarized is set), UINT32_MAX if none.
+ */
+static uint32_t
+first_free_bucket(const struct fib_vrf_ht *ht, uint32_t from, uint32_t to, bool unpolarized)
+{
+ uint32_t i, bidx;
+
+ for (i = from / 64; i * 64 < to; i++) {
+ uint64_t available = ~ht->full_bmp[i];
+
+ if (unpolarized)
+ available &= ~ht->polarized_bmp[i];
+ if (i == from / 64)
+ available &= UINT64_MAX << (from % 64);
+ if (available != 0) {
+ bidx = i * 64 + rte_ctz64(available);
+ return (bidx < to) ? bidx : UINT32_MAX;
+ }
+ }
+
+ return UINT32_MAX;
+}
+
+static inline uint32_t
+vrf_bucket_offset(const struct fib_vrf_ht *ht, uint16_t vrf_diff)
+{
+ return fib_vrf_bucket_idx(ht, fib_vrf_key(vrf_diff, 0));
+}
+
+static inline bool
+vrf_has_room(const struct fib_vrf_ht *ht, uint32_t bidx, uint16_t vrf,
+ uint16_t other_vrf)
+{
+ return !bmp_get(ht->full_bmp, bidx ^ vrf_bucket_offset(ht, vrf ^ other_vrf));
+}
+
+static bool
+records_have_room(const struct fib_vrf_ht *ht, uint16_t vrf, uint32_t bidx,
+ struct fib_vrf_node_ext *ext, uint16_t extra_vrf)
+{
+ struct fib_vrf_slot s;
+ unsigned int k;
+
+ if (extra_vrf != FIB_VRF_NO_VRF && !vrf_has_room(ht, bidx, vrf, extra_vrf))
+ return false;
+
+ if (ext == NULL)
+ return true;
+
+ slot_first(&s);
+ for (k = 0; k < ext->num_recs; k++, slot_next(ext, &s)) {
+ if (!vrf_has_room(ht, bidx, vrf, *slot_vrf(ext, &s)))
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Allocate a prefix ID whose record for vrf lands in a bucket with room (an
+ * unpolarized one when requested), and whose records for the VRFs referencing
+ * ext and for extra_vrf fit as well. The buckets are scanned from start,
+ * wrapping around.
+ */
+static int
+pid_alloc(const struct fib_vrf_ht *ht, uint16_t vrf, uint32_t start,
+ bool unpolarized, struct fib_vrf_node_ext *ext, uint16_t extra_vrf)
+{
+ uint32_t bidx, end = ht->mask + 1;
+ unsigned int pass;
+ int ret;
+
+ for (pass = 0; pass < 2; pass++) {
+ for (bidx = first_free_bucket(ht, start, end, unpolarized);
+ bidx != UINT32_MAX;
+ bidx = first_free_bucket(ht, bidx + 1, end, unpolarized)) {
+ if (!records_have_room(ht, vrf, bidx, ext, extra_vrf))
+ continue;
+ ret = pid_for_bucket(ht, vrf, bidx);
+ if (ret >= 0)
+ return ret;
+ }
+ end = start;
+ start = 0;
+ }
+
+ return -ENOSPC;
+}
+
+static void
+node_mark_polarized(struct fib_vrf_ht *ht, void *node)
+{
+ struct fib_vrf_node_ext *ext = node_ext(ht, node);
+ uint32_t prefix_id = node_pid(ht, node);
+ struct fib_vrf_slot s;
+ unsigned int k;
+
+ if (ext->num_recs < 2)
+ return;
+
+ slot_first(&s);
+ for (k = 0; k < ext->num_recs; k++, slot_next(ext, &s))
+ bmp_set(ht->polarized_bmp,
+ fib_vrf_bucket_idx(ht, fib_vrf_key(*slot_vrf(ext, &s), prefix_id)));
+}
+
+static void
+node_refresh_polarized(struct fib_vrf_ht *ht, void *node)
+{
+ struct fib_vrf_node_ext *ext = node_ext(ht, node);
+ uint32_t prefix_id = node_pid(ht, node);
+ struct fib_vrf_slot s;
+ unsigned int k;
+
+ slot_first(&s);
+ for (k = 0; k < ext->num_recs; k++, slot_next(ext, &s))
+ bucket_refresh_polarized(ht, fib_vrf_bucket_idx(ht,
+ fib_vrf_key(*slot_vrf(ext, &s), prefix_id)));
+}
+
+/*
+ * Closest existing prefix strictly covering ip/depth.
+ * Can not just call rib_lookup_exact and rib_get_parent
+ * since target prefix may not present in the RIB.
+ */
+static void *
+nearest_ancestor(const struct fib_vrf_ht *ht, void *rib, const void *ip,
+ uint8_t depth)
+{
+ alignas(uint64_t) uint8_t node_ip[FIB_VRF_IP_MAX_SZ];
+ void *node = ht->ops->rib_lookup(rib, ip);
+ uint8_t node_depth;
+
+ while (node != NULL) {
+ ht->ops->rib_get_prefix(node, node_ip, &node_depth);
+ if (node_depth < depth)
+ break;
+ node = ht->ops->rib_lookup_parent(node);
+ }
+ return node;
+}
+
+/* retrieve nexthop of the nearest ancestor of the node within a given VRF */
+static bool
+ancestor_nexthop(const struct fib_vrf_ht *ht, void *node, uint16_t vrf, uint64_t *nh)
+{
+ void *anc;
+
+ for (anc = ht->ops->rib_lookup_parent(node); anc != NULL;
+ anc = ht->ops->rib_lookup_parent(anc)) {
+ if (ext_get_nh(node_ext(ht, anc), vrf, nh))
+ return true;
+ }
+
+ return false;
+}
+
+static int
+node_move(struct fib_vrf_ht *ht, void *lpm, void *rib, void *node, uint32_t new_pid)
+{
+ struct fib_vrf_node_ext *ext = node_ext(ht, node);
+ alignas(uint64_t) uint8_t ip[FIB_VRF_IP_MAX_SZ];
+ uint32_t old_pid = node_pid(ht, node);
+ struct fib_vrf_slot s;
+ unsigned int k, n;
+ uint8_t depth;
+ int ret;
+
+ slot_first(&s);
+ for (n = 0; n < ext->num_recs; n++, slot_next(ext, &s)) {
+ ret = ht_upsert(ht, *slot_vrf(ext, &s), new_pid, *slot_nh(ext, &s));
+ if (ret != 0)
+ goto drop_records;
+ }
+
+ ret = rte_hash_add_key_data(ht->pid_map, &new_pid, node);
+ if (ret < 0)
+ goto drop_records;
+
+ /* Switch the LPM table to the new ID. */
+ ht->ops->rib_get_prefix(node, ip, &depth);
+ /* make sure all previous ht_upserts are visible before changing LPM part */
+ rte_atomic_thread_fence(rte_memory_order_release);
+ ret = ht->ops->lpm_modify(lpm, rib, ip, depth, new_pid, FIB_VRF_ADD);
+ if (ret != 0) {
+ rte_hash_del_key(ht->pid_map, &new_pid);
+ goto drop_records;
+ }
+
+ /* clean up old records with old prefix_id */
+ slot_first(&s);
+ for (k = 0; k < ext->num_recs; k++, slot_next(ext, &s))
+ ht_delete(ht, *slot_vrf(ext, &s), old_pid);
+
+ rte_hash_del_key(ht->pid_map, &old_pid);
+ node_mark_polarized(ht, node);
+
+ return 0;
+
+drop_records:
+ slot_first(&s);
+ for (k = 0; k < n; k++, slot_next(ext, &s))
+ ht_delete(ht, *slot_vrf(ext, &s), new_pid);
+
+ return ret;
+}
+
+/*
+ * Before a second VRF starts referencing a prefix whose single record sits in a
+ * polarized bucket, try to move that record to an unpolarized bucket by giving
+ * the prefix a new ID. The node is left untouched when no suitable prefix ID is
+ * available.
+ */
+static void
+node_relocate_if_polarized(struct fib_vrf_ht *ht, void *lpm, void *rib, void *node,
+ uint16_t new_vrf)
+{
+ struct fib_vrf_node_ext *ext = node_ext(ht, node);
+ uint16_t owner_vrf;
+ int ret;
+
+ if (ext->num_recs != 1 || ext->vrf[0] == new_vrf)
+ return;
+
+ owner_vrf = ext->vrf[0];
+ if (!bmp_get(ht->polarized_bmp,
+ fib_vrf_bucket_idx(ht, fib_vrf_key(owner_vrf, node_pid(ht, node)))))
+ return;
+
+ ret = pid_alloc(ht, owner_vrf, rte_rand_max(ht->mask + 1), true, NULL, new_vrf);
+ if (ret < 0)
+ return;
+
+ node_move(ht, lpm, rib, node, (uint32_t)ret);
+}
+
+static int
+record_upsert(struct fib_vrf_ht *ht, void *lpm, void *rib, uint16_t vrf,
+ uint32_t prefix_id, uint64_t next_hop)
+{
+ void *victims[FIB_VRF_HT_ASSOC];
+ const struct fib_vrf_bucket *b;
+ unsigned int i, j, n = 0;
+ int ret;
+
+ ret = ht_upsert(ht, vrf, prefix_id, next_hop);
+ if (ret != -ENOSPC)
+ return ret;
+
+ b = &ht->buckets[fib_vrf_bucket_idx(ht, fib_vrf_key(vrf, prefix_id))];
+ for (j = 0; j < FIB_VRF_HT_ASSOC; j++) {
+ uint32_t pid = (uint32_t)bucket_key(b, j);
+ void *node = pid_map_node(ht, pid);
+ uint16_t num_recs;
+
+ if (pid == prefix_id || node == NULL)
+ continue;
+
+ num_recs = node_ext(ht, node)->num_recs;
+
+ for (i = n; i > 0 && node_ext(ht, victims[i - 1])->num_recs > num_recs; i--)
+ victims[i] = victims[i - 1];
+ victims[i] = node;
+ n++;
+ }
+
+ for (i = 0; i < n; i++) {
+ struct fib_vrf_node_ext *ext = node_ext(ht, victims[i]);
+ uint32_t start = (ext->num_recs == 1) ? 0 : rte_rand_max(ht->mask + 1);
+
+ ret = pid_alloc(ht, ext->vrf[0], start, false, ext, FIB_VRF_NO_VRF);
+ if (ret < 0 || node_move(ht, lpm, rib, victims[i], (uint32_t)ret) != 0)
+ continue;
+
+ return ht_upsert(ht, vrf, prefix_id, next_hop);
+ }
+
+ return -ENOSPC;
+}
+
+/* Update the records of vrf in the prefixes covered by ip/depth that do not belong to vrf itself */
+static int
+subtree_set(struct fib_vrf_ht *ht, void *lpm, void *rib, const void *ip,
+ uint8_t depth, uint16_t vrf, uint64_t nh, bool have_nh)
+{
+ alignas(uint64_t) uint8_t child_ip[FIB_VRF_IP_MAX_SZ];
+ uint8_t child_depth;
+ void *child;
+ int ret;
+
+ for (child = ht->ops->rib_get_nxt_cover(rib, ip, depth, NULL);
+ child != NULL;
+ child = ht->ops->rib_get_nxt_cover(rib, ip, depth, child)) {
+ struct fib_vrf_node_ext *ext = node_ext(ht, child);
+
+ /* The subtree resolves through this route of vrf. */
+ if (ext_is_owner(ext, vrf))
+ continue;
+
+ if (have_nh) {
+ node_relocate_if_polarized(ht, lpm, rib, child, vrf);
+ ret = ext_set(ext, vrf, nh, false);
+ if (ret != 0)
+ return ret;
+ ret = record_upsert(ht, lpm, rib, vrf,
+ node_pid(ht, child), nh);
+ if (ret != 0)
+ return ret;
+ node_mark_polarized(ht, child);
+ } else {
+ ret = ext_del(ext, vrf);
+ /* Nothing inherited here, hence nothing below. */
+ if (ret == -ENOENT)
+ continue;
+ ht_delete(ht, vrf, node_pid(ht, child));
+ if (ret == 1)
+ node_refresh_polarized(ht, child);
+ }
+
+ ht->ops->rib_get_prefix(child, child_ip, &child_depth);
+ ret = subtree_set(ht, lpm, rib, child_ip, child_depth, vrf, nh, have_nh);
+ if (ret != 0)
+ return ret;
+ }
+
+ return 0;
+}
+
+/* Retire a prefix whose last owner is gone, with all its records. */
+static int
+remove_node(struct fib_vrf_ht *ht, void *lpm, void *rib, void *node, const void *ip, uint8_t depth)
+{
+ struct fib_vrf_node_ext *ext = node_ext(ht, node);
+ struct fib_vrf_chunk *chunk, *next;
+ uint16_t inline_vrf[FIB_VRF_EXT_RECS];
+ uint32_t prefix_id = node_pid(ht, node);
+ unsigned int i;
+ int ret;
+
+ /* The node is freed by the LPM delete, save what is needed. */
+ memcpy(inline_vrf, ext->vrf, sizeof(inline_vrf));
+ chunk = ext->head;
+
+ ret = ht->ops->lpm_modify(lpm, rib, ip, depth, 0, FIB_VRF_DEL);
+ if (ret != 0)
+ return ret;
+
+ rte_hash_del_key(ht->pid_map, &prefix_id);
+
+ for (i = 0; i < FIB_VRF_EXT_RECS; i++) {
+ if (inline_vrf[i] != FIB_VRF_NO_VRF)
+ ht_delete(ht, inline_vrf[i], prefix_id);
+ }
+ for (; chunk != NULL; chunk = next) {
+ for (i = 0; i < FIB_VRF_CHUNK_RECS; i++) {
+ if (chunk->vrf[i] != FIB_VRF_NO_VRF)
+ ht_delete(ht, chunk->vrf[i], prefix_id);
+ }
+ next = chunk->next;
+ rte_free(chunk);
+ }
+
+ return 0;
+}
+
+/* Drop the records of a prefix ID the LPM table never exposed. */
+static void
+drop_new_records(struct fib_vrf_ht *ht, uint16_t vrf, uint32_t prefix_id,
+ struct fib_vrf_node_ext *inherit, unsigned int n_inherit)
+{
+ struct fib_vrf_slot s;
+ unsigned int k, done = 0;
+
+ ht_delete(ht, vrf, prefix_id);
+ if (inherit == NULL)
+ return;
+
+ slot_first(&s);
+ for (k = 0; k < inherit->num_recs && done < n_inherit; k++, slot_next(inherit, &s)) {
+ if (*slot_vrf(inherit, &s) == vrf)
+ continue;
+
+ ht_delete(ht, *slot_vrf(inherit, &s), prefix_id);
+ done++;
+ }
+}
+
+static int
+route_add_new(struct fib_vrf_ht *ht, void *lpm, void *rib, uint16_t vrf, const void *ip,
+ uint8_t depth, uint64_t next_hop)
+{
+ struct fib_vrf_node_ext *ext, *pext = NULL;
+ struct fib_vrf_slot s;
+ void *parent, *node;
+ uint64_t old_nh = 0;
+ uint32_t prefix_id;
+ unsigned int k, n_inherit = 0;
+ bool have_old = false;
+ int ret;
+
+ /*
+ * The new prefix inherits every VRF referencing its nearest ancestor:
+ * those VRFs keep resolving to the ancestor nexthop in its range.
+ */
+ parent = nearest_ancestor(ht, rib, ip, depth);
+ if (parent != NULL) {
+ pext = node_ext(ht, parent);
+ have_old = ext_get_nh(pext, vrf, &old_nh);
+ n_inherit = pext->num_recs - (have_old ? 1 : 0);
+ }
+
+ /* A multi-VRF prefix goes to an unpolarized bucket if there is one. */
+ ret = -ENOSPC;
+ if (n_inherit > 0)
+ ret = pid_alloc(ht, vrf, 0, true, pext, FIB_VRF_NO_VRF);
+ if (ret < 0)
+ ret = pid_alloc(ht, vrf, 0, false, pext, FIB_VRF_NO_VRF);
+ if (ret < 0)
+ return ret;
+
+ prefix_id = (uint32_t)ret;
+
+ /* Records must resolve before the LPM table returns the ID. */
+ ret = ht_upsert(ht, vrf, prefix_id, next_hop);
+ if (ret != 0)
+ return ret;
+
+ if (pext != NULL) {
+ unsigned int done = 0;
+
+ slot_first(&s);
+ for (k = 0; k < pext->num_recs; k++, slot_next(pext, &s)) {
+ if (*slot_vrf(pext, &s) == vrf)
+ continue;
+ ret = ht_upsert(ht, *slot_vrf(pext, &s), prefix_id,
+ *slot_nh(pext, &s));
+ if (ret != 0) {
+ drop_new_records(ht, vrf, prefix_id, pext, done);
+ return ret;
+ }
+
+ done++;
+ }
+ }
+
+ /* The records must be visible before the LPM table returns the ID. */
+ rte_atomic_thread_fence(rte_memory_order_release);
+ ret = ht->ops->lpm_modify(lpm, rib, ip, depth, prefix_id, FIB_VRF_ADD);
+ if (ret != 0)
+ goto drop_records;
+
+ node = ht->ops->rib_lookup_exact(rib, ip, depth);
+ ext = node_ext(ht, node);
+ ext_init(ext);
+ ret = ext_reserve(ext, 1 + n_inherit);
+ if (ret != 0)
+ goto remove_lpm;
+
+ ext_set(ext, vrf, next_hop, true);
+ if (pext != NULL) {
+ slot_first(&s);
+ for (k = 0; k < pext->num_recs; k++, slot_next(pext, &s)) {
+ if (*slot_vrf(pext, &s) != vrf)
+ ext_set(ext, *slot_vrf(pext, &s), *slot_nh(pext, &s), false);
+ }
+ }
+
+ ret = rte_hash_add_key_data(ht->pid_map, &prefix_id, node);
+ if (ret < 0)
+ goto remove_lpm;
+
+ node_mark_polarized(ht, node);
+
+ /* Sub-prefixes that resolved vrf through the ancestor now use nh. */
+ ret = subtree_set(ht, lpm, rib, ip, depth, vrf, next_hop, true);
+ if (ret != 0) {
+ subtree_set(ht, lpm, rib, ip, depth, vrf, old_nh, have_old);
+ remove_node(ht, lpm, rib, node, ip, depth);
+ return ret;
+ }
+
+ return 0;
+
+remove_lpm:
+ ext_free_chunks(ext);
+ ht->ops->lpm_modify(lpm, rib, ip, depth, 0, FIB_VRF_DEL);
+drop_records:
+ drop_new_records(ht, vrf, prefix_id, pext, n_inherit);
+
+ return ret;
+}
+
+static int
+route_add_existing(struct fib_vrf_ht *ht, void *lpm, void *rib, void *node,
+ uint16_t vrf, const void *ip, uint8_t depth, uint64_t next_hop)
+{
+ struct fib_vrf_node_ext *ext = node_ext(ht, node);
+ uint64_t old_nh = 0;
+ uint32_t prefix_id;
+ bool have_old, was_owner;
+ int ret;
+
+ was_owner = ext_is_owner(ext, vrf);
+ have_old = ext_get_nh(ext, vrf, &old_nh);
+
+ if (!was_owner)
+ node_relocate_if_polarized(ht, lpm, rib, node, vrf);
+
+ prefix_id = node_pid(ht, node);
+
+ ret = ext_set(ext, vrf, next_hop, true);
+ if (ret != 0)
+ return ret;
+
+ ret = record_upsert(ht, lpm, rib, vrf, prefix_id, next_hop);
+ if (ret != 0) {
+ if (have_old)
+ ext_set(ext, vrf, old_nh, was_owner);
+ else
+ ext_del(ext, vrf);
+ return ret;
+ }
+
+ node_mark_polarized(ht, node);
+
+ if (was_owner && old_nh == next_hop)
+ return 0;
+
+ ret = subtree_set(ht, lpm, rib, ip, depth, vrf, next_hop, true);
+ if (ret != 0) {
+ subtree_set(ht, lpm, rib, ip, depth, vrf, old_nh, have_old);
+ /* Making room for the subtree records may have moved the node. */
+ prefix_id = node_pid(ht, node);
+ if (have_old) {
+ ext_set(ext, vrf, old_nh, was_owner);
+ ht_upsert(ht, vrf, prefix_id, old_nh);
+ } else {
+ ext_del(ext, vrf);
+ ht_delete(ht, vrf, prefix_id);
+ }
+
+ return ret;
+ }
+
+ return 0;
+}
+
+static int
+route_del(struct fib_vrf_ht *ht, void *lpm, void *rib, void *node,
+ uint16_t vrf, const void *ip, uint8_t depth)
+{
+ struct fib_vrf_node_ext *ext;
+ uint64_t ls_nh = 0;
+ bool have_less_specific;
+ int ret;
+
+ ext = node_ext(ht, node);
+ if (!ext_is_owner(ext, vrf))
+ return -ENOENT;
+
+ /* find nexthop of the less specific route covering given node for the given VRF */
+ have_less_specific = ancestor_nexthop(ht, node, vrf, &ls_nh);
+
+ if (ext_count_owners(ext) == 1) {
+ ret = remove_node(ht, lpm, rib, node, ip, depth);
+ if (ret != 0)
+ return ret;
+ } else {
+ uint32_t prefix_id = node_pid(ht, node);
+
+ if (have_less_specific) {
+ ext_set(ext, vrf, ls_nh, false);
+ ht_upsert(ht, vrf, prefix_id, ls_nh);
+ } else {
+ ext_del(ext, vrf);
+ ht_delete(ht, vrf, prefix_id);
+ }
+
+ if (ext->num_recs == 1)
+ node_refresh_polarized(ht, node);
+ }
+
+ /* Only updates and deletions are left, they cannot run out of space. */
+ return subtree_set(ht, lpm, rib, ip, depth, vrf, ls_nh, have_less_specific);
+}
+
+/* The default route is the per-VRF nexthop returned on a hash miss. */
+static void
+default_route_modify(struct fib_vrf_ht *ht, uint16_t vrf, uint64_t next_hop, enum fib_vrf_op op)
+{
+ ht->def_nh[vrf] = (op == FIB_VRF_ADD) ? next_hop : ht->conf_def_nh[vrf];
+}
+
+int
+fib_vrf_modify(struct fib_vrf_ht *ht, void *lpm, void *rib, void *vrf_rib, uint16_t vrf,
+ const void *ip, uint8_t depth, uint64_t next_hop, enum fib_vrf_op op)
+{
+ void *node, *vrf_node;
+ int ret = 0;
+ bool inserted = false;
+
+ if (ht == NULL || lpm == NULL || rib == NULL || vrf_rib == NULL ||
+ ip == NULL || vrf >= ht->num_vrfs)
+ return -EINVAL;
+
+ /* The RIB of the VRF mirrors its routes and tells which ones exist. */
+ vrf_node = ht->ops->rib_lookup_exact(vrf_rib, ip, depth);
+ switch (op) {
+ case FIB_VRF_ADD:
+ if (vrf_node == NULL) {
+ vrf_node = ht->ops->rib_insert(vrf_rib, ip, depth);
+ if (vrf_node == NULL)
+ return -rte_errno;
+
+ inserted = true;
+ }
+
+ if (depth == 0) {
+ default_route_modify(ht, vrf, next_hop, op);
+ } else {
+ node = ht->ops->rib_lookup_exact(rib, ip, depth);
+ if (node != NULL)
+ ret = route_add_existing(ht, lpm, rib, node, vrf,
+ ip, depth, next_hop);
+ else
+ ret = route_add_new(ht, lpm, rib, vrf, ip, depth, next_hop);
+ }
+
+ if (ret != 0 && inserted)
+ ht->ops->rib_remove(vrf_rib, ip, depth);
+
+ if (ret == 0)
+ ht->ops->rib_set_nh(vrf_node, next_hop);
+
+ break;
+ case FIB_VRF_DEL:
+ if (vrf_node == NULL)
+ return -ENOENT;
+
+ if (depth == 0)
+ default_route_modify(ht, vrf, next_hop, op);
+ else {
+ node = ht->ops->rib_lookup_exact(rib, ip, depth);
+ if (node == NULL)
+ return -ENOENT;
+
+ ret = route_del(ht, lpm, rib, node, vrf, ip, depth);
+ }
+
+ if (ret == 0)
+ ht->ops->rib_remove(vrf_rib, ip, depth);
+
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return ret;
+}
+
+size_t
+fib_vrf_rib_ext_sz(void)
+{
+ return sizeof(struct fib_vrf_node_ext);
+}
+
+static uint64_t *
+bitmap_alloc(const char *name, uint32_t nb_bits, int socket_id)
+{
+ return rte_zmalloc_socket(name, RTE_ALIGN_CEIL(nb_bits, 64) / 8,
+ RTE_CACHE_LINE_SIZE, socket_id);
+}
+
+int
+fib_vrf_ht_init(struct fib_vrf_ht *ht, const char *name, int socket_id,
+ const struct fib_vrf_ops *ops, uint16_t num_vrfs, uint64_t default_nh,
+ const uint64_t *vrf_default_nh, uint32_t max_routes)
+{
+ char mem_name[64];
+ struct rte_hash_parameters hash_params = {
+ .key_len = sizeof(uint32_t),
+ .hash_func = rte_hash_crc,
+ .socket_id = socket_id,
+ };
+ uint64_t nb_records, nb_buckets;
+ uint16_t v;
+
+ if (ht == NULL || name == NULL || ops == NULL || num_vrfs == 0 || max_routes == 0)
+ return -EINVAL;
+
+ memset(ht, 0, sizeof(*ht));
+ ht->ops = ops;
+ ht->num_vrfs = num_vrfs;
+
+ /*
+ * Worst case every VRF references every prefix: size the table for
+ * half load with that many records.
+ */
+ nb_records = (uint64_t)max_routes * num_vrfs * 2;
+ nb_buckets = nb_records / FIB_VRF_HT_ASSOC;
+ nb_buckets = RTE_MAX(nb_buckets, RTE_BIT64(FIB_VRF_HT_MIN_LOG2));
+ nb_buckets = RTE_MIN(nb_buckets, RTE_BIT64(FIB_VRF_HT_MAX_LOG2));
+
+ ht->bits = rte_log2_u64(nb_buckets);
+ if (RTE_BIT64(ht->bits) < nb_buckets)
+ ht->bits++;
+ ht->mask = RTE_BIT32(ht->bits) - 1;
+
+ snprintf(mem_name, sizeof(mem_name), "VRF_HT_%s", name);
+ ht->buckets = rte_zmalloc_socket(mem_name,
+ sizeof(struct fib_vrf_bucket) * RTE_BIT64(ht->bits),
+ RTE_CACHE_LINE_SIZE, socket_id);
+
+ snprintf(mem_name, sizeof(mem_name), "VRF_FULL_%s", name);
+ ht->full_bmp = bitmap_alloc(mem_name, RTE_BIT32(ht->bits), socket_id);
+
+ snprintf(mem_name, sizeof(mem_name), "VRF_POL_%s", name);
+ ht->polarized_bmp = bitmap_alloc(mem_name, RTE_BIT32(ht->bits),
+ socket_id);
+
+ snprintf(mem_name, sizeof(mem_name), "VRF_DEFNH_%s", name);
+ ht->def_nh = rte_zmalloc_socket(mem_name,
+ sizeof(uint64_t) * num_vrfs * 2, RTE_CACHE_LINE_SIZE, socket_id);
+ if (ht->buckets == NULL || ht->full_bmp == NULL ||
+ ht->polarized_bmp == NULL || ht->def_nh == NULL)
+ goto nomem;
+
+ ht->conf_def_nh = ht->def_nh + num_vrfs;
+ for (v = 0; v < num_vrfs; v++) {
+ ht->def_nh[v] = (vrf_default_nh != NULL) ?
+ vrf_default_nh[v] : default_nh;
+ ht->conf_def_nh[v] = ht->def_nh[v];
+ }
+
+ snprintf(mem_name, sizeof(mem_name), "VRF_PID_%s", name);
+ hash_params.name = mem_name;
+ /* x1.5 should be enough */
+ hash_params.entries = RTE_MAX(max_routes + max_routes / 2, 64U);
+ ht->pid_map = rte_hash_create(&hash_params);
+ if (ht->pid_map == NULL)
+ goto nomem;
+
+ return 0;
+
+nomem:
+ fib_vrf_ht_free(ht, NULL);
+
+ return -ENOMEM;
+}
+
+void
+fib_vrf_ht_free(struct fib_vrf_ht *ht, void *rib)
+{
+ void *node;
+
+ if (ht == NULL)
+ return;
+
+ /* Records of prefixes still present, chained outside the RIB nodes. */
+ if (rib != NULL) {
+ for (node = ht->ops->rib_walk(rib, NULL); node != NULL;
+ node = ht->ops->rib_walk(rib, node))
+ ext_free_chunks(node_ext(ht, node));
+ }
+
+ rte_hash_free(ht->pid_map);
+ rte_free(ht->def_nh);
+ rte_free(ht->polarized_bmp);
+ rte_free(ht->full_bmp);
+ rte_free(ht->buckets);
+ memset(ht, 0, sizeof(*ht));
+}
diff --git a/lib/fib/fib_vrf.h b/lib/fib/fib_vrf.h
new file mode 100644
index 0000000000..ea6a2bc09e
--- /dev/null
+++ b/lib/fib/fib_vrf.h
@@ -0,0 +1,277 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Intel Corporation
+ */
+
+#ifndef _FIB_VRF_H_
+#define _FIB_VRF_H_
+
+/**
+ * @file
+ * Hash based VRF resolution shared by the DIR24_8 (IPv4) and TRIE (IPv6)
+ * dataplanes.
+ *
+ * All VRFs share a single LPM table mapping an address to the prefix ID of the
+ * longest prefix present in any VRF (0 meaning no route at all). A hash table
+ * then maps (VRF, prefix ID) to the nexthop this VRF resolves to for that
+ * prefix, and a miss returns the per-VRF default nexthop.
+ *
+ * The dataplane keeps an internal RIB holding the prefixes of all the VRFs,
+ * with the prefix ID as node nexthop. Every node carries a fib_vrf_node_ext
+ * listing the VRFs that have a route either at the node prefix (owner records)
+ * or at a covering prefix (inherited records), together with the nexthop each
+ * of them resolves to. The invariant maintained by fib_vrf_modify() is: for
+ * every node N and every VRF V having a route covering N, the hash table holds
+ * (V, pid(N)) -> nexthop of the longest such route, and no record exists for
+ * VRFs without a covering route. The per-VRF RIBs exposed to the user are
+ * mirrors of the routes of each VRF and are maintained alongside.
+ *
+ * The address family specific bits (RIB access, LPM table update) are
+ * abstracted through fib_vrf_ops, so the same code drives both dataplanes.
+ */
+
+#include <stdalign.h>
+#include <stdbool.h>
+#include <stdint.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_hash_crc.h>
+#include <rte_prefetch.h>
+#include <rte_stdatomic.h>
+
+/** Number of records per hash bucket. */
+#define FIB_VRF_HT_ASSOC 8
+/** Largest address (in bytes) handled by the generic modify code. */
+#define FIB_VRF_IP_MAX_SZ 16
+/** Prefix IDs are stored in 4 byte LPM entries with one flag bit. */
+#define FIB_VRF_PREFIX_ID_MAX ((1U << 31) - 1)
+
+struct rte_hash;
+
+enum fib_vrf_op {
+ FIB_VRF_ADD,
+ FIB_VRF_DEL,
+};
+
+/**
+ * Address family specific operations. Addresses are passed as opaque pointers
+ * (uint32_t for IPv4, struct rte_ipv6_addr for IPv6). Node handles are opaque
+ * RIB node pointers.
+ */
+struct fib_vrf_ops {
+ /** Longest prefix match in the RIB, NULL if none. */
+ void *(*rib_lookup)(void *rib, const void *ip);
+ /** Exact match in the RIB, NULL if none. */
+ void *(*rib_lookup_exact)(void *rib, const void *ip, uint8_t depth);
+ /** Nearest valid ancestor of a node, NULL if none. */
+ void *(*rib_lookup_parent)(void *node);
+ /**
+ * Iterate over the prefixes directly covered by ip/depth (the
+ * RTE_RIB_GET_NXT_COVER semantic), last == NULL starts the walk.
+ */
+ void *(*rib_get_nxt_cover)(void *rib, const void *ip, uint8_t depth,
+ void *last);
+ /** Iterate over all the prefixes of the RIB, last == NULL starts. */
+ void *(*rib_walk)(void *rib, void *last);
+ /** Insert a prefix in the RIB, NULL on failure with rte_errno set. */
+ void *(*rib_insert)(void *rib, const void *ip, uint8_t depth);
+ /** Remove a prefix from the RIB. */
+ void (*rib_remove)(void *rib, const void *ip, uint8_t depth);
+ /** Copy the node prefix into ip (FIB_VRF_IP_MAX_SZ bytes) and depth. */
+ void (*rib_get_prefix)(const void *node, void *ip, uint8_t *depth);
+ /** Node nexthop, i.e. its prefix ID in the internal RIB. */
+ uint64_t (*rib_get_nh)(const void *node);
+ /** Set the node nexthop. */
+ void (*rib_set_nh)(void *node, uint64_t nh);
+ /** Node user extension area. */
+ void *(*rib_get_ext)(void *node);
+ /**
+ * Install (op == FIB_VRF_ADD, inserting the RIB node) or remove
+ * (op == FIB_VRF_DEL, removing the RIB node) ip/depth in the LPM table
+ * with prefix_id as its value.
+ */
+ int (*lpm_modify)(void *lpm, void *rib, const void *ip, uint8_t depth,
+ uint64_t prefix_id, enum fib_vrf_op op);
+};
+
+struct __rte_cache_aligned fib_vrf_bucket {
+ RTE_ATOMIC(uint64_t) key[FIB_VRF_HT_ASSOC];
+ uint64_t val[FIB_VRF_HT_ASSOC];
+};
+
+struct fib_vrf_ht {
+ /* Lookup fast path. */
+ struct fib_vrf_bucket *buckets;
+ uint64_t *def_nh; /**< Per-VRF default nexthop. */
+ uint32_t mask; /**< Number of buckets - 1. */
+ uint16_t num_vrfs;
+ /* Control path. */
+ uint32_t bits; /**< log2 of the number of buckets. */
+ uint64_t *full_bmp; /**< Bit set: bucket has no free slot. */
+ /** Bit set: bucket holds a record of a multi-VRF prefix. */
+ uint64_t *polarized_bmp;
+ uint64_t *conf_def_nh; /**< Configured default nexthops. */
+ struct rte_hash *pid_map; /**< prefix ID -> RIB node. */
+ const struct fib_vrf_ops *ops;
+};
+
+static inline uint64_t
+fib_vrf_key(uint16_t vrf, uint32_t prefix_id)
+{
+ return (uint64_t)vrf << 32 | prefix_id;
+}
+
+static inline uint32_t
+fib_vrf_bucket_idx(const struct fib_vrf_ht *ht, uint64_t key)
+{
+ return rte_hash_crc_8byte(key, 0) & ht->mask;
+}
+
+/*
+ * A record is published by writing its value before its key. The acquire
+ * load (a plain load on x86) makes sure that the value read after a key match
+ * is not an older one, as a plain load reordered before the key load could be
+ * on weakly ordered architectures.
+ */
+static inline uint64_t
+fib_vrf_bucket_search(const struct fib_vrf_ht *ht,
+ const struct fib_vrf_bucket *b, uint64_t key, uint16_t vrf)
+{
+ unsigned int j;
+
+ for (j = 0; j < FIB_VRF_HT_ASSOC; j++) {
+ if (rte_atomic_load_explicit(&b->key[j],
+ rte_memory_order_acquire) == key)
+ return b->val[j];
+ }
+ return ht->def_nh[vrf];
+}
+
+/**
+ * Resolve the nexthop of one VRF for the prefix ID returned by the LPM table.
+ */
+static inline uint64_t
+fib_vrf_ht_lookup(const struct fib_vrf_ht *ht, uint16_t vrf,
+ uint32_t prefix_id)
+{
+ uint64_t key;
+
+ if (unlikely(prefix_id == 0))
+ return ht->def_nh[vrf];
+ key = fib_vrf_key(vrf, prefix_id);
+ return fib_vrf_bucket_search(ht, &ht->buckets[fib_vrf_bucket_idx(ht, key)],
+ key, vrf);
+}
+
+#define FIB_VRF_RESOLVE_BURST 32
+
+/* Resolve up to FIB_VRF_RESOLVE_BURST prefix IDs, prefetching their buckets. */
+static inline void
+fib_vrf_ht_resolve_burst(const struct fib_vrf_ht *ht, const uint16_t *vrfs,
+ uint64_t *next_hops, unsigned int n)
+{
+ uint64_t keys[FIB_VRF_RESOLVE_BURST];
+ uint32_t bidx[FIB_VRF_RESOLVE_BURST];
+ unsigned int j;
+
+ for (j = 0; j < n; j++) {
+ keys[j] = fib_vrf_key(vrfs[j], (uint32_t)next_hops[j]);
+ bidx[j] = fib_vrf_bucket_idx(ht, keys[j]);
+ rte_prefetch0(&ht->buckets[bidx[j]]);
+ }
+ for (j = 0; j < n; j++) {
+ if (unlikely((uint32_t)next_hops[j] == 0)) {
+ next_hops[j] = ht->def_nh[vrfs[j]];
+ continue;
+ }
+ next_hops[j] = fib_vrf_bucket_search(ht, &ht->buckets[bidx[j]],
+ keys[j], vrfs[j]);
+ }
+}
+
+/**
+ * Turn the prefix IDs returned by the LPM table into nexthops in place.
+ *
+ * @param ht
+ * Hash table.
+ * @param vrfs
+ * Per address VRF, NULL means VRF 0 for every address.
+ * @param next_hops
+ * Prefix IDs on entry (0 = no route), nexthops on return.
+ * @param n
+ * Number of addresses.
+ */
+static inline void
+fib_vrf_ht_resolve(const struct fib_vrf_ht *ht, const uint16_t *vrfs,
+ uint64_t *next_hops, unsigned int n)
+{
+ unsigned int i;
+
+ if (unlikely(vrfs == NULL)) {
+ for (i = 0; i < n; i++)
+ next_hops[i] = fib_vrf_ht_lookup(ht, 0,
+ (uint32_t)next_hops[i]);
+ return;
+ }
+
+ for (i = 0; i + FIB_VRF_RESOLVE_BURST <= n; i += FIB_VRF_RESOLVE_BURST)
+ fib_vrf_ht_resolve_burst(ht, vrfs + i, next_hops + i,
+ FIB_VRF_RESOLVE_BURST);
+ if (i < n)
+ fib_vrf_ht_resolve_burst(ht, vrfs + i, next_hops + i, n - i);
+}
+
+#ifdef CC_AVX512_SUPPORT
+/** AVX512 flavour of fib_vrf_ht_resolve(). */
+void
+fib_vrf_ht_resolve_avx512(const struct fib_vrf_ht *ht, const uint16_t *vrfs,
+ uint64_t *next_hops, unsigned int n);
+#endif
+
+/** Size of the node extension of the internal RIB. */
+size_t
+fib_vrf_rib_ext_sz(void);
+
+/**
+ * Initialize a hash table. The LPM table must return prefix IDs allocated by
+ * fib_vrf_modify() and 0 for addresses without any route.
+ *
+ * @param num_vrfs
+ * Number of VRFs, at least 1.
+ * @param default_nh
+ * Default nexthop of every VRF, unless vrf_default_nh is set.
+ * @param vrf_default_nh
+ * Optional per-VRF default nexthops (num_vrfs entries).
+ * @param max_routes
+ * Maximum number of distinct prefixes (across all VRFs).
+ */
+int
+fib_vrf_ht_init(struct fib_vrf_ht *ht, const char *name, int socket_id,
+ const struct fib_vrf_ops *ops, uint16_t num_vrfs, uint64_t default_nh,
+ const uint64_t *vrf_default_nh, uint32_t max_routes);
+
+/**
+ * Release the hash table resources and the node extensions of the internal
+ * RIB, which must still be intact.
+ */
+void
+fib_vrf_ht_free(struct fib_vrf_ht *ht, void *rib);
+
+/**
+ * Add or delete a route of one VRF. ip must be masked to depth already.
+ *
+ * @param ht
+ * Hash table.
+ * @param lpm
+ * LPM table shared by all the VRFs.
+ * @param rib
+ * Internal RIB holding the prefixes of all the VRFs.
+ * @param vrf_rib
+ * RIB of the VRF, mirroring its routes.
+ */
+int
+fib_vrf_modify(struct fib_vrf_ht *ht, void *lpm, void *rib, void *vrf_rib,
+ uint16_t vrf, const void *ip, uint8_t depth, uint64_t next_hop,
+ enum fib_vrf_op op);
+
+#endif /* _FIB_VRF_H_ */
diff --git a/lib/fib/fib_vrf_avx512.c b/lib/fib/fib_vrf_avx512.c
new file mode 100644
index 0000000000..4a4d3b604a
--- /dev/null
+++ b/lib/fib/fib_vrf_avx512.c
@@ -0,0 +1,142 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2026 Intel Corporation
+ */
+
+#include <stdalign.h>
+
+#include <rte_bitops.h>
+#include <rte_vect.h>
+
+#include "fib_vrf.h"
+
+/* log2(sizeof(struct fib_vrf_bucket)) */
+#define BUCKET_SHIFT 7
+
+/* Chunks of 8 lookups hashed, and their buckets prefetched, ahead of search. */
+#define RESOLVE_AHEAD 3
+/* Hashed chunks kept for the search, a power of 2 above RESOLVE_AHEAD. */
+#define RESOLVE_RING 4
+
+/*
+ * Barrett reduction of the bit-reflected CRC32C of a 64-bit key: x^96 / P
+ * without its x^64 term, and P placing the remainder in bits 64-95.
+ */
+#define CRC32C_MU 0xa434f61c6f5389f8ULL
+#define CRC32C_POLY 0x105ec76f0ULL
+
+/* Searched for the prefix ID 0 (no route at all), never matches a key. */
+static const struct fib_vrf_bucket no_route_bucket = {
+ .key = { UINT64_MAX, UINT64_MAX, UINT64_MAX, UINT64_MAX,
+ UINT64_MAX, UINT64_MAX, UINT64_MAX, UINT64_MAX },
+};
+
+/* rte_hash_crc_8byte(key, 0) of 8 keys, in the low 32 bits of each lane. */
+static inline __m512i
+crc32c_x8(__m512i keys)
+{
+ const __m512i mu = _mm512_set1_epi64(CRC32C_MU);
+ const __m512i poly = _mm512_set1_epi64(CRC32C_POLY);
+ __m512i even, odd, quot;
+
+ /* The x^64 term of the constant contributes the key itself. */
+ even = _mm512_clmulepi64_epi128(keys, mu, 0x00);
+ odd = _mm512_clmulepi64_epi128(keys, mu, 0x01);
+ quot = _mm512_xor_si512(keys,
+ _mm512_slli_epi64(_mm512_unpacklo_epi64(even, odd), 1));
+
+ even = _mm512_clmulepi64_epi128(quot, poly, 0x00);
+ odd = _mm512_clmulepi64_epi128(quot, poly, 0x01);
+ return _mm512_unpackhi_epi64(even, odd);
+}
+
+/* Prefetch from registers: reloading the vector just stored stalls. */
+static inline void
+prefetch_x8(__m512i ptrs)
+{
+ __m256i lo = _mm512_castsi512_si256(ptrs);
+ __m256i hi = _mm512_extracti64x4_epi64(ptrs, 1);
+ __m128i p[4] = {
+ _mm256_castsi256_si128(lo), _mm256_extracti128_si256(lo, 1),
+ _mm256_castsi256_si128(hi), _mm256_extracti128_si256(hi, 1),
+ };
+ unsigned int i;
+
+ for (i = 0; i < RTE_DIM(p); i++) {
+ rte_prefetch0((const void *)_mm_cvtsi128_si64(p[i]));
+ rte_prefetch0((const void *)_mm_extract_epi64(p[i], 1));
+ }
+}
+
+/* Hash 8 lookups: store their keys and buckets, and prefetch the buckets. */
+static inline void
+hash_x8(const struct fib_vrf_ht *ht, const uint16_t *vrfs,
+ const uint64_t *next_hops, uint64_t *keys,
+ const struct fib_vrf_bucket **bkt)
+{
+ __m512i nh = _mm512_loadu_si512(next_hops);
+ __m512i vrf = (vrfs != NULL) ? _mm512_cvtepu16_epi64(
+ _mm_loadu_si128((const __m128i *)vrfs)) : _mm512_setzero_si512();
+ __m512i key = _mm512_or_si512(_mm512_slli_epi64(vrf, 32), nh);
+ __m512i ptr = _mm512_add_epi64(_mm512_set1_epi64((uintptr_t)ht->buckets),
+ _mm512_and_si512(_mm512_slli_epi64(crc32c_x8(key), BUCKET_SHIFT),
+ _mm512_set1_epi64((uint64_t)ht->mask << BUCKET_SHIFT)));
+
+ ptr = _mm512_mask_mov_epi64(ptr, _mm512_testn_epi64_mask(nh, nh),
+ _mm512_set1_epi64((uintptr_t)&no_route_bucket));
+ _mm512_store_si512(keys, key);
+ _mm512_store_si512(bkt, ptr);
+ prefetch_x8(ptr);
+}
+
+/* Search 8 hashed lookups in their buckets. */
+static inline void
+search_x8(const struct fib_vrf_ht *ht, const uint16_t *vrfs,
+ const uint64_t *keys, const struct fib_vrf_bucket * const *bkt,
+ uint64_t *next_hops)
+{
+ unsigned int i;
+
+ for (i = 0; i < 8; i++) {
+ const struct fib_vrf_bucket *b = bkt[i];
+ /* The keys are read as plain memory, like the LPM table entries. */
+ __mmask8 match = _mm512_cmpeq_epi64_mask(
+ _mm512_loadu_si512(RTE_CAST_PTR(const void *, b->key)),
+ _mm512_set1_epi64(keys[i]));
+
+ if (unlikely(match == 0))
+ next_hops[i] = ht->def_nh[(vrfs != NULL) ? vrfs[i] : 0];
+ else
+ next_hops[i] = b->val[rte_ctz32(match)];
+ }
+}
+
+void
+fib_vrf_ht_resolve_avx512(const struct fib_vrf_ht *ht, const uint16_t *vrfs,
+ uint64_t *next_hops, unsigned int n)
+{
+ alignas(64) uint64_t keys[RESOLVE_RING][8];
+ alignas(64) const struct fib_vrf_bucket *bkt[RESOLVE_RING][8];
+ unsigned int nb = RTE_ALIGN_FLOOR(n, 8);
+ unsigned int i, s;
+
+ RTE_BUILD_BUG_ON(sizeof(struct fib_vrf_bucket) != RTE_BIT32(BUCKET_SHIFT));
+ RTE_BUILD_BUG_ON(RESOLVE_AHEAD >= RESOLVE_RING);
+
+ for (i = 0; i < nb + RESOLVE_AHEAD * 8; i += 8) {
+ if (i < nb) {
+ s = (i / 8) % RESOLVE_RING;
+ hash_x8(ht, (vrfs != NULL) ? vrfs + i : NULL, next_hops + i,
+ keys[s], bkt[s]);
+ }
+ if (i >= RESOLVE_AHEAD * 8) {
+ unsigned int c = i - RESOLVE_AHEAD * 8;
+
+ s = (c / 8) % RESOLVE_RING;
+ search_x8(ht, (vrfs != NULL) ? vrfs + c : NULL, keys[s],
+ bkt[s], next_hops + c);
+ }
+ }
+ for (i = nb; i < n; i++)
+ next_hops[i] = fib_vrf_ht_lookup(ht, (vrfs != NULL) ? vrfs[i] : 0,
+ (uint32_t)next_hops[i]);
+}
diff --git a/lib/fib/meson.build b/lib/fib/meson.build
index f24f679039..1b8ab4859a 100644
--- a/lib/fib/meson.build
+++ b/lib/fib/meson.build
@@ -2,14 +2,32 @@
# Copyright(c) 2018 Vladimir Medvedkin <medvedkinv@gmail.com>
# Copyright(c) 2019 Intel Corporation
-sources = files('rte_fib.c', 'rte_fib6.c', 'dir24_8.c', 'dir8_16_8.c', 'trie.c')
+sources = files(
+ 'dir24_8.c',
+ 'dir24_8_vrf.c',
+ 'dir8_16_8.c',
+ 'fib_vrf.c',
+ 'rte_fib.c',
+ 'rte_fib6.c',
+ 'trie.c',
+)
headers = files('rte_fib.h', 'rte_fib6.h')
deps += ['rib']
deps += ['rcu']
deps += ['net']
+deps += ['hash']
if dpdk_conf.has('RTE_ARCH_X86_64')
- sources_avx512 += files('dir24_8_avx512.c', 'dir8_16_8_avx512.c', 'trie_avx512.c')
+ sources_avx512 += files(
+ 'dir24_8_avx512.c',
+ 'dir8_16_8_avx512.c',
+ 'fib_vrf_avx512.c',
+ 'trie_avx512.c',
+ )
+ # the VRF lookups hash the keys with VPCLMULQDQ
+ if cc.has_argument('-mvpclmulqdq')
+ cflags_avx512 += ['-mvpclmulqdq']
+ endif
elif dpdk_conf.has('RTE_ARCH_RISCV')
sources += files('dir24_8_rvv.c')
endif
diff --git a/lib/fib/rte_fib.c b/lib/fib/rte_fib.c
index 795031edcd..ac7f1b133b 100644
--- a/lib/fib/rte_fib.c
+++ b/lib/fib/rte_fib.c
@@ -4,6 +4,7 @@
*/
#include <stdint.h>
+#include <stdbool.h>
#include <string.h>
#include <sys/queue.h>
@@ -16,8 +17,11 @@
#include <rte_rib.h>
#include <rte_fib.h>
+#include <rte_vect.h>
+#include <rte_cpuflags.h>
#include "dir24_8.h"
+#include "dir24_8_vrf.h"
#include "dir8_16_8.h"
#include "fib_log.h"
@@ -50,6 +54,13 @@ struct rte_fib {
uint16_t max_vrfs; /**< Number of VRFs (0/1 = single). */
};
+/* Multi-VRF DIR24_8 shares one RIB and resolves VRFs through a hash table. */
+static inline bool
+fib_is_dir24_8_vrf(const struct rte_fib *fib)
+{
+ return fib->type == RTE_FIB_DIR24_8 && fib->max_vrfs > 1;
+}
+
static void
dummy_lookup(void *fib_p, const uint16_t *vrf_ids __rte_unused,
const uint32_t *ips, uint64_t *next_hops, const unsigned int n)
@@ -107,12 +118,23 @@ init_dataplane(struct rte_fib *fib, __rte_unused int socket_id,
fib->modify = dummy_modify;
return 0;
case RTE_FIB_DIR24_8:
- fib->dp = dir24_8_create(dp_name, socket_id, conf);
- if (fib->dp == NULL)
- return -rte_errno;
- fib->lookup = dir24_8_get_lookup_fn(fib->dp,
- RTE_FIB_LOOKUP_DEFAULT, !!(fib->flags & RTE_FIB_F_LOOKUP_NETWORK_ORDER));
- fib->modify = dir24_8_modify;
+ if (fib_is_dir24_8_vrf(fib)) {
+ fib->dp = dir24_8_vrf_create(dp_name, socket_id, conf);
+ if (fib->dp == NULL)
+ return -rte_errno;
+ fib->lookup = dir24_8_vrf_get_lookup_fn(fib->dp,
+ RTE_FIB_LOOKUP_DEFAULT,
+ !!(fib->flags & RTE_FIB_F_LOOKUP_NETWORK_ORDER));
+ fib->modify = dir24_8_vrf_modify;
+ } else {
+ fib->dp = dir24_8_create(dp_name, socket_id, conf);
+ if (fib->dp == NULL)
+ return -rte_errno;
+ fib->lookup = dir24_8_get_lookup_fn(fib->dp,
+ RTE_FIB_LOOKUP_DEFAULT,
+ !!(fib->flags & RTE_FIB_F_LOOKUP_NETWORK_ORDER));
+ fib->modify = dir24_8_modify;
+ }
return 0;
case RTE_FIB_DIR8_16_8:
fib->dp = dir8_16_8_create(dp_name, socket_id, conf);
@@ -165,19 +187,14 @@ rte_fib_lookup_bulk(struct rte_fib *fib, uint32_t *ips,
static void
free_dataplane(struct rte_fib *fib);
-/*
- * Allocate the RIB array and create the FIB's RIBs: ribs[0] is the main RIB
- * (named after the FIB), plus one independent RIB per extra VRF. A single-VRF
- * FIB still keeps a one-element array so ribs[0] is always the main RIB.
- */
static int
fib_create_ribs(struct rte_fib *fib, const char *name,
const struct rte_rib_conf *rib_conf, int socket_id)
{
- uint16_t v, num_vrfs = RTE_MAX(fib->max_vrfs, 1);
+ uint16_t v, num_ribs = RTE_MAX(fib->max_vrfs, 1);
fib->ribs = rte_zmalloc_socket(NULL,
- (size_t)num_vrfs * sizeof(struct rte_rib *),
+ (size_t)num_ribs * sizeof(struct rte_rib *),
RTE_CACHE_LINE_SIZE, socket_id);
if (fib->ribs == NULL)
return -ENOMEM;
@@ -186,12 +203,7 @@ fib_create_ribs(struct rte_fib *fib, const char *name,
if (fib->ribs[0] == NULL)
return -rte_errno;
- for (v = 1; v < num_vrfs; v++) {
- /*
- * Derive a short unique RIB name from the FIB pointer: the
- * user name plus a "_vN" suffix could overflow the RIB name
- * length limit.
- */
+ for (v = 1; v < num_ribs; v++) {
char rib_name[RTE_FIB_NAMESIZE];
snprintf(rib_name, sizeof(rib_name), "%p-v%u", (void *)fib, v);
@@ -199,18 +211,18 @@ fib_create_ribs(struct rte_fib *fib, const char *name,
if (fib->ribs[v] == NULL)
return -rte_errno;
}
+
return 0;
}
-/* Free every RIB and the RIB array. */
static void
fib_free_ribs(struct rte_fib *fib)
{
- uint16_t v, num_vrfs = RTE_MAX(fib->max_vrfs, 1);
+ uint16_t v, num_ribs = RTE_MAX(fib->max_vrfs, 1);
if (fib->ribs == NULL)
return;
- for (v = 0; v < num_vrfs; v++)
+ for (v = 0; v < num_ribs; v++)
rte_rib_free(fib->ribs[v]);
rte_free(fib->ribs);
fib->ribs = NULL;
@@ -233,7 +245,8 @@ rte_fib_create(const char *name, int socket_id, struct rte_fib_conf *conf)
(conf->flags & ~RTE_FIB_ALLOWED_FLAGS) ||
(conf->type > RTE_FIB_DIR8_16_8) ||
(conf->max_vrfs >= 2 &&
- conf->type != RTE_FIB_DIR8_16_8)) {
+ conf->type != RTE_FIB_DIR8_16_8 &&
+ conf->type != RTE_FIB_DIR24_8)) {
rte_errno = EINVAL;
return NULL;
}
@@ -243,7 +256,6 @@ rte_fib_create(const char *name, int socket_id, struct rte_fib_conf *conf)
snprintf(mem_name, sizeof(mem_name), "FIB_%s", name);
- /* allocate tailq entry */
te = rte_zmalloc("FIB_TAILQ_ENTRY", sizeof(*te), 0);
if (te == NULL) {
FIB_LOG(ERR,
@@ -252,7 +264,6 @@ rte_fib_create(const char *name, int socket_id, struct rte_fib_conf *conf)
return NULL;
}
- /* Allocate memory to store the FIB data structures. */
fib = rte_zmalloc_socket(mem_name,
sizeof(struct rte_fib), RTE_CACHE_LINE_SIZE, socket_id);
if (fib == NULL) {
@@ -287,7 +298,6 @@ rte_fib_create(const char *name, int socket_id, struct rte_fib_conf *conf)
rte_mcfg_tailq_write_lock();
- /* guarantee there's no existing */
TAILQ_FOREACH(tmp_te, fib_list, next) {
if (strncmp(name, ((struct rte_fib *)tmp_te->data)->name,
RTE_FIB_NAMESIZE) == 0) {
@@ -305,8 +315,9 @@ rte_fib_create(const char *name, int socket_id, struct rte_fib_conf *conf)
return fib;
free_ribs:
- fib_free_ribs(fib);
+ /* The dataplane may hold references to RIB nodes, free it first. */
free_dataplane(fib);
+ fib_free_ribs(fib);
free_fib:
rte_free(fib);
free_te:
@@ -348,7 +359,10 @@ free_dataplane(struct rte_fib *fib)
case RTE_FIB_DUMMY:
return;
case RTE_FIB_DIR24_8:
- dir24_8_free(fib->dp);
+ if (fib_is_dir24_8_vrf(fib))
+ dir24_8_vrf_free(fib->dp);
+ else
+ dir24_8_free(fib->dp);
return;
case RTE_FIB_DIR8_16_8:
dir8_16_8_free(fib->dp);
@@ -372,7 +386,6 @@ rte_fib_free(struct rte_fib *fib)
rte_mcfg_tailq_write_lock();
- /* find our tailq entry */
TAILQ_FOREACH(te, fib_list, next) {
if (te->data == (void *)fib)
break;
@@ -411,8 +424,12 @@ rte_fib_select_lookup(struct rte_fib *fib,
switch (fib->type) {
case RTE_FIB_DIR24_8:
- fn = dir24_8_get_lookup_fn(fib->dp, type,
- !!(fib->flags & RTE_FIB_F_LOOKUP_NETWORK_ORDER));
+ if (fib_is_dir24_8_vrf(fib))
+ fn = dir24_8_vrf_get_lookup_fn(fib->dp, type,
+ !!(fib->flags & RTE_FIB_F_LOOKUP_NETWORK_ORDER));
+ else
+ fn = dir24_8_get_lookup_fn(fib->dp, type,
+ !!(fib->flags & RTE_FIB_F_LOOKUP_NETWORK_ORDER));
if (fn == NULL)
return -EINVAL;
fib->lookup = fn;
@@ -438,6 +455,8 @@ rte_fib_rcu_qsbr_add(struct rte_fib *fib, struct rte_fib_rcu_config *cfg)
switch (fib->type) {
case RTE_FIB_DIR24_8:
+ if (fib_is_dir24_8_vrf(fib))
+ return -ENOTSUP;
return dir24_8_rcu_qsbr_add(fib->dp, cfg, fib->name);
case RTE_FIB_DIR8_16_8:
return dir8_16_8_rcu_qsbr_add(fib->dp, cfg, fib->name);
diff --git a/lib/fib/rte_fib.h b/lib/fib/rte_fib.h
index 948bef7600..0094afc269 100644
--- a/lib/fib/rte_fib.h
+++ b/lib/fib/rte_fib.h
@@ -94,8 +94,12 @@ enum rte_fib_lookup_type {
/**<
* Scalar implementation for the DIR8_16_8 algorithm
*/
- RTE_FIB_LOOKUP_DIR8_16_8_VECTOR_AVX512
+ RTE_FIB_LOOKUP_DIR8_16_8_VECTOR_AVX512,
/**< Vector (AVX512) lookup for the DIR8_16_8 algorithm */
+ RTE_FIB_LOOKUP_DIR24_8_VRF_SCALAR,
+ /**< Scalar lookup for the VRF-capable DIR24_8 algorithm */
+ RTE_FIB_LOOKUP_DIR24_8_VRF_VECTOR_AVX512
+ /**< Vector (AVX512) lookup for the VRF-capable DIR24_8 algorithm */
};
/** If set, fib lookup is expecting IPv4 address in network byte order */
@@ -124,8 +128,15 @@ struct rte_fib_conf {
};
unsigned int flags; /**< Optional feature flags from RTE_FIB_F_* */
/**
- * Number of VRFs to support. 0 or 1 means a single (default) VRF.
- * Values >= 2 require RTE_FIB_DIR8_16_8.
+ * Number of VRFs to support. 0 or 1 means a single VRF.
+ * Values >= 2 require RTE_FIB_DIR24_8 or RTE_FIB_DIR8_16_8.
+ *
+ * RTE_FIB_DIR8_16_8 keeps a root table per VRF.
+ *
+ * RTE_FIB_DIR24_8 shares a single DIR24_8 table across VRFs and
+ * resolves the per-VRF nexthops through a hash table: max_routes is
+ * then the number of distinct prefixes across all VRFs and
+ * dir24_8.nh_sz must be RTE_FIB_DIR24_8_8B.
*/
uint16_t max_vrfs;
/**
@@ -239,6 +250,7 @@ rte_fib_delete(struct rte_fib *fib, uint32_t ip, uint8_t depth);
* This is an array of eight byte values.
* If the lookup for the given IP failed, then corresponding element would
* contain default nexthop value configured for a FIB.
+ * For a multi-VRF FIB, all addresses are looked up in VRF 0.
* @param n
* Number of elements in ips (and next_hops) array to lookup.
* @return
--
2.43.0
^ permalink raw reply related [flat|nested] 45+ messages in thread
* Re: [RFC PATCH v2 3/7] app/test-fib: fix size of the tables for large counts
2026-10-14 18:39 ` [RFC PATCH v2 3/7] app/test-fib: fix size of the tables for large counts Vladimir Medvedkin
@ 2026-10-14 22:52 ` Stephen Hemminger
2026-10-15 14:21 ` Medvedkin, Vladimir
0 siblings, 1 reply; 45+ messages in thread
From: Stephen Hemminger @ 2026-10-14 22:52 UTC (permalink / raw)
To: Vladimir Medvedkin; +Cc: dev, rjarry, mb, maxime, konstantin.ananyev, stable
On Wed, 14 Oct 2026 18:39:17 +0000
Vladimir Medvedkin <vladimir.medvedkin@intel.com> wrote:
> diff --git a/app/test-fib/main.c b/app/test-fib/main.c
> index d85d94fdd1..0bc8dd2aa4 100644
> --- a/app/test-fib/main.c
> +++ b/app/test-fib/main.c
> @@ -1260,12 +1260,12 @@ main(int argc, char **argv)
> BURST_SZ);
>
> /* Alloc routes table*/
> - config.rt = rte_malloc(NULL, rt_ent_sz * config.nb_routes, 0);
> + config.rt = rte_malloc(NULL, (size_t)rt_ent_sz * config.nb_routes, 0);
> if (config.rt == NULL)
> rte_exit(-ENOMEM, "Can not alloc rt\n");
>
> /* Alloc table with ip's for lookup*/
> - config.lookup_tbl = rte_malloc(NULL, lookup_ent_sz *
> + config.lookup_tbl = rte_malloc(NULL, (size_t)lookup_ent_sz *
> config.nb_lookup_ips, 0);
if you are going to change these, then fix a couple related things.
- comments have bad spacing in there
- use rte_calloc() for arrays
- not sure why rte_malloc is used at all, this is test code. Doesn't need to be hugepages.
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH v2 0/7] VRF support in FIB library
2026-10-14 18:39 ` [RFC PATCH v2 0/7] " Vladimir Medvedkin
` (6 preceding siblings ...)
2026-10-14 18:39 ` [RFC PATCH v2 7/7] fib: add hash based VRF support for DIR24-8 Vladimir Medvedkin
@ 2026-10-14 22:55 ` Stephen Hemminger
2026-10-15 14:17 ` Medvedkin, Vladimir
7 siblings, 1 reply; 45+ messages in thread
From: Stephen Hemminger @ 2026-10-14 22:55 UTC (permalink / raw)
To: Vladimir Medvedkin; +Cc: dev, rjarry, mb, maxime, konstantin.ananyev
On Wed, 14 Oct 2026 18:39:14 +0000
Vladimir Medvedkin <vladimir.medvedkin@intel.com> wrote:
> This series adds multi-VRF support to the IPv4 FIB, allowing a single FIB
> instance to host multiple isolated routing tables.
>
> Today a FIB instance is a single routing table, so an application serving
> several VRFs needs one FIB per VRF. In a burst oriented datapath packets of
> one burst can belong to different VRFs, and the application either does
> per-packet lookups in different FIBs or regroups the burst by VRF before
> the lookup. Both are expensive. With this series all the VRFs live in one
> FIB, the bulk lookup takes a VRF ID per address, and the existing API
> operates on VRF 0.
>
> v1 replicated the 16M entry tbl24 per VRF. That kept the lookup on par with
> the single VRF case, but memory grew linearly with the number of VRFs,
> which was the main concern raised on v1. v2 resolves the issue by adding
> of two dataplane algorithms with different memory/lookup trade-offs.
>
> DIR8-16-8 is a new three level variant of DIR-24-8 with 8, 16 and 8 bit
> strides. Only the 256 entry root table (2 KB) is replicated per VRF; the
> tbl16 and tbl8 groups come from pools shared by all the VRFs, so memory
> follows the routes actually installed rather than the number of VRFs. The
> cost is an extra memory access for prefixes longer than /8.
>
> DIR-24-8 gets multi-VRF support by resolving the VRF after the longest
> prefix match, along the lines Konstantin suggested on v1. Instead of a
> vector of next hops per prefix, a single tbl24_8 shared by all the VRFs
> returns an internal prefix ID, and a hash table keyed by {VRF, prefix ID}
> returns the next hop, a miss meaning the VRF default next hop. The aim is
> to resolve the VRF with a single memory access, so buckets hold 8 records
> and there is no probing beyond the bucket.
The problem with existing FIB lookups is they are space intensive.
They tradeoff memory for cache misses. Every level of indirection adds
another cache miss.
Not sure they work well for VRF case at all. Normal LPM is more compact.
For normal mortal size machines having multiple VRF's is going to be huge.
How big are you envisioning this?
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH v2 0/7] VRF support in FIB library
2026-10-14 22:55 ` [RFC PATCH v2 0/7] VRF support in FIB library Stephen Hemminger
@ 2026-10-15 14:17 ` Medvedkin, Vladimir
0 siblings, 0 replies; 45+ messages in thread
From: Medvedkin, Vladimir @ 2026-10-15 14:17 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: dev, rjarry, mb, maxime, konstantin.ananyev
Hi Stephen,
On 10/14/2026 11:55 PM, Stephen Hemminger wrote:
> On Wed, 14 Oct 2026 18:39:14 +0000
> Vladimir Medvedkin <vladimir.medvedkin@intel.com> wrote:
>
>> This series adds multi-VRF support to the IPv4 FIB, allowing a single FIB
>> instance to host multiple isolated routing tables.
>>
>> Today a FIB instance is a single routing table, so an application serving
>> several VRFs needs one FIB per VRF. In a burst oriented datapath packets of
>> one burst can belong to different VRFs, and the application either does
>> per-packet lookups in different FIBs or regroups the burst by VRF before
>> the lookup. Both are expensive. With this series all the VRFs live in one
>> FIB, the bulk lookup takes a VRF ID per address, and the existing API
>> operates on VRF 0.
>>
>> v1 replicated the 16M entry tbl24 per VRF. That kept the lookup on par with
>> the single VRF case, but memory grew linearly with the number of VRFs,
>> which was the main concern raised on v1. v2 resolves the issue by adding
>> of two dataplane algorithms with different memory/lookup trade-offs.
>>
>> DIR8-16-8 is a new three level variant of DIR-24-8 with 8, 16 and 8 bit
>> strides. Only the 256 entry root table (2 KB) is replicated per VRF; the
>> tbl16 and tbl8 groups come from pools shared by all the VRFs, so memory
>> follows the routes actually installed rather than the number of VRFs. The
>> cost is an extra memory access for prefixes longer than /8.
>>
>> DIR-24-8 gets multi-VRF support by resolving the VRF after the longest
>> prefix match, along the lines Konstantin suggested on v1. Instead of a
>> vector of next hops per prefix, a single tbl24_8 shared by all the VRFs
>> returns an internal prefix ID, and a hash table keyed by {VRF, prefix ID}
>> returns the next hop, a miss meaning the VRF default next hop. The aim is
>> to resolve the VRF with a single memory access, so buckets hold 8 records
>> and there is no probing beyond the bucket.
>
> The problem with existing FIB lookups is they are space intensive.
comparing to what?
> They tradeoff memory for cache misses. Every level of indirection adds
> another cache miss.
This is not always a case. Also that's why we are using prefetching.
>
> Not sure they work well for VRF case at all.
I suspect this objection is applied for v1? v2 should resolve the issue
> Normal LPM is more compact.
more compact compared to what? Under "Normal LPM" I assume rte_lpm
library, if so, then comparing to existing FIB, which uses DIR24_8 with
4 byte nexthops (apples to apples), the memory footprint is exactly the
same.
> For normal mortal size machines having multiple VRF's is going to be huge.
> How big are you envisioning this?
From my experience when I was working as a network engineer in network
operators, an average PE router could have 2 big VRFs (about 800k
prefixes for v4) and thousands small VRFs with about hundreds prefixes each.
With this input an average size of a dataplane struct with:
DIR8_16_8 could be around 2Kb*1000 (VRFs) + 2*256*64k*8b(TBL16 for 2
VRFs with full view) + 1000*4*64k*8b(small VRFs needs around 4 TBL16 - 3
for RFC1918 + 1 customer public net) + 1000*100(prefixes more specific
than /24, off the top of my head)*256*8b. This corresponds to TBL0: 2Mb,
TBL16: ~2.5Gb, TBL8:~200Mb, so under 3Gb of RAM.
DIR24_8: single DIR24_8 with joint prefixes for all VRFs and 4b entries
- 64mb for TBL24 + a few Mb for TBL8 (since all prefixes are joint) +
hash table of size from 8Mb to 1Gb.
This sounds like mortal machine RAM usage to me.
--
Regards,
Vladimir
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [RFC PATCH v2 3/7] app/test-fib: fix size of the tables for large counts
2026-10-14 22:52 ` Stephen Hemminger
@ 2026-10-15 14:21 ` Medvedkin, Vladimir
0 siblings, 0 replies; 45+ messages in thread
From: Medvedkin, Vladimir @ 2026-10-15 14:21 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: dev, rjarry, mb, maxime, konstantin.ananyev, stable
Hi Stephen,
On 10/14/2026 11:52 PM, Stephen Hemminger wrote:
> On Wed, 14 Oct 2026 18:39:17 +0000
> Vladimir Medvedkin <vladimir.medvedkin@intel.com> wrote:
>
>> diff --git a/app/test-fib/main.c b/app/test-fib/main.c
>> index d85d94fdd1..0bc8dd2aa4 100644
>> --- a/app/test-fib/main.c
>> +++ b/app/test-fib/main.c
>> @@ -1260,12 +1260,12 @@ main(int argc, char **argv)
>> BURST_SZ);
>>
>> /* Alloc routes table*/
>> - config.rt = rte_malloc(NULL, rt_ent_sz * config.nb_routes, 0);
>> + config.rt = rte_malloc(NULL, (size_t)rt_ent_sz * config.nb_routes, 0);
>> if (config.rt == NULL)
>> rte_exit(-ENOMEM, "Can not alloc rt\n");
>>
>> /* Alloc table with ip's for lookup*/
>> - config.lookup_tbl = rte_malloc(NULL, lookup_ent_sz *
>> + config.lookup_tbl = rte_malloc(NULL, (size_t)lookup_ent_sz *
>> config.nb_lookup_ips, 0);
> if you are going to change these, then fix a couple related things.
> - comments have bad spacing in there
> - use rte_calloc() for arrays
will change
> - not sure why rte_malloc is used at all, this is test code. Doesn't need to be hugepages.
disagree. This array contains hundred millions of IP addresses and
occupies hundreds Mb. Also this array is used in performance benchmark.
I don't want to hit DTLB misses in performance tests.
--
Regards,
Vladimir
^ permalink raw reply [flat|nested] 45+ messages in thread
end of thread, other threads:[~2026-10-15 14:21 UTC | newest]
Thread overview: 45+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-03-22 15:42 [RFC PATCH 0/4] VRF support in FIB library Vladimir Medvedkin
2026-03-22 15:42 ` [RFC PATCH 1/4] fib: add multi-VRF support Vladimir Medvedkin
2026-03-23 15:48 ` Konstantin Ananyev
2026-03-23 19:06 ` Medvedkin, Vladimir
2026-03-23 22:22 ` Konstantin Ananyev
2026-03-25 14:09 ` Medvedkin, Vladimir
2026-03-26 10:13 ` Konstantin Ananyev
2026-03-27 18:32 ` Medvedkin, Vladimir
2026-03-22 15:42 ` [RFC PATCH 2/4] fib: add VRF functional and unit tests Vladimir Medvedkin
2026-03-22 16:40 ` Stephen Hemminger
2026-03-22 16:41 ` Stephen Hemminger
2026-03-22 15:42 ` [RFC PATCH 3/4] fib6: add multi-VRF support Vladimir Medvedkin
2026-03-22 15:42 ` [RFC PATCH 4/4] fib6: add VRF functional and unit tests Vladimir Medvedkin
2026-03-22 16:45 ` Stephen Hemminger
2026-03-22 16:43 ` [RFC PATCH 0/4] VRF support in FIB library Stephen Hemminger
2026-03-23 9:01 ` Morten Brørup
2026-03-23 11:32 ` Medvedkin, Vladimir
2026-03-23 11:16 ` Medvedkin, Vladimir
2026-03-23 9:54 ` Robin Jarry
2026-03-23 11:34 ` Medvedkin, Vladimir
2026-03-23 11:27 ` Maxime Leroy
2026-03-23 12:49 ` Medvedkin, Vladimir
2026-03-23 14:53 ` Maxime Leroy
2026-03-23 15:08 ` Robin Jarry
2026-03-23 15:27 ` Morten Brørup
2026-03-23 18:52 ` Medvedkin, Vladimir
2026-03-23 18:42 ` Medvedkin, Vladimir
2026-03-24 9:19 ` Maxime Leroy
2026-03-25 15:56 ` Medvedkin, Vladimir
2026-03-25 21:43 ` Maxime Leroy
2026-03-27 18:27 ` Medvedkin, Vladimir
2026-04-02 16:51 ` Maxime Leroy
2026-03-23 19:05 ` Stephen Hemminger
2026-10-14 18:39 ` [RFC PATCH v2 0/7] " Vladimir Medvedkin
2026-10-14 18:39 ` [RFC PATCH v2 1/7] fib: fix return value of a nexthop update Vladimir Medvedkin
2026-10-14 18:39 ` [RFC PATCH v2 2/7] app/test-fib: fix lookup of the last burst Vladimir Medvedkin
2026-10-14 18:39 ` [RFC PATCH v2 3/7] app/test-fib: fix size of the tables for large counts Vladimir Medvedkin
2026-10-14 22:52 ` Stephen Hemminger
2026-10-15 14:21 ` Medvedkin, Vladimir
2026-10-14 18:39 ` [RFC PATCH v2 4/7] fib: add DIR8-16-8 algorithm Vladimir Medvedkin
2026-10-14 18:39 ` [RFC PATCH v2 5/7] fib: add multi-VRF support Vladimir Medvedkin
2026-10-14 18:39 ` [RFC PATCH v2 6/7] fib: split DIR24-8 table update from the FIB wrapper Vladimir Medvedkin
2026-10-14 18:39 ` [RFC PATCH v2 7/7] fib: add hash based VRF support for DIR24-8 Vladimir Medvedkin
2026-10-14 22:55 ` [RFC PATCH v2 0/7] VRF support in FIB library Stephen Hemminger
2026-10-15 14:17 ` Medvedkin, Vladimir
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox