Re: [PATCH v4 0/1] mbuf: add runtime metadata dynamic-field storage

[Stephen Hemminger <stephen@networkplumber.org>]

On Thu, 8 Oct 2026 16:15:26 +0000
"Randy Tice (rtice)" <rtice@cisco.com> wrote:

> I did a brief look at this and still not convinced about the exact use case.
> What is the problem this is trying to solve and why can't it be done
> by using existing API’s.
> 
> #RT:
>   Our implementation currently requires an additional 256 bytes of metadata per
>   mbuf. That does not fit in the existing dynamic-field storage, so today we
>   carry a private patch to extend struct rte_mbuf. The goal of this work is to
>   replace that private struct change with a supported upstream mechanism.
> 
>   We did look at using mbuf private data first. It can work when the application
>   owns all mbuf pool creation, but it is not sufficient for our case without
>   another global/base reservation mechanism. Some mbuf pools are created by
>   drivers or libraries rather than directly by the application; the CNXK inline
>   IPsec/OOP meta pool is one example (NIX_INL_META_POOL, created through
>   cnxk_nix_inl_meta_pool_cb()). To make private data work there, we had to add
>   an EAL argument that reserved a base private size for all pktmbufs, including
>   PMD-created pools.
> 
>   Private data also lacks a central layout registry. If multiple modules use
>   private data, they must coordinate offsets out of band to avoid overlaying
>   each other. The dynamic-field registry solves that coordination problem, but
>   the existing copied dynamic-field area is too small and has copy/clone
>   semantics that are wrong for this metadata.
> 
>   That is why this version uses a globally configured per-mbuf metadata area
>   managed by the dynamic-field registry, with explicit metadata fields that are
>   not copied by generic copy/clone/attach paths. This direction came out of the
>   prior discussion with you, Morten, and me: avoid a Cisco-private mbuf struct
>   patch, avoid per-pool private-data layout coordination, and keep sizeof(struct
>   rte_mbuf) fixed.

You uncovered a design flaw in the CNXK driver and the 
proposed solution is wider than it needs to be.

All mbufs visible to application must come from mempools controlled by the application.
The design of CNXK driver is wrong, it shouldn't be using a private hidden pool.
It is ok for drivers to have hidden mempools that are used for non-visible things,
an example is the packet capture mempool where the mbufs only go into capture stream.

More long winded AI description:

On Thu, 8 Oct 2026 16:15:26 +0000
"Randy Tice (rtice)" <rtice@cisco.com> wrote:

> We did look at using mbuf private data first. It can work when the
> application owns all mbuf pool creation, but it is not sufficient for
> our case without another global/base reservation mechanism. Some mbuf
> pools are created by drivers or libraries rather than directly by the
> application; the CNXK inline IPsec/OOP meta pool is one example

That is the real bug. A driver should not hand the application mbufs
from a pool the application did not create. The Rx queue API already
takes the pool from the application, and rte_eth_rxconf can carry
more than one (rx_mempools). If cnxk needs a meta pool whose mbufs
reach the application, it should get it from the application, or at
minimum create it with the same priv_size as the Rx queue pool.

Internal pools are fine when the mbufs never cross the API, e.g.
dumpcap or the bonding LACP pool.

With that fixed, the existing private area does what you need. It is
per pool, sized by the application, and is not copied by copy, clone
or attach. Offset coordination inside it is the application's job,
and if a registry is wanted it can be layered on top of priv without
changing the mbuf layout.

So my answer to option 1 vs 2 is neither. I don't want a global
layout knob that puts a load in every inline helper and needs
per-driver range checks, to work around one driver.

What I would take:
 - cnxk: validate wqe_skip/later_skip/first_skip against priv_size
   and headroom. This is a bug today, independent of this series.
 - cnxk: meta pool comes from, or matches, the application pool.
 - ethdev: under RTE_ETHDEV_DEBUG_RX, check that received mbufs
   belong to a pool configured on that queue.

Nithin, Harman: do mbufs from NIX_INL_META_POOL get returned to the
application by rx_burst, or are they only consumed internally?