* Re: [PATCH v2] bus/uacce: support no-iommu mode
From: Thomas Monjalon @ 2026-06-01 20:21 UTC (permalink / raw)
To: Chengwen Feng; +Cc: dev, liuyonglong, qianweili
In-Reply-To: <20260518105857.23083-1-fengchengwen@huawei.com>
18/05/2026 12:58, Chengwen Feng:
> The uacce bus originally only supports devices with SVA capability.
> This patch extends the uacce bus to support no-iommu mode for devices
> without or not enabling IOMMU/SVA.
>
> For no-iommu mode UACCE devices:
> - The device api name is suffixed with _noiommu
> - The device flags bit1 was set (UACCE_DEV_FLAG_NOIOMMU)
>
> To support such devices, DPDK device drivers can mark
> RTE_UACCE_DRV_SUPPORT_NOIOMMU_MODE in drv_flags to declare capability
> of working with no-iommu devices.
>
> This commit also fixes typo UACCE_DEV_FLGA_SVA -> UACCE_DEV_FLAG_SVA.
>
> Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
Applied, thanks.
^ permalink raw reply
* Re: [PATCH 0/2] implement missing APIs on FreeBSD
From: Thomas Monjalon @ 2026-06-01 20:12 UTC (permalink / raw)
To: Bruce Richardson; +Cc: dev
In-Reply-To: <20260525171340.1701509-1-bruce.richardson@intel.com>
25/05/2026 19:13, Bruce Richardson:
> The APIs for converting a virtual address to an IOVA or a physical
> address were implemented only as stubs on FreeBSD, but unfortunately
> they are used by drivers such as "igc", part of e1000, causing the
> drivers to fail on BSD. A workaround would be to change the use of these
> functions with the rte_malloc equivalents (e.g. rte_malloc_virt2iova) -
> which do work on FreeBSD - but a better solution is just to replace the
> stubs with proper implementations in FreeBSD-specific EAL flags.
>
> Bruce Richardson (2):
> eal/freebsd: fix memseg addresses in IOVA_AS_VA mode
> eal/freebsd: implement virtual to IOVA translation fn
Applied, thanks.
^ permalink raw reply
* Re: [PATCH] ci: update versions of actions in GHA
From: Thomas Monjalon @ 2026-06-01 19:56 UTC (permalink / raw)
To: David Marchand; +Cc: dev, stable, Aaron Conole
In-Reply-To: <20260511161116.18652-1-david.marchand@redhat.com>
11/05/2026 18:11, David Marchand:
> GitHub started deprecating GHA actions based on Node 20.
> For now, only warnings are raised, but we can switch to more recent
> versions of the common actions, now:
> - cache v5
> - checkout v6
> - upload-artifact v7
> - setup-python v6
>
> Link: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
>
> Cc: stable@dpdk.org
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
Applied, thanks.
^ permalink raw reply
* Re: [PATCH v11] eal/x86: optimize memcpy of small sizes
From: Thomas Monjalon @ 2026-06-01 19:48 UTC (permalink / raw)
To: Morten Brørup
Cc: dev, Bruce Richardson, Konstantin Ananyev, Vipin Varghese,
Stephen Hemminger, Liangxing Wang, Thiyagarajan P,
Bala Murali Krishna, Anatoly Burakov, Vladimir Medvedkin,
Morten Brørup, Konstantin Ananyev
In-Reply-To: <20260521185631.116046-1-mb@smartsharesystems.com>
21/05/2026 20:56, Morten Brørup:
> The implementation for copying up to 64 bytes does not depend on address
> alignment with the size of the CPU's vector registers. Nonetheless, the
> exact same code for copying up to 64 bytes was present in both the aligned
> copy function and all the CPU vector register size specific variants of
> the unaligned copy functions.
> With this patch, the implementation for copying up to 64 bytes was
> consolidated into one instance, located in the common copy function,
> before checking alignment requirements.
> This provides three benefits:
> 1. No copy-paste in the source code.
> 2. A performance gain for copying up to 64 bytes, because the
> address alignment check is avoided in this case.
> 3. Reduced instruction memory footprint, because the compiler only
> generates one instance of the function for copying up to 64 bytes, instead
> of two instances (one in the unaligned copy function, and one in the
> aligned copy function).
>
> Furthermore, __rte_restrict was added to source and destination addresses.
>
> Also, the missing implementation of rte_mov48() was added.
>
> Until recently, some drivers required disabling stringop-overflow warnings
> when using rte_memcpy().
> For some strange reason, these warnings were disabled in the rte_memcpy
> header file, instead of in the problematic drivers.
> With series-38174 ("remove use of rte_memcpy from net/intel"), the
> problematic drivers were updated to use memcpy() instead of rte_memcpy(),
> so disabling these warnings is no longer required, and was removed.
>
> Regarding performance...
> The memcpy performance test (cache-to-cache copy) shows:
> Copying up to 15 bytes takes ca. 4.5 cycles, versus ca. 6.5 cycles before.
> Copying 8 bytes takes 4 cycles, versus 7 cycles before.
> Copying 16 bytes takes 2 cycles, versus 4 cycles before.
> Copying 64 bytes takes 4 cycles, versus 7 cycles before.
>
> Depends-on: series-38174 ("remove use of rte_memcpy from net/intel")
>
> Signed-off-by: Morten Brørup <mb@smartsharesystems.com>
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> Acked-by: Konstantin Ananyev <konstantin.ananyev@huawei.com>
Applied, thanks.
^ permalink raw reply
* RE: [PATCH v2] doc, eal, devtools: discourage new __rte_always_inline
From: Morten Brørup @ 2026-06-01 18:50 UTC (permalink / raw)
To: Stephen Hemminger, dev; +Cc: Thomas Monjalon
In-Reply-To: <20260601172104.311909-1-stephen@networkplumber.org>
> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> Sent: Monday, 1 June 2026 19.21
>
> Modern compilers at -O2 make good inlining decisions for small
> static inline functions; forced inlining via __rte_always_inline
> should be reserved for cases where it is required for correctness
> or for documented measured performance reasons.
>
> Document the policy in the coding style guide and add a
> checkpatches.sh entry that flags when new uses of the attribute
> are introduced. Checkpatches is not an absolute blocker to
> acceptance, only an indicator that more review is needed.
>
> Add additional comments about use of __rte_always_inline,
> __rte_noinline, __rte_hot, and __rte_cold to the rte_common.h
> to aid developers.
>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
Reviewed-by: Morten Brørup <mb@smartsharesystems.com>
^ permalink raw reply
* [PATCH v7] net/idpf: update for new mempool cache algorithm
From: Morten Brørup @ 2026-06-01 18:36 UTC (permalink / raw)
To: dev, Andrew Rybchenko, Bruce Richardson, Jingjing Wu,
Praveen Shetty, Hemant Agrawal, Sachin Saxena
Cc: Morten Brørup
In-Reply-To: <20260408141315.904381-1-mb@smartsharesystems.com>
As a consequence of the improved mempool cache algorithm, the PMD was
updated regarding how much to backfill the mempool cache in the AVX512
code path.
Signed-off-by: Morten Brørup <mb@smartsharesystems.com>
---
v7:
* Rebased.
v6:
* Moved driver changes out as separate patches, for easier review. (Bruce)
---
Depends-on: patch-164745 ("mempool: improve cache behaviour and performance")
---
.../net/intel/idpf/idpf_common_rxtx_avx512.c | 52 +++++++++++++++----
1 file changed, 42 insertions(+), 10 deletions(-)
diff --git a/drivers/net/intel/idpf/idpf_common_rxtx_avx512.c b/drivers/net/intel/idpf/idpf_common_rxtx_avx512.c
index 8db4c64106..5788a009ab 100644
--- a/drivers/net/intel/idpf/idpf_common_rxtx_avx512.c
+++ b/drivers/net/intel/idpf/idpf_common_rxtx_avx512.c
@@ -148,15 +148,31 @@ idpf_singleq_rearm(struct idpf_rx_queue *rxq)
/* Can this be satisfied from the cache? */
if (cache->len < IDPF_RXQ_REARM_THRESH) {
/* No. Backfill the cache first, and then fill from it */
- uint32_t req = IDPF_RXQ_REARM_THRESH + (cache->size -
- cache->len);
- /* How many do we require i.e. number to fill the cache + the request */
+ /* Backfill would exceed the cache bounce buffer limit? */
+ __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
+ if (unlikely(cache->size / 2 < IDPF_RXQ_REARM_THRESH)) {
+ idpf_singleq_rearm_common(rxq);
+ return;
+ }
+
+ /*
+ * Backfill the cache from the backend;
+ * move up the hot objects in the cache to the top half of the cache,
+ * and fetch (size / 2) objects to the bottom of the cache.
+ */
+ __rte_assume(cache->len < cache->size / 2);
+ rte_memcpy(&cache->objs[cache->size / 2], &cache->objs[0],
+ sizeof(void *) * cache->len);
int ret = rte_mempool_ops_dequeue_bulk
- (rxq->mp, &cache->objs[cache->len], req);
+ (rxq->mp, &cache->objs[0], cache->size / 2);
if (ret == 0) {
- cache->len += req;
+ cache->len += cache->size / 2;
} else {
+ /*
+ * No further action is required for roll back, as the objects moved
+ * in the cache were actually copied, and the cache remains intact.
+ */
if (rxq->rxrearm_nb + IDPF_RXQ_REARM_THRESH >=
rxq->nb_rx_desc) {
__m128i dma_addr0;
@@ -565,15 +581,31 @@ idpf_splitq_rearm(struct idpf_rx_queue *rx_bufq)
/* Can this be satisfied from the cache? */
if (cache->len < IDPF_RXQ_REARM_THRESH) {
/* No. Backfill the cache first, and then fill from it */
- uint32_t req = IDPF_RXQ_REARM_THRESH + (cache->size -
- cache->len);
- /* How many do we require i.e. number to fill the cache + the request */
+ /* Backfill would exceed the cache bounce buffer limit? */
+ __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
+ if (unlikely(cache->size / 2 < IDPF_RXQ_REARM_THRESH)) {
+ idpf_splitq_rearm_common(rx_bufq);
+ return;
+ }
+
+ /*
+ * Backfill the cache from the backend;
+ * move up the hot objects in the cache to the top half of the cache,
+ * and fetch (size / 2) objects to the bottom of the cache.
+ */
+ __rte_assume(cache->len < cache->size / 2);
+ rte_memcpy(&cache->objs[cache->size / 2], &cache->objs[0],
+ sizeof(void *) * cache->len);
int ret = rte_mempool_ops_dequeue_bulk
- (rx_bufq->mp, &cache->objs[cache->len], req);
+ (rx_bufq->mp, &cache->objs[0], cache->size / 2);
if (ret == 0) {
- cache->len += req;
+ cache->len += cache->size / 2;
} else {
+ /*
+ * No further action is required for roll back, as the objects moved
+ * in the cache were actually copied, and the cache remains intact.
+ */
if (rx_bufq->rxrearm_nb + IDPF_RXQ_REARM_THRESH >=
rx_bufq->nb_rx_desc) {
__m128i dma_addr0;
--
2.43.0
^ permalink raw reply related
* [PATCH v7] mempool/dpaa: update for new mempool cache algorithm
From: Morten Brørup @ 2026-06-01 18:36 UTC (permalink / raw)
To: dev, Andrew Rybchenko, Bruce Richardson, Jingjing Wu,
Praveen Shetty, Hemant Agrawal, Sachin Saxena
Cc: Morten Brørup
In-Reply-To: <20260601183621.252920-1-mb@smartsharesystems.com>
As a consequence of the improved mempool cache algorithm, the mempool
driver was updated to not modify the mempool cache's flushthresh field,
which is now obsolete, and modifying it has no effect.
Signed-off-by: Morten Brørup <mb@smartsharesystems.com>
---
v7:
* Rebased.
v6:
* Moved driver changes out as separate patches, for easier review. (Bruce)
---
Depends-on: patch-164745 ("mempool: improve cache behaviour and performance")
---
drivers/mempool/dpaa/dpaa_mempool.c | 14 --------------
1 file changed, 14 deletions(-)
diff --git a/drivers/mempool/dpaa/dpaa_mempool.c b/drivers/mempool/dpaa/dpaa_mempool.c
index 2f9395b3f4..2f8555a026 100644
--- a/drivers/mempool/dpaa/dpaa_mempool.c
+++ b/drivers/mempool/dpaa/dpaa_mempool.c
@@ -58,8 +58,6 @@ dpaa_mbuf_create_pool(struct rte_mempool *mp)
struct bman_pool_params params = {
.flags = BMAN_POOL_FLAG_DYNAMIC_BPID
};
- unsigned int lcore_id;
- struct rte_mempool_cache *cache;
MEMPOOL_INIT_FUNC_TRACE();
@@ -129,18 +127,6 @@ dpaa_mbuf_create_pool(struct rte_mempool *mp)
rte_memcpy(bp_info, (void *)&rte_dpaa_bpid_info[bpid],
sizeof(struct dpaa_bp_info));
mp->pool_data = (void *)bp_info;
- /* Update per core mempool cache threshold to optimal value which is
- * number of buffers that can be released to HW buffer pool in
- * a single API call.
- */
- for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
- cache = &mp->local_cache[lcore_id];
- DPAA_MEMPOOL_DEBUG("lCore %d: cache->flushthresh %d -> %d",
- lcore_id, cache->flushthresh,
- (uint32_t)(cache->size + DPAA_MBUF_MAX_ACQ_REL));
- if (cache->flushthresh)
- cache->flushthresh = cache->size + DPAA_MBUF_MAX_ACQ_REL;
- }
DPAA_MEMPOOL_INFO("BMAN pool created for bpid =%d", bpid);
return 0;
--
2.43.0
^ permalink raw reply related
* [PATCH v7] mempool/dpaa2: update for new mempool cache algorithm
From: Morten Brørup @ 2026-06-01 18:36 UTC (permalink / raw)
To: dev, Andrew Rybchenko, Bruce Richardson, Jingjing Wu,
Praveen Shetty, Hemant Agrawal, Sachin Saxena
Cc: Morten Brørup
In-Reply-To: <20260601183621.252920-1-mb@smartsharesystems.com>
As a consequence of the improved mempool cache algorithm, the mempool
driver was updated to not modify the mempool cache's flushthresh field,
which is now obsolete, and modifying it has no effect.
Signed-off-by: Morten Brørup <mb@smartsharesystems.com>
---
v7:
* Rebased.
v6:
* Moved driver changes out as separate patches, for easier review. (Bruce)
---
Depends-on: patch-164745 ("mempool: improve cache behaviour and performance")
---
drivers/mempool/dpaa2/dpaa2_hw_mempool.c | 14 --------------
1 file changed, 14 deletions(-)
diff --git a/drivers/mempool/dpaa2/dpaa2_hw_mempool.c b/drivers/mempool/dpaa2/dpaa2_hw_mempool.c
index 02b6741853..ee001d8ce0 100644
--- a/drivers/mempool/dpaa2/dpaa2_hw_mempool.c
+++ b/drivers/mempool/dpaa2/dpaa2_hw_mempool.c
@@ -54,8 +54,6 @@ rte_hw_mbuf_create_pool(struct rte_mempool *mp)
struct dpaa2_bp_info *bp_info;
struct dpbp_attr dpbp_attr;
uint32_t bpid;
- unsigned int lcore_id;
- struct rte_mempool_cache *cache;
int ret;
avail_dpbp = dpaa2_alloc_dpbp_dev();
@@ -152,18 +150,6 @@ rte_hw_mbuf_create_pool(struct rte_mempool *mp)
DPAA2_MEMPOOL_DEBUG("BP List created for bpid =%d", dpbp_attr.bpid);
h_bp_list = bp_list;
- /* Update per core mempool cache threshold to optimal value which is
- * number of buffers that can be released to HW buffer pool in
- * a single API call.
- */
- for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
- cache = &mp->local_cache[lcore_id];
- DPAA2_MEMPOOL_DEBUG("lCore %d: cache->flushthresh %d -> %d",
- lcore_id, cache->flushthresh,
- (uint32_t)(cache->size + DPAA2_MBUF_MAX_ACQ_REL));
- if (cache->flushthresh)
- cache->flushthresh = cache->size + DPAA2_MBUF_MAX_ACQ_REL;
- }
return 0;
err4:
--
2.43.0
^ permalink raw reply related
* RE: [PATCH v4 02/27] eal: reimplement rte_smp_*mb with rte_atomic_thread_fence
From: Konstantin Ananyev @ 2026-06-01 18:24 UTC (permalink / raw)
To: Stephen Hemminger, dev@dpdk.org
Cc: Thomas Monjalon, Wathsala Vithanage, Bibo Mao, David Christensen,
Sun Yuechi, Bruce Richardson
In-Reply-To: <20260526232542.620966-3-stephen@networkplumber.org>
> The rte_smp_mb(), rte_smp_wmb() and rte_smp_rmb() functions were
> flagged as deprecated by commit 3ec965b6de12 ("doc: update atomic
> operation deprecation") in 2021 but nothing came of it.
>
> Reimplement them as inline wrappers over rte_atomic_thread_fence()
> and drop the deprecation notice.
> The API is preserved; only the implementation changes.
>
> The wrapper provides stronger guarantees than previous code
> because there is no C11 equivalent to old rte_smp_qmb().
> Generated code is unchanged on x86; on arm64,
> release/acquire emit dmb ish instead of dmb ishst/ishld;
> the difference is below measurement noise.
>
> Drop restrictions on rte_smp_XX in checkpatch since they are
> no longer on deprecation cycle.
>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
> devtools/checkpatches.sh | 8 --
> doc/guides/rel_notes/deprecation.rst | 8 --
> lib/eal/arm/include/rte_atomic_32.h | 6 --
> lib/eal/arm/include/rte_atomic_64.h | 6 --
> lib/eal/include/generic/rte_atomic.h | 130 +++++--------------------
> lib/eal/loongarch/include/rte_atomic.h | 6 --
> lib/eal/ppc/include/rte_atomic.h | 6 --
> lib/eal/riscv/include/rte_atomic.h | 6 --
> lib/eal/x86/include/rte_atomic.h | 33 +++----
> 9 files changed, 37 insertions(+), 172 deletions(-)
>
> diff --git a/devtools/checkpatches.sh b/devtools/checkpatches.sh
> index f5dd77443f..81bb0fe4e8 100755
> --- a/devtools/checkpatches.sh
> +++ b/devtools/checkpatches.sh
> @@ -121,14 +121,6 @@ check_forbidden_additions() { # <patch>
> -f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \
> "$1" || res=1
>
> - # refrain from new additions of rte_smp_[r/w]mb()
> - awk -v FOLDERS="lib drivers app examples" \
> - -v EXPRESSIONS="rte_smp_(r|w)?mb\\\(" \
> - -v RET_ON_FAIL=1 \
> - -v MESSAGE='Using rte_smp_[r/w]mb' \
> - -f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \
> - "$1" || res=1
> -
> # refrain from using compiler __sync_xxx builtins
> awk -v FOLDERS="lib drivers app examples" \
> -v EXPRESSIONS="__sync_.*\\\(" \
> diff --git a/doc/guides/rel_notes/deprecation.rst
> b/doc/guides/rel_notes/deprecation.rst
> index 35c9b4e06c..2190419f79 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -47,14 +47,6 @@ Deprecation Notices
> operations must be used for patches that need to be merged in 20.08 onwards.
> This change will not introduce any performance degradation.
>
> -* rte_smp_*mb: These APIs provide full barrier functionality. However, many
> - use cases do not require full barriers. To support such use cases, DPDK has
> - adopted atomic operations from
> - https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html. These
> - operations and a new wrapper ``rte_atomic_thread_fence`` instead of
> - ``__atomic_thread_fence`` must be used for patches that need to be merged in
> - 20.08 onwards. This change will not introduce any performance degradation.
> -
> * lib: will fix extending some enum/define breaking the ABI. There are multiple
> samples in DPDK that enum/define terminated with a ``.*MAX.*`` value which is
> used by iterators, and arrays holding these values are sized with this
> diff --git a/lib/eal/arm/include/rte_atomic_32.h
> b/lib/eal/arm/include/rte_atomic_32.h
> index 696a539fef..4115271091 100644
> --- a/lib/eal/arm/include/rte_atomic_32.h
> +++ b/lib/eal/arm/include/rte_atomic_32.h
> @@ -17,12 +17,6 @@ extern "C" {
>
> #define rte_rmb() __sync_synchronize()
>
> -#define rte_smp_mb() rte_mb()
> -
> -#define rte_smp_wmb() rte_wmb()
> -
> -#define rte_smp_rmb() rte_rmb()
> -
> #define rte_io_mb() rte_mb()
>
> #define rte_io_wmb() rte_wmb()
> diff --git a/lib/eal/arm/include/rte_atomic_64.h
> b/lib/eal/arm/include/rte_atomic_64.h
> index 9f790238df..604e777bcd 100644
> --- a/lib/eal/arm/include/rte_atomic_64.h
> +++ b/lib/eal/arm/include/rte_atomic_64.h
> @@ -20,12 +20,6 @@ extern "C" {
>
> #define rte_rmb() asm volatile("dmb oshld" : : : "memory")
>
> -#define rte_smp_mb() asm volatile("dmb ish" : : : "memory")
> -
> -#define rte_smp_wmb() asm volatile("dmb ishst" : : : "memory")
> -
> -#define rte_smp_rmb() asm volatile("dmb ishld" : : : "memory")
> -
> #define rte_io_mb() rte_mb()
>
> #define rte_io_wmb() rte_wmb()
> diff --git a/lib/eal/include/generic/rte_atomic.h
> b/lib/eal/include/generic/rte_atomic.h
> index 292e52fade..1b04b43cbb 100644
> --- a/lib/eal/include/generic/rte_atomic.h
> +++ b/lib/eal/include/generic/rte_atomic.h
> @@ -59,55 +59,25 @@ static inline void rte_rmb(void);
> *
> * Guarantees that the LOAD and STORE operations that precede the
> * rte_smp_mb() call are globally visible across the lcores
> - * before the LOAD and STORE operations that follows it.
> - *
> - * @note
> - * This function is deprecated.
> - * It provides similar synchronization primitive as atomic fence,
> - * but has different syntax and memory ordering semantic. Hence
> - * deprecated for the simplicity of memory ordering semantics in use.
> - *
> - * rte_atomic_thread_fence(rte_memory_order_acq_rel) should be used
> instead.
> + * before the LOAD and STORE operations that follow it.
> */
> static inline void rte_smp_mb(void);
>
> /**
> * Write memory barrier between lcores
> *
> - * Guarantees that the STORE operations that precede the
> - * rte_smp_wmb() call are globally visible across the lcores
> - * before the STORE operations that follows it.
> - *
> - * @note
> - * This function is deprecated.
> - * It provides similar synchronization primitive as atomic fence,
> - * but has different syntax and memory ordering semantic. Hence
> - * deprecated for the simplicity of memory ordering semantics in use.
> - *
> - * rte_atomic_thread_fence(rte_memory_order_release) should be used
> instead.
> - * The fence also guarantees LOAD operations that precede the call
> - * are globally visible across the lcores before the STORE operations
> - * that follows it.
> + * Guarantees that the LOAD and STORE operations that precede the
> + * rte_smp_wmb() call are globally visible across the lcores before
> + * any STORE operations that follow it.
> */
> static inline void rte_smp_wmb(void);
>
> /**
> * Read memory barrier between lcores
> *
> - * Guarantees that the LOAD operations that precede the
> - * rte_smp_rmb() call are globally visible across the lcores
> - * before the LOAD operations that follows it.
> - *
> - * @note
> - * This function is deprecated.
> - * It provides similar synchronization primitive as atomic fence,
> - * but has different syntax and memory ordering semantic. Hence
> - * deprecated for the simplicity of memory ordering semantics in use.
> - *
> - * rte_atomic_thread_fence(rte_memory_order_acquire) should be used
> instead.
> - * The fence also guarantees LOAD operations that precede the call
> - * are globally visible across the lcores before the STORE operations
> - * that follows it.
> + * Guarantees that any LOAD operations that precede the rte_smp_rmb()
> + * call complete before LOAD and STORE operations that follow it
> + * become globally visible.
> */
> static inline void rte_smp_rmb(void);
> ///@}
> @@ -164,6 +134,24 @@ static inline void rte_io_rmb(void);
> */
> static inline void rte_atomic_thread_fence(rte_memory_order memorder);
>
> +static __rte_always_inline void
> +rte_smp_mb(void)
> +{
> + rte_atomic_thread_fence(rte_memory_order_seq_cst);
> +}
> +
> +static __rte_always_inline void
> +rte_smp_wmb(void)
> +{
> + rte_atomic_thread_fence(rte_memory_order_release);
> +}
> +
> +static __rte_always_inline void
> +rte_smp_rmb(void)
> +{
> + rte_atomic_thread_fence(rte_memory_order_acquire);
> +}
> +
> /*------------------------- 16 bit atomic operations -------------------------*/
>
> #ifndef RTE_TOOLCHAIN_MSVC
> @@ -184,9 +172,6 @@ static inline void
> rte_atomic_thread_fence(rte_memory_order memorder);
> * @return
> * Non-zero on success; 0 on failure.
> */
> -static inline int
> -rte_atomic16_cmpset(volatile uint16_t *dst, uint16_t exp, uint16_t src);
> -
> static inline int
> rte_atomic16_cmpset(volatile uint16_t *dst, uint16_t exp, uint16_t src)
> {
> @@ -303,9 +288,6 @@ rte_atomic16_sub(rte_atomic16_t *v, int16_t dec)
> * @param v
> * A pointer to the atomic counter.
> */
> -static inline void
> -rte_atomic16_inc(rte_atomic16_t *v);
> -
> static inline void
> rte_atomic16_inc(rte_atomic16_t *v)
> {
> @@ -318,9 +300,6 @@ rte_atomic16_inc(rte_atomic16_t *v)
> * @param v
> * A pointer to the atomic counter.
> */
> -static inline void
> -rte_atomic16_dec(rte_atomic16_t *v);
> -
> static inline void
> rte_atomic16_dec(rte_atomic16_t *v)
> {
> @@ -379,8 +358,6 @@ rte_atomic16_sub_return(rte_atomic16_t *v, int16_t dec)
> * @return
> * True if the result after the increment operation is 0; false otherwise.
> */
> -static inline int rte_atomic16_inc_and_test(rte_atomic16_t *v);
> -
> static inline int rte_atomic16_inc_and_test(rte_atomic16_t *v)
> {
> return rte_atomic_fetch_add_explicit((volatile __rte_atomic int16_t *)&v-
> >cnt, 1,
> @@ -398,8 +375,6 @@ static inline int
> rte_atomic16_inc_and_test(rte_atomic16_t *v)
> * @return
> * True if the result after the decrement operation is 0; false otherwise.
> */
> -static inline int rte_atomic16_dec_and_test(rte_atomic16_t *v);
> -
> static inline int rte_atomic16_dec_and_test(rte_atomic16_t *v)
> {
> return rte_atomic_fetch_sub_explicit((volatile __rte_atomic int16_t *)&v-
> >cnt, 1,
> @@ -417,8 +392,6 @@ static inline int
> rte_atomic16_dec_and_test(rte_atomic16_t *v)
> * @return
> * 0 if failed; else 1, success.
> */
> -static inline int rte_atomic16_test_and_set(rte_atomic16_t *v);
> -
> static inline int rte_atomic16_test_and_set(rte_atomic16_t *v)
> {
> return rte_atomic16_cmpset((volatile uint16_t *)&v->cnt, 0, 1);
> @@ -453,9 +426,6 @@ static inline void rte_atomic16_clear(rte_atomic16_t *v)
> * @return
> * Non-zero on success; 0 on failure.
> */
> -static inline int
> -rte_atomic32_cmpset(volatile uint32_t *dst, uint32_t exp, uint32_t src);
> -
> static inline int
> rte_atomic32_cmpset(volatile uint32_t *dst, uint32_t exp, uint32_t src)
> {
> @@ -572,9 +542,6 @@ rte_atomic32_sub(rte_atomic32_t *v, int32_t dec)
> * @param v
> * A pointer to the atomic counter.
> */
> -static inline void
> -rte_atomic32_inc(rte_atomic32_t *v);
> -
> static inline void
> rte_atomic32_inc(rte_atomic32_t *v)
> {
> @@ -587,9 +554,6 @@ rte_atomic32_inc(rte_atomic32_t *v)
> * @param v
> * A pointer to the atomic counter.
> */
> -static inline void
> -rte_atomic32_dec(rte_atomic32_t *v);
> -
> static inline void
> rte_atomic32_dec(rte_atomic32_t *v)
> {
> @@ -648,8 +612,6 @@ rte_atomic32_sub_return(rte_atomic32_t *v, int32_t dec)
> * @return
> * True if the result after the increment operation is 0; false otherwise.
> */
> -static inline int rte_atomic32_inc_and_test(rte_atomic32_t *v);
> -
> static inline int rte_atomic32_inc_and_test(rte_atomic32_t *v)
> {
> return rte_atomic_fetch_add_explicit((volatile __rte_atomic int32_t *)&v-
> >cnt, 1,
> @@ -667,8 +629,6 @@ static inline int
> rte_atomic32_inc_and_test(rte_atomic32_t *v)
> * @return
> * True if the result after the decrement operation is 0; false otherwise.
> */
> -static inline int rte_atomic32_dec_and_test(rte_atomic32_t *v);
> -
> static inline int rte_atomic32_dec_and_test(rte_atomic32_t *v)
> {
> return rte_atomic_fetch_sub_explicit((volatile __rte_atomic int32_t *)&v-
> >cnt, 1,
> @@ -686,8 +646,6 @@ static inline int
> rte_atomic32_dec_and_test(rte_atomic32_t *v)
> * @return
> * 0 if failed; else 1, success.
> */
> -static inline int rte_atomic32_test_and_set(rte_atomic32_t *v);
> -
> static inline int rte_atomic32_test_and_set(rte_atomic32_t *v)
> {
> return rte_atomic32_cmpset((volatile uint32_t *)&v->cnt, 0, 1);
> @@ -721,9 +679,6 @@ static inline void rte_atomic32_clear(rte_atomic32_t *v)
> * @return
> * Non-zero on success; 0 on failure.
> */
> -static inline int
> -rte_atomic64_cmpset(volatile uint64_t *dst, uint64_t exp, uint64_t src);
> -
> static inline int
> rte_atomic64_cmpset(volatile uint64_t *dst, uint64_t exp, uint64_t src)
> {
> @@ -770,9 +725,6 @@ typedef struct {
> * @param v
> * A pointer to the atomic counter.
> */
> -static inline void
> -rte_atomic64_init(rte_atomic64_t *v);
> -
> static inline void
> rte_atomic64_init(rte_atomic64_t *v)
> {
> @@ -798,9 +750,6 @@ rte_atomic64_init(rte_atomic64_t *v)
> * @return
> * The value of the counter.
> */
> -static inline int64_t
> -rte_atomic64_read(rte_atomic64_t *v);
> -
> static inline int64_t
> rte_atomic64_read(rte_atomic64_t *v)
> {
> @@ -828,9 +777,6 @@ rte_atomic64_read(rte_atomic64_t *v)
> * @param new_value
> * The new value of the counter.
> */
> -static inline void
> -rte_atomic64_set(rte_atomic64_t *v, int64_t new_value);
> -
> static inline void
> rte_atomic64_set(rte_atomic64_t *v, int64_t new_value)
> {
> @@ -856,9 +802,6 @@ rte_atomic64_set(rte_atomic64_t *v, int64_t new_value)
> * @param inc
> * The value to be added to the counter.
> */
> -static inline void
> -rte_atomic64_add(rte_atomic64_t *v, int64_t inc);
> -
> static inline void
> rte_atomic64_add(rte_atomic64_t *v, int64_t inc)
> {
> @@ -874,9 +817,6 @@ rte_atomic64_add(rte_atomic64_t *v, int64_t inc)
> * @param dec
> * The value to be subtracted from the counter.
> */
> -static inline void
> -rte_atomic64_sub(rte_atomic64_t *v, int64_t dec);
> -
> static inline void
> rte_atomic64_sub(rte_atomic64_t *v, int64_t dec)
> {
> @@ -890,9 +830,6 @@ rte_atomic64_sub(rte_atomic64_t *v, int64_t dec)
> * @param v
> * A pointer to the atomic counter.
> */
> -static inline void
> -rte_atomic64_inc(rte_atomic64_t *v);
> -
> static inline void
> rte_atomic64_inc(rte_atomic64_t *v)
> {
> @@ -905,9 +842,6 @@ rte_atomic64_inc(rte_atomic64_t *v)
> * @param v
> * A pointer to the atomic counter.
> */
> -static inline void
> -rte_atomic64_dec(rte_atomic64_t *v);
> -
> static inline void
> rte_atomic64_dec(rte_atomic64_t *v)
> {
> @@ -927,9 +861,6 @@ rte_atomic64_dec(rte_atomic64_t *v)
> * @return
> * The value of v after the addition.
> */
> -static inline int64_t
> -rte_atomic64_add_return(rte_atomic64_t *v, int64_t inc);
> -
> static inline int64_t
> rte_atomic64_add_return(rte_atomic64_t *v, int64_t inc)
> {
> @@ -950,9 +881,6 @@ rte_atomic64_add_return(rte_atomic64_t *v, int64_t inc)
> * @return
> * The value of v after the subtraction.
> */
> -static inline int64_t
> -rte_atomic64_sub_return(rte_atomic64_t *v, int64_t dec);
> -
> static inline int64_t
> rte_atomic64_sub_return(rte_atomic64_t *v, int64_t dec)
> {
> @@ -971,8 +899,6 @@ rte_atomic64_sub_return(rte_atomic64_t *v, int64_t dec)
> * @return
> * True if the result after the addition is 0; false otherwise.
> */
> -static inline int rte_atomic64_inc_and_test(rte_atomic64_t *v);
> -
> static inline int rte_atomic64_inc_and_test(rte_atomic64_t *v)
> {
> return rte_atomic64_add_return(v, 1) == 0;
> @@ -989,8 +915,6 @@ static inline int
> rte_atomic64_inc_and_test(rte_atomic64_t *v)
> * @return
> * True if the result after subtraction is 0; false otherwise.
> */
> -static inline int rte_atomic64_dec_and_test(rte_atomic64_t *v);
> -
> static inline int rte_atomic64_dec_and_test(rte_atomic64_t *v)
> {
> return rte_atomic64_sub_return(v, 1) == 0;
> @@ -1007,8 +931,6 @@ static inline int
> rte_atomic64_dec_and_test(rte_atomic64_t *v)
> * @return
> * 0 if failed; else 1, success.
> */
> -static inline int rte_atomic64_test_and_set(rte_atomic64_t *v);
> -
> static inline int rte_atomic64_test_and_set(rte_atomic64_t *v)
> {
> return rte_atomic64_cmpset((volatile uint64_t *)&v->cnt, 0, 1);
> @@ -1020,8 +942,6 @@ static inline int
> rte_atomic64_test_and_set(rte_atomic64_t *v)
> * @param v
> * A pointer to the atomic counter.
> */
> -static inline void rte_atomic64_clear(rte_atomic64_t *v);
> -
> static inline void rte_atomic64_clear(rte_atomic64_t *v)
> {
> rte_atomic64_set(v, 0);
> diff --git a/lib/eal/loongarch/include/rte_atomic.h
> b/lib/eal/loongarch/include/rte_atomic.h
> index 785a452c9e..a789e3ab4d 100644
> --- a/lib/eal/loongarch/include/rte_atomic.h
> +++ b/lib/eal/loongarch/include/rte_atomic.h
> @@ -18,12 +18,6 @@ extern "C" {
>
> #define rte_rmb() rte_mb()
>
> -#define rte_smp_mb() rte_mb()
> -
> -#define rte_smp_wmb() rte_mb()
> -
> -#define rte_smp_rmb() rte_mb()
> -
> #define rte_io_mb() rte_mb()
>
> #define rte_io_wmb() rte_mb()
> diff --git a/lib/eal/ppc/include/rte_atomic.h b/lib/eal/ppc/include/rte_atomic.h
> index 64f4c3d670..0e64db2a35 100644
> --- a/lib/eal/ppc/include/rte_atomic.h
> +++ b/lib/eal/ppc/include/rte_atomic.h
> @@ -24,12 +24,6 @@ extern "C" {
>
> #define rte_rmb() asm volatile("sync" : : : "memory")
>
> -#define rte_smp_mb() rte_mb()
> -
> -#define rte_smp_wmb() rte_wmb()
> -
> -#define rte_smp_rmb() rte_rmb()
> -
> #define rte_io_mb() rte_mb()
>
> #define rte_io_wmb() rte_wmb()
> diff --git a/lib/eal/riscv/include/rte_atomic.h b/lib/eal/riscv/include/rte_atomic.h
> index 061b175f33..04c40e4e9b 100644
> --- a/lib/eal/riscv/include/rte_atomic.h
> +++ b/lib/eal/riscv/include/rte_atomic.h
> @@ -23,12 +23,6 @@ extern "C" {
>
> #define rte_rmb() asm volatile("fence r, r" : : : "memory")
>
> -#define rte_smp_mb() rte_mb()
> -
> -#define rte_smp_wmb() rte_wmb()
> -
> -#define rte_smp_rmb() rte_rmb()
> -
> #define rte_io_mb() asm volatile("fence iorw, iorw" : : : "memory")
>
> #define rte_io_wmb() asm volatile("fence orw, ow" : : : "memory")
> diff --git a/lib/eal/x86/include/rte_atomic.h b/lib/eal/x86/include/rte_atomic.h
> index 4f05302c9f..f4d39ce4fe 100644
> --- a/lib/eal/x86/include/rte_atomic.h
> +++ b/lib/eal/x86/include/rte_atomic.h
> @@ -23,10 +23,6 @@
>
> #define rte_rmb() _mm_lfence()
>
> -#define rte_smp_wmb() rte_compiler_barrier()
> -
> -#define rte_smp_rmb() rte_compiler_barrier()
> -
> #ifdef __cplusplus
> extern "C" {
> #endif
> @@ -63,20 +59,6 @@ extern "C" {
> * So below we use that technique for rte_smp_mb() implementation.
> */
>
> -static __rte_always_inline void
> -rte_smp_mb(void)
> -{
> -#ifdef RTE_TOOLCHAIN_MSVC
> - _mm_mfence();
> -#else
> -#ifdef RTE_ARCH_I686
> - asm volatile("lock addl $0, -128(%%esp); " ::: "memory");
> -#else
> - asm volatile("lock addl $0, -128(%%rsp); " ::: "memory");
> -#endif
> -#endif
> -}
> -
> #define rte_io_mb() rte_mb()
>
> #define rte_io_wmb() rte_compiler_barrier()
> @@ -93,10 +75,19 @@ rte_smp_mb(void)
> static __rte_always_inline void
> rte_atomic_thread_fence(rte_memory_order memorder)
> {
> - if (memorder == rte_memory_order_seq_cst)
> - rte_smp_mb();
> - else
> + if (memorder == rte_memory_order_seq_cst) {
> +#ifdef RTE_TOOLCHAIN_MSVC
> + _mm_mfence();
> +#else
> +#ifdef RTE_ARCH_I686
> + asm volatile("lock addl $0, -128(%%esp); " ::: "memory");
> +#else
> + asm volatile("lock addl $0, -128(%%rsp); " ::: "memory");
> +#endif
> +#endif
> + } else {
> __rte_atomic_thread_fence(memorder);
> + }
> }
>
> #ifdef __cplusplus
> --
Acked-by: Konstantin Ananyev <konstantin.ananyev@huawei.com>
> 2.53.0
^ permalink raw reply
* RE: [PATCH v4 01/27] eal: use intrinsics for rte_atomic on all platforms
From: Konstantin Ananyev @ 2026-06-01 18:23 UTC (permalink / raw)
To: Stephen Hemminger, dev@dpdk.org
Cc: Wathsala Vithanage, Bibo Mao, David Christensen, Sun Yuechi,
Bruce Richardson
In-Reply-To: <20260526232542.620966-2-stephen@networkplumber.org>
> Next step is to deprecate the rte_atomicNN_*() family. Rather than
> maintaining both the inline asm and intrinsic fallbacks, drop the
> asm paths and use intrinsics everywhere.
>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
> lib/eal/arm/include/rte_atomic_32.h | 4 -
> lib/eal/arm/include/rte_atomic_64.h | 4 -
> lib/eal/include/generic/rte_atomic.h | 76 +---------
> lib/eal/loongarch/include/rte_atomic.h | 4 -
> lib/eal/ppc/include/rte_atomic.h | 173 -----------------------
> lib/eal/riscv/include/rte_atomic.h | 4 -
> lib/eal/x86/include/rte_atomic.h | 172 ----------------------
> lib/eal/x86/include/rte_atomic_32.h | 188 -------------------------
> lib/eal/x86/include/rte_atomic_64.h | 157 ---------------------
> 9 files changed, 6 insertions(+), 776 deletions(-)
>
> diff --git a/lib/eal/arm/include/rte_atomic_32.h
> b/lib/eal/arm/include/rte_atomic_32.h
> index 0b9a0dfa30..696a539fef 100644
> --- a/lib/eal/arm/include/rte_atomic_32.h
> +++ b/lib/eal/arm/include/rte_atomic_32.h
> @@ -5,10 +5,6 @@
> #ifndef _RTE_ATOMIC_ARM32_H_
> #define _RTE_ATOMIC_ARM32_H_
>
> -#ifndef RTE_FORCE_INTRINSICS
> -# error Platform must be built with RTE_FORCE_INTRINSICS
> -#endif
> -
> #include "generic/rte_atomic.h"
>
> #ifdef __cplusplus
> diff --git a/lib/eal/arm/include/rte_atomic_64.h
> b/lib/eal/arm/include/rte_atomic_64.h
> index 181bb60929..9f790238df 100644
> --- a/lib/eal/arm/include/rte_atomic_64.h
> +++ b/lib/eal/arm/include/rte_atomic_64.h
> @@ -6,10 +6,6 @@
> #ifndef _RTE_ATOMIC_ARM64_H_
> #define _RTE_ATOMIC_ARM64_H_
>
> -#ifndef RTE_FORCE_INTRINSICS
> -# error Platform must be built with RTE_FORCE_INTRINSICS
> -#endif
> -
> #include "generic/rte_atomic.h"
> #include <rte_branch_prediction.h>
> #include <rte_debug.h>
> diff --git a/lib/eal/include/generic/rte_atomic.h
> b/lib/eal/include/generic/rte_atomic.h
> index 0a4f3f8528..292e52fade 100644
> --- a/lib/eal/include/generic/rte_atomic.h
> +++ b/lib/eal/include/generic/rte_atomic.h
> @@ -187,13 +187,11 @@ static inline void
> rte_atomic_thread_fence(rte_memory_order memorder);
> static inline int
> rte_atomic16_cmpset(volatile uint16_t *dst, uint16_t exp, uint16_t src);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline int
> rte_atomic16_cmpset(volatile uint16_t *dst, uint16_t exp, uint16_t src)
> {
> return __sync_bool_compare_and_swap(dst, exp, src);
> }
> -#endif
>
> /**
> * Atomic exchange.
> @@ -211,15 +209,11 @@ rte_atomic16_cmpset(volatile uint16_t *dst, uint16_t
> exp, uint16_t src)
> * The original value at that location
> */
> static inline uint16_t
> -rte_atomic16_exchange(volatile uint16_t *dst, uint16_t val);
> -
> -#ifdef RTE_FORCE_INTRINSICS
> -static inline uint16_t
> rte_atomic16_exchange(volatile uint16_t *dst, uint16_t val)
> {
> - return rte_atomic_exchange_explicit(dst, val,
> rte_memory_order_seq_cst);
> + return rte_atomic_exchange_explicit((volatile __rte_atomic uint16_t
> *)dst,
> + val, rte_memory_order_seq_cst);
> }
> -#endif
>
> /**
> * The atomic counter structure.
> @@ -312,13 +306,11 @@ rte_atomic16_sub(rte_atomic16_t *v, int16_t dec)
> static inline void
> rte_atomic16_inc(rte_atomic16_t *v);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline void
> rte_atomic16_inc(rte_atomic16_t *v)
> {
> rte_atomic16_add(v, 1);
> }
> -#endif
>
> /**
> * Atomically decrement a counter by one.
> @@ -329,13 +321,11 @@ rte_atomic16_inc(rte_atomic16_t *v)
> static inline void
> rte_atomic16_dec(rte_atomic16_t *v);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline void
> rte_atomic16_dec(rte_atomic16_t *v)
> {
> rte_atomic16_sub(v, 1);
> }
> -#endif
>
> /**
> * Atomically add a 16-bit value to a counter and return the result.
> @@ -391,13 +381,11 @@ rte_atomic16_sub_return(rte_atomic16_t *v, int16_t
> dec)
> */
> static inline int rte_atomic16_inc_and_test(rte_atomic16_t *v);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline int rte_atomic16_inc_and_test(rte_atomic16_t *v)
> {
> return rte_atomic_fetch_add_explicit((volatile __rte_atomic int16_t *)&v-
> >cnt, 1,
> rte_memory_order_seq_cst) + 1 == 0;
> }
> -#endif
>
> /**
> * Atomically decrement a 16-bit counter by one and test.
> @@ -412,13 +400,11 @@ static inline int
> rte_atomic16_inc_and_test(rte_atomic16_t *v)
> */
> static inline int rte_atomic16_dec_and_test(rte_atomic16_t *v);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline int rte_atomic16_dec_and_test(rte_atomic16_t *v)
> {
> return rte_atomic_fetch_sub_explicit((volatile __rte_atomic int16_t *)&v-
> >cnt, 1,
> rte_memory_order_seq_cst) - 1 == 0;
> }
> -#endif
>
> /**
> * Atomically test and set a 16-bit atomic counter.
> @@ -433,12 +419,10 @@ static inline int
> rte_atomic16_dec_and_test(rte_atomic16_t *v)
> */
> static inline int rte_atomic16_test_and_set(rte_atomic16_t *v);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline int rte_atomic16_test_and_set(rte_atomic16_t *v)
> {
> return rte_atomic16_cmpset((volatile uint16_t *)&v->cnt, 0, 1);
> }
> -#endif
>
> /**
> * Atomically set a 16-bit counter to 0.
> @@ -472,13 +456,11 @@ static inline void rte_atomic16_clear(rte_atomic16_t
> *v)
> static inline int
> rte_atomic32_cmpset(volatile uint32_t *dst, uint32_t exp, uint32_t src);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline int
> rte_atomic32_cmpset(volatile uint32_t *dst, uint32_t exp, uint32_t src)
> {
> return __sync_bool_compare_and_swap(dst, exp, src);
> }
> -#endif
>
> /**
> * Atomic exchange.
> @@ -496,15 +478,11 @@ rte_atomic32_cmpset(volatile uint32_t *dst, uint32_t
> exp, uint32_t src)
> * The original value at that location
> */
> static inline uint32_t
> -rte_atomic32_exchange(volatile uint32_t *dst, uint32_t val);
> -
> -#ifdef RTE_FORCE_INTRINSICS
> -static inline uint32_t
> rte_atomic32_exchange(volatile uint32_t *dst, uint32_t val)
> {
> - return rte_atomic_exchange_explicit(dst, val,
> rte_memory_order_seq_cst);
> + return rte_atomic_exchange_explicit((volatile __rte_atomic uint32_t
> *)dst,
> + val, rte_memory_order_seq_cst);
> }
> -#endif
>
> /**
> * The atomic counter structure.
> @@ -597,13 +575,11 @@ rte_atomic32_sub(rte_atomic32_t *v, int32_t dec)
> static inline void
> rte_atomic32_inc(rte_atomic32_t *v);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline void
> rte_atomic32_inc(rte_atomic32_t *v)
> {
> rte_atomic32_add(v, 1);
> }
> -#endif
>
> /**
> * Atomically decrement a counter by one.
> @@ -614,13 +590,11 @@ rte_atomic32_inc(rte_atomic32_t *v)
> static inline void
> rte_atomic32_dec(rte_atomic32_t *v);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline void
> rte_atomic32_dec(rte_atomic32_t *v)
> {
> rte_atomic32_sub(v,1);
> }
> -#endif
>
> /**
> * Atomically add a 32-bit value to a counter and return the result.
> @@ -676,13 +650,11 @@ rte_atomic32_sub_return(rte_atomic32_t *v, int32_t
> dec)
> */
> static inline int rte_atomic32_inc_and_test(rte_atomic32_t *v);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline int rte_atomic32_inc_and_test(rte_atomic32_t *v)
> {
> return rte_atomic_fetch_add_explicit((volatile __rte_atomic int32_t *)&v-
> >cnt, 1,
> rte_memory_order_seq_cst) + 1 == 0;
> }
> -#endif
>
> /**
> * Atomically decrement a 32-bit counter by one and test.
> @@ -697,13 +669,11 @@ static inline int
> rte_atomic32_inc_and_test(rte_atomic32_t *v)
> */
> static inline int rte_atomic32_dec_and_test(rte_atomic32_t *v);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline int rte_atomic32_dec_and_test(rte_atomic32_t *v)
> {
> return rte_atomic_fetch_sub_explicit((volatile __rte_atomic int32_t *)&v-
> >cnt, 1,
> rte_memory_order_seq_cst) - 1 == 0;
> }
> -#endif
>
> /**
> * Atomically test and set a 32-bit atomic counter.
> @@ -718,12 +688,10 @@ static inline int
> rte_atomic32_dec_and_test(rte_atomic32_t *v)
> */
> static inline int rte_atomic32_test_and_set(rte_atomic32_t *v);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline int rte_atomic32_test_and_set(rte_atomic32_t *v)
> {
> return rte_atomic32_cmpset((volatile uint32_t *)&v->cnt, 0, 1);
> }
> -#endif
>
> /**
> * Atomically set a 32-bit counter to 0.
> @@ -756,13 +724,11 @@ static inline void rte_atomic32_clear(rte_atomic32_t
> *v)
> static inline int
> rte_atomic64_cmpset(volatile uint64_t *dst, uint64_t exp, uint64_t src);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline int
> rte_atomic64_cmpset(volatile uint64_t *dst, uint64_t exp, uint64_t src)
> {
> return __sync_bool_compare_and_swap(dst, exp, src);
> }
> -#endif
>
> /**
> * Atomic exchange.
> @@ -780,15 +746,11 @@ rte_atomic64_cmpset(volatile uint64_t *dst, uint64_t
> exp, uint64_t src)
> * The original value at that location
> */
> static inline uint64_t
> -rte_atomic64_exchange(volatile uint64_t *dst, uint64_t val);
> -
> -#ifdef RTE_FORCE_INTRINSICS
> -static inline uint64_t
> rte_atomic64_exchange(volatile uint64_t *dst, uint64_t val)
> {
> - return rte_atomic_exchange_explicit(dst, val,
> rte_memory_order_seq_cst);
> + return rte_atomic_exchange_explicit((volatile __rte_atomic uint64_t
> *)dst,
> + val, rte_memory_order_seq_cst);
> }
> -#endif
>
> /**
> * The atomic counter structure.
> @@ -811,7 +773,6 @@ typedef struct {
> static inline void
> rte_atomic64_init(rte_atomic64_t *v);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline void
> rte_atomic64_init(rte_atomic64_t *v)
> {
> @@ -828,7 +789,6 @@ rte_atomic64_init(rte_atomic64_t *v)
> }
> #endif
> }
> -#endif
>
> /**
> * Atomically read a 64-bit counter.
> @@ -841,7 +801,6 @@ rte_atomic64_init(rte_atomic64_t *v)
> static inline int64_t
> rte_atomic64_read(rte_atomic64_t *v);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline int64_t
> rte_atomic64_read(rte_atomic64_t *v)
> {
> @@ -860,7 +819,6 @@ rte_atomic64_read(rte_atomic64_t *v)
> return tmp;
> #endif
> }
> -#endif
>
> /**
> * Atomically set a 64-bit counter.
> @@ -873,7 +831,6 @@ rte_atomic64_read(rte_atomic64_t *v)
> static inline void
> rte_atomic64_set(rte_atomic64_t *v, int64_t new_value);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline void
> rte_atomic64_set(rte_atomic64_t *v, int64_t new_value)
> {
> @@ -890,7 +847,6 @@ rte_atomic64_set(rte_atomic64_t *v, int64_t new_value)
> }
> #endif
> }
> -#endif
>
> /**
> * Atomically add a 64-bit value to a counter.
> @@ -903,14 +859,12 @@ rte_atomic64_set(rte_atomic64_t *v, int64_t
> new_value)
> static inline void
> rte_atomic64_add(rte_atomic64_t *v, int64_t inc);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline void
> rte_atomic64_add(rte_atomic64_t *v, int64_t inc)
> {
> rte_atomic_fetch_add_explicit((volatile __rte_atomic int64_t *)&v->cnt,
> inc,
> rte_memory_order_seq_cst);
> }
> -#endif
>
> /**
> * Atomically subtract a 64-bit value from a counter.
> @@ -923,14 +877,12 @@ rte_atomic64_add(rte_atomic64_t *v, int64_t inc)
> static inline void
> rte_atomic64_sub(rte_atomic64_t *v, int64_t dec);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline void
> rte_atomic64_sub(rte_atomic64_t *v, int64_t dec)
> {
> rte_atomic_fetch_sub_explicit((volatile __rte_atomic int64_t *)&v->cnt,
> dec,
> rte_memory_order_seq_cst);
> }
> -#endif
>
> /**
> * Atomically increment a 64-bit counter by one and test.
> @@ -941,13 +893,11 @@ rte_atomic64_sub(rte_atomic64_t *v, int64_t dec)
> static inline void
> rte_atomic64_inc(rte_atomic64_t *v);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline void
> rte_atomic64_inc(rte_atomic64_t *v)
> {
> rte_atomic64_add(v, 1);
> }
> -#endif
>
> /**
> * Atomically decrement a 64-bit counter by one and test.
> @@ -958,13 +908,11 @@ rte_atomic64_inc(rte_atomic64_t *v)
> static inline void
> rte_atomic64_dec(rte_atomic64_t *v);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline void
> rte_atomic64_dec(rte_atomic64_t *v)
> {
> rte_atomic64_sub(v, 1);
> }
> -#endif
>
> /**
> * Add a 64-bit value to an atomic counter and return the result.
> @@ -982,14 +930,12 @@ rte_atomic64_dec(rte_atomic64_t *v)
> static inline int64_t
> rte_atomic64_add_return(rte_atomic64_t *v, int64_t inc);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline int64_t
> rte_atomic64_add_return(rte_atomic64_t *v, int64_t inc)
> {
> return rte_atomic_fetch_add_explicit((volatile __rte_atomic int64_t *)&v-
> >cnt, inc,
> rte_memory_order_seq_cst) + inc;
> }
> -#endif
>
> /**
> * Subtract a 64-bit value from an atomic counter and return the result.
> @@ -1007,14 +953,12 @@ rte_atomic64_add_return(rte_atomic64_t *v, int64_t
> inc)
> static inline int64_t
> rte_atomic64_sub_return(rte_atomic64_t *v, int64_t dec);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline int64_t
> rte_atomic64_sub_return(rte_atomic64_t *v, int64_t dec)
> {
> return rte_atomic_fetch_sub_explicit((volatile __rte_atomic int64_t *)&v-
> >cnt, dec,
> rte_memory_order_seq_cst) - dec;
> }
> -#endif
>
> /**
> * Atomically increment a 64-bit counter by one and test.
> @@ -1029,12 +973,10 @@ rte_atomic64_sub_return(rte_atomic64_t *v, int64_t
> dec)
> */
> static inline int rte_atomic64_inc_and_test(rte_atomic64_t *v);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline int rte_atomic64_inc_and_test(rte_atomic64_t *v)
> {
> return rte_atomic64_add_return(v, 1) == 0;
> }
> -#endif
>
> /**
> * Atomically decrement a 64-bit counter by one and test.
> @@ -1049,12 +991,10 @@ static inline int
> rte_atomic64_inc_and_test(rte_atomic64_t *v)
> */
> static inline int rte_atomic64_dec_and_test(rte_atomic64_t *v);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline int rte_atomic64_dec_and_test(rte_atomic64_t *v)
> {
> return rte_atomic64_sub_return(v, 1) == 0;
> }
> -#endif
>
> /**
> * Atomically test and set a 64-bit atomic counter.
> @@ -1069,12 +1009,10 @@ static inline int
> rte_atomic64_dec_and_test(rte_atomic64_t *v)
> */
> static inline int rte_atomic64_test_and_set(rte_atomic64_t *v);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline int rte_atomic64_test_and_set(rte_atomic64_t *v)
> {
> return rte_atomic64_cmpset((volatile uint64_t *)&v->cnt, 0, 1);
> }
> -#endif
>
> /**
> * Atomically set a 64-bit counter to 0.
> @@ -1084,12 +1022,10 @@ static inline int
> rte_atomic64_test_and_set(rte_atomic64_t *v)
> */
> static inline void rte_atomic64_clear(rte_atomic64_t *v);
>
> -#ifdef RTE_FORCE_INTRINSICS
> static inline void rte_atomic64_clear(rte_atomic64_t *v)
> {
> rte_atomic64_set(v, 0);
> }
> -#endif
>
> #endif
>
> diff --git a/lib/eal/loongarch/include/rte_atomic.h
> b/lib/eal/loongarch/include/rte_atomic.h
> index c8066a4612..785a452c9e 100644
> --- a/lib/eal/loongarch/include/rte_atomic.h
> +++ b/lib/eal/loongarch/include/rte_atomic.h
> @@ -5,10 +5,6 @@
> #ifndef RTE_ATOMIC_LOONGARCH_H
> #define RTE_ATOMIC_LOONGARCH_H
>
> -#ifndef RTE_FORCE_INTRINSICS
> -# error Platform must be built with RTE_FORCE_INTRINSICS
> -#endif
> -
> #include <rte_common.h>
> #include "generic/rte_atomic.h"
>
> diff --git a/lib/eal/ppc/include/rte_atomic.h b/lib/eal/ppc/include/rte_atomic.h
> index 10acc238f9..64f4c3d670 100644
> --- a/lib/eal/ppc/include/rte_atomic.h
> +++ b/lib/eal/ppc/include/rte_atomic.h
> @@ -43,179 +43,6 @@ rte_atomic_thread_fence(rte_memory_order memorder)
> }
>
> /*------------------------- 16 bit atomic operations -------------------------*/
> -#ifndef RTE_FORCE_INTRINSICS
> -static inline int
> -rte_atomic16_cmpset(volatile uint16_t *dst, uint16_t exp, uint16_t src)
> -{
> - return rte_atomic_compare_exchange_strong_explicit(dst, &exp, src,
> rte_memory_order_acquire,
> - rte_memory_order_acquire) ? 1 : 0;
> -}
> -
> -static inline int rte_atomic16_test_and_set(rte_atomic16_t *v)
> -{
> - return rte_atomic16_cmpset((volatile uint16_t *)&v->cnt, 0, 1);
> -}
> -
> -static inline void
> -rte_atomic16_inc(rte_atomic16_t *v)
> -{
> - rte_atomic_fetch_add_explicit(&v->cnt, 1, rte_memory_order_acquire);
> -}
> -
> -static inline void
> -rte_atomic16_dec(rte_atomic16_t *v)
> -{
> - rte_atomic_fetch_sub_explicit(&v->cnt, 1, rte_memory_order_acquire);
> -}
> -
> -static inline int rte_atomic16_inc_and_test(rte_atomic16_t *v)
> -{
> - return rte_atomic_fetch_add_explicit(&v->cnt, 1,
> rte_memory_order_acquire) + 1 == 0;
> -}
> -
> -static inline int rte_atomic16_dec_and_test(rte_atomic16_t *v)
> -{
> - return rte_atomic_fetch_sub_explicit(&v->cnt, 1,
> rte_memory_order_acquire) - 1 == 0;
> -}
> -
> -static inline uint16_t
> -rte_atomic16_exchange(volatile uint16_t *dst, uint16_t val)
> -{
> - return __atomic_exchange_2(dst, val, rte_memory_order_seq_cst);
> -}
> -
> -/*------------------------- 32 bit atomic operations -------------------------*/
> -
> -static inline int
> -rte_atomic32_cmpset(volatile uint32_t *dst, uint32_t exp, uint32_t src)
> -{
> - return rte_atomic_compare_exchange_strong_explicit(dst, &exp, src,
> rte_memory_order_acquire,
> - rte_memory_order_acquire) ? 1 : 0;
> -}
> -
> -static inline int rte_atomic32_test_and_set(rte_atomic32_t *v)
> -{
> - return rte_atomic32_cmpset((volatile uint32_t *)&v->cnt, 0, 1);
> -}
> -
> -static inline void
> -rte_atomic32_inc(rte_atomic32_t *v)
> -{
> - rte_atomic_fetch_add_explicit(&v->cnt, 1, rte_memory_order_acquire);
> -}
> -
> -static inline void
> -rte_atomic32_dec(rte_atomic32_t *v)
> -{
> - rte_atomic_fetch_sub_explicit(&v->cnt, 1, rte_memory_order_acquire);
> -}
> -
> -static inline int rte_atomic32_inc_and_test(rte_atomic32_t *v)
> -{
> - return rte_atomic_fetch_add_explicit(&v->cnt, 1,
> rte_memory_order_acquire) + 1 == 0;
> -}
> -
> -static inline int rte_atomic32_dec_and_test(rte_atomic32_t *v)
> -{
> - return rte_atomic_fetch_sub_explicit(&v->cnt, 1,
> rte_memory_order_acquire) - 1 == 0;
> -}
> -
> -static inline uint32_t
> -rte_atomic32_exchange(volatile uint32_t *dst, uint32_t val)
> -{
> - return __atomic_exchange_4(dst, val, rte_memory_order_seq_cst);
> -}
> -
> -/*------------------------- 64 bit atomic operations -------------------------*/
> -
> -static inline int
> -rte_atomic64_cmpset(volatile uint64_t *dst, uint64_t exp, uint64_t src)
> -{
> - return rte_atomic_compare_exchange_strong_explicit(dst, &exp, src,
> rte_memory_order_acquire,
> - rte_memory_order_acquire) ? 1 : 0;
> -}
> -
> -static inline void
> -rte_atomic64_init(rte_atomic64_t *v)
> -{
> - v->cnt = 0;
> -}
> -
> -static inline int64_t
> -rte_atomic64_read(rte_atomic64_t *v)
> -{
> - return v->cnt;
> -}
> -
> -static inline void
> -rte_atomic64_set(rte_atomic64_t *v, int64_t new_value)
> -{
> - v->cnt = new_value;
> -}
> -
> -static inline void
> -rte_atomic64_add(rte_atomic64_t *v, int64_t inc)
> -{
> - rte_atomic_fetch_add_explicit(&v->cnt, inc, rte_memory_order_acquire);
> -}
> -
> -static inline void
> -rte_atomic64_sub(rte_atomic64_t *v, int64_t dec)
> -{
> - rte_atomic_fetch_sub_explicit(&v->cnt, dec,
> rte_memory_order_acquire);
> -}
> -
> -static inline void
> -rte_atomic64_inc(rte_atomic64_t *v)
> -{
> - rte_atomic_fetch_add_explicit(&v->cnt, 1, rte_memory_order_acquire);
> -}
> -
> -static inline void
> -rte_atomic64_dec(rte_atomic64_t *v)
> -{
> - rte_atomic_fetch_sub_explicit(&v->cnt, 1, rte_memory_order_acquire);
> -}
> -
> -static inline int64_t
> -rte_atomic64_add_return(rte_atomic64_t *v, int64_t inc)
> -{
> - return rte_atomic_fetch_add_explicit(&v->cnt, inc,
> rte_memory_order_acquire) + inc;
> -}
> -
> -static inline int64_t
> -rte_atomic64_sub_return(rte_atomic64_t *v, int64_t dec)
> -{
> - return rte_atomic_fetch_sub_explicit(&v->cnt, dec,
> rte_memory_order_acquire) - dec;
> -}
> -
> -static inline int rte_atomic64_inc_and_test(rte_atomic64_t *v)
> -{
> - return rte_atomic_fetch_add_explicit(&v->cnt, 1,
> rte_memory_order_acquire) + 1 == 0;
> -}
> -
> -static inline int rte_atomic64_dec_and_test(rte_atomic64_t *v)
> -{
> - return rte_atomic_fetch_sub_explicit(&v->cnt, 1,
> rte_memory_order_acquire) - 1 == 0;
> -}
> -
> -static inline int rte_atomic64_test_and_set(rte_atomic64_t *v)
> -{
> - return rte_atomic64_cmpset((volatile uint64_t *)&v->cnt, 0, 1);
> -}
> -
> -static inline void rte_atomic64_clear(rte_atomic64_t *v)
> -{
> - v->cnt = 0;
> -}
> -
> -static inline uint64_t
> -rte_atomic64_exchange(volatile uint64_t *dst, uint64_t val)
> -{
> - return __atomic_exchange_8(dst, val, rte_memory_order_seq_cst);
> -}
> -
> -#endif
>
> #ifdef __cplusplus
> }
> diff --git a/lib/eal/riscv/include/rte_atomic.h b/lib/eal/riscv/include/rte_atomic.h
> index 66346ad474..061b175f33 100644
> --- a/lib/eal/riscv/include/rte_atomic.h
> +++ b/lib/eal/riscv/include/rte_atomic.h
> @@ -8,10 +8,6 @@
> #ifndef RTE_ATOMIC_RISCV_H
> #define RTE_ATOMIC_RISCV_H
>
> -#ifndef RTE_FORCE_INTRINSICS
> -# error Platform must be built with RTE_FORCE_INTRINSICS
> -#endif
> -
> #include <stdint.h>
> #include <rte_common.h>
> #include <rte_config.h>
> diff --git a/lib/eal/x86/include/rte_atomic.h b/lib/eal/x86/include/rte_atomic.h
> index e071e4234e..4f05302c9f 100644
> --- a/lib/eal/x86/include/rte_atomic.h
> +++ b/lib/eal/x86/include/rte_atomic.h
> @@ -111,178 +111,6 @@ rte_atomic_thread_fence(rte_memory_order
> memorder)
> extern "C" {
> #endif
>
> -#ifndef RTE_FORCE_INTRINSICS
> -static inline int
> -rte_atomic16_cmpset(volatile uint16_t *dst, uint16_t exp, uint16_t src)
> -{
> - uint8_t res;
> -
> - asm volatile(
> - MPLOCKED
> - "cmpxchgw %[src], %[dst];"
> - "sete %[res];"
> - : [res] "=a" (res), /* output */
> - [dst] "=m" (*dst)
> - : [src] "r" (src), /* input */
> - "a" (exp),
> - "m" (*dst)
> - : "memory"); /* no-clobber list */
> - return res;
> -}
> -
> -static inline uint16_t
> -rte_atomic16_exchange(volatile uint16_t *dst, uint16_t val)
> -{
> - asm volatile(
> - MPLOCKED
> - "xchgw %0, %1;"
> - : "=r" (val), "=m" (*dst)
> - : "0" (val), "m" (*dst)
> - : "memory"); /* no-clobber list */
> - return val;
> -}
> -
> -static inline int rte_atomic16_test_and_set(rte_atomic16_t *v)
> -{
> - return rte_atomic16_cmpset((volatile uint16_t *)&v->cnt, 0, 1);
> -}
> -
> -static inline void
> -rte_atomic16_inc(rte_atomic16_t *v)
> -{
> - asm volatile(
> - MPLOCKED
> - "incw %[cnt]"
> - : [cnt] "=m" (v->cnt) /* output */
> - : "m" (v->cnt) /* input */
> - );
> -}
> -
> -static inline void
> -rte_atomic16_dec(rte_atomic16_t *v)
> -{
> - asm volatile(
> - MPLOCKED
> - "decw %[cnt]"
> - : [cnt] "=m" (v->cnt) /* output */
> - : "m" (v->cnt) /* input */
> - );
> -}
> -
> -static inline int rte_atomic16_inc_and_test(rte_atomic16_t *v)
> -{
> - uint8_t ret;
> -
> - asm volatile(
> - MPLOCKED
> - "incw %[cnt] ; "
> - "sete %[ret]"
> - : [cnt] "+m" (v->cnt), /* output */
> - [ret] "=qm" (ret)
> - );
> - return ret != 0;
> -}
> -
> -static inline int rte_atomic16_dec_and_test(rte_atomic16_t *v)
> -{
> - uint8_t ret;
> -
> - asm volatile(MPLOCKED
> - "decw %[cnt] ; "
> - "sete %[ret]"
> - : [cnt] "+m" (v->cnt), /* output */
> - [ret] "=qm" (ret)
> - );
> - return ret != 0;
> -}
> -
> -/*------------------------- 32 bit atomic operations -------------------------*/
> -
> -static inline int
> -rte_atomic32_cmpset(volatile uint32_t *dst, uint32_t exp, uint32_t src)
> -{
> - uint8_t res;
> -
> - asm volatile(
> - MPLOCKED
> - "cmpxchgl %[src], %[dst];"
> - "sete %[res];"
> - : [res] "=a" (res), /* output */
> - [dst] "=m" (*dst)
> - : [src] "r" (src), /* input */
> - "a" (exp),
> - "m" (*dst)
> - : "memory"); /* no-clobber list */
> - return res;
> -}
> -
> -static inline uint32_t
> -rte_atomic32_exchange(volatile uint32_t *dst, uint32_t val)
> -{
> - asm volatile(
> - MPLOCKED
> - "xchgl %0, %1;"
> - : "=r" (val), "=m" (*dst)
> - : "0" (val), "m" (*dst)
> - : "memory"); /* no-clobber list */
> - return val;
> -}
> -
> -static inline int rte_atomic32_test_and_set(rte_atomic32_t *v)
> -{
> - return rte_atomic32_cmpset((volatile uint32_t *)&v->cnt, 0, 1);
> -}
> -
> -static inline void
> -rte_atomic32_inc(rte_atomic32_t *v)
> -{
> - asm volatile(
> - MPLOCKED
> - "incl %[cnt]"
> - : [cnt] "=m" (v->cnt) /* output */
> - : "m" (v->cnt) /* input */
> - );
> -}
> -
> -static inline void
> -rte_atomic32_dec(rte_atomic32_t *v)
> -{
> - asm volatile(
> - MPLOCKED
> - "decl %[cnt]"
> - : [cnt] "=m" (v->cnt) /* output */
> - : "m" (v->cnt) /* input */
> - );
> -}
> -
> -static inline int rte_atomic32_inc_and_test(rte_atomic32_t *v)
> -{
> - uint8_t ret;
> -
> - asm volatile(
> - MPLOCKED
> - "incl %[cnt] ; "
> - "sete %[ret]"
> - : [cnt] "+m" (v->cnt), /* output */
> - [ret] "=qm" (ret)
> - );
> - return ret != 0;
> -}
> -
> -static inline int rte_atomic32_dec_and_test(rte_atomic32_t *v)
> -{
> - uint8_t ret;
> -
> - asm volatile(MPLOCKED
> - "decl %[cnt] ; "
> - "sete %[ret]"
> - : [cnt] "+m" (v->cnt), /* output */
> - [ret] "=qm" (ret)
> - );
> - return ret != 0;
> -}
> -
> -#endif /* !RTE_FORCE_INTRINSICS */
>
> #ifdef __cplusplus
> }
> diff --git a/lib/eal/x86/include/rte_atomic_32.h
> b/lib/eal/x86/include/rte_atomic_32.h
> index 0f25863aa5..37d139f30d 100644
> --- a/lib/eal/x86/include/rte_atomic_32.h
> +++ b/lib/eal/x86/include/rte_atomic_32.h
> @@ -20,193 +20,5 @@
>
> /*------------------------- 64 bit atomic operations -------------------------*/
>
> -#ifndef RTE_FORCE_INTRINSICS
> -static inline int
> -rte_atomic64_cmpset(volatile uint64_t *dst, uint64_t exp, uint64_t src)
> -{
> - uint8_t res;
> - union {
> - struct {
> - uint32_t l32;
> - uint32_t h32;
> - };
> - uint64_t u64;
> - } _exp, _src;
> -
> - _exp.u64 = exp;
> - _src.u64 = src;
> -
> -#ifndef __PIC__
> - asm volatile (
> - MPLOCKED
> - "cmpxchg8b (%[dst]);"
> - "setz %[res];"
> - : [res] "=a" (res) /* result in eax */
> - : [dst] "S" (dst), /* esi */
> - "b" (_src.l32), /* ebx */
> - "c" (_src.h32), /* ecx */
> - "a" (_exp.l32), /* eax */
> - "d" (_exp.h32) /* edx */
> - : "memory" ); /* no-clobber list */
> -#else
> - asm volatile (
> - "xchgl %%ebx, %%edi;\n"
> - MPLOCKED
> - "cmpxchg8b (%[dst]);"
> - "setz %[res];"
> - "xchgl %%ebx, %%edi;\n"
> - : [res] "=a" (res) /* result in eax */
> - : [dst] "S" (dst), /* esi */
> - "D" (_src.l32), /* ebx */
> - "c" (_src.h32), /* ecx */
> - "a" (_exp.l32), /* eax */
> - "d" (_exp.h32) /* edx */
> - : "memory" ); /* no-clobber list */
> -#endif
> -
> - return res;
> -}
> -
> -static inline uint64_t
> -rte_atomic64_exchange(volatile uint64_t *dest, uint64_t val)
> -{
> - uint64_t old;
> -
> - do {
> - old = *dest;
> - } while (rte_atomic64_cmpset(dest, old, val) == 0);
> -
> - return old;
> -}
> -
> -static inline void
> -rte_atomic64_init(rte_atomic64_t *v)
> -{
> - int success = 0;
> - uint64_t tmp;
> -
> - while (success == 0) {
> - tmp = v->cnt;
> - success = rte_atomic64_cmpset((volatile uint64_t *)&v->cnt,
> - tmp, 0);
> - }
> -}
> -
> -static inline int64_t
> -rte_atomic64_read(rte_atomic64_t *v)
> -{
> - int success = 0;
> - uint64_t tmp;
> -
> - while (success == 0) {
> - tmp = v->cnt;
> - /* replace the value by itself */
> - success = rte_atomic64_cmpset((volatile uint64_t *)&v->cnt,
> - tmp, tmp);
> - }
> - return tmp;
> -}
> -
> -static inline void
> -rte_atomic64_set(rte_atomic64_t *v, int64_t new_value)
> -{
> - int success = 0;
> - uint64_t tmp;
> -
> - while (success == 0) {
> - tmp = v->cnt;
> - success = rte_atomic64_cmpset((volatile uint64_t *)&v->cnt,
> - tmp, new_value);
> - }
> -}
> -
> -static inline void
> -rte_atomic64_add(rte_atomic64_t *v, int64_t inc)
> -{
> - int success = 0;
> - uint64_t tmp;
> -
> - while (success == 0) {
> - tmp = v->cnt;
> - success = rte_atomic64_cmpset((volatile uint64_t *)&v->cnt,
> - tmp, tmp + inc);
> - }
> -}
> -
> -static inline void
> -rte_atomic64_sub(rte_atomic64_t *v, int64_t dec)
> -{
> - int success = 0;
> - uint64_t tmp;
> -
> - while (success == 0) {
> - tmp = v->cnt;
> - success = rte_atomic64_cmpset((volatile uint64_t *)&v->cnt,
> - tmp, tmp - dec);
> - }
> -}
> -
> -static inline void
> -rte_atomic64_inc(rte_atomic64_t *v)
> -{
> - rte_atomic64_add(v, 1);
> -}
> -
> -static inline void
> -rte_atomic64_dec(rte_atomic64_t *v)
> -{
> - rte_atomic64_sub(v, 1);
> -}
> -
> -static inline int64_t
> -rte_atomic64_add_return(rte_atomic64_t *v, int64_t inc)
> -{
> - int success = 0;
> - uint64_t tmp;
> -
> - while (success == 0) {
> - tmp = v->cnt;
> - success = rte_atomic64_cmpset((volatile uint64_t *)&v->cnt,
> - tmp, tmp + inc);
> - }
> -
> - return tmp + inc;
> -}
> -
> -static inline int64_t
> -rte_atomic64_sub_return(rte_atomic64_t *v, int64_t dec)
> -{
> - int success = 0;
> - uint64_t tmp;
> -
> - while (success == 0) {
> - tmp = v->cnt;
> - success = rte_atomic64_cmpset((volatile uint64_t *)&v->cnt,
> - tmp, tmp - dec);
> - }
> -
> - return tmp - dec;
> -}
> -
> -static inline int rte_atomic64_inc_and_test(rte_atomic64_t *v)
> -{
> - return rte_atomic64_add_return(v, 1) == 0;
> -}
> -
> -static inline int rte_atomic64_dec_and_test(rte_atomic64_t *v)
> -{
> - return rte_atomic64_sub_return(v, 1) == 0;
> -}
> -
> -static inline int rte_atomic64_test_and_set(rte_atomic64_t *v)
> -{
> - return rte_atomic64_cmpset((volatile uint64_t *)&v->cnt, 0, 1);
> -}
> -
> -static inline void rte_atomic64_clear(rte_atomic64_t *v)
> -{
> - rte_atomic64_set(v, 0);
> -}
> -#endif
>
> #endif /* _RTE_ATOMIC_I686_H_ */
> diff --git a/lib/eal/x86/include/rte_atomic_64.h
> b/lib/eal/x86/include/rte_atomic_64.h
> index 0a7a2131e0..1cd12695a2 100644
> --- a/lib/eal/x86/include/rte_atomic_64.h
> +++ b/lib/eal/x86/include/rte_atomic_64.h
> @@ -22,163 +22,6 @@
>
> /*------------------------- 64 bit atomic operations -------------------------*/
>
> -#ifndef RTE_FORCE_INTRINSICS
> -static inline int
> -rte_atomic64_cmpset(volatile uint64_t *dst, uint64_t exp, uint64_t src)
> -{
> - uint8_t res;
> -
> -
> - asm volatile(
> - MPLOCKED
> - "cmpxchgq %[src], %[dst];"
> - "sete %[res];"
> - : [res] "=a" (res), /* output */
> - [dst] "=m" (*dst)
> - : [src] "r" (src), /* input */
> - "a" (exp),
> - "m" (*dst)
> - : "memory"); /* no-clobber list */
> -
> - return res;
> -}
> -
> -static inline uint64_t
> -rte_atomic64_exchange(volatile uint64_t *dst, uint64_t val)
> -{
> - asm volatile(
> - MPLOCKED
> - "xchgq %0, %1;"
> - : "=r" (val), "=m" (*dst)
> - : "0" (val), "m" (*dst)
> - : "memory"); /* no-clobber list */
> - return val;
> -}
> -
> -static inline void
> -rte_atomic64_init(rte_atomic64_t *v)
> -{
> - v->cnt = 0;
> -}
> -
> -static inline int64_t
> -rte_atomic64_read(rte_atomic64_t *v)
> -{
> - return v->cnt;
> -}
> -
> -static inline void
> -rte_atomic64_set(rte_atomic64_t *v, int64_t new_value)
> -{
> - v->cnt = new_value;
> -}
> -
> -static inline void
> -rte_atomic64_add(rte_atomic64_t *v, int64_t inc)
> -{
> - asm volatile(
> - MPLOCKED
> - "addq %[inc], %[cnt]"
> - : [cnt] "=m" (v->cnt) /* output */
> - : [inc] "ir" (inc), /* input */
> - "m" (v->cnt)
> - );
> -}
> -
> -static inline void
> -rte_atomic64_sub(rte_atomic64_t *v, int64_t dec)
> -{
> - asm volatile(
> - MPLOCKED
> - "subq %[dec], %[cnt]"
> - : [cnt] "=m" (v->cnt) /* output */
> - : [dec] "ir" (dec), /* input */
> - "m" (v->cnt)
> - );
> -}
> -
> -static inline void
> -rte_atomic64_inc(rte_atomic64_t *v)
> -{
> - asm volatile(
> - MPLOCKED
> - "incq %[cnt]"
> - : [cnt] "=m" (v->cnt) /* output */
> - : "m" (v->cnt) /* input */
> - );
> -}
> -
> -static inline void
> -rte_atomic64_dec(rte_atomic64_t *v)
> -{
> - asm volatile(
> - MPLOCKED
> - "decq %[cnt]"
> - : [cnt] "=m" (v->cnt) /* output */
> - : "m" (v->cnt) /* input */
> - );
> -}
> -
> -static inline int64_t
> -rte_atomic64_add_return(rte_atomic64_t *v, int64_t inc)
> -{
> - int64_t prev = inc;
> -
> - asm volatile(
> - MPLOCKED
> - "xaddq %[prev], %[cnt]"
> - : [prev] "+r" (prev), /* output */
> - [cnt] "=m" (v->cnt)
> - : "m" (v->cnt) /* input */
> - );
> - return prev + inc;
> -}
> -
> -static inline int64_t
> -rte_atomic64_sub_return(rte_atomic64_t *v, int64_t dec)
> -{
> - return rte_atomic64_add_return(v, -dec);
> -}
> -
> -static inline int rte_atomic64_inc_and_test(rte_atomic64_t *v)
> -{
> - uint8_t ret;
> -
> - asm volatile(
> - MPLOCKED
> - "incq %[cnt] ; "
> - "sete %[ret]"
> - : [cnt] "+m" (v->cnt), /* output */
> - [ret] "=qm" (ret)
> - );
> -
> - return ret != 0;
> -}
> -
> -static inline int rte_atomic64_dec_and_test(rte_atomic64_t *v)
> -{
> - uint8_t ret;
> -
> - asm volatile(
> - MPLOCKED
> - "decq %[cnt] ; "
> - "sete %[ret]"
> - : [cnt] "+m" (v->cnt), /* output */
> - [ret] "=qm" (ret)
> - );
> - return ret != 0;
> -}
> -
> -static inline int rte_atomic64_test_and_set(rte_atomic64_t *v)
> -{
> - return rte_atomic64_cmpset((volatile uint64_t *)&v->cnt, 0, 1);
> -}
> -
> -static inline void rte_atomic64_clear(rte_atomic64_t *v)
> -{
> - v->cnt = 0;
> -}
> -#endif
>
> /*------------------------ 128 bit atomic operations -------------------------*/
>
> --
Acked-by: Konstantin Ananyev <konstantin.ananyev@huawei.com>
> 2.53.0
^ permalink raw reply
* RE: [PATCH v4 03/27] ring: unify memory model on C11, remove atomic32
From: Konstantin Ananyev @ 2026-06-01 18:18 UTC (permalink / raw)
To: Stephen Hemminger, dev@dpdk.org; +Cc: Wathsala Vithanage
In-Reply-To: <20260526232542.620966-4-stephen@networkplumber.org>
> Remove the RTE_USE_C11_MEM_MODEL build switch; C11 atomics are now
> the default for all platforms. Unifies __rte_ring_update_tail into
> the C11 form (atomic_store_release replaces the older rte_smp_wmb +
> plain store on the generic path) and renames rte_ring_generic_pvt.h
> to rte_ring_x86_pvt.h to reflect its new scope.
>
> Also splits the head-move helper into separate ST and MT variants,
> removing the runtime is_st branch from the MT retry loop.
> This gets small boost and scopes the following exception
> more tightly.
>
> Exception: on x86 with GCC, atomic_compare_exchange on the head CAS
> regresses MP/MC contended throughput by ~20% existing hand-written
> cmpxchg. As a workaround, GCC-on-x86 builds use the older
> __sync_bool_compare_and_swap builtin, which generates equivalent
> code to the original asm. Can be reverted if/when GCC gets
> fixed; similar issue was observed in Linux kernel.
>
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
> lib/ring/meson.build | 2 +-
> lib/ring/rte_ring_c11_pvt.h | 75 +++--------
> lib/ring/rte_ring_elem_pvt.h | 125 ++++++++++++++++--
> ..._ring_generic_pvt.h => rte_ring_x86_pvt.h} | 61 ++-------
> lib/ring/soring.c | 15 ++-
> 5 files changed, 158 insertions(+), 120 deletions(-)
> rename lib/ring/{rte_ring_generic_pvt.h => rte_ring_x86_pvt.h} (60%)
>
> diff --git a/lib/ring/meson.build b/lib/ring/meson.build
> index 21f2c12989..b178c963b8 100644
> --- a/lib/ring/meson.build
> +++ b/lib/ring/meson.build
> @@ -9,7 +9,7 @@ indirect_headers += files (
> 'rte_ring_elem.h',
> 'rte_ring_elem_pvt.h',
> 'rte_ring_c11_pvt.h',
> - 'rte_ring_generic_pvt.h',
> + 'rte_ring_x86_pvt.h',
> 'rte_ring_hts.h',
> 'rte_ring_hts_elem_pvt.h',
> 'rte_ring_peek.h',
> diff --git a/lib/ring/rte_ring_c11_pvt.h b/lib/ring/rte_ring_c11_pvt.h
> index 07b6efc416..3efe011f08 100644
> --- a/lib/ring/rte_ring_c11_pvt.h
> +++ b/lib/ring/rte_ring_c11_pvt.h
> @@ -15,35 +15,10 @@
> * @file rte_ring_c11_pvt.h
> * It is not recommended to include this file directly,
> * include <rte_ring.h> instead.
> - * Contains internal helper functions for MP/SP and MC/SC ring modes.
> + * Contains internal helper functions for MP and MC ring modes.
> * For more information please refer to <rte_ring.h>.
> */
>
> -/**
> - * @internal This function updates tail values.
> - */
> -static __rte_always_inline void
> -__rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t old_val,
> - uint32_t new_val, uint32_t single, uint32_t enqueue)
> -{
> - RTE_SET_USED(enqueue);
> -
> - /*
> - * If there are other enqueues/dequeues in progress that preceded us,
> - * we need to wait for them to complete
> - */
> - if (!single)
> - rte_wait_until_equal_32((uint32_t *)(uintptr_t)&ht->tail, old_val,
> - rte_memory_order_relaxed);
> -
> - /*
> - * R0: Establishes a synchronizing edge with load-acquire of tail at A1.
> - * Ensures that memory effects by this thread on ring elements array
> - * is observed by a different thread of the other type.
> - */
> - rte_atomic_store_explicit(&ht->tail, new_val,
> rte_memory_order_release);
> -}
> -
> /**
> * @internal This is a helper function that moves the producer/consumer head
> *
> @@ -72,14 +47,11 @@ __rte_ring_update_tail(struct rte_ring_headtail *ht,
> uint32_t old_val,
> * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only
> */
> static __rte_always_inline unsigned int
> -__rte_ring_headtail_move_head(struct rte_ring_headtail *d,
> +__rte_ring_headtail_move_head_mt(struct rte_ring_headtail *d,
> const struct rte_ring_headtail *s, uint32_t capacity,
> - unsigned int is_st, unsigned int n,
> - enum rte_ring_queue_behavior behavior,
> + unsigned int n, enum rte_ring_queue_behavior behavior,
> uint32_t *old_head, uint32_t *new_head, uint32_t *entries)
> {
> - uint32_t stail;
> - int success;
> unsigned int max = n;
>
> /*
> @@ -89,8 +61,7 @@ __rte_ring_headtail_move_head(struct rte_ring_headtail
> *d,
> * d->head.
> * If not, an unsafe partial order may ensue.
> */
> - *old_head = rte_atomic_load_explicit(&d->head,
> - rte_memory_order_acquire);
> + *old_head = rte_atomic_load_explicit(&d->head, rte_memory_order_acquire);
> do {
> /* Reset n to the initial burst count */
> n = max;
> @@ -101,15 +72,14 @@ __rte_ring_headtail_move_head(struct
> rte_ring_headtail *d,
> * ring elements array is observed by the time
> * this thread observes its tail update.
> */
> - stail = rte_atomic_load_explicit(&s->tail,
> - rte_memory_order_acquire);
> + uint32_t stail = rte_atomic_load_explicit(&s->tail,
> rte_memory_order_acquire);
>
> /* The subtraction is done between two unsigned 32bits value
> * (the result is always modulo 32 bits even if we have
> * *old_head > s->tail). So 'entries' is always between 0
> * and capacity (which is < size).
> */
> - *entries = (capacity + stail - *old_head);
> + *entries = capacity + stail - *old_head;
>
> /* check that we have enough room in ring */
> if (unlikely(n > *entries))
> @@ -120,25 +90,20 @@ __rte_ring_headtail_move_head(struct
> rte_ring_headtail *d,
> return 0;
>
> *new_head = *old_head + n;
> - if (is_st) {
> - d->head = *new_head;
> - success = 1;
> - } else
> - /* on failure, *old_head is updated */
> - /*
> - * R1/A2.
> - * R1: Establishes a synchronizing edge with A0 of a
> - * different thread.
> - * A2: Establishes a synchronizing edge with R1 of a
> - * different thread to observe same value for stail
> - * observed by that thread on CAS failure (to retry
> - * with an updated *old_head).
> - */
> - success =
> rte_atomic_compare_exchange_strong_explicit(
> - &d->head, old_head, *new_head,
> - rte_memory_order_release,
> - rte_memory_order_acquire);
> - } while (unlikely(success == 0));
> +
> + /* on failure, *old_head is updated */
> + /*
> + * R1/A2.
> + * R1: Establishes a synchronizing edge with A0 of a
> + * different thread.
> + * A2: Establishes a synchronizing edge with R1 of a
> + * different thread to observe same value for stail
> + * observed by that thread on CAS failure (to retry
> + * with an updated *old_head).
> + */
> + } while (unlikely(!rte_atomic_compare_exchange_strong_explicit(
> + &d->head, old_head, *new_head,
> + rte_memory_order_release,
> rte_memory_order_acquire)));
> return n;
> }
>
> diff --git a/lib/ring/rte_ring_elem_pvt.h b/lib/ring/rte_ring_elem_pvt.h
> index 6eafae121f..9d1da12a92 100644
> --- a/lib/ring/rte_ring_elem_pvt.h
> +++ b/lib/ring/rte_ring_elem_pvt.h
> @@ -299,17 +299,108 @@ __rte_ring_dequeue_elems(struct rte_ring *r,
> uint32_t cons_head,
> cons_head & r->mask, esize, num);
> }
>
> -/* Between load and load. there might be cpu reorder in weak model
> - * (powerpc/arm).
> - * There are 2 choices for the users
> - * 1.use rmb() memory barrier
> - * 2.use one-direction load_acquire/store_release barrier
> - * It depends on performance test results.
> +/**
> + * @internal This function updates tail values.
> */
> -#ifdef RTE_USE_C11_MEM_MODEL
> -#include "rte_ring_c11_pvt.h"
> +static __rte_always_inline void
> +__rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t old_val,
> + uint32_t new_val, uint32_t single, uint32_t enqueue)
> +{
> + RTE_SET_USED(enqueue);
> +
> + /*
> + * If there are other enqueues/dequeues in progress that preceded us,
> + * we need to wait for them to complete
> + */
> + if (!single)
> + rte_wait_until_equal_32((uint32_t *)(uintptr_t)&ht->tail, old_val,
> + rte_memory_order_relaxed);
> +
> + /*
> + * R0: Establishes a synchronizing edge with load-acquire of tail at A1.
> + * Ensures that memory effects by this thread on ring elements array
> + * is observed by a different thread of the other type.
> + */
> + rte_atomic_store_explicit(&ht->tail, new_val,
> rte_memory_order_release);
> +}
> +
> +/**
> + * @internal This is a helper function that moves the producer/consumer head
> + *
> + *
> + * This optimized version for single threaded case.
> + *
> + * @param d
> + * A pointer to the headtail structure with head value to be moved
> + * @param s
> + * A pointer to the counter-part headtail structure. Note that this
> + * function only reads tail value from it
> + * @param capacity
> + * Either ring capacity value (for producer), or zero (for consumer)
> + * @param n
> + * The number of elements we want to move head value on
> + * @param behavior
> + * RTE_RING_QUEUE_FIXED: Move on a fixed number of items
> + * RTE_RING_QUEUE_VARIABLE: Move on as many items as possible
> + * @param old_head
> + * Returns head value as it was before the move
> + * @param new_head
> + * Returns the new head value
> + * @param entries
> + * Returns the number of ring entries available BEFORE head was moved
> + * @return
> + * Actual number of objects the head was moved on
> + * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only
> + */
> +static __rte_always_inline unsigned int
> +__rte_ring_headtail_move_head_st(struct rte_ring_headtail *d,
> + const struct rte_ring_headtail *s, uint32_t capacity,
> + unsigned int n, enum rte_ring_queue_behavior behavior,
> + uint32_t *old_head, uint32_t *new_head, uint32_t *entries)
> +{
> + uint32_t stail;
> +
I really like the idea to split _st and _mt move_head into separate functions.
That makes code much cleaner an easier to understand and maintain.
Few comments on actual '_st' implementation below:
> + /*
> + * A0: Establishes a synchronizing edge with R1.
> + * Ensure that this thread observes same values
> + * to stail observed by the thread that updated
> + * d->head.
> + * If not, an unsafe partial order may ensue.
> + */
I believe that comment is not relevant for '_st',
there is no R1 anymore for '_st' - see below,
and no other thread except that one can move the head.
So, there is probably no point to use '_acquire' order here.
> + *old_head = rte_atomic_load_explicit(&d->head,
> rte_memory_order_acquire);
> +
> + /*
> + * A1: Establishes a synchronizing edge with R0.
> + * Ensures that other thread's memory effects on
> + * ring elements array is observed by the time
> + * this thread observes its tail update.
> + */
> + stail = rte_atomic_load_explicit(&s->tail, rte_memory_order_acquire);
> +
> + /* The subtraction is done between two unsigned 32bits value
> + * (the result is always modulo 32 bits even if we have
> + * *old_head > s->tail). So 'entries' is always between 0
> + * and capacity (which is < size).
> + */
> + *entries = capacity + stail - *old_head;
> +
> + /* check that we have enough room in ring */
> + if (unlikely(n > *entries))
> + n = (behavior == RTE_RING_QUEUE_FIXED) ? 0 : *entries;
> +
> + if (n > 0) {
> + *new_head = *old_head + n;
> + d->head = *new_head;
There is a bit of inconsistency with the 'load' operation above:
If we use atomic_load(&d->head. ...) then it would be better to use
atomic_store(&d->head,..., order_relaxed) here.
> + }
> +
> + return n;
> +}
> +
> +/* There are two choices because GCC optimizer does poorly on
> atomic_compare_exchange */
> +#if defined(RTE_TOOLCHAIN_GCC) && defined(RTE_ARCH_X86)
If we still need to use legacy code for x86, I think we need an explcit macro
to enable C11 for x86 (RTE_RING_FORCE_C11 or so):
to make sure that C11 version will still get tested and measured on x86.
> +#include "rte_ring_x86_pvt.h"
> #else
> -#include "rte_ring_generic_pvt.h"
> +#include "rte_ring_c11_pvt.h"
> #endif
I tried to look at compiler output for both cases, most of the code
looks nearly identical, one thing that I noticed:
C11 __rte_ring_headtail_move_head_mt() uses output
parameter: 'uint32_t *old_head' directly within CAS operation.
In x86_64 that cause gcc to generate extra instructions to
store return value of CAS (eax) within 'old_head' memory location,
even when CAS was not successfull and another attempt should be
performed. In some cases, even extra branch can be observed:
https://godbolt.org/z/4dTrqMjYe
In constrast, x86 specific version that uses
__sync_bool_compare_and_swap() doesn't exibit such problem,
as __sync_bool_compare_and_swap() doesn't update the 'old_head'
with new value, and we have to re-read it explicitly on each iteration.
I tried to overcome that problem by using local variable 'head' inside the loop,
and updaing '*old_head' value only at exit.
With such change gcc manages to avoid extra store(/branch),
see __rte_ring_headtail_move_head_mt_c11_v2() in the link above.
Can I ask you to re-run your perf test with the patch:
https://patchwork.dpdk.org/project/dpdk/patch/20260601181509.71007-1-konstantin.ananyev@huawei.com/
applied on top of your changes and see would it help in terms of performance?
From other side - if you'll point me to the exact tests you are running,
I am happy to repeat them on my box.
My preference would be to avoid arch/compiler specific versions, if possible.
> /**
> @@ -341,8 +432,12 @@ __rte_ring_move_prod_head(struct rte_ring *r,
> unsigned int is_sp,
> uint32_t *old_head, uint32_t *new_head,
> uint32_t *free_entries)
> {
> - return __rte_ring_headtail_move_head(&r->prod, &r->cons, r->capacity,
> - is_sp, n, behavior, old_head, new_head, free_entries);
> + if (is_sp)
> + return __rte_ring_headtail_move_head_st(&r->prod, &r->cons,
> r->capacity,
> + n, behavior, old_head, new_head, free_entries);
> + else
> + return __rte_ring_headtail_move_head_mt(&r->prod, &r->cons,
> r->capacity,
> + n, behavior, old_head, new_head, free_entries);
> }
>
> /**
> @@ -374,8 +469,12 @@ __rte_ring_move_cons_head(struct rte_ring *r,
> unsigned int is_sc,
> uint32_t *old_head, uint32_t *new_head,
> uint32_t *entries)
> {
> - return __rte_ring_headtail_move_head(&r->cons, &r->prod, 0,
> - is_sc, n, behavior, old_head, new_head, entries);
> + if (is_sc)
> + return __rte_ring_headtail_move_head_st(&r->cons, &r->prod,
> 0,
> + n, behavior, old_head, new_head, entries);
> + else
> + return __rte_ring_headtail_move_head_mt(&r->cons, &r->prod,
> 0,
> + n, behavior, old_head, new_head, entries);
> }
>
> /**
> diff --git a/lib/ring/rte_ring_generic_pvt.h b/lib/ring/rte_ring_x86_pvt.h
> similarity index 60%
> rename from lib/ring/rte_ring_generic_pvt.h
> rename to lib/ring/rte_ring_x86_pvt.h
> index affd2d5ba7..c8de108bbd 100644
> --- a/lib/ring/rte_ring_generic_pvt.h
> +++ b/lib/ring/rte_ring_x86_pvt.h
> @@ -7,39 +7,19 @@
> * Used as BSD-3 Licensed with permission from Kip Macy.
> */
>
> -#ifndef _RTE_RING_GENERIC_PVT_H_
> -#define _RTE_RING_GENERIC_PVT_H_
> +#ifndef _RTE_RING_X86_PVT_H_
> +#define _RTE_RING_X86_PVT_H_
>
> /**
> - * @file rte_ring_generic_pvt.h
> + * @file rte_ring_x86_pvt.h
> * It is not recommended to include this file directly,
> * include <rte_ring.h> instead.
> - * Contains internal helper functions for MP/SP and MC/SC ring modes.
> - * For more information please refer to <rte_ring.h>.
> + *
> + * Contains internal helper functions for MP and MC ring modes.
> + * It is GCC specific to workaround poor optimizer handling of C11 atomic
> + * compare_exchange.
> */
>
> -/**
> - * @internal This function updates tail values.
> - */
> -static __rte_always_inline void
> -__rte_ring_update_tail(struct rte_ring_headtail *ht, uint32_t old_val,
> - uint32_t new_val, uint32_t single, uint32_t enqueue)
> -{
> - if (enqueue)
> - rte_smp_wmb();
> - else
> - rte_smp_rmb();
> - /*
> - * If there are other enqueues/dequeues in progress that preceded us,
> - * we need to wait for them to complete
> - */
> - if (!single)
> - rte_wait_until_equal_32((volatile uint32_t *)(uintptr_t)&ht->tail,
> old_val,
> - rte_memory_order_relaxed);
> -
> - ht->tail = new_val;
> -}
> -
> /**
> * @internal This is a helper function that moves the producer/consumer head
> *
> @@ -50,8 +30,6 @@ __rte_ring_update_tail(struct rte_ring_headtail *ht,
> uint32_t old_val,
> * function only reads tail value from it
> * @param capacity
> * Either ring capacity value (for producer), or zero (for consumer)
> - * @param is_st
> - * Indicates whether multi-thread safe path is needed or not
> * @param n
> * The number of elements we want to move head value on
> * @param behavior
> @@ -68,14 +46,13 @@ __rte_ring_update_tail(struct rte_ring_headtail *ht,
> uint32_t old_val,
> * If behavior == RTE_RING_QUEUE_FIXED, this will be 0 or n only
> */
> static __rte_always_inline unsigned int
> -__rte_ring_headtail_move_head(struct rte_ring_headtail *d,
> +__rte_ring_headtail_move_head_mt(struct rte_ring_headtail *d,
> const struct rte_ring_headtail *s, uint32_t capacity,
> - unsigned int is_st, unsigned int n,
> + unsigned int n,
> enum rte_ring_queue_behavior behavior,
> uint32_t *old_head, uint32_t *new_head, uint32_t *entries)
> {
> unsigned int max = n;
> - int success;
>
> do {
> /* Reset n to the initial burst count */
> @@ -83,18 +60,13 @@ __rte_ring_headtail_move_head(struct rte_ring_headtail
> *d,
>
> *old_head = d->head;
>
> - /* add rmb barrier to avoid load/load reorder in weak
> - * memory model. It is noop on x86
> - */
> - rte_smp_rmb();
> -
> /*
> * The subtraction is done between two unsigned 32bits value
> * (the result is always modulo 32 bits even if we have
> * *old_head > s->tail). So 'entries' is always between 0
> * and capacity (which is < size).
> */
> - *entries = (capacity + s->tail - *old_head);
> + *entries = capacity + s->tail - *old_head;
>
> /* check that we have enough room in ring */
> if (unlikely(n > *entries))
> @@ -105,15 +77,10 @@ __rte_ring_headtail_move_head(struct
> rte_ring_headtail *d,
> return 0;
>
> *new_head = *old_head + n;
> - if (is_st) {
> - d->head = *new_head;
> - success = 1;
> - } else
> - success = rte_atomic32_cmpset(
> - (uint32_t *)(uintptr_t)&d->head,
> - *old_head, *new_head);
> - } while (unlikely(success == 0));
> + } while (unlikely(!__sync_bool_compare_and_swap(
> + (uint32_t *)(uintptr_t)&d->head,
> + *old_head, *new_head)));
> return n;
> }
>
> -#endif /* _RTE_RING_GENERIC_PVT_H_ */
> +#endif /* _RTE_RING_X86_PVT_H_ */
> diff --git a/lib/ring/soring.c b/lib/ring/soring.c
> index 3b90521bdb..0e8bbc03c1 100644
> --- a/lib/ring/soring.c
> +++ b/lib/ring/soring.c
> @@ -135,9 +135,12 @@ __rte_soring_move_prod_head(struct rte_soring *r,
> uint32_t num,
>
> switch (st) {
> case RTE_RING_SYNC_ST:
> + n = __rte_ring_headtail_move_head_st(&r->prod.ht, &r-
> >cons.ht,
> + r->capacity, num, behavior, head, next, free);
> + break;
> case RTE_RING_SYNC_MT:
> - n = __rte_ring_headtail_move_head(&r->prod.ht, &r->cons.ht,
> - r->capacity, st, num, behavior, head, next, free);
> + n = __rte_ring_headtail_move_head_mt(&r->prod.ht, &r-
> >cons.ht,
> + r->capacity, num, behavior, head, next, free);
> break;
> case RTE_RING_SYNC_MT_RTS:
> n = __rte_ring_rts_move_head(&r->prod.rts, &r->cons.ht,
> @@ -168,9 +171,13 @@ __rte_soring_move_cons_head(struct rte_soring *r,
> uint32_t stage, uint32_t num,
>
> switch (st) {
> case RTE_RING_SYNC_ST:
> + n = __rte_ring_headtail_move_head_st(&r->cons.ht,
> + &r->stage[stage].ht, 0, num, behavior,
> + head, next, avail);
> + break;
> case RTE_RING_SYNC_MT:
> - n = __rte_ring_headtail_move_head(&r->cons.ht,
> - &r->stage[stage].ht, 0, st, num, behavior,
> + n = __rte_ring_headtail_move_head_mt(&r->cons.ht,
> + &r->stage[stage].ht, 0, num, behavior,
> head, next, avail);
> break;
> case RTE_RING_SYNC_MT_RTS:
> --
> 2.53.0
^ permalink raw reply
* [PATCH] ring: avoid extra store at move head
From: Konstantin Ananyev @ 2026-06-01 18:15 UTC (permalink / raw)
To: dev; +Cc: stephen
C11 __rte_ring_headtail_move_head_mt() uses output
parameter: 'uint32_t *old_head' directly within CAS operation.
In x86_64 that cause gcc to generate extra instructions to
store return value of CAS (eax) within 'old_head' memory location,
even when CAS was not successful and another attempt should be
performed. In some cases, even extra branch can be observed.
To be more specific the code like that is generated:
// start of 'do { } while();' loop
.L2
...
lock cmpxchgl %r8d, (%rdi)
jne .L17 //
.L1: // <---- successful completion of CAS, finish
movl %edx, %eax
ret
.L17: // <---- unsuccessful completion of CAS, repeat
movl %eax, (%r9)
jmp .L2
In constrast, x86 specific version that uses
__sync_bool_compare_and_swap() doesn't exibit such problem,
as __sync_bool_compare_and_swap() doesn't update the 'old_head'
with new value, and we have to re-read it explicitly on each iteration.
Overcome that problem by using local variable 'head' inside the loop,
and updaing '*old_head' value only at exit.
With such change gcc manages to avoid extra store(/branch).
Depends-on: series-38225 ("deprecate rte_atomicNN family")
Signed-off-by: Konstantin Ananyev <konstantin.ananyev@huawei.com>
---
lib/ring/rte_ring_c11_pvt.h | 19 +++++++++++--------
lib/ring/rte_ring_elem_pvt.h | 5 -----
2 files changed, 11 insertions(+), 13 deletions(-)
diff --git a/lib/ring/rte_ring_c11_pvt.h b/lib/ring/rte_ring_c11_pvt.h
index 3efe011f08..ee98155bea 100644
--- a/lib/ring/rte_ring_c11_pvt.h
+++ b/lib/ring/rte_ring_c11_pvt.h
@@ -52,6 +52,7 @@ __rte_ring_headtail_move_head_mt(struct rte_ring_headtail *d,
unsigned int n, enum rte_ring_queue_behavior behavior,
uint32_t *old_head, uint32_t *new_head, uint32_t *entries)
{
+ uint32_t head;
unsigned int max = n;
/*
@@ -61,7 +62,7 @@ __rte_ring_headtail_move_head_mt(struct rte_ring_headtail *d,
* d->head.
* If not, an unsafe partial order may ensue.
*/
- *old_head = rte_atomic_load_explicit(&d->head, rte_memory_order_acquire);
+ head = rte_atomic_load_explicit(&d->head, rte_memory_order_acquire);
do {
/* Reset n to the initial burst count */
n = max;
@@ -76,10 +77,10 @@ __rte_ring_headtail_move_head_mt(struct rte_ring_headtail *d,
/* The subtraction is done between two unsigned 32bits value
* (the result is always modulo 32 bits even if we have
- * *old_head > s->tail). So 'entries' is always between 0
+ * head > s->tail). So 'entries' is always between 0
* and capacity (which is < size).
*/
- *entries = capacity + stail - *old_head;
+ *entries = capacity + stail - head;
/* check that we have enough room in ring */
if (unlikely(n > *entries))
@@ -87,11 +88,11 @@ __rte_ring_headtail_move_head_mt(struct rte_ring_headtail *d,
0 : *entries;
if (n == 0)
- return 0;
+ break;
- *new_head = *old_head + n;
+ *new_head = head + n;
- /* on failure, *old_head is updated */
+ /* on failure, head is updated */
/*
* R1/A2.
* R1: Establishes a synchronizing edge with A0 of a
@@ -99,11 +100,13 @@ __rte_ring_headtail_move_head_mt(struct rte_ring_headtail *d,
* A2: Establishes a synchronizing edge with R1 of a
* different thread to observe same value for stail
* observed by that thread on CAS failure (to retry
- * with an updated *old_head).
+ * with an updated head).
*/
} while (unlikely(!rte_atomic_compare_exchange_strong_explicit(
- &d->head, old_head, *new_head,
+ &d->head, &head, *new_head,
rte_memory_order_release, rte_memory_order_acquire)));
+
+ *old_head = head;
return n;
}
diff --git a/lib/ring/rte_ring_elem_pvt.h b/lib/ring/rte_ring_elem_pvt.h
index 9d1da12a92..51176b0405 100644
--- a/lib/ring/rte_ring_elem_pvt.h
+++ b/lib/ring/rte_ring_elem_pvt.h
@@ -396,12 +396,7 @@ __rte_ring_headtail_move_head_st(struct rte_ring_headtail *d,
return n;
}
-/* There are two choices because GCC optimizer does poorly on atomic_compare_exchange */
-#if defined(RTE_TOOLCHAIN_GCC) && defined(RTE_ARCH_X86)
-#include "rte_ring_x86_pvt.h"
-#else
#include "rte_ring_c11_pvt.h"
-#endif
/**
* @internal This function updates the producer head for enqueue
--
2.51.0
^ permalink raw reply related
* [PATCH v2] doc, eal, devtools: discourage new __rte_always_inline
From: Stephen Hemminger @ 2026-06-01 17:21 UTC (permalink / raw)
To: dev; +Cc: Stephen Hemminger, Thomas Monjalon
In-Reply-To: <20260527163734.599602-1-stephen@networkplumber.org>
Modern compilers at -O2 make good inlining decisions for small
static inline functions; forced inlining via __rte_always_inline
should be reserved for cases where it is required for correctness
or for documented measured performance reasons.
Document the policy in the coding style guide and add a
checkpatches.sh entry that flags when new uses of the attribute
are introduced. Checkpatches is not an absolute blocker to
acceptance, only an indicator that more review is needed.
Add additional comments about use of __rte_always_inline,
__rte_noinline, __rte_hot, and __rte_cold to the rte_common.h
to aid developers.
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
devtools/checkpatches.sh | 8 +++++
doc/guides/contributing/coding_style.rst | 26 ++++++++++++++++-
lib/eal/include/rte_common.h | 37 ++++++++++++++++++++++--
3 files changed, 68 insertions(+), 3 deletions(-)
diff --git a/devtools/checkpatches.sh b/devtools/checkpatches.sh
index f5dd77443f..2a3d364178 100755
--- a/devtools/checkpatches.sh
+++ b/devtools/checkpatches.sh
@@ -137,6 +137,14 @@ check_forbidden_additions() { # <patch>
-f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \
"$1" || res=1
+ # forbid new use of __rte_always_inline
+ awk -v FOLDERS="lib drivers app examples" \
+ -v EXPRESSIONS='\\<__rte_always_inline\\>' \
+ -v RET_ON_FAIL=1 \
+ -v MESSAGE='Adding __rte_always_inline; prefer plain inline' \
+ -f $(dirname $(readlink -f $0))/check-forbidden-tokens.awk \
+ "$1" || res=1
+
# refrain from using compiler __rte_atomic_thread_fence()
# It should be avoided on x86 for SMP case.
awk -v FOLDERS="lib drivers app examples" \
diff --git a/doc/guides/contributing/coding_style.rst b/doc/guides/contributing/coding_style.rst
index 243a3c2959..8f9e77d1c1 100644
--- a/doc/guides/contributing/coding_style.rst
+++ b/doc/guides/contributing/coding_style.rst
@@ -747,11 +747,35 @@ Static Variables and Functions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* All functions and variables that are local to a file must be declared as ``static`` because it can often help the compiler to do some optimizations (such as, inlining the code).
-* Functions that should be inlined should to be declared as ``static inline`` and can be defined in a .c or a .h file.
+* Functions that should be inlined should be declared as ``static inline`` and can be defined in a .c or a .h file.
.. note::
Static functions defined in a header file must be declared as ``static inline`` in order to prevent compiler warnings about the function being unused.
+Use of ``__rte_always_inline``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``__rte_always_inline`` attribute forces the compiler to inline a function regardless of its size or call-graph heuristics.
+Prefer plain ``inline`` (or no annotation at all for static functions) and let the compiler decide.
+Modern compilers at ``-O2`` make good inlining decisions for small ``static inline`` functions in headers,
+and forced inlining can hurt performance by inflating function bodies, increasing register pressure, and overriding profile-guided optimization.
+
+``__rte_always_inline`` should only be used when one of the following applies:
+
+* The function contains ``__rte_constant()`` checks that gate a constant-folded fast path,
+ and the optimization is lost if the function is not inlined into the caller.
+ Examples include byte-order helpers and length-dispatched copy/compare routines.
+
+* The function wraps inline assembly or a compiler intrinsic whose correctness depends on being inlined into the caller's register context (for example, intrinsics requiring a compile-time constant argument).
+
+* Measurement on a representative workload shows that the annotation is required to retain performance, and the reason is documented in the commit message that introduces it.
+
+Each use must be justified at the point it is introduced. Adding ``__rte_always_inline`` because nearby code uses it is not a justification;
+if the constant or intrinsic that requires inlining is several call levels up the call chain,
+restructure the code rather than annotating the entire chain.
+
+The complementary attribute ``__rte_noinline`` is useful for explicitly marking cold paths (error handling, initialization, slow-path fallbacks) where outlining the function can reduce instruction-cache pressure on the hot path.
+
Const Attribute
~~~~~~~~~~~~~~~
diff --git a/lib/eal/include/rte_common.h b/lib/eal/include/rte_common.h
index 71415346cf..e358be7fcf 100644
--- a/lib/eal/include/rte_common.h
+++ b/lib/eal/include/rte_common.h
@@ -482,7 +482,22 @@ static void __attribute__((destructor(RTE_PRIO(prio)), used)) func(void)
#endif
/**
- * Force a function to be inlined
+ * Force a function to be inlined, regardless of the compiler's size and
+ * call-graph heuristics.
+ *
+ * Prefer plain @c inline (or no annotation on a static function) and let the compiler decide.
+ * Modern compilers at -O2 inline small static functions well,
+ * and forcing it can hurt by inflating call sites, raising register pressure,
+ * and overriding profile-guided optimization.
+ *
+ * Reserve this attribute for cases where inlining is required for
+ * correctness, or for a documented and measured performance reason, e.g.
+ * - a constant-folded fast path gated by @c __rte_constant() that is lost
+ * unless the function is inlined into the caller;
+ * - a wrapper around inline asm or an intrinsic that needs a
+ * compile-time-constant argument from the caller's context.
+ *
+ * See the DPDK coding style guide for the full policy.
*/
#ifdef RTE_TOOLCHAIN_MSVC
#define __rte_always_inline __forceinline
@@ -491,7 +506,11 @@ static void __attribute__((destructor(RTE_PRIO(prio)), used)) func(void)
#endif
/**
- * Force a function to be noinlined
+ * Force a function not to be inlined.
+ *
+ * Useful for explicitly outlining cold paths such as error handling,
+ * initialization, slow-path fallbacks, so they do not bloat the hot path
+ * or add to its instruction-cache footprint.
*/
#ifdef RTE_TOOLCHAIN_MSVC
#define __rte_noinline __declspec(noinline)
@@ -501,6 +520,12 @@ static void __attribute__((destructor(RTE_PRIO(prio)), used)) func(void)
/**
* Hint function in the hot path
+ *
+ * The compiler may optimize the function more aggressively, treat calls
+ * to it as likely for branch prediction, and group it with other hot
+ * functions to improve instruction-cache locality. This affects code
+ * placement and prediction, not inlining; combine with an inline
+ * annotation if both are wanted.
*/
#ifdef RTE_TOOLCHAIN_MSVC
#define __rte_hot
@@ -510,6 +535,14 @@ static void __attribute__((destructor(RTE_PRIO(prio)), used)) func(void)
/**
* Hint function in the cold path
+ *
+ * The compiler optimizes the function for size rather than speed,
+ * marks branches that reach it as unlikely, and may move it to a separate
+ * section to keep it off the hot path and reduce instruction-cache
+ * pressure there.
+ *
+ * Suitable for error handling, logging, and setup/teardown code.
+ * Functions marked @c __rte_noreturn are already treated as cold.
*/
#ifdef RTE_TOOLCHAIN_MSVC
#define __rte_cold
--
2.53.0
^ permalink raw reply related
* Re: [v2] dts: add support for no link topology
From: Andrew Bailey @ 2026-06-01 17:16 UTC (permalink / raw)
To: Patrick Robb; +Cc: dev, dmarx, luca.vizzarro, probb
In-Reply-To: <20260530203757.22139-1-patrickrobb1997@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 822 bytes --]
Hi Patrick, Thanks for the review
> I don't understand the change above. Surely we do want to set the custom
> meson args for ICE DUTs when there is a sut_port_ingress, not when there
> isn't a sut_port_ingress, right?
>
> Yes, this was something I overlooked and will be updated in the next patch.
> Also, does the topology type affect whether we want to use these custom
> args, if the SUT driver is ice? I guess in a weird case we could have an
> ethdev test using an ice DUT which does not need a TG?
>
> My understanding is that if the link topology is no link, there will be no
ingress and the ports on the NIC would not be used for traffic. I do not
believe this argument will still be required if no traffic is being
handled. Please let me know if you disagree with this approach or if I am
missing something.
[-- Attachment #2: Type: text/html, Size: 1219 bytes --]
^ permalink raw reply
* Re: [PATCH v5 1/1] net/mana: add device reset support
From: Stephen Hemminger @ 2026-06-01 16:58 UTC (permalink / raw)
To: Wei Hu; +Cc: dev, longli, weh
In-Reply-To: <20260529142648.148407-2-weh@linux.microsoft.com>
On Fri, 29 May 2026 07:26:48 -0700
Wei Hu <weh@linux.microsoft.com> wrote:
> From: Wei Hu <weh@microsoft.com>
>
> Add support for handling hardware reset events in the MANA driver.
> When the MANA kernel driver receives a hardware service event, it
> initiates a device reset and notifies userspace via
> IBV_EVENT_DEVICE_FATAL. The DPDK driver handles this by performing
> an automatic teardown and recovery sequence.
>
> The reset flow has two phases. In the enter phase, running on the
> EAL interrupt thread, the driver transitions the device state,
> waits for data path threads to reach a quiescent state using RCU,
> stops queues, tears down IB resources, and frees per-queue MR
> caches. A control thread is then spawned to handle the exit phase:
> it waits for the hardware to recover, unregisters the interrupt
> handler, re-probes the PCI device, reinitializes MR caches, and
> restarts queues.
>
> A per-device mutex serializes the reset path with ethdev
> operations. The mutex uses PTHREAD_PROCESS_SHARED for multi-process
> support and is held across blocking IB verbs calls. Operations that
> cannot wait (configure, queue setup) return -EBUSY during reset,
> while dev_stop and dev_close join the reset thread before acquiring
> the lock to ensure proper sequencing. A CAS-based helper prevents
> double-join of the reset thread.
>
> Multi-process support is included: secondary processes unmap and
> remap doorbell pages via IPC during the reset enter and exit
> phases. Data path functions in both primary and secondary
> processes check the device state atomically and return early when
> the device is not active. RCU quiescent state tracking uses
> per-queue thread IDs in shared hugepage memory, covering both
> primary and secondary process data path threads.
>
> The driver uses ethdev recovery events to notify upper layers
> (e.g. netvsc) of the reset lifecycle: RTE_ETH_EVENT_ERR_RECOVERING
> on entry, RTE_ETH_EVENT_RECOVERY_SUCCESS or
> RTE_ETH_EVENT_RECOVERY_FAILED on completion. A PCI device removal
> event callback distinguishes hot-remove from service reset.
>
> Documentation for the device reset feature is added in the MANA
> NIC guide and the 26.07 release notes.
>
> Signed-off-by: Wei Hu <weh@microsoft.com>
> ---
I went a deeper with AI and tried to figure out a good way
to do what the driver is trying to do without reinventing so much.
This reset logic is considerably more complex than other DPDK drivers,
and most of the complexity looks self-inflicted. A few specific things.
The RCU use is not really RCU. thread_online/offline are called on every
rx/tx burst, and the "thread" token is the queue index, not a thread. So
it is a per-queue in-use flag paid for on the hottest path, plus a new
library dependency, to express "wait until no queue is mid-burst" -- which
the driver already half does by swapping the burst function to
mana_*_burst_removed and checking dev_state. Please drop the rcu use. If
you must drain readers, a per-queue atomic flag is lighter and local; the
fast path already has the dev_state acquire-load it needs.
The data path only needs the atomic, and that part is fine. The lock is
legitimate for serializing the teardown/rebuild against control ops, but
the way it is used is the problem: it is acquired in mana_intr_handler,
released in mana_reset_enter, re-acquired in mana_reset_thread, and
released in mana_reset_exit_delay. That cross-function, cross-thread
handoff is exactly why every function needs __rte_no_thread_safety_analysis.
Acquire and release the lock in the same function and the annotations all
go away. Turning off thread-safety analysis needs strong justification and
this does not have it.
Wrapping the ops in MANA_OPS_*_LOCK macros hides the lock/state protocol.
A single explicit helper at the top of each op is just as terse and stays
analyzable.
Two real bugs:
- Recovery and INTR_RMV events are sent via rte_eth_dev_callback_process
while reset_ops_lock is held. An app handling INTR_RMV or RECOVERY_FAILED
by calling dev_stop/dev_close will re-enter the lock (non-recursive) and
deadlock; on the recovery path the callback runs on the reset thread so
it also tries to join itself. Emit these events after dropping the lock,
as ERR_RECOVERING already does.
- thread_online is taken at the top of the burst functions but
thread_offline is only on some return paths. Any early return that
misses it leaves a token non-quiescent and rte_rcu_qsbr_check() in
mana_reset_enter spins forever. This is the kind of breakage the per-burst
bracketing invites -- another reason to drop it.
Also: reset_ops_lock is held across ibv_close_device and the PCI re-probe
(blocking under a sleeping mutex), and rte_alarm.h is included but no
longer used.
Please look at how hns3 or mlx5 structure reset/recovery. Matching the
common pattern means fixes can be made across drivers at once.
^ permalink raw reply
* [PATCH v7] mempool: improve cache behaviour and performance
From: Morten Brørup @ 2026-06-01 16:40 UTC (permalink / raw)
To: dev, Andrew Rybchenko, Bruce Richardson, Jingjing Wu,
Praveen Shetty, Hemant Agrawal, Sachin Saxena, Thomas Monjalon
Cc: Morten Brørup
In-Reply-To: <20260408141315.904381-1-mb@smartsharesystems.com>
This patch refactors the mempool cache to eliminate some unexpected
behaviour and reduce the mempool cache miss rate.
1.
The actual cache size was 1.5 times the cache size specified at run-time
mempool creation.
This was obviously not expected by application developers.
2.
In get operations, the check for when to use the cache as bounce buffer
did not respect the run-time configured cache size,
but compared to the build time maximum possible cache size
(RTE_MEMPOOL_CACHE_MAX_SIZE, default 512).
E.g. with a configured cache size of 32 objects, getting 256 objects
would first fetch 32 + 256 = 288 objects into the cache,
and then move the 256 objects from the cache to the destination memory,
instead of fetching the 256 objects directly to the destination memory.
This had a performance cost.
However, this is unlikely to occur in real applications, so it is not
important in itself.
3.
When putting objects into a mempool, and the mempool cache did not have
free space for so many objects,
the cache was flushed completely, and the new objects were then put into
the cache.
I.e. the cache drain level was zero.
This (complete cache flush) meant that a subsequent get operation (with
the same number of objects) completely emptied the cache,
so another subsequent get operation required replenishing the cache.
Similarly,
When getting objects from a mempool, and the mempool cache did not hold so
many objects,
the cache was replenished to cache->size + remaining objects,
and then (the remaining part of) the requested objects were fetched via
the cache,
which left the cache filled (to cache->size) at completion.
I.e. the cache refill level was cache->size (plus some, depending on
request size).
(1) was improved by generally comparing to cache->size instead of
cache->flushthresh, when considering the capacity of the cache.
The cache->flushthresh field is kept for API/ABI compatibility purposes,
and initialized to cache->size instead of cache->size * 1.5.
(2) was improved by generally comparing to cache->size / 2 instead of
RTE_MEMPOOL_CACHE_MAX_SIZE, when checking the bounce buffer limit.
(3) was improved by flushing and replenishing the cache by half its size,
so a flush/refill can be followed randomly by get or put requests.
This also reduced the number of objects in each flush/refill operation.
As a consequence of these changes, the size of the array holding the
objects in the cache (cache->objs[]) no longer needs to be
2 * RTE_MEMPOOL_CACHE_MAX_SIZE, and can be reduced to
RTE_MEMPOOL_CACHE_MAX_SIZE at an API/ABI breaking release.
Performance data:
With a real WAN Optimization application, where the number of allocated
packets varies (as they are held in e.g. shaper queues), the mempool
cache miss rate dropped from ca. 1/20 objects to ca. 1/48 objects.
This was deployed in production at an ISP, and using an effective cache
size of 384 objects.
Bugzilla ID: 1027
Fixes: ea5dd2744b90 ("mempool: cache optimisations")
Signed-off-by: Morten Brørup <mb@smartsharesystems.com>
Acked-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
---
v7:
* Rebased. Dependency no longer required. (Thomas)
v6:
* Moved driver changes out as separate patches, for easier review. (Bruce)
Tests using the Intel idpf PMD in AVX512 mode may fail with this patch.
* Reverted a small code comment change. The original was better. (Bruce)
* Reverted rte_mempool_create() description requiring the cache_size to be
an even number. There is no such requirement.
v5:
* Flush the cache from the bottom, where objects are colder, and move down
the remaining objects, which are hotter.
* In the Intel idpf PMD, move up the hot objects in the cache and refill
with cold objects at the bottom.
v4:
* Added Bugzilla ID.
* Added Fixes tag. For reference only.
* Moved fast-free related update of Intel common driver out as a separate
patch, and depend on that patch.
* Omitted unrelated changes to the Intel idpf AVX512 driver, specifically
fixing an indentation and adding mbuf instrumentation.
* Omitted unrelated changes to the mempool library, specifically adding
__rte_restrict and changing a couple of comments to proper sentences.
* Please checkpatches by swapping operators in a couple of comparisons.
v3:
* Fixed my copy-paste bug in idpf_splitq_rearm().
v2:
* Fixed issue found by abidiff:
Reverted cache objects array size reduction. Added a note instead.
* Added missing mbuf instrumentation to the Intel idpf AVX512 driver.
* Updated idpf_splitq_rearm() like idpf_singleq_rearm().
* Added a few more __rte_assume(). (Inspired by AI review)
* Updated NXP dpaa and dpaa2 mempool drivers to not set mempool cache
flush threshold.
* Added release notes.
* Added deprecation notes.
---
doc/guides/rel_notes/deprecation.rst | 7 +++
doc/guides/rel_notes/release_26_07.rst | 11 +++++
lib/mempool/rte_mempool.c | 14 +-----
lib/mempool/rte_mempool.h | 66 ++++++++++++++++----------
4 files changed, 61 insertions(+), 37 deletions(-)
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 17f90a6352..1b6cc181fb 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -158,3 +158,10 @@ Deprecation Notices
* net/iavf: The dynamic mbuf field used to detect LLDP packets on the
transmit path in the iavf PMD will be removed in a future release.
After removal, only packet type-based detection will be supported.
+
+* mempool: The ``flushthresh`` field in ``struct rte_mempool_cache``
+ is obsolete, and will be removed in DPDK 26.11.
+
+* mempool: The object array in ``struct rte_mempool_cache`` is oversize by
+ factor two, and will be reduced to ``RTE_MEMPOOL_CACHE_MAX_SIZE`` in
+ DPDK 26.11.
diff --git a/doc/guides/rel_notes/release_26_07.rst b/doc/guides/rel_notes/release_26_07.rst
index 8b4f8401e2..1b15c878f6 100644
--- a/doc/guides/rel_notes/release_26_07.rst
+++ b/doc/guides/rel_notes/release_26_07.rst
@@ -63,6 +63,17 @@ New Features
``rte_eal_init`` and the application is responsible for probing each device,
* ``--auto-probing`` enables the initial bus probing, which is the current default behavior.
+* **Changed effective size of mempool cache.**
+
+ * The effective size of a mempool cache was changed to match the specified size at mempool creation; the effective size was previously 50 % larger than requested.
+ * The ``flushthresh`` field of the ``struct rte_mempool_cache`` became obsolete, but was kept for API/ABI compatibility purposes.
+ * The effective size of the ``objs`` array in the ``struct rte_mempool_cache`` was reduced to ``RTE_MEMPOOL_CACHE_MAX_SIZE``, but its size was kept for API/ABI compatibility purposes.
+
+* **Improved mempool cache flush/refill algorithm.**
+
+ The mempool cache flush/refill algorithm was improved, to reduce the mempool cache miss rate for most application types.
+ Applications where each lcore only puts or gets to a mempool, e.g. pipelined applications where ethdev Rx and Tx run on separate lcores, should adapt to the new algorithm by doubling their configured mempool cache size, to avoid doubling their mempool cache miss rate.
+
* **Added LinkData sxe2 ethernet driver.**
Added network driver for the LinkData network adapters.
diff --git a/lib/mempool/rte_mempool.c b/lib/mempool/rte_mempool.c
index 3042d94c14..805b52cc58 100644
--- a/lib/mempool/rte_mempool.c
+++ b/lib/mempool/rte_mempool.c
@@ -52,11 +52,6 @@ static void
mempool_event_callback_invoke(enum rte_mempool_event event,
struct rte_mempool *mp);
-/* Note: avoid using floating point since that compiler
- * may not think that is constant.
- */
-#define CALC_CACHE_FLUSHTHRESH(c) (((c) * 3) / 2)
-
#if defined(RTE_ARCH_X86)
/*
* return the greatest common divisor between a and b (fast algorithm)
@@ -757,13 +752,8 @@ rte_mempool_free(struct rte_mempool *mp)
static void
mempool_cache_init(struct rte_mempool_cache *cache, uint32_t size)
{
- /* Check that cache have enough space for flush threshold */
- RTE_BUILD_BUG_ON(CALC_CACHE_FLUSHTHRESH(RTE_MEMPOOL_CACHE_MAX_SIZE) >
- RTE_SIZEOF_FIELD(struct rte_mempool_cache, objs) /
- RTE_SIZEOF_FIELD(struct rte_mempool_cache, objs[0]));
-
cache->size = size;
- cache->flushthresh = CALC_CACHE_FLUSHTHRESH(size);
+ cache->flushthresh = size; /* Obsolete; for API/ABI compatibility purposes only */
cache->len = 0;
}
@@ -850,7 +840,7 @@ rte_mempool_create_empty(const char *name, unsigned n, unsigned elt_size,
/* asked cache too big */
if (cache_size > RTE_MEMPOOL_CACHE_MAX_SIZE ||
- CALC_CACHE_FLUSHTHRESH(cache_size) > n) {
+ cache_size > n) {
rte_errno = EINVAL;
return NULL;
}
diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h
index 8c384d3453..26f47bf258 100644
--- a/lib/mempool/rte_mempool.h
+++ b/lib/mempool/rte_mempool.h
@@ -89,7 +89,7 @@ struct __rte_cache_aligned rte_mempool_debug_stats {
*/
struct __rte_cache_aligned rte_mempool_cache {
uint32_t size; /**< Size of the cache */
- uint32_t flushthresh; /**< Threshold before we flush excess elements */
+ uint32_t flushthresh; /**< Obsolete; for API/ABI compatibility purposes only */
uint32_t len; /**< Current cache count */
#ifdef RTE_LIBRTE_MEMPOOL_STATS
uint32_t unused;
@@ -107,8 +107,10 @@ struct __rte_cache_aligned rte_mempool_cache {
/**
* Cache objects
*
- * Cache is allocated to this size to allow it to overflow in certain
- * cases to avoid needless emptying of cache.
+ * Note:
+ * Cache is allocated at double size for API/ABI compatibility purposes only.
+ * When reducing its size at an API/ABI breaking release,
+ * remember to add a cache guard after it.
*/
alignas(RTE_CACHE_LINE_SIZE) void *objs[RTE_MEMPOOL_CACHE_MAX_SIZE * 2];
};
@@ -1047,11 +1049,16 @@ rte_mempool_free(struct rte_mempool *mp);
* If cache_size is non-zero, the rte_mempool library will try to
* limit the accesses to the common lockless pool, by maintaining a
* per-lcore object cache. This argument must be lower or equal to
- * RTE_MEMPOOL_CACHE_MAX_SIZE and n / 1.5.
+ * RTE_MEMPOOL_CACHE_MAX_SIZE and n.
* The access to the per-lcore table is of course
* faster than the multi-producer/consumer pool. The cache can be
* disabled if the cache_size argument is set to 0; it can be useful to
* avoid losing objects in cache.
+ * Note:
+ * Mempool put/get requests of more than cache_size / 2 objects may be
+ * partially or fully served directly by the multi-producer/consumer
+ * pool, to avoid the overhead of copying the objects twice (instead of
+ * once) when using the cache as a bounce buffer.
* @param private_data_size
* The size of the private data appended after the mempool
* structure. This is useful for storing some private data after the
@@ -1390,22 +1397,30 @@ rte_mempool_do_generic_put(struct rte_mempool *mp, void * const *obj_table,
RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_bulk, 1);
RTE_MEMPOOL_CACHE_STAT_ADD(cache, put_objs, n);
- __rte_assume(cache->flushthresh <= RTE_MEMPOOL_CACHE_MAX_SIZE * 2);
- __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE * 2);
- __rte_assume(cache->len <= cache->flushthresh);
- if (likely(cache->len + n <= cache->flushthresh)) {
+ __rte_assume(cache->size <= RTE_MEMPOOL_CACHE_MAX_SIZE);
+ __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
+ __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE);
+ __rte_assume(cache->len <= cache->size);
+ if (likely(cache->len + n <= cache->size)) {
/* Sufficient room in the cache for the objects. */
cache_objs = &cache->objs[cache->len];
cache->len += n;
- } else if (n <= cache->flushthresh) {
+ } else if (n <= cache->size / 2) {
/*
- * The cache is big enough for the objects, but - as detected by
- * the comparison above - has insufficient room for them.
- * Flush the cache to make room for the objects.
+ * The number of objects is within the cache bounce buffer limit,
+ * but - as detected by the comparison above - the cache has
+ * insufficient room for them.
+ * Flush the cache to the backend to make room for the objects;
+ * flush (size / 2) objects from the bottom of the cache, where
+ * objects are less hot, and move down the remaining objects, which
+ * are more hot, from the upper half of the cache.
*/
- cache_objs = &cache->objs[0];
- rte_mempool_ops_enqueue_bulk(mp, cache_objs, cache->len);
- cache->len = n;
+ __rte_assume(cache->len > cache->size / 2);
+ rte_mempool_ops_enqueue_bulk(mp, &cache->objs[0], cache->size / 2);
+ rte_memcpy(&cache->objs[0], &cache->objs[cache->size / 2],
+ sizeof(void *) * (cache->len - cache->size / 2));
+ cache_objs = &cache->objs[cache->len - cache->size / 2];
+ cache->len = cache->len - cache->size / 2 + n;
} else {
/* The request itself is too big for the cache. */
goto driver_enqueue_stats_incremented;
@@ -1524,7 +1539,7 @@ rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,
/* The cache is a stack, so copy will be in reverse order. */
cache_objs = &cache->objs[cache->len];
- __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE * 2);
+ __rte_assume(cache->len <= RTE_MEMPOOL_CACHE_MAX_SIZE);
if (likely(n <= cache->len)) {
/* The entire request can be satisfied from the cache. */
RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);
@@ -1548,13 +1563,13 @@ rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,
for (index = 0; index < len; index++)
*obj_table++ = *--cache_objs;
- /* Dequeue below would overflow mem allocated for cache? */
- if (unlikely(remaining > RTE_MEMPOOL_CACHE_MAX_SIZE))
+ /* Dequeue below would exceed the cache bounce buffer limit? */
+ __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
+ if (unlikely(remaining > cache->size / 2))
goto driver_dequeue;
- /* Fill the cache from the backend; fetch size + remaining objects. */
- ret = rte_mempool_ops_dequeue_bulk(mp, cache->objs,
- cache->size + remaining);
+ /* Fill the cache from the backend; fetch (size / 2) objects. */
+ ret = rte_mempool_ops_dequeue_bulk(mp, cache->objs, cache->size / 2);
if (unlikely(ret < 0)) {
/*
* We are buffer constrained, and not able to fetch all that.
@@ -1568,10 +1583,11 @@ rte_mempool_do_generic_get(struct rte_mempool *mp, void **obj_table,
RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_bulk, 1);
RTE_MEMPOOL_CACHE_STAT_ADD(cache, get_success_objs, n);
- __rte_assume(cache->size <= RTE_MEMPOOL_CACHE_MAX_SIZE);
- __rte_assume(remaining <= RTE_MEMPOOL_CACHE_MAX_SIZE);
- cache_objs = &cache->objs[cache->size + remaining];
- cache->len = cache->size;
+ __rte_assume(cache->size / 2 <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
+ __rte_assume(remaining <= RTE_MEMPOOL_CACHE_MAX_SIZE / 2);
+ __rte_assume(remaining <= cache->size / 2);
+ cache_objs = &cache->objs[cache->size / 2];
+ cache->len = cache->size / 2 - remaining;
for (index = 0; index < remaining; index++)
*obj_table++ = *--cache_objs;
--
2.43.0
^ permalink raw reply related
* Re: [PATCH v5 1/1] net/mana: add device reset support
From: Stephen Hemminger @ 2026-06-01 16:31 UTC (permalink / raw)
To: Wei Hu; +Cc: dev, longli, weh
In-Reply-To: <20260529142648.148407-2-weh@linux.microsoft.com>
On Fri, 29 May 2026 07:26:48 -0700
Wei Hu <weh@linux.microsoft.com> wrote:
> +#define MANA_OPS_1_LOCK(_func) \
> +static int \
> +_func##_lock(struct rte_eth_dev *dev) \
> +{ \
> + struct mana_priv *priv = dev->data->dev_private; \
> + int ret; \
> + if (!pthread_mutex_trylock(&priv->reset_ops_lock)) { \
> + if (rte_atomic_load_explicit(&priv->dev_state, \
> + rte_memory_order_acquire) != \
> + MANA_DEV_ACTIVE) { \
> + pthread_mutex_unlock(&priv->reset_ops_lock); \
> + return -EBUSY; \
> + } \
> + ret = _func(dev); \
> + pthread_mutex_unlock(&priv->reset_ops_lock); \
> + } else { \
> + ret = -EBUSY; \
> + } \
> + return ret; \
> +}
> +
> +MANA_OPS_1_LOCK(mana_dev_configure)
> +
> +MANA_OPS_1_LOCK(mana_dev_start)
I strongly dislike wrapping locking in macros.
Macros make code harder to analyze and hide things.
Also, using pthread_mutex here needs seems like a bigger hammer than needed.
It looks like reset is just an atomic flag and as long as it was in shared
memory simple atomic operators would suffice.
In Linux there are many ways to express the same thing
but in general if most drivers follow the same patterns it helps
when there is an issue to be able to fix it globally.
Also any code which turns off thread safety analysis is a red flag
for me. It needs strong justification.
Overall this reset logic looks more complex than other drivers.
^ permalink raw reply
* [PATCH v2] net: fix GTP Tunnel parse out-of-bounds read
From: Stephen Hemminger @ 2026-06-01 16:17 UTC (permalink / raw)
To: dev; +Cc: Stephen Hemminger, stable, Andrew Rybchenko, Jie Hai
In-Reply-To: <20260409161556.141251-1-stephen@networkplumber.org>
If packet is fragmented across multiple mbufs or the packet
has only GTP header the code would reference outside
the incoming mbuf.
Send GTP packet:
- Valid GTP header (8 bytes)
- msg_type = 0xff
- e=1, s=1, pn=1 (sets gtp_len = 12)
- Total packet size = 10 bytes
Read at gh + 12 accesses 2 bytes beyond packet end.
The fix is to use rte_pktmbuf_read in a manner similar
to the read of the GTP header.
Fixes: 64ed7f854cf4 ("net: add tunnel packet type parsing")
Cc: stable@dpdk.org
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
Reviewed-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
---
v2 - change variable name of l3 header byte
lib/net/rte_net.c | 16 +++++++++++-----
1 file changed, 11 insertions(+), 5 deletions(-)
diff --git a/lib/net/rte_net.c b/lib/net/rte_net.c
index 458b4814a9..0a91e92ba0 100644
--- a/lib/net/rte_net.c
+++ b/lib/net/rte_net.c
@@ -219,8 +219,7 @@ ptype_tunnel_with_udp(uint16_t *proto, const struct rte_mbuf *m,
case RTE_GTPU_UDP_PORT: {
const struct rte_gtp_hdr *gh;
struct rte_gtp_hdr gh_copy;
- uint8_t gtp_len;
- uint8_t ip_ver;
+ uint32_t gtp_len;
gh = rte_pktmbuf_read(m, *off, sizeof(*gh), &gh_copy);
if (unlikely(gh == NULL))
return 0;
@@ -231,9 +230,17 @@ ptype_tunnel_with_udp(uint16_t *proto, const struct rte_mbuf *m,
* Check message type. If message type is 0xff, it is
* a GTP data packet. If not, it is a GTP control packet
*/
+ *off += gtp_len;
if (gh->msg_type == 0xff) {
- ip_ver = *(const uint8_t *)((const char *)gh + gtp_len);
- ip_ver = (ip_ver) & 0xf0;
+ const uint8_t *l3_byte;
+ uint8_t l3_copy, ip_ver;
+
+ /* read first byte of l3 header */
+ l3_byte = rte_pktmbuf_read(m, *off, sizeof(uint8_t), &l3_copy);
+ if (unlikely(l3_byte == NULL))
+ return 0;
+
+ ip_ver = *l3_byte & 0xf0;
if (ip_ver == RTE_GTP_TYPE_IPV4)
*proto = rte_cpu_to_be_16(RTE_ETHER_TYPE_IPV4);
else if (ip_ver == RTE_GTP_TYPE_IPV6)
@@ -243,7 +250,6 @@ ptype_tunnel_with_udp(uint16_t *proto, const struct rte_mbuf *m,
} else {
*proto = 0;
}
- *off += gtp_len;
hdr_lens->inner_l2_len = gtp_len + sizeof(struct rte_udp_hdr);
hdr_lens->tunnel_len = gtp_len;
if (port_no == RTE_GTPC_UDP_PORT)
--
2.53.0
^ permalink raw reply related
* [PATCH v2] eal: fix data race in hugepage prefault
From: Stephen Hemminger @ 2026-06-01 16:00 UTC (permalink / raw)
To: dev
Cc: Stephen Hemminger, stable, Michal Sieron, Thomas Monjalon,
Anatoly Burakov, Bruce Richardson
In-Reply-To: <20260520170812.759638-1-stephen@networkplumber.org>
The prefault step in alloc_seg() reads a value from the hugepage and
writes it back unchanged to force the kernel to commit the backing
page. The read and write were not atomic, which races with concurrent
access to the same physical page from a secondary process attaching
to the hugetlbfs-backed mapping during rte_eal_init().
Replace the non-atomic load+store with a single atomic fetch-or of
zero. This touches the page with an atomic read-modify-write without
changing its contents, eliminating the race while preserving the
original intent of forcing a write fault.
Fixes: 0f1631be24bd ("mem: fix page fault trigger")
Cc: stable@dpdk.org
Reported-by: Michal Sieron <michal.sieron@nokia.com>
Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
.mailmap | 1 +
lib/eal/linux/eal_memalloc.c | 8 +++++---
2 files changed, 6 insertions(+), 3 deletions(-)
diff --git a/.mailmap b/.mailmap
index 43febb9030..3c45e365d3 100644
--- a/.mailmap
+++ b/.mailmap
@@ -1094,6 +1094,7 @@ Michal Mazurek <maz@semihalf.com>
Michal Michalik <michal.michalik@intel.com>
Michal Nowak <michal2.nowak@intel.com>
Michal Schmidt <mschmidt@redhat.com>
+Michal Sieron <michal.sieron@nokia.com>
Michal Swiatkowski <michal.swiatkowski@intel.com>
Michal Wilczynski <michal.wilczynski@intel.com>
Michał Mirosław <michal.miroslaw@atendesoftware.pl> <mirq-linux@rere.qmqm.pl>
diff --git a/lib/eal/linux/eal_memalloc.c b/lib/eal/linux/eal_memalloc.c
index a39bc31c7b..7359a41d3f 100644
--- a/lib/eal/linux/eal_memalloc.c
+++ b/lib/eal/linux/eal_memalloc.c
@@ -25,6 +25,7 @@
#include <linux/falloc.h>
#include <linux/mman.h> /* for hugetlb-related mmap flags */
+#include <rte_atomic.h>
#include <rte_common.h>
#include <rte_log.h>
#include <rte_eal.h>
@@ -597,10 +598,11 @@ alloc_seg(struct rte_memseg *ms, void *addr, int socket_id,
/* we need to trigger a write to the page to enforce page fault and
* ensure that page is accessible to us, but we can't overwrite value
- * that is already there, so read the old value, and write itback.
- * kernel populates the page with zeroes initially.
+ * that is already there.
+ * Use an atomic OR with zero to touch the page without changing its contents.
*/
- *(volatile int *)addr = *(volatile int *)addr;
+ (void)rte_atomic_fetch_or_explicit((__rte_atomic uint64_t *)addr, 0,
+ rte_memory_order_relaxed);
iova = rte_mem_virt2iova(addr);
if (iova == RTE_BAD_PHYS_ADDR) {
--
2.53.0
^ permalink raw reply related
* Re: [PATCH] stack: rightsize cache guard
From: Thomas Monjalon @ 2026-06-01 15:58 UTC (permalink / raw)
To: Morten Brørup; +Cc: dev
In-Reply-To: <20260504071539.252926-1-mb@smartsharesystems.com>
04/05/2026 09:15, Morten Brørup:
> Using 2 cache lines as cache guard after the table holding the stack
> elements may be too many or too few. Instead, use the number of cache
> lines specified in the build configuration.
>
> Signed-off-by: Morten Brørup <mb@smartsharesystems.com>
Applied, thanks.
^ permalink raw reply
* Re: [PATCH v4] ring: fix zero-copy burst API documentation
From: Thomas Monjalon @ 2026-06-01 15:44 UTC (permalink / raw)
To: Zhiguang Jin
Cc: Konstantin Ananyev, Wathsala Vithanage, Dharmik Thakkar,
Honnappa Nagarahalli, dev, stable
In-Reply-To: <20260521030009.23491-1-jinzhiguang@kylinos.cn>
21/05/2026 05:00, Zhiguang Jin:
> These are burst APIs relying on RTE_RING_QUEUE_VARIABLE behavior, they
> operate on a best-effort basis and return the actual number of
> objects processed (between 0 and n).
>
> Update description to match implementation.
>
> Fixes: 47bec9a5ca9f ("ring: add zero copy API")
> Cc: stable@dpdk.org
>
> Signed-off-by: Zhiguang Jin <jinzhiguang@kylinos.cn>
> Acked-by: Konstantin Ananyev <konstantin.ananyev@huawei.com>
Applied, thanks.
^ permalink raw reply
* Re: [PATCH v5 0/2] few improvemnts for SORING lib
From: Thomas Monjalon @ 2026-06-01 15:41 UTC (permalink / raw)
To: Konstantin Ananyev; +Cc: dev, wathsala.vithanage, mb
In-Reply-To: <20260505154728.82235-1-konstantin.ananyev@huawei.com>
> Konstantin Ananyev (2):
> ring: make soring to always finalize its own stage
> ring: introduce peek API for soring
Applied, thanks.
^ permalink raw reply
* Re: [PATCH v3 00/20]net/sxe2: added Linkdata sxe2 ethernet driver
From: Stephen Hemminger @ 2026-06-01 15:40 UTC (permalink / raw)
To: liujie5; +Cc: dev
In-Reply-To: <20260601084950.269887-1-liujie5@linkdatatechnology.com>
On Mon, 1 Jun 2026 16:49:30 +0800
liujie5@linkdatatechnology.com wrote:
> From: Jie Liu <liujie5@linkdatatechnology.com>
>
> This patch set implements core functionality for the SXE PMD,
> including basic driver framework, data path setup, and advanced
> offload features (VLAN, RSS, DCB, PTP etc.).
>
> V3:
> - Addressed AI comments
>
> Jie Liu (20):
> net/sxe2: support AVX512 vectorized path for Rx and Tx
> net/sxe2: add AVX2 vector data path for Rx and Tx
> drivers: add supported packet types get callback
> net/sxe2: support L2 filtering and MAC config
> drivers: support RSS feature
> net/sxe2: support TM hierarchy and shaping
> net/sxe2: support IPsec inline protocol offload
> net/sxe2: support statistics and multi-process
> drivers: interrupt handling
> net/sxe2: add NEON vec Rx/Tx burst functions
> drivers: add support for VF representors
> net/sxe2: add support for custom UDP tunnel ports
> net/sxe2: support firmware version reading
> net/sxe2: implement get monitor address
> common/sxe2: add shared SFP module definitions
> net/sxe2: support SFP module info and EEPROM access
> net/sxe2: implement private dump info
> net/sxe2: add mbuf validation in Tx debug mode
> drivers: add testpmd commands for private features
> net/sxe2: update sxe2 feature matrix docs
>
> doc/guides/nics/features/sxe2.ini | 66 +
> drivers/common/sxe2/sxe2_common.c | 156 ++
> drivers/common/sxe2/sxe2_common.h | 4 +
> drivers/common/sxe2/sxe2_flow_public.h | 633 +++++++
> drivers/common/sxe2/sxe2_ioctl_chnl.c | 178 +-
> drivers/common/sxe2/sxe2_ioctl_chnl_func.h | 18 +
> drivers/common/sxe2/sxe2_msg.h | 118 ++
> drivers/common/sxe2/sxe2_ptype.h | 1793 ++++++++++++++++++
> drivers/net/sxe2/meson.build | 56 +-
> drivers/net/sxe2/sxe2_cmd_chnl.c | 1587 +++++++++++++++-
> drivers/net/sxe2/sxe2_cmd_chnl.h | 139 ++
> drivers/net/sxe2/sxe2_drv_cmd.h | 521 +++++-
> drivers/net/sxe2/sxe2_dump.c | 304 +++
> drivers/net/sxe2/sxe2_dump.h | 12 +
> drivers/net/sxe2/sxe2_ethdev.c | 1531 +++++++++++++++-
> drivers/net/sxe2/sxe2_ethdev.h | 112 +-
> drivers/net/sxe2/sxe2_ethdev_repr.c | 610 ++++++
> drivers/net/sxe2/sxe2_ethdev_repr.h | 32 +
> drivers/net/sxe2/sxe2_filter.c | 897 +++++++++
> drivers/net/sxe2/sxe2_filter.h | 100 +
> drivers/net/sxe2/sxe2_flow.c | 1391 ++++++++++++++
> drivers/net/sxe2/sxe2_flow.h | 30 +
> drivers/net/sxe2/sxe2_flow_define.h | 144 ++
> drivers/net/sxe2/sxe2_flow_parse_action.c | 1182 ++++++++++++
> drivers/net/sxe2/sxe2_flow_parse_action.h | 23 +
> drivers/net/sxe2/sxe2_flow_parse_engine.c | 106 ++
> drivers/net/sxe2/sxe2_flow_parse_engine.h | 13 +
> drivers/net/sxe2/sxe2_flow_parse_pattern.c | 1935 ++++++++++++++++++++
> drivers/net/sxe2/sxe2_flow_parse_pattern.h | 46 +
> drivers/net/sxe2/sxe2_ipsec.c | 1565 ++++++++++++++++
> drivers/net/sxe2/sxe2_ipsec.h | 254 +++
> drivers/net/sxe2/sxe2_irq.c | 1025 +++++++++++
> drivers/net/sxe2/sxe2_irq.h | 25 +
> drivers/net/sxe2/sxe2_mac.c | 535 ++++++
> drivers/net/sxe2/sxe2_mac.h | 84 +
> drivers/net/sxe2/sxe2_mp.c | 414 +++++
> drivers/net/sxe2/sxe2_mp.h | 67 +
> drivers/net/sxe2/sxe2_queue.c | 17 +-
> drivers/net/sxe2/sxe2_rss.c | 584 ++++++
> drivers/net/sxe2/sxe2_rss.h | 81 +
> drivers/net/sxe2/sxe2_rx.c | 38 +
> drivers/net/sxe2/sxe2_rx.h | 2 +
> drivers/net/sxe2/sxe2_security.c | 335 ++++
> drivers/net/sxe2/sxe2_security.h | 77 +
> drivers/net/sxe2/sxe2_stats.c | 591 ++++++
> drivers/net/sxe2/sxe2_stats.h | 39 +
> drivers/net/sxe2/sxe2_switchdev.c | 332 ++++
> drivers/net/sxe2/sxe2_switchdev.h | 33 +
> drivers/net/sxe2/sxe2_testpmd.c | 733 ++++++++
> drivers/net/sxe2/sxe2_testpmd_lib.c | 969 ++++++++++
> drivers/net/sxe2/sxe2_testpmd_lib.h | 142 ++
> drivers/net/sxe2/sxe2_tm.c | 1169 ++++++++++++
> drivers/net/sxe2/sxe2_tm.h | 78 +
> drivers/net/sxe2/sxe2_tx.c | 7 +
> drivers/net/sxe2/sxe2_txrx.c | 176 +-
> drivers/net/sxe2/sxe2_txrx.h | 4 +
> drivers/net/sxe2/sxe2_txrx_check_mbuf.c | 595 ++++++
> drivers/net/sxe2/sxe2_txrx_check_mbuf.h | 38 +
> drivers/net/sxe2/sxe2_txrx_poll.c | 243 ++-
> drivers/net/sxe2/sxe2_txrx_vec.c | 46 +-
> drivers/net/sxe2/sxe2_txrx_vec.h | 38 +-
> drivers/net/sxe2/sxe2_txrx_vec_avx2.c | 777 ++++++++
> drivers/net/sxe2/sxe2_txrx_vec_avx512.c | 897 +++++++++
> drivers/net/sxe2/sxe2_txrx_vec_common.h | 1 +
> drivers/net/sxe2/sxe2_txrx_vec_neon.c | 707 +++++++
> drivers/net/sxe2/sxe2_vsi.c | 146 ++
> drivers/net/sxe2/sxe2_vsi.h | 12 +-
> drivers/net/sxe2/sxe2vf_regs.h | 82 +
> 68 files changed, 26571 insertions(+), 124 deletions(-)
> create mode 100644 drivers/common/sxe2/sxe2_flow_public.h
> create mode 100644 drivers/common/sxe2/sxe2_msg.h
> create mode 100644 drivers/common/sxe2/sxe2_ptype.h
> create mode 100644 drivers/net/sxe2/sxe2_dump.c
> create mode 100644 drivers/net/sxe2/sxe2_dump.h
> create mode 100644 drivers/net/sxe2/sxe2_ethdev_repr.c
> create mode 100644 drivers/net/sxe2/sxe2_ethdev_repr.h
> create mode 100644 drivers/net/sxe2/sxe2_filter.c
> create mode 100644 drivers/net/sxe2/sxe2_filter.h
> create mode 100644 drivers/net/sxe2/sxe2_flow.c
> create mode 100644 drivers/net/sxe2/sxe2_flow.h
> create mode 100644 drivers/net/sxe2/sxe2_flow_define.h
> create mode 100644 drivers/net/sxe2/sxe2_flow_parse_action.c
> create mode 100644 drivers/net/sxe2/sxe2_flow_parse_action.h
> create mode 100644 drivers/net/sxe2/sxe2_flow_parse_engine.c
> create mode 100644 drivers/net/sxe2/sxe2_flow_parse_engine.h
> create mode 100644 drivers/net/sxe2/sxe2_flow_parse_pattern.c
> create mode 100644 drivers/net/sxe2/sxe2_flow_parse_pattern.h
> create mode 100644 drivers/net/sxe2/sxe2_ipsec.c
> create mode 100644 drivers/net/sxe2/sxe2_ipsec.h
> create mode 100644 drivers/net/sxe2/sxe2_irq.c
> create mode 100644 drivers/net/sxe2/sxe2_mac.c
> create mode 100644 drivers/net/sxe2/sxe2_mac.h
> create mode 100644 drivers/net/sxe2/sxe2_mp.c
> create mode 100644 drivers/net/sxe2/sxe2_mp.h
> create mode 100644 drivers/net/sxe2/sxe2_rss.c
> create mode 100644 drivers/net/sxe2/sxe2_rss.h
> create mode 100644 drivers/net/sxe2/sxe2_security.c
> create mode 100644 drivers/net/sxe2/sxe2_security.h
> create mode 100644 drivers/net/sxe2/sxe2_stats.c
> create mode 100644 drivers/net/sxe2/sxe2_stats.h
> create mode 100644 drivers/net/sxe2/sxe2_switchdev.c
> create mode 100644 drivers/net/sxe2/sxe2_switchdev.h
> create mode 100644 drivers/net/sxe2/sxe2_testpmd.c
> create mode 100644 drivers/net/sxe2/sxe2_testpmd_lib.c
> create mode 100644 drivers/net/sxe2/sxe2_testpmd_lib.h
> create mode 100644 drivers/net/sxe2/sxe2_tm.c
> create mode 100644 drivers/net/sxe2/sxe2_tm.h
> create mode 100644 drivers/net/sxe2/sxe2_txrx_check_mbuf.c
> create mode 100644 drivers/net/sxe2/sxe2_txrx_check_mbuf.h
> create mode 100644 drivers/net/sxe2/sxe2_txrx_vec_avx2.c
> create mode 100644 drivers/net/sxe2/sxe2_txrx_vec_avx512.c
> create mode 100644 drivers/net/sxe2/sxe2_txrx_vec_neon.c
> create mode 100644 drivers/net/sxe2/sxe2vf_regs.h
>
Thanks for fixing everything from review.
But CI builds are failing with clang
FAILED: [code=1] drivers/libtmp_rte_net_sxe2.a.p/net_sxe2_sxe2_filter.c.o
ccache clang -Idrivers/libtmp_rte_net_sxe2.a.p -Idrivers -I../drivers -Idrivers/net/sxe2 -I../drivers/net/sxe2 -Idrivers/common/sxe2 -I../drivers/common/sxe2 -Ilib/ethdev -I../lib/ethdev -Ilib/eal/common -I../lib/eal/common -I. -I.. -Iconfig -I../config -Ilib/eal/include -I../lib/eal/include -Ilib/eal/linux/include -I../lib/eal/linux/include -Ilib/eal/x86/include -I../lib/eal/x86/include -I../kernel/linux -Ilib/eal -I../lib/eal -Ilib/kvargs -I../lib/kvargs -Ilib/log -I../lib/log -Ilib/metrics -I../lib/metrics -Ilib/telemetry -I../lib/telemetry -Ilib/argparse -I../lib/argparse -Ilib/net -I../lib/net -Ilib/mbuf -I../lib/mbuf -Ilib/mempool -I../lib/mempool -Ilib/ring -I../lib/ring -Ilib/meter -I../lib/meter -Idrivers/bus/pci -I../drivers/bus/pci -I../drivers/bus/pci/linux -Ilib/pci -I../lib/pci -Idrivers/bus/vdev -I../drivers/bus/vdev -Ilib/hash -I../lib/hash -Ilib/rcu -I../lib/rcu -Ilib/cryptodev -I../lib/cryptodev -Ilib/security -I../lib/security -Ilib/cmdline -I../lib/cmdline -Xclang -fcolor-diagnostics -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -Werror -std=c11 -O2 -g -include rte_config.h -Wvla -Wcast-qual -Wcomma -Wdeprecated -Wformat -Wformat-nonliteral -Wformat-security -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpointer-arith -Wshadow -Wsign-compare -Wstrict-prototypes -Wundef -Wwrite-strings -Wno-missing-field-initializers -D_GNU_SOURCE -fPIC -march=corei7 -mrtm -DALLOW_EXPERIMENTAL_API -DALLOW_INTERNAL_API -Wno-address-of-packed-member -g -DCC_AVX512_SUPPORT -DRTE_LOG_DEFAULT_LOGTYPE=pmd.net.sxe2 -DRTE_ANNOTATE_LOCKS -Wthread-safety -MD -MQ drivers/libtmp_rte_net_sxe2.a.p/net_sxe2_sxe2_filter.c.o -MF drivers/libtmp_rte_net_sxe2.a.p/net_sxe2_sxe2_filter.c.o.d -o drivers/libtmp_rte_net_sxe2.a.p/net_sxe2_sxe2_filter.c.o -c ../drivers/net/sxe2/sxe2_filter.c
../drivers/net/sxe2/sxe2_filter.c:401:1: error: expected statement
}
^
1 error generated.
^ permalink raw reply
* Re: [PATCH] ethdev: promote experimental API's to stable
From: Stephen Hemminger @ 2026-06-01 15:30 UTC (permalink / raw)
To: David Marchand; +Cc: Andrew Rybchenko, dev, Thomas Monjalon
In-Reply-To: <CAJFAV8ztviupv0rcjvL9MkBQSzY+yfak0RK8GzNbe-8s5JiyEw@mail.gmail.com>
On Mon, 1 Jun 2026 13:55:13 +0200
David Marchand <david.marchand@redhat.com> wrote:
> On Wed, 27 May 2026 at 16:44, Stephen Hemminger
> <stephen@networkplumber.org> wrote:
> > + * ``rte_eth_macaddrs_get``
>
> I am not enthousiastic on marking this stable.
> It more or less sets in stone that the mac addresses are stored in an array.
The device could store the mac addresses in any form.
Your right that right now the implementation of the API depends on dev->data->mac_addrs.
But if that changed, having API to walk the macs would be good.
Maybe rte_eth_macaddrs_get should scrub out empty zero slots now?
^ permalink raw reply
* Re: [PATCH] ethdev: promote experimental API's to stable
From: Stephen Hemminger @ 2026-06-01 15:26 UTC (permalink / raw)
To: David Marchand; +Cc: Andrew Rybchenko, dev, Thomas Monjalon
In-Reply-To: <CAJFAV8ztviupv0rcjvL9MkBQSzY+yfak0RK8GzNbe-8s5JiyEw@mail.gmail.com>
On Mon, 1 Jun 2026 13:55:13 +0200
David Marchand <david.marchand@redhat.com> wrote:
> On Wed, 27 May 2026 at 16:44, Stephen Hemminger
> <stephen@networkplumber.org> wrote:
> > + * ``rte_eth_macaddrs_get``
>
> I am not enthousiastic on marking this stable.
> It more or less sets in stone that the mac addresses are stored in an array.
So either we kill it or make it stable, having experimental API for so long
seems like a todo list that never gets done.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox