[PATCH 0/5] add versioned symbols for recently stabilized APIs

[PATCH 0/5] add versioned symbols for recently stabilized APIs

[Dariusz Sosnowski]

Main goal of this patchset is to address https://bugs.dpdk.org/show_bug.cgi?id=1957
but it also handles other recently stabilized symbols and has some minor fixes:

- Patch 1 - Fix RTE_VERSION_EXPERIMENTAL_SYMBOL macro on clang.
- Patch 2 - Allow function versioning inside drivers.
- Patch 3 - Version the function symbols stabilized in
  https://git.dpdk.org/dpdk/commit/?id=e8cab133645f5466ef75e511629add43b68a5027
- Patch 4 - Introduce versioning macros for global variable symbols.
- Patch 5 - Version the function and variable symbols stabilized in
  https://git.dpdk.org/dpdk/commit/?id=4ee2f5c1cedf9ee7f39afa667f71b07f4004ba5c

Issue is still not fully fixed for stabilized global variables:
rte_flow_dynf_metadata_offs and rte_flow_dynf_metadata_mask.
Patch 4 and 5 address the bug for these global variables,
by providing a single storage for both EXPERIMENTAL and
DPDK_26 variable symbol versions.
This is achieved through symbol aliasing.
But this solution is limited only to executables compiled with clang.

clang and gcc have a different default behavior regarding relocations
of global variables exposed by shared libraries.

With clang, R_X86_64_GLOB_DAT relocations are generated for executables:

   $ readelf -sW build-26.07/lib/librte_ethdev.so | grep rte_flow_dynf_metadata_offs
       113: 00000000000ea4c0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@@DPDK_26
       116: 00000000000ea4c0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@EXPERIMENTAL
       970: 00000000000ea4c0     4 OBJECT  LOCAL  DEFAULT   24 rte_flow_dynf_metadata_offs_impl
      1212: 00000000000ea4c0     4 OBJECT  LOCAL  DEFAULT   24 rte_flow_dynf_metadata_offs_v26
      1325: 00000000000ea4c0     4 OBJECT  LOCAL  DEFAULT   24 rte_flow_dynf_metadata_offs_exp
      1415: 00000000000ea4c0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@@DPDK_26
      1705: 00000000000ea4c0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@EXPERIMENTAL

    $ readelf -rW build-26.07/drivers/librte_net_mlx5.so | grep rte_flow_dynf_metadata_offs
    0000000003ed5f18  0000001600000006 R_X86_64_GLOB_DAT      0000000000000000 rte_flow_dynf_metadata_offs@DPDK_26 + 0

    $ readelf -rW build-25.11/app/dpdk-testpmd | grep rte_flow_dynf_metadata_offs
--> 000000000028ef70  0000011300000006 R_X86_64_GLOB_DAT      0000000000000000 rte_flow_dynf_metadata_offs@EXPERIMENTAL + 0

With gcc, R_X86_64_COPY relocations are generated:

    $ readelf -sW build-26.07/lib/librte_ethdev.so | grep rte_flow_dynf_metadata_offs
       113: 00000000000e74e0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@@DPDK_26
       116: 00000000000e74e0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@EXPERIMENTAL
      1471: 00000000000e74e0     4 OBJECT  LOCAL  DEFAULT   24 rte_flow_dynf_metadata_offs_impl
      2134: 00000000000e74e0     4 OBJECT  LOCAL  DEFAULT   24 rte_flow_dynf_metadata_offs_v26
      2247: 00000000000e74e0     4 OBJECT  LOCAL  DEFAULT   24 rte_flow_dynf_metadata_offs_exp
      2337: 00000000000e74e0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@@DPDK_26
      2627: 00000000000e74e0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@EXPERIMENTAL

    $ readelf -rW build-26.07/drivers/librte_net_mlx5.so | grep rte_flow_dynf_metadata_offs
    00000000046dbef0  0000001600000006 R_X86_64_GLOB_DAT      0000000000000000 rte_flow_dynf_metadata_offs@DPDK_26 + 0

    $ readelf -rW build-25.11/app/dpdk-testpmd | grep rte_flow_dynf_metadata_offs
--> 000000000029b540  000001d200000005 R_X86_64_COPY          000000000029b540 rte_flow_dynf_metadata_offs@EXPERIMENTAL + 0

With copy relocations (testpmd linked through gcc) the following happens:

- When variable symbol (with EXPERIMENTAL version) gets resolved inside executable,
  global variable gets copied from read-only data to executable's BSS section.
  Executable will access this variable through BSS.
- When variable symbol (with DPDK_26 version) gets resolved inside a library,
  global variable is accessed indirectly through GOT.
  It is stored inside BSS section of the shared library.

So executable and libraries refer to different storage,
eventually leading to inconsistent runtime behavior.
Problems only appears when executable and library require
different versions of global variable symbol.
If testpmd from 26.07 is used with libraries from 26.07,
GOT entry for these variables will point to copied variable.

Without copy relocations (testpmd linked through clang) both
executable and libraries access the global variable indirectly through GOT.
Runtime behavior is consistent, regardless of the mix of variable symbol versions.

The only other solution I could find was to use dlsym() inside libraries
to dynamically resolve the location rte_flow_dynf_metadata_offs and rte_flow_dynf_metadata_mask,
but this solution sounds like an overkill.
Essentially this would require moving to getter/setter functions for these variables
inside the library.

I would appreciate any feedback or suggestions if anybody had encountered a similar issue before.

Dariusz Sosnowski (5):
  eal: fix macro for versioned experimental symbol
  drivers: support function versioning
  net/mlx5: fix stabilized function versions
  eal: support aliases for versioned variable symbols
  ethdev: fix promoted flow metadata symbols

 buildtools/gen-version-map.py        | 11 ++++++++++
 drivers/meson.build                  |  8 +++++++
 drivers/net/mlx5/meson.build         |  2 ++
 drivers/net/mlx5/mlx5_driver_event.c | 22 ++++++++++++++-----
 drivers/net/mlx5/mlx5_flow.c         | 18 ++++++++++-----
 lib/eal/common/eal_export.h          | 24 +++++++++++++++++++-
 lib/ethdev/meson.build               |  2 ++
 lib/ethdev/rte_flow.c                | 33 ++++++++++++++++++----------
 8 files changed, 96 insertions(+), 24 deletions(-)

--
2.47.3

[PATCH 1/5] eal: fix macro for versioned experimental symbol

[Dariusz Sosnowski]

Add a missing semicolon after __asm__ block in
RTE_VERSION_EXPERIMENTAL_SYMBOL macro.
It's lack triggers the following compilation error with clang:

    ../lib/ethdev/rte_flow.c:320:1: error: expected ';' after top-level asm block
      320 | RTE_VERSION_EXPERIMENTAL_SYMBOL(int, rte_flow_dynf_metadata_register, (void))
          | ^
    ../lib/eal/common/eal_export.h:75:74: note: expanded from macro 'RTE_VERSION_EXPERIMENTAL_SYMBOL'
       75 | __asm__(".symver " RTE_STR(name) "_exp, " RTE_STR(name) "@EXPERIMENTAL") \
          |                                                                          ^
    ../lib/eal/include/rte_common.h:237:20: note: expanded from macro '\
    __rte_used'
      237 | #define __rte_used __attribute__((used))
          |                    ^

Fixes: e30e194c4d06 ("eal: rework function versioning macros")
Cc: david.marchand@redhat.com

Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
---
 lib/eal/common/eal_export.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/eal/common/eal_export.h b/lib/eal/common/eal_export.h
index 888fd9f9ed..7971bf8d7a 100644
--- a/lib/eal/common/eal_export.h
+++ b/lib/eal/common/eal_export.h
@@ -72,7 +72,7 @@ __rte_used type name ## _v ## ver args; \
 type name ## _v ## ver args
 
 #define RTE_VERSION_EXPERIMENTAL_SYMBOL(type, name, args) VERSIONING_WARN \
-__asm__(".symver " RTE_STR(name) "_exp, " RTE_STR(name) "@EXPERIMENTAL") \
+__asm__(".symver " RTE_STR(name) "_exp, " RTE_STR(name) "@EXPERIMENTAL"); \
 __rte_used type name ## _exp args; \
 type name ## _exp args
 
-- 
2.47.3

[PATCH 2/5] drivers: support function versioning

[Dariusz Sosnowski]

Add support for enabling function versioning
(through use_function_versioning meson variable) for drivers,
similar to libraries.

Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
---
 drivers/meson.build | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/drivers/meson.build b/drivers/meson.build
index 4d95604ecd..a63d93372a 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -171,6 +171,7 @@ foreach subpath:subdirs
         pkgconfig_extra_libs = []
         testpmd_sources = []
         require_iova_in_mbuf = true
+        use_function_versioning = false
         # for handling base code files which may need extra cflags
         base_sources = []
         base_cflags = []
@@ -273,6 +274,13 @@ foreach subpath:subdirs
         endif
         dpdk_conf.set(lib_name.to_upper(), 1)

+        if developer_mode and is_windows and use_function_versioning
+            message('@0@: Function versioning is not supported by Windows.'.format(name))
+        endif
+        if use_function_versioning
+            cflags += '-DRTE_USE_FUNCTION_VERSIONING'
+        endif
+
         dpdk_extra_ldflags += pkgconfig_extra_libs

         dpdk_headers += headers
--
2.47.3

[PATCH 3/5] net/mlx5: fix stabilized function versions

[Dariusz Sosnowski]

Offending patch stabilized the following function symbols:

- rte_pmd_mlx5_driver_event_cb_register
- rte_pmd_mlx5_driver_event_cb_unregister
- rte_pmd_mlx5_enable_steering
- rte_pmd_mlx5_disable_steering

These function symbols were introduced in 25.11.
Any application using these functions, linked against 25.11 version,
would fail when used with 26.07 libraries, because only DPDK_26 versions
of these symbols were exported.

This patch fixes that by adding proper function symbol versioning
to these symbols.

Fixes: e8cab133645f ("net/mlx5: promote some private API to stable")

Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
---
 drivers/net/mlx5/meson.build         |  2 ++
 drivers/net/mlx5/mlx5_driver_event.c | 22 ++++++++++++++++------
 drivers/net/mlx5/mlx5_flow.c         | 18 ++++++++++++------
 3 files changed, 30 insertions(+), 12 deletions(-)

diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build
index 82a7dfe782..0fa6322779 100644
--- a/drivers/net/mlx5/meson.build
+++ b/drivers/net/mlx5/meson.build
@@ -2,6 +2,8 @@
 # Copyright 2018 6WIND S.A.
 # Copyright 2018 Mellanox Technologies, Ltd
 
+use_function_versioning = true
+
 if not (is_linux or is_windows)
     build = false
     reason = 'only supported on Linux and Windows'
diff --git a/drivers/net/mlx5/mlx5_driver_event.c b/drivers/net/mlx5/mlx5_driver_event.c
index 89e49331c8..d0e22d6151 100644
--- a/drivers/net/mlx5/mlx5_driver_event.c
+++ b/drivers/net/mlx5/mlx5_driver_event.c
@@ -236,9 +236,8 @@ notify_existing_devices(rte_pmd_mlx5_driver_event_callback_t cb, void *opaque)
 		notify_existing_queues(port_id, cb, opaque);
 }
 
-RTE_EXPORT_SYMBOL(rte_pmd_mlx5_driver_event_cb_register)
-int
-rte_pmd_mlx5_driver_event_cb_register(rte_pmd_mlx5_driver_event_callback_t cb, void *opaque)
+RTE_DEFAULT_SYMBOL(26, int, rte_pmd_mlx5_driver_event_cb_register,
+		   (rte_pmd_mlx5_driver_event_callback_t cb, void *opaque))
 {
 	struct registered_cb *r;
 
@@ -264,9 +263,14 @@ rte_pmd_mlx5_driver_event_cb_register(rte_pmd_mlx5_driver_event_callback_t cb, v
 	return 0;
 }
 
-RTE_EXPORT_SYMBOL(rte_pmd_mlx5_driver_event_cb_unregister)
-int
-rte_pmd_mlx5_driver_event_cb_unregister(rte_pmd_mlx5_driver_event_callback_t cb)
+RTE_VERSION_EXPERIMENTAL_SYMBOL(int, rte_pmd_mlx5_driver_event_cb_register,
+				(rte_pmd_mlx5_driver_event_callback_t cb, void *opaque))
+{
+	return rte_pmd_mlx5_driver_event_cb_register(cb, opaque);
+}
+
+RTE_DEFAULT_SYMBOL(26, int, rte_pmd_mlx5_driver_event_cb_unregister,
+		   (rte_pmd_mlx5_driver_event_callback_t cb))
 {
 	struct registered_cb *r;
 	bool found = false;
@@ -289,6 +293,12 @@ rte_pmd_mlx5_driver_event_cb_unregister(rte_pmd_mlx5_driver_event_callback_t cb)
 	return 0;
 }
 
+RTE_VERSION_EXPERIMENTAL_SYMBOL(int, rte_pmd_mlx5_driver_event_cb_unregister,
+				(rte_pmd_mlx5_driver_event_callback_t cb))
+{
+	return rte_pmd_mlx5_driver_event_cb_unregister(cb);
+}
+
 RTE_FINI(rte_pmd_mlx5_driver_event_cb_cleanup) {
 	struct registered_cb *r;
 
diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c
index a95dd9dc94..4b984df892 100644
--- a/drivers/net/mlx5/mlx5_flow.c
+++ b/drivers/net/mlx5/mlx5_flow.c
@@ -12506,9 +12506,7 @@ flow_disable_steering_run_on_related(struct rte_eth_dev *dev,
 	}
 }
 
-RTE_EXPORT_SYMBOL(rte_pmd_mlx5_disable_steering)
-void
-rte_pmd_mlx5_disable_steering(void)
+RTE_DEFAULT_SYMBOL(26, void, rte_pmd_mlx5_disable_steering, (void))
 {
 	uint16_t port_id;
 
@@ -12532,9 +12530,12 @@ rte_pmd_mlx5_disable_steering(void)
 	mlx5_steering_disabled = true;
 }
 
-RTE_EXPORT_SYMBOL(rte_pmd_mlx5_enable_steering)
-int
-rte_pmd_mlx5_enable_steering(void)
+RTE_VERSION_EXPERIMENTAL_SYMBOL(void, rte_pmd_mlx5_disable_steering, (void))
+{
+	rte_pmd_mlx5_disable_steering();
+}
+
+RTE_DEFAULT_SYMBOL(26, int, rte_pmd_mlx5_enable_steering, (void))
 {
 	uint16_t port_id;
 
@@ -12551,6 +12552,11 @@ rte_pmd_mlx5_enable_steering(void)
 	return 0;
 }
 
+RTE_VERSION_EXPERIMENTAL_SYMBOL(int, rte_pmd_mlx5_enable_steering, (void))
+{
+	return rte_pmd_mlx5_enable_steering();
+}
+
 bool
 mlx5_vport_rx_metadata_passing_enabled(const struct mlx5_dev_ctx_shared *sh)
 {
-- 
2.47.3

[PATCH 4/5] eal: support aliases for versioned variable symbols

[Dariusz Sosnowski]

Existing symbol versioning macros are not suitable for versioning
exported global variables.

Specifically, if existing macros are used for versioning
global variable symbol promoted from experimental to stable,
result would be multiple variables with separate storage defined.
If an application was linked against older DPDK and had copy
relocations, this would yield an inconsistent behavior:

- Application would use experimental symbol version,
  with storage set up in BSS section in application.
- Library would use latest symbol version,
  with storage set up in BSS section of shared object.

This patch adds versioning macros which utilize symbol aliasing.
Specifically, a new variable (with version suffix) is defined
as an alias to private (static) variable inside the library.
Variable symbol versions are attached to these alias variables.

Following macros are added:

- RTE_VERSION_EXPERIMENTAL_SYMBOL_ALIAS
- RTE_DEFAULT_SYMBOL_ALIAS

Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
---
 buildtools/gen-version-map.py | 11 +++++++++++
 lib/eal/common/eal_export.h   | 22 ++++++++++++++++++++++
 2 files changed, 33 insertions(+)

diff --git a/buildtools/gen-version-map.py b/buildtools/gen-version-map.py
index 57e08a8c0f..aa88e69179 100755
--- a/buildtools/gen-version-map.py
+++ b/buildtools/gen-version-map.py
@@ -14,8 +14,12 @@
 export_int_sym_regexp = re.compile(r"^RTE_EXPORT_INTERNAL_SYMBOL\(([^)]+)\)")
 export_sym_regexp = re.compile(r"^RTE_EXPORT_SYMBOL\(([^)]+)\)")
 ver_sym_regexp = re.compile(r"^RTE_VERSION_SYMBOL\(([^,]+), [^,]+, ([^,]+),")
+
 ver_exp_sym_regexp = re.compile(r"^RTE_VERSION_EXPERIMENTAL_SYMBOL\([^,]+, ([^,]+),")
+ver_exp_sym_alias_regexp = re.compile(r"^RTE_VERSION_EXPERIMENTAL_SYMBOL_ALIAS\([^,]+, ([^,]+),")
+
 default_sym_regexp = re.compile(r"^RTE_DEFAULT_SYMBOL\(([^,]+), [^,]+, ([^,]+),")
+default_sym_alias_regexp = re.compile(r"^RTE_DEFAULT_SYMBOL_ALIAS\(([^,]+), [^,]+, ([^,]+),")
 
 parser = argparse.ArgumentParser(
     description=__doc__,
@@ -73,10 +77,17 @@
         elif ver_exp_sym_regexp.match(ln):
             node = "EXPERIMENTAL"
             symbol = ver_exp_sym_regexp.match(ln).group(1)
+        elif ver_exp_sym_alias_regexp.match(ln):
+            node = "EXPERIMENTAL"
+            symbol = ver_exp_sym_alias_regexp.match(ln).group(1)
         elif default_sym_regexp.match(ln):
             abi = default_sym_regexp.match(ln).group(1)
             node = f"DPDK_{abi}"
             symbol = default_sym_regexp.match(ln).group(2)
+        elif default_sym_alias_regexp.match(ln):
+            abi = default_sym_alias_regexp.match(ln).group(1)
+            node = f"DPDK_{abi}"
+            symbol = default_sym_alias_regexp.match(ln).group(2)
 
         if not symbol:
             continue
diff --git a/lib/eal/common/eal_export.h b/lib/eal/common/eal_export.h
index 7971bf8d7a..5b458f81c6 100644
--- a/lib/eal/common/eal_export.h
+++ b/lib/eal/common/eal_export.h
@@ -63,6 +63,14 @@ __attribute__((__symver__(RTE_STR(name) "@@DPDK_" RTE_STR(ver)))) \
 type name ## _v ## ver args; \
 type name ## _v ## ver args
 
+#define RTE_VERSION_EXPERIMENTAL_SYMBOL_ALIAS(type, name, orig) VERSIONING_WARN \
+extern type name ## _exp __attribute((alias(RTE_STR(orig)), \
+				      __symver__(RTE_STR(name) "@EXPERIMENTAL")))
+
+#define RTE_DEFAULT_SYMBOL_ALIAS(ver, type, name, orig) VERSIONING_WARN \
+extern type name ## _v ## ver __attribute((alias(RTE_STR(orig)), \
+					   __symver__(RTE_STR(name) "@@DPDK_" RTE_STR(ver))))
+
 #else /* !__has_attribute(symver) */
 
 /* Use asm tag to create symbol table entry */
@@ -81,6 +89,14 @@ __asm__(".symver " RTE_STR(name) "_v" RTE_STR(ver) ", " RTE_STR(name) "@@DPDK_"
 __rte_used type name ## _v ## ver args; \
 type name ## _v ## ver args
 
+#define RTE_DEFAULT_SYMBOL_ALIAS(ver, type, name, orig) VERSIONING_WARN \
+extern type name ## _v ## ver __attribute__((alias(RTE_STR(orig)))); \
+__asm__(".symver " RTE_STR(name) "_v" RTE_STR(ver) ", " RTE_STR(name) "@@DPDK_" RTE_STR(ver));
+
+#define RTE_VERSION_EXPERIMENTAL_SYMBOL_ALIAS(type, name, orig) VERSIONING_WARN \
+extern type name ## _exp __attribute__((alias(RTE_STR(orig)))); \
+__asm__(".symver " RTE_STR(name) "_exp, " RTE_STR(name) "@EXPERIMENTAL");
+
 #endif /* __has_attribute(symver) */
 
 #else /* !RTE_BUILD_SHARED_LIB */
@@ -97,6 +113,12 @@ type name ## _exp args
 #define RTE_DEFAULT_SYMBOL(ver, type, name, args) VERSIONING_WARN \
 type name args
 
+#define RTE_VERSION_EXPERIMENTAL_SYMBOL_ALIAS(type, name, orig) VERSIONING_WARN \
+extern type name ## _exp __attribute__((alias(RTE_STR(orig))));
+
+#define RTE_DEFAULT_SYMBOL_ALIAS(ver, type, name, orig) VERSIONING_WARN \
+extern type name __attribute__((alias(RTE_STR(orig))));
+
 #endif /* RTE_BUILD_SHARED_LIB */
 
 #endif /* EAL_EXPORT_H */
-- 
2.47.3

[PATCH 5/5] ethdev: fix promoted flow metadata symbols

[Dariusz Sosnowski]

Offending patch stabilized the following symbols:

- 1 function symbol:
    - rte_flow_dynf_metadata_register
- 2 global variable symbols:
    - rte_flow_dynf_metadata_offs
    - rte_flow_dynf_metadata_mask

Any application using these flow metadata symbols,
which was linked dynamically against 25.11 version of ethdev
library and using current version of ethdev library
would fail on symbol resolution, because EXPERIMENTAL versions
were not exported.
Specifically, on application start up
variable symbol lookup error happens:

/tmp/dpdk-25.11/usr/local/bin/dpdk-testpmd:
  symbol lookup error: /tmp/dpdk-25.11/usr/local/bin/dpdk-testpmd:
    undefined symbol: rte_flow_dynf_metadata_offs, version EXPERIMENTAL

This error occurss because symbol lookup for global variables
happens on application startup.

This patch addresses that by adding versioned aliases
for the following variable symbols:

- rte_flow_dynf_metadata_offs
- rte_flow_dynf_metadata_mask

Versioned function symbols are also added
for rte_flow_dynf_metadata_register().

Bugzilla ID: 1957
Fixes: 4ee2f5c1cedf ("ethdev: promote flow metadata API to stable")

Reported-by: Yu Jiang <yux.jiang@intel.com>
Signed-off-by: Dariusz Sosnowski <dsosnowski@nvidia.com>
---
 lib/ethdev/meson.build |  2 ++
 lib/ethdev/rte_flow.c  | 33 ++++++++++++++++++++++-----------
 2 files changed, 24 insertions(+), 11 deletions(-)

diff --git a/lib/ethdev/meson.build b/lib/ethdev/meson.build
index 8ba6c708a2..63fd866af9 100644
--- a/lib/ethdev/meson.build
+++ b/lib/ethdev/meson.build
@@ -1,6 +1,8 @@
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2017 Intel Corporation
 
+use_function_versioning = true
+
 sources = files(
         'ethdev_driver.c',
         'ethdev_private.c',
diff --git a/lib/ethdev/rte_flow.c b/lib/ethdev/rte_flow.c
index ec0fe08355..a8c01ffe8a 100644
--- a/lib/ethdev/rte_flow.c
+++ b/lib/ethdev/rte_flow.c
@@ -23,12 +23,20 @@
 #define FLOW_LOG RTE_ETHDEV_LOG_LINE
 
 /* Mbuf dynamic field name for metadata. */
-RTE_EXPORT_SYMBOL(rte_flow_dynf_metadata_offs)
-int32_t rte_flow_dynf_metadata_offs = -1;
+static int32_t rte_flow_dynf_metadata_offs_impl = -1;
+
+RTE_DEFAULT_SYMBOL_ALIAS(26, int32_t, rte_flow_dynf_metadata_offs,
+			 rte_flow_dynf_metadata_offs_impl);
+RTE_VERSION_EXPERIMENTAL_SYMBOL_ALIAS(int32_t, rte_flow_dynf_metadata_offs,
+				      rte_flow_dynf_metadata_offs_impl);
 
 /* Mbuf dynamic field flag bit number for metadata. */
-RTE_EXPORT_SYMBOL(rte_flow_dynf_metadata_mask)
-uint64_t rte_flow_dynf_metadata_mask;
+static uint64_t rte_flow_dynf_metadata_mask_impl = 0;
+
+RTE_DEFAULT_SYMBOL_ALIAS(26, uint64_t, rte_flow_dynf_metadata_mask,
+			 rte_flow_dynf_metadata_mask_impl);
+RTE_VERSION_EXPERIMENTAL_SYMBOL_ALIAS(uint64_t, rte_flow_dynf_metadata_mask,
+				      rte_flow_dynf_metadata_mask_impl);
 
 /**
  * Flow elements description tables.
@@ -281,9 +289,7 @@ static const struct rte_flow_desc_data rte_flow_desc_action[] = {
 	MK_FLOW_ACTION(JUMP_TO_TABLE_INDEX, sizeof(struct rte_flow_action_jump_to_table_index)),
 };
 
-RTE_EXPORT_SYMBOL(rte_flow_dynf_metadata_register)
-int
-rte_flow_dynf_metadata_register(void)
+RTE_DEFAULT_SYMBOL(26, int, rte_flow_dynf_metadata_register, (void))
 {
 	int offset;
 	int flag;
@@ -303,19 +309,24 @@ rte_flow_dynf_metadata_register(void)
 	flag = rte_mbuf_dynflag_register(&desc_flag);
 	if (flag < 0)
 		goto error;
-	rte_flow_dynf_metadata_offs = offset;
-	rte_flow_dynf_metadata_mask = RTE_BIT64(flag);
+	rte_flow_dynf_metadata_offs_impl = offset;
+	rte_flow_dynf_metadata_mask_impl = RTE_BIT64(flag);
 
 	rte_flow_trace_dynf_metadata_register(offset, RTE_BIT64(flag));
 
 	return 0;
 
 error:
-	rte_flow_dynf_metadata_offs = -1;
-	rte_flow_dynf_metadata_mask = UINT64_C(0);
+	rte_flow_dynf_metadata_offs_impl = -1;
+	rte_flow_dynf_metadata_mask_impl = UINT64_C(0);
 	return -rte_errno;
 }
 
+RTE_VERSION_EXPERIMENTAL_SYMBOL(int, rte_flow_dynf_metadata_register, (void))
+{
+	return rte_flow_dynf_metadata_register();
+}
+
 static inline void
 fts_enter(struct rte_eth_dev *dev)
 {
-- 
2.47.3

Re: [PATCH 0/5] add versioned symbols for recently stabilized APIs

[Stephen Hemminger]

On Tue, 23 Jun 2026 13:37:46 +0200
Dariusz Sosnowski <dsosnowski@nvidia.com> wrote:

> Main goal of this patchset is to address https://bugs.dpdk.org/show_bug.cgi?id=1957
> but it also handles other recently stabilized symbols and has some minor fixes:
> 
> - Patch 1 - Fix RTE_VERSION_EXPERIMENTAL_SYMBOL macro on clang.
> - Patch 2 - Allow function versioning inside drivers.
> - Patch 3 - Version the function symbols stabilized in
>   https://git.dpdk.org/dpdk/commit/?id=e8cab133645f5466ef75e511629add43b68a5027
> - Patch 4 - Introduce versioning macros for global variable symbols.
> - Patch 5 - Version the function and variable symbols stabilized in
>   https://git.dpdk.org/dpdk/commit/?id=4ee2f5c1cedf9ee7f39afa667f71b07f4004ba5c
> 
> Issue is still not fully fixed for stabilized global variables:
> rte_flow_dynf_metadata_offs and rte_flow_dynf_metadata_mask.
> Patch 4 and 5 address the bug for these global variables,
> by providing a single storage for both EXPERIMENTAL and
> DPDK_26 variable symbol versions.
> This is achieved through symbol aliasing.
> But this solution is limited only to executables compiled with clang.
> 
> clang and gcc have a different default behavior regarding relocations
> of global variables exposed by shared libraries.
> 
> With clang, R_X86_64_GLOB_DAT relocations are generated for executables:
> 
>    $ readelf -sW build-26.07/lib/librte_ethdev.so | grep rte_flow_dynf_metadata_offs
>        113: 00000000000ea4c0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@@DPDK_26
>        116: 00000000000ea4c0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@EXPERIMENTAL
>        970: 00000000000ea4c0     4 OBJECT  LOCAL  DEFAULT   24 rte_flow_dynf_metadata_offs_impl
>       1212: 00000000000ea4c0     4 OBJECT  LOCAL  DEFAULT   24 rte_flow_dynf_metadata_offs_v26
>       1325: 00000000000ea4c0     4 OBJECT  LOCAL  DEFAULT   24 rte_flow_dynf_metadata_offs_exp
>       1415: 00000000000ea4c0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@@DPDK_26
>       1705: 00000000000ea4c0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@EXPERIMENTAL
> 
>     $ readelf -rW build-26.07/drivers/librte_net_mlx5.so | grep rte_flow_dynf_metadata_offs
>     0000000003ed5f18  0000001600000006 R_X86_64_GLOB_DAT      0000000000000000 rte_flow_dynf_metadata_offs@DPDK_26 + 0
> 
>     $ readelf -rW build-25.11/app/dpdk-testpmd | grep rte_flow_dynf_metadata_offs
> --> 000000000028ef70  0000011300000006 R_X86_64_GLOB_DAT      0000000000000000 rte_flow_dynf_metadata_offs@EXPERIMENTAL + 0  
> 
> With gcc, R_X86_64_COPY relocations are generated:
> 
>     $ readelf -sW build-26.07/lib/librte_ethdev.so | grep rte_flow_dynf_metadata_offs
>        113: 00000000000e74e0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@@DPDK_26
>        116: 00000000000e74e0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@EXPERIMENTAL
>       1471: 00000000000e74e0     4 OBJECT  LOCAL  DEFAULT   24 rte_flow_dynf_metadata_offs_impl
>       2134: 00000000000e74e0     4 OBJECT  LOCAL  DEFAULT   24 rte_flow_dynf_metadata_offs_v26
>       2247: 00000000000e74e0     4 OBJECT  LOCAL  DEFAULT   24 rte_flow_dynf_metadata_offs_exp
>       2337: 00000000000e74e0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@@DPDK_26
>       2627: 00000000000e74e0     4 OBJECT  GLOBAL DEFAULT   24 rte_flow_dynf_metadata_offs@EXPERIMENTAL
> 
>     $ readelf -rW build-26.07/drivers/librte_net_mlx5.so | grep rte_flow_dynf_metadata_offs
>     00000000046dbef0  0000001600000006 R_X86_64_GLOB_DAT      0000000000000000 rte_flow_dynf_metadata_offs@DPDK_26 + 0
> 
>     $ readelf -rW build-25.11/app/dpdk-testpmd | grep rte_flow_dynf_metadata_offs
> --> 000000000029b540  000001d200000005 R_X86_64_COPY          000000000029b540 rte_flow_dynf_metadata_offs@EXPERIMENTAL + 0  
> 
> With copy relocations (testpmd linked through gcc) the following happens:
> 
> - When variable symbol (with EXPERIMENTAL version) gets resolved inside executable,
>   global variable gets copied from read-only data to executable's BSS section.
>   Executable will access this variable through BSS.
> - When variable symbol (with DPDK_26 version) gets resolved inside a library,
>   global variable is accessed indirectly through GOT.
>   It is stored inside BSS section of the shared library.
> 
> So executable and libraries refer to different storage,
> eventually leading to inconsistent runtime behavior.
> Problems only appears when executable and library require
> different versions of global variable symbol.
> If testpmd from 26.07 is used with libraries from 26.07,
> GOT entry for these variables will point to copied variable.
> 
> Without copy relocations (testpmd linked through clang) both
> executable and libraries access the global variable indirectly through GOT.
> Runtime behavior is consistent, regardless of the mix of variable symbol versions.
> 
> The only other solution I could find was to use dlsym() inside libraries
> to dynamically resolve the location rte_flow_dynf_metadata_offs and rte_flow_dynf_metadata_mask,
> but this solution sounds like an overkill.
> Essentially this would require moving to getter/setter functions for these variables
> inside the library.
> 
> I would appreciate any feedback or suggestions if anybody had encountered a similar issue before.
> 
> Dariusz Sosnowski (5):
>   eal: fix macro for versioned experimental symbol
>   drivers: support function versioning
>   net/mlx5: fix stabilized function versions
>   eal: support aliases for versioned variable symbols
>   ethdev: fix promoted flow metadata symbols
> 
>  buildtools/gen-version-map.py        | 11 ++++++++++
>  drivers/meson.build                  |  8 +++++++
>  drivers/net/mlx5/meson.build         |  2 ++
>  drivers/net/mlx5/mlx5_driver_event.c | 22 ++++++++++++++-----
>  drivers/net/mlx5/mlx5_flow.c         | 18 ++++++++++-----
>  lib/eal/common/eal_export.h          | 24 +++++++++++++++++++-
>  lib/ethdev/meson.build               |  2 ++
>  lib/ethdev/rte_flow.c                | 33 ++++++++++++++++++----------
>  8 files changed, 96 insertions(+), 24 deletions(-)
> 
> --
> 2.47.3
> 

The bugfix is good, but not sure the rest is needed right now.
It is getting late to add more stuff for 26.07 and in 26.11 function versioning
will not be needed.

Re: [PATCH 1/5] eal: fix macro for versioned experimental symbol

[Stephen Hemminger]

On Tue, 23 Jun 2026 13:37:47 +0200
Dariusz Sosnowski <dsosnowski@nvidia.com> wrote:

> Add a missing semicolon after __asm__ block in
> RTE_VERSION_EXPERIMENTAL_SYMBOL macro.
> It's lack triggers the following compilation error with clang:
> 
>     ../lib/ethdev/rte_flow.c:320:1: error: expected ';' after top-level asm block
>       320 | RTE_VERSION_EXPERIMENTAL_SYMBOL(int, rte_flow_dynf_metadata_register, (void))
>           | ^
>     ../lib/eal/common/eal_export.h:75:74: note: expanded from macro 'RTE_VERSION_EXPERIMENTAL_SYMBOL'
>        75 | __asm__(".symver " RTE_STR(name) "_exp, " RTE_STR(name) "@EXPERIMENTAL") \
>           |                                                                          ^
>     ../lib/eal/include/rte_common.h:237:20: note: expanded from macro '\
>     __rte_used'
>       237 | #define __rte_used __attribute__((used))
>           |                    ^
> 
> Fixes: e30e194c4d06 ("eal: rework function versioning macros")
> Cc: david.marchand@redhat.com
> 
> Signed-

I didn't see this because clang doesn't have symver support.
Which version of clang is this?

Re: [PATCH 0/5] add versioned symbols for recently stabilized APIs

[David Marchand]

Hello Dariusz,

On Tue, 23 Jun 2026 at 13:38, Dariusz Sosnowski <dsosnowski@nvidia.com> wrote:
>
> Main goal of this patchset is to address https://bugs.dpdk.org/show_bug.cgi?id=1957

It is expected that experimental symbols may disappear overnight, and
this bug could also be closed as NOTABUG.

On the other hand, we do state in the doc that compatibility could be
provided when stabilising an experimental API, so ok.. let's try.

> but it also handles other recently stabilized symbols and has some minor fixes:
>
> - Patch 1 - Fix RTE_VERSION_EXPERIMENTAL_SYMBOL macro on clang.

Ouch... /me hides.


> - Patch 2 - Allow function versioning inside drivers.
> - Patch 3 - Version the function symbols stabilized in
>   https://git.dpdk.org/dpdk/commit/?id=e8cab133645f5466ef75e511629add43b68a5027
> - Patch 4 - Introduce versioning macros for global variable symbols.
> - Patch 5 - Version the function and variable symbols stabilized in
>   https://git.dpdk.org/dpdk/commit/?id=4ee2f5c1cedf9ee7f39afa667f71b07f4004ba5c
>
> Issue is still not fully fixed for stabilized global variables:
> rte_flow_dynf_metadata_offs and rte_flow_dynf_metadata_mask.

Well, symbol versioning is not something for variables.
Exposing global variables was a mistake from the start...
Those were exported for "performance" reasons as those are accessed
via inline helpers (but I am not sure there were benchmarks showing
the benefits).

I am for forbidding exports of global variables from now, unless some
really good performance benchmark is provided (@techboard for info).


Now, in practice for your issue, rather than reintroducing symbol
aliases (technical solution that I dropped when refactoring the
macros), I think we can do with some middle ground approach:
- leaving the inline helpers as "stable" (not __rte_experimental),
- restoring the EXPERIMENTAL version on the global variables, this
will restore the location of those symbols from the previous ABI pov,
and the checks won't catch this discrepancy anyway,
- during 26.11, drop the EXPERIMENTAL version on those variables,


In other words, stopping at your patch 3 of the series, then adding:

$ git diff
diff --git a/lib/ethdev/rte_flow.c b/lib/ethdev/rte_flow.c
index ec0fe08355..8bd21ccd31 100644
--- a/lib/ethdev/rte_flow.c
+++ b/lib/ethdev/rte_flow.c
@@ -23,11 +23,11 @@
 #define FLOW_LOG RTE_ETHDEV_LOG_LINE

 /* Mbuf dynamic field name for metadata. */
-RTE_EXPORT_SYMBOL(rte_flow_dynf_metadata_offs)
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_flow_dynf_metadata_offs, 19.11)
 int32_t rte_flow_dynf_metadata_offs = -1;

 /* Mbuf dynamic field flag bit number for metadata. */
-RTE_EXPORT_SYMBOL(rte_flow_dynf_metadata_mask)
+RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_flow_dynf_metadata_mask, 19.11)
 uint64_t rte_flow_dynf_metadata_mask;

 /**

> Patch 4 and 5 address the bug for these global variables,
> by providing a single storage for both EXPERIMENTAL and
> DPDK_26 variable symbol versions.
> This is achieved through symbol aliasing.
> But this solution is limited only to executables compiled with clang.
>
> clang and gcc have a different default behavior regarding relocations
> of global variables exposed by shared libraries.
>

Yeah... not even thinking about adding MSVC in the list...


-- 
David Marchand

Re: [PATCH 1/5] eal: fix macro for versioned experimental symbol

[Dariusz Sosnowski]

On Tue, Jun 23, 2026 at 06:50:00AM -0700, Stephen Hemminger wrote:
> On Tue, 23 Jun 2026 13:37:47 +0200
> Dariusz Sosnowski <dsosnowski@nvidia.com> wrote:
> 
> > Add a missing semicolon after __asm__ block in
> > RTE_VERSION_EXPERIMENTAL_SYMBOL macro.
> > It's lack triggers the following compilation error with clang:
> > 
> >     ../lib/ethdev/rte_flow.c:320:1: error: expected ';' after top-level asm block
> >       320 | RTE_VERSION_EXPERIMENTAL_SYMBOL(int, rte_flow_dynf_metadata_register, (void))
> >           | ^
> >     ../lib/eal/common/eal_export.h:75:74: note: expanded from macro 'RTE_VERSION_EXPERIMENTAL_SYMBOL'
> >        75 | __asm__(".symver " RTE_STR(name) "_exp, " RTE_STR(name) "@EXPERIMENTAL") \
> >           |                                                                          ^
> >     ../lib/eal/include/rte_common.h:237:20: note: expanded from macro '\
> >     __rte_used'
> >       237 | #define __rte_used __attribute__((used))
> >           |                    ^
> > 
> > Fixes: e30e194c4d06 ("eal: rework function versioning macros")
> > Cc: david.marchand@redhat.com
> > 
> > Signed-
> 
> I didn't see this because clang doesn't have symver support.
> Which version of clang is this?

clang 19 available on Debian 13:

  $ clang --version
  Debian clang version 19.1.7 (3+b1)
  Target: x86_64-pc-linux-gnu
  Thread model: posix
  InstalledDir: /usr/lib/llvm-19/bin

Re: [PATCH 0/5] add versioned symbols for recently stabilized APIs

[Dariusz Sosnowski]

On Tue, Jun 23, 2026 at 03:50:52PM +0200, David Marchand wrote:
> Hello Dariusz,
> 
> On Tue, 23 Jun 2026 at 13:38, Dariusz Sosnowski <dsosnowski@nvidia.com> wrote:
> >
> > Main goal of this patchset is to address https://bugs.dpdk.org/show_bug.cgi?id=1957
> 
> It is expected that experimental symbols may disappear overnight, and
> this bug could also be closed as NOTABUG.
> 
> On the other hand, we do state in the doc that compatibility could be
> provided when stabilising an experimental API, so ok.. let's try.
> 
> > but it also handles other recently stabilized symbols and has some minor fixes:
> >
> > - Patch 1 - Fix RTE_VERSION_EXPERIMENTAL_SYMBOL macro on clang.
> 
> Ouch... /me hides.
> 
> 
> > - Patch 2 - Allow function versioning inside drivers.
> > - Patch 3 - Version the function symbols stabilized in
> >   https://git.dpdk.org/dpdk/commit/?id=e8cab133645f5466ef75e511629add43b68a5027
> > - Patch 4 - Introduce versioning macros for global variable symbols.
> > - Patch 5 - Version the function and variable symbols stabilized in
> >   https://git.dpdk.org/dpdk/commit/?id=4ee2f5c1cedf9ee7f39afa667f71b07f4004ba5c
> >
> > Issue is still not fully fixed for stabilized global variables:
> > rte_flow_dynf_metadata_offs and rte_flow_dynf_metadata_mask.
> 
> Well, symbol versioning is not something for variables.
> Exposing global variables was a mistake from the start...

After fighting with this issue for some time,
I am coming to the similar conclusion :)

> Those were exported for "performance" reasons as those are accessed
> via inline helpers (but I am not sure there were benchmarks showing
> the benefits).
> 
> I am for forbidding exports of global variables from now, unless some
> really good performance benchmark is provided (@techboard for info).

Sounds like a good proposal IMO.
Especially since, from a quick glance, almost all existing variables
expose values not expected to change frequently at runtime.
For example, like here, mbuf dynamic field offset.
These could be retrieved once and stored somewhere locally
(Rx/Tx queue context for example).

> 
> 
> Now, in practice for your issue, rather than reintroducing symbol
> aliases (technical solution that I dropped when refactoring the
> macros), I think we can do with some middle ground approach:
> - leaving the inline helpers as "stable" (not __rte_experimental),
> - restoring the EXPERIMENTAL version on the global variables, this
> will restore the location of those symbols from the previous ABI pov,
> and the checks won't catch this discrepancy anyway,
> - during 26.11, drop the EXPERIMENTAL version on those variables,
> 
> 
> In other words, stopping at your patch 3 of the series, then adding:
> 
> $ git diff
> diff --git a/lib/ethdev/rte_flow.c b/lib/ethdev/rte_flow.c
> index ec0fe08355..8bd21ccd31 100644
> --- a/lib/ethdev/rte_flow.c
> +++ b/lib/ethdev/rte_flow.c
> @@ -23,11 +23,11 @@
>  #define FLOW_LOG RTE_ETHDEV_LOG_LINE
> 
>  /* Mbuf dynamic field name for metadata. */
> -RTE_EXPORT_SYMBOL(rte_flow_dynf_metadata_offs)
> +RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_flow_dynf_metadata_offs, 19.11)
>  int32_t rte_flow_dynf_metadata_offs = -1;
> 
>  /* Mbuf dynamic field flag bit number for metadata. */
> -RTE_EXPORT_SYMBOL(rte_flow_dynf_metadata_mask)
> +RTE_EXPORT_EXPERIMENTAL_SYMBOL(rte_flow_dynf_metadata_mask, 19.11)
>  uint64_t rte_flow_dynf_metadata_mask;
> 
>  /**

Thank you for the suggestion.
That looks good to me.
I'll prepare a v2.

> 
> > Patch 4 and 5 address the bug for these global variables,
> > by providing a single storage for both EXPERIMENTAL and
> > DPDK_26 variable symbol versions.
> > This is achieved through symbol aliasing.
> > But this solution is limited only to executables compiled with clang.
> >
> > clang and gcc have a different default behavior regarding relocations
> > of global variables exposed by shared libraries.
> >
> 
> Yeah... not even thinking about adding MSVC in the list...
> 
> 
> -- 
> David Marchand
> 

Re: [PATCH 1/5] eal: fix macro for versioned experimental symbol

[Stephen Hemminger]

On Tue, 23 Jun 2026 17:26:22 +0200
Dariusz Sosnowski <dsosnowski@nvidia.com> wrote:

> On Tue, Jun 23, 2026 at 06:50:00AM -0700, Stephen Hemminger wrote:
> > On Tue, 23 Jun 2026 13:37:47 +0200
> > Dariusz Sosnowski <dsosnowski@nvidia.com> wrote:
> >   
> > > Add a missing semicolon after __asm__ block in
> > > RTE_VERSION_EXPERIMENTAL_SYMBOL macro.
> > > It's lack triggers the following compilation error with clang:
> > > 
> > >     ../lib/ethdev/rte_flow.c:320:1: error: expected ';' after top-level asm block
> > >       320 | RTE_VERSION_EXPERIMENTAL_SYMBOL(int, rte_flow_dynf_metadata_register, (void))
> > >           | ^
> > >     ../lib/eal/common/eal_export.h:75:74: note: expanded from macro 'RTE_VERSION_EXPERIMENTAL_SYMBOL'
> > >        75 | __asm__(".symver " RTE_STR(name) "_exp, " RTE_STR(name) "@EXPERIMENTAL") \
> > >           |                                                                          ^
> > >     ../lib/eal/include/rte_common.h:237:20: note: expanded from macro '\
> > >     __rte_used'
> > >       237 | #define __rte_used __attribute__((used))
> > >           |                    ^
> > > 
> > > Fixes: e30e194c4d06 ("eal: rework function versioning macros")
> > > Cc: david.marchand@redhat.com
> > > 
> > > Signed-  
> > 
> > I didn't see this because clang doesn't have symver support.
> > Which version of clang is this?  
> 
> clang 19 available on Debian 13:
> 
>   $ clang --version
>   Debian clang version 19.1.7 (3+b1)
>   Target: x86_64-pc-linux-gnu
>   Thread model: posix
>   InstalledDir: /usr/lib/llvm-19/bin

Ok, that was the asm block not the symver attribute.