From: Tariq Toukan <tariqt@nvidia.com>
To: "David S. Miller" <davem@davemloft.net>,
Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>,
Eric Dumazet <edumazet@google.com>,
"Andrew Lunn" <andrew+netdev@lunn.ch>
Cc: <netdev@vger.kernel.org>, Saeed Mahameed <saeedm@nvidia.com>,
Gal Pressman <gal@nvidia.com>,
Leon Romanovsky <leonro@nvidia.com>,
Jiri Pirko <jiri@resnulli.us>,
Carolina Jubran <cjubran@nvidia.com>,
Cosmin Ratiu <cratiu@nvidia.com>,
Tariq Toukan <tariqt@nvidia.com>
Subject: [PATCH net-next 3/8] devlink: Extend devlink rate API with traffic classes bandwidth management
Date: Wed, 13 Nov 2024 20:00:28 +0200 [thread overview]
Message-ID: <20241113180034.714102-4-tariqt@nvidia.com> (raw)
In-Reply-To: <20241113180034.714102-1-tariqt@nvidia.com>
From: Carolina Jubran <cjubran@nvidia.com>
Introduce support for specifying bandwidth proportions between traffic
classes (TC) in the devlink-rate API. This new option allows users to
allocate bandwidth across multiple traffic classes in a single command.
This feature provides a more granular control over traffic management,
especially for scenarios requiring Enhanced Transmission Selection.
Users can now define a specific bandwidth share for each traffic class,
such as allocating 20% for TC0 (TCP/UDP) and 80% for TC5 (RoCE).
Example:
DEV=pci/0000:08:00.0
$ devlink port function rate add $DEV/vfs_group tx_share 10Gbit \
tx_max 50Gbit tc-bw 0:20 1:0 2:0 3:0 4:0 5:80 6:0 7:0
$ devlink port function rate set $DEV/vfs_group \
tc-bw 0:20 1:0 2:0 3:0 4:0 5:10 6:60 7:0
Signed-off-by: Carolina Jubran <cjubran@nvidia.com>
Reviewed-by: Cosmin Ratiu <cratiu@nvidia.com>
Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
---
Documentation/netlink/specs/devlink.yaml | 50 ++++++++++++++++++++
include/net/devlink.h | 6 +++
include/uapi/linux/devlink.h | 10 ++++
net/devlink/netlink_gen.c | 21 +++++++--
net/devlink/netlink_gen.h | 1 +
net/devlink/rate.c | 60 ++++++++++++++++++++++--
6 files changed, 141 insertions(+), 7 deletions(-)
diff --git a/Documentation/netlink/specs/devlink.yaml b/Documentation/netlink/specs/devlink.yaml
index 09fbb4c03fc8..41fdc2514f69 100644
--- a/Documentation/netlink/specs/devlink.yaml
+++ b/Documentation/netlink/specs/devlink.yaml
@@ -817,6 +817,34 @@ attribute-sets:
-
name: rate-tx-weight
type: u32
+ -
+ name: rate-tc-0-bw
+ type: u32
+ -
+ name: rate-tc-1-bw
+ type: u32
+ -
+ name: rate-tc-2-bw
+ type: u32
+ -
+ name: rate-tc-3-bw
+ type: u32
+ -
+ name: rate-tc-4-bw
+ type: u32
+ -
+ name: rate-tc-5-bw
+ type: u32
+ -
+ name: rate-tc-6-bw
+ type: u32
+ -
+ name: rate-tc-7-bw
+ type: u32
+ -
+ name: rate-tc-bw
+ type: nest
+ nested-attributes: dl-rate-tc-bw-values
-
name: region-direct
type: flag
@@ -1225,6 +1253,26 @@ attribute-sets:
-
name: flash
type: flag
+ -
+ name: dl-rate-tc-bw-values
+ subset-of: devlink
+ attributes:
+ -
+ name: rate-tc-0-bw
+ -
+ name: rate-tc-1-bw
+ -
+ name: rate-tc-2-bw
+ -
+ name: rate-tc-3-bw
+ -
+ name: rate-tc-4-bw
+ -
+ name: rate-tc-5-bw
+ -
+ name: rate-tc-6-bw
+ -
+ name: rate-tc-7-bw
operations:
enum-model: directional
@@ -2148,6 +2196,7 @@ operations:
- rate-tx-max
- rate-tx-priority
- rate-tx-weight
+ - rate-tc-bw
- rate-parent-node-name
-
@@ -2168,6 +2217,7 @@ operations:
- rate-tx-max
- rate-tx-priority
- rate-tx-weight
+ - rate-tc-bw
- rate-parent-node-name
-
diff --git a/include/net/devlink.h b/include/net/devlink.h
index fbb9a2668e24..917bc006a5a4 100644
--- a/include/net/devlink.h
+++ b/include/net/devlink.h
@@ -117,6 +117,8 @@ struct devlink_rate {
u32 tx_priority;
u32 tx_weight;
+
+ u32 tc_bw[IEEE_8021QAZ_MAX_TCS];
};
struct devlink_port {
@@ -1469,6 +1471,8 @@ struct devlink_ops {
u32 tx_priority, struct netlink_ext_ack *extack);
int (*rate_leaf_tx_weight_set)(struct devlink_rate *devlink_rate, void *priv,
u32 tx_weight, struct netlink_ext_ack *extack);
+ int (*rate_leaf_tc_bw_set)(struct devlink_rate *devlink_rate, void *priv,
+ u32 *tc_bw, struct netlink_ext_ack *extack);
int (*rate_node_tx_share_set)(struct devlink_rate *devlink_rate, void *priv,
u64 tx_share, struct netlink_ext_ack *extack);
int (*rate_node_tx_max_set)(struct devlink_rate *devlink_rate, void *priv,
@@ -1477,6 +1481,8 @@ struct devlink_ops {
u32 tx_priority, struct netlink_ext_ack *extack);
int (*rate_node_tx_weight_set)(struct devlink_rate *devlink_rate, void *priv,
u32 tx_weight, struct netlink_ext_ack *extack);
+ int (*rate_node_tc_bw_set)(struct devlink_rate *devlink_rate, void *priv,
+ u32 *tc_bw, struct netlink_ext_ack *extack);
int (*rate_node_new)(struct devlink_rate *rate_node, void **priv,
struct netlink_ext_ack *extack);
int (*rate_node_del)(struct devlink_rate *rate_node, void *priv,
diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h
index 9401aa343673..c369726a262a 100644
--- a/include/uapi/linux/devlink.h
+++ b/include/uapi/linux/devlink.h
@@ -614,6 +614,16 @@ enum devlink_attr {
DEVLINK_ATTR_REGION_DIRECT, /* flag */
+ DEVLINK_ATTR_RATE_TC_0_BW, /* u32 */
+ DEVLINK_ATTR_RATE_TC_1_BW, /* u32 */
+ DEVLINK_ATTR_RATE_TC_2_BW, /* u32 */
+ DEVLINK_ATTR_RATE_TC_3_BW, /* u32 */
+ DEVLINK_ATTR_RATE_TC_4_BW, /* u32 */
+ DEVLINK_ATTR_RATE_TC_5_BW, /* u32 */
+ DEVLINK_ATTR_RATE_TC_6_BW, /* u32 */
+ DEVLINK_ATTR_RATE_TC_7_BW, /* u32 */
+ DEVLINK_ATTR_RATE_TC_BW, /* nested */
+
/* Add new attributes above here, update the spec in
* Documentation/netlink/specs/devlink.yaml and re-generate
* net/devlink/netlink_gen.c.
diff --git a/net/devlink/netlink_gen.c b/net/devlink/netlink_gen.c
index f9786d51f68f..546766bdd836 100644
--- a/net/devlink/netlink_gen.c
+++ b/net/devlink/netlink_gen.c
@@ -18,6 +18,17 @@ const struct nla_policy devlink_dl_port_function_nl_policy[DEVLINK_PORT_FN_ATTR_
[DEVLINK_PORT_FN_ATTR_CAPS] = NLA_POLICY_BITFIELD32(15),
};
+const struct nla_policy devlink_dl_rate_tc_bw_nl_policy[DEVLINK_ATTR_RATE_TC_7_BW + 1] = {
+ [DEVLINK_ATTR_RATE_TC_0_BW] = { .type = NLA_U32, },
+ [DEVLINK_ATTR_RATE_TC_1_BW] = { .type = NLA_U32, },
+ [DEVLINK_ATTR_RATE_TC_2_BW] = { .type = NLA_U32, },
+ [DEVLINK_ATTR_RATE_TC_3_BW] = { .type = NLA_U32, },
+ [DEVLINK_ATTR_RATE_TC_4_BW] = { .type = NLA_U32, },
+ [DEVLINK_ATTR_RATE_TC_5_BW] = { .type = NLA_U32, },
+ [DEVLINK_ATTR_RATE_TC_6_BW] = { .type = NLA_U32, },
+ [DEVLINK_ATTR_RATE_TC_7_BW] = { .type = NLA_U32, },
+};
+
const struct nla_policy devlink_dl_selftest_id_nl_policy[DEVLINK_ATTR_SELFTEST_ID_FLASH + 1] = {
[DEVLINK_ATTR_SELFTEST_ID_FLASH] = { .type = NLA_FLAG, },
};
@@ -496,7 +507,7 @@ static const struct nla_policy devlink_rate_get_dump_nl_policy[DEVLINK_ATTR_DEV_
};
/* DEVLINK_CMD_RATE_SET - do */
-static const struct nla_policy devlink_rate_set_nl_policy[DEVLINK_ATTR_RATE_TX_WEIGHT + 1] = {
+static const struct nla_policy devlink_rate_set_nl_policy[DEVLINK_ATTR_RATE_TC_BW + 1] = {
[DEVLINK_ATTR_BUS_NAME] = { .type = NLA_NUL_STRING, },
[DEVLINK_ATTR_DEV_NAME] = { .type = NLA_NUL_STRING, },
[DEVLINK_ATTR_RATE_NODE_NAME] = { .type = NLA_NUL_STRING, },
@@ -505,10 +516,11 @@ static const struct nla_policy devlink_rate_set_nl_policy[DEVLINK_ATTR_RATE_TX_W
[DEVLINK_ATTR_RATE_TX_PRIORITY] = { .type = NLA_U32, },
[DEVLINK_ATTR_RATE_TX_WEIGHT] = { .type = NLA_U32, },
[DEVLINK_ATTR_RATE_PARENT_NODE_NAME] = { .type = NLA_NUL_STRING, },
+ [DEVLINK_ATTR_RATE_TC_BW] = NLA_POLICY_NESTED(devlink_dl_rate_tc_bw_nl_policy),
};
/* DEVLINK_CMD_RATE_NEW - do */
-static const struct nla_policy devlink_rate_new_nl_policy[DEVLINK_ATTR_RATE_TX_WEIGHT + 1] = {
+static const struct nla_policy devlink_rate_new_nl_policy[DEVLINK_ATTR_RATE_TC_BW + 1] = {
[DEVLINK_ATTR_BUS_NAME] = { .type = NLA_NUL_STRING, },
[DEVLINK_ATTR_DEV_NAME] = { .type = NLA_NUL_STRING, },
[DEVLINK_ATTR_RATE_NODE_NAME] = { .type = NLA_NUL_STRING, },
@@ -517,6 +529,7 @@ static const struct nla_policy devlink_rate_new_nl_policy[DEVLINK_ATTR_RATE_TX_W
[DEVLINK_ATTR_RATE_TX_PRIORITY] = { .type = NLA_U32, },
[DEVLINK_ATTR_RATE_TX_WEIGHT] = { .type = NLA_U32, },
[DEVLINK_ATTR_RATE_PARENT_NODE_NAME] = { .type = NLA_NUL_STRING, },
+ [DEVLINK_ATTR_RATE_TC_BW] = NLA_POLICY_NESTED(devlink_dl_rate_tc_bw_nl_policy),
};
/* DEVLINK_CMD_RATE_DEL - do */
@@ -1164,7 +1177,7 @@ const struct genl_split_ops devlink_nl_ops[74] = {
.doit = devlink_nl_rate_set_doit,
.post_doit = devlink_nl_post_doit,
.policy = devlink_rate_set_nl_policy,
- .maxattr = DEVLINK_ATTR_RATE_TX_WEIGHT,
+ .maxattr = DEVLINK_ATTR_RATE_TC_BW,
.flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
},
{
@@ -1174,7 +1187,7 @@ const struct genl_split_ops devlink_nl_ops[74] = {
.doit = devlink_nl_rate_new_doit,
.post_doit = devlink_nl_post_doit,
.policy = devlink_rate_new_nl_policy,
- .maxattr = DEVLINK_ATTR_RATE_TX_WEIGHT,
+ .maxattr = DEVLINK_ATTR_RATE_TC_BW,
.flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO,
},
{
diff --git a/net/devlink/netlink_gen.h b/net/devlink/netlink_gen.h
index 8f2bd50ddf5e..084ad5026d4f 100644
--- a/net/devlink/netlink_gen.h
+++ b/net/devlink/netlink_gen.h
@@ -14,6 +14,7 @@
/* Common nested types */
extern const struct nla_policy devlink_dl_port_function_nl_policy[DEVLINK_PORT_FN_ATTR_CAPS + 1];
extern const struct nla_policy devlink_dl_selftest_id_nl_policy[DEVLINK_ATTR_SELFTEST_ID_FLASH + 1];
+extern const struct nla_policy devlink_dl_rate_tc_bw_nl_policy[DEVLINK_ATTR_RATE_TC_7_BW + 1];
/* Ops table for devlink */
extern const struct genl_split_ops devlink_nl_ops[74];
diff --git a/net/devlink/rate.c b/net/devlink/rate.c
index 8828ffaf6cbc..9bffda6783f9 100644
--- a/net/devlink/rate.c
+++ b/net/devlink/rate.c
@@ -86,7 +86,9 @@ static int devlink_nl_rate_fill(struct sk_buff *msg,
int flags, struct netlink_ext_ack *extack)
{
struct devlink *devlink = devlink_rate->devlink;
+ struct nlattr *nla_tc_bw;
void *hdr;
+ int i;
hdr = genlmsg_put(msg, portid, seq, &devlink_nl_family, flags, cmd);
if (!hdr)
@@ -124,10 +126,19 @@ static int devlink_nl_rate_fill(struct sk_buff *msg,
devlink_rate->tx_weight))
goto nla_put_failure;
- if (devlink_rate->parent)
- if (nla_put_string(msg, DEVLINK_ATTR_RATE_PARENT_NODE_NAME,
- devlink_rate->parent->name))
+ nla_tc_bw = nla_nest_start(msg, DEVLINK_ATTR_RATE_TC_BW);
+ if (!nla_tc_bw)
+ goto nla_put_failure;
+
+ for (i = 0; i < IEEE_8021QAZ_MAX_TCS; i++) {
+ if (nla_put_u32(msg, DEVLINK_ATTR_RATE_TC_0_BW + i,
+ devlink_rate->tc_bw[i])) {
+ nla_nest_cancel(msg, nla_tc_bw);
goto nla_put_failure;
+ }
+ }
+
+ nla_nest_end(msg, nla_tc_bw);
genlmsg_end(msg, hdr);
return 0;
@@ -321,6 +332,7 @@ static int devlink_nl_rate_set(struct devlink_rate *devlink_rate,
struct genl_info *info)
{
struct nlattr *nla_parent, **attrs = info->attrs;
+ u32 tc_bw[IEEE_8021QAZ_MAX_TCS];
int err = -EOPNOTSUPP;
u32 priority;
u32 weight;
@@ -380,6 +392,36 @@ static int devlink_nl_rate_set(struct devlink_rate *devlink_rate,
devlink_rate->tx_weight = weight;
}
+ if (attrs[DEVLINK_ATTR_RATE_TC_BW]) {
+ struct nlattr *nla_tc_bw = attrs[DEVLINK_ATTR_RATE_TC_BW];
+ struct nlattr *tb[DEVLINK_ATTR_RATE_TC_7_BW + 1];
+ int i;
+
+ err = nla_parse_nested(tb, DEVLINK_ATTR_RATE_TC_7_BW, nla_tc_bw,
+ devlink_dl_rate_tc_bw_nl_policy, info->extack);
+ if (err)
+ return err;
+
+ for (i = 0; i < IEEE_8021QAZ_MAX_TCS; i++) {
+ if (tb[DEVLINK_ATTR_RATE_TC_0_BW + i])
+ tc_bw[i] = nla_get_u32(tb[DEVLINK_ATTR_RATE_TC_0_BW + i]);
+ else
+ tc_bw[i] = 0;
+ }
+
+ if (devlink_rate_is_leaf(devlink_rate))
+ err = ops->rate_leaf_tc_bw_set(devlink_rate, devlink_rate->priv,
+ tc_bw, info->extack);
+ else if (devlink_rate_is_node(devlink_rate))
+ err = ops->rate_node_tc_bw_set(devlink_rate, devlink_rate->priv,
+ tc_bw, info->extack);
+
+ if (err)
+ return err;
+
+ memcpy(devlink_rate->tc_bw, tc_bw, sizeof(tc_bw));
+ }
+
nla_parent = attrs[DEVLINK_ATTR_RATE_PARENT_NODE_NAME];
if (nla_parent) {
err = devlink_nl_rate_parent_node_set(devlink_rate, info,
@@ -423,6 +465,12 @@ static bool devlink_rate_set_ops_supported(const struct devlink_ops *ops,
"TX weight set isn't supported for the leafs");
return false;
}
+ if (attrs[DEVLINK_ATTR_RATE_TC_BW] && !ops->rate_leaf_tc_bw_set) {
+ NL_SET_ERR_MSG_ATTR(info->extack,
+ attrs[DEVLINK_ATTR_RATE_TC_BW],
+ "TC bandwidth set isn't supported for the leafs");
+ return false;
+ }
} else if (type == DEVLINK_RATE_TYPE_NODE) {
if (attrs[DEVLINK_ATTR_RATE_TX_SHARE] && !ops->rate_node_tx_share_set) {
NL_SET_ERR_MSG(info->extack, "TX share set isn't supported for the nodes");
@@ -449,6 +497,12 @@ static bool devlink_rate_set_ops_supported(const struct devlink_ops *ops,
"TX weight set isn't supported for the nodes");
return false;
}
+ if (attrs[DEVLINK_ATTR_RATE_TC_BW] && !ops->rate_node_tc_bw_set) {
+ NL_SET_ERR_MSG_ATTR(info->extack,
+ attrs[DEVLINK_ATTR_RATE_TC_BW],
+ "TC bandwidth set isn't supported for the nodes");
+ return false;
+ }
} else {
WARN(1, "Unknown type of rate object");
return false;
--
2.44.0
next prev parent reply other threads:[~2024-11-13 18:01 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-11-13 18:00 [PATCH net-next 0/8] net/mlx5: ConnectX-8 SW Steering + Rate management on traffic classes Tariq Toukan
2024-11-13 18:00 ` [PATCH net-next 1/8] net/mlx5: DR, expand SWS STE callbacks and consolidate common structs Tariq Toukan
2024-11-13 18:00 ` [PATCH net-next 2/8] net/mlx5: DR, add support for ConnectX-8 steering Tariq Toukan
2024-11-13 18:00 ` Tariq Toukan [this message]
2024-11-13 23:22 ` [PATCH net-next 3/8] devlink: Extend devlink rate API with traffic classes bandwidth management kernel test robot
2024-11-14 0:05 ` kernel test robot
2024-11-14 0:46 ` kernel test robot
2024-11-14 9:48 ` Jiri Pirko
2024-11-13 18:00 ` [PATCH net-next 4/8] net/mlx5:Add no-op implementation for setting tc-bw on rate objects Tariq Toukan
2024-11-13 18:00 ` [PATCH net-next 5/8] net/mlx5: Add support for new scheduling elements Tariq Toukan
2024-11-13 18:00 ` [PATCH net-next 6/8] net/mlx5: Add support for setting tc-bw on nodes Tariq Toukan
2024-11-13 18:00 ` [PATCH net-next 7/8] net/mlx5: Add traffic class scheduling support for vport QoS Tariq Toukan
2024-11-13 18:00 ` [PATCH net-next 8/8] net/mlx5: Manage TC arbiter nodes and provide full support for tc-bw Tariq Toukan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20241113180034.714102-4-tariqt@nvidia.com \
--to=tariqt@nvidia.com \
--cc=andrew+netdev@lunn.ch \
--cc=cjubran@nvidia.com \
--cc=cratiu@nvidia.com \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=gal@nvidia.com \
--cc=jiri@resnulli.us \
--cc=kuba@kernel.org \
--cc=leonro@nvidia.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=saeedm@nvidia.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.