From: Liu Jian <liujian56@huawei.com>
To: <john.fastabend@gmail.com>, <jakub@cloudflare.com>,
<ast@kernel.org>, <daniel@iogearbox.net>, <andrii@kernel.org>,
<martin.lau@linux.dev>, <song@kernel.org>,
<yonghong.song@linux.dev>, <kpsingh@kernel.org>, <sdf@google.com>,
<haoluo@google.com>, <jolsa@kernel.org>, <davem@davemloft.net>,
<edumazet@google.com>, <kuba@kernel.org>, <pabeni@redhat.com>,
<dsahern@kernel.org>
Cc: <netdev@vger.kernel.org>, <bpf@vger.kernel.org>, <liujian56@huawei.com>
Subject: [PATCH bpf-next v5 1/7] bpf, sockmap: add BPF_F_PERMANENT flag for skmsg redirect
Date: Wed, 27 Sep 2023 17:30:07 +0800 [thread overview]
Message-ID: <20230927093013.1951659-2-liujian56@huawei.com> (raw)
In-Reply-To: <20230927093013.1951659-1-liujian56@huawei.com>
If the sockmap msg redirection function is used only to forward packets
and no other operation, the execution result of the BPF_SK_MSG_VERDICT
program is the same each time. In this case, the BPF program only needs to
be run once. Add BPF_F_PERMANENT flag to bpf_msg_redirect_map() and
bpf_msg_redirect_hash() to implement this ability.
Then we can enable this function in the bpf program as follows:
bpf_msg_redirect_hash(xx, xx, xx, BPF_F_INGRESS | BPF_F_PERMANENT);
Test results using netperf TCP_STREAM mode:
for i in 1 64 128 512 1k 2k 32k 64k 100k 500k 1m;then
netperf -T 1,2 -t TCP_STREAM -H 127.0.0.1 -l 20 -- -m $i -s 100m,100m -S 100m,100m
done
before:
3.84 246.52 496.89 1885.03 3415.29 6375.03 40749.09 48764.40 51611.34 55678.26 55992.78
after:
4.43 279.20 555.82 2080.79 3870.70 7105.44 41836.41 49709.75 51861.56 55211.00 54566.85
Signed-off-by: Liu Jian <liujian56@huawei.com>
---
include/linux/skmsg.h | 1 +
include/uapi/linux/bpf.h | 45 ++++++++++++++++++++++++++--------
net/core/skmsg.c | 6 ++++-
net/core/sock_map.c | 4 +--
net/ipv4/tcp_bpf.c | 12 +++++----
tools/include/uapi/linux/bpf.h | 45 ++++++++++++++++++++++++++--------
6 files changed, 85 insertions(+), 28 deletions(-)
diff --git a/include/linux/skmsg.h b/include/linux/skmsg.h
index c1637515a8a4..acd7de85608b 100644
--- a/include/linux/skmsg.h
+++ b/include/linux/skmsg.h
@@ -83,6 +83,7 @@ struct sk_psock {
u32 cork_bytes;
u32 eval;
bool redir_ingress; /* undefined if sk_redir is null */
+ bool redir_permanent;
struct sk_msg *cork;
struct sk_psock_progs progs;
#if IS_ENABLED(CONFIG_BPF_STREAM_PARSER)
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 70bfa997e896..cec6c34f4486 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -3029,11 +3029,23 @@ union bpf_attr {
* socket level. If the message *msg* is allowed to pass (i.e. if
* the verdict eBPF program returns **SK_PASS**), redirect it to
* the socket referenced by *map* (of type
- * **BPF_MAP_TYPE_SOCKMAP**) at index *key*. Both ingress and
- * egress interfaces can be used for redirection. The
- * **BPF_F_INGRESS** value in *flags* is used to make the
- * distinction (ingress path is selected if the flag is present,
- * egress path otherwise). This is the only flag supported for now.
+ * **BPF_MAP_TYPE_SOCKMAP**) at index *key*.
+ *
+ * The following *flags* are supported:
+ *
+ * **BPF_F_INGRESS**
+ * Both ingress and egress interfaces can be used for redirection.
+ * The **BPF_F_INGRESS** value in *flags* is used to make the
+ * distinction. Ingress path is selected if the flag is present,
+ * egress path otherwise.
+ * **BPF_F_PERMANENT**
+ * Indicates that redirect verdict and the target socket should be
+ * remembered. The verdict program will not be run for subsequent
+ * packets, unless an error occurs when forwarding packets.
+ *
+ * **BPF_F_PERMANENT** cannot be use together with
+ * **bpf_msg_apply_bytes**\ () and **bpf_msg_cork_bytes**\ (). If
+ * **BPF_F_PERMANENT** is set apply_bytes and cork_bytes are ignored.
* Return
* **SK_PASS** on success, or **SK_DROP** on error.
*
@@ -3301,11 +3313,23 @@ union bpf_attr {
* socket level. If the message *msg* is allowed to pass (i.e. if
* the verdict eBPF program returns **SK_PASS**), redirect it to
* the socket referenced by *map* (of type
- * **BPF_MAP_TYPE_SOCKHASH**) using hash *key*. Both ingress and
- * egress interfaces can be used for redirection. The
- * **BPF_F_INGRESS** value in *flags* is used to make the
- * distinction (ingress path is selected if the flag is present,
- * egress path otherwise). This is the only flag supported for now.
+ * **BPF_MAP_TYPE_SOCKHASH**) using hash *key*.
+ *
+ * The following *flags* are supported:
+ *
+ * **BPF_F_INGRESS**
+ * Both ingress and egress interfaces can be used for redirection.
+ * The **BPF_F_INGRESS** value in *flags* is used to make the
+ * distinction. Ingress path is selected if the flag is present,
+ * egress path otherwise.
+ * **BPF_F_PERMANENT**
+ * Indicates that redirect verdict and the target socket should be
+ * remembered. The verdict program will not be run for subsequent
+ * packets, unless an error occurs when forwarding packets.
+ *
+ * **BPF_F_PERMANENT** cannot be use together with
+ * **bpf_msg_apply_bytes**\ () and **bpf_msg_cork_bytes**\ (). If
+ * **BPF_F_PERMANENT** is set apply_bytes and cork_bytes are ignored.
* Return
* **SK_PASS** on success, or **SK_DROP** on error.
*
@@ -5906,6 +5930,7 @@ enum {
/* BPF_FUNC_clone_redirect and BPF_FUNC_redirect flags. */
enum {
BPF_F_INGRESS = (1ULL << 0),
+ BPF_F_PERMANENT = (1ULL << 1),
};
/* BPF_FUNC_skb_set_tunnel_key and BPF_FUNC_skb_get_tunnel_key flags. */
diff --git a/net/core/skmsg.c b/net/core/skmsg.c
index 6c31eefbd777..22e55eec3089 100644
--- a/net/core/skmsg.c
+++ b/net/core/skmsg.c
@@ -878,7 +878,11 @@ int sk_psock_msg_verdict(struct sock *sk, struct sk_psock *psock,
msg->sk = sk;
ret = bpf_prog_run_pin_on_cpu(prog, msg);
ret = sk_psock_map_verd(ret, msg->sk_redir);
- psock->apply_bytes = msg->apply_bytes;
+ psock->redir_permanent = msg->flags & BPF_F_PERMANENT;
+ if (psock->redir_permanent)
+ msg->cork_bytes = msg->apply_bytes = 0;
+ else
+ psock->apply_bytes = msg->apply_bytes;
if (ret == __SK_REDIRECT) {
if (psock->sk_redir) {
sock_put(psock->sk_redir);
diff --git a/net/core/sock_map.c b/net/core/sock_map.c
index cb11750b1df5..5b2539b04628 100644
--- a/net/core/sock_map.c
+++ b/net/core/sock_map.c
@@ -662,7 +662,7 @@ BPF_CALL_4(bpf_msg_redirect_map, struct sk_msg *, msg,
{
struct sock *sk;
- if (unlikely(flags & ~(BPF_F_INGRESS)))
+ if (unlikely(flags & ~(BPF_F_INGRESS | BPF_F_PERMANENT)))
return SK_DROP;
sk = __sock_map_lookup_elem(map, key);
@@ -1261,7 +1261,7 @@ BPF_CALL_4(bpf_msg_redirect_hash, struct sk_msg *, msg,
{
struct sock *sk;
- if (unlikely(flags & ~(BPF_F_INGRESS)))
+ if (unlikely(flags & ~(BPF_F_INGRESS | BPF_F_PERMANENT)))
return SK_DROP;
sk = __sock_hash_lookup_elem(map, key);
diff --git a/net/ipv4/tcp_bpf.c b/net/ipv4/tcp_bpf.c
index 81f0dff69e0b..bbbbafaa1929 100644
--- a/net/ipv4/tcp_bpf.c
+++ b/net/ipv4/tcp_bpf.c
@@ -419,8 +419,10 @@ static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock,
if (!psock->apply_bytes) {
/* Clean up before releasing the sock lock. */
eval = psock->eval;
- psock->eval = __SK_NONE;
- psock->sk_redir = NULL;
+ if (!psock->redir_permanent) {
+ psock->eval = __SK_NONE;
+ psock->sk_redir = NULL;
+ }
}
if (psock->cork) {
cork = true;
@@ -434,7 +436,7 @@ static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock,
msg, tosend, flags);
sent = origsize - msg->sg.size;
- if (eval == __SK_REDIRECT)
+ if (!psock->redir_permanent && eval == __SK_REDIRECT)
sock_put(sk_redir);
lock_sock(sk);
@@ -460,8 +462,8 @@ static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock,
}
if (likely(!ret)) {
- if (!psock->apply_bytes) {
- psock->eval = __SK_NONE;
+ if (!psock->apply_bytes && !psock->redir_permanent) {
+ psock->eval = __SK_NONE;
if (psock->sk_redir) {
sock_put(psock->sk_redir);
psock->sk_redir = NULL;
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index 70bfa997e896..cec6c34f4486 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -3029,11 +3029,23 @@ union bpf_attr {
* socket level. If the message *msg* is allowed to pass (i.e. if
* the verdict eBPF program returns **SK_PASS**), redirect it to
* the socket referenced by *map* (of type
- * **BPF_MAP_TYPE_SOCKMAP**) at index *key*. Both ingress and
- * egress interfaces can be used for redirection. The
- * **BPF_F_INGRESS** value in *flags* is used to make the
- * distinction (ingress path is selected if the flag is present,
- * egress path otherwise). This is the only flag supported for now.
+ * **BPF_MAP_TYPE_SOCKMAP**) at index *key*.
+ *
+ * The following *flags* are supported:
+ *
+ * **BPF_F_INGRESS**
+ * Both ingress and egress interfaces can be used for redirection.
+ * The **BPF_F_INGRESS** value in *flags* is used to make the
+ * distinction. Ingress path is selected if the flag is present,
+ * egress path otherwise.
+ * **BPF_F_PERMANENT**
+ * Indicates that redirect verdict and the target socket should be
+ * remembered. The verdict program will not be run for subsequent
+ * packets, unless an error occurs when forwarding packets.
+ *
+ * **BPF_F_PERMANENT** cannot be use together with
+ * **bpf_msg_apply_bytes**\ () and **bpf_msg_cork_bytes**\ (). If
+ * **BPF_F_PERMANENT** is set apply_bytes and cork_bytes are ignored.
* Return
* **SK_PASS** on success, or **SK_DROP** on error.
*
@@ -3301,11 +3313,23 @@ union bpf_attr {
* socket level. If the message *msg* is allowed to pass (i.e. if
* the verdict eBPF program returns **SK_PASS**), redirect it to
* the socket referenced by *map* (of type
- * **BPF_MAP_TYPE_SOCKHASH**) using hash *key*. Both ingress and
- * egress interfaces can be used for redirection. The
- * **BPF_F_INGRESS** value in *flags* is used to make the
- * distinction (ingress path is selected if the flag is present,
- * egress path otherwise). This is the only flag supported for now.
+ * **BPF_MAP_TYPE_SOCKHASH**) using hash *key*.
+ *
+ * The following *flags* are supported:
+ *
+ * **BPF_F_INGRESS**
+ * Both ingress and egress interfaces can be used for redirection.
+ * The **BPF_F_INGRESS** value in *flags* is used to make the
+ * distinction. Ingress path is selected if the flag is present,
+ * egress path otherwise.
+ * **BPF_F_PERMANENT**
+ * Indicates that redirect verdict and the target socket should be
+ * remembered. The verdict program will not be run for subsequent
+ * packets, unless an error occurs when forwarding packets.
+ *
+ * **BPF_F_PERMANENT** cannot be use together with
+ * **bpf_msg_apply_bytes**\ () and **bpf_msg_cork_bytes**\ (). If
+ * **BPF_F_PERMANENT** is set apply_bytes and cork_bytes are ignored.
* Return
* **SK_PASS** on success, or **SK_DROP** on error.
*
@@ -5906,6 +5930,7 @@ enum {
/* BPF_FUNC_clone_redirect and BPF_FUNC_redirect flags. */
enum {
BPF_F_INGRESS = (1ULL << 0),
+ BPF_F_PERMANENT = (1ULL << 1),
};
/* BPF_FUNC_skb_set_tunnel_key and BPF_FUNC_skb_get_tunnel_key flags. */
--
2.34.1
next prev parent reply other threads:[~2023-09-27 9:27 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-09-27 9:30 [PATCH bpf-next v5 0/7] add BPF_F_PERMANENT flag for sockmap skmsg redirect Liu Jian
2023-09-27 9:30 ` Liu Jian [this message]
2023-10-03 4:27 ` [PATCH bpf-next v5 1/7] bpf, sockmap: add BPF_F_PERMANENT flag for " John Fastabend
2023-10-14 12:05 ` liujian (CE)
2023-10-13 12:34 ` Jakub Sitnicki
2023-09-27 9:30 ` [PATCH bpf-next v5 2/7] selftests/bpf: Add txmsg permanently test for sockmap Liu Jian
2023-09-27 9:30 ` [PATCH bpf-next v5 3/7] selftests/bpf: Add txmsg redir " Liu Jian
2023-09-27 9:30 ` [PATCH bpf-next v5 4/7] selftests/bpf: add skmsg verdict tests Liu Jian
2023-09-27 9:30 ` [PATCH bpf-next v5 5/7] selftests/bpf: add two skmsg verdict tests for BPF_F_PERMANENT flag Liu Jian
2023-09-27 9:30 ` [PATCH bpf-next v5 6/7] selftests/bpf: add tests for verdict skmsg to itself Liu Jian
2023-09-27 9:30 ` [PATCH bpf-next v5 7/7] selftests/bpf: add tests for verdict skmsg to closed socket Liu Jian
2023-10-03 4:31 ` [PATCH bpf-next v5 0/7] add BPF_F_PERMANENT flag for sockmap skmsg redirect John Fastabend
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230927093013.1951659-2-liujian56@huawei.com \
--to=liujian56@huawei.com \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=davem@davemloft.net \
--cc=dsahern@kernel.org \
--cc=edumazet@google.com \
--cc=haoluo@google.com \
--cc=jakub@cloudflare.com \
--cc=john.fastabend@gmail.com \
--cc=jolsa@kernel.org \
--cc=kpsingh@kernel.org \
--cc=kuba@kernel.org \
--cc=martin.lau@linux.dev \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=sdf@google.com \
--cc=song@kernel.org \
--cc=yonghong.song@linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).