* Re: arm32 build failure after 992aa864dca068554802a65a467a2640985cc213
From: Shalom Toledo @ 2019-06-18 12:05 UTC (permalink / raw)
To: Nathan Chancellor, Ido Schimmel, Jiri Pirko
Cc: Petr Machata, David S. Miller, netdev@vger.kernel.org,
linux-kernel@vger.kernel.org
In-Reply-To: <20190618014604.GA17174@archlinux-epyc>
On 18/06/2019 4:46, Nathan Chancellor wrote:
> Hi all,
>
> A 32-bit ARM allyesconfig fails to link after commit 992aa864dca0
> ("mlxsw: spectrum_ptp: Add implementation for physical hardware clock
> operations") because of 64-bit division:
>
> arm-linux-gnueabi-ld:
> drivers/net/ethernet/mellanox/mlxsw/spectrum_ptp.o: in function
> `mlxsw_sp1_ptp_phc_settime':
> spectrum_ptp.c:(.text+0x39c): undefined reference to `__aeabi_uldivmod'
>
> The following diff fixes it but I have no idea if it is proper or not
> (hence reaching out before sending it, in case one of you has a more
> proper idea).
>
> Cheers,
> Nathan
>
> ---
>
> diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_ptp.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_ptp.c
> index 2a9bbc90225e..65686f0b6834 100644
> --- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_ptp.c
> +++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_ptp.c
> @@ -87,7 +87,7 @@ mlxsw_sp1_ptp_phc_settime(struct mlxsw_sp_ptp_clock *clock, u64 nsec)
> u32 next_sec;
> int err;
>
> - next_sec = nsec / NSEC_PER_SEC + 1;
> + next_sec = (u32)div64_u64(nsec, NSEC_PER_SEC + 1);
> next_sec_in_nsec = next_sec * NSEC_PER_SEC;
>
> spin_lock(&clock->lock);
>
>
Thanks for the report. I will send a fix.
^ permalink raw reply
* next: arm64: build error: implicit declaration of function '__cookie_v6_init_sequence'
From: Naresh Kamboju @ 2019-06-18 12:05 UTC (permalink / raw)
To: Linux-Next Mailing List, Netdev, bpf
Cc: David S. Miller, kuznet, yoshfuji, ast, Daniel Borkmann, kafai,
Song Liu, Yonghong Song, open list
Linux -next build failed on arm64 and arm.
In file included from net/ipv6/af_inet6.c:41:0:
include/linux/netfilter_ipv6.h: In function 'nf_ipv6_cookie_init_sequence':
include/linux/netfilter_ipv6.h:174:9: error: implicit declaration of
function '__cookie_v6_init_sequence'; did you mean
'cookie_init_sequence'? [-Werror=implicit-function-declaration]
return __cookie_v6_init_sequence(iph, th, mssp);
^~~~~~~~~~~~~~~~~~~~~~~~~
cookie_init_sequence
include/linux/netfilter_ipv6.h: In function 'nf_cookie_v6_check':
include/linux/netfilter_ipv6.h:189:9: error: implicit declaration of
function '__cookie_v6_check'; did you mean '__cookie_v4_check'?
[-Werror=implicit-function-declaration]
return __cookie_v6_check(iph, th, cookie);
^~~~~~~~~~~~~~~~~
__cookie_v4_check
CC net/core/net-traces.o
CC drivers/char/tpm/eventlog/acpi.o
CC fs/proc/page.o
cc1: some warnings being treated as errors
scripts/Makefile.build:278: recipe for target 'net/ipv6/af_inet6.o' failed
make[3]: *** [net/ipv6/af_inet6.o] Error 1
scripts/Makefile.build:489: recipe for target 'net/ipv6' failed
make[2]: *** [net/ipv6] Error 2
Best regards
Naresh Kamboju
^ permalink raw reply
* Re: 4.19: udpgso_bench_tx: setsockopt zerocopy: Unknown error 524
From: Willem de Bruijn @ 2019-06-18 12:31 UTC (permalink / raw)
To: Naresh Kamboju
Cc: David S. Miller, Netdev, open list,
open list:KERNEL SELFTEST FRAMEWORK, Fred Klassen
In-Reply-To: <CA+G9fYs2+-yeYcx7oe228oo9GfDgTuPL1=TemT3R20tzCmcjsw@mail.gmail.com>
On Tue, Jun 18, 2019 at 7:27 AM Naresh Kamboju
<naresh.kamboju@linaro.org> wrote:
>
> selftests: net: udpgso_bench.sh failed on 4.19, 4.14, 4.9 and 4.4 branches.
> PASS on stable branch 5.1, mainline and next.
> This failure is started happening on 4.19 and older kernel branches after
> kselftest upgrade to version 5.1
Does version 5.1 here mean running tests from Linux 5.1, against older kernels?
> Is there any possibilities to backport ?
>
> Error:
> udpgso_bench_tx: setsockopt zerocopy: Unknown error 524
MSG_ZEROCOPY for UDP was added in commit b5947e5d1e71 ("udp:
msg_zerocopy") in Linux 5.0.
The selftest was expanded with this feature in commit db63e489c7aa
("selftests: extend zerocopy tests to udp"), also in Linux 5.0.
Those tests are not expected to pass on older kernels.
^ permalink raw reply
* [PATCH net-next] mlxsw: spectrum_ptp: Fix compilation on 32-bit ARM
From: Shalom Toledo @ 2019-06-18 12:45 UTC (permalink / raw)
To: netdev@vger.kernel.org
Cc: davem@davemloft.net, Jiri Pirko, Ido Schimmel, mlxsw,
natechancellor@gmail.com
Compilation on 32-bit ARM fails after commit 992aa864dca0 ("mlxsw:
spectrum_ptp: Add implementation for physical hardware clock operations")
because of 64-bit division:
arm-linux-gnueabi-ld:
drivers/net/ethernet/mellanox/mlxsw/spectrum_ptp.o: in function
`mlxsw_sp1_ptp_phc_settime': spectrum_ptp.c:(.text+0x39c): undefined
reference to `__aeabi_uldivmod'
Fix by using div_u64().
Fixes: 992aa864dca0 ("mlxsw: spectrum_ptp: Add implementation for physical hardware clock operations")
Signed-off-by: Shalom Toledo <shalomt@mellanox.com>
Reviewed-by: Ido Schimmel <idosch@mellanox.com>
Reported-by: Nathan Chancellor <natechancellor@gmail.com>
---
drivers/net/ethernet/mellanox/mlxsw/spectrum_ptp.c | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_ptp.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_ptp.c
index 2a9bbc90225e..bb6c0cb25771 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_ptp.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_ptp.c
@@ -81,13 +81,12 @@ static int
mlxsw_sp1_ptp_phc_settime(struct mlxsw_sp_ptp_clock *clock, u64 nsec)
{
struct mlxsw_core *mlxsw_core = clock->core;
+ u64 next_sec, next_sec_in_nsec, cycles;
char mtutc_pl[MLXSW_REG_MTUTC_LEN];
char mtpps_pl[MLXSW_REG_MTPPS_LEN];
- u64 next_sec_in_nsec, cycles;
- u32 next_sec;
int err;
- next_sec = nsec / NSEC_PER_SEC + 1;
+ next_sec = div_u64(nsec, NSEC_PER_SEC) + 1;
next_sec_in_nsec = next_sec * NSEC_PER_SEC;
spin_lock(&clock->lock);
--
2.20.1
^ permalink raw reply related
* Re: [PATCH net-next v1 08/11] xdp: tracking page_pool resources and safe removal
From: Ivan Khoronzhuk @ 2019-06-18 12:54 UTC (permalink / raw)
To: Tariq Toukan
Cc: Jesper Dangaard Brouer, netdev@vger.kernel.org, Ilias Apalodimas,
Toke Høiland-Jørgensen, toshiaki.makita1@gmail.com,
grygorii.strashko@ti.com, mcroce@redhat.com
In-Reply-To: <a02856c1-46e7-4691-6bb9-e0efb388981f@mellanox.com>
On Sun, Jun 16, 2019 at 10:56:25AM +0000, Tariq Toukan wrote:
Hi Tariq
>
>
>On 6/15/2019 12:33 PM, Ivan Khoronzhuk wrote:
>> On Thu, Jun 13, 2019 at 08:28:42PM +0200, Jesper Dangaard Brouer wrote:
>> Hi, Jesper
>>
>>> This patch is needed before we can allow drivers to use page_pool for
>>> DMA-mappings. Today with page_pool and XDP return API, it is possible to
>>> remove the page_pool object (from rhashtable), while there are still
>>> in-flight packet-pages. This is safely handled via RCU and failed
>>> lookups in
>>> __xdp_return() fallback to call put_page(), when page_pool object is
>>> gone.
>>> In-case page is still DMA mapped, this will result in page note getting
>>> correctly DMA unmapped.
>>>
>>> To solve this, the page_pool is extended with tracking in-flight
>>> pages. And
>>> XDP disconnect system queries page_pool and waits, via workqueue, for all
>>> in-flight pages to be returned.
>>>
>>> To avoid killing performance when tracking in-flight pages, the implement
>>> use two (unsigned) counters, that in placed on different cache-lines, and
>>> can be used to deduct in-flight packets. This is done by mapping the
>>> unsigned "sequence" counters onto signed Two's complement arithmetic
>>> operations. This is e.g. used by kernel's time_after macros, described in
>>> kernel commit 1ba3aab3033b and 5a581b367b5, and also explained in
>>> RFC1982.
>>>
>>> The trick is these two incrementing counters only need to be read and
>>> compared, when checking if it's safe to free the page_pool structure.
>>> Which
>>> will only happen when driver have disconnected RX/alloc side. Thus, on a
>>> non-fast-path.
>>>
>>> It is chosen that page_pool tracking is also enabled for the non-DMA
>>> use-case, as this can be used for statistics later.
>>>
>>> After this patch, using page_pool requires more strict resource
>>> "release",
>>> e.g. via page_pool_release_page() that was introduced in this
>>> patchset, and
>>> previous patches implement/fix this more strict requirement.
>>>
>>> Drivers no-longer call page_pool_destroy(). Drivers already call
>>> xdp_rxq_info_unreg() which call xdp_rxq_info_unreg_mem_model(), which
>>> will
>>> attempt to disconnect the mem id, and if attempt fails schedule the
>>> disconnect for later via delayed workqueue.
>>>
>>> Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
>>> ---
>>> drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 3 -
>>> include/net/page_pool.h | 41 ++++++++++---
>>> net/core/page_pool.c | 62
>>> +++++++++++++++-----
>>> net/core/xdp.c | 65
>>> +++++++++++++++++++--
>>> 4 files changed, 136 insertions(+), 35 deletions(-)
>>>
>>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>>> b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>>> index 2f647be292b6..6c9d4d7defbc 100644
>>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>>
>> [...]
>>
>>> --- a/net/core/xdp.c
>>> +++ b/net/core/xdp.c
>>> @@ -38,6 +38,7 @@ struct xdp_mem_allocator {
>>> };
>>> struct rhash_head node;
>>> struct rcu_head rcu;
>>> + struct delayed_work defer_wq;
>>> };
>>>
>>> static u32 xdp_mem_id_hashfn(const void *data, u32 len, u32 seed)
>>> @@ -79,13 +80,13 @@ static void __xdp_mem_allocator_rcu_free(struct
>>> rcu_head *rcu)
>>>
>>> xa = container_of(rcu, struct xdp_mem_allocator, rcu);
>>>
>>> + /* Allocator have indicated safe to remove before this is called */
>>> + if (xa->mem.type == MEM_TYPE_PAGE_POOL)
>>> + page_pool_free(xa->page_pool);
>>> +
>>
>> What would you recommend to do for the following situation:
>>
>> Same receive queue is shared between 2 network devices. The receive ring is
>> filled by pages from page_pool, but you don't know the actual port (ndev)
>> filling this ring, because a device is recognized only after packet is
>> received.
>>
>> The API is so that xdp rxq is bind to network device, each frame has
>> reference
>> on it, so rxq ndev must be static. That means each netdev has it's own rxq
>> instance even no need in it. Thus, after your changes, page must be
>> returned to
>> the pool it was taken from, or released from old pool and recycled in
>> new one
>> somehow.
>>
>> And that is inconvenience at least. It's hard to move pages between
>> pools w/o
>> performance penalty. No way to use common pool either, as unreg_rxq now
>> drops
>> the pool and 2 rxqa can't reference same pool.
>>
>
>Within the single netdev, separate page_pool instances are anyway
>created for different RX rings, working under different NAPI's.
The circumstances are so that same RX ring is shared between 2
netdevs... and netdev can be known only after descriptor/packet is
received. Thus, while filling RX ring, there is no actual device,
but when packet is received it has to be recycled to appropriate
net device pool. Before this change there were no difference from
which pool the page was allocated to fill RX ring, as there were no
owner. After this change there is owner - netdev page pool.
For cpsw the dma unmap is common for both netdevs and no difference
who is freeing the page, but there is difference which pool it's
freed to.
So that, while filling RX ring the page is taken from page pool of
ndev1, but packet is received for ndev2, it has to be later
returned/recycled to page pool of ndev1, but when xdp buffer is
handed over to xdp prog the xdp_rxq_info has reference on ndev2 ...
And no way to predict the final ndev before packet is received, so no
way to choose appropriate page pool as now it becomes page owner.
So, while RX ring filling, the page/dma recycling is needed but should
be some way to identify page owner only after receiving packet.
Roughly speaking, something like:
pool->pages_state_hold_cnt++;
outside of page allocation API, after packet is received.
and free of the counter while allocation (w/o owing the page).
--
Regards,
Ivan Khoronzhuk
^ permalink raw reply
* [RFC bpf-next 0/7] Programming socket lookup with BPF
From: Jakub Sitnicki @ 2019-06-18 13:00 UTC (permalink / raw)
To: netdev, bpf; +Cc: kernel-team
We have been exploring an idea of making listening socket
lookup (inet_lookup) programmable with BPF.
Why? At last Netdev Marek talked [1] about two limitations of bind() API
we're hitting when running services on our edge servers:
1) sharing a port between two services
Services are accepting connections on different (disjoint) IP ranges but
use the same port. Say, packets to 192.0.2.0/24 tcp/80 go to NGINX,
while 198.51.100.0/24 tcp/80 is handled by Apache. Servers are running
as different users, in a flat single-netns setup.
2) receiving traffic on all ports
Proxy accepts connections a specific IP range but on any port [2].
In both cases we've found that bind() and a combination of INADDR_ANY,
SO_REUSEADDR, or SO_REUSEPORT doesn't allow for the setup we need, short of
binding each service to every IP:port pair combination :-)
We've resorted at first to custom patches [3], and more recently to traffic
steering with TPROXY. Not without pain points:
- XDP programs using bpf_sk_lookup helpers, like load balancers, can't
find the listening socket to check for SYN cookies with TPROXY redirect.
- TPROXY takes a reference to the listening socket on dispatch, which
raises lock contention concerns.
- Traffic steering configuration is split over several iptables rules, at
least one per service, which makes configuration changes error prone.
Now back to the patch set, it introduces a new BPF program type, dubbed
inet_lookup, that runs before listening socket lookup, and can override the
destination IP:port pair used as lookup key. Program attaches to netns in
scope of which the lookup happens.
What an inet_lookup program might look like? For the mentioned scenario
with two HTTP servers sharing port 80:
#define NET1 (IP4(192, 0, 2, 0) >> 8)
#define NET2 (IP4(198, 51, 100, 0) >> 8)
SEC("inet_lookup/demo_two_http_servers")
int demo_two_http_servers(struct bpf_inet_lookup *ctx)
{
if (ctx->family != AF_INET)
return BPF_OK;
if (ctx->local_port != 80)
return BPF_OK;
switch (bpf_ntohl(ctx->local_ip4) >> 8) {
case NET1:
ctx->local_ip4 = bpf_htonl(IP4(127, 0, 0, 1));
ctx->local_port = 81;
return BPF_REDIRECT;
case NET2:
ctx->local_ip4 = bpf_htonl(IP4(127, 0, 0, 1));
ctx->local_port = 82;
return BPF_REDIRECT;
}
return BPF_OK;
}
What are the downsides?
- BPF program, if attached, runs on the receive hot path,
- introspection is worse than for TPROXY iptables rules.
Also UDP packet steering has to be reworked. In current form we run the
inet_lookup program before checking for any connected UDP sockets, which is
unexpected.
The patches, while still in their early stages, show what we're trying to
solve. We're reaching out early for feedback to see what are the technical
concerns and if we can address them.
Just in time for the coming Netconf conference.
Thanks,
Jakub
[1] https://netdevconf.org/0x13/session.html?panel-industry-perspectives
[2] https://blog.cloudflare.com/how-we-built-spectrum/
[3] https://www.spinics.net/lists/netdev/msg370789.html
Jakub Sitnicki (7):
bpf: Introduce inet_lookup program type
ipv4: Run inet_lookup bpf program on socket lookup
ipv6: Run inet_lookup bpf program on socket lookup
bpf: Sync linux/bpf.h to tools/
libbpf: Add support for inet_lookup program type
bpf: Test destination address remapping with inet_lookup
bpf: Add verifier tests for inet_lookup context access
include/linux/bpf_types.h | 1 +
include/linux/filter.h | 17 +
include/net/inet6_hashtables.h | 39 ++
include/net/inet_hashtables.h | 39 ++
include/net/net_namespace.h | 3 +
include/uapi/linux/bpf.h | 27 +
kernel/bpf/syscall.c | 10 +
net/core/filter.c | 216 ++++++++
net/ipv4/inet_hashtables.c | 11 +-
net/ipv4/udp.c | 1 +
net/ipv6/inet6_hashtables.c | 11 +-
net/ipv6/udp.c | 6 +-
tools/include/uapi/linux/bpf.h | 27 +
tools/lib/bpf/libbpf.c | 4 +
tools/lib/bpf/libbpf.h | 2 +
tools/lib/bpf/libbpf.map | 2 +
tools/lib/bpf/libbpf_probes.c | 1 +
tools/testing/selftests/bpf/.gitignore | 1 +
tools/testing/selftests/bpf/Makefile | 6 +-
.../selftests/bpf/progs/inet_lookup_prog.c | 68 +++
.../testing/selftests/bpf/test_inet_lookup.c | 392 ++++++++++++++
.../testing/selftests/bpf/test_inet_lookup.sh | 35 ++
.../selftests/bpf/verifier/ctx_inet_lookup.c | 511 ++++++++++++++++++
23 files changed, 1418 insertions(+), 12 deletions(-)
create mode 100644 tools/testing/selftests/bpf/progs/inet_lookup_prog.c
create mode 100644 tools/testing/selftests/bpf/test_inet_lookup.c
create mode 100755 tools/testing/selftests/bpf/test_inet_lookup.sh
create mode 100644 tools/testing/selftests/bpf/verifier/ctx_inet_lookup.c
--
2.20.1
^ permalink raw reply
* [RFC bpf-next 1/7] bpf: Introduce inet_lookup program type
From: Jakub Sitnicki @ 2019-06-18 13:00 UTC (permalink / raw)
To: netdev, bpf; +Cc: kernel-team, Marek Majkowski
In-Reply-To: <20190618130050.8344-1-jakub@cloudflare.com>
Add a new BPF program type that attaches to the network namespace. Its
purpose is to run before the listening socket lookup so that we can
override the destination IP and/or port from BPF.
The inet_lookup program receives the lookup 4-tuple via context as input
for making a decision. It is allowed to overwrite the local IP & port that
are used for looking up a listening socket.
The program is not called anywhere yet. We hook it up to ipv4 and ipv6
stacks in subsequent patches.
Suggested-by: Marek Majkowski <marek@cloudflare.com>
Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
---
include/linux/bpf_types.h | 1 +
include/linux/filter.h | 17 +++
include/net/net_namespace.h | 3 +
include/uapi/linux/bpf.h | 27 +++++
kernel/bpf/syscall.c | 10 ++
net/core/filter.c | 216 ++++++++++++++++++++++++++++++++++++
6 files changed, 274 insertions(+)
diff --git a/include/linux/bpf_types.h b/include/linux/bpf_types.h
index 5a9975678d6f..9f1424146de8 100644
--- a/include/linux/bpf_types.h
+++ b/include/linux/bpf_types.h
@@ -36,6 +36,7 @@ BPF_PROG_TYPE(BPF_PROG_TYPE_LIRC_MODE2, lirc_mode2)
#endif
#ifdef CONFIG_INET
BPF_PROG_TYPE(BPF_PROG_TYPE_SK_REUSEPORT, sk_reuseport)
+BPF_PROG_TYPE(BPF_PROG_TYPE_INET_LOOKUP, inet_lookup)
#endif
BPF_MAP_TYPE(BPF_MAP_TYPE_ARRAY, array_map_ops)
diff --git a/include/linux/filter.h b/include/linux/filter.h
index 43b45d6db36d..f826fca6cc1c 100644
--- a/include/linux/filter.h
+++ b/include/linux/filter.h
@@ -1199,4 +1199,21 @@ struct bpf_sysctl_kern {
u64 tmp_reg;
};
+#ifdef CONFIG_INET
+struct bpf_inet_lookup_kern {
+ unsigned short family;
+ __be32 saddr;
+ struct in6_addr saddr6;
+ __be16 sport;
+ __be32 daddr;
+ struct in6_addr daddr6;
+ unsigned short hnum;
+};
+
+int inet_lookup_attach_bpf(const union bpf_attr *attr, struct bpf_prog *prog);
+int inet_lookup_detach_bpf(const union bpf_attr *attr);
+int inet_lookup_query_bpf(const union bpf_attr *attr,
+ union bpf_attr __user *uattr);
+#endif
+
#endif /* __LINUX_FILTER_H__ */
diff --git a/include/net/net_namespace.h b/include/net/net_namespace.h
index abb4f92456e1..6f2e5ecc8b08 100644
--- a/include/net/net_namespace.h
+++ b/include/net/net_namespace.h
@@ -167,6 +167,9 @@ struct net {
#endif
#ifdef CONFIG_XDP_SOCKETS
struct netns_xdp xdp;
+#endif
+#ifdef CONFIG_BPF_SYSCALL
+ struct bpf_prog __rcu *inet_lookup_prog;
#endif
struct sock *diag_nlsk;
atomic_t fnhe_genid;
diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index d0a23476f887..7776f36a43d1 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -170,6 +170,7 @@ enum bpf_prog_type {
BPF_PROG_TYPE_FLOW_DISSECTOR,
BPF_PROG_TYPE_CGROUP_SYSCTL,
BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE,
+ BPF_PROG_TYPE_INET_LOOKUP,
};
enum bpf_attach_type {
@@ -192,6 +193,7 @@ enum bpf_attach_type {
BPF_LIRC_MODE2,
BPF_FLOW_DISSECTOR,
BPF_CGROUP_SYSCTL,
+ BPF_INET_LOOKUP,
__MAX_BPF_ATTACH_TYPE
};
@@ -3066,6 +3068,31 @@ struct bpf_tcp_sock {
*/
};
+/* User accessible data for inet_lookup programs.
+ * New fields must be added at the end.
+ */
+struct bpf_inet_lookup {
+ __u32 family;
+ __u32 remote_ip4; /* Allows 1,2,4-byte read but no write.
+ * Stored in network byte order.
+ */
+ __u32 local_ip4; /* Allows 1,2,4-byte read and 4-byte write.
+ * Stored in network byte order.
+ */
+ __u32 remote_ip6[4]; /* Allows 1,2,4-byte read but no write.
+ * Stored in network byte order.
+ */
+ __u32 local_ip6[4]; /* Allows 1,2,4-byte read and 4-byte write.
+ * Stored in network byte order.
+ */
+ __u32 remote_port; /* Allows 4-byte read but no write.
+ * Stored in network byte order.
+ */
+ __u32 local_port; /* Allows 4-byte read and write.
+ * Stored in host byte order.
+ */
+};
+
struct bpf_sock_tuple {
union {
struct {
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 4c53cbd3329d..57813c539a41 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -1916,6 +1916,9 @@ static int bpf_prog_attach(const union bpf_attr *attr)
case BPF_CGROUP_SYSCTL:
ptype = BPF_PROG_TYPE_CGROUP_SYSCTL;
break;
+ case BPF_INET_LOOKUP:
+ ptype = BPF_PROG_TYPE_INET_LOOKUP;
+ break;
default:
return -EINVAL;
}
@@ -1940,6 +1943,9 @@ static int bpf_prog_attach(const union bpf_attr *attr)
case BPF_PROG_TYPE_FLOW_DISSECTOR:
ret = skb_flow_dissector_bpf_prog_attach(attr, prog);
break;
+ case BPF_PROG_TYPE_INET_LOOKUP:
+ ret = inet_lookup_attach_bpf(attr, prog);
+ break;
default:
ret = cgroup_bpf_prog_attach(attr, ptype, prog);
}
@@ -1997,6 +2003,8 @@ static int bpf_prog_detach(const union bpf_attr *attr)
case BPF_CGROUP_SYSCTL:
ptype = BPF_PROG_TYPE_CGROUP_SYSCTL;
break;
+ case BPF_INET_LOOKUP:
+ return inet_lookup_detach_bpf(attr);
default:
return -EINVAL;
}
@@ -2036,6 +2044,8 @@ static int bpf_prog_query(const union bpf_attr *attr,
return lirc_prog_query(attr, uattr);
case BPF_FLOW_DISSECTOR:
return skb_flow_dissector_prog_query(attr, uattr);
+ case BPF_INET_LOOKUP:
+ return inet_lookup_query_bpf(attr, uattr);
default:
return -EINVAL;
}
diff --git a/net/core/filter.c b/net/core/filter.c
index 8c18f2781afa..439e8eccb018 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -8716,4 +8716,220 @@ const struct bpf_verifier_ops sk_reuseport_verifier_ops = {
const struct bpf_prog_ops sk_reuseport_prog_ops = {
};
+
+static DEFINE_MUTEX(inet_lookup_prog_mutex);
+
+int inet_lookup_attach_bpf(const union bpf_attr *attr, struct bpf_prog *prog)
+{
+ struct net *net = current->nsproxy->net_ns;
+ struct bpf_prog *attached;
+
+ mutex_lock(&inet_lookup_prog_mutex);
+ attached = rcu_dereference_protected(net->inet_lookup_prog,
+ lockdep_is_held(&inet_lookup_prog_mutex));
+ if (attached) {
+ /* Only one BPF program can be attached at a time */
+ mutex_unlock(&inet_lookup_prog_mutex);
+ return -EEXIST;
+ }
+ rcu_assign_pointer(net->inet_lookup_prog, prog);
+ mutex_unlock(&inet_lookup_prog_mutex);
+ return 0;
+}
+
+int inet_lookup_detach_bpf(const union bpf_attr *attr)
+{
+ struct net *net = current->nsproxy->net_ns;
+ struct bpf_prog *attached;
+
+ mutex_lock(&inet_lookup_prog_mutex);
+ attached = rcu_dereference_protected(net->inet_lookup_prog,
+ lockdep_is_held(&inet_lookup_prog_mutex));
+ if (!attached) {
+ mutex_unlock(&inet_lookup_prog_mutex);
+ return -ENOENT;
+ }
+ bpf_prog_put(attached);
+ RCU_INIT_POINTER(net->inet_lookup_prog, NULL);
+ mutex_unlock(&inet_lookup_prog_mutex);
+
+ return 0;
+}
+
+int inet_lookup_query_bpf(const union bpf_attr *attr,
+ union bpf_attr __user *uattr)
+{
+ return -EOPNOTSUPP; /* TODO: Not implemented. */
+}
+
+static const struct bpf_func_proto *
+inet_lookup_func_proto(enum bpf_func_id func_id,
+ const struct bpf_prog *prog)
+{
+ return bpf_base_func_proto(func_id);
+}
+
+static bool inet_lookup_is_valid_access(int off, int size,
+ enum bpf_access_type type,
+ const struct bpf_prog *prog,
+ struct bpf_insn_access_aux *info)
+{
+ const int size_default = sizeof(__u32);
+
+ if (off < 0 || off >= sizeof(struct bpf_inet_lookup))
+ return false;
+ if (off % size != 0)
+ return false;
+
+ switch (off) {
+ case bpf_ctx_range(struct bpf_inet_lookup, remote_ip4):
+ case bpf_ctx_range_till(struct bpf_inet_lookup,
+ remote_ip6[0], remote_ip6[3]):
+ if (type != BPF_READ)
+ return false;
+ if (!bpf_ctx_narrow_access_ok(off, size, size_default))
+ return false;
+ bpf_ctx_record_field_size(info, size_default);
+ break;
+
+ case bpf_ctx_range(struct bpf_inet_lookup, local_ip4):
+ case bpf_ctx_range_till(struct bpf_inet_lookup,
+ local_ip6[0], local_ip6[3]):
+ if (type == BPF_READ) {
+ if (!bpf_ctx_narrow_access_ok(off, size, size_default))
+ return false;
+ bpf_ctx_record_field_size(info, size_default);
+ } else {
+ if (size != size_default)
+ return false;
+ }
+ break;
+
+ case bpf_ctx_range(struct bpf_inet_lookup, family):
+ case bpf_ctx_range(struct bpf_inet_lookup, remote_port):
+ if (type != BPF_READ)
+ return false;
+ if (size != size_default)
+ return false;
+ break;
+
+ case bpf_ctx_range(struct bpf_inet_lookup, local_port):
+ if (size != size_default)
+ return false;
+ break;
+
+ default:
+ return false;
+ }
+
+ return true;
+}
+
+#define LOAD_FIELD_OFF(STRUCT, FIELD, OFF) ({ \
+ *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(STRUCT, FIELD), \
+ si->dst_reg, si->src_reg, \
+ bpf_target_off(STRUCT, FIELD, \
+ FIELD_SIZEOF(STRUCT, \
+ FIELD), \
+ target_size) + (OFF)); \
+})
+
+#define LOAD_FIELD(STRUCT, FIELD) LOAD_FIELD_OFF(STRUCT, FIELD, 0)
+
+#define STORE_FIELD_OFF(STRUCT, FIELD, OFF) ({ \
+ *insn++ = BPF_STX_MEM(BPF_FIELD_SIZEOF(STRUCT, FIELD), \
+ si->dst_reg, si->src_reg, \
+ bpf_target_off(STRUCT, FIELD, \
+ FIELD_SIZEOF(STRUCT, \
+ FIELD), \
+ target_size) + (OFF)); \
+})
+
+#define STORE_FIELD(STRUCT, FIELD) STORE_FIELD_OFF(STRUCT, FIELD, 0)
+
+/* TODO: Handle 1,2-byte reads from {local,remote}_ip[46]. */
+static u32 inet_lookup_convert_ctx_access(enum bpf_access_type type,
+ const struct bpf_insn *si,
+ struct bpf_insn *insn_buf,
+ struct bpf_prog *prog,
+ u32 *target_size)
+{
+ struct bpf_insn *insn = insn_buf;
+ int off;
+
+ switch (si->off) {
+ case offsetof(struct bpf_inet_lookup, family):
+ LOAD_FIELD(struct bpf_inet_lookup_kern, family);
+ break;
+
+ case offsetof(struct bpf_inet_lookup, remote_ip4):
+ LOAD_FIELD(struct bpf_inet_lookup_kern, saddr);
+ break;
+
+ case offsetof(struct bpf_inet_lookup, local_ip4):
+ if (type == BPF_READ)
+ LOAD_FIELD(struct bpf_inet_lookup_kern, daddr);
+ else
+ STORE_FIELD(struct bpf_inet_lookup_kern, daddr);
+ break;
+
+ case bpf_ctx_range_till(struct bpf_inet_lookup,
+ remote_ip6[0], remote_ip6[3]):
+#if IS_ENABLED(CONFIG_IPV6)
+ off = si->off;
+ off -= offsetof(struct bpf_inet_lookup, remote_ip6[0]);
+
+ LOAD_FIELD_OFF(struct bpf_inet_lookup_kern,
+ saddr6.s6_addr32[0], off);
+#else
+ (void)off;
+
+ *insn++ = BPF_MOV32_IMM(si->dst_reg, 0);
+#endif
+ break;
+
+ case bpf_ctx_range_till(struct bpf_inet_lookup,
+ local_ip6[0], local_ip6[3]):
+#if IS_ENABLED(CONFIG_IPV6)
+ off = si->off;
+ off -= offsetof(struct bpf_inet_lookup, local_ip6[0]);
+
+ if (type == BPF_READ)
+ LOAD_FIELD_OFF(struct bpf_inet_lookup_kern,
+ daddr6.s6_addr32[0], off);
+ else
+ STORE_FIELD_OFF(struct bpf_inet_lookup_kern,
+ daddr6.s6_addr32[0], off);
+#else
+ (void)off;
+
+ if (type == BPF_READ)
+ *insn++ = BPF_MOV32_IMM(si->dst_reg, 0);
+#endif
+ break;
+
+ case offsetof(struct bpf_inet_lookup, remote_port):
+ LOAD_FIELD(struct bpf_inet_lookup_kern, sport);
+ break;
+
+ case offsetof(struct bpf_inet_lookup, local_port):
+ if (type == BPF_READ)
+ LOAD_FIELD(struct bpf_inet_lookup_kern, hnum);
+ else
+ STORE_FIELD(struct bpf_inet_lookup_kern, hnum);
+ break;
+ }
+
+ return insn - insn_buf;
+}
+
+const struct bpf_prog_ops inet_lookup_prog_ops = {
+};
+
+const struct bpf_verifier_ops inet_lookup_verifier_ops = {
+ .get_func_proto = inet_lookup_func_proto,
+ .is_valid_access = inet_lookup_is_valid_access,
+ .convert_ctx_access = inet_lookup_convert_ctx_access,
+};
+
#endif /* CONFIG_INET */
--
2.20.1
^ permalink raw reply related
* [RFC bpf-next 2/7] ipv4: Run inet_lookup bpf program on socket lookup
From: Jakub Sitnicki @ 2019-06-18 13:00 UTC (permalink / raw)
To: netdev, bpf; +Cc: kernel-team, Marek Majkowski
In-Reply-To: <20190618130050.8344-1-jakub@cloudflare.com>
Run a BPF program before looking up the listening socket, or in case of udp
before looking up any socket, connected or not. The program is allowed to
change the destination address & port we use as keys for the lookup,
providing its return code is BPF_REDIRECT.
This allows us to redirect traffic destined to a set of addresses and ports
without bindings sockets to all the addresses and ports we want to receive
on.
Suggested-by: Marek Majkowski <marek@cloudflare.com>
Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
---
include/net/inet_hashtables.h | 39 +++++++++++++++++++++++++++++++++++
net/ipv4/inet_hashtables.c | 11 ++++++----
net/ipv4/udp.c | 1 +
3 files changed, 47 insertions(+), 4 deletions(-)
diff --git a/include/net/inet_hashtables.h b/include/net/inet_hashtables.h
index babb14136705..7d8b58b2ded0 100644
--- a/include/net/inet_hashtables.h
+++ b/include/net/inet_hashtables.h
@@ -418,4 +418,43 @@ int __inet_hash_connect(struct inet_timewait_death_row *death_row,
int inet_hash_connect(struct inet_timewait_death_row *death_row,
struct sock *sk);
+
+#ifdef CONFIG_BPF_SYSCALL
+static inline void inet_lookup_run_bpf(struct net *net,
+ const __be32 saddr,
+ const __be16 sport,
+ __be32 *daddr,
+ unsigned short *hnum)
+{
+ struct bpf_inet_lookup_kern ctx = {
+ .family = AF_INET,
+ .saddr = saddr,
+ .sport = sport,
+ .daddr = *daddr,
+ .hnum = *hnum,
+ };
+ struct bpf_prog *prog;
+ int ret = BPF_OK;
+
+ rcu_read_lock();
+ prog = rcu_dereference(net->inet_lookup_prog);
+ if (prog)
+ ret = BPF_PROG_RUN(prog, &ctx);
+ rcu_read_unlock();
+
+ if (ret == BPF_REDIRECT) {
+ *daddr = ctx.daddr;
+ *hnum = ctx.hnum;
+ }
+}
+#else
+static inline void inet_lookup_run_bpf(struct sk_buff *skb,
+ const __be32 saddr,
+ const __be16 sport,
+ __be32 *daddr,
+ unsigned short *hnum)
+{
+}
+#endif /* CONFIG_BPF_SYSCALL */
+
#endif /* _INET_HASHTABLES_H */
diff --git a/net/ipv4/inet_hashtables.c b/net/ipv4/inet_hashtables.c
index 942265d65eb3..ae3f1da1b4f6 100644
--- a/net/ipv4/inet_hashtables.c
+++ b/net/ipv4/inet_hashtables.c
@@ -300,24 +300,27 @@ struct sock *__inet_lookup_listener(struct net *net,
const int dif, const int sdif)
{
struct inet_listen_hashbucket *ilb2;
+ unsigned short hnum2 = hnum;
struct sock *result = NULL;
+ __be32 daddr2 = daddr;
unsigned int hash2;
- hash2 = ipv4_portaddr_hash(net, daddr, hnum);
+ inet_lookup_run_bpf(net, saddr, sport, &daddr2, &hnum2);
+ hash2 = ipv4_portaddr_hash(net, daddr2, hnum2);
ilb2 = inet_lhash2_bucket(hashinfo, hash2);
result = inet_lhash2_lookup(net, ilb2, skb, doff,
- saddr, sport, daddr, hnum,
+ saddr, sport, daddr2, hnum2,
dif, sdif);
if (result)
goto done;
/* Lookup lhash2 with INADDR_ANY */
- hash2 = ipv4_portaddr_hash(net, htonl(INADDR_ANY), hnum);
+ hash2 = ipv4_portaddr_hash(net, htonl(INADDR_ANY), hnum2);
ilb2 = inet_lhash2_bucket(hashinfo, hash2);
result = inet_lhash2_lookup(net, ilb2, skb, doff,
- saddr, sport, htonl(INADDR_ANY), hnum,
+ saddr, sport, htonl(INADDR_ANY), hnum2,
dif, sdif);
done:
if (unlikely(IS_ERR(result)))
diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c
index 8fb250ed53d4..c4f4c94525ec 100644
--- a/net/ipv4/udp.c
+++ b/net/ipv4/udp.c
@@ -467,6 +467,7 @@ struct sock *__udp4_lib_lookup(struct net *net, __be32 saddr,
struct udp_hslot *hslot2;
bool exact_dif = udp_lib_exact_dif_match(net, skb);
+ inet_lookup_run_bpf(net, saddr, sport, &daddr, &hnum);
hash2 = ipv4_portaddr_hash(net, daddr, hnum);
slot2 = hash2 & udptable->mask;
hslot2 = &udptable->hash2[slot2];
--
2.20.1
^ permalink raw reply related
* [RFC bpf-next 4/7] bpf: Sync linux/bpf.h to tools/
From: Jakub Sitnicki @ 2019-06-18 13:00 UTC (permalink / raw)
To: netdev, bpf; +Cc: kernel-team
In-Reply-To: <20190618130050.8344-1-jakub@cloudflare.com>
Newly added program and context type is needed for tests in subsequent
patches.
Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
---
tools/include/uapi/linux/bpf.h | 27 +++++++++++++++++++++++++++
1 file changed, 27 insertions(+)
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index d0a23476f887..7776f36a43d1 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -170,6 +170,7 @@ enum bpf_prog_type {
BPF_PROG_TYPE_FLOW_DISSECTOR,
BPF_PROG_TYPE_CGROUP_SYSCTL,
BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE,
+ BPF_PROG_TYPE_INET_LOOKUP,
};
enum bpf_attach_type {
@@ -192,6 +193,7 @@ enum bpf_attach_type {
BPF_LIRC_MODE2,
BPF_FLOW_DISSECTOR,
BPF_CGROUP_SYSCTL,
+ BPF_INET_LOOKUP,
__MAX_BPF_ATTACH_TYPE
};
@@ -3066,6 +3068,31 @@ struct bpf_tcp_sock {
*/
};
+/* User accessible data for inet_lookup programs.
+ * New fields must be added at the end.
+ */
+struct bpf_inet_lookup {
+ __u32 family;
+ __u32 remote_ip4; /* Allows 1,2,4-byte read but no write.
+ * Stored in network byte order.
+ */
+ __u32 local_ip4; /* Allows 1,2,4-byte read and 4-byte write.
+ * Stored in network byte order.
+ */
+ __u32 remote_ip6[4]; /* Allows 1,2,4-byte read but no write.
+ * Stored in network byte order.
+ */
+ __u32 local_ip6[4]; /* Allows 1,2,4-byte read and 4-byte write.
+ * Stored in network byte order.
+ */
+ __u32 remote_port; /* Allows 4-byte read but no write.
+ * Stored in network byte order.
+ */
+ __u32 local_port; /* Allows 4-byte read and write.
+ * Stored in host byte order.
+ */
+};
+
struct bpf_sock_tuple {
union {
struct {
--
2.20.1
^ permalink raw reply related
* [RFC bpf-next 3/7] ipv6: Run inet_lookup bpf program on socket lookup
From: Jakub Sitnicki @ 2019-06-18 13:00 UTC (permalink / raw)
To: netdev, bpf; +Cc: kernel-team, Marek Majkowski
In-Reply-To: <20190618130050.8344-1-jakub@cloudflare.com>
Following the ipv4 changes, run a BPF program attached to netns in context
of which we're doing the socket lookup so that it can rewrite the
destination IP and port we use as keys for the lookup.
The program is called before the listening socket lookup for TCP, and
before connected or not-connected socket lookup for UDP.
Suggested-by: Marek Majkowski <marek@cloudflare.com>
Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
---
include/net/inet6_hashtables.h | 39 ++++++++++++++++++++++++++++++++++
net/ipv6/inet6_hashtables.c | 11 ++++++----
net/ipv6/udp.c | 6 ++++--
3 files changed, 50 insertions(+), 6 deletions(-)
diff --git a/include/net/inet6_hashtables.h b/include/net/inet6_hashtables.h
index 9db98af46985..ab06961d33a9 100644
--- a/include/net/inet6_hashtables.h
+++ b/include/net/inet6_hashtables.h
@@ -108,6 +108,45 @@ struct sock *inet6_lookup(struct net *net, struct inet_hashinfo *hashinfo,
const int dif);
int inet6_hash(struct sock *sk);
+
+#ifdef CONFIG_BPF_SYSCALL
+static inline void inet6_lookup_run_bpf(struct net *net,
+ const struct in6_addr *saddr,
+ const __be16 sport,
+ struct in6_addr *daddr,
+ unsigned short *hnum)
+{
+ struct bpf_inet_lookup_kern ctx = {
+ .family = AF_INET6,
+ .saddr6 = *saddr,
+ .sport = sport,
+ .daddr6 = *daddr,
+ .hnum = *hnum,
+ };
+ struct bpf_prog *prog;
+ int ret = BPF_OK;
+
+ rcu_read_lock();
+ prog = rcu_dereference(net->inet_lookup_prog);
+ if (prog)
+ ret = BPF_PROG_RUN(prog, &ctx);
+ rcu_read_unlock();
+
+ if (ret == BPF_REDIRECT) {
+ *daddr = ctx.daddr6;
+ *hnum = ctx.hnum;
+ }
+}
+#else
+static inline void inet6_lookup_run_bpf(struct net *net,
+ const struct in6_addr *saddr,
+ const __be16 sport,
+ struct in6_addr *daddr,
+ unsigned short *hnum)
+{
+}
+#endif /* CONFIG_BPF_SYSCALL */
+
#endif /* IS_ENABLED(CONFIG_IPV6) */
#define INET6_MATCH(__sk, __net, __saddr, __daddr, __ports, __dif, __sdif) \
diff --git a/net/ipv6/inet6_hashtables.c b/net/ipv6/inet6_hashtables.c
index f3515ebe9b3a..280a9b8bf914 100644
--- a/net/ipv6/inet6_hashtables.c
+++ b/net/ipv6/inet6_hashtables.c
@@ -158,24 +158,27 @@ struct sock *inet6_lookup_listener(struct net *net,
const unsigned short hnum, const int dif, const int sdif)
{
struct inet_listen_hashbucket *ilb2;
+ struct in6_addr daddr2 = *daddr;
+ unsigned short hnum2 = hnum;
struct sock *result = NULL;
unsigned int hash2;
- hash2 = ipv6_portaddr_hash(net, daddr, hnum);
+ inet6_lookup_run_bpf(net, saddr, sport, &daddr2, &hnum2);
+ hash2 = ipv6_portaddr_hash(net, &daddr2, hnum2);
ilb2 = inet_lhash2_bucket(hashinfo, hash2);
result = inet6_lhash2_lookup(net, ilb2, skb, doff,
- saddr, sport, daddr, hnum,
+ saddr, sport, &daddr2, hnum2,
dif, sdif);
if (result)
goto done;
/* Lookup lhash2 with in6addr_any */
- hash2 = ipv6_portaddr_hash(net, &in6addr_any, hnum);
+ hash2 = ipv6_portaddr_hash(net, &in6addr_any, hnum2);
ilb2 = inet_lhash2_bucket(hashinfo, hash2);
result = inet6_lhash2_lookup(net, ilb2, skb, doff,
- saddr, sport, &in6addr_any, hnum,
+ saddr, sport, &in6addr_any, hnum2,
dif, sdif);
done:
if (unlikely(IS_ERR(result)))
diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c
index 07fa579dfb96..6c0030ba83c6 100644
--- a/net/ipv6/udp.c
+++ b/net/ipv6/udp.c
@@ -196,17 +196,19 @@ struct sock *__udp6_lib_lookup(struct net *net,
struct sk_buff *skb)
{
unsigned short hnum = ntohs(dport);
+ struct in6_addr daddr2 = *daddr;
unsigned int hash2, slot2;
struct udp_hslot *hslot2;
struct sock *result;
bool exact_dif = udp6_lib_exact_dif_match(net, skb);
- hash2 = ipv6_portaddr_hash(net, daddr, hnum);
+ inet6_lookup_run_bpf(net, saddr, sport, &daddr2, &hnum);
+ hash2 = ipv6_portaddr_hash(net, &daddr2, hnum);
slot2 = hash2 & udptable->mask;
hslot2 = &udptable->hash2[slot2];
result = udp6_lib_lookup2(net, saddr, sport,
- daddr, hnum, dif, sdif, exact_dif,
+ &daddr2, hnum, dif, sdif, exact_dif,
hslot2, skb);
if (!result) {
hash2 = ipv6_portaddr_hash(net, &in6addr_any, hnum);
--
2.20.1
^ permalink raw reply related
* [RFC bpf-next 5/7] libbpf: Add support for inet_lookup program type
From: Jakub Sitnicki @ 2019-06-18 13:00 UTC (permalink / raw)
To: netdev, bpf; +Cc: kernel-team
In-Reply-To: <20190618130050.8344-1-jakub@cloudflare.com>
Make libbpf aware of the newly added program type. Reserve a section name
for it.
Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
---
tools/lib/bpf/libbpf.c | 4 ++++
tools/lib/bpf/libbpf.h | 2 ++
tools/lib/bpf/libbpf.map | 2 ++
tools/lib/bpf/libbpf_probes.c | 1 +
4 files changed, 9 insertions(+)
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index e725fa86b189..84dfdfc0a971 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -2244,6 +2244,7 @@ static bool bpf_prog_type__needs_kver(enum bpf_prog_type type)
case BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE:
case BPF_PROG_TYPE_PERF_EVENT:
case BPF_PROG_TYPE_CGROUP_SYSCTL:
+ case BPF_PROG_TYPE_INET_LOOKUP:
return false;
case BPF_PROG_TYPE_KPROBE:
default:
@@ -3110,6 +3111,7 @@ BPF_PROG_TYPE_FNS(tracepoint, BPF_PROG_TYPE_TRACEPOINT);
BPF_PROG_TYPE_FNS(raw_tracepoint, BPF_PROG_TYPE_RAW_TRACEPOINT);
BPF_PROG_TYPE_FNS(xdp, BPF_PROG_TYPE_XDP);
BPF_PROG_TYPE_FNS(perf_event, BPF_PROG_TYPE_PERF_EVENT);
+BPF_PROG_TYPE_FNS(inet_lookup, BPF_PROG_TYPE_INET_LOOKUP);
void bpf_program__set_expected_attach_type(struct bpf_program *prog,
enum bpf_attach_type type)
@@ -3197,6 +3199,8 @@ static const struct {
BPF_CGROUP_UDP6_SENDMSG),
BPF_EAPROG_SEC("cgroup/sysctl", BPF_PROG_TYPE_CGROUP_SYSCTL,
BPF_CGROUP_SYSCTL),
+ BPF_EAPROG_SEC("inet_lookup", BPF_PROG_TYPE_INET_LOOKUP,
+ BPF_INET_LOOKUP),
};
#undef BPF_PROG_SEC_IMPL
diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h
index 2e594a0fa961..283dac0f6d13 100644
--- a/tools/lib/bpf/libbpf.h
+++ b/tools/lib/bpf/libbpf.h
@@ -240,6 +240,7 @@ LIBBPF_API int bpf_program__set_sched_cls(struct bpf_program *prog);
LIBBPF_API int bpf_program__set_sched_act(struct bpf_program *prog);
LIBBPF_API int bpf_program__set_xdp(struct bpf_program *prog);
LIBBPF_API int bpf_program__set_perf_event(struct bpf_program *prog);
+LIBBPF_API int bpf_program__set_inet_lookup(struct bpf_program *prog);
LIBBPF_API void bpf_program__set_type(struct bpf_program *prog,
enum bpf_prog_type type);
LIBBPF_API void
@@ -254,6 +255,7 @@ LIBBPF_API bool bpf_program__is_sched_cls(struct bpf_program *prog);
LIBBPF_API bool bpf_program__is_sched_act(struct bpf_program *prog);
LIBBPF_API bool bpf_program__is_xdp(struct bpf_program *prog);
LIBBPF_API bool bpf_program__is_perf_event(struct bpf_program *prog);
+LIBBPF_API bool bpf_program__is_inet_lookup(struct bpf_program *prog);
/*
* No need for __attribute__((packed)), all members of 'bpf_map_def'
diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map
index 2c6d835620d2..e55d8e5d6fd4 100644
--- a/tools/lib/bpf/libbpf.map
+++ b/tools/lib/bpf/libbpf.map
@@ -67,6 +67,7 @@ LIBBPF_0.0.1 {
bpf_prog_test_run;
bpf_prog_test_run_xattr;
bpf_program__fd;
+ bpf_program__is_inet_lookup;
bpf_program__is_kprobe;
bpf_program__is_perf_event;
bpf_program__is_raw_tracepoint;
@@ -84,6 +85,7 @@ LIBBPF_0.0.1 {
bpf_program__priv;
bpf_program__set_expected_attach_type;
bpf_program__set_ifindex;
+ bpf_program__set_inet_lookup;
bpf_program__set_kprobe;
bpf_program__set_perf_event;
bpf_program__set_prep;
diff --git a/tools/lib/bpf/libbpf_probes.c b/tools/lib/bpf/libbpf_probes.c
index 5e2aa83f637a..5094f32d33a7 100644
--- a/tools/lib/bpf/libbpf_probes.c
+++ b/tools/lib/bpf/libbpf_probes.c
@@ -101,6 +101,7 @@ probe_load(enum bpf_prog_type prog_type, const struct bpf_insn *insns,
case BPF_PROG_TYPE_SK_REUSEPORT:
case BPF_PROG_TYPE_FLOW_DISSECTOR:
case BPF_PROG_TYPE_CGROUP_SYSCTL:
+ case BPF_PROG_TYPE_INET_LOOKUP:
default:
break;
}
--
2.20.1
^ permalink raw reply related
* [RFC bpf-next 7/7] bpf: Add verifier tests for inet_lookup context access
From: Jakub Sitnicki @ 2019-06-18 13:00 UTC (permalink / raw)
To: netdev, bpf; +Cc: kernel-team
In-Reply-To: <20190618130050.8344-1-jakub@cloudflare.com>
Exercise verifier access checks for bpf_inet_lookup context object fields.
Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
---
.../selftests/bpf/verifier/ctx_inet_lookup.c | 511 ++++++++++++++++++
1 file changed, 511 insertions(+)
create mode 100644 tools/testing/selftests/bpf/verifier/ctx_inet_lookup.c
diff --git a/tools/testing/selftests/bpf/verifier/ctx_inet_lookup.c b/tools/testing/selftests/bpf/verifier/ctx_inet_lookup.c
new file mode 100644
index 000000000000..b4555fb03e17
--- /dev/null
+++ b/tools/testing/selftests/bpf/verifier/ctx_inet_lookup.c
@@ -0,0 +1,511 @@
+{
+ "valid 1,2,4-byte read bpf_inet_lookup remote_ip4",
+ .insns = {
+ /* 4-byte read */
+ BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, remote_ip4)),
+ /* 2-byte read */
+ BPF_LDX_MEM(BPF_H, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, remote_ip4)),
+ BPF_LDX_MEM(BPF_H, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, remote_ip4) + 2),
+ /* 1-byte read */
+ BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, remote_ip4)),
+ BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, remote_ip4) + 3),
+ BPF_EXIT_INSN(),
+ },
+ .result = ACCEPT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 8-byte read bpf_inet_lookup remote_ip4",
+ .insns = {
+ /* 8-byte read */
+ BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, remote_ip4)),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 8-byte write bpf_inet_lookup remote_ip4",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 0x7f000001U),
+ /* 4-byte write */
+ BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, remote_ip4)),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 4-byte write bpf_inet_lookup remote_ip4",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 0x7f000001U),
+ /* 4-byte write */
+ BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, remote_ip4)),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 2-byte write bpf_inet_lookup remote_ip4",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 0x7f000001U),
+ /* 2-byte write */
+ BPF_STX_MEM(BPF_H, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, remote_ip4)),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 1-byte write bpf_inet_lookup remote_ip4",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 0x7f000001U),
+ /* 1-byte write */
+ BPF_STX_MEM(BPF_B, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, remote_ip4)),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "valid 1,2,4-byte read bpf_inet_lookup local_ip4",
+ .insns = {
+ /* 4-byte read */
+ BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, local_ip4)),
+ /* 2-byte read */
+ BPF_LDX_MEM(BPF_H, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, local_ip4)),
+ BPF_LDX_MEM(BPF_H, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, local_ip4) + 2),
+ /* 1-byte read */
+ BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, local_ip4)),
+ BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, local_ip4) + 3),
+ BPF_MOV64_IMM(BPF_REG_0, 0),
+ BPF_EXIT_INSN(),
+ },
+ .result = ACCEPT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 8-byte read bpf_inet_lookup local_ip4",
+ .insns = {
+ BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, local_ip4)),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "valid 4-byte write bpf_inet_lookup local_ip4",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 0x7f000001U),
+ BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, local_ip4)),
+ BPF_EXIT_INSN(),
+ },
+ .result = ACCEPT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 8-byte write bpf_inet_lookup local_ip4",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 0x7f000001U),
+ BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, local_ip4)),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 2-byte write bpf_inet_lookup local_ip4",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 0x7f000001U),
+ BPF_STX_MEM(BPF_H, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, local_ip4)),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 1-byte write bpf_inet_lookup local_ip4",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 0x7f000001U),
+ BPF_STX_MEM(BPF_B, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, local_ip4)),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "valid 1,2,4-byte read bpf_inet_lookup remote_ip6",
+ .insns = {
+ /* 4-byte read */
+ BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, remote_ip6[0])),
+ BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, remote_ip6[3])),
+ /* 2-byte read */
+ BPF_LDX_MEM(BPF_H, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, remote_ip6[0])),
+ BPF_LDX_MEM(BPF_H, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup,
+ remote_ip6[3]) + 2),
+ /* 1-byte read */
+ BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, remote_ip6[0])),
+ BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup,
+ remote_ip6[3]) + 3),
+ BPF_EXIT_INSN(),
+ },
+ .result = ACCEPT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 8-byte read bpf_inet_lookup remote_ip6",
+ .insns = {
+ BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, remote_ip6[0])),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 8-byte write bpf_inet_lookup remote_ip6",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 0x00000001U),
+ BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, remote_ip6[0])),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 4-byte write bpf_inet_lookup remote_ip6",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 0x00000001U),
+ BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, remote_ip6[0])),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 2-byte write bpf_inet_lookup remote_ip6",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 0x00000001U),
+ BPF_STX_MEM(BPF_H, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, remote_ip6[0])),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 1-byte write bpf_inet_lookup remote_ip6",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 0x00000001U),
+ BPF_STX_MEM(BPF_B, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, remote_ip6[0])),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "valid 1,2,4-byte read bpf_inet_lookup local_ip6",
+ .insns = {
+ /* 4-byte read */
+ BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, local_ip6[0])),
+ BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, local_ip6[3])),
+ /* 2-byte read */
+ BPF_LDX_MEM(BPF_H, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, local_ip6[0])),
+ BPF_LDX_MEM(BPF_H, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, local_ip6[3]) + 2),
+ /* 1-byte read */
+ BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, local_ip6[0])),
+ BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, local_ip6[3]) + 3),
+ BPF_EXIT_INSN(),
+ },
+ .result = ACCEPT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 8-byte read bpf_inet_lookup local_ip6",
+ .insns = {
+ BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, local_ip6[0])),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 8-byte write bpf_inet_lookup local_ip6",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 0x00000001U),
+ BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, local_ip6[0])),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "valid 4-byte write bpf_inet_lookup local_ip6",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 0x00000001U),
+ BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, local_ip6[0])),
+ BPF_EXIT_INSN(),
+ },
+ .result = ACCEPT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 2-byte write bpf_inet_lookup local_ip6",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 0x00000001U),
+ BPF_STX_MEM(BPF_H, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, local_ip6[0])),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 1-byte write bpf_inet_lookup local_ip6",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 0x00000001U),
+ BPF_STX_MEM(BPF_B, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, local_ip6[0])),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "valid 4-byte read bpf_inet_lookup remote_port",
+ .insns = {
+ BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, remote_port)),
+ BPF_EXIT_INSN(),
+ },
+ .result = ACCEPT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 8-byte read bpf_inet_lookup remote_port",
+ .insns = {
+ BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, remote_port)),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 2-byte read bpf_inet_lookup remote_port",
+ .insns = {
+ BPF_LDX_MEM(BPF_H, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, remote_port)),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 1-byte read bpf_inet_lookup remote_port",
+ .insns = {
+ BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, remote_port)),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 8-byte write bpf_inet_lookup remote_port",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 1234),
+ BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, remote_port)),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 4-byte write bpf_inet_lookup remote_port",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 1234),
+ BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, remote_port)),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 2-byte write bpf_inet_lookup remote_port",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 1234),
+ BPF_STX_MEM(BPF_H, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, remote_port)),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 1-byte write bpf_inet_lookup remote_port",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 1234),
+ BPF_STX_MEM(BPF_B, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, remote_port)),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "valid 4-byte read bpf_inet_lookup local_port",
+ .insns = {
+ BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, local_port)),
+ BPF_EXIT_INSN(),
+ },
+ .result = ACCEPT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 8-byte read bpf_inet_lookup local_port",
+ .insns = {
+ BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, local_port)),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 2-byte read bpf_inet_lookup local_port",
+ .insns = {
+ BPF_LDX_MEM(BPF_H, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, local_port)),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 1-byte read bpf_inet_lookup local_port",
+ .insns = {
+ BPF_LDX_MEM(BPF_B, BPF_REG_0, BPF_REG_1,
+ offsetof(struct bpf_inet_lookup, local_port)),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "valid 4-byte write bpf_inet_lookup local_port",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 1234),
+ BPF_STX_MEM(BPF_W, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, local_port)),
+ BPF_EXIT_INSN(),
+ },
+ .result = ACCEPT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 8-byte write bpf_inet_lookup local_port",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 1234),
+ BPF_STX_MEM(BPF_DW, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, local_port)),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 2-byte write bpf_inet_lookup local_port",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 1234),
+ BPF_STX_MEM(BPF_H, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, local_port)),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
+{
+ "invalid 1-byte write bpf_inet_lookup local_port",
+ .insns = {
+ BPF_MOV64_IMM(BPF_REG_0, 1234),
+ BPF_STX_MEM(BPF_B, BPF_REG_1, BPF_REG_0,
+ offsetof(struct bpf_inet_lookup, local_port)),
+ BPF_EXIT_INSN(),
+ },
+ .errstr = "invalid bpf_context access",
+ .result = REJECT,
+ .prog_type = BPF_PROG_TYPE_INET_LOOKUP,
+},
--
2.20.1
^ permalink raw reply related
* [RFC bpf-next 6/7] bpf: Test destination address remapping with inet_lookup
From: Jakub Sitnicki @ 2019-06-18 13:00 UTC (permalink / raw)
To: netdev, bpf; +Cc: kernel-team
In-Reply-To: <20190618130050.8344-1-jakub@cloudflare.com>
Check if destination IP address and/or port remapping happens when an
inet_lookup program is attached to the net namespace.
Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
---
tools/testing/selftests/bpf/.gitignore | 1 +
tools/testing/selftests/bpf/Makefile | 6 +-
.../selftests/bpf/progs/inet_lookup_prog.c | 68 +++
.../testing/selftests/bpf/test_inet_lookup.c | 392 ++++++++++++++++++
.../testing/selftests/bpf/test_inet_lookup.sh | 35 ++
5 files changed, 500 insertions(+), 2 deletions(-)
create mode 100644 tools/testing/selftests/bpf/progs/inet_lookup_prog.c
create mode 100644 tools/testing/selftests/bpf/test_inet_lookup.c
create mode 100755 tools/testing/selftests/bpf/test_inet_lookup.sh
diff --git a/tools/testing/selftests/bpf/.gitignore b/tools/testing/selftests/bpf/.gitignore
index 7470327edcfe..c1492cb188e5 100644
--- a/tools/testing/selftests/bpf/.gitignore
+++ b/tools/testing/selftests/bpf/.gitignore
@@ -39,3 +39,4 @@ libbpf.so.*
test_hashmap
test_btf_dump
xdping
+test_inet_lookup
diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index ae5c3e576c2f..aa68dc091888 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -61,7 +61,8 @@ TEST_PROGS := test_kmod.sh \
test_tcp_check_syncookie.sh \
test_tc_tunnel.sh \
test_tc_edt.sh \
- test_xdping.sh
+ test_xdping.sh \
+ test_inet_lookup.sh
TEST_PROGS_EXTENDED := with_addr.sh \
with_tunnels.sh \
@@ -70,7 +71,8 @@ TEST_PROGS_EXTENDED := with_addr.sh \
# Compile but not part of 'make run_tests'
TEST_GEN_PROGS_EXTENDED = test_libbpf_open test_sock_addr test_skb_cgroup_id_user \
- flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user
+ flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \
+ test_inet_lookup
include ../lib.mk
diff --git a/tools/testing/selftests/bpf/progs/inet_lookup_prog.c b/tools/testing/selftests/bpf/progs/inet_lookup_prog.c
new file mode 100644
index 000000000000..5f3d2b1f94eb
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/inet_lookup_prog.c
@@ -0,0 +1,68 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/bpf.h>
+#include <sys/socket.h>
+
+#include "bpf_endian.h"
+#include "bpf_helpers.h"
+
+#define IP4(a, b, c, d) ((__u32)( \
+ ((__u32)((a) & (__u32)0xffUL) << 24) | \
+ ((__u32)((b) & (__u32)0xffUL) << 16) | \
+ ((__u32)((c) & (__u32)0xffUL) << 8) | \
+ ((__u32)((d) & (__u32)0xffUL) << 0)))
+
+static const __u32 CONNECT_PORT = 7007;
+static const __u32 LISTEN_PORT = 8008;
+
+static const __u32 CONNECT_IP4 = IP4(127, 0, 0, 1);
+static const __u32 LISTEN_IP4 = IP4(127, 0, 0, 2);
+
+/* Remap destination port CONNECT_PORT -> LISTEN_PORT. */
+SEC("inet_lookup/remap_port")
+int inet4_remap_port(struct bpf_inet_lookup *ctx)
+{
+ if (ctx->local_port != CONNECT_PORT)
+ return BPF_OK;
+
+ ctx->local_port = LISTEN_PORT;
+ return BPF_REDIRECT;
+}
+
+/* Remap destination IP CONNECT_IP4 -> LISTEN_IP4. */
+SEC("inet_lookup/remap_ip4")
+int inet4_remap_ip(struct bpf_inet_lookup *ctx)
+{
+ if (ctx->family != AF_INET)
+ return BPF_OK;
+ if (ctx->local_port != CONNECT_PORT)
+ return BPF_OK;
+ if (ctx->local_ip4 != bpf_htonl(CONNECT_IP4))
+ return BPF_OK;
+
+ ctx->local_ip4 = bpf_htonl(LISTEN_IP4);
+ return BPF_REDIRECT;
+}
+
+/* Remap destination IP CONNECT_IP6 -> LISTEN_IP6. */
+SEC("inet_lookup/remap_ip6")
+int inet6_remap_ip(struct bpf_inet_lookup *ctx)
+{
+ if (ctx->family != AF_INET6)
+ return BPF_OK;
+ if (ctx->local_port != CONNECT_PORT)
+ return BPF_OK;
+ if (ctx->local_ip6[0] != bpf_htonl(0xfd000000) ||
+ ctx->local_ip6[1] != bpf_htonl(0x00000000) ||
+ ctx->local_ip6[2] != bpf_htonl(0x00000000) ||
+ ctx->local_ip6[3] != bpf_htonl(0x00000001))
+ return BPF_OK;
+
+ ctx->local_ip6[0] = bpf_htonl(0xfd000000);
+ ctx->local_ip6[1] = bpf_htonl(0x00000000);
+ ctx->local_ip6[2] = bpf_htonl(0x00000000);
+ ctx->local_ip6[3] = bpf_htonl(0x00000002);
+ return BPF_REDIRECT;
+}
+
+char _license[] SEC("license") = "GPL";
+__u32 _version SEC("version") = 1;
diff --git a/tools/testing/selftests/bpf/test_inet_lookup.c b/tools/testing/selftests/bpf/test_inet_lookup.c
new file mode 100644
index 000000000000..d4e440655252
--- /dev/null
+++ b/tools/testing/selftests/bpf/test_inet_lookup.c
@@ -0,0 +1,392 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Echo test with the server not receiving at the same IP:port as the
+ * client sends the request to. Use BPF inet_lookup program to remap
+ * IP/port on socket lookup and direct the packets to the server.
+ */
+
+#include <arpa/inet.h>
+#include <errno.h>
+#include <error.h>
+#include <stdio.h>
+#include <unistd.h>
+
+#include <bpf/libbpf.h>
+#include <bpf/bpf.h>
+
+#include "bpf_rlimit.h"
+#include "bpf_util.h"
+
+#define BPF_FILE "./inet_lookup_prog.o"
+#define MAX_ERROR_LEN 256
+
+#define EXT_IP4 "127.0.0.1"
+#define INT_IP4 "127.0.0.2"
+#define EXT_IP6 "fd00::1"
+#define INT_IP6 "fd00::2"
+#define EXT_PORT 7007
+#define INT_PORT 8008
+
+struct inet_addr {
+ const char *ip;
+ unsigned short port;
+};
+
+struct test {
+ const char *desc;
+ const char *bpf_prog;
+
+ struct {
+ int family;
+ int type;
+ } socket;
+
+ struct inet_addr recv_at;
+ struct inet_addr send_to;
+};
+
+static const struct test tests[] = {
+ {
+ .desc = "TCP IPv4 remap port",
+ .bpf_prog = "inet_lookup/remap_port",
+ .socket = { AF_INET, SOCK_STREAM },
+ .recv_at = { EXT_IP4, INT_PORT },
+ .send_to = { EXT_IP4, EXT_PORT },
+ },
+ {
+ .desc = "TCP IPv4 remap IP",
+ .bpf_prog = "inet_lookup/remap_ip4",
+ .socket = { AF_INET, SOCK_STREAM },
+ .recv_at = { INT_IP4, EXT_PORT },
+ .send_to = { EXT_IP4, EXT_PORT },
+ },
+ {
+ .desc = "TCP IPv6 remap port",
+ .bpf_prog = "inet_lookup/remap_port",
+ .socket = { AF_INET6, SOCK_STREAM },
+ .recv_at = { EXT_IP6, INT_PORT },
+ .send_to = { EXT_IP6, EXT_PORT },
+ },
+ {
+ .desc = "TCP IPv6 remap IP",
+ .bpf_prog = "inet_lookup/remap_ip6",
+ .socket = { AF_INET6, SOCK_STREAM },
+ .recv_at = { INT_IP6, EXT_PORT },
+ .send_to = { EXT_IP6, EXT_PORT },
+ },
+ {
+ .desc = "UDP IPv4 remap port",
+ .bpf_prog = "inet_lookup/remap_port",
+ .socket = { AF_INET, SOCK_DGRAM },
+ .recv_at = { EXT_IP4, INT_PORT },
+ .send_to = { EXT_IP4, EXT_PORT },
+ },
+ {
+ .desc = "UDP IPv4 remap IP",
+ .bpf_prog = "inet_lookup/remap_ip4",
+ .socket = { AF_INET, SOCK_DGRAM },
+ .recv_at = { INT_IP4, EXT_PORT },
+ .send_to = { EXT_IP4, EXT_PORT },
+ },
+ {
+ .desc = "UDP IPv6 remap port",
+ .bpf_prog = "inet_lookup/remap_port",
+ .socket = { AF_INET6, SOCK_DGRAM },
+ .recv_at = { EXT_IP6, INT_PORT },
+ .send_to = { EXT_IP6, EXT_PORT },
+ },
+ {
+ .desc = "UDP IPv6 remap IP",
+ .bpf_prog = "inet_lookup/remap_ip6",
+ .socket = { AF_INET6, SOCK_DGRAM },
+ .recv_at = { INT_IP6, EXT_PORT },
+ .send_to = { EXT_IP6, EXT_PORT },
+ },
+};
+
+static void make_addr(int family, const char *ip, int port,
+ struct sockaddr_storage *ss, int *sz)
+{
+ struct sockaddr_in *addr4;
+ struct sockaddr_in6 *addr6;
+
+ switch (family) {
+ case AF_INET:
+ addr4 = (struct sockaddr_in *)ss;
+ addr4->sin_family = AF_INET;
+ addr4->sin_port = htons(port);
+ if (!inet_pton(AF_INET, ip, &addr4->sin_addr))
+ error(1, errno, "inet_pton failed: %s", ip);
+ *sz = sizeof(*addr4);
+ break;
+ case AF_INET6:
+ addr6 = (struct sockaddr_in6 *)ss;
+ addr6->sin6_family = AF_INET6;
+ addr6->sin6_port = htons(port);
+ if (!inet_pton(AF_INET6, ip, &addr6->sin6_addr))
+ error(1, errno, "inet_pton failed: %s", ip);
+ *sz = sizeof(*addr6);
+ break;
+ default:
+ error(1, 0, "unsupported family %d", family);
+ }
+}
+
+static int make_server(int family, int type, const char *ip, int port)
+{
+ struct sockaddr_storage ss = {0};
+ int fd, opt, sz;
+
+ make_addr(family, ip, port, &ss, &sz);
+
+ fd = socket(family, type, 0);
+ if (fd < 0)
+ error(1, errno, "failed to create listen socket");
+
+ opt = 1;
+ if (setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, &opt, sizeof(opt)))
+ error(1, errno, "failed to set SO_REUSEPORT");
+ if (family == AF_INET && type == SOCK_DGRAM) {
+ if (setsockopt(fd, SOL_IP, IP_RECVORIGDSTADDR,
+ &opt, sizeof(opt)))
+ error(1, errno, "failed to set IP_RECVORIGDSTADDR");
+ }
+ if (family == AF_INET6 && type == SOCK_DGRAM) {
+ if (setsockopt(fd, SOL_IPV6, IPV6_RECVORIGDSTADDR,
+ &opt, sizeof(opt)))
+ error(1, errno, "failed to set IPV6_RECVORIGDSTADDR");
+ }
+
+ if (bind(fd, (struct sockaddr *)&ss, sz))
+ error(1, errno, "failed to bind listen socket");
+
+ if (type == SOCK_STREAM && listen(fd, 1))
+ error(1, errno, "failed to listen on port %d", port);
+
+ return fd;
+}
+
+static int make_client(int family, int type, const char *ip, int port)
+{
+ struct sockaddr_storage ss = {0};
+ struct sockaddr *sa;
+ int fd, sz;
+
+ make_addr(family, ip, port, &ss, &sz);
+ sa = (struct sockaddr *)&ss;
+
+ fd = socket(family, type, 0);
+ if (fd < 0)
+ error(1, errno, "failed to create socket");
+
+ if (connect(fd, sa, sz))
+ error(1, errno, "failed to connect socket");
+
+ return fd;
+}
+
+static void send_byte(int fd)
+{
+ if (send(fd, "a", 1, 0) < 1)
+ error(1, errno, "failed to send message");
+}
+
+static void recv_byte(int fd)
+{
+ char buf[1];
+
+ if (recv(fd, buf, sizeof(buf), 0) < 1)
+ error(1, errno, "failed to receive message");
+}
+
+static void tcp_recv_send(int server_fd)
+{
+ char buf[1];
+ size_t len;
+ ssize_t n;
+ int fd;
+
+ fd = accept(server_fd, NULL, NULL);
+ if (fd < 0)
+ error(1, errno, "failed to accept");
+
+ len = sizeof(buf);
+ n = recv(fd, buf, len, 0);
+ if (n < 0)
+ error(1, errno, "failed to receive");
+ if (n < len)
+ error(1, 0, "partial receive");
+
+ n = send(fd, buf, len, 0);
+ if (n < 0)
+ error(1, errno, "failed to send");
+ if (n < len)
+ error(1, 0, "partial send");
+
+ close(fd);
+}
+
+static void udp_recv_send(int server_fd)
+{
+ char cmsg_buf[CMSG_SPACE(sizeof(struct sockaddr_storage))];
+ struct sockaddr_storage *dst_addr = NULL;
+ struct sockaddr_storage src_addr;
+ struct msghdr msg = { 0 };
+ struct iovec iov = { 0 };
+ struct cmsghdr *cm;
+ char buf[1];
+ ssize_t n;
+ int fd;
+
+ iov.iov_base = buf;
+ iov.iov_len = sizeof(buf);
+
+ msg.msg_name = &src_addr;
+ msg.msg_namelen = sizeof(src_addr);
+ msg.msg_iov = &iov;
+ msg.msg_iovlen = 1;
+ msg.msg_control = cmsg_buf;
+ msg.msg_controllen = sizeof(cmsg_buf);
+
+ n = recvmsg(server_fd, &msg, 0);
+ if (n < 0)
+ error(1, errno, "failed to receive");
+ if (n < sizeof(buf))
+ error(1, 0, "partial receive");
+ if (msg.msg_flags & MSG_CTRUNC)
+ error(1, errno, "truncated cmsg");
+
+ for (cm = CMSG_FIRSTHDR(&msg); cm; cm = CMSG_NXTHDR(&msg, cm)) {
+ if ((cm->cmsg_level == SOL_IP &&
+ cm->cmsg_type == IP_ORIGDSTADDR) ||
+ (cm->cmsg_level == SOL_IPV6 &&
+ cm->cmsg_type == IPV6_ORIGDSTADDR)) {
+ dst_addr = (struct sockaddr_storage *)CMSG_DATA(cm);
+ break;
+ }
+ error(0, 0, "ignored cmsg at level %d type %d",
+ cm->cmsg_level, cm->cmsg_type);
+ }
+ if (!dst_addr)
+ error(1, 0, "failed to get destination address");
+
+ /* Reply from original destination address. */
+ fd = socket(dst_addr->ss_family, SOCK_DGRAM, 0);
+ if (fd < 0)
+ error(1, errno, "failed to create socket");
+
+ if (bind(fd, (struct sockaddr *)dst_addr, sizeof(*dst_addr)))
+ error(1, errno, "failed to bind socket");
+
+ msg.msg_control = NULL;
+ msg.msg_controllen = 0;
+ n = sendmsg(fd, &msg, 0);
+ if (n < 0)
+ error(1, errno, "failed to send");
+ if (n < sizeof(buf))
+ error(1, 0, "partial send");
+
+ close(fd);
+}
+
+static void tcp_echo(int client_fd, int server_fd)
+{
+ send_byte(client_fd);
+ tcp_recv_send(server_fd);
+ recv_byte(client_fd);
+}
+
+static void udp_echo(int client_fd, int server_fd)
+{
+ send_byte(client_fd);
+ udp_recv_send(server_fd);
+ recv_byte(client_fd);
+}
+
+static struct bpf_object *load_progs(void)
+{
+ char buf[MAX_ERROR_LEN];
+ struct bpf_object *obj;
+ int prog_fd;
+ int err;
+
+ err = bpf_prog_load(BPF_FILE, BPF_PROG_TYPE_UNSPEC, &obj, &prog_fd);
+ if (err) {
+ libbpf_strerror(err, buf, ARRAY_SIZE(buf));
+ error(1, 0, "failed to open bpf file: %s", buf);
+ }
+
+ return obj;
+}
+
+static void attach_prog(struct bpf_object *obj, const char *sec)
+{
+ enum bpf_attach_type attach_type;
+ struct bpf_program *prog;
+ char buf[MAX_ERROR_LEN];
+ int target_fd = -1;
+ int prog_fd;
+ int err;
+
+ prog = bpf_object__find_program_by_title(obj, sec);
+ err = libbpf_get_error(prog);
+ if (err) {
+ libbpf_strerror(err, buf, ARRAY_SIZE(buf));
+ error(1, 0, "failed to find section \"%s\": %s", sec, buf);
+ }
+
+ err = libbpf_attach_type_by_name(sec, &attach_type);
+ if (err) {
+ libbpf_strerror(err, buf, ARRAY_SIZE(buf));
+ error(1, 0, "failed to identify attach type: %s", buf);
+ }
+
+ prog_fd = bpf_program__fd(prog);
+ if (prog_fd < 0)
+ error(1, errno, "failed to get prog fd");
+
+ err = bpf_prog_detach(target_fd, attach_type);
+ if (err && err != -EPERM)
+ error(1, -err, "failed to detach prog");
+
+ err = bpf_prog_attach(prog_fd, target_fd, attach_type, 0);
+ if (err)
+ error(1, -err, "failed to attach prog");
+}
+
+static void run_test(const struct test *t, struct bpf_object *obj)
+{
+ int client_fd, server_fd;
+
+ fprintf(stderr, "%s\n", t->desc);
+ attach_prog(obj, t->bpf_prog);
+
+ server_fd = make_server(t->socket.family, t->socket.type,
+ t->recv_at.ip, t->recv_at.port);
+ client_fd = make_client(t->socket.family, t->socket.type,
+ t->send_to.ip, t->send_to.port);
+
+ if (t->socket.type == SOCK_STREAM)
+ tcp_echo(client_fd, server_fd);
+ else
+ udp_echo(client_fd, server_fd);
+
+ close(client_fd);
+ close(server_fd);
+}
+
+int main(void)
+{
+ struct bpf_object *obj;
+ const struct test *t;
+
+ obj = load_progs();
+
+ for (t = tests; t < tests + ARRAY_SIZE(tests); t++)
+ run_test(t, obj);
+
+ bpf_object__unload(obj);
+
+ fprintf(stderr, "PASS\n");
+ return 0;
+}
diff --git a/tools/testing/selftests/bpf/test_inet_lookup.sh b/tools/testing/selftests/bpf/test_inet_lookup.sh
new file mode 100755
index 000000000000..5efb42fbdf59
--- /dev/null
+++ b/tools/testing/selftests/bpf/test_inet_lookup.sh
@@ -0,0 +1,35 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+if [[ $EUID -ne 0 ]]; then
+ echo "This script must be run as root"
+ echo "FAIL"
+ exit 1
+fi
+
+# Run the script in a dedicated network namespace.
+if [[ -z $(ip netns identify $$) ]]; then
+ ../net/in_netns.sh "$0" "$@"
+ exit $?
+fi
+
+readonly IP6_1="fd00::1"
+readonly IP6_2="fd00::2"
+
+setup()
+{
+ ip -6 addr add ${IP6_1}/128 dev lo
+ ip -6 addr add ${IP6_2}/128 dev lo
+}
+
+cleanup()
+{
+ ip -6 addr del ${IP6_1}/128 dev lo
+ ip -6 addr del ${IP6_2}/128 dev lo
+}
+
+trap cleanup EXIT
+setup
+
+./test_inet_lookup
+exit $?
--
2.20.1
^ permalink raw reply related
* Re: [PATCH] mm: mempolicy: handle vma with unmovable pages mapped correctly in mbind
From: Michal Hocko @ 2019-06-18 13:02 UTC (permalink / raw)
To: Yang Shi
Cc: akpm, vbabka, mgorman, linux-mm, linux-kernel, Eric Dumazet,
David S. Miller, netdev
In-Reply-To: <1560797290-42267-1-git-send-email-yang.shi@linux.alibaba.com>
[Cc networking people - see a question about setsockopt below]
On Tue 18-06-19 02:48:10, Yang Shi wrote:
> When running syzkaller internally, we ran into the below bug on 4.9.x
> kernel:
>
> kernel BUG at mm/huge_memory.c:2124!
What is the BUG_ON because I do not see any BUG_ON neither in v4.9 nor
the latest stable/linux-4.9.y
> invalid opcode: 0000 [#1] SMP KASAN
[...]
> Code: c7 80 1c 02 00 e8 26 0a 76 01 <0f> 0b 48 c7 c7 40 46 45 84 e8 4c
> RIP [<ffffffff81895d6b>] split_huge_page_to_list+0x8fb/0x1030 mm/huge_memory.c:2124
> RSP <ffff88006899f980>
>
> with the below test:
>
> ---8<---
>
> uint64_t r[1] = {0xffffffffffffffff};
>
> int main(void)
> {
> syscall(__NR_mmap, 0x20000000, 0x1000000, 3, 0x32, -1, 0);
> intptr_t res = 0;
> res = syscall(__NR_socket, 0x11, 3, 0x300);
> if (res != -1)
> r[0] = res;
> *(uint32_t*)0x20000040 = 0x10000;
> *(uint32_t*)0x20000044 = 1;
> *(uint32_t*)0x20000048 = 0xc520;
> *(uint32_t*)0x2000004c = 1;
> syscall(__NR_setsockopt, r[0], 0x107, 0xd, 0x20000040, 0x10);
> syscall(__NR_mmap, 0x20fed000, 0x10000, 0, 0x8811, r[0], 0);
> *(uint64_t*)0x20000340 = 2;
> syscall(__NR_mbind, 0x20ff9000, 0x4000, 0x4002, 0x20000340,
> 0x45d4, 3);
> return 0;
> }
>
> ---8<---
>
> Actually the test does:
>
> mmap(0x20000000, 16777216, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x20000000
> socket(AF_PACKET, SOCK_RAW, 768) = 3
> setsockopt(3, SOL_PACKET, PACKET_TX_RING, {block_size=65536, block_nr=1, frame_size=50464, frame_nr=1}, 16) = 0
> mmap(0x20fed000, 65536, PROT_NONE, MAP_SHARED|MAP_FIXED|MAP_POPULATE|MAP_DENYWRITE, 3, 0) = 0x20fed000
> mbind(..., MPOL_MF_STRICT|MPOL_MF_MOVE) = 0
Ughh. Do I get it right that that this setsockopt allows an arbitrary
contiguous memory allocation size to be requested by a unpriviledged
user? Or am I missing something that restricts there any restriction?
> The setsockopt() would allocate compound pages (16 pages in this test)
> for packet tx ring, then the mmap() would call packet_mmap() to map the
> pages into the user address space specifed by the mmap() call.
>
> When calling mbind(), it would scan the vma to queue the pages for
> migration to the new node. It would split any huge page since 4.9
> doesn't support THP migration, however, the packet tx ring compound
> pages are not THP and even not movable. So, the above bug is triggered.
>
> However, the later kernel is not hit by this issue due to the commit
> d44d363f65780f2ac2ec672164555af54896d40d ("mm: don't assume anonymous
> pages have SwapBacked flag"), which just removes the PageSwapBacked
> check for a different reason.
>
> But, there is a deeper issue. According to the semantic of mbind(), it
> should return -EIO if MPOL_MF_MOVE or MPOL_MF_MOVE_ALL was specified and
> the kernel was unable to move all existing pages in the range. The tx ring
> of the packet socket is definitely not movable, however, mbind returns
> success for this case.
>
> Although the most socket file associates with non-movable pages, but XDP
> may have movable pages from gup. So, it sounds not fine to just check
> the underlying file type of vma in vma_migratable().
>
> Change migrate_page_add() to check if the page is movable or not, if it
> is unmovable, just return -EIO. We don't have to check non-LRU movable
> pages since just zsmalloc and virtio-baloon support this. And, they
> should be not able to reach here.
You are not checking whether the page is movable, right? You only rely
on PageLRU check which is not really an equivalent thing. There are
movable pages which are not LRU and also pages might be off LRU
temporarily for many reasons so this could lead to false positives.
So I do not think this fix is correct. Blowing up on a BUG_ON is
definitely not a right thing to do but we should rely on migrate_pages
to fail the migration and report the failure based on that.
> With this change the above test would return -EIO as expected.
>
> Signed-off-by: Yang Shi <yang.shi@linux.alibaba.com>
> ---
> include/linux/mempolicy.h | 3 ++-
> mm/mempolicy.c | 22 +++++++++++++++++-----
> 2 files changed, 19 insertions(+), 6 deletions(-)
>
> diff --git a/include/linux/mempolicy.h b/include/linux/mempolicy.h
> index 5228c62..cce7ba3 100644
> --- a/include/linux/mempolicy.h
> +++ b/include/linux/mempolicy.h
> @@ -198,7 +198,8 @@ static inline bool vma_migratable(struct vm_area_struct *vma)
> if (vma->vm_file &&
> gfp_zone(mapping_gfp_mask(vma->vm_file->f_mapping))
> < policy_zone)
> - return false;
> + return false;
> +
Any reason to make this change?
> return true;
> }
>
> diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> index 2219e74..4d9e17d 100644
> --- a/mm/mempolicy.c
> +++ b/mm/mempolicy.c
> @@ -403,7 +403,7 @@ void mpol_rebind_mm(struct mm_struct *mm, nodemask_t *new)
> },
> };
>
> -static void migrate_page_add(struct page *page, struct list_head *pagelist,
> +static int migrate_page_add(struct page *page, struct list_head *pagelist,
> unsigned long flags);
>
> struct queue_pages {
> @@ -467,7 +467,9 @@ static int queue_pages_pmd(pmd_t *pmd, spinlock_t *ptl, unsigned long addr,
> goto unlock;
> }
>
> - migrate_page_add(page, qp->pagelist, flags);
> + ret = migrate_page_add(page, qp->pagelist, flags);
> + if (ret)
> + goto unlock;
> } else
> ret = -EIO;
> unlock:
> @@ -521,7 +523,9 @@ static int queue_pages_pte_range(pmd_t *pmd, unsigned long addr,
> if (flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) {
> if (!vma_migratable(vma))
> break;
> - migrate_page_add(page, qp->pagelist, flags);
> + ret = migrate_page_add(page, qp->pagelist, flags);
> + if (ret)
> + break;
> } else
> break;
> }
> @@ -940,10 +944,15 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask,
> /*
> * page migration, thp tail pages can be passed.
> */
> -static void migrate_page_add(struct page *page, struct list_head *pagelist,
> +static int migrate_page_add(struct page *page, struct list_head *pagelist,
> unsigned long flags)
> {
> struct page *head = compound_head(page);
> +
> + /* Non-movable page may reach here. */
> + if (!PageLRU(head))
> + return -EIO;
> +
> /*
> * Avoid migrating a page that is shared with others.
> */
> @@ -955,6 +964,8 @@ static void migrate_page_add(struct page *page, struct list_head *pagelist,
> hpage_nr_pages(head));
> }
> }
> +
> + return 0;
> }
>
> /* page allocation callback for NUMA node migration */
> @@ -1157,9 +1168,10 @@ static struct page *new_page(struct page *page, unsigned long start)
> }
> #else
>
> -static void migrate_page_add(struct page *page, struct list_head *pagelist,
> +static int migrate_page_add(struct page *page, struct list_head *pagelist,
> unsigned long flags)
> {
> + return -EIO;
> }
>
> int do_migrate_pages(struct mm_struct *mm, const nodemask_t *from,
> --
> 1.8.3.1
>
--
Michal Hocko
SUSE Labs
^ permalink raw reply
* [PATCH net-next v2 01/12] net: page_pool: add helper function to retrieve dma addresses
From: Jesper Dangaard Brouer @ 2019-06-18 13:05 UTC (permalink / raw)
To: netdev, Ilias Apalodimas, Toke Høiland-Jørgensen,
Tariq Toukan, Jesper Dangaard Brouer
Cc: toshiaki.makita1, grygorii.strashko, ivan.khoronzhuk, mcroce
In-Reply-To: <156086304827.27760.11339786046465638081.stgit@firesoul>
From: Ilias Apalodimas <ilias.apalodimas@linaro.org>
On a previous patch dma addr was stored in 'struct page'.
Use that to retrieve DMA addresses used by network drivers
Signed-off-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
include/net/page_pool.h | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/include/net/page_pool.h b/include/net/page_pool.h
index 694d055e01ef..b885d86cb7a1 100644
--- a/include/net/page_pool.h
+++ b/include/net/page_pool.h
@@ -132,6 +132,11 @@ static inline void page_pool_recycle_direct(struct page_pool *pool,
__page_pool_put_page(pool, page, true);
}
+static inline dma_addr_t page_pool_get_dma_addr(struct page *page)
+{
+ return page->dma_addr;
+}
+
static inline bool is_page_pool_compiled_in(void)
{
#ifdef CONFIG_PAGE_POOL
^ permalink raw reply related
* [PATCH net-next v2 00/12] xdp: page_pool fixes and in-flight accounting
From: Jesper Dangaard Brouer @ 2019-06-18 13:05 UTC (permalink / raw)
To: netdev, Ilias Apalodimas, Toke Høiland-Jørgensen,
Tariq Toukan, Jesper Dangaard Brouer
Cc: toshiaki.makita1, grygorii.strashko, ivan.khoronzhuk, mcroce
This patchset fix page_pool API and users, such that drivers can use it for
DMA-mapping. A number of places exist, where the DMA-mapping would not get
released/unmapped, all these are fixed. This occurs e.g. when an xdp_frame
gets converted to an SKB. As network stack doesn't have any callback for XDP
memory models.
The patchset also address a shutdown race-condition. Today removing a XDP
memory model, based on page_pool, is only delayed one RCU grace period. This
isn't enough as redirected xdp_frames can still be in-flight on different
queues (remote driver TX, cpumap or veth).
We stress that when drivers use page_pool for DMA-mapping, then they MUST
use one packet per page. This might change in the future, but more work lies
ahead, before we can lift this restriction.
This patchset change the page_pool API to be more strict, as in-flight page
accounting is added.
---
Ilias Apalodimas (2):
net: page_pool: add helper function to retrieve dma addresses
net: page_pool: add helper function to unmap dma addresses
Jesper Dangaard Brouer (10):
xdp: fix leak of IDA cyclic id if rhashtable_insert_slow fails
xdp: page_pool related fix to cpumap
veth: use xdp_release_frame for XDP_PASS
page_pool: introduce page_pool_free and use in mlx5
mlx5: more strict use of page_pool API
xdp: tracking page_pool resources and safe removal
xdp: force mem allocator removal and periodic warning
xdp: add tracepoints for XDP mem
page_pool: add tracepoints for page_pool with details need by XDP
page_pool: make sure struct device is stable
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 12 +-
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 3 -
drivers/net/veth.c | 1
include/net/page_pool.h | 69 +++++++++++-
include/net/xdp.h | 15 +++
include/net/xdp_priv.h | 23 ++++
include/trace/events/page_pool.h | 87 +++++++++++++++
include/trace/events/xdp.h | 115 ++++++++++++++++++++
kernel/bpf/cpumap.c | 3 +
net/core/net-traces.c | 4 +
net/core/page_pool.c | 95 +++++++++++++++--
net/core/xdp.c | 120 ++++++++++++++++++---
12 files changed, 502 insertions(+), 45 deletions(-)
create mode 100644 include/net/xdp_priv.h
create mode 100644 include/trace/events/page_pool.h
--
^ permalink raw reply
* [PATCH net-next v2 02/12] net: page_pool: add helper function to unmap dma addresses
From: Jesper Dangaard Brouer @ 2019-06-18 13:05 UTC (permalink / raw)
To: netdev, Ilias Apalodimas, Toke Høiland-Jørgensen,
Tariq Toukan, Jesper Dangaard Brouer
Cc: toshiaki.makita1, grygorii.strashko, ivan.khoronzhuk, mcroce
In-Reply-To: <156086304827.27760.11339786046465638081.stgit@firesoul>
From: Ilias Apalodimas <ilias.apalodimas@linaro.org>
On a previous patch dma addr was stored in 'struct page'.
Use that to unmap DMA addresses used by network drivers
Signed-off-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
include/net/page_pool.h | 1 +
net/core/page_pool.c | 7 +++++++
2 files changed, 8 insertions(+)
diff --git a/include/net/page_pool.h b/include/net/page_pool.h
index b885d86cb7a1..ad218cef88c5 100644
--- a/include/net/page_pool.h
+++ b/include/net/page_pool.h
@@ -110,6 +110,7 @@ static inline struct page *page_pool_dev_alloc_pages(struct page_pool *pool)
struct page_pool *page_pool_create(const struct page_pool_params *params);
void page_pool_destroy(struct page_pool *pool);
+void page_pool_unmap_page(struct page_pool *pool, struct page *page);
/* Never call this directly, use helpers below */
void __page_pool_put_page(struct page_pool *pool,
diff --git a/net/core/page_pool.c b/net/core/page_pool.c
index 5b2252c6d49b..205af7bd6d09 100644
--- a/net/core/page_pool.c
+++ b/net/core/page_pool.c
@@ -190,6 +190,13 @@ static void __page_pool_clean_page(struct page_pool *pool,
page->dma_addr = 0;
}
+/* unmap the page and clean our state */
+void page_pool_unmap_page(struct page_pool *pool, struct page *page)
+{
+ __page_pool_clean_page(pool, page);
+}
+EXPORT_SYMBOL(page_pool_unmap_page);
+
/* Return a page to the page allocator, cleaning up our state */
static void __page_pool_return_page(struct page_pool *pool, struct page *page)
{
^ permalink raw reply related
* [PATCH net-next v2 03/12] xdp: fix leak of IDA cyclic id if rhashtable_insert_slow fails
From: Jesper Dangaard Brouer @ 2019-06-18 13:05 UTC (permalink / raw)
To: netdev, Ilias Apalodimas, Toke Høiland-Jørgensen,
Tariq Toukan, Jesper Dangaard Brouer
Cc: toshiaki.makita1, grygorii.strashko, ivan.khoronzhuk, mcroce
In-Reply-To: <156086304827.27760.11339786046465638081.stgit@firesoul>
Fix error handling case, where inserting ID with rhashtable_insert_slow
fails in xdp_rxq_info_reg_mem_model, which leads to never releasing the IDA
ID, as the lookup in xdp_rxq_info_unreg_mem_model fails and thus
ida_simple_remove() is never called.
Fix by releasing ID via ida_simple_remove(), and mark xdp_rxq->mem.id with
zero, which is already checked in xdp_rxq_info_unreg_mem_model().
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Reviewed-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
---
net/core/xdp.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/net/core/xdp.c b/net/core/xdp.c
index 4b2b194f4f1f..762abeb89847 100644
--- a/net/core/xdp.c
+++ b/net/core/xdp.c
@@ -301,6 +301,8 @@ int xdp_rxq_info_reg_mem_model(struct xdp_rxq_info *xdp_rxq,
/* Insert allocator into ID lookup table */
ptr = rhashtable_insert_slow(mem_id_ht, &id, &xdp_alloc->node);
if (IS_ERR(ptr)) {
+ ida_simple_remove(&mem_id_pool, xdp_rxq->mem.id);
+ xdp_rxq->mem.id = 0;
errno = PTR_ERR(ptr);
goto err;
}
^ permalink raw reply related
* [PATCH net-next v2 04/12] xdp: page_pool related fix to cpumap
From: Jesper Dangaard Brouer @ 2019-06-18 13:05 UTC (permalink / raw)
To: netdev, Ilias Apalodimas, Toke Høiland-Jørgensen,
Tariq Toukan, Jesper Dangaard Brouer
Cc: toshiaki.makita1, grygorii.strashko, ivan.khoronzhuk, mcroce
In-Reply-To: <156086304827.27760.11339786046465638081.stgit@firesoul>
When converting an xdp_frame into an SKB, and sending this into the network
stack, then the underlying XDP memory model need to release associated
resources, because the network stack don't have callbacks for XDP memory
models. The only memory model that needs this is page_pool, when a driver
use the DMA-mapping feature.
Introduce page_pool_release_page(), which basically does the same as
page_pool_unmap_page(). Add xdp_release_frame() as the XDP memory model
interface for calling it, if the memory model match MEM_TYPE_PAGE_POOL, to
save the function call overhead for others. Have cpumap call
xdp_release_frame() before xdp_scrub_frame().
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
include/net/page_pool.h | 15 ++++++++++++++-
include/net/xdp.h | 15 +++++++++++++++
kernel/bpf/cpumap.c | 3 +++
net/core/xdp.c | 15 +++++++++++++++
4 files changed, 47 insertions(+), 1 deletion(-)
diff --git a/include/net/page_pool.h b/include/net/page_pool.h
index ad218cef88c5..e240fac4c5b9 100644
--- a/include/net/page_pool.h
+++ b/include/net/page_pool.h
@@ -110,7 +110,6 @@ static inline struct page *page_pool_dev_alloc_pages(struct page_pool *pool)
struct page_pool *page_pool_create(const struct page_pool_params *params);
void page_pool_destroy(struct page_pool *pool);
-void page_pool_unmap_page(struct page_pool *pool, struct page *page);
/* Never call this directly, use helpers below */
void __page_pool_put_page(struct page_pool *pool,
@@ -133,6 +132,20 @@ static inline void page_pool_recycle_direct(struct page_pool *pool,
__page_pool_put_page(pool, page, true);
}
+/* Disconnects a page (from a page_pool). API users can have a need
+ * to disconnect a page (from a page_pool), to allow it to be used as
+ * a regular page (that will eventually be returned to the normal
+ * page-allocator via put_page).
+ */
+void page_pool_unmap_page(struct page_pool *pool, struct page *page);
+static inline void page_pool_release_page(struct page_pool *pool,
+ struct page *page)
+{
+#ifdef CONFIG_PAGE_POOL
+ page_pool_unmap_page(pool, page);
+#endif
+}
+
static inline dma_addr_t page_pool_get_dma_addr(struct page *page)
{
return page->dma_addr;
diff --git a/include/net/xdp.h b/include/net/xdp.h
index 0f25b3675c5c..a06e5f2dfcc5 100644
--- a/include/net/xdp.h
+++ b/include/net/xdp.h
@@ -129,6 +129,21 @@ void xdp_return_frame(struct xdp_frame *xdpf);
void xdp_return_frame_rx_napi(struct xdp_frame *xdpf);
void xdp_return_buff(struct xdp_buff *xdp);
+/* When sending xdp_frame into the network stack, then there is no
+ * return point callback, which is needed to release e.g. DMA-mapping
+ * resources with page_pool. Thus, have explicit function to release
+ * frame resources.
+ */
+void __xdp_release_frame(void *data, struct xdp_mem_info *mem);
+static inline void xdp_release_frame(struct xdp_frame *xdpf)
+{
+ struct xdp_mem_info *mem = &xdpf->mem;
+
+ /* Curr only page_pool needs this */
+ if (mem->type == MEM_TYPE_PAGE_POOL)
+ __xdp_release_frame(xdpf->data, mem);
+}
+
int xdp_rxq_info_reg(struct xdp_rxq_info *xdp_rxq,
struct net_device *dev, u32 queue_index);
void xdp_rxq_info_unreg(struct xdp_rxq_info *xdp_rxq);
diff --git a/kernel/bpf/cpumap.c b/kernel/bpf/cpumap.c
index cf727d77c6c6..b7a3eab4f7c8 100644
--- a/kernel/bpf/cpumap.c
+++ b/kernel/bpf/cpumap.c
@@ -209,6 +209,9 @@ static struct sk_buff *cpu_map_build_skb(struct bpf_cpu_map_entry *rcpu,
* - RX ring dev queue index (skb_record_rx_queue)
*/
+ /* Until page_pool get SKB return path, release DMA here */
+ xdp_release_frame(xdpf);
+
/* Allow SKB to reuse area used by xdp_frame */
xdp_scrub_frame(xdpf);
diff --git a/net/core/xdp.c b/net/core/xdp.c
index 762abeb89847..179d90570afe 100644
--- a/net/core/xdp.c
+++ b/net/core/xdp.c
@@ -381,6 +381,21 @@ void xdp_return_buff(struct xdp_buff *xdp)
}
EXPORT_SYMBOL_GPL(xdp_return_buff);
+/* Only called for MEM_TYPE_PAGE_POOL see xdp.h */
+void __xdp_release_frame(void *data, struct xdp_mem_info *mem)
+{
+ struct xdp_mem_allocator *xa;
+ struct page *page;
+
+ rcu_read_lock();
+ xa = rhashtable_lookup(mem_id_ht, &mem->id, mem_id_rht_params);
+ page = virt_to_head_page(data);
+ if (xa)
+ page_pool_release_page(xa->page_pool, page);
+ rcu_read_unlock();
+}
+EXPORT_SYMBOL_GPL(__xdp_release_frame);
+
int xdp_attachment_query(struct xdp_attachment_info *info,
struct netdev_bpf *bpf)
{
^ permalink raw reply related
* [PATCH net-next v2 05/12] veth: use xdp_release_frame for XDP_PASS
From: Jesper Dangaard Brouer @ 2019-06-18 13:05 UTC (permalink / raw)
To: netdev, Ilias Apalodimas, Toke Høiland-Jørgensen,
Tariq Toukan, Jesper Dangaard Brouer
Cc: toshiaki.makita1, grygorii.strashko, ivan.khoronzhuk, mcroce
In-Reply-To: <156086304827.27760.11339786046465638081.stgit@firesoul>
Like cpumap use xdp_release_frame() when an xdp_frame got
converted into an SKB and send towars the network stack.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
drivers/net/veth.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/net/veth.c b/drivers/net/veth.c
index 52110e54e621..c6916bf1017b 100644
--- a/drivers/net/veth.c
+++ b/drivers/net/veth.c
@@ -547,6 +547,7 @@ static struct sk_buff *veth_xdp_rcv_one(struct veth_rq *rq,
goto err;
}
+ xdp_release_frame(frame);
xdp_scrub_frame(frame);
skb->protocol = eth_type_trans(skb, rq->dev);
err:
^ permalink raw reply related
* [PATCH net-next v2 06/12] page_pool: introduce page_pool_free and use in mlx5
From: Jesper Dangaard Brouer @ 2019-06-18 13:05 UTC (permalink / raw)
To: netdev, Ilias Apalodimas, Toke Høiland-Jørgensen,
Tariq Toukan, Jesper Dangaard Brouer
Cc: toshiaki.makita1, grygorii.strashko, ivan.khoronzhuk, mcroce
In-Reply-To: <156086304827.27760.11339786046465638081.stgit@firesoul>
In case driver fails to register the page_pool with XDP return API (via
xdp_rxq_info_reg_mem_model()), then the driver can free the page_pool
resources more directly than calling page_pool_destroy(), which does a
unnecessarily RCU free procedure.
This patch is preparing for removing page_pool_destroy(), from driver
invocation.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 6 +++---
include/net/page_pool.h | 11 +++++++++++
net/core/page_pool.c | 15 +++++++++++----
3 files changed, 25 insertions(+), 7 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 457cc39423f2..07de9ca4c53c 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -545,8 +545,10 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c,
}
err = xdp_rxq_info_reg_mem_model(&rq->xdp_rxq,
MEM_TYPE_PAGE_POOL, rq->page_pool);
- if (err)
+ if (err) {
+ page_pool_free(rq->page_pool);
goto err_free;
+ }
for (i = 0; i < wq_sz; i++) {
if (rq->wq_type == MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ) {
@@ -611,8 +613,6 @@ static int mlx5e_alloc_rq(struct mlx5e_channel *c,
if (rq->xdp_prog)
bpf_prog_put(rq->xdp_prog);
xdp_rxq_info_unreg(&rq->xdp_rxq);
- if (rq->page_pool)
- page_pool_destroy(rq->page_pool);
mlx5_wq_destroy(&rq->wq_ctrl);
return err;
diff --git a/include/net/page_pool.h b/include/net/page_pool.h
index e240fac4c5b9..754d980700df 100644
--- a/include/net/page_pool.h
+++ b/include/net/page_pool.h
@@ -111,6 +111,17 @@ struct page_pool *page_pool_create(const struct page_pool_params *params);
void page_pool_destroy(struct page_pool *pool);
+void __page_pool_free(struct page_pool *pool);
+static inline void page_pool_free(struct page_pool *pool)
+{
+ /* When page_pool isn't compiled-in, net/core/xdp.c doesn't
+ * allow registering MEM_TYPE_PAGE_POOL, but shield linker.
+ */
+#ifdef CONFIG_PAGE_POOL
+ __page_pool_free(pool);
+#endif
+}
+
/* Never call this directly, use helpers below */
void __page_pool_put_page(struct page_pool *pool,
struct page *page, bool allow_direct);
diff --git a/net/core/page_pool.c b/net/core/page_pool.c
index 205af7bd6d09..41391b5dc14c 100644
--- a/net/core/page_pool.c
+++ b/net/core/page_pool.c
@@ -292,17 +292,24 @@ static void __page_pool_empty_ring(struct page_pool *pool)
}
}
+void __page_pool_free(struct page_pool *pool)
+{
+ WARN(pool->alloc.count, "API usage violation");
+ WARN(!ptr_ring_empty(&pool->ring), "ptr_ring is not empty");
+
+ ptr_ring_cleanup(&pool->ring, NULL);
+ kfree(pool);
+}
+EXPORT_SYMBOL(__page_pool_free);
+
static void __page_pool_destroy_rcu(struct rcu_head *rcu)
{
struct page_pool *pool;
pool = container_of(rcu, struct page_pool, rcu);
- WARN(pool->alloc.count, "API usage violation");
-
__page_pool_empty_ring(pool);
- ptr_ring_cleanup(&pool->ring, NULL);
- kfree(pool);
+ __page_pool_free(pool);
}
/* Cleanup and release resources */
^ permalink raw reply related
* [PATCH net-next v2 07/12] mlx5: more strict use of page_pool API
From: Jesper Dangaard Brouer @ 2019-06-18 13:05 UTC (permalink / raw)
To: netdev, Ilias Apalodimas, Toke Høiland-Jørgensen,
Tariq Toukan, Jesper Dangaard Brouer
Cc: toshiaki.makita1, grygorii.strashko, ivan.khoronzhuk, mcroce
In-Reply-To: <156086304827.27760.11339786046465638081.stgit@firesoul>
The mlx5 driver is using page_pool, but not for DMA-mapping (currently), and
is a little too relaxed about returning or releasing page resources, as it
is not strictly necessary, when not using DMA-mappings.
As this patchset is working towards tracking page_pool resources, to know
about in-flight frames on shutdown. Then fix places where mlx5 leak
page_pool resource.
In case of dma_mapping_error, then recycle into page_pool.
In mlx5e_free_rq() moved the page_pool_destroy() call to after the
mlx5e_page_release() calls, as it is more correct.
In mlx5e_page_release() when no recycle was requested, then release page
from the page_pool, via page_pool_release_page().
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Reviewed-by: Tariq Toukan <tariqt@mellanox.com>
---
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 9 +++++----
drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 3 ++-
2 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 07de9ca4c53c..2f647be292b6 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -625,10 +625,6 @@ static void mlx5e_free_rq(struct mlx5e_rq *rq)
if (rq->xdp_prog)
bpf_prog_put(rq->xdp_prog);
- xdp_rxq_info_unreg(&rq->xdp_rxq);
- if (rq->page_pool)
- page_pool_destroy(rq->page_pool);
-
switch (rq->wq_type) {
case MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ:
kvfree(rq->mpwqe.info);
@@ -645,6 +641,11 @@ static void mlx5e_free_rq(struct mlx5e_rq *rq)
mlx5e_page_release(rq, dma_info, false);
}
+
+ xdp_rxq_info_unreg(&rq->xdp_rxq);
+ if (rq->page_pool)
+ page_pool_destroy(rq->page_pool);
+
mlx5_wq_destroy(&rq->wq_ctrl);
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index 13133e7f088e..8331ff2ffdc6 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -247,7 +247,7 @@ static inline int mlx5e_page_alloc_mapped(struct mlx5e_rq *rq,
dma_info->addr = dma_map_page(rq->pdev, dma_info->page, 0,
PAGE_SIZE, rq->buff.map_dir);
if (unlikely(dma_mapping_error(rq->pdev, dma_info->addr))) {
- put_page(dma_info->page);
+ page_pool_recycle_direct(rq->page_pool, dma_info->page);
dma_info->page = NULL;
return -ENOMEM;
}
@@ -271,6 +271,7 @@ void mlx5e_page_release(struct mlx5e_rq *rq, struct mlx5e_dma_info *dma_info,
page_pool_recycle_direct(rq->page_pool, dma_info->page);
} else {
mlx5e_page_dma_unmap(rq, dma_info);
+ page_pool_release_page(rq->page_pool, dma_info->page);
put_page(dma_info->page);
}
}
^ permalink raw reply related
* [PATCH net-next v2 09/12] xdp: force mem allocator removal and periodic warning
From: Jesper Dangaard Brouer @ 2019-06-18 13:05 UTC (permalink / raw)
To: netdev, Ilias Apalodimas, Toke Høiland-Jørgensen,
Tariq Toukan, Jesper Dangaard Brouer
Cc: toshiaki.makita1, grygorii.strashko, ivan.khoronzhuk, mcroce
In-Reply-To: <156086304827.27760.11339786046465638081.stgit@firesoul>
If bugs exists or are introduced later e.g. by drivers misusing the API,
then we want to warn about the issue, such that developer notice. This patch
will generate a bit of noise in form of periodic pr_warn every 30 seconds.
It is not nice to have this stall warning running forever. Thus, this patch
will (after 120 attempts) force disconnect the mem id (from the rhashtable)
and free the page_pool object. This will cause fallback to the put_page() as
before, which only potentially leak DMA-mappings, if objects are really
stuck for this long. In that unlikely case, a WARN_ONCE should show us the
call stack.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
net/core/page_pool.c | 18 +++++++++++++++++-
net/core/xdp.c | 37 +++++++++++++++++++++++++++++++------
2 files changed, 48 insertions(+), 7 deletions(-)
diff --git a/net/core/page_pool.c b/net/core/page_pool.c
index 8679e24fd665..42c3b0a5a259 100644
--- a/net/core/page_pool.c
+++ b/net/core/page_pool.c
@@ -330,11 +330,27 @@ static void __page_pool_empty_ring(struct page_pool *pool)
}
}
+static void __warn_in_flight(struct page_pool *pool)
+{
+ u32 release_cnt = atomic_read(&pool->pages_state_release_cnt);
+ u32 hold_cnt = READ_ONCE(pool->pages_state_hold_cnt);
+ s32 distance;
+
+ distance = _distance(hold_cnt, release_cnt);
+
+ /* Drivers should fix this, but only problematic when DMA is used */
+ WARN(1, "Still in-flight pages:%d hold:%u released:%u",
+ distance, hold_cnt, release_cnt);
+}
+
void __page_pool_free(struct page_pool *pool)
{
WARN(pool->alloc.count, "API usage violation");
WARN(!ptr_ring_empty(&pool->ring), "ptr_ring is not empty");
- WARN(!__page_pool_safe_to_destroy(pool), "still in-flight pages");
+
+ /* Can happen due to forced shutdown */
+ if (!__page_pool_safe_to_destroy(pool))
+ __warn_in_flight(pool);
ptr_ring_cleanup(&pool->ring, NULL);
kfree(pool);
diff --git a/net/core/xdp.c b/net/core/xdp.c
index 2b7bad227030..53bce4fa776a 100644
--- a/net/core/xdp.c
+++ b/net/core/xdp.c
@@ -39,6 +39,9 @@ struct xdp_mem_allocator {
struct rhash_head node;
struct rcu_head rcu;
struct delayed_work defer_wq;
+ unsigned long defer_start;
+ unsigned long defer_warn;
+ int disconnect_cnt;
};
static u32 xdp_mem_id_hashfn(const void *data, u32 len, u32 seed)
@@ -95,7 +98,7 @@ static void __xdp_mem_allocator_rcu_free(struct rcu_head *rcu)
kfree(xa);
}
-bool __mem_id_disconnect(int id)
+bool __mem_id_disconnect(int id, bool force)
{
struct xdp_mem_allocator *xa;
bool safe_to_remove = true;
@@ -108,29 +111,47 @@ bool __mem_id_disconnect(int id)
WARN(1, "Request remove non-existing id(%d), driver bug?", id);
return true;
}
+ xa->disconnect_cnt++;
/* Detects in-flight packet-pages for page_pool */
if (xa->mem.type == MEM_TYPE_PAGE_POOL)
safe_to_remove = page_pool_request_shutdown(xa->page_pool);
- if (safe_to_remove &&
+ /* TODO: Tracepoint will be added here in next-patch */
+
+ if ((safe_to_remove || force) &&
!rhashtable_remove_fast(mem_id_ht, &xa->node, mem_id_rht_params))
call_rcu(&xa->rcu, __xdp_mem_allocator_rcu_free);
mutex_unlock(&mem_id_lock);
- return safe_to_remove;
+ return (safe_to_remove|force);
}
#define DEFER_TIME (msecs_to_jiffies(1000))
+#define DEFER_WARN_INTERVAL (30 * HZ)
+#define DEFER_MAX_RETRIES 120
static void mem_id_disconnect_defer_retry(struct work_struct *wq)
{
struct delayed_work *dwq = to_delayed_work(wq);
struct xdp_mem_allocator *xa = container_of(dwq, typeof(*xa), defer_wq);
+ bool force = false;
+
+ if (xa->disconnect_cnt > DEFER_MAX_RETRIES)
+ force = true;
- if (__mem_id_disconnect(xa->mem.id))
+ if (__mem_id_disconnect(xa->mem.id, force))
return;
+ /* Periodic warning */
+ if (time_after_eq(jiffies, xa->defer_warn)) {
+ int sec = (s32)((u32)jiffies - (u32)xa->defer_start) / HZ;
+
+ pr_warn("%s() stalled mem.id=%u shutdown %d attempts %d sec\n",
+ __func__, xa->mem.id, xa->disconnect_cnt, sec);
+ xa->defer_warn = jiffies + DEFER_WARN_INTERVAL;
+ }
+
/* Still not ready to be disconnected, retry later */
schedule_delayed_work(&xa->defer_wq, DEFER_TIME);
}
@@ -153,7 +174,7 @@ void xdp_rxq_info_unreg_mem_model(struct xdp_rxq_info *xdp_rxq)
if (id == 0)
return;
- if (__mem_id_disconnect(id))
+ if (__mem_id_disconnect(id, false))
return;
/* Could not disconnect, defer new disconnect attempt to later */
@@ -164,6 +185,8 @@ void xdp_rxq_info_unreg_mem_model(struct xdp_rxq_info *xdp_rxq)
mutex_unlock(&mem_id_lock);
return;
}
+ xa->defer_start = jiffies;
+ xa->defer_warn = jiffies + DEFER_WARN_INTERVAL;
INIT_DELAYED_WORK(&xa->defer_wq, mem_id_disconnect_defer_retry);
mutex_unlock(&mem_id_lock);
@@ -388,10 +411,12 @@ static void __xdp_return(void *data, struct xdp_mem_info *mem, bool napi_direct,
/* mem->id is valid, checked in xdp_rxq_info_reg_mem_model() */
xa = rhashtable_lookup(mem_id_ht, &mem->id, mem_id_rht_params);
page = virt_to_head_page(data);
- if (xa) {
+ if (likely(xa)) {
napi_direct &= !xdp_return_frame_no_direct();
page_pool_put_page(xa->page_pool, page, napi_direct);
} else {
+ /* Hopefully stack show who to blame for late return */
+ WARN_ONCE(1, "page_pool gone mem.id=%d", mem->id);
put_page(page);
}
rcu_read_unlock();
^ permalink raw reply related
* [PATCH net-next v2 08/12] xdp: tracking page_pool resources and safe removal
From: Jesper Dangaard Brouer @ 2019-06-18 13:05 UTC (permalink / raw)
To: netdev, Ilias Apalodimas, Toke Høiland-Jørgensen,
Tariq Toukan, Jesper Dangaard Brouer
Cc: toshiaki.makita1, grygorii.strashko, ivan.khoronzhuk, mcroce
In-Reply-To: <156086304827.27760.11339786046465638081.stgit@firesoul>
This patch is needed before we can allow drivers to use page_pool for
DMA-mappings. Today with page_pool and XDP return API, it is possible to
remove the page_pool object (from rhashtable), while there are still
in-flight packet-pages. This is safely handled via RCU and failed lookups in
__xdp_return() fallback to call put_page(), when page_pool object is gone.
In-case page is still DMA mapped, this will result in page note getting
correctly DMA unmapped.
To solve this, the page_pool is extended with tracking in-flight pages. And
XDP disconnect system queries page_pool and waits, via workqueue, for all
in-flight pages to be returned.
To avoid killing performance when tracking in-flight pages, the implement
use two (unsigned) counters, that in placed on different cache-lines, and
can be used to deduct in-flight packets. This is done by mapping the
unsigned "sequence" counters onto signed Two's complement arithmetic
operations. This is e.g. used by kernel's time_after macros, described in
kernel commit 1ba3aab3033b and 5a581b367b5, and also explained in RFC1982.
The trick is these two incrementing counters only need to be read and
compared, when checking if it's safe to free the page_pool structure. Which
will only happen when driver have disconnected RX/alloc side. Thus, on a
non-fast-path.
It is chosen that page_pool tracking is also enabled for the non-DMA
use-case, as this can be used for statistics later.
After this patch, using page_pool requires more strict resource "release",
e.g. via page_pool_release_page() that was introduced in this patchset, and
previous patches implement/fix this more strict requirement.
Drivers no-longer call page_pool_destroy(). Drivers already call
xdp_rxq_info_unreg() which call xdp_rxq_info_unreg_mem_model(), which will
attempt to disconnect the mem id, and if attempt fails schedule the
disconnect for later via delayed workqueue.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Reviewed-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
---
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 3 -
include/net/page_pool.h | 41 ++++++++++---
net/core/page_pool.c | 62 +++++++++++++++-----
net/core/xdp.c | 65 +++++++++++++++++++--
4 files changed, 136 insertions(+), 35 deletions(-)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 2f647be292b6..6c9d4d7defbc 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -643,9 +643,6 @@ static void mlx5e_free_rq(struct mlx5e_rq *rq)
}
xdp_rxq_info_unreg(&rq->xdp_rxq);
- if (rq->page_pool)
- page_pool_destroy(rq->page_pool);
-
mlx5_wq_destroy(&rq->wq_ctrl);
}
diff --git a/include/net/page_pool.h b/include/net/page_pool.h
index 754d980700df..f09b3f1994e6 100644
--- a/include/net/page_pool.h
+++ b/include/net/page_pool.h
@@ -16,14 +16,16 @@
* page_pool_alloc_pages() call. Drivers should likely use
* page_pool_dev_alloc_pages() replacing dev_alloc_pages().
*
- * If page_pool handles DMA mapping (use page->private), then API user
- * is responsible for invoking page_pool_put_page() once. In-case of
- * elevated refcnt, the DMA state is released, assuming other users of
- * the page will eventually call put_page().
+ * API keeps track of in-flight pages, in-order to let API user know
+ * when it is safe to dealloactor page_pool object. Thus, API users
+ * must make sure to call page_pool_release_page() when a page is
+ * "leaving" the page_pool. Or call page_pool_put_page() where
+ * appropiate. For maintaining correct accounting.
*
- * If no DMA mapping is done, then it can act as shim-layer that
- * fall-through to alloc_page. As no state is kept on the page, the
- * regular put_page() call is sufficient.
+ * API user must only call page_pool_put_page() once on a page, as it
+ * will either recycle the page, or in case of elevated refcnt, it
+ * will release the DMA mapping and in-flight state accounting. We
+ * hope to lift this requirement in the future.
*/
#ifndef _NET_PAGE_POOL_H
#define _NET_PAGE_POOL_H
@@ -66,9 +68,10 @@ struct page_pool_params {
};
struct page_pool {
- struct rcu_head rcu;
struct page_pool_params p;
+ u32 pages_state_hold_cnt;
+
/*
* Data structure for allocation side
*
@@ -96,6 +99,8 @@ struct page_pool {
* TODO: Implement bulk return pages into this structure.
*/
struct ptr_ring ring;
+
+ atomic_t pages_state_release_cnt;
};
struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp);
@@ -109,8 +114,6 @@ static inline struct page *page_pool_dev_alloc_pages(struct page_pool *pool)
struct page_pool *page_pool_create(const struct page_pool_params *params);
-void page_pool_destroy(struct page_pool *pool);
-
void __page_pool_free(struct page_pool *pool);
static inline void page_pool_free(struct page_pool *pool)
{
@@ -143,6 +146,24 @@ static inline void page_pool_recycle_direct(struct page_pool *pool,
__page_pool_put_page(pool, page, true);
}
+/* API user MUST have disconnected alloc-side (not allowed to call
+ * page_pool_alloc_pages()) before calling this. The free-side can
+ * still run concurrently, to handle in-flight packet-pages.
+ *
+ * A request to shutdown can fail (with false) if there are still
+ * in-flight packet-pages.
+ */
+bool __page_pool_request_shutdown(struct page_pool *pool);
+static inline bool page_pool_request_shutdown(struct page_pool *pool)
+{
+ /* When page_pool isn't compiled-in, net/core/xdp.c doesn't
+ * allow registering MEM_TYPE_PAGE_POOL, but shield linker.
+ */
+#ifdef CONFIG_PAGE_POOL
+ return __page_pool_request_shutdown(pool);
+#endif
+}
+
/* Disconnects a page (from a page_pool). API users can have a need
* to disconnect a page (from a page_pool), to allow it to be used as
* a regular page (that will eventually be returned to the normal
diff --git a/net/core/page_pool.c b/net/core/page_pool.c
index 41391b5dc14c..8679e24fd665 100644
--- a/net/core/page_pool.c
+++ b/net/core/page_pool.c
@@ -43,6 +43,8 @@ static int page_pool_init(struct page_pool *pool,
if (ptr_ring_init(&pool->ring, ring_qsize, GFP_KERNEL) < 0)
return -ENOMEM;
+ atomic_set(&pool->pages_state_release_cnt, 0);
+
return 0;
}
@@ -151,6 +153,9 @@ static struct page *__page_pool_alloc_pages_slow(struct page_pool *pool,
page->dma_addr = dma;
skip_dma_map:
+ /* Track how many pages are held 'in-flight' */
+ pool->pages_state_hold_cnt++;
+
/* When page just alloc'ed is should/must have refcnt 1. */
return page;
}
@@ -173,6 +178,33 @@ struct page *page_pool_alloc_pages(struct page_pool *pool, gfp_t gfp)
}
EXPORT_SYMBOL(page_pool_alloc_pages);
+/* Calculate distance between two u32 values, valid if distance is below 2^(31)
+ * https://en.wikipedia.org/wiki/Serial_number_arithmetic#General_Solution
+ */
+#define _distance(a, b) (s32)((a) - (b))
+
+static s32 page_pool_inflight(struct page_pool *pool)
+{
+ u32 release_cnt = atomic_read(&pool->pages_state_release_cnt);
+ u32 hold_cnt = READ_ONCE(pool->pages_state_hold_cnt);
+ s32 distance;
+
+ distance = _distance(hold_cnt, release_cnt);
+
+ /* TODO: Add tracepoint here */
+ return distance;
+}
+
+static bool __page_pool_safe_to_destroy(struct page_pool *pool)
+{
+ s32 inflight = page_pool_inflight(pool);
+
+ /* The distance should not be able to become negative */
+ WARN(inflight < 0, "Negative(%d) inflight packet-pages", inflight);
+
+ return (inflight == 0);
+}
+
/* Cleanup page_pool state from page */
static void __page_pool_clean_page(struct page_pool *pool,
struct page *page)
@@ -180,7 +212,7 @@ static void __page_pool_clean_page(struct page_pool *pool,
dma_addr_t dma;
if (!(pool->p.flags & PP_FLAG_DMA_MAP))
- return;
+ goto skip_dma_unmap;
dma = page->dma_addr;
/* DMA unmap */
@@ -188,11 +220,16 @@ static void __page_pool_clean_page(struct page_pool *pool,
PAGE_SIZE << pool->p.order, pool->p.dma_dir,
DMA_ATTR_SKIP_CPU_SYNC);
page->dma_addr = 0;
+skip_dma_unmap:
+ atomic_inc(&pool->pages_state_release_cnt);
}
/* unmap the page and clean our state */
void page_pool_unmap_page(struct page_pool *pool, struct page *page)
{
+ /* When page is unmapped, this implies page will not be
+ * returned to page_pool.
+ */
__page_pool_clean_page(pool, page);
}
EXPORT_SYMBOL(page_pool_unmap_page);
@@ -201,6 +238,7 @@ EXPORT_SYMBOL(page_pool_unmap_page);
static void __page_pool_return_page(struct page_pool *pool, struct page *page)
{
__page_pool_clean_page(pool, page);
+
put_page(page);
/* An optimization would be to call __free_pages(page, pool->p.order)
* knowing page is not part of page-cache (thus avoiding a
@@ -296,24 +334,17 @@ void __page_pool_free(struct page_pool *pool)
{
WARN(pool->alloc.count, "API usage violation");
WARN(!ptr_ring_empty(&pool->ring), "ptr_ring is not empty");
+ WARN(!__page_pool_safe_to_destroy(pool), "still in-flight pages");
ptr_ring_cleanup(&pool->ring, NULL);
kfree(pool);
}
EXPORT_SYMBOL(__page_pool_free);
-static void __page_pool_destroy_rcu(struct rcu_head *rcu)
-{
- struct page_pool *pool;
-
- pool = container_of(rcu, struct page_pool, rcu);
-
- __page_pool_empty_ring(pool);
- __page_pool_free(pool);
-}
-
-/* Cleanup and release resources */
-void page_pool_destroy(struct page_pool *pool)
+/* Request to shutdown: release pages cached by page_pool, and check
+ * for in-flight pages
+ */
+bool __page_pool_request_shutdown(struct page_pool *pool)
{
struct page *page;
@@ -331,7 +362,6 @@ void page_pool_destroy(struct page_pool *pool)
*/
__page_pool_empty_ring(pool);
- /* An xdp_mem_allocator can still ref page_pool pointer */
- call_rcu(&pool->rcu, __page_pool_destroy_rcu);
+ return __page_pool_safe_to_destroy(pool);
}
-EXPORT_SYMBOL(page_pool_destroy);
+EXPORT_SYMBOL(__page_pool_request_shutdown);
diff --git a/net/core/xdp.c b/net/core/xdp.c
index 179d90570afe..2b7bad227030 100644
--- a/net/core/xdp.c
+++ b/net/core/xdp.c
@@ -38,6 +38,7 @@ struct xdp_mem_allocator {
};
struct rhash_head node;
struct rcu_head rcu;
+ struct delayed_work defer_wq;
};
static u32 xdp_mem_id_hashfn(const void *data, u32 len, u32 seed)
@@ -79,13 +80,13 @@ static void __xdp_mem_allocator_rcu_free(struct rcu_head *rcu)
xa = container_of(rcu, struct xdp_mem_allocator, rcu);
+ /* Allocator have indicated safe to remove before this is called */
+ if (xa->mem.type == MEM_TYPE_PAGE_POOL)
+ page_pool_free(xa->page_pool);
+
/* Allow this ID to be reused */
ida_simple_remove(&mem_id_pool, xa->mem.id);
- /* Notice, driver is expected to free the *allocator,
- * e.g. page_pool, and MUST also use RCU free.
- */
-
/* Poison memory */
xa->mem.id = 0xFFFF;
xa->mem.type = 0xF0F0;
@@ -94,6 +95,46 @@ static void __xdp_mem_allocator_rcu_free(struct rcu_head *rcu)
kfree(xa);
}
+bool __mem_id_disconnect(int id)
+{
+ struct xdp_mem_allocator *xa;
+ bool safe_to_remove = true;
+
+ mutex_lock(&mem_id_lock);
+
+ xa = rhashtable_lookup_fast(mem_id_ht, &id, mem_id_rht_params);
+ if (!xa) {
+ mutex_unlock(&mem_id_lock);
+ WARN(1, "Request remove non-existing id(%d), driver bug?", id);
+ return true;
+ }
+
+ /* Detects in-flight packet-pages for page_pool */
+ if (xa->mem.type == MEM_TYPE_PAGE_POOL)
+ safe_to_remove = page_pool_request_shutdown(xa->page_pool);
+
+ if (safe_to_remove &&
+ !rhashtable_remove_fast(mem_id_ht, &xa->node, mem_id_rht_params))
+ call_rcu(&xa->rcu, __xdp_mem_allocator_rcu_free);
+
+ mutex_unlock(&mem_id_lock);
+ return safe_to_remove;
+}
+
+#define DEFER_TIME (msecs_to_jiffies(1000))
+
+static void mem_id_disconnect_defer_retry(struct work_struct *wq)
+{
+ struct delayed_work *dwq = to_delayed_work(wq);
+ struct xdp_mem_allocator *xa = container_of(dwq, typeof(*xa), defer_wq);
+
+ if (__mem_id_disconnect(xa->mem.id))
+ return;
+
+ /* Still not ready to be disconnected, retry later */
+ schedule_delayed_work(&xa->defer_wq, DEFER_TIME);
+}
+
void xdp_rxq_info_unreg_mem_model(struct xdp_rxq_info *xdp_rxq)
{
struct xdp_mem_allocator *xa;
@@ -112,16 +153,28 @@ void xdp_rxq_info_unreg_mem_model(struct xdp_rxq_info *xdp_rxq)
if (id == 0)
return;
+ if (__mem_id_disconnect(id))
+ return;
+
+ /* Could not disconnect, defer new disconnect attempt to later */
mutex_lock(&mem_id_lock);
xa = rhashtable_lookup_fast(mem_id_ht, &id, mem_id_rht_params);
- if (xa && !rhashtable_remove_fast(mem_id_ht, &xa->node, mem_id_rht_params))
- call_rcu(&xa->rcu, __xdp_mem_allocator_rcu_free);
+ if (!xa) {
+ mutex_unlock(&mem_id_lock);
+ return;
+ }
+ INIT_DELAYED_WORK(&xa->defer_wq, mem_id_disconnect_defer_retry);
mutex_unlock(&mem_id_lock);
+ schedule_delayed_work(&xa->defer_wq, DEFER_TIME);
}
EXPORT_SYMBOL_GPL(xdp_rxq_info_unreg_mem_model);
+/* This unregister operation will also cleanup and destroy the
+ * allocator. The page_pool_free() operation is first called when it's
+ * safe to remove, possibly deferred to a workqueue.
+ */
void xdp_rxq_info_unreg(struct xdp_rxq_info *xdp_rxq)
{
/* Simplify driver cleanup code paths, allow unreg "unused" */
^ permalink raw reply related
* [PATCH net-next v2 10/12] xdp: add tracepoints for XDP mem
From: Jesper Dangaard Brouer @ 2019-06-18 13:05 UTC (permalink / raw)
To: netdev, Ilias Apalodimas, Toke Høiland-Jørgensen,
Tariq Toukan, Jesper Dangaard Brouer
Cc: toshiaki.makita1, grygorii.strashko, ivan.khoronzhuk, mcroce
In-Reply-To: <156086304827.27760.11339786046465638081.stgit@firesoul>
These tracepoints make it easier to troubleshoot XDP mem id disconnect.
The xdp:mem_disconnect tracepoint cannot be replaced via kprobe. It is
placed at the last stable place for the pointer to struct xdp_mem_allocator,
just before it's scheduled for RCU removal. It also extract info on
'safe_to_remove' and 'force'.
Detailed info about in-flight pages is not available at this layer. The next
patch will added tracepoints needed at the page_pool layer for this.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
---
include/net/xdp_priv.h | 23 +++++++++
include/trace/events/xdp.h | 115 ++++++++++++++++++++++++++++++++++++++++++++
net/core/xdp.c | 21 ++------
3 files changed, 143 insertions(+), 16 deletions(-)
create mode 100644 include/net/xdp_priv.h
diff --git a/include/net/xdp_priv.h b/include/net/xdp_priv.h
new file mode 100644
index 000000000000..6a8cba6ea79a
--- /dev/null
+++ b/include/net/xdp_priv.h
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __LINUX_NET_XDP_PRIV_H__
+#define __LINUX_NET_XDP_PRIV_H__
+
+#include <linux/rhashtable.h>
+
+/* Private to net/core/xdp.c, but used by trace/events/xdp.h */
+struct xdp_mem_allocator {
+ struct xdp_mem_info mem;
+ union {
+ void *allocator;
+ struct page_pool *page_pool;
+ struct zero_copy_allocator *zc_alloc;
+ };
+ int disconnect_cnt;
+ unsigned long defer_start;
+ struct rhash_head node;
+ struct rcu_head rcu;
+ struct delayed_work defer_wq;
+ unsigned long defer_warn;
+};
+
+#endif /* __LINUX_NET_XDP_PRIV_H__ */
diff --git a/include/trace/events/xdp.h b/include/trace/events/xdp.h
index e95cb86b65cf..bb5e380e2ef3 100644
--- a/include/trace/events/xdp.h
+++ b/include/trace/events/xdp.h
@@ -269,6 +269,121 @@ TRACE_EVENT(xdp_devmap_xmit,
__entry->from_ifindex, __entry->to_ifindex, __entry->err)
);
+/* Expect users already include <net/xdp.h>, but not xdp_priv.h */
+#include <net/xdp_priv.h>
+
+#define __MEM_TYPE_MAP(FN) \
+ FN(PAGE_SHARED) \
+ FN(PAGE_ORDER0) \
+ FN(PAGE_POOL) \
+ FN(ZERO_COPY)
+
+#define __MEM_TYPE_TP_FN(x) \
+ TRACE_DEFINE_ENUM(MEM_TYPE_##x);
+#define __MEM_TYPE_SYM_FN(x) \
+ { MEM_TYPE_##x, #x },
+#define __MEM_TYPE_SYM_TAB \
+ __MEM_TYPE_MAP(__MEM_TYPE_SYM_FN) { -1, 0 }
+__MEM_TYPE_MAP(__MEM_TYPE_TP_FN)
+
+TRACE_EVENT(mem_disconnect,
+
+ TP_PROTO(const struct xdp_mem_allocator *xa,
+ bool safe_to_remove, bool force),
+
+ TP_ARGS(xa, safe_to_remove, force),
+
+ TP_STRUCT__entry(
+ __field(const struct xdp_mem_allocator *, xa)
+ __field(u32, mem_id)
+ __field(u32, mem_type)
+ __field(const void *, allocator)
+ __field(bool, safe_to_remove)
+ __field(bool, force)
+ __field(int, disconnect_cnt)
+ ),
+
+ TP_fast_assign(
+ __entry->xa = xa;
+ __entry->mem_id = xa->mem.id;
+ __entry->mem_type = xa->mem.type;
+ __entry->allocator = xa->allocator;
+ __entry->safe_to_remove = safe_to_remove;
+ __entry->force = force;
+ __entry->disconnect_cnt = xa->disconnect_cnt;
+ ),
+
+ TP_printk("mem_id=%d mem_type=%s allocator=%p"
+ " safe_to_remove=%s force=%s disconnect_cnt=%d",
+ __entry->mem_id,
+ __print_symbolic(__entry->mem_type, __MEM_TYPE_SYM_TAB),
+ __entry->allocator,
+ __entry->safe_to_remove ? "true" : "false",
+ __entry->force ? "true" : "false",
+ __entry->disconnect_cnt
+ )
+);
+
+TRACE_EVENT(mem_connect,
+
+ TP_PROTO(const struct xdp_mem_allocator *xa,
+ const struct xdp_rxq_info *rxq),
+
+ TP_ARGS(xa, rxq),
+
+ TP_STRUCT__entry(
+ __field(const struct xdp_mem_allocator *, xa)
+ __field(u32, mem_id)
+ __field(u32, mem_type)
+ __field(const void *, allocator)
+ __field(const struct xdp_rxq_info *, rxq)
+ __field(int, ifindex)
+ ),
+
+ TP_fast_assign(
+ __entry->xa = xa;
+ __entry->mem_id = xa->mem.id;
+ __entry->mem_type = xa->mem.type;
+ __entry->allocator = xa->allocator;
+ __entry->rxq = rxq;
+ __entry->ifindex = rxq->dev->ifindex;
+ ),
+
+ TP_printk("mem_id=%d mem_type=%s allocator=%p"
+ " ifindex=%d",
+ __entry->mem_id,
+ __print_symbolic(__entry->mem_type, __MEM_TYPE_SYM_TAB),
+ __entry->allocator,
+ __entry->ifindex
+ )
+);
+
+TRACE_EVENT(mem_return_failed,
+
+ TP_PROTO(const struct xdp_mem_info *mem,
+ const struct page *page),
+
+ TP_ARGS(mem, page),
+
+ TP_STRUCT__entry(
+ __field(const struct page *, page)
+ __field(u32, mem_id)
+ __field(u32, mem_type)
+ ),
+
+ TP_fast_assign(
+ __entry->page = page;
+ __entry->mem_id = mem->id;
+ __entry->mem_type = mem->type;
+ ),
+
+ TP_printk("mem_id=%d mem_type=%s page=%p",
+ __entry->mem_id,
+ __print_symbolic(__entry->mem_type, __MEM_TYPE_SYM_TAB),
+ __entry->page
+ )
+);
+
#endif /* _TRACE_XDP_H */
#include <trace/define_trace.h>
diff --git a/net/core/xdp.c b/net/core/xdp.c
index 53bce4fa776a..4ad9e48b76a8 100644
--- a/net/core/xdp.c
+++ b/net/core/xdp.c
@@ -14,6 +14,8 @@
#include <net/page_pool.h>
#include <net/xdp.h>
+#include <net/xdp_priv.h> /* struct xdp_mem_allocator */
+#include <trace/events/xdp.h>
#define REG_STATE_NEW 0x0
#define REG_STATE_REGISTERED 0x1
@@ -29,21 +31,6 @@ static int mem_id_next = MEM_ID_MIN;
static bool mem_id_init; /* false */
static struct rhashtable *mem_id_ht;
-struct xdp_mem_allocator {
- struct xdp_mem_info mem;
- union {
- void *allocator;
- struct page_pool *page_pool;
- struct zero_copy_allocator *zc_alloc;
- };
- struct rhash_head node;
- struct rcu_head rcu;
- struct delayed_work defer_wq;
- unsigned long defer_start;
- unsigned long defer_warn;
- int disconnect_cnt;
-};
-
static u32 xdp_mem_id_hashfn(const void *data, u32 len, u32 seed)
{
const u32 *k = data;
@@ -117,7 +104,7 @@ bool __mem_id_disconnect(int id, bool force)
if (xa->mem.type == MEM_TYPE_PAGE_POOL)
safe_to_remove = page_pool_request_shutdown(xa->page_pool);
- /* TODO: Tracepoint will be added here in next-patch */
+ trace_mem_disconnect(xa, safe_to_remove, force);
if ((safe_to_remove || force) &&
!rhashtable_remove_fast(mem_id_ht, &xa->node, mem_id_rht_params))
@@ -385,6 +372,7 @@ int xdp_rxq_info_reg_mem_model(struct xdp_rxq_info *xdp_rxq,
mutex_unlock(&mem_id_lock);
+ trace_mem_connect(xdp_alloc, xdp_rxq);
return 0;
err:
mutex_unlock(&mem_id_lock);
@@ -417,6 +405,7 @@ static void __xdp_return(void *data, struct xdp_mem_info *mem, bool napi_direct,
} else {
/* Hopefully stack show who to blame for late return */
WARN_ONCE(1, "page_pool gone mem.id=%d", mem->id);
+ trace_mem_return_failed(mem, page);
put_page(page);
}
rcu_read_unlock();
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox