* [PATCH] page_pool: fix compile warning when CONFIG_PAGE_POOL is disabled
From: Jesper Dangaard Brouer @ 2019-06-19 22:15 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: netdev
Kbuild test robot reported compile warning:
warning: no return statement in function returning non-void
in function page_pool_request_shutdown, when CONFIG_PAGE_POOL is disabled.
The fix makes the code a little more verbose, with a descriptive variable.
Fixes: 99c07c43c4ea ("xdp: tracking page_pool resources and safe removal")
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Reported-by: kbuild test robot <lkp@intel.com>
---
include/net/page_pool.h | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/include/net/page_pool.h b/include/net/page_pool.h
index f09b3f1994e6..f07c518ef8a5 100644
--- a/include/net/page_pool.h
+++ b/include/net/page_pool.h
@@ -156,12 +156,12 @@ static inline void page_pool_recycle_direct(struct page_pool *pool,
bool __page_pool_request_shutdown(struct page_pool *pool);
static inline bool page_pool_request_shutdown(struct page_pool *pool)
{
- /* When page_pool isn't compiled-in, net/core/xdp.c doesn't
- * allow registering MEM_TYPE_PAGE_POOL, but shield linker.
- */
+ bool safe_to_remove = false;
+
#ifdef CONFIG_PAGE_POOL
- return __page_pool_request_shutdown(pool);
+ safe_to_remove = __page_pool_request_shutdown(pool);
#endif
+ return safe_to_remove;
}
/* Disconnects a page (from a page_pool). API users can have a need
^ permalink raw reply related
* Re: [PATCH bpf-next v7 1/9] bpf: implement getsockopt and setsockopt hooks
From: Stanislav Fomichev @ 2019-06-19 22:20 UTC (permalink / raw)
To: Andrii Nakryiko
Cc: Stanislav Fomichev, Networking, bpf, David S. Miller,
Alexei Starovoitov, Daniel Borkmann, Martin Lau
In-Reply-To: <CAEf4BzbTfCG3Mk9=78=Kh5u4ofxdYePm=xVEcN8g38cXQ_gq6Q@mail.gmail.com>
On 06/19, Andrii Nakryiko wrote:
> On Wed, Jun 19, 2019 at 1:17 PM Stanislav Fomichev <sdf@fomichev.me> wrote:
> >
> > On 06/19, Andrii Nakryiko wrote:
> > > On Wed, Jun 19, 2019 at 10:00 AM Stanislav Fomichev <sdf@google.com> wrote:
> > > >
> > > > Implement new BPF_PROG_TYPE_CGROUP_SOCKOPT program type and
> > > > BPF_CGROUP_{G,S}ETSOCKOPT cgroup hooks.
> > > >
> > > > BPF_CGROUP_SETSOCKOPT get a read-only view of the setsockopt arguments.
> > > > BPF_CGROUP_GETSOCKOPT can modify the supplied buffer.
> > > > Both of them reuse existing PTR_TO_PACKET{,_END} infrastructure.
> > > >
> > > > The buffer memory is pre-allocated (because I don't think there is
> > > > a precedent for working with __user memory from bpf). This might be
> > > > slow to do for each {s,g}etsockopt call, that's why I've added
> > > > __cgroup_bpf_prog_array_is_empty that exits early if there is nothing
> > > > attached to a cgroup. Note, however, that there is a race between
> > > > __cgroup_bpf_prog_array_is_empty and BPF_PROG_RUN_ARRAY where cgroup
> > > > program layout might have changed; this should not be a problem
> > > > because in general there is a race between multiple calls to
> > > > {s,g}etsocktop and user adding/removing bpf progs from a cgroup.
> > > >
> > > > The return code of the BPF program is handled as follows:
> > > > * 0: EPERM
> > > > * 1: success, continue with next BPF program in the cgroup chain
> > > >
> > > > v7:
> > > > * return only 0 or 1 (Alexei Starovoitov)
> > > > * always run all progs (Alexei Starovoitov)
> > > > * use optval=0 as kernel bypass in setsockopt (Alexei Starovoitov)
> > > > (decided to use optval=-1 instead, optval=0 might be a valid input)
> > > > * call getsockopt hook after kernel handlers (Alexei Starovoitov)
> > > >
> > > > v6:
> > > > * rework cgroup chaining; stop as soon as bpf program returns
> > > > 0 or 2; see patch with the documentation for the details
> > > > * drop Andrii's and Martin's Acked-by (not sure they are comfortable
> > > > with the new state of things)
> > >
> > > I like the general approach, just overall unclear about seemingly
> > > artificial restrictions I mentioned below.
> > >
> > > >
> > > > v5:
> > > > * skip copy_to_user() and put_user() when ret == 0 (Martin Lau)
> > > >
> > > > v4:
> > > > * don't export bpf_sk_fullsock helper (Martin Lau)
> > > > * size != sizeof(__u64) for uapi pointers (Martin Lau)
> > > > * offsetof instead of bpf_ctx_range when checking ctx access (Martin Lau)
> > > >
> > > > v3:
> > > > * typos in BPF_PROG_CGROUP_SOCKOPT_RUN_ARRAY comments (Andrii Nakryiko)
> > > > * reverse christmas tree in BPF_PROG_CGROUP_SOCKOPT_RUN_ARRAY (Andrii
> > > > Nakryiko)
> > > > * use __bpf_md_ptr instead of __u32 for optval{,_end} (Martin Lau)
> > > > * use BPF_FIELD_SIZEOF() for consistency (Martin Lau)
> > > > * new CG_SOCKOPT_ACCESS macro to wrap repeated parts
> > > >
> > > > v2:
> > > > * moved bpf_sockopt_kern fields around to remove a hole (Martin Lau)
> > > > * aligned bpf_sockopt_kern->buf to 8 bytes (Martin Lau)
> > > > * bpf_prog_array_is_empty instead of bpf_prog_array_length (Martin Lau)
> > > > * added [0,2] return code check to verifier (Martin Lau)
> > > > * dropped unused buf[64] from the stack (Martin Lau)
> > > > * use PTR_TO_SOCKET for bpf_sockopt->sk (Martin Lau)
> > > > * dropped bpf_target_off from ctx rewrites (Martin Lau)
> > > > * use return code for kernel bypass (Martin Lau & Andrii Nakryiko)
> > > >
> > > > Cc: Martin Lau <kafai@fb.com>
> > > > Signed-off-by: Stanislav Fomichev <sdf@google.com>
> > > > ---
> > >
> > > <snip>
> > >
> > > >
> > > > +struct bpf_sockopt_kern {
> > > > + struct sock *sk;
> > > > + u8 *optval;
> > > > + u8 *optval_end;
> > > > + s32 level;
> > > > + s32 optname;
> > > > + u32 optlen;
> > >
> > > Optlen is used below as signed integer, so switch it to s32?
> > Good catch, should be s32 here and below, thanks!
> >
> > > > + s32 retval;
> > > > +
> > > > + /* Small on-stack optval buffer to avoid small allocations.
> > > > + */
> > > > + u8 buf[64] __aligned(8);
> > > > +};
> > > > +
> > >
> > > <snip>
> > >
> > > >
> > > > +struct bpf_sockopt {
> > > > + __bpf_md_ptr(struct bpf_sock *, sk);
> > > > + __bpf_md_ptr(void *, optval);
> > > > + __bpf_md_ptr(void *, optval_end);
> > > > +
> > > > + __s32 level;
> > > > + __s32 optname;
> > > > + __u32 optlen;
> > >
> > > Same as above, we expect BPF program to be able to set it to -1, so __s32?
> > >
> > > > + __s32 retval;
> > > > +};
> > > > +
> > > > #endif /* _UAPI__LINUX_BPF_H__ */
> > > > diff --git a/kernel/bpf/cgroup.c b/kernel/bpf/cgroup.c
> > >
> > > <snip>
> > >
> > > > +
> > > > + if (ctx.optlen == -1)
> > > > + /* optlen set to -1, bypass kernel */
> > > > + ret = 1;
> > > > + else if (ctx.optlen == optlen)
> > > > + /* optlen not changed, run kernel handler */
> > > > + ret = 0;
> > > > + else
> > > > + /* any other value is rejected */
> > > > + ret = -EFAULT;
> > >
> > > I'm consufed about this assymetry between getsockopt and setsockopt
> > > behavior. Why we are disallowing setsockopt from changing optlen (and
> > > value itself)? Is there any harm in allowing that? Imagining some use
> > > case that provides transparent "support" for some option, you'd need
> > > to be able to intercept and provide custom values both for setsockopt
> > > and getsockopt. So unless I'm missing some security implications, why
> > > not make both sides able to write?
> > Because kernel setsockopt handlers use get_user to read the data. We
> > can definitely allow changing optval+optlen, but we'd have to copy
> > that data back to userspace to let kernel handle it. I'm not sure how
> > userspace might feel about it. Can it be a buffer in the readonly
> > elf section?
>
> Ah, ok, now I see why :) Yeah, I guess it can be in read-only section.
> Alright, I don't see an easy solution to that, I guess we can live
> with that for now.
>
> >
> > > Similar will apply w.r.t. retval, why can't setsockopt return EINVAL
> > > to reject some options? This seems very useful and very similar to
> > > what sysctl BPF hooks do.
> > I was just being defensive because I'm not sure what's the use-case.
> > We can already return EPERM, why do we need to return a different
> > error code? Are we comfortable letting progs return arbitrary number?
> > Or you just want to allow a bunch of pre-defined error codes?
> >
> > I haven't seen the ability to return arbitrary error from the sysctl
> > hooks, but maybe I didn't look hard enough.
>
> Yeah, seems like sysctl is only 0 or EPERM. I missed for a moment that
> there is return value from BPF program and retval from the context. I
> think it's good enough as is.
>
> >
> > > > +
> > > > +out:
> > > > + sockopt_free_buf(&ctx);
> > > > + return ret;
> > > > +}
> > > > +EXPORT_SYMBOL(__cgroup_bpf_run_filter_setsockopt);
> > > > +
> > > > +int __cgroup_bpf_run_filter_getsockopt(struct sock *sk, int level,
> > > > + int optname, char __user *optval,
> > > > + int __user *optlen, int max_optlen,
> > > > + int retval)
> > > > +{
> > >
> > > <snip>
> > >
> > > > +
> > > > + if (ctx.optlen > max_optlen) {
> > > > + ret = -EFAULT;
> > > > + goto out;
> > > > + }
> > > > +
> > > > + /* BPF programs only allowed to set retval to 0, not some
> > > > + * arbitrary value.
> > > > + */
> > > > + if (ctx.retval != 0 && ctx.retval != retval) {
> > >
> > > Lookin at manpage of getsockopt, seems like at least two error codes
> > > are relevant and generally useful for BPF program to be able to
> > > return: EINVAL and ENOPROTOOPT? Why we are disallowing anything but 0
> > > (or preserving original retval)?
> > I was thinking about simple use-case where it's either BPF that
> > handles the opt or the kernel. And then it's BFP returning success or
> > EPERM. I don't think I understand why BPF needs to be able to
> > return different error codes. We can certainly do that if you think
> > that it makes sense; alternatively, we can start with 0 or kernel retval
> > and relax the requirements if someone really needs that in the future.
> >
> > (I don't have a strong opinion here tbh).
>
> As replied above, EPERM is probably good enough for practical
> purposes, I was being a bit pedantic :)
Sounds good! I was also debating whether to allow BPF programs
to set arbitrary retval, but didn't find any good example on
why we need it :-)
> > > > + ret = -EFAULT;
> > > > + goto out;
> > > > + }
> > > > +
> > > > + if (copy_to_user(optval, ctx.optval, ctx.optlen) ||
> > > > + put_user(ctx.optlen, optlen)) {
> > > > + ret = -EFAULT;
> > > > + goto out;
> > > > + }
> > > > +
> > > > + ret = ctx.retval;
> > > > +
> > > > +out:
> > > > + sockopt_free_buf(&ctx);
> > > > + return ret;
> > > > +}
> > > > +EXPORT_SYMBOL(__cgroup_bpf_run_filter_getsockopt);
> > > > +
> > >
> > > <snip>
^ permalink raw reply
* [PATCH v2 net-next 0/5] ipv6: avoid taking refcnt on dst during route lookup
From: Wei Wang @ 2019-06-19 22:31 UTC (permalink / raw)
To: David Miller, netdev
Cc: Eric Dumazet, Mahesh Bandewar, Martin KaFai Lau, David Ahern,
Wei Wang
From: Wei Wang <weiwan@google.com>
Ipv6 route lookup code always grabs refcnt on the dst for the caller.
But for certain cases, grabbing refcnt is not always necessary if the
call path is rcu protected and the caller does not cache the dst.
Another issue in the route lookup logic is:
When there are multiple custom rules, we have to do the lookup into
each table associated to each rule individually. And when we can't
find the route in one table, we grab and release refcnt on
net->ipv6.ip6_null_entry before going to the next table.
This operation is completely redundant, and causes false issue because
net->ipv6.ip6_null_entry is a shared object.
This patch set introduces a new flag RT6_LOOKUP_F_DST_NOREF for route
lookup callers to set, to avoid any manipulation on the dst refcnt. And
it converts the major input and output path to use it.
The performance gain is noticable.
I ran synflood tests between 2 hosts under the same switch. Both hosts
have 20G mlx NIC, and 8 tx/rx queues.
Sender sends pure SYN flood with random src IPs and ports using trafgen.
Receiver has a simple TCP listener on the target port.
Both hosts have multiple custom rules:
- For incoming packets, only local table is traversed.
- For outgoing packets, 3 tables are traversed to find the route.
The packet processing rate on the receiver is as follows:
- Before the fix: 3.78Mpps
- After the fix: 5.50Mpps
v1->v2:
- Added a helper ip6_rt_put_flags() in patch 3 suggested by David Miller
Wei Wang (5):
ipv6: introduce RT6_LOOKUP_F_DST_NOREF flag in ip6_pol_route()
ipv6: initialize rt6->rt6i_uncached in all pre-allocated dst entries
ipv6: honor RT6_LOOKUP_F_DST_NOREF in rule lookup logic
ipv6: convert rx data path to not take refcnt on dst
ipv6: convert major tx path to use RT6_LOOKUP_F_DST_NOREF
drivers/net/vrf.c | 11 ++---
include/net/ip6_route.h | 36 +++++++++++++++-
include/net/l3mdev.h | 11 +++--
net/ipv6/fib6_rules.c | 12 +++---
net/ipv6/route.c | 93 +++++++++++++++++++----------------------
net/l3mdev/l3mdev.c | 22 +++++-----
6 files changed, 108 insertions(+), 77 deletions(-)
--
2.22.0.410.gd8fdbe21b5-goog
^ permalink raw reply
* [PATCH v2 net-next 1/5] ipv6: introduce RT6_LOOKUP_F_DST_NOREF flag in ip6_pol_route()
From: Wei Wang @ 2019-06-19 22:31 UTC (permalink / raw)
To: David Miller, netdev
Cc: Eric Dumazet, Mahesh Bandewar, Martin KaFai Lau, David Ahern,
Wei Wang
In-Reply-To: <20190619223158.35829-1-tracywwnj@gmail.com>
From: Wei Wang <weiwan@google.com>
This new flag is to instruct the route lookup function to not take
refcnt on the dst entry. The user which does route lookup with this flag
must properly use rcu protection.
ip6_pol_route() is the major route lookup function for both tx and rx
path.
In this function:
Do not take refcnt on dst if RT6_LOOKUP_F_DST_NOREF flag is set, and
directly return the route entry. The caller should be holding rcu lock
when using this flag, and decide whether to take refcnt or not.
One note on the dst cache in the uncached_list:
As uncached_list does not consume refcnt, one refcnt is always returned
back to the caller even if RT6_LOOKUP_F_DST_NOREF flag is set.
Uncached dst is only possible in the output path. So in such call path,
caller MUST check if the dst is in the uncached_list before assuming
that there is no refcnt taken on the returned dst.
Signed-off-by: Wei Wang <weiwan@google.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Acked-by: Mahesh Bandewar <maheshb@google.com>
---
include/net/ip6_route.h | 1 +
net/ipv6/route.c | 73 +++++++++++++++++------------------------
2 files changed, 31 insertions(+), 43 deletions(-)
diff --git a/include/net/ip6_route.h b/include/net/ip6_route.h
index 7375a165fd98..82bced2fc1e3 100644
--- a/include/net/ip6_route.h
+++ b/include/net/ip6_route.h
@@ -36,6 +36,7 @@ struct route_info {
#define RT6_LOOKUP_F_SRCPREF_PUBLIC 0x00000010
#define RT6_LOOKUP_F_SRCPREF_COA 0x00000020
#define RT6_LOOKUP_F_IGNORE_LINKSTATE 0x00000040
+#define RT6_LOOKUP_F_DST_NOREF 0x00000080
/* We do not (yet ?) support IPv6 jumbograms (RFC 2675)
* Unlike IPv4, hdr->seg_len doesn't include the IPv6 header
diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index c4d285fe0adc..9dcbc56e4151 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -1391,9 +1391,6 @@ static struct rt6_info *rt6_get_pcpu_route(const struct fib6_result *res)
pcpu_rt = this_cpu_read(*res->nh->rt6i_pcpu);
- if (pcpu_rt)
- ip6_hold_safe(NULL, &pcpu_rt);
-
return pcpu_rt;
}
@@ -1403,12 +1400,9 @@ static struct rt6_info *rt6_make_pcpu_route(struct net *net,
struct rt6_info *pcpu_rt, *prev, **p;
pcpu_rt = ip6_rt_pcpu_alloc(res);
- if (!pcpu_rt) {
- dst_hold(&net->ipv6.ip6_null_entry->dst);
- return net->ipv6.ip6_null_entry;
- }
+ if (!pcpu_rt)
+ return NULL;
- dst_hold(&pcpu_rt->dst);
p = this_cpu_ptr(res->nh->rt6i_pcpu);
prev = cmpxchg(p, NULL, pcpu_rt);
BUG_ON(prev);
@@ -2189,9 +2183,12 @@ struct rt6_info *ip6_pol_route(struct net *net, struct fib6_table *table,
const struct sk_buff *skb, int flags)
{
struct fib6_result res = {};
- struct rt6_info *rt;
+ struct rt6_info *rt = NULL;
int strict = 0;
+ WARN_ON_ONCE((flags & RT6_LOOKUP_F_DST_NOREF) &&
+ !rcu_read_lock_held());
+
strict |= flags & RT6_LOOKUP_F_IFACE;
strict |= flags & RT6_LOOKUP_F_IGNORE_LINKSTATE;
if (net->ipv6.devconf_all->forwarding == 0)
@@ -2200,23 +2197,15 @@ struct rt6_info *ip6_pol_route(struct net *net, struct fib6_table *table,
rcu_read_lock();
fib6_table_lookup(net, table, oif, fl6, &res, strict);
- if (res.f6i == net->ipv6.fib6_null_entry) {
- rt = net->ipv6.ip6_null_entry;
- rcu_read_unlock();
- dst_hold(&rt->dst);
- return rt;
- }
+ if (res.f6i == net->ipv6.fib6_null_entry)
+ goto out;
fib6_select_path(net, &res, fl6, oif, false, skb, strict);
/*Search through exception table */
rt = rt6_find_cached_rt(&res, &fl6->daddr, &fl6->saddr);
if (rt) {
- if (ip6_hold_safe(net, &rt))
- dst_use_noref(&rt->dst, jiffies);
-
- rcu_read_unlock();
- return rt;
+ goto out;
} else if (unlikely((fl6->flowi6_flags & FLOWI_FLAG_KNOWN_NH) &&
!res.nh->fib_nh_gw_family)) {
/* Create a RTF_CACHE clone which will not be
@@ -2224,40 +2213,38 @@ struct rt6_info *ip6_pol_route(struct net *net, struct fib6_table *table,
* the daddr in the skb during the neighbor look-up is different
* from the fl6->daddr used to look-up route here.
*/
- struct rt6_info *uncached_rt;
+ rt = ip6_rt_cache_alloc(&res, &fl6->daddr, NULL);
- uncached_rt = ip6_rt_cache_alloc(&res, &fl6->daddr, NULL);
-
- rcu_read_unlock();
-
- if (uncached_rt) {
- /* Uncached_rt's refcnt is taken during ip6_rt_cache_alloc()
- * No need for another dst_hold()
+ if (rt) {
+ /* 1 refcnt is taken during ip6_rt_cache_alloc().
+ * As rt6_uncached_list_add() does not consume refcnt,
+ * this refcnt is always returned to the caller even
+ * if caller sets RT6_LOOKUP_F_DST_NOREF flag.
*/
- rt6_uncached_list_add(uncached_rt);
+ rt6_uncached_list_add(rt);
atomic_inc(&net->ipv6.rt6_stats->fib_rt_uncache);
- } else {
- uncached_rt = net->ipv6.ip6_null_entry;
- dst_hold(&uncached_rt->dst);
- }
+ rcu_read_unlock();
- return uncached_rt;
+ return rt;
+ }
} else {
/* Get a percpu copy */
-
- struct rt6_info *pcpu_rt;
-
local_bh_disable();
- pcpu_rt = rt6_get_pcpu_route(&res);
+ rt = rt6_get_pcpu_route(&res);
- if (!pcpu_rt)
- pcpu_rt = rt6_make_pcpu_route(net, &res);
+ if (!rt)
+ rt = rt6_make_pcpu_route(net, &res);
local_bh_enable();
- rcu_read_unlock();
-
- return pcpu_rt;
}
+out:
+ if (!rt)
+ rt = net->ipv6.ip6_null_entry;
+ if (!(flags & RT6_LOOKUP_F_DST_NOREF))
+ ip6_hold_safe(net, &rt);
+ rcu_read_unlock();
+
+ return rt;
}
EXPORT_SYMBOL_GPL(ip6_pol_route);
--
2.22.0.410.gd8fdbe21b5-goog
^ permalink raw reply related
* [PATCH v2 net-next 2/5] ipv6: initialize rt6->rt6i_uncached in all pre-allocated dst entries
From: Wei Wang @ 2019-06-19 22:31 UTC (permalink / raw)
To: David Miller, netdev
Cc: Eric Dumazet, Mahesh Bandewar, Martin KaFai Lau, David Ahern,
Wei Wang
In-Reply-To: <20190619223158.35829-1-tracywwnj@gmail.com>
From: Wei Wang <weiwan@google.com>
Initialize rt6->rt6i_uncached on the following pre-allocated dsts:
net->ipv6.ip6_null_entry
net->ipv6.ip6_prohibit_entry
net->ipv6.ip6_blk_hole_entry
This is a preparation patch for later commits to be able to distinguish
dst entries in uncached list by doing:
!list_empty(rt6->rt6i_uncached)
Signed-off-by: Wei Wang <weiwan@google.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Acked-by: Mahesh Bandewar <maheshb@google.com>
---
net/ipv6/route.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index 9dcbc56e4151..33dc8af9a4bf 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -6010,6 +6010,7 @@ static int __net_init ip6_route_net_init(struct net *net)
net->ipv6.ip6_null_entry->dst.ops = &net->ipv6.ip6_dst_ops;
dst_init_metrics(&net->ipv6.ip6_null_entry->dst,
ip6_template_metrics, true);
+ INIT_LIST_HEAD(&net->ipv6.ip6_null_entry->rt6i_uncached);
#ifdef CONFIG_IPV6_MULTIPLE_TABLES
net->ipv6.fib6_has_custom_rules = false;
@@ -6021,6 +6022,7 @@ static int __net_init ip6_route_net_init(struct net *net)
net->ipv6.ip6_prohibit_entry->dst.ops = &net->ipv6.ip6_dst_ops;
dst_init_metrics(&net->ipv6.ip6_prohibit_entry->dst,
ip6_template_metrics, true);
+ INIT_LIST_HEAD(&net->ipv6.ip6_prohibit_entry->rt6i_uncached);
net->ipv6.ip6_blk_hole_entry = kmemdup(&ip6_blk_hole_entry_template,
sizeof(*net->ipv6.ip6_blk_hole_entry),
@@ -6030,6 +6032,7 @@ static int __net_init ip6_route_net_init(struct net *net)
net->ipv6.ip6_blk_hole_entry->dst.ops = &net->ipv6.ip6_dst_ops;
dst_init_metrics(&net->ipv6.ip6_blk_hole_entry->dst,
ip6_template_metrics, true);
+ INIT_LIST_HEAD(&net->ipv6.ip6_blk_hole_entry->rt6i_uncached);
#endif
net->ipv6.sysctl.flush_delay = 0;
--
2.22.0.410.gd8fdbe21b5-goog
^ permalink raw reply related
* [PATCH v2 net-next 3/5] ipv6: honor RT6_LOOKUP_F_DST_NOREF in rule lookup logic
From: Wei Wang @ 2019-06-19 22:31 UTC (permalink / raw)
To: David Miller, netdev
Cc: Eric Dumazet, Mahesh Bandewar, Martin KaFai Lau, David Ahern,
Wei Wang
In-Reply-To: <20190619223158.35829-1-tracywwnj@gmail.com>
From: Wei Wang <weiwan@google.com>
This patch specifically converts the rule lookup logic to honor this
flag and not release refcnt when traversing each rule and calling
lookup() on each routing table.
Similar to previous patch, we also need some special handling of dst
entries in uncached list because there is always 1 refcnt taken for them
even if RT6_LOOKUP_F_DST_NOREF flag is set.
Signed-off-by: Wei Wang <weiwan@google.com>
---
include/net/ip6_route.h | 10 ++++++++++
net/ipv6/fib6_rules.c | 12 +++++++-----
2 files changed, 17 insertions(+), 5 deletions(-)
diff --git a/include/net/ip6_route.h b/include/net/ip6_route.h
index 82bced2fc1e3..0709835c01ad 100644
--- a/include/net/ip6_route.h
+++ b/include/net/ip6_route.h
@@ -94,6 +94,16 @@ static inline struct dst_entry *ip6_route_output(struct net *net,
return ip6_route_output_flags(net, sk, fl6, 0);
}
+/* Only conditionally release dst if flags indicates
+ * !RT6_LOOKUP_F_DST_NOREF or dst is in uncached_list.
+ */
+static inline void ip6_rt_put_flags(struct rt6_info *rt, int flags)
+{
+ if (!(flags & RT6_LOOKUP_F_DST_NOREF) ||
+ !list_empty(&rt->rt6i_uncached))
+ ip6_rt_put(rt);
+}
+
struct dst_entry *ip6_route_lookup(struct net *net, struct flowi6 *fl6,
const struct sk_buff *skb, int flags);
struct rt6_info *ip6_pol_route(struct net *net, struct fib6_table *table,
diff --git a/net/ipv6/fib6_rules.c b/net/ipv6/fib6_rules.c
index bcfae13409b5..d22b6c140f23 100644
--- a/net/ipv6/fib6_rules.c
+++ b/net/ipv6/fib6_rules.c
@@ -113,14 +113,15 @@ struct dst_entry *fib6_rule_lookup(struct net *net, struct flowi6 *fl6,
rt = lookup(net, net->ipv6.fib6_local_tbl, fl6, skb, flags);
if (rt != net->ipv6.ip6_null_entry && rt->dst.error != -EAGAIN)
return &rt->dst;
- ip6_rt_put(rt);
+ ip6_rt_put_flags(rt, flags);
rt = lookup(net, net->ipv6.fib6_main_tbl, fl6, skb, flags);
if (rt->dst.error != -EAGAIN)
return &rt->dst;
- ip6_rt_put(rt);
+ ip6_rt_put_flags(rt, flags);
}
- dst_hold(&net->ipv6.ip6_null_entry->dst);
+ if (!(flags & RT6_LOOKUP_F_DST_NOREF))
+ dst_hold(&net->ipv6.ip6_null_entry->dst);
return &net->ipv6.ip6_null_entry->dst;
}
@@ -237,13 +238,14 @@ static int __fib6_rule_action(struct fib_rule *rule, struct flowi *flp,
goto out;
}
again:
- ip6_rt_put(rt);
+ ip6_rt_put_flags(rt, flags);
err = -EAGAIN;
rt = NULL;
goto out;
discard_pkt:
- dst_hold(&rt->dst);
+ if (!(flags & RT6_LOOKUP_F_DST_NOREF))
+ dst_hold(&rt->dst);
out:
res->rt6 = rt;
return err;
--
2.22.0.410.gd8fdbe21b5-goog
^ permalink raw reply related
* [PATCH v2 net-next 4/5] ipv6: convert rx data path to not take refcnt on dst
From: Wei Wang @ 2019-06-19 22:31 UTC (permalink / raw)
To: David Miller, netdev
Cc: Eric Dumazet, Mahesh Bandewar, Martin KaFai Lau, David Ahern,
Wei Wang
In-Reply-To: <20190619223158.35829-1-tracywwnj@gmail.com>
From: Wei Wang <weiwan@google.com>
ip6_route_input() is the key function to do the route lookup in the
rx data path. All the callers to this function are already holding rcu
lock. So it is fairly easy to convert it to not take refcnt on the dst:
We pass in flag RT6_LOOKUP_F_DST_NOREF and do skb_dst_set_noref().
This saves a few atomic inc or dec operations and should boost
performance overall.
This also makes the logic more aligned with v4.
Signed-off-by: Wei Wang <weiwan@google.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Acked-by: Mahesh Bandewar <maheshb@google.com>
---
net/ipv6/route.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index 33dc8af9a4bf..d2b287635aab 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -2375,11 +2375,12 @@ u32 rt6_multipath_hash(const struct net *net, const struct flowi6 *fl6,
return mhash >> 1;
}
+/* Called with rcu held */
void ip6_route_input(struct sk_buff *skb)
{
const struct ipv6hdr *iph = ipv6_hdr(skb);
struct net *net = dev_net(skb->dev);
- int flags = RT6_LOOKUP_F_HAS_SADDR;
+ int flags = RT6_LOOKUP_F_HAS_SADDR | RT6_LOOKUP_F_DST_NOREF;
struct ip_tunnel_info *tun_info;
struct flowi6 fl6 = {
.flowi6_iif = skb->dev->ifindex,
@@ -2401,8 +2402,8 @@ void ip6_route_input(struct sk_buff *skb)
if (unlikely(fl6.flowi6_proto == IPPROTO_ICMPV6))
fl6.mp_hash = rt6_multipath_hash(net, &fl6, skb, flkeys);
skb_dst_drop(skb);
- skb_dst_set(skb,
- ip6_route_input_lookup(net, skb->dev, &fl6, skb, flags));
+ skb_dst_set_noref(skb, ip6_route_input_lookup(net, skb->dev,
+ &fl6, skb, flags));
}
static struct rt6_info *ip6_pol_route_output(struct net *net,
--
2.22.0.410.gd8fdbe21b5-goog
^ permalink raw reply related
* [PATCH v2 net-next 5/5] ipv6: convert major tx path to use RT6_LOOKUP_F_DST_NOREF
From: Wei Wang @ 2019-06-19 22:31 UTC (permalink / raw)
To: David Miller, netdev
Cc: Eric Dumazet, Mahesh Bandewar, Martin KaFai Lau, David Ahern,
Wei Wang
In-Reply-To: <20190619223158.35829-1-tracywwnj@gmail.com>
From: Wei Wang <weiwan@google.com>
For tx path, in most cases, we still have to take refcnt on the dst
cause the caller is caching the dst somewhere. But it still is
beneficial to make use of RT6_LOOKUP_F_DST_NOREF flag while doing the
route lookup. It is cause this flag prevents manipulating refcnt on
net->ipv6.ip6_null_entry when doing fib6_rule_lookup() to traverse each
routing table. The null_entry is a shared object and constant updates on
it cause false sharing.
We converted the current major lookup function ip6_route_output_flags()
to make use of RT6_LOOKUP_F_DST_NOREF.
Together with the change in the rx path, we see noticable performance
boost:
I ran synflood tests between 2 hosts under the same switch. Both hosts
have 20G mlx NIC, and 8 tx/rx queues.
Sender sends pure SYN flood with random src IPs and ports using trafgen.
Receiver has a simple TCP listener on the target port.
Both hosts have multiple custom rules:
- For incoming packets, only local table is traversed.
- For outgoing packets, 3 tables are traversed to find the route.
The packet processing rate on the receiver is as follows:
- Before the fix: 3.78Mpps
- After the fix: 5.50Mpps
Signed-off-by: Wei Wang <weiwan@google.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Acked-by: Mahesh Bandewar <maheshb@google.com>
---
drivers/net/vrf.c | 11 ++++++-----
include/net/ip6_route.h | 25 +++++++++++++++++++++++--
include/net/l3mdev.h | 11 +++++++----
net/ipv6/route.c | 10 ++++++----
net/l3mdev/l3mdev.c | 22 +++++++++++-----------
5 files changed, 53 insertions(+), 26 deletions(-)
diff --git a/drivers/net/vrf.c b/drivers/net/vrf.c
index 11b9525dff27..1d1ac78b167e 100644
--- a/drivers/net/vrf.c
+++ b/drivers/net/vrf.c
@@ -1072,12 +1072,14 @@ static struct sk_buff *vrf_l3_rcv(struct net_device *vrf_dev,
#if IS_ENABLED(CONFIG_IPV6)
/* send to link-local or multicast address via interface enslaved to
* VRF device. Force lookup to VRF table without changing flow struct
+ * No refcnt is taken on the dst.
*/
-static struct dst_entry *vrf_link_scope_lookup(const struct net_device *dev,
- struct flowi6 *fl6)
+static struct dst_entry *vrf_link_scope_lookup_noref(
+ const struct net_device *dev,
+ struct flowi6 *fl6)
{
struct net *net = dev_net(dev);
- int flags = RT6_LOOKUP_F_IFACE;
+ int flags = RT6_LOOKUP_F_IFACE | RT6_LOOKUP_F_DST_NOREF;
struct dst_entry *dst = NULL;
struct rt6_info *rt;
@@ -1087,7 +1089,6 @@ static struct dst_entry *vrf_link_scope_lookup(const struct net_device *dev,
*/
if (fl6->flowi6_oif == dev->ifindex) {
dst = &net->ipv6.ip6_null_entry->dst;
- dst_hold(dst);
return dst;
}
@@ -1107,7 +1108,7 @@ static const struct l3mdev_ops vrf_l3mdev_ops = {
.l3mdev_l3_rcv = vrf_l3_rcv,
.l3mdev_l3_out = vrf_l3_out,
#if IS_ENABLED(CONFIG_IPV6)
- .l3mdev_link_scope_lookup = vrf_link_scope_lookup,
+ .l3mdev_link_scope_lookup_noref = vrf_link_scope_lookup_noref,
#endif
};
diff --git a/include/net/ip6_route.h b/include/net/ip6_route.h
index 0709835c01ad..66802ecd81e5 100644
--- a/include/net/ip6_route.h
+++ b/include/net/ip6_route.h
@@ -84,8 +84,29 @@ struct dst_entry *ip6_route_input_lookup(struct net *net,
struct flowi6 *fl6,
const struct sk_buff *skb, int flags);
-struct dst_entry *ip6_route_output_flags(struct net *net, const struct sock *sk,
- struct flowi6 *fl6, int flags);
+struct dst_entry *ip6_route_output_flags_noref(struct net *net,
+ const struct sock *sk,
+ struct flowi6 *fl6, int flags);
+
+static inline struct dst_entry *ip6_route_output_flags(struct net *net,
+ const struct sock *sk,
+ struct flowi6 *fl6,
+ int flags) {
+ struct dst_entry *dst;
+ struct rt6_info *rt6;
+
+ rcu_read_lock();
+ dst = ip6_route_output_flags_noref(net, sk, fl6, flags);
+ rt6 = (struct rt6_info *)dst;
+ /* For dst cached in uncached_list, refcnt is already taken. */
+ if (list_empty(&rt6->rt6i_uncached) && !dst_hold_safe(dst)) {
+ dst = &net->ipv6.ip6_null_entry->dst;
+ dst_hold(dst);
+ }
+ rcu_read_unlock();
+
+ return dst;
+}
static inline struct dst_entry *ip6_route_output(struct net *net,
const struct sock *sk,
diff --git a/include/net/l3mdev.h b/include/net/l3mdev.h
index e942372b077b..d8c37317bb86 100644
--- a/include/net/l3mdev.h
+++ b/include/net/l3mdev.h
@@ -31,8 +31,9 @@ struct l3mdev_ops {
u16 proto);
/* IPv6 ops */
- struct dst_entry * (*l3mdev_link_scope_lookup)(const struct net_device *dev,
- struct flowi6 *fl6);
+ struct dst_entry * (*l3mdev_link_scope_lookup_noref)(
+ const struct net_device *dev,
+ struct flowi6 *fl6);
};
#ifdef CONFIG_NET_L3_MASTER_DEV
@@ -140,7 +141,8 @@ static inline bool netif_index_is_l3_master(struct net *net, int ifindex)
return rc;
}
-struct dst_entry *l3mdev_link_scope_lookup(struct net *net, struct flowi6 *fl6);
+struct dst_entry *l3mdev_link_scope_lookup_noref(struct net *net,
+ struct flowi6 *fl6);
static inline
struct sk_buff *l3mdev_l3_rcv(struct sk_buff *skb, u16 proto)
@@ -251,7 +253,8 @@ static inline bool netif_index_is_l3_master(struct net *net, int ifindex)
}
static inline
-struct dst_entry *l3mdev_link_scope_lookup(struct net *net, struct flowi6 *fl6)
+struct dst_entry *l3mdev_link_scope_lookup_noref(struct net *net,
+ struct flowi6 *fl6)
{
return NULL;
}
diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index d2b287635aab..602d00794b30 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -2415,8 +2415,9 @@ static struct rt6_info *ip6_pol_route_output(struct net *net,
return ip6_pol_route(net, table, fl6->flowi6_oif, fl6, skb, flags);
}
-struct dst_entry *ip6_route_output_flags(struct net *net, const struct sock *sk,
- struct flowi6 *fl6, int flags)
+struct dst_entry *ip6_route_output_flags_noref(struct net *net,
+ const struct sock *sk,
+ struct flowi6 *fl6, int flags)
{
bool any_src;
@@ -2424,13 +2425,14 @@ struct dst_entry *ip6_route_output_flags(struct net *net, const struct sock *sk,
(IPV6_ADDR_MULTICAST | IPV6_ADDR_LINKLOCAL)) {
struct dst_entry *dst;
- dst = l3mdev_link_scope_lookup(net, fl6);
+ dst = l3mdev_link_scope_lookup_noref(net, fl6);
if (dst)
return dst;
}
fl6->flowi6_iif = LOOPBACK_IFINDEX;
+ flags |= RT6_LOOKUP_F_DST_NOREF;
any_src = ipv6_addr_any(&fl6->saddr);
if ((sk && sk->sk_bound_dev_if) || rt6_need_strict(&fl6->daddr) ||
(fl6->flowi6_oif && any_src))
@@ -2443,7 +2445,7 @@ struct dst_entry *ip6_route_output_flags(struct net *net, const struct sock *sk,
return fib6_rule_lookup(net, fl6, NULL, flags, ip6_pol_route_output);
}
-EXPORT_SYMBOL_GPL(ip6_route_output_flags);
+EXPORT_SYMBOL_GPL(ip6_route_output_flags_noref);
struct dst_entry *ip6_blackhole_route(struct net *net, struct dst_entry *dst_orig)
{
diff --git a/net/l3mdev/l3mdev.c b/net/l3mdev/l3mdev.c
index cfc9fcb97465..06133426549b 100644
--- a/net/l3mdev/l3mdev.c
+++ b/net/l3mdev/l3mdev.c
@@ -114,35 +114,35 @@ u32 l3mdev_fib_table_by_index(struct net *net, int ifindex)
EXPORT_SYMBOL_GPL(l3mdev_fib_table_by_index);
/**
- * l3mdev_link_scope_lookup - IPv6 route lookup based on flow for link
- * local and multicast addresses
+ * l3mdev_link_scope_lookup_noref - IPv6 route lookup based on flow
+ * for link local and multicast addresses
* @net: network namespace for device index lookup
* @fl6: IPv6 flow struct for lookup
+ * This function does not hold refcnt on the returned dst.
+ * Caller must hold rcu_read_lock().
*/
-struct dst_entry *l3mdev_link_scope_lookup(struct net *net,
- struct flowi6 *fl6)
+struct dst_entry *l3mdev_link_scope_lookup_noref(struct net *net,
+ struct flowi6 *fl6)
{
struct dst_entry *dst = NULL;
struct net_device *dev;
+ WARN_ON_ONCE(!rcu_read_lock_held());
if (fl6->flowi6_oif) {
- rcu_read_lock();
-
dev = dev_get_by_index_rcu(net, fl6->flowi6_oif);
if (dev && netif_is_l3_slave(dev))
dev = netdev_master_upper_dev_get_rcu(dev);
if (dev && netif_is_l3_master(dev) &&
- dev->l3mdev_ops->l3mdev_link_scope_lookup)
- dst = dev->l3mdev_ops->l3mdev_link_scope_lookup(dev, fl6);
-
- rcu_read_unlock();
+ dev->l3mdev_ops->l3mdev_link_scope_lookup_noref)
+ dst = dev->l3mdev_ops->
+ l3mdev_link_scope_lookup_noref(dev, fl6);
}
return dst;
}
-EXPORT_SYMBOL_GPL(l3mdev_link_scope_lookup);
+EXPORT_SYMBOL_GPL(l3mdev_link_scope_lookup_noref);
/**
* l3mdev_fib_rule_match - Determine if flowi references an
--
2.22.0.410.gd8fdbe21b5-goog
^ permalink raw reply related
* Re: [PATCH v2 net-next 3/5] ipv6: honor RT6_LOOKUP_F_DST_NOREF in rule lookup logic
From: David Ahern @ 2019-06-19 23:12 UTC (permalink / raw)
To: Wei Wang, David Miller, netdev
Cc: Eric Dumazet, Mahesh Bandewar, Martin KaFai Lau, Wei Wang
In-Reply-To: <20190619223158.35829-4-tracywwnj@gmail.com>
On 6/19/19 4:31 PM, Wei Wang wrote:
> diff --git a/net/ipv6/fib6_rules.c b/net/ipv6/fib6_rules.c
> index bcfae13409b5..d22b6c140f23 100644
> --- a/net/ipv6/fib6_rules.c
> +++ b/net/ipv6/fib6_rules.c
> @@ -113,14 +113,15 @@ struct dst_entry *fib6_rule_lookup(struct net *net, struct flowi6 *fl6,
> rt = lookup(net, net->ipv6.fib6_local_tbl, fl6, skb, flags);
> if (rt != net->ipv6.ip6_null_entry && rt->dst.error != -EAGAIN)
> return &rt->dst;
> - ip6_rt_put(rt);
> + ip6_rt_put_flags(rt, flags);
> rt = lookup(net, net->ipv6.fib6_main_tbl, fl6, skb, flags);
> if (rt->dst.error != -EAGAIN)
> return &rt->dst;
> - ip6_rt_put(rt);
> + ip6_rt_put_flags(rt, flags);
> }
>
> - dst_hold(&net->ipv6.ip6_null_entry->dst);
> + if (!(flags & RT6_LOOKUP_F_DST_NOREF))
> + dst_hold(&net->ipv6.ip6_null_entry->dst);
> return &net->ipv6.ip6_null_entry->dst;
> }
>
What about the fib6_rule_lookup when CONFIG_IPV6_MULTIPLE_TABLES is
configured off? If caller passes in RT6_LOOKUP_F_DST_NOREF that version
is still taking a reference in the error path and does a put after the
lookup.
^ permalink raw reply
* RE: [Intel-wired-lan] [PATCH][next][V2] ixgbe: fix potential u32 overflow on shift
From: Bowers, AndrewX @ 2019-06-19 23:13 UTC (permalink / raw)
To: intel-wired-lan@lists.osuosl.org, netdev@vger.kernel.org
Cc: kernel-janitors@vger.kernel.org, linux-kernel@vger.kernel.org
In-Reply-To: <20190607181920.23339-1-colin.king@canonical.com>
> -----Original Message-----
> From: Intel-wired-lan [mailto:intel-wired-lan-bounces@osuosl.org] On
> Behalf Of Colin King
> Sent: Friday, June 7, 2019 11:19 AM
> To: Keller, Jacob E <jacob.e.keller@intel.com>; Kirsher, Jeffrey T
> <jeffrey.t.kirsher@intel.com>; David S . Miller <davem@davemloft.net>;
> intel-wired-lan@lists.osuosl.org; netdev@vger.kernel.org
> Cc: kernel-janitors@vger.kernel.org; linux-kernel@vger.kernel.org
> Subject: [Intel-wired-lan] [PATCH][next][V2] ixgbe: fix potential u32
> overflow on shift
>
> From: Colin Ian King <colin.king@canonical.com>
>
> The u32 variable rem is being shifted using u32 arithmetic however it is being
> passed to div_u64 that expects the expression to be a u64.
> The 32 bit shift may potentially overflow, so cast rem to a u64 before shifting
> to avoid this. Also remove comment about overflow.
>
> Addresses-Coverity: ("Unintentional integer overflow")
> Fixes: cd4583206990 ("ixgbe: implement support for SDP/PPS output on X550
> hardware")
> Fixes: 68d9676fc04e ("ixgbe: fix PTP SDP pin setup on X540 hardware")
> Signed-off-by: Colin Ian King <colin.king@canonical.com>
> ---
>
> V2: update comment
>
> ---
> drivers/net/ethernet/intel/ixgbe/ixgbe_ptp.c | 14 ++++----------
> 1 file changed, 4 insertions(+), 10 deletions(-)
Tested-by: Andrew Bowers <andrewx.bowers@intel.com>
^ permalink raw reply
* Re: [PATCH v3 0/7] Hexdump Enhancements
From: Alastair D'Silva @ 2019-06-19 23:15 UTC (permalink / raw)
To: Joe Perches
Cc: Jani Nikula, Joonas Lahtinen, Rodrigo Vivi, David Airlie,
Daniel Vetter, Dan Carpenter, Karsten Keil, Jassi Brar,
Tom Lendacky, David S. Miller, Jose Abreu, Kalle Valo,
Stanislaw Gruszka, Benson Leung, Enric Balletbo i Serra,
James E.J. Bottomley, Martin K. Petersen, Greg Kroah-Hartman,
Alexander Viro, Petr Mladek, Sergey Senozhatsky, Steven Rostedt,
David Laight, Andrew Morton, intel-gfx, dri-devel, linux-kernel,
netdev, ath10k, linux-wireless, linux-scsi, linux-fbdev, devel,
linux-fsdevel
In-Reply-To: <9a000734375c0801fc16b71f4be1235f9b857772.camel@perches.com>
On Wed, 2019-06-19 at 09:31 -0700, Joe Perches wrote:
> On Mon, 2019-06-17 at 12:04 +1000, Alastair D'Silva wrote:
> > From: Alastair D'Silva <alastair@d-silva.org>
> >
> > Apologies for the large CC list, it's a heads up for those
> > responsible
> > for subsystems where a prototype change in generic code causes a
> > change
> > in those subsystems.
> >
> > This series enhances hexdump.
>
> Still not a fan of these patches.
I'm afraid there's not too much action I can take on that, I'm happy to
address specific issues though.
>
> > These improve the readability of the dumped data in certain
> > situations
> > (eg. wide terminals are available, many lines of empty bytes exist,
> > etc).
>
> Changing hexdump's last argument from bool to int is odd.
>
Think of it as replacing a single boolean with many booleans.
> Perhaps a new function should be added instead of changing
> the existing hexdump.
>
There's only a handful of consumers, I don't think there is a value-add
in creating more wrappers vs updating the existing callers.
--
Alastair D'Silva mob: 0423 762 819
skype: alastair_dsilva
Twitter: @EvilDeece
blog: http://alastair.d-silva.org
^ permalink raw reply
* Re: [PATCH v2 net-next 5/5] ipv6: convert major tx path to use RT6_LOOKUP_F_DST_NOREF
From: David Ahern @ 2019-06-19 23:21 UTC (permalink / raw)
To: Wei Wang, David Miller, netdev
Cc: Eric Dumazet, Mahesh Bandewar, Martin KaFai Lau, Wei Wang
In-Reply-To: <20190619223158.35829-6-tracywwnj@gmail.com>
On 6/19/19 4:31 PM, Wei Wang wrote:
> diff --git a/include/net/l3mdev.h b/include/net/l3mdev.h
> index e942372b077b..d8c37317bb86 100644
> --- a/include/net/l3mdev.h
> +++ b/include/net/l3mdev.h
> @@ -31,8 +31,9 @@ struct l3mdev_ops {
> u16 proto);
>
> /* IPv6 ops */
> - struct dst_entry * (*l3mdev_link_scope_lookup)(const struct net_device *dev,
> - struct flowi6 *fl6);
> + struct dst_entry * (*l3mdev_link_scope_lookup_noref)(
> + const struct net_device *dev,
> + struct flowi6 *fl6);
I would prefer to add a comment about not taking a references vs adding
the _noref extension. There is only 1 user for 1 context and the name
length is getting out of hand (as evidenced by the line wrapping below).
^ permalink raw reply
* RE: [Intel-wired-lan] [PATCH] ixgbe: Avoid NULL pointer dereference with VF on non-IPsec hw
From: Bowers, AndrewX @ 2019-06-19 23:25 UTC (permalink / raw)
To: netdev@vger.kernel.org, intel-wired-lan@lists.osuosl.org,
linux-kernel@vger.kernel.org
In-Reply-To: <20190522232258.10353-1-dann.frazier@canonical.com>
> -----Original Message-----
> From: Intel-wired-lan [mailto:intel-wired-lan-bounces@osuosl.org] On
> Behalf Of dann frazier
> Sent: Wednesday, May 22, 2019 4:23 PM
> To: Kirsher, Jeffrey T <jeffrey.t.kirsher@intel.com>; David S. Miller
> <davem@davemloft.net>; Shannon Nelson <shannon.nelson@oracle.com>
> Cc: netdev@vger.kernel.org; intel-wired-lan@lists.osuosl.org; linux-
> kernel@vger.kernel.org
> Subject: [Intel-wired-lan] [PATCH] ixgbe: Avoid NULL pointer dereference
> with VF on non-IPsec hw
>
> An ipsec structure will not be allocated if the hardware does not support
> offload. Fixes the following Oops:
>
> [ 191.045452] Unable to handle kernel NULL pointer dereference at virtual
> address 0000000000000000 [ 191.054232] Mem abort info:
> [ 191.057014] ESR = 0x96000004
> [ 191.060057] Exception class = DABT (current EL), IL = 32 bits
> [ 191.065963] SET = 0, FnV = 0
> [ 191.069004] EA = 0, S1PTW = 0
> [ 191.072132] Data abort info:
> [ 191.074999] ISV = 0, ISS = 0x00000004
> [ 191.078822] CM = 0, WnR = 0
> [ 191.081780] user pgtable: 4k pages, 48-bit VAs, pgdp = 0000000043d9e467 [
> 191.088382] [0000000000000000] pgd=0000000000000000 [ 191.093252]
> Internal error: Oops: 96000004 [#1] SMP [ 191.098119] Modules linked in:
> vhost_net vhost tap vfio_pci vfio_virqfd vfio_iommu_type1 vfio
> xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4
> nf_nat xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
> ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter devlink
> ebtables ip6table_filter ip6_tables iptable_filter bpfilter ipmi_ssif
> nls_iso8859_1 input_leds joydev ipmi_si hns_roce_hw_v2 ipmi_devintf
> hns_roce ipmi_msghandler cppc_cpufreq sch_fq_codel ib_iser rdma_cm
> iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
> ip_tables x_tables autofs4 ses enclosure btrfs zstd_compress raid10 raid456
> async_raid6_recov async_memcpy async_pq async_xor async_tx xor
> hid_generic usbhid hid raid6_pq libcrc32c raid1 raid0 multipath linear ixgbevf
> hibmc_drm ttm [ 191.168607] drm_kms_helper aes_ce_blk aes_ce_cipher
> syscopyarea crct10dif_ce sysfillrect ghash_ce qla2xxx sysimgblt sha2_ce
> sha256_arm64 hisi_sas_v3_hw fb_sys_fops sha1_ce uas nvme_fc mpt3sas
> ixgbe drm hisi_sas_main nvme_fabrics usb_storage hclge scsi_transport_fc
> ahci libsas hnae3 raid_class libahci xfrm_algo scsi_transport_sas mdio
> aes_neon_bs aes_neon_blk crypto_simd cryptd aes_arm64 [ 191.202952]
> CPU: 94 PID: 0 Comm: swapper/94 Not tainted 4.19.0-rc1+ #11 [ 191.209553]
> Hardware name: Huawei D06 /D06, BIOS Hisilicon D06 UEFI RC0 - V1.20.01
> 04/26/2019 [ 191.218064] pstate: 20400089 (nzCv daIf +PAN -UAO) [
> 191.222873] pc : ixgbe_ipsec_vf_clear+0x60/0xd0 [ixgbe] [ 191.228093] lr :
> ixgbe_msg_task+0x2d0/0x1088 [ixgbe] [ 191.233044] sp : ffff000009b3bcd0 [
> 191.236346] x29: ffff000009b3bcd0 x28: 0000000000000000 [ 191.241647] x27:
> ffff000009628000 x26: 0000000000000000 [ 191.246946] x25: ffff803f652d7600
> x24: 0000000000000004 [ 191.252246] x23: ffff803f6a718900 x22:
> 0000000000000000 [ 191.257546] x21: 0000000000000000 x20:
> 0000000000000000 [ 191.262845] x19: 0000000000000000 x18:
> 0000000000000000 [ 191.268144] x17: 0000000000000000 x16:
> 0000000000000000 [ 191.273443] x15: 0000000000000000 x14:
> 0000000100000026 [ 191.278742] x13: 0000000100000025 x12:
> ffff8a5f7fbe0df0 [ 191.284042] x11: 000000010000000b x10:
> 0000000000000040 [ 191.289341] x9 : 0000000000001100 x8 : ffff803f6a824fd8
> [ 191.294640] x7 : ffff803f6a825098 x6 : 0000000000000001 [ 191.299939] x5 :
> ffff000000f0ffc0 x4 : 0000000000000000 [ 191.305238] x3 : ffff000028c00000 x2
> : ffff803f652d7600 [ 191.310538] x1 : 0000000000000000 x0 : ffff000000f205f0 [
> 191.315838] Process swapper/94 (pid: 0, stack limit = 0x00000000addfed5a) [
> 191.322613] Call trace:
> [ 191.325055] ixgbe_ipsec_vf_clear+0x60/0xd0 [ixgbe] [ 191.329927]
> ixgbe_msg_task+0x2d0/0x1088 [ixgbe] [ 191.334536]
> ixgbe_msix_other+0x274/0x330 [ixgbe] [ 191.339233]
> __handle_irq_event_percpu+0x78/0x270
> [ 191.343924] handle_irq_event_percpu+0x40/0x98 [ 191.348355]
> handle_irq_event+0x50/0xa8 [ 191.352180] handle_fasteoi_irq+0xbc/0x148
> [ 191.356263] generic_handle_irq+0x34/0x50 [ 191.360259]
> __handle_domain_irq+0x68/0xc0 [ 191.364343] gic_handle_irq+0x84/0x180
> [ 191.368079] el1_irq+0xe8/0x180 [ 191.371208] arch_cpu_idle+0x30/0x1a8
> [ 191.374860] do_idle+0x1dc/0x2a0 [ 191.378077]
> cpu_startup_entry+0x2c/0x30 [ 191.381988]
> secondary_start_kernel+0x150/0x1e0
> [ 191.386506] Code: 6b15003f 54000320 f1404a9f 54000060 (79400260)
>
> Fixes: eda0333ac2930 ("ixgbe: add VF IPsec management")
> Signed-off-by: dann frazier <dann.frazier@canonical.com>
> ---
> drivers/net/ethernet/intel/ixgbe/ixgbe_ipsec.c | 3 +++
> 1 file changed, 3 insertions(+)
Tested-by: Andrew Bowers <andrewx.bowers@intel.com>
^ permalink raw reply
* Re: [PATCH v2 net-next 3/5] ipv6: honor RT6_LOOKUP_F_DST_NOREF in rule lookup logic
From: Wei Wang @ 2019-06-19 23:25 UTC (permalink / raw)
To: David Ahern
Cc: Wei Wang, David Miller, Linux Kernel Network Developers,
Eric Dumazet, Mahesh Bandewar, Martin KaFai Lau
In-Reply-To: <857851ee-c610-0b19-b5b1-1948bebe84ab@gmail.com>
On Wed, Jun 19, 2019 at 4:12 PM David Ahern <dsahern@gmail.com> wrote:
>
> On 6/19/19 4:31 PM, Wei Wang wrote:
> > diff --git a/net/ipv6/fib6_rules.c b/net/ipv6/fib6_rules.c
> > index bcfae13409b5..d22b6c140f23 100644
> > --- a/net/ipv6/fib6_rules.c
> > +++ b/net/ipv6/fib6_rules.c
> > @@ -113,14 +113,15 @@ struct dst_entry *fib6_rule_lookup(struct net *net, struct flowi6 *fl6,
> > rt = lookup(net, net->ipv6.fib6_local_tbl, fl6, skb, flags);
> > if (rt != net->ipv6.ip6_null_entry && rt->dst.error != -EAGAIN)
> > return &rt->dst;
> > - ip6_rt_put(rt);
> > + ip6_rt_put_flags(rt, flags);
> > rt = lookup(net, net->ipv6.fib6_main_tbl, fl6, skb, flags);
> > if (rt->dst.error != -EAGAIN)
> > return &rt->dst;
> > - ip6_rt_put(rt);
> > + ip6_rt_put_flags(rt, flags);
> > }
> >
> > - dst_hold(&net->ipv6.ip6_null_entry->dst);
> > + if (!(flags & RT6_LOOKUP_F_DST_NOREF))
> > + dst_hold(&net->ipv6.ip6_null_entry->dst);
> > return &net->ipv6.ip6_null_entry->dst;
> > }
> >
>
> What about the fib6_rule_lookup when CONFIG_IPV6_MULTIPLE_TABLES is
> configured off? If caller passes in RT6_LOOKUP_F_DST_NOREF that version
> is still taking a reference in the error path and does a put after the
> lookup.
True. I missed that part. Thanks for catching it. Will update.
^ permalink raw reply
* Re: [PATCH v2 net-next 5/5] ipv6: convert major tx path to use RT6_LOOKUP_F_DST_NOREF
From: Wei Wang @ 2019-06-19 23:28 UTC (permalink / raw)
To: David Ahern
Cc: Wei Wang, David Miller, Linux Kernel Network Developers,
Eric Dumazet, Mahesh Bandewar, Martin KaFai Lau
In-Reply-To: <c0860def-4b20-610f-3fde-f9601eb1600e@gmail.com>
On Wed, Jun 19, 2019 at 4:21 PM David Ahern <dsahern@gmail.com> wrote:
>
> On 6/19/19 4:31 PM, Wei Wang wrote:
> > diff --git a/include/net/l3mdev.h b/include/net/l3mdev.h
> > index e942372b077b..d8c37317bb86 100644
> > --- a/include/net/l3mdev.h
> > +++ b/include/net/l3mdev.h
> > @@ -31,8 +31,9 @@ struct l3mdev_ops {
> > u16 proto);
> >
> > /* IPv6 ops */
> > - struct dst_entry * (*l3mdev_link_scope_lookup)(const struct net_device *dev,
> > - struct flowi6 *fl6);
> > + struct dst_entry * (*l3mdev_link_scope_lookup_noref)(
> > + const struct net_device *dev,
> > + struct flowi6 *fl6);
>
> I would prefer to add a comment about not taking a references vs adding
> the _noref extension. There is only 1 user for 1 context and the name
> length is getting out of hand (as evidenced by the line wrapping below).
Hmm OK. I was trying to make it clear to make sure future callers do
not misuse it.
^ permalink raw reply
* Re: [PATCH net v5 3/6] ipv4: Dump route exceptions if requested
From: Stefano Brivio @ 2019-06-19 23:57 UTC (permalink / raw)
To: David Ahern
Cc: David Miller, Jianlin Shi, Wei Wang, Martin KaFai Lau,
Eric Dumazet, Matti Vaittinen, netdev
In-Reply-To: <b85db470-81c2-3abe-a68b-154711147656@gmail.com>
On Tue, 18 Jun 2019 08:48:23 -0600
David Ahern <dsahern@gmail.com> wrote:
> > +++ b/net/ipv4/fib_trie.c
> > @@ -2000,28 +2000,92 @@ void fib_free_table(struct fib_table *tb)
> > call_rcu(&tb->rcu, __trie_free_rcu);
> > }
> >
> > +static int fib_dump_fnhe_from_leaf(struct fib_alias *fa, struct sk_buff *skb,
> > + struct netlink_callback *cb,
> > + int *fa_index, int fa_start)
> > +{
> > + struct net *net = sock_net(cb->skb->sk);
> > + struct fib_info *fi = fa->fa_info;
> > + struct fnhe_hash_bucket *bucket;
> > + struct fib_nh_common *nhc;
> > + int i, genid;
> > +
> > + if (!fi || fi->fib_flags & RTNH_F_DEAD)
> > + return 0;
> > +
> > + nhc = fib_info_nhc(fi, 0);
>
> This should be a loop over fi->fib_nhs for net:
> for (i = 0; i < fi->fib_nhs; i++) {
> nhc = fib_info_nhc(fi, 0);
> ...
>
> and a loop over fib_info_num_path(fi) for net-next:
> for (i = 0; i < fib_info_num_path(fi); i++) {
> nhc = fib_info_nhc(fi, 0);
> ...
Right, I started this from net-next and only later "adapted" to net
clearly in the wrong way. Thanks for providing both expressions. Fixed
in v6.
>
> > + if (nhc->nhc_flags & RTNH_F_DEAD)
> > + return 0;
>
> And then the loop over the exception bucket could be a helper in route.c
> in which case you don't need to export rt_fill_info and nhc_exceptions
> code does not spread to fib_trie.c
Cleaner I guess, changed in v6.
> > +
> > + bucket = rcu_dereference(nhc->nhc_exceptions);
> > + if (!bucket)
> > + return 0;
> > +
> > + genid = fnhe_genid(net);
> > +
> > + for (i = 0; i < FNHE_HASH_SIZE; i++) {
> > + struct fib_nh_exception *fnhe;
> > +
> > + for (fnhe = rcu_dereference(bucket[i].chain); fnhe;
> > + fnhe = rcu_dereference(fnhe->fnhe_next)) {
> > + struct flowi4 fl4 = {};
>
> rather than pass an empty flow struct, update rt_fill_info to handle a
> NULL fl4; it's only a few checks.
Added patch and changed in v6.
--
Stefano
^ permalink raw reply
* Re: [PATCH net v5 5/6] ipv6: Dump route exceptions if requested
From: Stefano Brivio @ 2019-06-19 23:57 UTC (permalink / raw)
To: David Ahern
Cc: David Miller, Jianlin Shi, Wei Wang, Martin KaFai Lau,
Eric Dumazet, Matti Vaittinen, netdev
In-Reply-To: <333b0a08-07dd-3c70-1268-2d9eb5646564@gmail.com>
On Tue, 18 Jun 2019 09:19:53 -0600
David Ahern <dsahern@gmail.com> wrote:
> On 6/18/19 7:20 AM, Stefano Brivio wrote:
> > diff --git a/net/ipv6/route.c b/net/ipv6/route.c
> > index 0f60eb3a2873..7375f3b7d310 100644
> > --- a/net/ipv6/route.c
> > +++ b/net/ipv6/route.c
> > @@ -4854,33 +4854,94 @@ static bool fib6_info_uses_dev(const struct fib6_info *f6i,
> > return false;
> > }
> >
> > -int rt6_dump_route(struct fib6_info *rt, void *p_arg)
> > +/* Return -1 if done with node, number of handled routes on partial dump */
> > +int rt6_dump_route(struct fib6_info *rt, void *p_arg, unsigned int skip)
>
> Changing the return code of rt6_dump_route should be a separate patch.
I guess the purpose would be to highlight how existing cases are
changed, but that looks rather trivial to me. Anyway, changed in v6.
> > + if (filter->dump_routes) {
> > + if (skip) {
> > + skip--;
> > + } else {
> > + if (rt6_fill_node(net, arg->skb, rt, NULL, NULL, NULL,
> > + 0, RTM_NEWROUTE,
> > + NETLINK_CB(arg->cb->skb).portid,
> > + arg->cb->nlh->nlmsg_seq, flags)) {
> > + return 0;
> > + }
> > + count++;
> > + }
> > + }
> > +
> > + if (!filter->dump_exceptions)
> > + return -1;
> > +
>
> And the dump of the exception bucket should be a standalone function.
> You will see why with net-next (it is per fib6_nh).
Sure, no way around it now, changed in v6.
--
Stefano
^ permalink raw reply
* [PATCH net-next v6 00/11] Fix listing (IPv4, IPv6) and flushing (IPv6) of cached route exceptions
From: Stefano Brivio @ 2019-06-19 23:59 UTC (permalink / raw)
To: David Miller
Cc: Jianlin Shi, Wei Wang, David Ahern, Martin KaFai Lau,
Eric Dumazet, Matti Vaittinen, netdev
For IPv6 cached routes, the commands 'ip -6 route list cache' and
'ip -6 route flush cache' don't work at all after route exceptions have
been moved to a separate hash table in commit 2b760fcf5cfb ("ipv6: hook
up exception table to store dst cache").
For IPv4 cached routes, the command 'ip route list cache' has also
stopped working in kernel 3.5 after commit 4895c771c7f0 ("ipv4: Add FIB
nexthop exceptions.") introduced storage for route exceptions as a
separate entity.
Fix this by allowing userspace to clearly request cached routes with
the RTM_F_CLONED flag used as a filter (in conjuction with strict
checking) and by retrieving and dumping cached routes if requested.
If strict checking is not requested (iproute2 < 5.0.0), we don't have a
way to consistently filter results on other selectors (e.g. on tables),
so skip filtering entirely and dump both regular routes and exceptions.
For IPv4, cache flushing uses a completely different mechanism, so it
wasn't affected. Listing of exception routes (modified routes pre-3.5) was
tested against these versions of kernel and iproute2:
iproute2
kernel 4.14.0 4.15.0 4.19.0 5.0.0 5.1.0
3.5-rc4 + + + + +
4.4
4.9
4.14
4.15
4.19
5.0
5.1
fixed + + + + +
For IPv6, a separate iproute2 patch is required. Versions of iproute2
and kernel tested:
iproute2
kernel 4.14.0 4.15.0 4.19.0 5.0.0 5.1.0 5.1.0, patched
3.18 list + + + + + +
flush + + + + + +
4.4 list + + + + + +
flush + + + + + +
4.9 list + + + + + +
flush + + + + + +
4.14 list + + + + + +
flush + + + + + +
4.15 list
flush
4.19 list
flush
5.0 list
flush
5.1 list
flush
with list + + + + + +
fix flush + + + +
v6: Target for net-next, rebase and adapt to nexthop objects for IPv6 paths.
Merge selftests into this series (as they were addressed for net-next).
A number of minor changes detailed in logs of single patches.
v5: Skip filtering altogether if no strict checking is requested: selecting
routes or exceptions only would be inconsistent with the fact we can't
filter on tables. Drop 1/8 (non-strict dump filter function no longer
needed), replace 2/8 (don't use NLM_F_MATCH, decide to skip routes or
exceptions in filter function), drop 6/8 (2/8 is enough for IPv6 too).
Introduce dump_routes and dump_exceptions flags in filter, adapt other
patches to that.
v4: Fix the listing issue also for IPv4, making the behaviour consistent
with IPv6. Honour NLM_F_MATCH as per RFC 3549 and allow usage of
RTM_F_CLONED filter. Split patches into smaller logical changes.
v3: Drop check on RTM_F_CLONED and rework logic of return values of
rt6_dump_route()
v2: Add count of routes handled in partial dumps, and skip them, in patch 1/2.
Stefano Brivio (11):
fib_frontend, ip6_fib: Select routes or exceptions dump from
RTM_F_CLONED
ipv4/fib_frontend: Allow RTM_F_CLONED flag to be used for filtering
ipv4/route: Allow NULL flowinfo in rt_fill_info()
ipv4: Dump route exceptions if requested
Revert "net/ipv6: Bail early if user only wants cloned entries"
ipv6/route: Don't match on fc_nh_id if not set in ip6_route_del()
ipv6/route: Change return code of rt6_dump_route() for partial node
dumps
ipv6: Dump route exceptions if requested
ip6_fib: Don't discard nodes with valid routing information in
fib6_locate_1()
selftests: pmtu: Introduce list_flush_ipv4_exception test case
selftests: pmtu: Make list_flush_ipv6_exception test more demanding
include/net/ip6_fib.h | 1 +
include/net/ip6_route.h | 2 +-
include/net/ip_fib.h | 2 +
include/net/route.h | 4 +
net/ipv4/fib_frontend.c | 12 ++-
net/ipv4/fib_trie.c | 60 +++++++++++---
net/ipv4/route.c | 113 +++++++++++++++++++------
net/ipv6/ip6_fib.c | 27 ++++--
net/ipv6/route.c | 123 +++++++++++++++++++++++++---
tools/testing/selftests/net/pmtu.sh | 82 +++++++++++++++++--
10 files changed, 354 insertions(+), 72 deletions(-)
--
2.20.1
^ permalink raw reply
* [PATCH net-next v6 01/11] fib_frontend, ip6_fib: Select routes or exceptions dump from RTM_F_CLONED
From: Stefano Brivio @ 2019-06-19 23:59 UTC (permalink / raw)
To: David Miller
Cc: Jianlin Shi, Wei Wang, David Ahern, Martin KaFai Lau,
Eric Dumazet, Matti Vaittinen, netdev
In-Reply-To: <cover.1560987611.git.sbrivio@redhat.com>
The following patches add back the ability to dump IPv4 and IPv6 exception
routes, and we need to allow selection of regular routes or exceptions.
Use RTM_F_CLONED as filter to decide whether to dump routes or exceptions:
iproute2 passes it in dump requests (except for IPv6 cache flush requests,
this will be fixed in iproute2) and this used to work as long as
exceptions were stored directly in the FIB, for both IPv4 and IPv6.
Caveat: if strict checking is not requested (that is, if the dump request
doesn't go through ip_valid_fib_dump_req()), we can't filter on protocol,
tables or route types.
In this case, filtering on RTM_F_CLONED would be inconsistent: we would
fix 'ip route list cache' by returning exception routes and at the same
time introduce another bug in case another selector is present, e.g. on
'ip route list cache table main' we would return all exception routes,
without filtering on tables.
Keep this consistent by applying no filters at all, and dumping both
routes and exceptions, if strict checking is not requested. iproute2
currently filters results anyway, and no unwanted results will be
presented to the user. The kernel will just dump more data than needed.
v6: Rebase onto net-next, no changes
v5: New patch: add dump_routes and dump_exceptions flags in filter and
simply clear the unwanted one if strict checking is enabled, don't
ignore NLM_F_MATCH and don't set filter_set if NLM_F_MATCH is set.
Skip filtering altogether if no strict checking is requested:
selecting routes or exceptions only would be inconsistent with the
fact we can't filter on tables.
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
Reviewed-by: David Ahern <dsahern@gmail.com>
---
include/net/ip_fib.h | 2 ++
net/ipv4/fib_frontend.c | 8 +++++++-
net/ipv6/ip6_fib.c | 3 ++-
3 files changed, 11 insertions(+), 2 deletions(-)
diff --git a/include/net/ip_fib.h b/include/net/ip_fib.h
index 7e1e621a56df..4c81846ccce8 100644
--- a/include/net/ip_fib.h
+++ b/include/net/ip_fib.h
@@ -245,6 +245,8 @@ struct fib_dump_filter {
/* filter_set is an optimization that an entry is set */
bool filter_set;
bool dump_all_families;
+ bool dump_routes;
+ bool dump_exceptions;
unsigned char protocol;
unsigned char rt_type;
unsigned int flags;
diff --git a/net/ipv4/fib_frontend.c b/net/ipv4/fib_frontend.c
index 108191667531..ed7fb5fd885c 100644
--- a/net/ipv4/fib_frontend.c
+++ b/net/ipv4/fib_frontend.c
@@ -912,10 +912,15 @@ int ip_valid_fib_dump_req(struct net *net, const struct nlmsghdr *nlh,
NL_SET_ERR_MSG(extack, "Invalid values in header for FIB dump request");
return -EINVAL;
}
+
if (rtm->rtm_flags & ~(RTM_F_CLONED | RTM_F_PREFIX)) {
NL_SET_ERR_MSG(extack, "Invalid flags for FIB dump request");
return -EINVAL;
}
+ if (rtm->rtm_flags & RTM_F_CLONED)
+ filter->dump_routes = false;
+ else
+ filter->dump_exceptions = false;
filter->dump_all_families = (rtm->rtm_family == AF_UNSPEC);
filter->flags = rtm->rtm_flags;
@@ -962,9 +967,10 @@ EXPORT_SYMBOL_GPL(ip_valid_fib_dump_req);
static int inet_dump_fib(struct sk_buff *skb, struct netlink_callback *cb)
{
+ struct fib_dump_filter filter = { .dump_routes = true,
+ .dump_exceptions = true };
const struct nlmsghdr *nlh = cb->nlh;
struct net *net = sock_net(skb->sk);
- struct fib_dump_filter filter = {};
unsigned int h, s_h;
unsigned int e = 0, s_e;
struct fib_table *tb;
diff --git a/net/ipv6/ip6_fib.c b/net/ipv6/ip6_fib.c
index 1d16a01eccf5..f2a3cc4c8f12 100644
--- a/net/ipv6/ip6_fib.c
+++ b/net/ipv6/ip6_fib.c
@@ -552,9 +552,10 @@ static int fib6_dump_table(struct fib6_table *table, struct sk_buff *skb,
static int inet6_dump_fib(struct sk_buff *skb, struct netlink_callback *cb)
{
+ struct rt6_rtnl_dump_arg arg = { .filter.dump_exceptions = true,
+ .filter.dump_routes = true };
const struct nlmsghdr *nlh = cb->nlh;
struct net *net = sock_net(skb->sk);
- struct rt6_rtnl_dump_arg arg = {};
unsigned int h, s_h;
unsigned int e = 0, s_e;
struct fib6_walker *w;
--
2.20.1
^ permalink raw reply related
* [PATCH net-next v6 02/11] ipv4/fib_frontend: Allow RTM_F_CLONED flag to be used for filtering
From: Stefano Brivio @ 2019-06-19 23:59 UTC (permalink / raw)
To: David Miller
Cc: Jianlin Shi, Wei Wang, David Ahern, Martin KaFai Lau,
Eric Dumazet, Matti Vaittinen, netdev
In-Reply-To: <cover.1560987611.git.sbrivio@redhat.com>
This functionally reverts the check introduced by commit
e8ba330ac0c5 ("rtnetlink: Update fib dumps for strict data checking")
as modified by commit e4e92fb160d7 ("net/ipv4: Bail early if user only
wants prefix entries").
As we are preparing to fix listing of IPv4 cached routes, we need to
give userspace a way to request them.
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
Reviewed-by: David Ahern <dsahern@gmail.com>
---
v6: Rebase onto net-next, no changes
v5: No changes
v4: New patch
net/ipv4/fib_frontend.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/net/ipv4/fib_frontend.c b/net/ipv4/fib_frontend.c
index ed7fb5fd885c..317339cd7f03 100644
--- a/net/ipv4/fib_frontend.c
+++ b/net/ipv4/fib_frontend.c
@@ -987,8 +987,8 @@ static int inet_dump_fib(struct sk_buff *skb, struct netlink_callback *cb)
filter.flags = rtm->rtm_flags & (RTM_F_PREFIX | RTM_F_CLONED);
}
- /* fib entries are never clones and ipv4 does not use prefix flag */
- if (filter.flags & (RTM_F_PREFIX | RTM_F_CLONED))
+ /* ipv4 does not use prefix flag */
+ if (filter.flags & RTM_F_PREFIX)
return skb->len;
if (filter.table_id) {
--
2.20.1
^ permalink raw reply related
* [PATCH net-next v6 03/11] ipv4/route: Allow NULL flowinfo in rt_fill_info()
From: Stefano Brivio @ 2019-06-19 23:59 UTC (permalink / raw)
To: David Miller
Cc: Jianlin Shi, Wei Wang, David Ahern, Martin KaFai Lau,
Eric Dumazet, Matti Vaittinen, netdev
In-Reply-To: <cover.1560987611.git.sbrivio@redhat.com>
In the next patch, we're going to use rt_fill_info() to dump exception
routes upon RTM_GETROUTE with NLM_F_ROOT, meaning userspace is requesting
a dump and not a specific route selection, which in turn implies the input
interface is not relevant. Update rt_fill_info() to handle a NULL
flowinfo.
Suggested-by: David Ahern <dsahern@gmail.com>
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
---
v6: New patch
net/ipv4/route.c | 57 ++++++++++++++++++++++++++----------------------
1 file changed, 31 insertions(+), 26 deletions(-)
diff --git a/net/ipv4/route.c b/net/ipv4/route.c
index 66cbe8a7a168..052a80373b1d 100644
--- a/net/ipv4/route.c
+++ b/net/ipv4/route.c
@@ -2699,7 +2699,8 @@ static int rt_fill_info(struct net *net, __be32 dst, __be32 src,
r->rtm_family = AF_INET;
r->rtm_dst_len = 32;
r->rtm_src_len = 0;
- r->rtm_tos = fl4->flowi4_tos;
+ if (fl4)
+ r->rtm_tos = fl4->flowi4_tos;
r->rtm_table = table_id < 256 ? table_id : RT_TABLE_COMPAT;
if (nla_put_u32(skb, RTA_TABLE, table_id))
goto nla_put_failure;
@@ -2727,7 +2728,7 @@ static int rt_fill_info(struct net *net, __be32 dst, __be32 src,
nla_put_u32(skb, RTA_FLOW, rt->dst.tclassid))
goto nla_put_failure;
#endif
- if (!rt_is_input_route(rt) &&
+ if (fl4 && !rt_is_input_route(rt) &&
fl4->saddr != src) {
if (nla_put_in_addr(skb, RTA_PREFSRC, fl4->saddr))
goto nla_put_failure;
@@ -2767,36 +2768,40 @@ static int rt_fill_info(struct net *net, __be32 dst, __be32 src,
if (rtnetlink_put_metrics(skb, metrics) < 0)
goto nla_put_failure;
- if (fl4->flowi4_mark &&
- nla_put_u32(skb, RTA_MARK, fl4->flowi4_mark))
- goto nla_put_failure;
-
- if (!uid_eq(fl4->flowi4_uid, INVALID_UID) &&
- nla_put_u32(skb, RTA_UID,
- from_kuid_munged(current_user_ns(), fl4->flowi4_uid)))
- goto nla_put_failure;
+ if (fl4) {
+ if (fl4->flowi4_mark &&
+ nla_put_u32(skb, RTA_MARK, fl4->flowi4_mark))
+ goto nla_put_failure;
- error = rt->dst.error;
+ if (!uid_eq(fl4->flowi4_uid, INVALID_UID) &&
+ nla_put_u32(skb, RTA_UID,
+ from_kuid_munged(current_user_ns(),
+ fl4->flowi4_uid)))
+ goto nla_put_failure;
- if (rt_is_input_route(rt)) {
+ if (rt_is_input_route(rt)) {
#ifdef CONFIG_IP_MROUTE
- if (ipv4_is_multicast(dst) && !ipv4_is_local_multicast(dst) &&
- IPV4_DEVCONF_ALL(net, MC_FORWARDING)) {
- int err = ipmr_get_route(net, skb,
- fl4->saddr, fl4->daddr,
- r, portid);
-
- if (err <= 0) {
- if (err == 0)
- return 0;
- goto nla_put_failure;
- }
- } else
+ if (ipv4_is_multicast(dst) &&
+ !ipv4_is_local_multicast(dst) &&
+ IPV4_DEVCONF_ALL(net, MC_FORWARDING)) {
+ int err = ipmr_get_route(net, skb,
+ fl4->saddr, fl4->daddr,
+ r, portid);
+
+ if (err <= 0) {
+ if (err == 0)
+ return 0;
+ goto nla_put_failure;
+ }
+ } else
#endif
- if (nla_put_u32(skb, RTA_IIF, fl4->flowi4_iif))
- goto nla_put_failure;
+ if (nla_put_u32(skb, RTA_IIF, fl4->flowi4_iif))
+ goto nla_put_failure;
+ }
}
+ error = rt->dst.error;
+
if (rtnl_put_cacheinfo(skb, &rt->dst, 0, expires, error) < 0)
goto nla_put_failure;
--
2.20.1
^ permalink raw reply related
* [PATCH net-next v6 04/11] ipv4: Dump route exceptions if requested
From: Stefano Brivio @ 2019-06-19 23:59 UTC (permalink / raw)
To: David Miller
Cc: Jianlin Shi, Wei Wang, David Ahern, Martin KaFai Lau,
Eric Dumazet, Matti Vaittinen, netdev
In-Reply-To: <cover.1560987611.git.sbrivio@redhat.com>
Since commit 4895c771c7f0 ("ipv4: Add FIB nexthop exceptions."), cached
exception routes are stored as a separate entity, so they are not dumped
on a FIB dump, even if the RTM_F_CLONED flag is passed.
This implies that the command 'ip route list cache' doesn't return any
result anymore.
If the RTM_F_CLONED is passed, and strict checking requested, retrieve
nexthop exception routes and dump them. If no strict checking is
requested, filtering can't be performed consistently: dump everything in
that case.
With this, we need to add an argument to the netlink callback in order to
track how many entries were already dumped for the last leaf included in
a partial netlink dump.
Note that this is only as accurate as the existing tracking mechanism for
leaves: if a partial dump is restarted after exceptions are removed or
expired, we might skip some non-dumped entries. To improve this, we could
attach a 'sernum' attribute (similar to the one used for IPv6) to nexthop
entities, and bump this counter whenever exceptions change.
Listing of exception routes (modified routes pre-3.5) was tested against
these versions of kernel and iproute2:
iproute2
kernel 4.14.0 4.15.0 4.19.0 5.0.0 5.1.0
3.5-rc4 + + + + +
4.4
4.9
4.14
4.15
4.19
5.0
5.1
fixed + + + + +
v6:
- Rebased onto net-next
- Loop over nexthop paths too. Move loop over fnhe buckets to route.c,
avoids need to export rt_fill_info() and to touch exceptions from
fib_trie.c. Pass NULL as flow to rt_fill_info(), it now allows that
(all suggested by David Ahern)
Fixes: 4895c771c7f0 ("ipv4: Add FIB nexthop exceptions.")
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
---
include/net/route.h | 4 +++
net/ipv4/fib_trie.c | 60 ++++++++++++++++++++++++++++++++++++---------
net/ipv4/route.c | 56 ++++++++++++++++++++++++++++++++++++++++++
3 files changed, 108 insertions(+), 12 deletions(-)
diff --git a/include/net/route.h b/include/net/route.h
index 065b47754f05..e7f65388a6d4 100644
--- a/include/net/route.h
+++ b/include/net/route.h
@@ -44,6 +44,7 @@
#define RT_CONN_FLAGS_TOS(sk,tos) (RT_TOS(tos) | sock_flag(sk, SOCK_LOCALROUTE))
struct fib_nh;
+struct fib_alias;
struct fib_info;
struct uncached_list;
struct rtable {
@@ -230,6 +231,9 @@ void fib_modify_prefix_metric(struct in_ifaddr *ifa, u32 new_metric);
void rt_add_uncached_list(struct rtable *rt);
void rt_del_uncached_list(struct rtable *rt);
+int fnhe_dump_buckets(struct fib_alias *fa, int nhsel, struct sk_buff *skb,
+ struct netlink_callback *cb, int *fa_index, int fa_start);
+
static inline void ip_rt_put(struct rtable *rt)
{
/* dst_release() accepts a NULL parameter.
diff --git a/net/ipv4/fib_trie.c b/net/ipv4/fib_trie.c
index 94e5d83db4db..03f51e5192e5 100644
--- a/net/ipv4/fib_trie.c
+++ b/net/ipv4/fib_trie.c
@@ -2078,28 +2078,51 @@ void fib_free_table(struct fib_table *tb)
call_rcu(&tb->rcu, __trie_free_rcu);
}
+static int fib_dump_fnhe_from_leaf(struct fib_alias *fa, struct sk_buff *skb,
+ struct netlink_callback *cb,
+ int *fa_index, int fa_start)
+{
+ struct fib_info *fi = fa->fa_info;
+ int nhsel;
+
+ if (!fi || fi->fib_flags & RTNH_F_DEAD)
+ return 0;
+
+ for (nhsel = 0; nhsel < fib_info_num_path(fi); nhsel++) {
+ int err;
+
+ err = fnhe_dump_buckets(fa, nhsel, skb, cb, fa_index, fa_start);
+ if (err)
+ return err;
+ }
+
+ return 0;
+}
+
static int fn_trie_dump_leaf(struct key_vector *l, struct fib_table *tb,
struct sk_buff *skb, struct netlink_callback *cb,
struct fib_dump_filter *filter)
{
unsigned int flags = NLM_F_MULTI;
__be32 xkey = htonl(l->key);
+ int i, s_i, i_fa, s_fa, err;
struct fib_alias *fa;
- int i, s_i;
- if (filter->filter_set)
+ if (filter->filter_set ||
+ !filter->dump_exceptions || !filter->dump_routes)
flags |= NLM_F_DUMP_FILTERED;
s_i = cb->args[4];
+ s_fa = cb->args[5];
i = 0;
/* rcu_read_lock is hold by caller */
hlist_for_each_entry_rcu(fa, &l->leaf, fa_list) {
- int err;
-
if (i < s_i)
goto next;
+ i_fa = 0;
+
if (tb->tb_id != fa->tb_id)
goto next;
@@ -2116,21 +2139,34 @@ static int fn_trie_dump_leaf(struct key_vector *l, struct fib_table *tb,
goto next;
}
- err = fib_dump_info(skb, NETLINK_CB(cb->skb).portid,
- cb->nlh->nlmsg_seq, RTM_NEWROUTE,
- tb->tb_id, fa->fa_type,
- xkey, KEYLENGTH - fa->fa_slen,
- fa->fa_tos, fa->fa_info, flags);
- if (err < 0) {
- cb->args[4] = i;
- return err;
+ if (filter->dump_routes && !s_fa) {
+ err = fib_dump_info(skb, NETLINK_CB(cb->skb).portid,
+ cb->nlh->nlmsg_seq, RTM_NEWROUTE,
+ tb->tb_id, fa->fa_type,
+ xkey, KEYLENGTH - fa->fa_slen,
+ fa->fa_tos, fa->fa_info, flags);
+ if (err < 0)
+ goto stop;
+ i_fa++;
}
+
+ if (filter->dump_exceptions) {
+ err = fib_dump_fnhe_from_leaf(fa, skb, cb, &i_fa, s_fa);
+ if (err < 0)
+ goto stop;
+ }
+
next:
i++;
}
cb->args[4] = i;
return skb->len;
+
+stop:
+ cb->args[4] = i;
+ cb->args[5] = i_fa;
+ return err;
}
/* rcu_read_lock needs to be hold by caller from readside */
diff --git a/net/ipv4/route.c b/net/ipv4/route.c
index 052a80373b1d..b3961a135dfc 100644
--- a/net/ipv4/route.c
+++ b/net/ipv4/route.c
@@ -2813,6 +2813,62 @@ static int rt_fill_info(struct net *net, __be32 dst, __be32 src,
return -EMSGSIZE;
}
+int fnhe_dump_buckets(struct fib_alias *fa, int nhsel, struct sk_buff *skb,
+ struct netlink_callback *cb, int *fa_index, int fa_start)
+{
+ struct net *net = sock_net(cb->skb->sk);
+ struct fnhe_hash_bucket *bucket;
+ struct fib_nh_common *nhc;
+ int i, genid;
+
+ nhc = fib_info_nhc(fa->fa_info, nhsel);
+ if (nhc->nhc_flags & RTNH_F_DEAD)
+ return 0;
+
+ bucket = rcu_dereference(nhc->nhc_exceptions);
+ if (!bucket)
+ return 0;
+
+ genid = fnhe_genid(net);
+
+ for (i = 0; i < FNHE_HASH_SIZE; i++) {
+ struct fib_nh_exception *fnhe;
+
+ for (fnhe = rcu_dereference(bucket[i].chain); fnhe;
+ fnhe = rcu_dereference(fnhe->fnhe_next)) {
+ struct rtable *rt;
+ int err;
+
+ if (*fa_index < fa_start)
+ goto next;
+
+ if (fnhe->fnhe_genid != genid)
+ goto next;
+
+ if (fnhe->fnhe_expires &&
+ time_after(jiffies, fnhe->fnhe_expires))
+ goto next;
+
+ rt = rcu_dereference(fnhe->fnhe_rth_input);
+ if (!rt)
+ rt = rcu_dereference(fnhe->fnhe_rth_output);
+ if (!rt)
+ goto next;
+
+ err = rt_fill_info(net, fnhe->fnhe_daddr, 0, rt,
+ fa->tb_id, NULL, skb,
+ NETLINK_CB(cb->skb).portid,
+ cb->nlh->nlmsg_seq);
+ if (err)
+ return err;
+next:
+ (*fa_index)++;
+ }
+ }
+
+ return 0;
+}
+
static struct sk_buff *inet_rtm_getroute_build_skb(__be32 src, __be32 dst,
u8 ip_proto, __be16 sport,
__be16 dport)
--
2.20.1
^ permalink raw reply related
* [PATCH net-next v6 05/11] Revert "net/ipv6: Bail early if user only wants cloned entries"
From: Stefano Brivio @ 2019-06-19 23:59 UTC (permalink / raw)
To: David Miller
Cc: Jianlin Shi, Wei Wang, David Ahern, Martin KaFai Lau,
Eric Dumazet, Matti Vaittinen, netdev
In-Reply-To: <cover.1560987611.git.sbrivio@redhat.com>
This reverts commit 08e814c9e8eb5a982cbd1e8f6bd255d97c51026f: as we
are preparing to fix listing and dumping of IPv6 cached routes, we
need to allow RTM_F_CLONED as a flag to match routes against while
dumping them.
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
Reviewed-by: David Ahern <dsahern@gmail.com>
---
v6: No changes
v5: No changes
v4: New patch
net/ipv6/ip6_fib.c | 7 ++-----
1 file changed, 2 insertions(+), 5 deletions(-)
diff --git a/net/ipv6/ip6_fib.c b/net/ipv6/ip6_fib.c
index f2a3cc4c8f12..2baca6ef9e01 100644
--- a/net/ipv6/ip6_fib.c
+++ b/net/ipv6/ip6_fib.c
@@ -572,13 +572,10 @@ static int inet6_dump_fib(struct sk_buff *skb, struct netlink_callback *cb)
} else if (nlmsg_len(nlh) >= sizeof(struct rtmsg)) {
struct rtmsg *rtm = nlmsg_data(nlh);
- arg.filter.flags = rtm->rtm_flags & (RTM_F_PREFIX|RTM_F_CLONED);
+ if (rtm->rtm_flags & RTM_F_PREFIX)
+ arg.filter.flags = RTM_F_PREFIX;
}
- /* fib entries are never clones */
- if (arg.filter.flags & RTM_F_CLONED)
- goto out;
-
w = (void *)cb->args[2];
if (!w) {
/* New dump:
--
2.20.1
^ permalink raw reply related
* [PATCH net-next v6 06/11] ipv6/route: Don't match on fc_nh_id if not set in ip6_route_del()
From: Stefano Brivio @ 2019-06-19 23:59 UTC (permalink / raw)
To: David Miller
Cc: Jianlin Shi, Wei Wang, David Ahern, Martin KaFai Lau,
Eric Dumazet, Matti Vaittinen, netdev
In-Reply-To: <cover.1560987611.git.sbrivio@redhat.com>
If fc_nh_id isn't set, we shouldn't try to match against it. This
actually matters just for the RTF_CACHE case below (where this is
already handled): if iproute2 gets a route exception and tries to
delete it, it won't reference its fc_nh_id, even if a nexthop
object might be associated to the originating route.
Fixes: 5b98324ebe29 ("ipv6: Allow routes to use nexthop objects")
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
---
v6: New patch
net/ipv6/route.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index c4d285fe0adc..86859023cd01 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -3827,7 +3827,8 @@ static int ip6_route_del(struct fib6_config *cfg,
for_each_fib6_node_rt_rcu(fn) {
struct fib6_nh *nh;
- if (rt->nh && rt->nh->id != cfg->fc_nh_id)
+ if (rt->nh && cfg->fc_nh_id &&
+ rt->nh->id != cfg->fc_nh_id)
continue;
if (cfg->fc_flags & RTF_CACHE) {
--
2.20.1
^ permalink raw reply related
* [PATCH net-next v6 07/11] ipv6/route: Change return code of rt6_dump_route() for partial node dumps
From: Stefano Brivio @ 2019-06-19 23:59 UTC (permalink / raw)
To: David Miller
Cc: Jianlin Shi, Wei Wang, David Ahern, Martin KaFai Lau,
Eric Dumazet, Matti Vaittinen, netdev
In-Reply-To: <cover.1560987611.git.sbrivio@redhat.com>
In the next patch, we are going to add optional dump of exceptions to
rt6_dump_route().
Change the return code of rt6_dump_route() to accomodate partial node
dumps: we might dump multiple routes per node, and might be able to dump
only a given number of them, so fib6_dump_node() will need to know how
many routes have been dumped on partial dump, to restart the dump from the
point where it was interrupted.
Note that fib6_dump_node() is the only caller and already handles all
non-negative return codes as success: those become -1 to signal that we're
done with the node. If we fail, return 0, as we were unable to dump the
single route in the node, but we're not done with it.
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
---
v6: New patch
net/ipv6/ip6_fib.c | 2 +-
net/ipv6/route.c | 16 ++++++++++------
2 files changed, 11 insertions(+), 7 deletions(-)
diff --git a/net/ipv6/ip6_fib.c b/net/ipv6/ip6_fib.c
index 2baca6ef9e01..0e9a2ec69336 100644
--- a/net/ipv6/ip6_fib.c
+++ b/net/ipv6/ip6_fib.c
@@ -464,7 +464,7 @@ static int fib6_dump_node(struct fib6_walker *w)
for_each_fib6_walker_rt(w) {
res = rt6_dump_route(rt, w->args);
- if (res < 0) {
+ if (res >= 0) {
/* Frame is full, suspend walking */
w->leaf = rt;
return 1;
diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index 86859023cd01..1282f5a55b08 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -5503,6 +5503,7 @@ static bool fib6_info_uses_dev(const struct fib6_info *f6i,
return false;
}
+/* Return -1 if done with node, number of handled routes on partial dump */
int rt6_dump_route(struct fib6_info *rt, void *p_arg)
{
struct rt6_rtnl_dump_arg *arg = (struct rt6_rtnl_dump_arg *) p_arg;
@@ -5511,25 +5512,28 @@ int rt6_dump_route(struct fib6_info *rt, void *p_arg)
struct net *net = arg->net;
if (rt == net->ipv6.fib6_null_entry)
- return 0;
+ return -1;
if ((filter->flags & RTM_F_PREFIX) &&
!(rt->fib6_flags & RTF_PREFIX_RT)) {
/* success since this is not a prefix route */
- return 1;
+ return -1;
}
if (filter->filter_set) {
if ((filter->rt_type && rt->fib6_type != filter->rt_type) ||
(filter->dev && !fib6_info_uses_dev(rt, filter->dev)) ||
(filter->protocol && rt->fib6_protocol != filter->protocol)) {
- return 1;
+ return -1;
}
flags |= NLM_F_DUMP_FILTERED;
}
- return rt6_fill_node(net, arg->skb, rt, NULL, NULL, NULL, 0,
- RTM_NEWROUTE, NETLINK_CB(arg->cb->skb).portid,
- arg->cb->nlh->nlmsg_seq, flags);
+ if (rt6_fill_node(net, arg->skb, rt, NULL, NULL, NULL, 0, RTM_NEWROUTE,
+ NETLINK_CB(arg->cb->skb).portid,
+ arg->cb->nlh->nlmsg_seq, flags))
+ return 0;
+
+ return -1;
}
static int inet6_rtm_valid_getroute_req(struct sk_buff *skb,
--
2.20.1
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox