From: Alexei Starovoitov <alexei.starovoitov@gmail.com>
To: Daniel Xu <dxu@dxuuu.xyz>
Cc: Andrii Nakryiko <andrii@kernel.org>,
Alexei Starovoitov <ast@kernel.org>,
Florian Westphal <fw@strlen.de>,
"David S. Miller" <davem@davemloft.net>,
Pablo Neira Ayuso <pablo@netfilter.org>,
Paolo Abeni <pabeni@redhat.com>,
Daniel Borkmann <daniel@iogearbox.net>,
Eric Dumazet <edumazet@google.com>,
Jakub Kicinski <kuba@kernel.org>,
Jozsef Kadlecsik <kadlec@netfilter.org>,
Martin KaFai Lau <martin.lau@linux.dev>,
Song Liu <song@kernel.org>, Yonghong Song <yhs@fb.com>,
John Fastabend <john.fastabend@gmail.com>,
KP Singh <kpsingh@kernel.org>,
Stanislav Fomichev <sdf@google.com>, Hao Luo <haoluo@google.com>,
Jiri Olsa <jolsa@kernel.org>, bpf <bpf@vger.kernel.org>,
LKML <linux-kernel@vger.kernel.org>,
netfilter-devel <netfilter-devel@vger.kernel.org>,
coreteam@netfilter.org,
Network Development <netdev@vger.kernel.org>,
David Ahern <dsahern@kernel.org>
Subject: Re: [PATCH bpf-next v4 2/6] netfilter: bpf: Support BPF_F_NETFILTER_IP_DEFRAG in netfilter link
Date: Thu, 13 Jul 2023 16:10:03 -0700 [thread overview]
Message-ID: <CAADnVQJQZ2jQSWByVvi3N2ZOoL0XDSJzx5biSVvq=inS7OSW7A@mail.gmail.com> (raw)
In-Reply-To: <wltfmammaf5g4gumsbna4kmwo6dtd24g472o7kgkug42dhwcy2@32fmd7q6kvg4>
On Wed, Jul 12, 2023 at 9:33 PM Daniel Xu <dxu@dxuuu.xyz> wrote:
>
> On Wed, Jul 12, 2023 at 06:26:13PM -0700, Alexei Starovoitov wrote:
> > On Wed, Jul 12, 2023 at 6:22 PM Daniel Xu <dxu@dxuuu.xyz> wrote:
> > >
> > > Hi Alexei,
> > >
> > > On Wed, Jul 12, 2023 at 05:43:49PM -0700, Alexei Starovoitov wrote:
> > > > On Wed, Jul 12, 2023 at 4:44 PM Daniel Xu <dxu@dxuuu.xyz> wrote:
> > > > > +#if IS_ENABLED(CONFIG_NF_DEFRAG_IPV6)
> > > > > + case NFPROTO_IPV6:
> > > > > + rcu_read_lock();
> > > > > + v6_hook = rcu_dereference(nf_defrag_v6_hook);
> > > > > + if (!v6_hook) {
> > > > > + rcu_read_unlock();
> > > > > + err = request_module("nf_defrag_ipv6");
> > > > > + if (err)
> > > > > + return err < 0 ? err : -EINVAL;
> > > > > +
> > > > > + rcu_read_lock();
> > > > > + v6_hook = rcu_dereference(nf_defrag_v6_hook);
> > > > > + if (!v6_hook) {
> > > > > + WARN_ONCE(1, "nf_defrag_ipv6_hooks bad registration");
> > > > > + err = -ENOENT;
> > > > > + goto out_v6;
> > > > > + }
> > > > > + }
> > > > > +
> > > > > + err = v6_hook->enable(link->net);
> > > >
> > > > I was about to apply, but luckily caught this issue in my local test:
> > > >
> > > > [ 18.462448] BUG: sleeping function called from invalid context at
> > > > kernel/locking/mutex.c:283
> > > > [ 18.463238] in_atomic(): 0, irqs_disabled(): 0, non_block: 0, pid:
> > > > 2042, name: test_progs
> > > > [ 18.463927] preempt_count: 0, expected: 0
> > > > [ 18.464249] RCU nest depth: 1, expected: 0
> > > > [ 18.464631] CPU: 15 PID: 2042 Comm: test_progs Tainted: G
> > > > O 6.4.0-04319-g6f6ec4fa00dc #4896
> > > > [ 18.465480] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
> > > > BIOS rel-1.12.0-59-gc9ba5276e321-prebuilt.qemu.org 04/01/2014
> > > > [ 18.466531] Call Trace:
> > > > [ 18.466767] <TASK>
> > > > [ 18.466975] dump_stack_lvl+0x32/0x40
> > > > [ 18.467325] __might_resched+0x129/0x180
> > > > [ 18.467691] mutex_lock+0x1a/0x40
> > > > [ 18.468057] nf_defrag_ipv4_enable+0x16/0x70
> > > > [ 18.468467] bpf_nf_link_attach+0x141/0x300
> > > > [ 18.468856] __sys_bpf+0x133e/0x26d0
> > > >
> > > > You cannot call mutex under rcu_read_lock.
> > >
> > > Whoops, my bad. I think this patch should fix it:
> > >
> > > ```
> > > From 7e8927c44452db07ddd7cf0e30bb49215fc044ed Mon Sep 17 00:00:00 2001
> > > Message-ID: <7e8927c44452db07ddd7cf0e30bb49215fc044ed.1689211250.git.dxu@dxuuu.xyz>
> > > From: Daniel Xu <dxu@dxuuu.xyz>
> > > Date: Wed, 12 Jul 2023 19:17:35 -0600
> > > Subject: [PATCH] netfilter: bpf: Don't hold rcu_read_lock during
> > > enable/disable
> > >
> > > ->enable()/->disable() takes a mutex which can sleep. You can't sleep
> > > during RCU read side critical section.
> > >
> > > Our refcnt on the module will protect us from ->enable()/->disable()
> > > from going away while we call it.
> > >
> > > Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
> > > ---
> > > net/netfilter/nf_bpf_link.c | 10 ++++++++--
> > > 1 file changed, 8 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/net/netfilter/nf_bpf_link.c b/net/netfilter/nf_bpf_link.c
> > > index 77ffbf26ba3d..79704cc596aa 100644
> > > --- a/net/netfilter/nf_bpf_link.c
> > > +++ b/net/netfilter/nf_bpf_link.c
> > > @@ -60,9 +60,12 @@ static int bpf_nf_enable_defrag(struct bpf_nf_link *link)
> > > goto out_v4;
> > > }
> > >
> > > + rcu_read_unlock();
> > > err = v4_hook->enable(link->net);
> > > if (err)
> > > module_put(v4_hook->owner);
> > > +
> > > + return err;
> > > out_v4:
> > > rcu_read_unlock();
> > > return err;
> > > @@ -92,9 +95,12 @@ static int bpf_nf_enable_defrag(struct bpf_nf_link *link)
> > > goto out_v6;
> > > }
> > >
> > > + rcu_read_unlock();
> > > err = v6_hook->enable(link->net);
> > > if (err)
> > > module_put(v6_hook->owner);
> > > +
> > > + return err;
> > > out_v6:
> > > rcu_read_unlock();
> > > return err;
> > > @@ -114,11 +120,11 @@ static void bpf_nf_disable_defrag(struct bpf_nf_link *link)
> > > case NFPROTO_IPV4:
> > > rcu_read_lock();
> > > v4_hook = rcu_dereference(nf_defrag_v4_hook);
> > > + rcu_read_unlock();
> > > if (v4_hook) {
> > > v4_hook->disable(link->net);
> > > module_put(v4_hook->owner);
> > > }
> > > - rcu_read_unlock();
> > >
> > > break;
> > > #endif
> > > @@ -126,11 +132,11 @@ static void bpf_nf_disable_defrag(struct bpf_nf_link *link)
> > > case NFPROTO_IPV6:
> > > rcu_read_lock();
> > > v6_hook = rcu_dereference(nf_defrag_v6_hook);
> > > + rcu_read_unlock();
> >
> > No. v6_hook is gone as soon as you unlock it.
>
> I think we're protected here by the try_module_get() on the enable path.
> And we only disable defrag if enabling succeeds. The module shouldn't
> be able to deregister its hooks until we call the module_put() later.
>
> I think READ_ONCE() would've been more appropriate but I wasn't sure if
> that was ok given nf_defrag_v(4|6)_hook is written to by
> rcu_assign_pointer() and I was assuming symmetry is necessary.
Why is rcu_assign_pointer() used?
If it's not RCU protected, what is the point of rcu_*() accessors
and rcu_read_lock() ?
In general, the pattern:
rcu_read_lock();
ptr = rcu_dereference(...);
rcu_read_unlock();
ptr->..
is a bug. 100%.
next prev parent reply other threads:[~2023-07-13 23:10 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-07-12 23:43 [PATCH bpf-next v4 0/6] Support defragmenting IPv(4|6) packets in BPF Daniel Xu
2023-07-12 23:43 ` [PATCH bpf-next v4 1/6] netfilter: defrag: Add glue hooks for enabling/disabling defrag Daniel Xu
2023-07-12 23:43 ` [PATCH bpf-next v4 2/6] netfilter: bpf: Support BPF_F_NETFILTER_IP_DEFRAG in netfilter link Daniel Xu
2023-07-13 0:43 ` Alexei Starovoitov
2023-07-13 1:22 ` Daniel Xu
2023-07-13 1:26 ` Alexei Starovoitov
2023-07-13 4:33 ` Daniel Xu
2023-07-13 23:10 ` Alexei Starovoitov [this message]
2023-07-13 23:42 ` Daniel Xu
2023-07-14 9:47 ` Florian Westphal
2023-07-18 21:45 ` Daniel Xu
2023-07-12 23:43 ` [PATCH bpf-next v4 3/6] netfilter: bpf: Prevent defrag module unload while link active Daniel Xu
2023-07-12 23:43 ` [PATCH bpf-next v4 4/6] bpf: selftests: Support not connecting client socket Daniel Xu
2023-07-12 23:44 ` [PATCH bpf-next v4 5/6] bpf: selftests: Support custom type and proto for client sockets Daniel Xu
2023-07-12 23:44 ` [PATCH bpf-next v4 6/6] bpf: selftests: Add defrag selftests Daniel Xu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAADnVQJQZ2jQSWByVvi3N2ZOoL0XDSJzx5biSVvq=inS7OSW7A@mail.gmail.com' \
--to=alexei.starovoitov@gmail.com \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=coreteam@netfilter.org \
--cc=daniel@iogearbox.net \
--cc=davem@davemloft.net \
--cc=dsahern@kernel.org \
--cc=dxu@dxuuu.xyz \
--cc=edumazet@google.com \
--cc=fw@strlen.de \
--cc=haoluo@google.com \
--cc=john.fastabend@gmail.com \
--cc=jolsa@kernel.org \
--cc=kadlec@netfilter.org \
--cc=kpsingh@kernel.org \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=martin.lau@linux.dev \
--cc=netdev@vger.kernel.org \
--cc=netfilter-devel@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=pablo@netfilter.org \
--cc=sdf@google.com \
--cc=song@kernel.org \
--cc=yhs@fb.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).