* Re: [PATCH 5.10 104/157] mptcp: put subflow sock on connect error
[not found] ` <20210322121937.071435221@linuxfoundation.org>
@ 2021-03-24 8:32 ` Naresh Kamboju
2021-03-24 9:04 ` Florian Westphal
2021-03-24 9:26 ` Greg Kroah-Hartman
0 siblings, 2 replies; 5+ messages in thread
From: Naresh Kamboju @ 2021-03-24 8:32 UTC (permalink / raw)
To: Greg Kroah-Hartman, Florian Westphal, David S. Miller
Cc: open list, linux-stable, Mat Martineau, Sasha Levin,
Anders Roxell, lkft-triage, Netdev, Arnd Bergmann
On Mon, 22 Mar 2021 at 18:15, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
>
> From: Florian Westphal <fw@strlen.de>
>
> [ Upstream commit f07157792c633b528de5fc1dbe2e4ea54f8e09d4 ]
>
> mptcp_add_pending_subflow() performs a sock_hold() on the subflow,
> then adds the subflow to the join list.
>
> Without a sock_put the subflow sk won't be freed in case connect() fails.
>
> unreferenced object 0xffff88810c03b100 (size 3000):
> [..]
> sk_prot_alloc.isra.0+0x2f/0x110
> sk_alloc+0x5d/0xc20
> inet6_create+0x2b7/0xd30
> __sock_create+0x17f/0x410
> mptcp_subflow_create_socket+0xff/0x9c0
> __mptcp_subflow_connect+0x1da/0xaf0
> mptcp_pm_nl_work+0x6e0/0x1120
> mptcp_worker+0x508/0x9a0
>
> Fixes: 5b950ff4331ddda ("mptcp: link MPC subflow into msk only after accept")
> Signed-off-by: Florian Westphal <fw@strlen.de>
> Signed-off-by: Mat Martineau <mathew.j.martineau@linux.intel.com>
> Signed-off-by: David S. Miller <davem@davemloft.net>
> Signed-off-by: Sasha Levin <sashal@kernel.org>
I have reported the following warnings and kernel crash on 5.10.26-rc2 [1]
The bisect reported that issue pointing out to this commit.
commit 460916534896e6d4f80a37152e0948db33376873
mptcp: put subflow sock on connect error
This problem is specific to 5.10.26-rc2.
Warning:
--------
[ 1040.114695] refcount_t: addition on 0; use-after-free.
[ 1040.119857] WARNING: CPU: 3 PID: 31925 at
/usr/src/kernel/lib/refcount.c:25 refcount_warn_saturate+0xd7/0x100
[ 1040.129769] Modules linked in: act_mirred cls_u32 sch_netem sch_etf
ip6table_nat xt_nat iptable_nat nf_nat ip6table_filter xt_conntrack
nf_conntrack nf_defrag_ipv4 libcrc32c ip6_tables nf_defrag_ipv6 sch_fq
iptable_filter xt_mark ip_tables cls_bpf sch_ingress algif_hash
x86_pkg_temp_thermal fuse [last unloaded: test_blackhole_dev]
[ 1040.159030] CPU: 3 PID: 31925 Comm: mptcp_connect Tainted: G
W K 5.10.26-rc2 #1
[ 1040.167459] Hardware name: Supermicro SYS-5019S-ML/X11SSH-F, BIOS
2.2 05/23/2018
[ 1040.174851] RIP: 0010:refcount_warn_saturate+0xd7/0x100
And
Kernel Panic:
-------------
[ 1069.557485] BUG: kernel NULL pointer dereference, address: 0000000000000010
[ 1069.564446] #PF: supervisor read access in kernel mode
[ 1069.569583] #PF: error_code(0x0000) - not-present page
[ 1069.574714] PGD 0 P4D 0
[ 1069.577246] Oops: 0000 [#1] SMP PTI
[ 1069.580730] CPU: 1 PID: 17 Comm: ksoftirqd/1 Tainted: G W
K 5.10.26-rc2 #1
[ 1069.588719] Hardware name: Supermicro SYS-5019S-ML/X11SSH-F, BIOS
2.2 05/23/2018
[ 1069.596106] RIP: 0010:selinux_socket_sock_rcv_skb+0x3f/0x290
...
[ 1069.961697] Kernel panic - not syncing: Fatal exception in interrupt
[ 1069.968083] Kernel Offset: 0x18600000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)
steps to reproduce:
--------------------------
- cd /opt/kselftests/mainline/net/mptcp
- ./mptcp_join.sh || true
Reported-by: Naresh Kamboju <naresh.kamboju@linaro.org>
crash test link:
https://lkft.validation.linaro.org/scheduler/job/2436164
Revert this commit and test job:
https://lkft.validation.linaro.org/scheduler/job/2437401#L1207
> ---
> net/mptcp/subflow.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/net/mptcp/subflow.c b/net/mptcp/subflow.c
> index 16adba172fb9..591546d0953f 100644
> --- a/net/mptcp/subflow.c
> +++ b/net/mptcp/subflow.c
> @@ -1133,6 +1133,7 @@ int __mptcp_subflow_connect(struct sock *sk, const struct mptcp_addr_info *loc,
> spin_lock_bh(&msk->join_list_lock);
> list_add_tail(&subflow->node, &msk->join_list);
> spin_unlock_bh(&msk->join_list_lock);
> + sock_put(mptcp_subflow_tcp_sock(subflow));
>
> return err;
>
url:
[1] https://lore.kernel.org/stable/20210323182123.3ce89282@yaviniv.e18.physik.tu-muenchen.de/T/#m7994b86b52391a746e7d5be214885a5a1b2f9713
--
Linaro LKFT
https://lkft.linaro.org
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 5.10 104/157] mptcp: put subflow sock on connect error
2021-03-24 8:32 ` [PATCH 5.10 104/157] mptcp: put subflow sock on connect error Naresh Kamboju
@ 2021-03-24 9:04 ` Florian Westphal
2021-03-24 9:22 ` Greg Kroah-Hartman
2021-03-24 9:26 ` Greg Kroah-Hartman
1 sibling, 1 reply; 5+ messages in thread
From: Florian Westphal @ 2021-03-24 9:04 UTC (permalink / raw)
To: Naresh Kamboju
Cc: Greg Kroah-Hartman, Florian Westphal, David S. Miller, open list,
linux-stable, Mat Martineau, Sasha Levin, Anders Roxell,
lkft-triage, Netdev, Arnd Bergmann
Naresh Kamboju <naresh.kamboju@linaro.org> wrote:
> On Mon, 22 Mar 2021 at 18:15, Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> >
> > From: Florian Westphal <fw@strlen.de>
> >
> > [ Upstream commit f07157792c633b528de5fc1dbe2e4ea54f8e09d4 ]
> >
> > mptcp_add_pending_subflow() performs a sock_hold() on the subflow,
> > then adds the subflow to the join list.
> >
> > Without a sock_put the subflow sk won't be freed in case connect() fails.
> >
> > unreferenced object 0xffff88810c03b100 (size 3000):
> > [..]
> > sk_prot_alloc.isra.0+0x2f/0x110
> > sk_alloc+0x5d/0xc20
> > inet6_create+0x2b7/0xd30
> > __sock_create+0x17f/0x410
> > mptcp_subflow_create_socket+0xff/0x9c0
> > __mptcp_subflow_connect+0x1da/0xaf0
> > mptcp_pm_nl_work+0x6e0/0x1120
> > mptcp_worker+0x508/0x9a0
> >
> > Fixes: 5b950ff4331ddda ("mptcp: link MPC subflow into msk only after accept")
I don't see this change in 5.10, so why is this fix queued up?
> I have reported the following warnings and kernel crash on 5.10.26-rc2 [1]
> The bisect reported that issue pointing out to this commit.
>
> commit 460916534896e6d4f80a37152e0948db33376873
> mptcp: put subflow sock on connect error
>
> This problem is specific to 5.10.26-rc2.
>
> Warning:
> --------
> [ 1040.114695] refcount_t: addition on 0; use-after-free.
> [ 1040.119857] WARNING: CPU: 3 PID: 31925 at
> /usr/src/kernel/lib/refcount.c:25 refcount_warn_saturate+0xd7/0x100
> [ 1040.129769] Modules linked in: act_mirred cls_u32 sch_netem sch_etf
> ip6table_nat xt_nat iptable_nat nf_nat ip6table_filter xt_conntrack
> nf_conntrack nf_defrag_ipv4 libcrc32c ip6_tables nf_defrag_ipv6 sch_fq
> iptable_filter xt_mark ip_tables cls_bpf sch_ingress algif_hash
> x86_pkg_temp_thermal fuse [last unloaded: test_blackhole_dev]
> [ 1040.159030] CPU: 3 PID: 31925 Comm: mptcp_connect Tainted: G
> W K 5.10.26-rc2 #1
> [ 1040.167459] Hardware name: Supermicro SYS-5019S-ML/X11SSH-F, BIOS
> 2.2 05/23/2018
> [ 1040.174851] RIP: 0010:refcount_warn_saturate+0xd7/0x100
>
> And
>
> Kernel Panic:
> -------------
> [ 1069.557485] BUG: kernel NULL pointer dereference, address: 0000000000000010
> [ 1069.564446] #PF: supervisor read access in kernel mode
> [ 1069.569583] #PF: error_code(0x0000) - not-present page
> [ 1069.574714] PGD 0 P4D 0
> [ 1069.577246] Oops: 0000 [#1] SMP PTI
> > index 16adba172fb9..591546d0953f 100644
> > --- a/net/mptcp/subflow.c
> > +++ b/net/mptcp/subflow.c
> > @@ -1133,6 +1133,7 @@ int __mptcp_subflow_connect(struct sock *sk, const struct mptcp_addr_info *loc,
> > spin_lock_bh(&msk->join_list_lock);
> > list_add_tail(&subflow->node, &msk->join_list);
> > spin_unlock_bh(&msk->join_list_lock);
> > + sock_put(mptcp_subflow_tcp_sock(subflow));
> >
> > return err;
Crash is not surprising, the backport puts the socket in the 'success' path
(list_add_tail).
I don't see why this is in -stable, the faulty commit is not there?
The upstream patch is:
list_del(&subflow->node);
spin_unlock_bh(&msk->join_list_lock);
+ sock_put(mptcp_subflow_tcp_sock(subflow));
[ Note the 'list_del', this is in the error unwind path ]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 5.10 104/157] mptcp: put subflow sock on connect error
2021-03-24 9:04 ` Florian Westphal
@ 2021-03-24 9:22 ` Greg Kroah-Hartman
2021-03-24 14:28 ` Sasha Levin
0 siblings, 1 reply; 5+ messages in thread
From: Greg Kroah-Hartman @ 2021-03-24 9:22 UTC (permalink / raw)
To: Florian Westphal
Cc: Naresh Kamboju, David S. Miller, open list, linux-stable,
Mat Martineau, Sasha Levin, Anders Roxell, lkft-triage, Netdev,
Arnd Bergmann
On Wed, Mar 24, 2021 at 10:04:12AM +0100, Florian Westphal wrote:
> Naresh Kamboju <naresh.kamboju@linaro.org> wrote:
> > On Mon, 22 Mar 2021 at 18:15, Greg Kroah-Hartman
> > <gregkh@linuxfoundation.org> wrote:
> > >
> > > From: Florian Westphal <fw@strlen.de>
> > >
> > > [ Upstream commit f07157792c633b528de5fc1dbe2e4ea54f8e09d4 ]
> > >
> > > mptcp_add_pending_subflow() performs a sock_hold() on the subflow,
> > > then adds the subflow to the join list.
> > >
> > > Without a sock_put the subflow sk won't be freed in case connect() fails.
> > >
> > > unreferenced object 0xffff88810c03b100 (size 3000):
> > > [..]
> > > sk_prot_alloc.isra.0+0x2f/0x110
> > > sk_alloc+0x5d/0xc20
> > > inet6_create+0x2b7/0xd30
> > > __sock_create+0x17f/0x410
> > > mptcp_subflow_create_socket+0xff/0x9c0
> > > __mptcp_subflow_connect+0x1da/0xaf0
> > > mptcp_pm_nl_work+0x6e0/0x1120
> > > mptcp_worker+0x508/0x9a0
> > >
> > > Fixes: 5b950ff4331ddda ("mptcp: link MPC subflow into msk only after accept")
>
> I don't see this change in 5.10, so why is this fix queued up?
>
> > I have reported the following warnings and kernel crash on 5.10.26-rc2 [1]
> > The bisect reported that issue pointing out to this commit.
> >
> > commit 460916534896e6d4f80a37152e0948db33376873
> > mptcp: put subflow sock on connect error
> >
> > This problem is specific to 5.10.26-rc2.
> >
> > Warning:
> > --------
> > [ 1040.114695] refcount_t: addition on 0; use-after-free.
> > [ 1040.119857] WARNING: CPU: 3 PID: 31925 at
> > /usr/src/kernel/lib/refcount.c:25 refcount_warn_saturate+0xd7/0x100
> > [ 1040.129769] Modules linked in: act_mirred cls_u32 sch_netem sch_etf
> > ip6table_nat xt_nat iptable_nat nf_nat ip6table_filter xt_conntrack
> > nf_conntrack nf_defrag_ipv4 libcrc32c ip6_tables nf_defrag_ipv6 sch_fq
> > iptable_filter xt_mark ip_tables cls_bpf sch_ingress algif_hash
> > x86_pkg_temp_thermal fuse [last unloaded: test_blackhole_dev]
> > [ 1040.159030] CPU: 3 PID: 31925 Comm: mptcp_connect Tainted: G
> > W K 5.10.26-rc2 #1
> > [ 1040.167459] Hardware name: Supermicro SYS-5019S-ML/X11SSH-F, BIOS
> > 2.2 05/23/2018
> > [ 1040.174851] RIP: 0010:refcount_warn_saturate+0xd7/0x100
> >
> > And
> >
> > Kernel Panic:
> > -------------
> > [ 1069.557485] BUG: kernel NULL pointer dereference, address: 0000000000000010
> > [ 1069.564446] #PF: supervisor read access in kernel mode
> > [ 1069.569583] #PF: error_code(0x0000) - not-present page
> > [ 1069.574714] PGD 0 P4D 0
> > [ 1069.577246] Oops: 0000 [#1] SMP PTI
> > > index 16adba172fb9..591546d0953f 100644
> > > --- a/net/mptcp/subflow.c
> > > +++ b/net/mptcp/subflow.c
> > > @@ -1133,6 +1133,7 @@ int __mptcp_subflow_connect(struct sock *sk, const struct mptcp_addr_info *loc,
> > > spin_lock_bh(&msk->join_list_lock);
> > > list_add_tail(&subflow->node, &msk->join_list);
> > > spin_unlock_bh(&msk->join_list_lock);
> > > + sock_put(mptcp_subflow_tcp_sock(subflow));
> > >
> > > return err;
>
> Crash is not surprising, the backport puts the socket in the 'success' path
> (list_add_tail).
>
> I don't see why this is in -stable, the faulty commit is not there?
>
> The upstream patch is:
> list_del(&subflow->node);
> spin_unlock_bh(&msk->join_list_lock);
> + sock_put(mptcp_subflow_tcp_sock(subflow));
>
> [ Note the 'list_del', this is in the error unwind path ]
Odd, I think something went wrong with Sasha's scripts.
I've dropped this, and the other two mptcp patches, from the 5.10 queue
and let's see if that helps. I'll do a new -rc now as well after my
build tests finish...
thanks,
greg k-h
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 5.10 104/157] mptcp: put subflow sock on connect error
2021-03-24 8:32 ` [PATCH 5.10 104/157] mptcp: put subflow sock on connect error Naresh Kamboju
2021-03-24 9:04 ` Florian Westphal
@ 2021-03-24 9:26 ` Greg Kroah-Hartman
1 sibling, 0 replies; 5+ messages in thread
From: Greg Kroah-Hartman @ 2021-03-24 9:26 UTC (permalink / raw)
To: Naresh Kamboju
Cc: Florian Westphal, David S. Miller, open list, linux-stable,
Mat Martineau, Sasha Levin, Anders Roxell, lkft-triage, Netdev,
Arnd Bergmann
On Wed, Mar 24, 2021 at 02:02:06PM +0530, Naresh Kamboju wrote:
> On Mon, 22 Mar 2021 at 18:15, Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> >
> > From: Florian Westphal <fw@strlen.de>
> >
> > [ Upstream commit f07157792c633b528de5fc1dbe2e4ea54f8e09d4 ]
> >
> > mptcp_add_pending_subflow() performs a sock_hold() on the subflow,
> > then adds the subflow to the join list.
> >
> > Without a sock_put the subflow sk won't be freed in case connect() fails.
> >
> > unreferenced object 0xffff88810c03b100 (size 3000):
> > [..]
> > sk_prot_alloc.isra.0+0x2f/0x110
> > sk_alloc+0x5d/0xc20
> > inet6_create+0x2b7/0xd30
> > __sock_create+0x17f/0x410
> > mptcp_subflow_create_socket+0xff/0x9c0
> > __mptcp_subflow_connect+0x1da/0xaf0
> > mptcp_pm_nl_work+0x6e0/0x1120
> > mptcp_worker+0x508/0x9a0
> >
> > Fixes: 5b950ff4331ddda ("mptcp: link MPC subflow into msk only after accept")
> > Signed-off-by: Florian Westphal <fw@strlen.de>
> > Signed-off-by: Mat Martineau <mathew.j.martineau@linux.intel.com>
> > Signed-off-by: David S. Miller <davem@davemloft.net>
> > Signed-off-by: Sasha Levin <sashal@kernel.org>
>
> I have reported the following warnings and kernel crash on 5.10.26-rc2 [1]
> The bisect reported that issue pointing out to this commit.
>
> commit 460916534896e6d4f80a37152e0948db33376873
> mptcp: put subflow sock on connect error
>
> This problem is specific to 5.10.26-rc2.
Thank you for tracking this down!
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 5.10 104/157] mptcp: put subflow sock on connect error
2021-03-24 9:22 ` Greg Kroah-Hartman
@ 2021-03-24 14:28 ` Sasha Levin
0 siblings, 0 replies; 5+ messages in thread
From: Sasha Levin @ 2021-03-24 14:28 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Florian Westphal, Naresh Kamboju, David S. Miller, open list,
linux-stable, Mat Martineau, Anders Roxell, lkft-triage, Netdev,
Arnd Bergmann
On Wed, Mar 24, 2021 at 10:22:29AM +0100, Greg Kroah-Hartman wrote:
>On Wed, Mar 24, 2021 at 10:04:12AM +0100, Florian Westphal wrote:
>> Naresh Kamboju <naresh.kamboju@linaro.org> wrote:
>> > On Mon, 22 Mar 2021 at 18:15, Greg Kroah-Hartman
>> > <gregkh@linuxfoundation.org> wrote:
>> > >
>> > > From: Florian Westphal <fw@strlen.de>
>> > >
>> > > [ Upstream commit f07157792c633b528de5fc1dbe2e4ea54f8e09d4 ]
>> > >
>> > > mptcp_add_pending_subflow() performs a sock_hold() on the subflow,
>> > > then adds the subflow to the join list.
>> > >
>> > > Without a sock_put the subflow sk won't be freed in case connect() fails.
>> > >
>> > > unreferenced object 0xffff88810c03b100 (size 3000):
>> > > [..]
>> > > sk_prot_alloc.isra.0+0x2f/0x110
>> > > sk_alloc+0x5d/0xc20
>> > > inet6_create+0x2b7/0xd30
>> > > __sock_create+0x17f/0x410
>> > > mptcp_subflow_create_socket+0xff/0x9c0
>> > > __mptcp_subflow_connect+0x1da/0xaf0
>> > > mptcp_pm_nl_work+0x6e0/0x1120
>> > > mptcp_worker+0x508/0x9a0
>> > >
>> > > Fixes: 5b950ff4331ddda ("mptcp: link MPC subflow into msk only after accept")
>>
>> I don't see this change in 5.10, so why is this fix queued up?
>>
>> > I have reported the following warnings and kernel crash on 5.10.26-rc2 [1]
>> > The bisect reported that issue pointing out to this commit.
>> >
>> > commit 460916534896e6d4f80a37152e0948db33376873
>> > mptcp: put subflow sock on connect error
>> >
>> > This problem is specific to 5.10.26-rc2.
>> >
>> > Warning:
>> > --------
>> > [ 1040.114695] refcount_t: addition on 0; use-after-free.
>> > [ 1040.119857] WARNING: CPU: 3 PID: 31925 at
>> > /usr/src/kernel/lib/refcount.c:25 refcount_warn_saturate+0xd7/0x100
>> > [ 1040.129769] Modules linked in: act_mirred cls_u32 sch_netem sch_etf
>> > ip6table_nat xt_nat iptable_nat nf_nat ip6table_filter xt_conntrack
>> > nf_conntrack nf_defrag_ipv4 libcrc32c ip6_tables nf_defrag_ipv6 sch_fq
>> > iptable_filter xt_mark ip_tables cls_bpf sch_ingress algif_hash
>> > x86_pkg_temp_thermal fuse [last unloaded: test_blackhole_dev]
>> > [ 1040.159030] CPU: 3 PID: 31925 Comm: mptcp_connect Tainted: G
>> > W K 5.10.26-rc2 #1
>> > [ 1040.167459] Hardware name: Supermicro SYS-5019S-ML/X11SSH-F, BIOS
>> > 2.2 05/23/2018
>> > [ 1040.174851] RIP: 0010:refcount_warn_saturate+0xd7/0x100
>> >
>> > And
>> >
>> > Kernel Panic:
>> > -------------
>> > [ 1069.557485] BUG: kernel NULL pointer dereference, address: 0000000000000010
>> > [ 1069.564446] #PF: supervisor read access in kernel mode
>> > [ 1069.569583] #PF: error_code(0x0000) - not-present page
>> > [ 1069.574714] PGD 0 P4D 0
>> > [ 1069.577246] Oops: 0000 [#1] SMP PTI
>> > > index 16adba172fb9..591546d0953f 100644
>> > > --- a/net/mptcp/subflow.c
>> > > +++ b/net/mptcp/subflow.c
>> > > @@ -1133,6 +1133,7 @@ int __mptcp_subflow_connect(struct sock *sk, const struct mptcp_addr_info *loc,
>> > > spin_lock_bh(&msk->join_list_lock);
>> > > list_add_tail(&subflow->node, &msk->join_list);
>> > > spin_unlock_bh(&msk->join_list_lock);
>> > > + sock_put(mptcp_subflow_tcp_sock(subflow));
>> > >
>> > > return err;
>>
>> Crash is not surprising, the backport puts the socket in the 'success' path
>> (list_add_tail).
>>
>> I don't see why this is in -stable, the faulty commit is not there?
>>
>> The upstream patch is:
>> list_del(&subflow->node);
>> spin_unlock_bh(&msk->join_list_lock);
>> + sock_put(mptcp_subflow_tcp_sock(subflow));
>>
>> [ Note the 'list_del', this is in the error unwind path ]
>
>Odd, I think something went wrong with Sasha's scripts.
I brought in the commit it depends on as a dependency for something
else, but ended up throwing it away, forgetting about this commit. Sorry
:(
--
Thanks,
Sasha
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-03-24 14:29 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20210322121933.746237845@linuxfoundation.org>
[not found] ` <20210322121937.071435221@linuxfoundation.org>
2021-03-24 8:32 ` [PATCH 5.10 104/157] mptcp: put subflow sock on connect error Naresh Kamboju
2021-03-24 9:04 ` Florian Westphal
2021-03-24 9:22 ` Greg Kroah-Hartman
2021-03-24 14:28 ` Sasha Levin
2021-03-24 9:26 ` Greg Kroah-Hartman
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).