From: Jiri Pirko <jiri@resnulli.us>
To: John Fastabend <john.fastabend@gmail.com>
Cc: Cong Wang <xiyou.wangcong@gmail.com>,
Jakub Kicinski <kubakici@wp.pl>,
"netdev@vger.kernel.org" <netdev@vger.kernel.org>
Subject: Re: RCU callback crashes
Date: Wed, 20 Dec 2017 21:18:41 +0100 [thread overview]
Message-ID: <20171220201841.GG1760@nanopsycho> (raw)
In-Reply-To: <d82f1fdf-5d7b-db1b-6efb-85ff957f8a38@gmail.com>
Wed, Dec 20, 2017 at 09:14:49PM CET, john.fastabend@gmail.com wrote:
>On 12/20/2017 11:59 AM, Jiri Pirko wrote:
>> Wed, Dec 20, 2017 at 07:17:50PM CET, xiyou.wangcong@gmail.com wrote:
>>> On Tue, Dec 19, 2017 at 10:34 PM, Jakub Kicinski <kubakici@wp.pl> wrote:
>>>> Ah, no object debug but KASAN on produces this:
>>>>
>>>
>>>
>>> I bet it is an ingress qdisc which is being freed?
>>>
>>>
>>>
>>>> [ 39.268209] BUG: KASAN: use-after-free in cpu_needs_another_gp+0x246/0x2b0
>>>> [ 39.275965] Read of size 8 at addr ffff8803aa64f138 by task swapper/13/0
>>>> [ 39.283524]
>>>> [ 39.285256] CPU: 13 PID: 0 Comm: swapper/13 Not tainted 4.15.0-rc3-perf-00955-g1d0b01347dd5-dirty #8
>>>> [ 39.295535] Hardware name: Dell Inc. PowerEdge R730/072T6D, BIOS 2.3.4 11/08/2016
>>>> [ 39.303969] Call Trace:
>>>> [ 39.306769] <IRQ>
>>>> [ 39.309088] dump_stack+0xa6/0x118
>>>> [ 39.312957] ? _atomic_dec_and_lock+0xe8/0xe8
>>>> [ 39.317895] ? cpu_needs_another_gp+0x246/0x2b0
>>>> [ 39.323030] print_address_description+0x6a/0x270
>>>> [ 39.328380] ? cpu_needs_another_gp+0x246/0x2b0
>>>> [ 39.333510] kasan_report+0x23f/0x350
>>>> [ 39.337672] cpu_needs_another_gp+0x246/0x2b0
>>>> ...
>>>> [ 39.383026] rcu_process_callbacks+0x1a0/0x620
>>>> ...
>>>
>>>
>>> This is confusing.
>>>
>>> I guess it is q->miniqp which is freed in qdisc_graft() without properly
>>> waiting for rcu readers?
>>
>> miniqp is inside qdisc private data:
>> struct ingress_sched_data {
>> struct tcf_block *block;
>> struct tcf_block_ext_info block_info;
>> struct mini_Qdisc_pair miniqp;
>> };
>>
>> That is freed along with the qdisc itself in:
>> qdisc_destroy->qdisc_free
>>
>> Before miniq, tp was checked in the rcu reader path. In case it was not
>> null, q was processed. In slow patch, tp is freed after rcu grace period:
>> tcf_proto_destroy->kfree_rcu
>>
>> I assumed that since q is processed in rcu reader, it is also freed after
>> a grace period, but now looking at the code I don't see it happening
>> like that.
>>
>> So I think that change to miniq made the existing race window
>> a bit wider and easier to hit.
>>
>> I believe that calling kfree_rcu by call_rcu should resolve this.
>>
>
>Hi,
>
>Just sent a patch to complete qdisc_destroy from rcu callback. This
>is needed to resolve a race with the lockless qdisc patches.
>
>But I guess it should fix the miniq issue as well?
Yes, it should.
next prev parent reply other threads:[~2017-12-20 20:18 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-12-20 1:59 RCU callback crashes Jakub Kicinski
2017-12-20 6:11 ` Jiri Pirko
2017-12-20 6:22 ` Jakub Kicinski
2017-12-20 6:34 ` Jakub Kicinski
2017-12-20 18:04 ` John Fastabend
2017-12-20 20:17 ` Jakub Kicinski
2017-12-20 20:23 ` John Fastabend
2017-12-20 22:38 ` Cong Wang
2017-12-20 18:17 ` Cong Wang
2017-12-20 18:31 ` Cong Wang
2017-12-21 0:03 ` Cong Wang
2017-12-21 0:08 ` Jakub Kicinski
2017-12-21 0:37 ` Jakub Kicinski
2017-12-21 0:41 ` Jakub Kicinski
2017-12-21 0:50 ` Jakub Kicinski
2017-12-21 7:27 ` Cong Wang
2017-12-21 16:26 ` John Fastabend
2017-12-21 16:56 ` Jakub Kicinski
2017-12-21 20:17 ` Cong Wang
2017-12-21 21:31 ` Cong Wang
2017-12-21 21:45 ` Jakub Kicinski
2017-12-21 7:24 ` Cong Wang
2017-12-20 19:59 ` Jiri Pirko
2017-12-20 20:14 ` John Fastabend
2017-12-20 20:18 ` Jiri Pirko [this message]
2017-12-20 22:25 ` Cong Wang
2017-12-20 20:15 ` Jiri Pirko
2017-12-20 20:18 ` John Fastabend
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20171220201841.GG1760@nanopsycho \
--to=jiri@resnulli.us \
--cc=john.fastabend@gmail.com \
--cc=kubakici@wp.pl \
--cc=netdev@vger.kernel.org \
--cc=xiyou.wangcong@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).