From: David Miller <davem@davemloft.net>
To: ast@plumgrid.com
Cc: edumazet@google.com, netdev@vger.kernel.org
Subject: Re: [PATCH net] ipv4: fix race in concurrent ip_route_input_slow()
Date: Wed, 20 Nov 2013 15:29:18 -0500 (EST) [thread overview]
Message-ID: <20131120.152918.493706024928330813.davem@davemloft.net> (raw)
In-Reply-To: <1384917154-11049-1-git-send-email-ast@plumgrid.com>
From: Alexei Starovoitov <ast@plumgrid.com>
Date: Tue, 19 Nov 2013 19:12:34 -0800
> CPUs can ask for local route via ip_route_input_noref() concurrently.
> if nh_rth_input is not cached yet, CPUs will proceed to allocate
> equivalent DSTs on 'lo' and then will try to cache them in nh_rth_input
> via rt_cache_route()
> Most of the time they succeed, but on occasion the following two lines:
> orig = *p;
> prev = cmpxchg(p, orig, rt);
> in rt_cache_route() do race and one of the cpus fails to complete cmpxchg.
> But ip_route_input_slow() doesn't check the return code of rt_cache_route(),
> so dst is leaking. dst_destroy() is never called and 'lo' device
> refcnt doesn't go to zero, which can be seen in the logs as:
> unregister_netdevice: waiting for lo to become free. Usage count = 1
> Adding mdelay() between above two lines makes it easily reproducible.
> Fix it similar to nh_pcpu_rth_output case.
>
> Fixes: d2d68ba9fe8b ("ipv4: Cache input routes in fib_info nexthops.")
> Signed-off-by: Alexei Starovoitov <ast@plumgrid.com>
> ---
>
> David,
>
> looks like caacf05e5ad1 ("ipv4: Properly purge netdev references on uncached routes.")
> fixed the race for nexthop/rth_output, but missed it for rth_input.
> I'm not sure what was the assumption why it's not needed there.
> We're definitely seeing it every 12-24hr during nightly tests.
> There are several bugs on ubuntu and debian forums with similar description.
> Some were closed, since folks struggled to reproduce it.
> It took us more than a month to debug it.
> Please queue for stable.
Your analysis is accurate and your fix is absolutely correct, applied
and queued up for -stable, thanks!
prev parent reply other threads:[~2013-11-20 20:29 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-11-20 3:12 [PATCH net] ipv4: fix race in concurrent ip_route_input_slow() Alexei Starovoitov
2013-11-20 20:29 ` David Miller [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20131120.152918.493706024928330813.davem@davemloft.net \
--to=davem@davemloft.net \
--cc=ast@plumgrid.com \
--cc=edumazet@google.com \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).