All of lore.kernel.org
 help / color / mirror / Atom feed
From: Daniel Borkmann <daniel@iogearbox.net>
To: Shahar Klein <shahark@mellanox.com>, netdev@vger.kernel.org
Cc: Roi Dayan <roid@mellanox.com>, David Miller <davem@davemloft.net>,
	Cong Wang <xiyou.wangcong@gmail.com>,
	Jiri Pirko <jiri@mellanox.com>,
	John Fastabend <john.fastabend@gmail.com>,
	Or Gerlitz <ogerlitz@mellanox.com>,
	Hadar Hen Zion <hadarh@mellanox.com>
Subject: Re: Soft lockup in tc_classify
Date: Mon, 12 Dec 2016 14:28:43 +0100	[thread overview]
Message-ID: <584EA60B.80803@iogearbox.net> (raw)
In-Reply-To: <7394f89e-e8a5-5fb2-ee04-63bf1c4ef6e7@mellanox.com>

Hi Shahar,

On 12/12/2016 10:43 AM, Shahar Klein wrote:
> Hi All,
>
> sorry for the spam, the first time was sent with html part and was rejected.
>
> We observed an issue where a classifier instance next member is pointing back to itself, causing a CPU soft lockup.
> We found it by running traffic on many udp connections and then adding a new flower rule using tc.
>
> We added a quick workaround to verify it:
>
> In tc_classify:
>
>          for (; tp; tp = rcu_dereference_bh(tp->next)) {
>                  int err;
> +               if (tp == tp->next)
> +                     RCU_INIT_POINTER(tp->next, NULL);
>
>
> We also had a print here showing tp->next is pointing to tp. With this workaround we are not hitting the issue anymore.
> We are not sure we fully understand the mechanism here - with the rtnl and rcu locks.
> We'll appreciate your help solving this issue.

Note that there's still the RCU fix missing for the deletion race that
Cong will still send out, but you say that the only thing you do is to
add a single rule, but no other operation in involved during that test?

Do you have a script and kernel .config for reproducing this?

Thanks,
Daniel

  reply	other threads:[~2016-12-12 13:28 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <c1c394d3-3aea-52a8-89e3-be57d4d46b8e@mellanox.com>
2016-12-12  9:43 ` Soft lockup in tc_classify Shahar Klein
2016-12-12 13:28   ` Daniel Borkmann [this message]
2016-12-12 16:04     ` Shahar Klein
2016-12-12 19:07       ` Cong Wang
2016-12-13 11:59         ` Shahar Klein
2016-12-12 21:18     ` Or Gerlitz
2016-12-12 22:51       ` Cong Wang
2016-12-19 16:39         ` Shahar Klein
2016-12-19 17:58           ` Cong Wang
2016-12-20  6:22             ` Shahar Klein
2016-12-20 11:47               ` Daniel Borkmann
2016-12-21  6:44                 ` Shahar Klein
2016-12-21  7:03                   ` Cong Wang
2016-12-21 10:15                     ` Daniel Borkmann
2016-12-21 12:58                       ` Shahar Klein
2016-12-21 13:18                         ` Daniel Borkmann
2016-12-21 11:25                     ` Shahar Klein

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=584EA60B.80803@iogearbox.net \
    --to=daniel@iogearbox.net \
    --cc=davem@davemloft.net \
    --cc=hadarh@mellanox.com \
    --cc=jiri@mellanox.com \
    --cc=john.fastabend@gmail.com \
    --cc=netdev@vger.kernel.org \
    --cc=ogerlitz@mellanox.com \
    --cc=roid@mellanox.com \
    --cc=shahark@mellanox.com \
    --cc=xiyou.wangcong@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.