All of lore.kernel.org
 help / color / mirror / Atom feed
From: Vlad Buslov <vladbu@mellanox.com>
To: Vlad Buslov <vladbu@mellanox.com>
Cc: Pablo Neira Ayuso <pablo@netfilter.org>,
	"netdev\@vger.kernel.org" <netdev@vger.kernel.org>,
	Jiri Pirko <jiri@resnulli.us>,
	Saeed Mahameed <saeedm@mellanox.com>,
	Roi Dayan <roid@mellanox.com>, Majd Dibbiny <majd@mellanox.com>,
	Maor Dickman <maord@mellanox.com>, wenxu <wenxu@ucloud.cn>
Subject: Re: Crash in indirect block infra after unloading driver module
Date: Wed, 24 Jun 2020 15:37:20 +0300	[thread overview]
Message-ID: <vbfk0zwy70f.fsf@mellanox.com> (raw)
In-Reply-To: <vbfmu4s8xgt.fsf@mellanox.com>


On Wed 24 Jun 2020 at 15:22, Vlad Buslov <vladbu@mellanox.com> wrote:
> On Wed 24 Jun 2020 at 13:30, Pablo Neira Ayuso <pablo@netfilter.org> wrote:
>> On Wed, Jun 24, 2020 at 01:22:29PM +0300, Vlad Buslov wrote:
>>> Hi Pablo,
>>> 
>>> I've encountered a new issue with indirect offloads infrastructure. The
>>> issue is that on driver offload its indirect callbacks are not removed
>>> from blocks and any following offloads operations on block that has such
>>> callback in its offloads cb list causes call to unmapped address.
>>> 
>>> Steps to reproduce:
>>> 
>>> echo 1 >/sys/class/net/ens1f0/device/sriov_numvfs
>>> echo 0000:81:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind
>>> devlink dev eswitch set pci/0000:81:00.0 mode switchdev
>>> 
>>> ip link add vxlan1 type vxlan dstport 4789 external
>>> ip addr add 192.168.1.1 dev ens1f0
>>> link set up dev ens1f0
>>> ip link set up dev ens1f0
>>> tc qdisc add dev vxlan1 ingress
>>> tc filter add dev vxlan1 protocol ip ingress flower enc_src_ip 192.168.1.2 enc_dst_ip 192.168.1.1 enc_key_id 42 enc_dst_port 4789 action tunnel_key unset action mirred egress redirect dev ens1f0_0
>>> tc -s filter show dev vxlan1 ingress
>>> 
>>> rmmod mlx5_ib
>>> rmmod mlx5_core
>>> tc -s filter show dev vxlan1 ingress
>>
>> On module removal, the representors are gone and the ->cleanup
>> callback should be exercised, this callback removes the flow_block and
>> removes the rules in the driver.
>>
>> Can you check if the ->cleanup callback is exercised?
>
> I added some traces. On module unload mlx5e_cleanup_rep_tx() and
> flow_indr_dev_unregister() are called, but not tc_block_indr_cleanup().
> Maybe this is the problem that wenxu fixed in one of his patches? I'll
> try to reproduce on net branch.

Indeed, on net branch tc_block_indr_cleanup() is called and crash is not
reproduced. It seems to be fixed by a1db217861f3 ("net: flow_offload:
fix flow_indr_dev_unregister path"). 

>
>>
>>> Resulting dmesg:
>>> 
>>> [  153.747853] BUG: unable to handle page fault for address: ffffffffc114cee0
>>> [  153.747975] #PF: supervisor instruction fetch in kernel mode
>>> [  153.748071] #PF: error_code(0x0010) - not-present page
>>> [  153.748189] PGD 5b6c12067 P4D 5b6c12067 PUD 5b6c14067 PMD 35b76b067 PTE 0
>>> [  153.748328] Oops: 0010 [#1] SMP KASAN PTI
>>> [  153.748403] CPU: 1 PID: 1909 Comm: tc Not tainted 5.8.0-rc1+ #1170
>>> [  153.748507] Hardware name: Supermicro SYS-2028TP-DECR/X10DRT-P, BIOS 2.0b 03/30/2017
>>> [  153.748638] RIP: 0010:0xffffffffc114cee0
>>> [  153.748709] Code: Bad RIP value.
>>> [  153.748767] RSP: 0018:ffff88834895ef00 EFLAGS: 00010246
>>> [  153.748858] RAX: 0000000000000000 RBX: ffff888330a30078 RCX: ffffffffb2da70ba
>>> [  153.748975] RDX: ffff888333635d80 RSI: ffff88834895efa0 RDI: 0000000000000002
>>> [  153.752948] RBP: 0000000000000002 R08: 0000000000000001 R09: ffffed106614600c
>>> [  153.759173] R10: ffff888330a3005f R11: ffffed106614600b R12: ffff88834895efa0
>>> [  153.765419] R13: 0000000000000000 R14: ffffffffc114cee0 R15: ffff8883470efe00
>>> [  153.771689] FS:  00007f6f6ac12480(0000) GS:ffff888362e40000(0000) knlGS:0000000000000000
>>> [  153.777983] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [  153.784187] CR2: ffffffffc114ceb6 CR3: 000000035eb9e005 CR4: 00000000001606e0
>>> [  153.790567] Call Trace:
>>> [  153.796844]  ? tc_setup_cb_call+0xd8/0x170
>>> [  153.803164]  ? fl_hw_update_stats+0x117/0x280 [cls_flower]
>>> [  153.809516]  ? 0xffffffffc1328000
>>> [  153.815766]  ? _find_next_bit.constprop.0+0x3e/0xf0
>>> [  153.822079]  ? __nla_reserve+0x4c/0x60
>> [...]
>>> 
>>> I can come up with something to fix mlx5 but it looks like all other
>>> drivers that support indirect devices are also susceptible to similar
>>> issue.
>>
>> How does the fix you have in mind look like?
>
> To call flow_indr_dev_unregister() from right path. But you are right,
> it is already called, so we just need to determine why it doesn't
> perform the proper cleanup.
>
>>
>> Thanks.


      reply	other threads:[~2020-06-24 12:37 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-06-24 10:22 Crash in indirect block infra after unloading driver module Vlad Buslov
2020-06-24 10:30 ` Pablo Neira Ayuso
2020-06-24 12:22   ` Vlad Buslov
2020-06-24 12:37     ` Vlad Buslov [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=vbfk0zwy70f.fsf@mellanox.com \
    --to=vladbu@mellanox.com \
    --cc=jiri@resnulli.us \
    --cc=majd@mellanox.com \
    --cc=maord@mellanox.com \
    --cc=netdev@vger.kernel.org \
    --cc=pablo@netfilter.org \
    --cc=roid@mellanox.com \
    --cc=saeedm@mellanox.com \
    --cc=wenxu@ucloud.cn \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.