Crash in indirect block infra after unloading driver module

Crash in indirect block infra after unloading driver module

[Vlad Buslov]

Hi Pablo,

I've encountered a new issue with indirect offloads infrastructure. The
issue is that on driver offload its indirect callbacks are not removed
from blocks and any following offloads operations on block that has such
callback in its offloads cb list causes call to unmapped address.

Steps to reproduce:

echo 1 >/sys/class/net/ens1f0/device/sriov_numvfs
echo 0000:81:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind
devlink dev eswitch set pci/0000:81:00.0 mode switchdev

ip link add vxlan1 type vxlan dstport 4789 external
ip addr add 192.168.1.1 dev ens1f0
link set up dev ens1f0
ip link set up dev ens1f0
tc qdisc add dev vxlan1 ingress
tc filter add dev vxlan1 protocol ip ingress flower enc_src_ip 192.168.1.2 enc_dst_ip 192.168.1.1 enc_key_id 42 enc_dst_port 4789 action tunnel_key unset action mirred egress redirect dev ens1f0_0
tc -s filter show dev vxlan1 ingress

rmmod mlx5_ib
rmmod mlx5_core
tc -s filter show dev vxlan1 ingress


Resulting dmesg:

[  153.747853] BUG: unable to handle page fault for address: ffffffffc114cee0
[  153.747975] #PF: supervisor instruction fetch in kernel mode
[  153.748071] #PF: error_code(0x0010) - not-present page
[  153.748189] PGD 5b6c12067 P4D 5b6c12067 PUD 5b6c14067 PMD 35b76b067 PTE 0
[  153.748328] Oops: 0010 [#1] SMP KASAN PTI
[  153.748403] CPU: 1 PID: 1909 Comm: tc Not tainted 5.8.0-rc1+ #1170
[  153.748507] Hardware name: Supermicro SYS-2028TP-DECR/X10DRT-P, BIOS 2.0b 03/30/2017
[  153.748638] RIP: 0010:0xffffffffc114cee0
[  153.748709] Code: Bad RIP value.
[  153.748767] RSP: 0018:ffff88834895ef00 EFLAGS: 00010246
[  153.748858] RAX: 0000000000000000 RBX: ffff888330a30078 RCX: ffffffffb2da70ba
[  153.748975] RDX: ffff888333635d80 RSI: ffff88834895efa0 RDI: 0000000000000002
[  153.752948] RBP: 0000000000000002 R08: 0000000000000001 R09: ffffed106614600c
[  153.759173] R10: ffff888330a3005f R11: ffffed106614600b R12: ffff88834895efa0
[  153.765419] R13: 0000000000000000 R14: ffffffffc114cee0 R15: ffff8883470efe00
[  153.771689] FS:  00007f6f6ac12480(0000) GS:ffff888362e40000(0000) knlGS:0000000000000000
[  153.777983] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  153.784187] CR2: ffffffffc114ceb6 CR3: 000000035eb9e005 CR4: 00000000001606e0
[  153.790567] Call Trace:
[  153.796844]  ? tc_setup_cb_call+0xd8/0x170
[  153.803164]  ? fl_hw_update_stats+0x117/0x280 [cls_flower]
[  153.809516]  ? 0xffffffffc1328000
[  153.815766]  ? _find_next_bit.constprop.0+0x3e/0xf0
[  153.822079]  ? __nla_reserve+0x4c/0x60
[  153.828283]  ? memcpy+0x39/0x60
[  153.834311]  ? fl_dump+0x335/0x350 [cls_flower]
[  153.840389]  ? fl_tmplt_dump+0x110/0x110 [cls_flower]
[  153.846483]  ? memcpy+0x39/0x60
[  153.852516]  ? tcf_fill_node+0x2ea/0x420
[  153.858564]  ? tcf_chain_tp_delete_empty+0x170/0x170
[  153.864558]  ? tcf_node_dump+0xf9/0x110
[  153.870481]  ? fl_walk+0xf6/0x240 [cls_flower]
[  153.876357]  ? fl_put+0x10/0x10 [cls_flower]
[  153.882173]  ? __mutex_lock_slowpath+0x10/0x10
[  153.887922]  ? tcf_chain_dump+0x237/0x450
[  153.893595]  ? tcf_block_release+0x50/0x50
[  153.899141]  ? tfilter_notify+0x160/0x160
[  153.904630]  ? tc_dump_tfilter+0x388/0x4a0
[  153.910024]  ? tcf_chain_dump+0x450/0x450
[  153.915390]  ? __mutex_lock_slowpath+0x10/0x10
[  153.920657]  ? netlink_dump+0x2ea/0x670
[  153.925851]  ? __netlink_sendskb+0x70/0x70
[  153.931028]  ? __mutex_lock_slowpath+0x10/0x10
[  153.936117]  ? __alloc_skb+0xc3/0x310
[  153.941199]  ? __netlink_dump_start+0x2f6/0x3c0
[  153.946247]  ? rtnetlink_rcv_msg+0x383/0x510
[  153.951222]  ? tcf_chain_dump+0x450/0x450
[  153.956124]  ? rtnl_calcit.isra.0+0x1c0/0x1c0
[  153.961010]  ? kmem_cache_free+0x84/0x2a0
[  153.965787]  ? skb_free_datagram+0x12/0x60
[  153.970576]  ? netlink_recvmsg+0x281/0x690
[  153.975256]  ? tcf_chain_dump+0x450/0x450
[  153.979861]  ? netlink_compare+0x53/0x70
[  153.984447]  ? netlink_rcv_skb+0xd0/0x200
[  153.988872]  ? rtnl_calcit.isra.0+0x1c0/0x1c0
[  153.993233]  ? netlink_ack+0x460/0x460
[  153.997573]  ? __kasan_kmalloc.constprop.0+0xc2/0xd0
[  154.001936]  ? netlink_deliver_tap+0x48/0x390
[  154.006298]  ? netlink_unicast+0x2d8/0x3d0
[  154.010553]  ? netlink_attachskb+0x3d0/0x3d0
[  154.014747]  ? __virt_addr_valid+0xbb/0x130
[  154.018886]  ? netlink_sendmsg+0x3af/0x690
[  154.022997]  ? netlink_unicast+0x3d0/0x3d0
[  154.026993]  ? import_iovec+0x135/0x1e0
[  154.030898]  ? netlink_unicast+0x3d0/0x3d0
[  154.034776]  ? sock_sendmsg+0x96/0xa0
[  154.038526]  ? ____sys_sendmsg+0x388/0x420
[  154.042282]  ? kernel_sendmsg+0x30/0x30
[  154.045980]  ? __copy_msghdr_from_user+0x260/0x260
[  154.049702]  ? kernel_recvmsg+0x60/0x60
[  154.053397]  ? copy_msghdr_from_user+0xa7/0x100
[  154.057097]  ? ___sys_sendmsg+0xe0/0x140
[  154.060807]  ? sendmsg_copy_msghdr+0x30/0x30
[  154.064517]  ? ___sys_recvmsg+0xdc/0x130
[  154.068198]  ? recvmsg_copy_msghdr+0x40/0x40
[  154.071852]  ? _raw_read_lock_irqsave+0x50/0x50
[  154.075540]  ? __count_memcg_events+0x33/0x100
[  154.079222]  ? handle_mm_fault+0x66d/0x1eb0
[  154.082884]  ? __fget_light+0x9e/0xf0
[  154.086533]  ? __sys_sendmsg+0xb3/0x130
[  154.090164]  ? __sys_sendmsg_sock+0x60/0x60
[  154.093768]  ? do_syscall_64+0x4d/0x90
[  154.097357]  ? entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  154.101006] Modules linked in: act_mirred act_tunnel_key cls_flower sch_ingress vxlan ip6_udp_tunnel udp_tunnel nfsv3 nfs_acl nfs lockd grace fscache tun bridge stp llc sunrpc rdma_ucm rdma_cm iw_cm ib_cm ib_uverbs ib_core intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass igb crct10dif_pclmul crc32_pclmul mlxfw crc32c_intel ipmi_ssif ptp iTCO_wdt pps_core ses ioatdma ghash_clmulni_intel mei_me iTCO_vendor_suppo
rt enclosure i2c_i801 ipmi_si intel_cstate joydev mei i2c_smbus lpc_ich pcspkr intel_uncore dca wmi ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad ast i2c_algo_bit drm_vram_helper drm_kms_helper drm_ttm_helper ttm drm mpt3sas raid_class scsi_transport_sas [last unloaded: mlx5_core]
[  154.132864] CR2: ffffffffc114cee0
[  154.137879] ---[ end trace 1c03f81270107f93 ]---
[  154.162631] RIP: 0010:0xffffffffc114cee0
[  154.167703] Code: Bad RIP value.
[  154.172705] RSP: 0018:ffff88834895ef00 EFLAGS: 00010246
[  154.177817] RAX: 0000000000000000 RBX: ffff888330a30078 RCX: ffffffffb2da70ba
[  154.183006] RDX: ffff888333635d80 RSI: ffff88834895efa0 RDI: 0000000000000002
[  154.188259] RBP: 0000000000000002 R08: 0000000000000001 R09: ffffed106614600c
[  154.193585] R10: ffff888330a3005f R11: ffffed106614600b R12: ffff88834895efa0
[  154.198928] R13: 0000000000000000 R14: ffffffffc114cee0 R15: ffff8883470efe00
[  154.204315] FS:  00007f6f6ac12480(0000) GS:ffff888362e40000(0000) knlGS:0000000000000000
[  154.209867] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  154.215417] CR2: ffffffffc114ceb6 CR3: 000000035eb9e005 CR4:
00000000001606e0


I can come up with something to fix mlx5 but it looks like all other
drivers that support indirect devices are also susceptible to similar
issue.

Regards,
Vlad

Re: Crash in indirect block infra after unloading driver module

[Pablo Neira Ayuso]

On Wed, Jun 24, 2020 at 01:22:29PM +0300, Vlad Buslov wrote:
> Hi Pablo,
> 
> I've encountered a new issue with indirect offloads infrastructure. The
> issue is that on driver offload its indirect callbacks are not removed
> from blocks and any following offloads operations on block that has such
> callback in its offloads cb list causes call to unmapped address.
> 
> Steps to reproduce:
> 
> echo 1 >/sys/class/net/ens1f0/device/sriov_numvfs
> echo 0000:81:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind
> devlink dev eswitch set pci/0000:81:00.0 mode switchdev
> 
> ip link add vxlan1 type vxlan dstport 4789 external
> ip addr add 192.168.1.1 dev ens1f0
> link set up dev ens1f0
> ip link set up dev ens1f0
> tc qdisc add dev vxlan1 ingress
> tc filter add dev vxlan1 protocol ip ingress flower enc_src_ip 192.168.1.2 enc_dst_ip 192.168.1.1 enc_key_id 42 enc_dst_port 4789 action tunnel_key unset action mirred egress redirect dev ens1f0_0
> tc -s filter show dev vxlan1 ingress
> 
> rmmod mlx5_ib
> rmmod mlx5_core
> tc -s filter show dev vxlan1 ingress

On module removal, the representors are gone and the ->cleanup
callback should be exercised, this callback removes the flow_block and
removes the rules in the driver.

Can you check if the ->cleanup callback is exercised?

> Resulting dmesg:
> 
> [  153.747853] BUG: unable to handle page fault for address: ffffffffc114cee0
> [  153.747975] #PF: supervisor instruction fetch in kernel mode
> [  153.748071] #PF: error_code(0x0010) - not-present page
> [  153.748189] PGD 5b6c12067 P4D 5b6c12067 PUD 5b6c14067 PMD 35b76b067 PTE 0
> [  153.748328] Oops: 0010 [#1] SMP KASAN PTI
> [  153.748403] CPU: 1 PID: 1909 Comm: tc Not tainted 5.8.0-rc1+ #1170
> [  153.748507] Hardware name: Supermicro SYS-2028TP-DECR/X10DRT-P, BIOS 2.0b 03/30/2017
> [  153.748638] RIP: 0010:0xffffffffc114cee0
> [  153.748709] Code: Bad RIP value.
> [  153.748767] RSP: 0018:ffff88834895ef00 EFLAGS: 00010246
> [  153.748858] RAX: 0000000000000000 RBX: ffff888330a30078 RCX: ffffffffb2da70ba
> [  153.748975] RDX: ffff888333635d80 RSI: ffff88834895efa0 RDI: 0000000000000002
> [  153.752948] RBP: 0000000000000002 R08: 0000000000000001 R09: ffffed106614600c
> [  153.759173] R10: ffff888330a3005f R11: ffffed106614600b R12: ffff88834895efa0
> [  153.765419] R13: 0000000000000000 R14: ffffffffc114cee0 R15: ffff8883470efe00
> [  153.771689] FS:  00007f6f6ac12480(0000) GS:ffff888362e40000(0000) knlGS:0000000000000000
> [  153.777983] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  153.784187] CR2: ffffffffc114ceb6 CR3: 000000035eb9e005 CR4: 00000000001606e0
> [  153.790567] Call Trace:
> [  153.796844]  ? tc_setup_cb_call+0xd8/0x170
> [  153.803164]  ? fl_hw_update_stats+0x117/0x280 [cls_flower]
> [  153.809516]  ? 0xffffffffc1328000
> [  153.815766]  ? _find_next_bit.constprop.0+0x3e/0xf0
> [  153.822079]  ? __nla_reserve+0x4c/0x60
[...]
> 
> I can come up with something to fix mlx5 but it looks like all other
> drivers that support indirect devices are also susceptible to similar
> issue.

How does the fix you have in mind look like?

Thanks.

Re: Crash in indirect block infra after unloading driver module

[Vlad Buslov]

On Wed 24 Jun 2020 at 13:30, Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> On Wed, Jun 24, 2020 at 01:22:29PM +0300, Vlad Buslov wrote:
>> Hi Pablo,
>> 
>> I've encountered a new issue with indirect offloads infrastructure. The
>> issue is that on driver offload its indirect callbacks are not removed
>> from blocks and any following offloads operations on block that has such
>> callback in its offloads cb list causes call to unmapped address.
>> 
>> Steps to reproduce:
>> 
>> echo 1 >/sys/class/net/ens1f0/device/sriov_numvfs
>> echo 0000:81:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind
>> devlink dev eswitch set pci/0000:81:00.0 mode switchdev
>> 
>> ip link add vxlan1 type vxlan dstport 4789 external
>> ip addr add 192.168.1.1 dev ens1f0
>> link set up dev ens1f0
>> ip link set up dev ens1f0
>> tc qdisc add dev vxlan1 ingress
>> tc filter add dev vxlan1 protocol ip ingress flower enc_src_ip 192.168.1.2 enc_dst_ip 192.168.1.1 enc_key_id 42 enc_dst_port 4789 action tunnel_key unset action mirred egress redirect dev ens1f0_0
>> tc -s filter show dev vxlan1 ingress
>> 
>> rmmod mlx5_ib
>> rmmod mlx5_core
>> tc -s filter show dev vxlan1 ingress
>
> On module removal, the representors are gone and the ->cleanup
> callback should be exercised, this callback removes the flow_block and
> removes the rules in the driver.
>
> Can you check if the ->cleanup callback is exercised?

I added some traces. On module unload mlx5e_cleanup_rep_tx() and
flow_indr_dev_unregister() are called, but not tc_block_indr_cleanup().
Maybe this is the problem that wenxu fixed in one of his patches? I'll
try to reproduce on net branch.

>
>> Resulting dmesg:
>> 
>> [  153.747853] BUG: unable to handle page fault for address: ffffffffc114cee0
>> [  153.747975] #PF: supervisor instruction fetch in kernel mode
>> [  153.748071] #PF: error_code(0x0010) - not-present page
>> [  153.748189] PGD 5b6c12067 P4D 5b6c12067 PUD 5b6c14067 PMD 35b76b067 PTE 0
>> [  153.748328] Oops: 0010 [#1] SMP KASAN PTI
>> [  153.748403] CPU: 1 PID: 1909 Comm: tc Not tainted 5.8.0-rc1+ #1170
>> [  153.748507] Hardware name: Supermicro SYS-2028TP-DECR/X10DRT-P, BIOS 2.0b 03/30/2017
>> [  153.748638] RIP: 0010:0xffffffffc114cee0
>> [  153.748709] Code: Bad RIP value.
>> [  153.748767] RSP: 0018:ffff88834895ef00 EFLAGS: 00010246
>> [  153.748858] RAX: 0000000000000000 RBX: ffff888330a30078 RCX: ffffffffb2da70ba
>> [  153.748975] RDX: ffff888333635d80 RSI: ffff88834895efa0 RDI: 0000000000000002
>> [  153.752948] RBP: 0000000000000002 R08: 0000000000000001 R09: ffffed106614600c
>> [  153.759173] R10: ffff888330a3005f R11: ffffed106614600b R12: ffff88834895efa0
>> [  153.765419] R13: 0000000000000000 R14: ffffffffc114cee0 R15: ffff8883470efe00
>> [  153.771689] FS:  00007f6f6ac12480(0000) GS:ffff888362e40000(0000) knlGS:0000000000000000
>> [  153.777983] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [  153.784187] CR2: ffffffffc114ceb6 CR3: 000000035eb9e005 CR4: 00000000001606e0
>> [  153.790567] Call Trace:
>> [  153.796844]  ? tc_setup_cb_call+0xd8/0x170
>> [  153.803164]  ? fl_hw_update_stats+0x117/0x280 [cls_flower]
>> [  153.809516]  ? 0xffffffffc1328000
>> [  153.815766]  ? _find_next_bit.constprop.0+0x3e/0xf0
>> [  153.822079]  ? __nla_reserve+0x4c/0x60
> [...]
>> 
>> I can come up with something to fix mlx5 but it looks like all other
>> drivers that support indirect devices are also susceptible to similar
>> issue.
>
> How does the fix you have in mind look like?

To call flow_indr_dev_unregister() from right path. But you are right,
it is already called, so we just need to determine why it doesn't
perform the proper cleanup.

>
> Thanks.

Re: Crash in indirect block infra after unloading driver module

[Vlad Buslov]

On Wed 24 Jun 2020 at 15:22, Vlad Buslov <vladbu@mellanox.com> wrote:
> On Wed 24 Jun 2020 at 13:30, Pablo Neira Ayuso <pablo@netfilter.org> wrote:
>> On Wed, Jun 24, 2020 at 01:22:29PM +0300, Vlad Buslov wrote:
>>> Hi Pablo,
>>> 
>>> I've encountered a new issue with indirect offloads infrastructure. The
>>> issue is that on driver offload its indirect callbacks are not removed
>>> from blocks and any following offloads operations on block that has such
>>> callback in its offloads cb list causes call to unmapped address.
>>> 
>>> Steps to reproduce:
>>> 
>>> echo 1 >/sys/class/net/ens1f0/device/sriov_numvfs
>>> echo 0000:81:00.2 > /sys/bus/pci/drivers/mlx5_core/unbind
>>> devlink dev eswitch set pci/0000:81:00.0 mode switchdev
>>> 
>>> ip link add vxlan1 type vxlan dstport 4789 external
>>> ip addr add 192.168.1.1 dev ens1f0
>>> link set up dev ens1f0
>>> ip link set up dev ens1f0
>>> tc qdisc add dev vxlan1 ingress
>>> tc filter add dev vxlan1 protocol ip ingress flower enc_src_ip 192.168.1.2 enc_dst_ip 192.168.1.1 enc_key_id 42 enc_dst_port 4789 action tunnel_key unset action mirred egress redirect dev ens1f0_0
>>> tc -s filter show dev vxlan1 ingress
>>> 
>>> rmmod mlx5_ib
>>> rmmod mlx5_core
>>> tc -s filter show dev vxlan1 ingress
>>
>> On module removal, the representors are gone and the ->cleanup
>> callback should be exercised, this callback removes the flow_block and
>> removes the rules in the driver.
>>
>> Can you check if the ->cleanup callback is exercised?
>
> I added some traces. On module unload mlx5e_cleanup_rep_tx() and
> flow_indr_dev_unregister() are called, but not tc_block_indr_cleanup().
> Maybe this is the problem that wenxu fixed in one of his patches? I'll
> try to reproduce on net branch.

Indeed, on net branch tc_block_indr_cleanup() is called and crash is not
reproduced. It seems to be fixed by a1db217861f3 ("net: flow_offload:
fix flow_indr_dev_unregister path"). 

>
>>
>>> Resulting dmesg:
>>> 
>>> [  153.747853] BUG: unable to handle page fault for address: ffffffffc114cee0
>>> [  153.747975] #PF: supervisor instruction fetch in kernel mode
>>> [  153.748071] #PF: error_code(0x0010) - not-present page
>>> [  153.748189] PGD 5b6c12067 P4D 5b6c12067 PUD 5b6c14067 PMD 35b76b067 PTE 0
>>> [  153.748328] Oops: 0010 [#1] SMP KASAN PTI
>>> [  153.748403] CPU: 1 PID: 1909 Comm: tc Not tainted 5.8.0-rc1+ #1170
>>> [  153.748507] Hardware name: Supermicro SYS-2028TP-DECR/X10DRT-P, BIOS 2.0b 03/30/2017
>>> [  153.748638] RIP: 0010:0xffffffffc114cee0
>>> [  153.748709] Code: Bad RIP value.
>>> [  153.748767] RSP: 0018:ffff88834895ef00 EFLAGS: 00010246
>>> [  153.748858] RAX: 0000000000000000 RBX: ffff888330a30078 RCX: ffffffffb2da70ba
>>> [  153.748975] RDX: ffff888333635d80 RSI: ffff88834895efa0 RDI: 0000000000000002
>>> [  153.752948] RBP: 0000000000000002 R08: 0000000000000001 R09: ffffed106614600c
>>> [  153.759173] R10: ffff888330a3005f R11: ffffed106614600b R12: ffff88834895efa0
>>> [  153.765419] R13: 0000000000000000 R14: ffffffffc114cee0 R15: ffff8883470efe00
>>> [  153.771689] FS:  00007f6f6ac12480(0000) GS:ffff888362e40000(0000) knlGS:0000000000000000
>>> [  153.777983] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [  153.784187] CR2: ffffffffc114ceb6 CR3: 000000035eb9e005 CR4: 00000000001606e0
>>> [  153.790567] Call Trace:
>>> [  153.796844]  ? tc_setup_cb_call+0xd8/0x170
>>> [  153.803164]  ? fl_hw_update_stats+0x117/0x280 [cls_flower]
>>> [  153.809516]  ? 0xffffffffc1328000
>>> [  153.815766]  ? _find_next_bit.constprop.0+0x3e/0xf0
>>> [  153.822079]  ? __nla_reserve+0x4c/0x60
>> [...]
>>> 
>>> I can come up with something to fix mlx5 but it looks like all other
>>> drivers that support indirect devices are also susceptible to similar
>>> issue.
>>
>> How does the fix you have in mind look like?
>
> To call flow_indr_dev_unregister() from right path. But you are right,
> it is already called, so we just need to determine why it doesn't
> perform the proper cleanup.
>
>>
>> Thanks.