Re: [bug report] [rdma] RXE ODP test hangs with new DMA map API

[Daisuke Matsuda <dskmtsd@gmail.com>]

Some additional information:

NULL Pointer dereference is observed while calling ibv_reg_mr(3) to allocate a MR for ODP,
and that triggered the interruption which resulted in the stuck in uverbs_destroy_ufile_hw().
==========
[  488.242907] BUG: kernel NULL pointer dereference, address: 00000000000002fc
[  488.242923] #PF: supervisor read access in kernel mode
[  488.242932] #PF: error_code(0x0000) - not-present page
[  488.242940] PGD 1028eb067 P4D 1028eb067 PUD 105da0067 PMD 0
[  488.242951] Oops: Oops: 0000 [#1] SMP NOPTI
[  488.242960] CPU: 3 UID: 1000 PID: 1854 Comm: python3 Tainted: G        W           6.15.0-rc1+ #11 PREEMPT(voluntary)
[  488.242976] Tainted: [W]=WARN
[  488.242981] Hardware name: Trigkey Key N/Key N, BIOS KEYN101 09/02/2024
[  488.242990] RIP: 0010:hmm_dma_map_alloc+0x25/0x100
[  488.243002] Code: 90 90 90 90 90 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 49 89 d6 49 c1 e6 0c 41 55 41 54 53 49 39 ce 0f 82 c6 00 00 00 49 89 fc <f6> 87 fc 02 00 00 20 0f 84 af 00 00 00 49 89 f5 48 89 d3 49 89 cf
[  488.243024] RSP: 0018:ffffd3d3420eb830 EFLAGS: 00010246
[  488.243032] RAX: 0000000000001000 RBX: ffff8b727c7f7400 RCX: 0000000000001000
[  488.243042] RDX: 0000000000000001 RSI: ffff8b727c7f74b0 RDI: 0000000000000000
[  488.243052] RBP: ffffd3d3420eb858 R08: 0000000000000000 R09: 0000000000000000
[  488.243062] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[  488.243072] R13: 00007262a622a000 R14: 0000000000001000 R15: ffff8b727c7f74b0
[  488.243082] FS:  00007262a62a1080(0000) GS:ffff8b762ac3e000(0000) knlGS:0000000000000000
[  488.243094] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  488.243102] CR2: 00000000000002fc CR3: 000000010a1f0004 CR4: 0000000000f72ef0
[  488.243113] PKRU: 55555554
[  488.243117] Call Trace:
[  488.243123]  <TASK>
[  488.243131]  ib_init_umem_odp+0xb6/0x110 [ib_uverbs]
[  488.243151]  ib_umem_odp_get+0xf0/0x150 [ib_uverbs]
[  488.243166]  rxe_odp_mr_init_user+0x71/0x170 [rdma_rxe]
[  488.243181]  rxe_reg_user_mr+0x217/0x2e0 [rdma_rxe]
[  488.243194]  ib_uverbs_reg_mr+0x19e/0x2e0 [ib_uverbs]
[  488.243209]  ib_uverbs_handler_UVERBS_METHOD_INVOKE_WRITE+0xd9/0x150 [ib_uverbs]
[  488.243227]  ib_uverbs_cmd_verbs+0xd19/0xee0 [ib_uverbs]
[  488.243243]  ? mmap_region+0x63/0xd0
[  488.243251]  ? __pfx_ib_uverbs_handler_UVERBS_METHOD_INVOKE_WRITE+0x10/0x10 [ib_uverbs]
[  488.243272]  ib_uverbs_ioctl+0xba/0x130 [ib_uverbs]
[  488.243287]  __x64_sys_ioctl+0xa4/0xe0
[  488.243295]  x64_sys_call+0x1178/0x2660
[  488.243303]  do_syscall_64+0x7e/0x170
[  488.243310]  ? syscall_exit_to_user_mode+0x4e/0x250
[  488.243319]  ? do_syscall_64+0x8a/0x170
[  488.243326]  ? do_syscall_64+0x8a/0x170
[  488.243332]  ? syscall_exit_to_user_mode+0x4e/0x250
[  488.243340]  ? do_syscall_64+0x8a/0x170
[  488.243347]  ? syscall_exit_to_user_mode+0x4e/0x250
[  488.243355]  ? do_syscall_64+0x8a/0x170
[  488.243361]  ? do_user_addr_fault+0x1d2/0x8d0
[  488.243369]  ? irqentry_exit_to_user_mode+0x43/0x250
[  488.243378]  ? irqentry_exit+0x43/0x50
[  488.243384]  ? exc_page_fault+0x93/0x1d0
[  488.243391]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  488.243400] RIP: 0033:0x7262a6124ded
[  488.243406] Code: 04 25 28 00 00 00 48 89 45 c8 31 c0 48 8d 45 10 c7 45 b0 10 00 00 00 48 89 45 b8 48 8d 45 d0 48 89 45 c0 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1a 48 8b 45 c8 64 48 2b 04 25 28 00 00 00
[  488.243428] RSP: 002b:00007fffd08c3960 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[  488.243439] RAX: ffffffffffffffda RBX: 00007fffd08c39f0 RCX: 00007262a6124ded
[  488.243449] RDX: 00007fffd08c3a10 RSI: 00000000c0181b01 RDI: 0000000000000007
[  488.243459] RBP: 00007fffd08c39b0 R08: 0000000014107820 R09: 00007fffd08c3b44
[  488.243469] R10: 000000000000000c R11: 0000000000000246 R12: 00007fffd08c3b44
[  488.243478] R13: 000000000000000c R14: 00007fffd08c3b58 R15: 0000000014107960
[  488.243489]  </TASK>
[  488.243492] Modules linked in: rdma_rxe ip6_udp_tunnel udp_tunnel ib_uverbs ib_core xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo xt_addrtype nft_compat nf_tables qrtr xe snd_hda_codec_realtek drm_ttm_helper snd_hda_codec_generic drm_suballoc_helper snd_hda_scodec_component gpu_sched drm_gpuvm drm_exec drm_gpusvm snd_sof_pci_intel_tgl snd_sof_pci_intel_cnl snd_sof_intel_hda_generic snd_sof_pci snd_sof_xtensa_dsp snd_sof_intel_hda_common snd_soc_hdac_hda snd_sof_intel_hda snd_hda_codec_hdmi rtw88_8821ce rtw88_8821c snd_sof rtw88_pci i915 rtw88_core intel_rapl_msr snd_sof_utils intel_rapl_common snd_soc_acpi_intel_match snd_soc_acpi x86_pkg_temp_thermal snd_soc_acpi_intel_sdca_quirks intel_powerclamp snd_sof_intel_hda_mlink mac80211 snd_hda_ext_core snd_soc_sdca binfmt_misc snd_soc_core snd_compress coretemp kvm_intel snd_hda_intel nls_iso8859_1 snd_intel_dspcfg btusb kvm snd_hda_codec btrtl btintel cmdlinepart snd_hwdep libarc4 btbcm snd_hda_core spi_nor
[  488.243531]  btmtk cfg80211 bluetooth snd_pcm mtd i2c_algo_bit ee1004 snd_timer rapl wmi_bmof drm_buddy snd i2c_i801 mei_me ttm intel_pmc_core mei intel_cstate spi_intel_pci i2c_smbus soundcore spi_intel drm_display_helper pmt_telemetry igen6_edac video pmt_class intel_vsec wmi acpi_tad acpi_pad mac_hid sch_fq_codel overlay iptable_filter ip6table_filter ip6_tables br_netfilter bridge stp llc arp_tables dm_multipath msr efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 spi_pxa2xx_platform dw_dmac dw_dmac_core 8250_dw spi_pxa2xx_core polyval_clmulni polyval_generic ghash_clmulni_intel r8169 sha256_ssse3 intel_lpss_pci sha1_ssse3 intel_lpss ahci idma64 realtek libahci pinctrl_alderlake aesni_intel crypto_simd cryptd
[  488.245175] CR2: 00000000000002fc
[  488.245901] ---[ end trace 0000000000000000 ]---
[  488.345979] RIP: 0010:hmm_dma_map_alloc+0x25/0x100
[  488.346754] Code: 90 90 90 90 90 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 49 89 d6 49 c1 e6 0c 41 55 41 54 53 49 39 ce 0f 82 c6 00 00 00 49 89 fc <f6> 87 fc 02 00 00 20 0f 84 af 00 00 00 49 89 f5 48 89 d3 49 89 cf
[  488.347530] RSP: 0018:ffffd3d3420eb830 EFLAGS: 00010246
[  488.348320] RAX: 0000000000001000 RBX: ffff8b727c7f7400 RCX: 0000000000001000
[  488.349112] RDX: 0000000000000001 RSI: ffff8b727c7f74b0 RDI: 0000000000000000
[  488.349894] RBP: ffffd3d3420eb858 R08: 0000000000000000 R09: 0000000000000000
[  488.350669] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[  488.351439] R13: 00007262a622a000 R14: 0000000000001000 R15: ffff8b727c7f74b0
[  488.352210] FS:  00007262a62a1080(0000) GS:ffff8b762ac3e000(0000) knlGS:0000000000000000
[  488.352963] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  488.353699] CR2: 00000000000002fc CR3: 000000010a1f0004 CR4: 0000000000f72ef0
[  488.354436] PKRU: 55555554
[  488.355173] note: python3[1854] exited with irqs disabled
==========

The root cause is that dev is NULL in hmm_dma_map_alloc() because
RXE passes NULL for dma_device when calling ib_register_device(),
which is allowed according to the comment in ib_register_device()
==========
int hmm_dma_map_alloc(struct device *dev, struct hmm_dma_map *map,
                       size_t nr_entries, size_t dma_entry_size)
{
         bool dma_need_sync = false;
         bool use_iova;

         if (!(nr_entries * PAGE_SIZE / dma_entry_size))
                 return -EINVAL;

         /*
          * The HMM API violates our normal DMA buffer ownership rules and can't
          * transfer buffer ownership.  The dma_addressing_limited() check is a
          * best approximation to ensure no swiotlb buffering happens.
          */
#ifdef CONFIG_DMA_NEED_SYNC
         dma_need_sync = !dev->dma_skip_sync; // <--- NULL pointer dereference
#endif /* CONFIG_DMA_NEED_SYNC */
         if (dma_need_sync || dma_addressing_limited(dev)) // <--- NULL pointer dereference
                 return -EOPNOTSUPP;
==========

I have found that the issue can be resolved by adding some NULL checks in hmm.c,
so I will make a patch and post the fix in a day or so.

Thanks,
Daisuke

On 2025/05/21 21:48, Daisuke Matsuda wrote:
> Hi,
> 
> After these two patches are merged to the for-next tree, RXE ODP test always hangs:
>    RDMA/core: Convert UMEM ODP DMA mapping to caching IOVA and page linkage
>    RDMA/umem: Store ODP access mask information in PFN
> cf. https://lore.kernel.org/linux-rdma/cover.1745831017.git.leon@kernel.org/
> 
> Here is the console log:
> ```
> $ ./build/bin/run_tests.py -v -k odp
> test_odp_dc_traffic (tests.test_mlx5_dc.DCTest.test_odp_dc_traffic) ... skipped 'Can not run the test over non MLX5 device'
> test_devx_rc_qp_odp_traffic (tests.test_mlx5_devx.Mlx5DevxRcTrafficTest.test_devx_rc_qp_odp_traffic) ... skipped 'Can not run the test over non MLX5 device'
> test_odp_mkey_list_new_api (tests.test_mlx5_mkey.Mlx5MkeyTest.test_odp_mkey_list_new_api)
> Create Mkeys above ODP MR, configure it with memory layout using the new API and ... skipped 'Could not open mlx5 context (This is not an MLX5 device)'
> test_odp_async_prefetch_rc_traffic (tests.test_odp.OdpTestCase.test_odp_async_prefetch_rc_traffic) ...
> 
> 
> ```
> 
> It looks that the python process is somehow stuck in uverbs_destroy_ufile_hw():
> ```
> $ sudo cat /proc/1845/task/1845/stack
> [<0>] uverbs_destroy_ufile_hw+0x24/0x100 [ib_uverbs]
> [<0>] ib_uverbs_close+0x1b/0xc0 [ib_uverbs]
> [<0>] __fput+0xea/0x2d0
> [<0>] ____fput+0x15/0x20
> [<0>] task_work_run+0x5d/0xa0
> [<0>] do_exit+0x316/0xa50
> [<0>] make_task_dead+0x81/0x160
> [<0>] rewind_stack_and_make_dead+0x16/0x20




> ```
> 
> I am not sure about the root cause but hope we can fix this before the next merge window.
> 
> Thanks,
> Daisuke