Re: [PATCH v3 00/30] Introduce GPU SVM and Xe SVM implementation

[Matthew Brost <matthew.brost@intel.com>]

On Fri, Jan 17, 2025 at 11:47:41AM +0200, Gwan-gyeong Mun wrote:
> Hi,
> This kernel oops, which I reported before, was caused by my incorrect
> modification (incorrect applying of review comments) of this patch
> "[v3,19/30] drm/xe: Add SVM device memory mirroring"
> ( the kernel oops occurred because the xe_drm_pagemap_map_dma() and
> xe_devm_add() functions were built in the form of empty functions. )
> 
> This issue disappeared after proper patch modifications were applied.
> So please ignore the previously reported this kernel oops.
> 

My post unfortunately had some bugs masked by Kconfig issue Himal
pointed out. If you want to test of this code I suggest you pull this
branch [1] - it should be stable with all fixes I mention on this rev.

Matt

[1] https://gitlab.freedesktop.org/mbrost/xe-kernel-driver-svm-stable-1-13-25/-/tree/stable-1.13.25?ref_type=heads

> Br,
> 
> G.G.
> 
> On 1/7/25 2:19 PM, Gwan-gyeong Mun wrote:
> > Hi Matthew Brost,
> > 
> > After applying this patch series and the following to the latest drm-
> > tip, while testing[1] with the mentioned IGT, I faced a kernel oops[3].
> > It makes prevent progressing of the mentioned igt tests.
> > Could you please check the following oops log?
> > 
> > (1) apply comments of "[v3,05/30] drm/gpusvm: Add support for GPU Shared
> > Virtual Memory"
> > (2) apply comments of "[v3,15/30] drm/xe: Add unbind to SVM garbage
> > collector"
> > (3) drop "[v3,27/30] drm/xe: Basic SVM BO eviction" patch
> > 
> > The kernel config used, the entire dmesg, and detailed information can
> > be found in [2].
> > 
> > br,
> > 
> > G.G.
> > 
> > [1] used igt command: xe_exec_system_allocator --run-subtest once-malloc
> > [2] https://gitlab.freedesktop.org/elongbug/drm-tip/-/snippets/7823
> > 
> > [3] kernel oops dmesg
> > [   51.365230] Console: switching to colour VGA+ 80x25
> > [   51.367772] [IGT] xe_exec_system_allocator: executing
> > [   51.383611] [IGT] xe_exec_system_allocator: starting subtest once-malloc
> > [   51.386066] xe 0000:00:04.0: [drm:vm_bind_ioctl_ops_create [xe]]
> > op=0, addr=0x0000000000000000, range=0x0001000000000000,
> > bo_offset_or_userptr=0x0000000000000000
> > [   51.386171] xe 0000:00:04.0: [drm:vm_bind_ioctl_ops_create [xe]] MAP:
> > addr=0x0000000000000000, range=0x0001000000000000
> > [   51.389429] xe 0000:00:04.0: [drm:xe_svm_handle_pagefault [xe]] PAGE
> > FAULT: asid=1, gpusvm=0xffff8881775e9188, vram=0,0,
> > seqno=9223372036854775807, start=0x005584e8400000, end=0x005584e8410000,
> > size=65536
> > [   51.389529] xe 0000:00:04.0: [drm:xe_svm_handle_pagefault [xe]]
> > ALLOCATE VRAM: asid=1, gpusvm=0xffff8881775e9188, vram=0,0,
> > seqno=9223372036854775807, start=0x005584e8400000, end=0x005584e8410000,
> > size=65536
> > [   51.389935] xe 0000:00:04.0: [drm:xe_svm_handle_pagefault [xe]] ALLOC
> > VRAM: asid=1, gpusvm=0xffff8881775e9188, pfn=3126960, npages=16
> > [   51.390048] xe 0000:00:04.0: [drm:xe_svm_invalidate [xe]] INVALIDATE:
> > asid=1, gpusvm=0xffff8881775e9188, seqno=3, start=0x00005584e8400000,
> > end=0x00005584e8410000, event=6
> > [   51.390440] xe 0000:00:04.0: [drm:xe_svm_invalidate [xe]] NOTIFIER:
> > asid=1, gpusvm=0xffff8881775e9188, vram=0,0, seqno=9223372036854775807,
> > start=0x005584e8400000, end=0x005584e8410000, size=65536
> > [   51.390948] Oops: general protection fault, probably for non-
> > canonical address 0x3fff88842fc80000: 0000 [#1] PREEMPT SMP NOPTI
> > [   51.391624] CPU: 1 UID: 0 PID: 76 Comm: kworker/u17:0 Not tainted
> > 6.13.0-rc4-drm-tip-test+ #48
> > [   51.392088] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
> > BIOS 1.15.0-1 04/01/2014
> > [   51.392527] Workqueue: xe_gt_page_fault_work_queue pf_queue_work_func
> > [xe]
> > [   51.392947] RIP: 0010:zone_device_page_init+0x5d/0x240
> > [   51.393228] Code: 04 dd ff e8 d5 d2 a1 00 5a 85 c0 0f 85 ba 00 00 00
> > e8 d7 bb df ff 85 c0 0f 84 9d 01 00 00 48 8b 45 38 a8 03 0f 85 ec 00 00
> > 00 <65> 48 ff 00 e8 aa d2 a1 00 85 c0 0f 85 0d 01 00 00 48 c7 c7 20 cb
> > [   51.394247] RSP: 0018:ffffc9000039fb48 EFLAGS: 00010246
> > [   51.394570] RAX: 4000000000000000 RBX: ffffea000bedac00 RCX:
> > 0000000000000000
> > [   51.394950] RDX: 0000000000000046 RSI: ffffffff824c67b4 RDI:
> > ffffffff824e58f5
> > [   51.395328] RBP: ffffea000bedac08 R08: 0000000000000015 R09:
> > 0000000000000004
> > [   51.395709] R10: 0000000000000001 R11: 0000000000000004 R12:
> > 0000000000000001
> > [   51.396093] R13: ffff888170fd8d40 R14: ffff88817f922640 R15:
> > ffffea000bedac00
> > [   51.396472] FS:  0000000000000000(0000) GS:ffff88842fc80000(0000)
> > knlGS:0000000000000000
> > [   51.396925] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [   51.397237] CR2: 0000563d1f7ecbe4 CR3: 000000017c212000 CR4:
> > 0000000000750ef0
> > [   51.397618] PKRU: 55555554
> > [   51.397768] Call Trace:
> > [   51.397904]  <TASK>
> > [   51.398024]  ? __die_body.cold+0x19/0x26
> > [   51.398238]  ? die_addr+0x38/0x60
> > [   51.398420]  ? exc_general_protection+0x19e/0x450
> > [   51.398678]  ? asm_exc_general_protection+0x22/0x30
> > [   51.398942]  ? zone_device_page_init+0x5d/0x240
> > [   51.399188]  ? zone_device_page_init+0x49/0x240
> > [   51.399433]  drm_gpusvm_migrate_to_devmem+0x379/0x9e0 [drm_gpusvm]
> > [   51.399768]  xe_svm_handle_pagefault+0x62c/0xa60 [xe]
> > [   51.400110]  ? xe_vm_find_overlapping_vma+0xa4/0x1d0 [xe]
> > [   51.400475]  pf_queue_work_func+0x1ba/0x450 [xe]
> > [   51.400777]  process_one_work+0x1fe/0x580
> > [   51.400996]  worker_thread+0x1d1/0x3b0
> > [   51.401201]  ? __pfx_worker_thread+0x10/0x10
> > [   51.401433]  kthread+0xeb/0x120
> > [   51.401609]  ? __pfx_kthread+0x10/0x10
> > [   51.401813]  ret_from_fork+0x2d/0x50
> > [   51.402008]  ? __pfx_kthread+0x10/0x10
> > [   51.402211]  ret_from_fork_asm+0x1a/0x30
> > [   51.402427]  </TASK>
> > [   51.402551] Modules linked in: xe drm_ttm_helper gpu_sched
> > drm_suballoc_helper drm_gpuvm drm_exec drm_gpusvm i2c_algo_bit drm_buddy
> > video wmi ttm drm_display_helper drm_kms_helper crct10dif_pclmul
> > crc32_pclmul e1000 ghash_clmulni_intel i2c_piix4 i2c_smbus fuse
> > [   51.403779] ---[ end trace 0000000000000000 ]---
> > [   51.404106] RIP: 0010:zone_device_page_init+0x5d/0x240
> > [   51.404393] Code: 04 dd ff e8 d5 d2 a1 00 5a 85 c0 0f 85 ba 00 00 00
> > e8 d7 bb df ff 85 c0 0f 84 9d 01 00 00 48 8b 45 38 a8 03 0f 85 ec 00 00
> > 00 <65> 48 ff 00 e8 aa d2 a1 00 85 c0 0f 85 0d 01 00 00 48 c7 c7 20 cb
> > [   51.405408] RSP: 0018:ffffc9000039fb48 EFLAGS: 00010246
> > [   51.405725] RAX: 4000000000000000 RBX: ffffea000bedac00 RCX:
> > 0000000000000000
> > [   51.406110] RDX: 0000000000000046 RSI: ffffffff824c67b4 RDI:
> > ffffffff824e58f5
> > [   51.406518] RBP: ffffea000bedac08 R08: 0000000000000015 R09:
> > 0000000000000004
> > [   51.406905] R10: 0000000000000001 R11: 0000000000000004 R12:
> > 0000000000000001
> > [   51.407312] R13: ffff888170fd8d40 R14: ffff88817f922640 R15:
> > ffffea000bedac00
> > [   51.407691] FS:  0000000000000000(0000) GS:ffff88842fc80000(0000)
> > knlGS:0000000000000000
> > [   51.408135] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [   51.408484] CR2: 0000563d1f7ecbe4 CR3: 000000017c212000 CR4:
> > 0000000000750ef0
> > [   51.408877] PKRU: 55555554
> > [   51.409047] BUG: sleeping function called from invalid context at ./
> > include/linux/percpu-rwsem.h:49
> > [   51.409528] in_atomic(): 0, irqs_disabled(): 0, non_block: 0, pid:
> > 76, name: kworker/u17:0
> > [   51.409976] preempt_count: 0, expected: 0
> > [   51.410212] RCU nest depth: 1, expected: 0
> > [   51.410435] INFO: lockdep is turned off.
> > [   51.410648] CPU: 1 UID: 0 PID: 76 Comm: kworker/u17:0 Tainted: G
> > D            6.13.0-rc4-drm-tip-test+ #48
> > [   51.411180] Tainted: [D]=DIE
> > [   51.411338] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
> > BIOS 1.15.0-1 04/01/2014
> > [   51.411859] Workqueue: xe_gt_page_fault_work_queue pf_queue_work_func
> > [xe]
> > [   51.412269] Call Trace:
> > [   51.412404]  <TASK>
> > [   51.412525]  dump_stack_lvl+0x69/0xa0
> > [   51.412724]  __might_resched.cold+0xe5/0x120
> > [   51.412956]  exit_signals+0x1a/0x360
> > [   51.413150]  do_exit+0x122/0xbd0
> > [   51.413328]  ? __pfx_worker_thread+0x10/0x10
> > [   51.413562]  make_task_dead+0x88/0x90
> > [   51.413783]  rewind_stack_and_make_dead+0x16/0x20
> > [   51.414045] RIP: 0000:0x0
> > [   51.414191] Code: Unable to access opcode bytes at 0xffffffffffffffd6.
> > [   51.414595] RSP: 0000:0000000000000000 EFLAGS: 00000000 ORIG_RAX:
> > 0000000000000000
> > [   51.414993] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
> > 0000000000000000
> > [   51.415369] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
> > 0000000000000000
> > [   51.415746] RBP: 0000000000000000 R08: 0000000000000000 R09:
> > 0000000000000000
> > [   51.416123] R10: 0000000000000000 R11: 0000000000000000 R12:
> > 0000000000000000
> > [   51.416501] R13: 0000000000000000 R14: 0000000000000000 R15:
> > 0000000000000000
> > [   51.416899]  </TASK>
> > 
> > 
> > On 12/18/24 1:33 AM, Matthew Brost wrote:
> > > Version 3 of GPU SVM has been promoted to the proper series from an RFC.
> > > Thanks to everyone (especially Sima and Thomas) for their numerous
> > > reviews on revision 1, 2 and for helping to address many design issues.
> > > 
> > > This version has been tested with IGT [1] on PVC, BMG, and LNL. Also
> > > tested with level0 (UMD) PR [2].
> > > 
> > > Major changes in v2:
> > > - Dropped mmap write abuse
> > > - core MM locking and retry loops instead of driver locking to avoid
> > > races
> > > - Removed physical to virtual references
> > > - Embedded structure/ops for drm_gpusvm_devmem
> > > - Fixed mremap and fork issues
> > > - Added DRM pagemap
> > > - Included RFC documentation in the kernel doc
> > > 
> > > Major changes in v3:
> > > - Move GPU SVM and DRM pagemap to DRM level
> > > - Mostly addresses Thomas's feedback, lots of small changes documented
> > >    in each individual patch change log
> > > 
> > > Known issues in v3:
> > > - Check pages still exists, changed to threshold in this version which
> > >    is better but still need to root cause cross process page finding on
> > >    small user allocations.
> > > - Dropped documentation patch, fairly large rewrite and will send out
> > >    independently once finished.
> > > 
> > > Matt
> > > 
> > > [1] https://patchwork.freedesktop.org/series/137545/#rev3
> > > [2] https://github.com/intel/compute-runtime/pull/782
> > > 
> > > Matthew Brost (27):
> > >    drm/xe: Retry BO allocation
> > >    mm/migrate: Add migrate_device_pfns
> > >    mm/migrate: Trylock device page in do_swap_page
> > >    drm/gpusvm: Add support for GPU Shared Virtual Memory
> > >    drm/xe: Select DRM_GPUSVM Kconfig
> > >    drm/xe/uapi: Add DRM_XE_VM_BIND_FLAG_CPU_ADDR_MIRROR flag
> > >    drm/xe: Add SVM init / close / fini to faulting VMs
> > >    drm/xe: Nuke VM's mapping upon close
> > >    drm/xe: Add SVM range invalidation and page fault handler
> > >    drm/gpuvm: Add DRM_GPUVA_OP_DRIVER
> > >    drm/xe: Add (re)bind to SVM page fault handler
> > >    drm/xe: Add SVM garbage collector
> > >    drm/xe: Add unbind to SVM garbage collector
> > >    drm/xe: Do not allow CPU address mirror VMA unbind if the GPU has
> > >      bindings
> > >    drm/xe: Enable CPU address mirror uAPI
> > >    drm/xe: Add migrate layer functions for SVM support
> > >    drm/xe: Add SVM device memory mirroring
> > >    drm/xe: Add drm_gpusvm_devmem to xe_bo
> > >    drm/xe: Add GPUSVM device memory copy vfunc functions
> > >    drm/xe: Add Xe SVM populate_devmem_pfn GPU SVM vfunc
> > >    drm/xe: Add Xe SVM devmem_release GPU SVM vfunc
> > >    drm/xe: Add BO flags required for SVM
> > >    drm/xe: Add SVM VRAM migration
> > >    drm/xe: Basic SVM BO eviction
> > >    drm/xe: Add SVM debug
> > >    drm/xe: Add modparam for SVM notifier size
> > >    drm/xe: Add always_migrate_to_vram modparam
> > > 
> > > Thomas Hellström (3):
> > >    drm/pagemap: Add DRM pagemap
> > >    drm/xe: Add dma_addr res cursor
> > >    drm/xe: Add drm_pagemap ops to SVM
> > > 
> > >   drivers/gpu/drm/Kconfig                     |    8 +
> > >   drivers/gpu/drm/Makefile                    |    1 +
> > >   drivers/gpu/drm/drm_gpusvm.c                | 2220 +++++++++++++++++++
> > >   drivers/gpu/drm/xe/Kconfig                  |   10 +
> > >   drivers/gpu/drm/xe/Makefile                 |    1 +
> > >   drivers/gpu/drm/xe/xe_bo.c                  |   20 +-
> > >   drivers/gpu/drm/xe/xe_bo.h                  |    1 +
> > >   drivers/gpu/drm/xe/xe_bo_types.h            |    4 +
> > >   drivers/gpu/drm/xe/xe_device_types.h        |   15 +
> > >   drivers/gpu/drm/xe/xe_gt_pagefault.c        |   17 +-
> > >   drivers/gpu/drm/xe/xe_gt_tlb_invalidation.c |   24 +
> > >   drivers/gpu/drm/xe/xe_gt_tlb_invalidation.h |    2 +
> > >   drivers/gpu/drm/xe/xe_migrate.c             |  175 ++
> > >   drivers/gpu/drm/xe/xe_migrate.h             |   10 +
> > >   drivers/gpu/drm/xe/xe_module.c              |    7 +
> > >   drivers/gpu/drm/xe/xe_module.h              |    2 +
> > >   drivers/gpu/drm/xe/xe_pt.c                  |  393 +++-
> > >   drivers/gpu/drm/xe/xe_pt.h                  |    5 +
> > >   drivers/gpu/drm/xe/xe_pt_types.h            |    2 +
> > >   drivers/gpu/drm/xe/xe_res_cursor.h          |  116 +-
> > >   drivers/gpu/drm/xe/xe_svm.c                 |  948 ++++++++
> > >   drivers/gpu/drm/xe/xe_svm.h                 |   83 +
> > >   drivers/gpu/drm/xe/xe_tile.c                |    5 +
> > >   drivers/gpu/drm/xe/xe_vm.c                  |  375 +++-
> > >   drivers/gpu/drm/xe/xe_vm.h                  |   15 +-
> > >   drivers/gpu/drm/xe/xe_vm_types.h            |   57 +
> > >   include/drm/drm_gpusvm.h                    |  445 ++++
> > >   include/drm/drm_gpuvm.h                     |    5 +
> > >   include/drm/drm_pagemap.h                   |  103 +
> > >   include/linux/migrate.h                     |    1 +
> > >   include/uapi/drm/xe_drm.h                   |   19 +-
> > >   mm/memory.c                                 |   13 +-
> > >   mm/migrate_device.c                         |  116 +-
> > >   33 files changed, 5061 insertions(+), 157 deletions(-)
> > >   create mode 100644 drivers/gpu/drm/drm_gpusvm.c
> > >   create mode 100644 drivers/gpu/drm/xe/xe_svm.c
> > >   create mode 100644 drivers/gpu/drm/xe/xe_svm.h
> > >   create mode 100644 include/drm/drm_gpusvm.h
> > >   create mode 100644 include/drm/drm_pagemap.h
> > > 
> > 
>