From: Felix Kuehling <felix.kuehling@amd.com>
To: Zhu Lingshan <lingshan.zhu@amd.com>, alexander.deucher@amd.com
Cc: ray.huang@amd.com, amd-gfx@lists.freedesktop.org
Subject: Re: [PATCH V6 00/18] amdkfd: Implement kfd multiple contexts
Date: Thu, 30 Oct 2025 17:15:33 -0400 [thread overview]
Message-ID: <9598909f-1d07-4df8-a09c-ec3e25727b72@amd.com> (raw)
In-Reply-To: <20251022073043.13009-1-lingshan.zhu@amd.com>
https://github.com/AMD-ROCm-Internal/rocm-systems/pull/78
https://github.com/AMD-ROCm-Internal/rocm-systems/pull/110
This repository is not publicly accessible. Can these user mode patches be pushed to a publicly accessible repository?
Regards,
Felix
On 2025-10-22 03:30, Zhu Lingshan wrote:
> Currently kfd manages kfd_process in a one context (kfd_process)
> per program manner, thus each user space program
> only onws one kfd context (kfd_process).
>
> This model works fine for most of the programs, but imperfect
> for a hypervisor like QEMU. Because all programs in the guest
> user space share the same only one kfd context, which is
> problematic, including but not limited to:
>
> As illustrated in Figure 1, all guest user space programs share the same fd of /dev/kfd
> and the same kfd_process, and the same PASID leading to the same
> GPU_VM address space. Therefore the IOVA range of each
> guest user space programs are not isolated,
> they can attack each other through GPU DMA.
>
>
> +----------------------------------------------------------------------------------+
> | |
> | +-----------+ +-----------+ +------------+ +------------+ |
> | | | | | | | | | |
> | | Program 1 | | Program 2 | | Program 3 | | Program N | |
> | | | | | | | | | |
> | +----+------+ +--------+--+ +--+---------+ +-----+------+ |
> | | | | | |
> | | | | | Guest |
> | | | | | |
> +-------+----------------------+------------+----------------------+---------------+
> | | | |
> | | | |
> | | | |
> | | | |
> | +--+------------+---+ |
> | | file descriptor | |
> +-------------------+ of /dev/kfd +------------------+
> | opened by QEMU |
> | |
> +---------+---------+ User Space
> | QEMU
> |
> ---------------------------------------+-----------------------------------------------------
> | Kernel Space
> | KFD Module
> |
> +--------+--------+
> | |
> | kfd_process |<------------------The only one KFD context
> | |
> +--------+--------+
> |
> +--------+--------+
> | PASID |
> +--------+--------+
> |
> +--------+--------+
> | GPU_VM |
> +-----------------+
>
> Fiture 1
>
>
> This series implements a multiple contexts solution:
> - Allow each program to create and hold multiple contexts (kfd processes)
> - Each context has its own fd of /dev/kfd and an exclusive kfd_process,
> which is a secondary kfd context. So that PASID/GPU VM isolates their IOVA address spaces.
> Therefore, they can not attack each other through GPU DMA.
>
> The design is illustrated in Figure 2 below:
>
> +---------------------------------------------------------------------------------------------------------+
> | |
> | |
> | |
> | +----------------------------------------------------------------------------------+ |
> | | | |
> | | +-----------+ +-----------+ +-----------+ +-----------+ | |
> | | | | | | | | | | | |
> | | | Program 1 | | Program 2 | | Program 3 | | Program N | | |
> | | | | | | | | | | | |
> | | +-----+-----+ +-----+-----+ +-----+-----+ +-----+-----+ | |
> | | | | | | | |
> | | | | | | Guest | |
> | | | | | | | |
> | +-------+------------------+-----------------+----------------+--------------------+ |
> | | | | | QEMU |
> | | | | | |
> +---------------+------------------+-----------------+----------------+--------------------------+--------+
> | | | | |
> | | | | |
> | | | | |
> +---+----+ +---+----+ +---+----+ +---+----+ +---+-----+
> | | | | | | | | | Primary |
> | FD 1 | | FD 2 | | FD 3 | | FD 4 | | FD |
> | | | | | | | | | |
> +---+----+ +---+----+ +---+----+ +----+---+ +----+----+
> | | | | | User Space
> | | | | |
> -------------------+------------------+-----------------+-----------------+--------------------------+----------------------------
> | | | | | Kernel SPace
> | | | | |
> | | | | |
> +--------------------------------------------------------------------------------------------------------------------------+
> | +------+------+ +------+------+ +------+------+ +------+------+ +------+------+ |
> | | Secondary | | Secondary | | Secondary | | Secondary | | Primary | KFD Module |
> | |kfd_process 1| |kfd_process 2| |kfd_process 3| |kfd_process 4| | kfd_process | |
> | | | | | | | | | | | |
> | +------+------+ +------+------+ +------+------+ +------+------+ +------+------+ |
> | | | | | | |
> | +------+------+ +------+------+ +------+------+ +------+------+ +------+------+ |
> | | PASID | | PASID | | PASID | | PASID | | PASID | |
> | +------+------+ +------+------+ +------+------+ +------+------+ +------+------+ |
> | | | | | | |
> | | | | | | |
> | +------+------+ +------+------+ +------+------+ +------+------+ +------+------+ |
> | | GPU_VM | | GPU_VM | | GPU_VM | | GPU_VM | | GPU_VM | |
> | +-------------+ +-------------+ +-------------+ +-------------+ +-------------+ |
> | |
> +--------------------------------------------------------------------------------------------------------------------------+
>
> Figure 2
>
> The relevant reference user space rocm changes could be found at:
> https://github.com/AMD-ROCm-Internal/rocm-systems/pull/78
> https://github.com/AMD-ROCm-Internal/rocm-systems/pull/110
>
> Changes from V5:
> 1) change upper limit of ida_alloc_range() to KFD_CONTEXT_ID_PRIMARY - 1
> 2) in set_debug_trap ioctl, check whether the target kfd_process is a
> primary one after checking validity of the target pointer
> 3) in set_debug_trap ioctl, check whether the kfd_process which
> the ioctl is processed on is a primary one
> 4) in AMDKFD_IOC_CREATE_PROCESS ioctl, place filep->private_data
> and related code in the critical zone that protected by kfd_processes_mutex
> 5) add Felix's reviewed-by tag to the series except patch 8,17 and 18
>
> Changes from V4:
> 1) rename process_id to context_id in struct kfd_process
> 2) remove primary flag in struct kfd_process
> 3) reject set_debug_trap ioctl request when
> the target kfd_process is non-primary
> 4) other than default 0, assign context_id 0xFFFF to the primary kfd process
>
> Zhu Lingshan (18):
> amdkfd: enlarge the hashtable of kfd_process
> amdkfd: mark the first kfd_process as the primary one
> amdkfd: find_process_by_mm always return the primary context
> amdkfd: Introduce kfd_create_process_sysfs as a separate function
> amdkfd: destroy kfd secondary contexts through fd close
> amdkfd: process svm ioctl only on the primary kfd process
> amdkfd: process USERPTR allocation only on the primary kfd process
> amdkfd: identify a secondary kfd process by its id
> amdkfd: find kfd_process by filep->private_data in kfd_mmap
> amdkfd: remove DIQ support
> amdkfd: process pointer of a HIQ should be NULL
> amdkfd: remove test_kq
> amdkfd: introduce new helper kfd_lookup_process_by_id
> amdkfd: record kfd context id into kfd process_info
> amdkfd: record kfd context id in amdkfd_fence
> amdkfd: fence handler evict and restore a kfd process by its context
> id
> amdkfd: process debug trap ioctl only on a primary context
> amdkfd: introduce new ioctl AMDKFD_IOC_CREATE_PROCESS
>
> drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h | 8 +-
> .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_fence.c | 10 +-
> .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 8 +-
> drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 90 ++++++-
> drivers/gpu/drm/amd/amdkfd/kfd_device.c | 7 +-
> .../drm/amd/amdkfd/kfd_device_queue_manager.c | 6 +-
> drivers/gpu/drm/amd/amdkfd/kfd_kernel_queue.c | 61 +----
> .../drm/amd/amdkfd/kfd_packet_manager_v9.c | 4 -
> .../drm/amd/amdkfd/kfd_packet_manager_vi.c | 4 -
> drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 18 +-
> drivers/gpu/drm/amd/amdkfd/kfd_process.c | 239 +++++++++++++-----
> .../amd/amdkfd/kfd_process_queue_manager.c | 39 +--
> drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 2 +-
> include/uapi/linux/kfd_ioctl.h | 8 +-
> 14 files changed, 304 insertions(+), 200 deletions(-)
>
prev parent reply other threads:[~2025-10-30 21:15 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-10-22 7:30 [PATCH V6 00/18] amdkfd: Implement kfd multiple contexts Zhu Lingshan
2025-10-22 7:30 ` [PATCH V6 01/18] amdkfd: enlarge the hashtable of kfd_process Zhu Lingshan
2025-10-22 7:30 ` [PATCH V6 02/18] amdkfd: mark the first kfd_process as the primary one Zhu Lingshan
2025-10-22 7:30 ` [PATCH V6 03/18] amdkfd: find_process_by_mm always return the primary context Zhu Lingshan
2025-10-22 7:30 ` [PATCH V6 04/18] amdkfd: Introduce kfd_create_process_sysfs as a separate function Zhu Lingshan
2025-10-22 7:30 ` [PATCH V6 05/18] amdkfd: destroy kfd secondary contexts through fd close Zhu Lingshan
2025-10-22 7:30 ` [PATCH V6 06/18] amdkfd: process svm ioctl only on the primary kfd process Zhu Lingshan
2025-10-22 7:30 ` [PATCH V6 07/18] amdkfd: process USERPTR allocation " Zhu Lingshan
2025-10-22 7:30 ` [PATCH V6 08/18] amdkfd: identify a secondary kfd process by its id Zhu Lingshan
2025-10-30 21:17 ` Felix Kuehling
2025-10-22 7:30 ` [PATCH V6 09/18] amdkfd: find kfd_process by filep->private_data in kfd_mmap Zhu Lingshan
2025-10-22 7:30 ` [PATCH V6 10/18] amdkfd: remove DIQ support Zhu Lingshan
2025-10-22 7:30 ` [PATCH V6 11/18] amdkfd: process pointer of a HIQ should be NULL Zhu Lingshan
2025-10-22 7:30 ` [PATCH V6 12/18] amdkfd: remove test_kq Zhu Lingshan
2025-10-22 7:30 ` [PATCH V6 13/18] amdkfd: introduce new helper kfd_lookup_process_by_id Zhu Lingshan
2025-10-22 7:30 ` [PATCH V6 14/18] amdkfd: record kfd context id into kfd process_info Zhu Lingshan
2025-10-22 7:30 ` [PATCH V6 15/18] amdkfd: record kfd context id in amdkfd_fence Zhu Lingshan
2025-10-22 7:30 ` [PATCH V6 16/18] amdkfd: fence handler evict and restore a kfd process by its context id Zhu Lingshan
2025-10-22 7:30 ` [PATCH V6 17/18] amdkfd: process debug trap ioctl only on a primary context Zhu Lingshan
2025-10-30 21:18 ` Felix Kuehling
2025-10-22 7:30 ` [PATCH V6 18/18] amdkfd: introduce new ioctl AMDKFD_IOC_CREATE_PROCESS Zhu Lingshan
2025-10-30 21:28 ` Felix Kuehling
2025-10-30 21:15 ` Felix Kuehling [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=9598909f-1d07-4df8-a09c-ec3e25727b72@amd.com \
--to=felix.kuehling@amd.com \
--cc=alexander.deucher@amd.com \
--cc=amd-gfx@lists.freedesktop.org \
--cc=lingshan.zhu@amd.com \
--cc=ray.huang@amd.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox