drm/amd/kfd: AMDKFD Sphinx documentation

drm/amd/kfd: AMDKFD Sphinx documentation

[David.Francis]

Make existing documentation comments available through the
sphinx docs, and add new high-level documentation for 
AMDKFD features

[PATCH 1/3] drm/amd/kfd: Update documentation to match kernel standards

[David.Francis]

From: David Francis <David.Francis@amd.com>

The kernel documentation system requires things to be just so
in order to incorporate documentation into the sphinx system.

There were some comments in amdkfd which didn't meet that
standard. Tweak them so they do.

Signed-off-by: David Francis <David.Francis@amd.com>
Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 27 +++++++++++++---
 drivers/gpu/drm/amd/amdkfd/kfd_interrupt.c   |  4 +--
 include/uapi/linux/kfd_ioctl.h               | 33 +++++++++++---------
 3 files changed, 43 insertions(+), 21 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c b/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
index 8aebe408c544..075e4c80dd04 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
@@ -37,11 +37,14 @@
 #include <linux/mman.h>
 #include <linux/processor.h>
 
-/*
+/**
+ * DOC: Memory_Features
+ *
  * The primary memory I/O features being added for revisions of gfxip
  * beyond 7.0 (Kaveri) are:
  *
  * Access to ATC/IOMMU mapped memory w/ associated extension of VA to 48b
+ * ----------------------------------------------------------------------
  *
  * “Flat” shader memory access – These are new shader vector memory
  * operations that do not reference a T#/V# so a “pointer” is what is
@@ -55,6 +58,7 @@
  *
  *
  * System Unified Address - SUA
+ * ----------------------------
  *
  * The standard usage for GPU virtual addresses are that they are mapped by
  * a set of page tables we call GPUVM and these page tables are managed by
@@ -72,13 +76,16 @@
  * There are three fundamental address modes of operation for a given VMID
  * (process) on the GPU:
  *
- *	HSA64 – 64b pointers and the default address space is ATC
- *	HSA32 – 32b pointers and the default address space is ATC
- *	GPUVM – 64b pointers and the default address space is GPUVM (driver
- *		model mode)
+ * HSA64 – 64b pointers and the default address space is ATC
+ *
+ * HSA32 – 32b pointers and the default address space is ATC
+ *
+ * GPUVM – 64b pointers and the default address space is GPUVM (driver
+ * model mode)
  *
  *
  * HSA64 - ATC/IOMMU 64b
+ * ---------------------
  *
  * A 64b pointer in the AMD64/IA64 CPU architecture is not fully utilized
  * by the CPU so an AMD CPU can only access the high area
@@ -99,6 +106,7 @@
  *
  *
  * 64b to 49b Address conversion
+ * -----------------------------
  *
  * Note that there are still significant portions of unused regions (holes)
  * in the 64b address space even for the GPU.  There are several places in
@@ -147,6 +155,7 @@
  *
  *
  * “Spare” aperture (APE1)
+ * -----------------------
  *
  * We use the GPUVM aperture to differentiate ATC vs. GPUVM, but we also use
  * apertures to set the Mtype field for S_LOAD/FLAT_* ops which is input to the
@@ -160,6 +169,7 @@
  *
  *
  * General Aperture definitions and rules
+ * --------------------------------------
  *
  * An aperture register definition consists of a Base, Limit, Mtype, and
  * usually an ATC bit indicating which translation tables that aperture uses.
@@ -187,6 +197,7 @@
  *
  *
  * HSA32 - ATC/IOMMU 32b
+ * ---------------------
  *
  * For HSA32 mode, the pointers are interpreted as 32 bits and use a single GPR
  * instead of two for the S_LOAD and FLAT_* ops. The entire GPUVM space of 40b
@@ -196,6 +207,7 @@
  *
  *
  * GPUVM 64b mode (driver model)
+ * -----------------------------
  *
  * This mode is related to HSA64 in that the difference really is that
  * the default aperture is GPUVM (ATC==0) and not ATC space.
@@ -204,6 +216,7 @@
  *
  *
  * Device Unified Address - DUA
+ * ----------------------------
  *
  * Device unified address (DUA) is the name of the feature that maps the
  * Shared(LDS) memory and Private(Scratch) memory into the overall address
@@ -220,6 +233,7 @@
  *
  *
  * HSA64 Apertures for FLAT_* vector ops
+ * -------------------------------------
  *
  * For HSA64 SUA mode, the Shared and Private apertures are always placed
  * in the hole w/ a limited selection of possible locations. The requests
@@ -231,6 +245,7 @@
  *
  *
  * HSA32 Apertures for FLAT_* vector ops
+ * -------------------------------------
  *
  * In HSA32 mode, the Private and Shared apertures can be placed anywhere
  * in the 32b space except at 0 (Private or Shared Base at zero disables
@@ -239,6 +254,7 @@
  *
  *
  * GPUVM Apertures for FLAT_* vector ops
+ * -------------------------------------
  *
  * In GPUVM mode, the Shared/Private apertures are specified identically
  * to HSA64 mode where they are always in the hole at a limited selection
@@ -246,6 +262,7 @@
  *
  *
  * Aperture Definitions for SUA and DUA
+ * ------------------------------------
  *
  * The interpretation of the aperture register definitions for a given
  * VMID is a function of the “SUA Mode” which is one of HSA64, HSA32, or
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_interrupt.c b/drivers/gpu/drm/amd/amdkfd/kfd_interrupt.c
index a9466d154395..d53ffff2635a 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_interrupt.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_interrupt.c
@@ -21,8 +21,8 @@
  * OTHER DEALINGS IN THE SOFTWARE.
  */
 
-/*
- * KFD Interrupts.
+/**
+ * DOC: KFD_Interrupts
  *
  * AMD GPUs deliver interrupts by pushing an interrupt description onto the
  * interrupt ring and then sending an interrupt. KGD receives the interrupt
diff --git a/include/uapi/linux/kfd_ioctl.h b/include/uapi/linux/kfd_ioctl.h
index afd8ff29c74f..248c4b2f85d6 100644
--- a/include/uapi/linux/kfd_ioctl.h
+++ b/include/uapi/linux/kfd_ioctl.h
@@ -367,7 +367,8 @@ struct kfd_ioctl_acquire_vm_args {
 #define KFD_IOC_ALLOC_MEM_FLAGS_COHERENT	(1 << 26)
 #define KFD_IOC_ALLOC_MEM_FLAGS_UNCACHED	(1 << 25)
 
-/* Allocate memory for later SVM (shared virtual memory) mapping.
+/**
+ * struct kfd_ioctl_alloc_memory_of_gpu_args - Allocate memory for later SVM (shared virtual memory) mapping.
  *
  * @va_addr:     virtual address of the memory to be allocated
  *               all later mappings on all GPUs will use this address
@@ -396,7 +397,8 @@ struct kfd_ioctl_free_memory_of_gpu_args {
 	__u64 handle;		/* to KFD */
 };
 
-/* Map memory to one or more GPUs
+/**
+ * struct kfd_ioctl_map_memory_to_gpu_args - Map memory to one or more GPUs
  *
  * @handle:                memory handle returned by alloc
  * @device_ids_array_ptr:  array of gpu_ids (__u32 per device)
@@ -544,7 +546,7 @@ enum kfd_criu_op {
 };
 
 /**
- * kfd_ioctl_criu_args - Arguments perform CRIU operation
+ * struct kfd_ioctl_criu_args - Arguments perform CRIU operation
  * @devices:		[in/out] User pointer to memory location for devices information.
  * 			This is an array of type kfd_criu_device_bucket.
  * @bos:		[in/out] User pointer to memory location for BOs information
@@ -616,7 +618,7 @@ enum kfd_mmio_remap {
 #define KFD_IOCTL_SVM_FLAG_GPU_ALWAYS_MAPPED   0x00000040
 
 /**
- * kfd_ioctl_svm_op - SVM ioctl operations
+ * enum kfd_ioctl_svm_op - SVM ioctl operations
  *
  * @KFD_IOCTL_SVM_OP_SET_ATTR: Modify one or more attributes
  * @KFD_IOCTL_SVM_OP_GET_ATTR: Query one or more attributes
@@ -626,7 +628,8 @@ enum kfd_ioctl_svm_op {
 	KFD_IOCTL_SVM_OP_GET_ATTR
 };
 
-/** kfd_ioctl_svm_location - Enum for preferred and prefetch locations
+/**
+ * enum kfd_ioctl_svm_location - Enum for preferred and prefetch locations
  *
  * GPU IDs are used to specify GPUs as preferred and prefetch locations.
  * Below definitions are used for system memory or for leaving the preferred
@@ -638,7 +641,7 @@ enum kfd_ioctl_svm_location {
 };
 
 /**
- * kfd_ioctl_svm_attr_type - SVM attribute types
+ * enum kfd_ioctl_svm_attr_type - SVM attribute types
  *
  * @KFD_IOCTL_SVM_ATTR_PREFERRED_LOC: gpuid of the preferred location, 0 for
  *                                    system memory
@@ -667,7 +670,7 @@ enum kfd_ioctl_svm_attr_type {
 };
 
 /**
- * kfd_ioctl_svm_attribute - Attributes as pairs of type and value
+ * struct kfd_ioctl_svm_attribute - Attributes as pairs of type and value
  *
  * The meaning of the @value depends on the attribute type.
  *
@@ -680,14 +683,16 @@ struct kfd_ioctl_svm_attribute {
 };
 
 /**
- * kfd_ioctl_svm_args - Arguments for SVM ioctl
+ * struct kfd_ioctl_svm_args - Arguments for SVM ioctl
+ *
+ * @op: specifies the operation to perform (see enum kfd_ioctl_svm_op).
+ * @start_addr: starting virtual address of SVM range
+ * @size: size of SVM range
  *
- * @op specifies the operation to perform (see enum
- * @kfd_ioctl_svm_op).  @start_addr and @size are common for all
- * operations.
+ * @attrs: a variable number of attributes to set / get
+ * @nattr: the number of attributes.
  *
- * A variable number of attributes can be given in @attrs.
- * @nattr specifies the number of attributes. New attributes can be
+ * New attributes can be
  * added in the future without breaking the ABI. If unknown attributes
  * are given, the function returns -EINVAL.
  *
@@ -727,7 +732,7 @@ struct kfd_ioctl_svm_args {
 };
 
 /**
- * kfd_ioctl_set_xnack_mode_args - Arguments for set_xnack_mode
+ * struct kfd_ioctl_set_xnack_mode_args - Arguments for set_xnack_mode
  *
  * @xnack_enabled:       [in/out] Whether to enable XNACK mode for this process
  *
-- 
2.25.1

[PATCH 2/3] drm/amd/kfd: Add documentation comments to KFD

[David.Francis]

From: David Francis <David.Francis@amd.com>

Add six long comments outlining the basic features of the
driver, to aid new developers.

Signed-off-by: David Francis <David.Francis@amd.com>
Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>
---
 drivers/gpu/drm/amd/amdkfd/kfd_chardev.c  | 74 +++++++++++++++++++++++
 drivers/gpu/drm/amd/amdkfd/kfd_device.c   | 25 ++++++++
 drivers/gpu/drm/amd/amdkfd/kfd_queue.c    | 57 +++++++++++++++++
 drivers/gpu/drm/amd/amdkfd/kfd_topology.c | 33 ++++++++++
 4 files changed, 189 insertions(+)

diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
index 6abfe10229a2..ea25a47b62dc 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
@@ -1031,6 +1031,80 @@ static int kfd_ioctl_get_available_memory(struct file *filep,
 	return 0;
 }
 
+/**
+ * DOC: Memory_Types
+ *
+ * There are many different types of memory that KFD can manage, each with
+ * slightly different interfaces
+ *
+ * VRAM and GTT
+ * ------------
+ *
+ * VRAM and GTT can be allocated with the AMDKFD_IOC_ALLOC_MEMORY_OF_GPU ioctl.
+ * This ioctl returns a handle used to refer to the memory in future kfd ioctls,
+ * as well as a mmap_offset used for mapping the allocation on the CPU. VRAM
+ * memory is located on the GPU, while GTT memory is located in host memory.
+ * Once memory is allocated, it must be mapped with the
+ * AMD_KFD_IOC_MAP_MEMORY_TO_GPU ioctl before the GPU can access it.
+ *
+ * Doorbell and MMIO
+ * -----------------
+ *
+ * Each process is assigned two pages of doorbell memory used to signal that
+ * usermode queues have awaiting packets. AMDKFD_IOC_ALLOC_MEMORY_OF_GPU
+ * associates these pages with a virtual address. They must still be mapped if
+ * the GPU is to access them.
+ *
+ * There is one page of MMIO memory per GPU that is accessible to userspace by
+ * the same means.
+ *
+ * userptr
+ * -------
+ *
+ * userptr memory is user-allocated system memory, alloacted with malloc or
+ * similar. As with doorbell and MMIO memory, AMDKFD_IOC_ALLOC_MEMORY_OF_GPU
+ * does not allocate the memory; instead it registers existing memory for
+ * mapping.
+ *
+ * SVM
+ * ---
+ *
+ * SVM is a different memory-allocation API available on GFX9+. Like userptr
+ * memory, SVM maps existing user-managed memory onto the GPU.
+ *
+ * XNACK is an SVM feature that is disabled by default as it has a performance
+ * cost. When XNACK is enabled, SVM memory can perform recoverable page faults,
+ * allowing KFD to allocate memory without reserving physical address space,
+ * performing the physical allocation only on page fault. With XNACK, SVM
+ * uses the Heterogenous Memory Manager (HMM) to migrate pages back and forth
+ * between the device and the host in reponse to memory pressure and page faults.
+ *
+ * Scratch
+ * -------
+ *
+ * Scratch memory is VRAM memory on a GPU reserved for holding intermediate
+ * values during a shader's execution. A user (usually ROCr) can allocate
+ * scratch memory by allocating VRAM memory and then using the
+ * AMDKFD_IOC_SET_SCRATCH_BACKING_VA ioctl.
+ */
+
+/**
+ * DOC: Memory_Implementation
+ *
+ * The GPU page tables need to be kept in sync with the CPU page tables; if a
+ * page is moved, swapped, or evicted by linux's normal memory manager, a callback
+ * is made into kfd, which must pause hardware access to the memory while the
+ * operation is in progress.
+ *
+ * Compute shaders can cause thrashing if the total memory in use exceeds the
+ * GPU or system's memory limits. Because user command submission is via
+ * usermode queues, with no driver involvement, all memory must be physically
+ * resident at all times (this is different from the graphics approach, which
+ * can swap memory on and off the GPU as needed). KFD prevents overcommitment
+ * of memory by keeping an account of how much memory processes have allocated,
+ * and refusing to allocate beyond a threshold.
+ */
+
 static int kfd_ioctl_alloc_memory_of_gpu(struct file *filep,
 					struct kfd_process *p, void *data)
 {
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
index f5853835f03a..76d1842c9333 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
@@ -37,6 +37,31 @@
 
 #define MQD_SIZE_ALIGNED 768
 
+/**
+ * DOC: Discovery
+ *
+ * There are two phases of initialization and topology discovery in KFD. The
+ * first, module_init occurs when the module is built into the kernel (on boot
+ * or modprobe). The second, device_init, occurs when linux discovers a PCI
+ * device that is an AMD GPU (on boot or hotplug).
+ *
+ * module_init begins when the amdgpu driver is initialized (amdgpu_drv.c),
+ * which calls kfd_init() in kfd_module.c. At this time, the chardev is created
+ * so that ioctls can be submitted, and the topology is queried, creating the
+ * sysfs layout. Some AMD APUs make their topology information
+ * available through a BIOS structure called a CRAT table. If no CRAT table is
+ * found, KFD will make one up with the information available to it. Discrete
+ * GPUs are not discovered at this time; only CPUs and APUs. At this
+ * point, AMDGPU registers itself as a PCIe driver.
+ *
+ * device_init begins when linux finds a device with a PCIe ID matching an entry
+ * amdgpu is registered for. If the device contains compute functionality,
+ * amdgpu will call kgd2kfd_probe() and kgd2kfd_device_init() in kfd_device.c
+ * (kgd2kfd stands for Kernel Graphics Driver to Kernel Fusion Driver) to set up
+ * shared resources such as non-compute doorbells and add the new device to the
+ * topology.
+ */
+
 /*
  * kfd_locked is used to lock the kfd driver during suspend or reset
  * once locked, kfd driver will stop any further GPU execution.
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_queue.c b/drivers/gpu/drm/amd/amdkfd/kfd_queue.c
index 0f6992b1895c..3c1a2be18d4c 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_queue.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_queue.c
@@ -25,6 +25,63 @@
 #include <linux/slab.h>
 #include "kfd_priv.h"
 
+/**
+ * DOC: Queue_Interface
+ *
+ * A process can create queues with the ioctl AMDKFD_IOC_CREATE_QUEUE, which
+ * returns a queue id used as a handle, the addresses of the read and write
+ * pointers, and the doorbell location. Up to 256 processes can have queues,
+ * and each process can have up to 1024 queues.
+ *
+ * A doorbell is a 64-bit memory-mapped register on the GPU that a
+ * process can write to to signal that the corresponding queue has packets
+ * waiting in it.
+ *
+ * A queue can be either an compute queue, used for computation, or an SDMA queue,
+ * used for data transfers.
+ *
+ * Most HSA queues take commands in the form of 64-byte AQL packets. Most
+ * commonly, this will be a kernel dispatch packet containing a pointer to the
+ * kernel to be executed. A kernel is a small program that performs an
+ * elementary operation such as a vector sum, matrix multiplication, or
+ * scatter-gather operation. A single user program may be made up of many
+ * kernels. Other AQL packets include barrier packets used for synchronization
+ * between shaders and PM4_IB packets that flush the cache, used for profiling.
+ * Packets in the same queue will begin execution in order, but can run
+ * concurrently. DMA queues are similar, but use a different packet format.
+ *
+ * A queue contains a ringbuffer with read and write pointers used to submit
+ * packets (the size of the ringbuffer is specified when the queue is
+ * created). To write to a queue, a process first atomically moves the
+ * write pointer forward. Then, it writes each of the packets to the buffer,
+ * leaving the headers empty so that if hardware attempts to consume the packets
+ * at this point, it will find them invalid. Then it writes the headers and
+ * signals the doorbell.
+ *
+ * In addition to the user mode queues described here, there are kernel mode
+ * queues used internally by KFD to communicate with various elements of the
+ * hardware; these work similarly.
+ */
+
+/**
+ * DOC: Queue_Implementation
+ *
+ * Although there may be thousands of queues attached to processes, the 
+ * hardware engines have a limited number of queue slots, usually 32 or fewer 
+ * compute and 10 or fewer DMA per GPU. The hardware will detect doorbell 
+ * signals directly only from queues mapped to an engine. The hardware
+ * scheduler will periodically poll for unmapped queues with work waiting and
+ * map them, unmapping empty queues to make room.
+ *
+ * Compute shaders can be interrupted partway through execution by the hardware
+ * scheduler. In that case, the shader's current state will be saved to a
+ * usermode buffer so it can be restored at a later time. These buffers are
+ * large, and each queue requires its own buffer, so queues are memory-expensive
+ * objects. The context save/restore process is initiated with a trap handler on
+ * the GPU. The trap handler itself is located in the driver, written in SP3
+ * assembly code.
+ */
+
 void print_queue_properties(struct queue_properties *q)
 {
 	if (!q)
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
index 25990bec600d..8c2910b98ece 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
@@ -42,6 +42,39 @@
 #include "amdgpu_ras.h"
 #include "amdgpu.h"
 
+/**
+ * DOC: Topology
+ *
+ * The GPU component of an APU or iGPU, or a discrete GPU, is a GPU device. The
+ * CPU is also a device (known as "System" or "Host" to KFD).
+ *
+ * A node is a memory domain. Most devices are a single node, but certain GPUs
+ * may contain multiple nodes, depending on how they are configured. Each GPU
+ * node has its own L2 data cache.
+ *
+ * A GPU contains multiple Shader Engines (SEs). Each shader engine has its own
+ * sub-scheduler to divide up work within the SE.
+ *
+ * A Shader Engine contains multiple Compute Units (CUs). All processing in a
+ * CU will share all caches, such that two threads running in the same
+ * CU will be able to easily communicate and synchronize
+ *
+ * A Compute Unit contains multiple Single Instruction Multiple Data units
+ * (SIMDs). A SIMD can run programs that perform operations on 32- or 64-element
+ * vectors. A program running on a single SIMD is called a wavefront.
+ *
+ * In addition to the processing capabilities, the topology also includes the
+ * IO links between nodes. GPU nodes may be connected to each other or the
+ * system via XGMI or PCIe links.
+ *
+ * Topology information is available through sysfs at
+ * /sys/devices/virtual/kfd/kfd/topology or through symbolic link at
+ * /sys/class/kfd/kfd/topology. The generation_id field in that directory is
+ * incremented each time the topology is updated. To ensure a consistent view of
+ * the topology, user programs should read generation_id before and after
+ * checking the topology, and retry if the values are not the same.
+ */
+
 /* topology_device_list - Master list of all topology devices */
 static struct list_head topology_device_list;
 static struct kfd_system_properties sys_props;
-- 
2.25.1

[PATCH 3/3] drm/amd/kfd: Add sphinx documentation for amdkfd

[David.Francis]

From: David Francis <David.Francis@amd.com>

Add a new series of documentation entries in the AMDGPU
section describing AMDKFD, the compute driver for AMD
GPUs

Signed-off-by: David Francis <David.Francis@amd.com>
Reviewed-by: Felix Kuehling <Felix.Kuehling@amd.com>
---
 Documentation/gpu/amdgpu/index.rst          |  1 +
 Documentation/gpu/amdgpu/kfd/index.rst      | 14 +++++++++
 Documentation/gpu/amdgpu/kfd/interrupts.rst | 19 ++++++++++++
 Documentation/gpu/amdgpu/kfd/memory.rst     | 33 +++++++++++++++++++++
 Documentation/gpu/amdgpu/kfd/queues.rst     | 21 +++++++++++++
 Documentation/gpu/amdgpu/kfd/topology.rst   | 13 ++++++++
 6 files changed, 101 insertions(+)
 create mode 100644 Documentation/gpu/amdgpu/kfd/index.rst
 create mode 100644 Documentation/gpu/amdgpu/kfd/interrupts.rst
 create mode 100644 Documentation/gpu/amdgpu/kfd/memory.rst
 create mode 100644 Documentation/gpu/amdgpu/kfd/queues.rst
 create mode 100644 Documentation/gpu/amdgpu/kfd/topology.rst

diff --git a/Documentation/gpu/amdgpu/index.rst b/Documentation/gpu/amdgpu/index.rst
index a24e1cfa7407..e1152c40d78b 100644
--- a/Documentation/gpu/amdgpu/index.rst
+++ b/Documentation/gpu/amdgpu/index.rst
@@ -10,6 +10,7 @@ Next (GCN) architecture.
    module-parameters
    driver-core
    display/index
+   kfd/index
    xgmi
    ras
    thermal
diff --git a/Documentation/gpu/amdgpu/kfd/index.rst b/Documentation/gpu/amdgpu/kfd/index.rst
new file mode 100644
index 000000000000..c9bfc52c547a
--- /dev/null
+++ b/Documentation/gpu/amdgpu/kfd/index.rst
@@ -0,0 +1,14 @@
+===========================================
+drm/amd/amdkfd - Kernel Fusion Driver (KFD)
+===========================================
+
+KFD is the non-graphics component of AMDGPU. It allows users to run
+their own programs (called compute shaders) on
+AMD GPUs, including discrete GPUs, integrated GPUs, and APUs.
+
+.. toctree::
+
+   topology.rst
+   memory.rst
+   queues.rst
+   interrupts.rst
\ No newline at end of file
diff --git a/Documentation/gpu/amdgpu/kfd/interrupts.rst b/Documentation/gpu/amdgpu/kfd/interrupts.rst
new file mode 100644
index 000000000000..f59b985ca3f1
--- /dev/null
+++ b/Documentation/gpu/amdgpu/kfd/interrupts.rst
@@ -0,0 +1,19 @@
+======================
+Interrupts and Signals
+======================
+
+Signals
+-------
+
+Signals are locations in memory used for communication between CPUs
+and GPUs. Uses of signals include indicating that a kernel has completed
+and indicating that more scratch memory is rqeuired.
+
+CPUs and GPUs can poll signals to await their signalling. Processes can also
+set up an event, which will trigger a CPU interrupt when a signal arrives.
+
+Interrupts
+----------
+
+.. kernel-doc:: drivers/gpu/drm/amd/amdkfd/kfd_interrupt.c
+   :doc: KFD_Interrupts
\ No newline at end of file
diff --git a/Documentation/gpu/amdgpu/kfd/memory.rst b/Documentation/gpu/amdgpu/kfd/memory.rst
new file mode 100644
index 000000000000..2d6b3724d3fb
--- /dev/null
+++ b/Documentation/gpu/amdgpu/kfd/memory.rst
@@ -0,0 +1,33 @@
+=============================
+Memory Allocation and Mapping
+=============================
+
+AMD GPUs contain their own page tables used by the GPU's processors; memory must
+be mapped on these page tables before the GPU can access it. The GPUs also
+contain VRAM memory of their own, which can be allocated. By mapping memory on
+both the CPU and GPU, processes can load data onto or off of the GPU.
+
+Interface
+=========
+
+.. kernel-doc:: drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
+   :doc: Memory_Types
+
+Implementation
+==============
+
+.. kernel-doc:: drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
+   :doc: Memory_Implementation
+
+Memory Features
+===============
+
+.. kernel-doc:: drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c
+   :doc: Memory_Features
+
+
+ioctls
+======
+
+.. kernel-doc:: include/uapi/linux/kfd_ioctl.h
+   :identifiers: kfd_ioctl_alloc_memory_of_gpu_args kfd_ioctl_map_memory_to_gpu_args kfd_ioctl_svm_args kfd_ioctl_set_xnack_mode_args
\ No newline at end of file
diff --git a/Documentation/gpu/amdgpu/kfd/queues.rst b/Documentation/gpu/amdgpu/kfd/queues.rst
new file mode 100644
index 000000000000..dbd18029e0c8
--- /dev/null
+++ b/Documentation/gpu/amdgpu/kfd/queues.rst
@@ -0,0 +1,21 @@
+===============
+Usermode Queues
+===============
+
+Queues (HSA queues and DMA queues) are the means by which users may submit
+programs to be run on the GPU. Once a queue is set up, users can submit commands
+to the hardware through it with no driver involvement. This is different from
+the way that graphics shaders are loaded, which involves an ioctl each time.
+
+Interface
+=========
+
+.. kernel-doc:: drivers/gpu/drm/amd/amdkfd/kfd_queue.c
+   :doc: Queue_Interface
+
+Implementation
+==============
+
+.. kernel-doc:: drivers/gpu/drm/amd/amdkfd/kfd_queue.c
+   :doc: Queue_Implementation
+
diff --git a/Documentation/gpu/amdgpu/kfd/topology.rst b/Documentation/gpu/amdgpu/kfd/topology.rst
new file mode 100644
index 000000000000..309f2d01fed9
--- /dev/null
+++ b/Documentation/gpu/amdgpu/kfd/topology.rst
@@ -0,0 +1,13 @@
+====================
+AMD Compute Topology
+====================
+
+.. kernel-doc:: drivers/gpu/drm/amd/amdkfd/kfd_topology.c
+   :doc: Topology
+
+
+Discovery and Initialization
+============================
+
+.. kernel-doc:: drivers/gpu/drm/amd/amdkfd/kfd_device.c
+   :doc: Discovery
-- 
2.25.1