[Intel-gfx] [PATCH 0/3] drm/doc/rfc: i915 VM_BIND feature design + uapi

[Intel-gfx] [PATCH 0/3] drm/doc/rfc: i915 VM_BIND feature design + uapi

[Niranjana Vishwanathapura]

This is the i915 driver VM_BIND feature design RFC patch series along
with the required uapi definition and description of intended use cases.

This series is an updated version of the below RFC series. It address
the review feedback by adding execbuf3 ioctl for vm_bind, adding
multiple queues support for vm_bind/unbind ioctls and some formatting
and documentation updates.
https://www.spinics.net/lists/dri-devel/msg347731.html

Signed-off-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>

Niranjana Vishwanathapura (3):
  drm/doc/rfc: VM_BIND feature design document
  drm/i915: Update i915 uapi documentation
  drm/doc/rfc: VM_BIND uapi definition

 Documentation/driver-api/dma-buf.rst   |   2 +
 Documentation/gpu/rfc/i915_vm_bind.h   | 490 +++++++++++++++++++++++++
 Documentation/gpu/rfc/i915_vm_bind.rst | 309 ++++++++++++++++
 Documentation/gpu/rfc/index.rst        |   4 +
 include/uapi/drm/i915_drm.h            | 203 +++++++---
 5 files changed, 963 insertions(+), 45 deletions(-)
 create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
 create mode 100644 Documentation/gpu/rfc/i915_vm_bind.rst

-- 
2.21.0.rc0.32.g243a4c7e27

[Intel-gfx] [PATCH 1/3] drm/doc/rfc: VM_BIND feature design document

[Niranjana Vishwanathapura]

VM_BIND design document with description of intended use cases.

Signed-off-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>
---
 Documentation/driver-api/dma-buf.rst   |   2 +
 Documentation/gpu/rfc/i915_vm_bind.rst | 309 +++++++++++++++++++++++++
 Documentation/gpu/rfc/index.rst        |   4 +
 3 files changed, 315 insertions(+)
 create mode 100644 Documentation/gpu/rfc/i915_vm_bind.rst

diff --git a/Documentation/driver-api/dma-buf.rst b/Documentation/driver-api/dma-buf.rst
index 36a76cbe9095..64cb924ec5bb 100644
--- a/Documentation/driver-api/dma-buf.rst
+++ b/Documentation/driver-api/dma-buf.rst
@@ -200,6 +200,8 @@ DMA Fence uABI/Sync File
 .. kernel-doc:: include/linux/sync_file.h
    :internal:
 
+.. _indefinite_dma_fences:
+
 Indefinite DMA Fences
 ~~~~~~~~~~~~~~~~~~~~~
 
diff --git a/Documentation/gpu/rfc/i915_vm_bind.rst b/Documentation/gpu/rfc/i915_vm_bind.rst
new file mode 100644
index 000000000000..8f911298dda5
--- /dev/null
+++ b/Documentation/gpu/rfc/i915_vm_bind.rst
@@ -0,0 +1,309 @@
+==========================================
+I915 VM_BIND feature design and use cases
+==========================================
+
+VM_BIND feature
+================
+DRM_I915_GEM_VM_BIND/UNBIND ioctls allows UMD to bind/unbind GEM buffer
+objects (BOs) or sections of a BOs at specified GPU virtual addresses on a
+specified address space (VM). These mappings (also referred to as persistent
+mappings) will be persistent across multiple GPU submissions (execbuf calls)
+issued by the UMD, without user having to provide a list of all required
+mappings during each submission (as required by older execbuf mode).
+
+VM_BIND/UNBIND ioctls will support 'in' and 'out' fences to allow userpace
+to specify how the binding/unbinding should sync with other operations
+like the GPU job submission. These fences will be timeline 'drm_syncobj's
+for non-Compute contexts (See struct drm_i915_vm_bind_ext_timeline_fences).
+For Compute contexts, they will be user/memory fences (See struct
+drm_i915_vm_bind_ext_user_fence).
+
+VM_BIND feature is advertised to user via I915_PARAM_HAS_VM_BIND.
+User has to opt-in for VM_BIND mode of binding for an address space (VM)
+during VM creation time via I915_VM_CREATE_FLAGS_USE_VM_BIND extension.
+
+VM_BIND/UNBIND ioctls support multiple queues and the binding/unbinding
+operations submitted are completed asynchronously after any specified 'in'
+fences are signaled. The operations submitted on a queue are completed in
+the order of submission. But operations submitted on different queues can
+get completed out of the submission order. Any 'out' fences returned by the
+VM_BIND/UNBIND ioctls are signaled once the operation is complete. Due to
+serialization, completion of an operation will also indicate that all
+previous operations on that queue are also complete.
+
+VM_BIND features include:
+
+* Multiple Virtual Address (VA) mappings can map to the same physical pages
+  of an object (aliasing).
+* VA mapping can map to a partial section of the BO (partial binding).
+* Support capture of persistent mappings in the dump upon GPU error.
+* TLB is flushed upon unbind completion. Batching of TLB flushes in some
+  use cases will be helpful.
+* Multiple queues where each can make progress independent of the other.
+* Asynchronous vm_bind and vm_unbind support with 'in' and 'out' fences.
+* Support for userptr gem objects (no special uapi is required for this).
+
+Execbuf ioctl in VM_BIND mode
+-------------------------------
+A VM in VM_BIND mode will not support older execbuf mode of binding.
+The execbuf ioctl handling in VM_BIND mode differs significantly from the
+older execbuf2 ioctl (See struct drm_i915_gem_execbuffer2).
+Hence, a new execbuf3 ioctl has been added to support VM_BIND mode. (See
+struct drm_i915_gem_execbuffer3). The execbuf3 ioctl will not accept any
+execlist. Hence, no support for implicit sync. It is expected that the below
+work will be able to support requirements of object dependency setting in all
+use cases:
+
+"dma-buf: Add an API for exporting sync files"
+(https://lwn.net/Articles/859290/)
+
+The execbuf3 ioctl directly specifies the batch addresses instead of as
+object handles as in execbuf2 ioctl. The execbuf3 ioctl will also not
+support many of the older features like in/out/submit fences, fence array,
+default gem context and many more (see struct drm_i915_gem_execbuffer3).
+
+In VM_BIND mode, VA allocation is completely managed by the user instead of
+the i915 driver. Hence all VA assignment, eviction are not applicable in
+VM_BIND mode. Also, for determining object activeness, VM_BIND mode will not
+be using the i915_vma active reference tracking. It will instead use dma-resv
+object for that (See `VM_BIND dma_resv usage`_).
+
+So, a lot of existing code supporting execbuf2 ioctl, like relocations, VA
+evictions, vma lookup table, implicit sync, vma active reference tracking etc.,
+are not applicable for execbuf3 ioctl. Hence, all execbuf3 specific handling
+should be in a separate file and only functionalities common to these ioctls
+can be the shared code if possible.
+
+VM_PRIVATE objects
+-------------------
+By default, BOs can be mapped on multiple VMs and can also be dma-buf
+exported. Hence these BOs are referred to as Shared BOs.
+During each execbuf submission, the request fence must be added to the
+dma-resv fence list of all shared BOs mapped on the VM.
+
+VM_BIND feature introduces an optimization where user can create BO which
+is private to a specified VM via I915_GEM_CREATE_EXT_VM_PRIVATE flag during
+BO creation. Unlike Shared BOs, these VM private BOs can only be mapped on
+the VM they are private to and can't be dma-buf exported.
+All private BOs of a VM share the dma-resv object. Hence during each execbuf
+submission, they need only one dma-resv fence list updated. Thus, the fast
+path (where required mappings are already bound) submission latency is O(1)
+w.r.t the number of VM private BOs.
+
+VM_BIND locking hirarchy
+-------------------------
+The locking design here supports the older (execlist based) execbuf mode, the
+newer VM_BIND mode, the VM_BIND mode with GPU page faults and possible future
+system allocator support (See `Shared Virtual Memory (SVM) support`_).
+The older execbuf mode and the newer VM_BIND mode without page faults manages
+residency of backing storage using dma_fence. The VM_BIND mode with page faults
+and the system allocator support do not use any dma_fence at all.
+
+VM_BIND locking order is as below.
+
+1) Lock-A: A vm_bind mutex will protect vm_bind lists. This lock is taken in
+   vm_bind/vm_unbind ioctl calls, in the execbuf path and while releasing the
+   mapping.
+
+   In future, when GPU page faults are supported, we can potentially use a
+   rwsem instead, so that multiple page fault handlers can take the read side
+   lock to lookup the mapping and hence can run in parallel.
+   The older execbuf mode of binding do not need this lock.
+
+2) Lock-B: The object's dma-resv lock will protect i915_vma state and needs to
+   be held while binding/unbinding a vma in the async worker and while updating
+   dma-resv fence list of an object. Note that private BOs of a VM will all
+   share a dma-resv object.
+
+   The future system allocator support will use the HMM prescribed locking
+   instead.
+
+3) Lock-C: Spinlock/s to protect some of the VM's lists like the list of
+   invalidated vmas (due to eviction and userptr invalidation) etc.
+
+When GPU page faults are supported, the execbuf path do not take any of these
+locks. There we will simply smash the new batch buffer address into the ring and
+then tell the scheduler run that. The lock taking only happens from the page
+fault handler, where we take lock-A in read mode, whichever lock-B we need to
+find the backing storage (dma_resv lock for gem objects, and hmm/core mm for
+system allocator) and some additional locks (lock-D) for taking care of page
+table races. Page fault mode should not need to ever manipulate the vm lists,
+so won't ever need lock-C.
+
+VM_BIND LRU handling
+---------------------
+We need to ensure VM_BIND mapped objects are properly LRU tagged to avoid
+performance degradation. We will also need support for bulk LRU movement of
+VM_BIND objects to avoid additional latencies in execbuf path.
+
+The page table pages are similar to VM_BIND mapped objects (See
+`Evictable page table allocations`_) and are maintained per VM and needs to
+be pinned in memory when VM is made active (ie., upon an execbuf call with
+that VM). So, bulk LRU movement of page table pages is also needed.
+
+The i915 shrinker LRU has stopped being an LRU. So, it should also be moved
+over to the ttm LRU in some fashion to make sure we once again have a reasonable
+and consistent memory aging and reclaim architecture.
+
+VM_BIND dma_resv usage
+-----------------------
+Fences needs to be added to all VM_BIND mapped objects. During each execbuf
+submission, they are added with DMA_RESV_USAGE_BOOKKEEP usage to prevent
+over sync (See enum dma_resv_usage). One can override it with either
+DMA_RESV_USAGE_READ or DMA_RESV_USAGE_WRITE usage during object dependency
+setting (either through explicit or implicit mechanism).
+
+When vm_bind is called for a non-private object while the VM is already
+active, the fences need to be copied from VM's shared dma-resv object
+(common to all private objects of the VM) to this non-private object.
+If this results in performance degradation, then some optimization will
+be needed here. This is not a problem for VM's private objects as they use
+shared dma-resv object which is always updated on each execbuf submission.
+
+Also, in VM_BIND mode, use dma-resv apis for determining object activeness
+(See dma_resv_test_signaled() and dma_resv_wait_timeout()) and do not use the
+older i915_vma active reference tracking which is deprecated. This should be
+easier to get it working with the current TTM backend. We can remove the
+i915_vma active reference tracking fully while supporting TTM backend for igfx.
+
+Evictable page table allocations
+---------------------------------
+Make pagetable allocations evictable and manage them similar to VM_BIND
+mapped objects. Page table pages are similar to persistent mappings of a
+VM (difference here are that the page table pages will not have an i915_vma
+structure and after swapping pages back in, parent page link needs to be
+updated).
+
+Mesa use case
+--------------
+VM_BIND can potentially reduce the CPU overhead in Mesa (both Vulkan and Iris),
+hence improving performance of CPU-bound applications. It also allows us to
+implement Vulkan's Sparse Resources. With increasing GPU hardware performance,
+reducing CPU overhead becomes more impactful.
+
+
+VM_BIND Compute support
+========================
+
+User/Memory Fence
+------------------
+The idea is to take a user specified virtual address and install an interrupt
+handler to wake up the current task when the memory location passes the user
+supplied filter. User/Memory fence is a <address, value> pair. To signal the
+user fence, specified value will be written at the specified virtual address
+and wakeup the waiting process. User can wait on a user fence with the
+gem_wait_user_fence ioctl.
+
+It also allows the user to emit their own MI_FLUSH/PIPE_CONTROL notify
+interrupt within their batches after updating the value to have sub-batch
+precision on the wakeup. Each batch can signal a user fence to indicate
+the completion of next level batch. The completion of very first level batch
+needs to be signaled by the command streamer. The user must provide the
+user/memory fence for this via the DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE
+extension of execbuf3 ioctl, so that KMD can setup the command streamer to
+signal it.
+
+User/Memory fence can also be supplied to the kernel driver to signal/wake up
+the user process after completion of an asynchronous operation.
+
+When VM_BIND ioctl was provided with a user/memory fence via the
+I915_VM_BIND_EXT_USER_FENCE extension, it will be signaled upon the completion
+of binding of that mapping. All async binds/unbinds are serialized, hence
+signaling of user/memory fence also indicate the completion of all previous
+binds/unbinds.
+
+This feature will be derived from the below original work:
+https://patchwork.freedesktop.org/patch/349417/
+
+Long running Compute contexts
+------------------------------
+Usage of dma-fence expects that they complete in reasonable amount of time.
+Compute on the other hand can be long running. Hence it is appropriate for
+compute to use user/memory fence and dma-fence usage will be limited to
+in-kernel consumption only. This requires an execbuf uapi extension to pass
+in user fence (See struct drm_i915_gem_execbuffer3_ext_user_fence). Compute
+must opt-in for this mechanism with I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
+flag during context creation. The dma-fence based user interfaces like gem_wait
+ioctl and execbuf out fence are not allowed on long running contexts. Implicit
+sync is not valid as well and is anyway not supported in VM_BIND mode.
+
+Where GPU page faults are not available, kernel driver upon buffer invalidation
+will initiate a suspend (preemption) of long running context with a dma-fence
+attached to it. And upon completion of that suspend fence, finish the
+invalidation, revalidate the BO and then resume the compute context. This is
+done by having a per-context preempt fence (also called suspend fence) proxying
+as i915_request fence. This suspend fence is enabled when someone tries to wait
+on it, which then triggers the context preemption.
+
+As this support for context suspension using a preempt fence and the resume work
+for the compute mode contexts can get tricky to get it right, it is better to
+add this support in drm scheduler so that multiple drivers can make use of it.
+That means, it will have a dependency on i915 drm scheduler conversion with GuC
+scheduler backend. This should be fine, as the plan is to support compute mode
+contexts only with GuC scheduler backend (at least initially). This is much
+easier to support with VM_BIND mode compared to the current heavier execbuf
+path resource attachment.
+
+Low Latency Submission
+-----------------------
+Allows compute UMD to directly submit GPU jobs instead of through execbuf
+ioctl. This is made possible by VM_BIND is not being synchronized against
+execbuf. VM_BIND allows bind/unbind of mappings required for the directly
+submitted jobs.
+
+Other VM_BIND use cases
+========================
+
+Debugger
+---------
+With debug event interface user space process (debugger) is able to keep track
+of and act upon resources created by another process (debugged) and attached
+to GPU via vm_bind interface.
+
+GPU page faults
+----------------
+GPU page faults when supported (in future), will only be supported in the
+VM_BIND mode. While both the older execbuf mode and the newer VM_BIND mode of
+binding will require using dma-fence to ensure residency, the GPU page faults
+mode when supported, will not use any dma-fence as residency is purely managed
+by installing and removing/invalidating page table entries.
+
+Page level hints settings
+--------------------------
+VM_BIND allows any hints setting per mapping instead of per BO.
+Possible hints include read-only mapping, placement and atomicity.
+Sub-BO level placement hint will be even more relevant with
+upcoming GPU on-demand page fault support.
+
+Page level Cache/CLOS settings
+-------------------------------
+VM_BIND allows cache/CLOS settings per mapping instead of per BO.
+
+Shared Virtual Memory (SVM) support
+------------------------------------
+VM_BIND interface can be used to map system memory directly (without gem BO
+abstraction) using the HMM interface. SVM is only supported with GPU page
+faults enabled.
+
+
+Broder i915 cleanups
+=====================
+Supporting this whole new vm_bind mode of binding which comes with its own
+use cases to support and the locking requirements requires proper integration
+with the existing i915 driver. This calls for some broader i915 driver
+cleanups/simplifications for maintainability of the driver going forward.
+Here are few things identified and are being looked into.
+
+- Remove vma lookup cache (eb->gem_context->handles_vma). VM_BIND feature
+  do not use it and complexity it brings in is probably more than the
+  performance advantage we get in legacy execbuf case.
+- Remove vma->open_count counting
+- Remove i915_vma active reference tracking. VM_BIND feature will not be using
+  it. Instead use underlying BO's dma-resv fence list to determine if a i915_vma
+  is active or not.
+
+
+VM_BIND UAPI
+=============
+
+.. kernel-doc:: Documentation/gpu/rfc/i915_vm_bind.h
diff --git a/Documentation/gpu/rfc/index.rst b/Documentation/gpu/rfc/index.rst
index 91e93a705230..7d10c36b268d 100644
--- a/Documentation/gpu/rfc/index.rst
+++ b/Documentation/gpu/rfc/index.rst
@@ -23,3 +23,7 @@ host such documentation:
 .. toctree::
 
     i915_scheduler.rst
+
+.. toctree::
+
+    i915_vm_bind.rst
-- 
2.21.0.rc0.32.g243a4c7e27

[Intel-gfx] [PATCH 2/3] drm/i915: Update i915 uapi documentation

[Niranjana Vishwanathapura]

Add some missing i915 upai documentation which the new
i915 VM_BIND feature documentation will be refer to.

Signed-off-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>
---
 include/uapi/drm/i915_drm.h | 203 ++++++++++++++++++++++++++++--------
 1 file changed, 158 insertions(+), 45 deletions(-)

diff --git a/include/uapi/drm/i915_drm.h b/include/uapi/drm/i915_drm.h
index de49b68b4fc8..9cf3bf47c7f2 100644
--- a/include/uapi/drm/i915_drm.h
+++ b/include/uapi/drm/i915_drm.h
@@ -751,14 +751,27 @@ typedef struct drm_i915_irq_wait {
 
 /* Must be kept compact -- no holes and well documented */
 
-typedef struct drm_i915_getparam {
+/**
+ * struct drm_i915_getparam - Driver parameter query structure.
+ */
+struct drm_i915_getparam {
+	/** @param: Driver parameter to query. */
 	__s32 param;
-	/*
+
+	/**
+	 * @value: Address of memory where queried value should be put.
+	 *
 	 * WARNING: Using pointers instead of fixed-size u64 means we need to write
 	 * compat32 code. Don't repeat this mistake.
 	 */
 	int __user *value;
-} drm_i915_getparam_t;
+};
+
+/**
+ * typedef drm_i915_getparam_t - Driver parameter query structure.
+ * See struct drm_i915_getparam.
+ */
+typedef struct drm_i915_getparam drm_i915_getparam_t;
 
 /* Ioctl to set kernel params:
  */
@@ -1239,76 +1252,119 @@ struct drm_i915_gem_exec_object2 {
 	__u64 rsvd2;
 };
 
+/**
+ * struct drm_i915_gem_exec_fence - An input or output fence for the execbuf
+ * ioctl.
+ *
+ * The request will wait for input fence to signal before submission.
+ *
+ * The returned output fence will be signaled after the completion of the
+ * request.
+ */
 struct drm_i915_gem_exec_fence {
-	/**
-	 * User's handle for a drm_syncobj to wait on or signal.
-	 */
+	/** @handle: User's handle for a drm_syncobj to wait on or signal. */
 	__u32 handle;
 
+	/**
+	 * @flags: Supported flags are:
+	 *
+	 * I915_EXEC_FENCE_WAIT:
+	 * Wait for the input fence before request submission.
+	 *
+	 * I915_EXEC_FENCE_SIGNAL:
+	 * Return request completion fence as output
+	 */
+	__u32 flags;
 #define I915_EXEC_FENCE_WAIT            (1<<0)
 #define I915_EXEC_FENCE_SIGNAL          (1<<1)
 #define __I915_EXEC_FENCE_UNKNOWN_FLAGS (-(I915_EXEC_FENCE_SIGNAL << 1))
-	__u32 flags;
 };
 
-/*
- * See drm_i915_gem_execbuffer_ext_timeline_fences.
- */
-#define DRM_I915_GEM_EXECBUFFER_EXT_TIMELINE_FENCES 0
-
-/*
+/**
+ * struct drm_i915_gem_execbuffer_ext_timeline_fences - Timeline fences
+ * for execbuf ioctl.
+ *
  * This structure describes an array of drm_syncobj and associated points for
  * timeline variants of drm_syncobj. It is invalid to append this structure to
  * the execbuf if I915_EXEC_FENCE_ARRAY is set.
  */
 struct drm_i915_gem_execbuffer_ext_timeline_fences {
+#define DRM_I915_GEM_EXECBUFFER_EXT_TIMELINE_FENCES 0
+	/** @base: Extension link. See struct i915_user_extension. */
 	struct i915_user_extension base;
 
 	/**
-	 * Number of element in the handles_ptr & value_ptr arrays.
+	 * @fence_count: Number of elements in the @handles_ptr & @value_ptr
+	 * arrays.
 	 */
 	__u64 fence_count;
 
 	/**
-	 * Pointer to an array of struct drm_i915_gem_exec_fence of length
-	 * fence_count.
+	 * @handles_ptr: Pointer to an array of struct drm_i915_gem_exec_fence
+	 * of length @fence_count.
 	 */
 	__u64 handles_ptr;
 
 	/**
-	 * Pointer to an array of u64 values of length fence_count. Values
-	 * must be 0 for a binary drm_syncobj. A Value of 0 for a timeline
-	 * drm_syncobj is invalid as it turns a drm_syncobj into a binary one.
+	 * @values_ptr: Pointer to an array of u64 values of length
+	 * @fence_count.
+	 * Values must be 0 for a binary drm_syncobj. A Value of 0 for a
+	 * timeline drm_syncobj is invalid as it turns a drm_syncobj into a
+	 * binary one.
 	 */
 	__u64 values_ptr;
 };
 
+/**
+ * struct drm_i915_gem_execbuffer2 - Structure for DRM_I915_GEM_EXECBUFFER2
+ * ioctl.
+ */
 struct drm_i915_gem_execbuffer2 {
-	/**
-	 * List of gem_exec_object2 structs
-	 */
+	/** @buffers_ptr: Pointer to a list of gem_exec_object2 structs */
 	__u64 buffers_ptr;
+
+	/** @buffer_count: Number of elements in @buffers_ptr array */
 	__u32 buffer_count;
 
-	/** Offset in the batchbuffer to start execution from. */
+	/**
+	 * @batch_start_offset: Offset in the batchbuffer to start execution
+	 * from.
+	 */
 	__u32 batch_start_offset;
-	/** Bytes used in batchbuffer from batch_start_offset */
+
+	/**
+	 * @batch_len: Length in bytes of the batch buffer, starting from the
+	 * @batch_start_offset. If 0, length is assumed to be the batch buffer
+	 * object size.
+	 */
 	__u32 batch_len;
+
+	/** @DR1: deprecated */
 	__u32 DR1;
+
+	/** @DR4: deprecated */
 	__u32 DR4;
+
+	/** @num_cliprects: See @cliprects_ptr */
 	__u32 num_cliprects;
+
 	/**
-	 * This is a struct drm_clip_rect *cliprects if I915_EXEC_FENCE_ARRAY
-	 * & I915_EXEC_USE_EXTENSIONS are not set.
+	 * @cliprects_ptr: Kernel clipping was a DRI1 misfeature.
+	 *
+	 * It is invalid to use this field if I915_EXEC_FENCE_ARRAY or
+	 * I915_EXEC_USE_EXTENSIONS flags are not set.
 	 *
 	 * If I915_EXEC_FENCE_ARRAY is set, then this is a pointer to an array
-	 * of struct drm_i915_gem_exec_fence and num_cliprects is the length
-	 * of the array.
+	 * of &drm_i915_gem_exec_fence and @num_cliprects is the length of the
+	 * array.
 	 *
 	 * If I915_EXEC_USE_EXTENSIONS is set, then this is a pointer to a
-	 * single struct i915_user_extension and num_cliprects is 0.
+	 * single &i915_user_extension and num_cliprects is 0.
 	 */
 	__u64 cliprects_ptr;
+
+	/** @flags: Execbuf flags */
+	__u64 flags;
 #define I915_EXEC_RING_MASK              (0x3f)
 #define I915_EXEC_DEFAULT                (0<<0)
 #define I915_EXEC_RENDER                 (1<<0)
@@ -1326,10 +1382,6 @@ struct drm_i915_gem_execbuffer2 {
 #define I915_EXEC_CONSTANTS_REL_GENERAL (0<<6) /* default */
 #define I915_EXEC_CONSTANTS_ABSOLUTE 	(1<<6)
 #define I915_EXEC_CONSTANTS_REL_SURFACE (2<<6) /* gen4/5 only */
-	__u64 flags;
-	__u64 rsvd1; /* now used for context info */
-	__u64 rsvd2;
-};
 
 /** Resets the SO write offset registers for transform feedback on gen7. */
 #define I915_EXEC_GEN7_SOL_RESET	(1<<8)
@@ -1432,9 +1484,23 @@ struct drm_i915_gem_execbuffer2 {
  * drm_i915_gem_execbuffer_ext enum.
  */
 #define I915_EXEC_USE_EXTENSIONS	(1 << 21)
-
 #define __I915_EXEC_UNKNOWN_FLAGS (-(I915_EXEC_USE_EXTENSIONS << 1))
 
+	/** @rsvd1: Context id */
+	__u64 rsvd1;
+
+	/**
+	 * @rsvd2: in and out sync_file file descriptors.
+	 *
+	 * When I915_EXEC_FENCE_IN or I915_EXEC_FENCE_SUBMIT flag is set, the
+	 * lower 32 bits of this field will have the in sync_file fd (input).
+	 *
+	 * When I915_EXEC_FENCE_OUT flag is set, the upper 32 bits of this
+	 * field will have the out sync_file fd (output).
+	 */
+	__u64 rsvd2;
+};
+
 #define I915_EXEC_CONTEXT_ID_MASK	(0xffffffff)
 #define i915_execbuffer2_set_context_id(eb2, context) \
 	(eb2).rsvd1 = context & I915_EXEC_CONTEXT_ID_MASK
@@ -1814,19 +1880,56 @@ struct drm_i915_gem_context_create {
 	__u32 pad;
 };
 
+/**
+ * struct drm_i915_gem_context_create_ext - Structure for creating contexts.
+ */
 struct drm_i915_gem_context_create_ext {
-	__u32 ctx_id; /* output: id of new context*/
+	/** @ctx_id: Id of the created context (output) */
+	__u32 ctx_id;
+
+	/**
+	 * @flags: Supported flags are:
+	 *
+	 * I915_CONTEXT_CREATE_FLAGS_USE_EXTENSIONS:
+	 * Extensions may be appended to this structure and driver must check
+	 * for those. See @extensions.
+	 *
+	 * I915_CONTEXT_CREATE_FLAGS_SINGLE_TIMELINE:
+	 * Created context will have single timeline.
+	 */
 	__u32 flags;
 #define I915_CONTEXT_CREATE_FLAGS_USE_EXTENSIONS	(1u << 0)
 #define I915_CONTEXT_CREATE_FLAGS_SINGLE_TIMELINE	(1u << 1)
 #define I915_CONTEXT_CREATE_FLAGS_UNKNOWN \
 	(-(I915_CONTEXT_CREATE_FLAGS_SINGLE_TIMELINE << 1))
+
+	/**
+	 * @extensions: Zero-terminated chain of extensions.
+	 * 
+	 * I915_CONTEXT_CREATE_EXT_SETPARAM:
+	 * Context parameter to set or query during context creation.
+	 * See struct drm_i915_gem_context_create_ext_setparam.
+	 * 
+	 * I915_CONTEXT_CREATE_EXT_CLONE:
+	 * This extension has been removed. On the off chance someone somewhere
+	 * has attempted to use it, never re-use this extension number.
+	 */
 	__u64 extensions;
+#define I915_CONTEXT_CREATE_EXT_SETPARAM 0
+#define I915_CONTEXT_CREATE_EXT_CLONE 1
 };
 
+/**
+ * struct drm_i915_gem_context_param - Context parameter to set or query.
+ */
 struct drm_i915_gem_context_param {
+	/** @ctx_id: Context id */
 	__u32 ctx_id;
+
+	/** @size: Size of the parameter @value
 	__u32 size;
+
+	/** @param: Parameter to set or query */
 	__u64 param;
 #define I915_CONTEXT_PARAM_BAN_PERIOD	0x1
 /* I915_CONTEXT_PARAM_NO_ZEROMAP has been removed.  On the off chance
@@ -1973,6 +2076,7 @@ struct drm_i915_gem_context_param {
 #define I915_CONTEXT_PARAM_PROTECTED_CONTENT    0xd
 /* Must be kept compact -- no holes and well documented */
 
+	/** @value: Context parameter value to be set or queried */
 	__u64 value;
 };
 
@@ -2371,23 +2475,29 @@ struct i915_context_param_engines {
 	struct i915_engine_class_instance engines[N__]; \
 } __attribute__((packed)) name__
 
+/**
+ * struct drm_i915_gem_context_create_ext_setparam - Context parameter
+ * to set or query during context creation.
+ */
 struct drm_i915_gem_context_create_ext_setparam {
-#define I915_CONTEXT_CREATE_EXT_SETPARAM 0
+	/** @base: Extension link. See struct i915_user_extension. */ 
 	struct i915_user_extension base;
+
+	/**
+	 * @param: Context parameter to set or query.
+	 * See struct drm_i915_gem_context_param.
+	 */
 	struct drm_i915_gem_context_param param;
 };
 
-/* This API has been removed.  On the off chance someone somewhere has
- * attempted to use it, never re-use this extension number.
- */
-#define I915_CONTEXT_CREATE_EXT_CLONE 1
-
 struct drm_i915_gem_context_destroy {
 	__u32 ctx_id;
 	__u32 pad;
 };
 
-/*
+/**
+ * struct drm_i915_gem_vm_control - Structure to create or destroy VM.
+ *
  * DRM_I915_GEM_VM_CREATE -
  *
  * Create a new virtual memory address space (ppGTT) for use within a context
@@ -2397,20 +2507,23 @@ struct drm_i915_gem_context_destroy {
  * The id of new VM (bound to the fd) for use with I915_CONTEXT_PARAM_VM is
  * returned in the outparam @id.
  *
- * No flags are defined, with all bits reserved and must be zero.
- *
  * An extension chain maybe provided, starting with @extensions, and terminated
  * by the @next_extension being 0. Currently, no extensions are defined.
  *
  * DRM_I915_GEM_VM_DESTROY -
  *
- * Destroys a previously created VM id, specified in @id.
+ * Destroys a previously created VM id, specified in @vm_id.
  *
  * No extensions or flags are allowed currently, and so must be zero.
  */
 struct drm_i915_gem_vm_control {
+	/** @extensions: Zero-terminated chain of extensions. */
 	__u64 extensions;
+
+	/** @flags: reserved for future usage, currently MBZ */
 	__u32 flags;
+
+	/** @vm_id: Id of the VM created or to be destroyed */
 	__u32 vm_id;
 };
 
-- 
2.21.0.rc0.32.g243a4c7e27

Re: [Intel-gfx] [PATCH 2/3] drm/i915: Update i915 uapi documentation

[Matthew Auld]

On 10/06/2022 08:07, Niranjana Vishwanathapura wrote:
> Add some missing i915 upai documentation which the new
> i915 VM_BIND feature documentation will be refer to.
> 
> Signed-off-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>

This one looks to be standalone. If no objections should we go ahead and 
merge this one?

> ---
>   include/uapi/drm/i915_drm.h | 203 ++++++++++++++++++++++++++++--------
>   1 file changed, 158 insertions(+), 45 deletions(-)
> 
> diff --git a/include/uapi/drm/i915_drm.h b/include/uapi/drm/i915_drm.h
> index de49b68b4fc8..9cf3bf47c7f2 100644
> --- a/include/uapi/drm/i915_drm.h
> +++ b/include/uapi/drm/i915_drm.h
> @@ -751,14 +751,27 @@ typedef struct drm_i915_irq_wait {
>   
>   /* Must be kept compact -- no holes and well documented */
>   
> -typedef struct drm_i915_getparam {
> +/**
> + * struct drm_i915_getparam - Driver parameter query structure.
> + */
> +struct drm_i915_getparam {
> +	/** @param: Driver parameter to query. */
>   	__s32 param;
> -	/*
> +
> +	/**
> +	 * @value: Address of memory where queried value should be put.
> +	 *
>   	 * WARNING: Using pointers instead of fixed-size u64 means we need to write
>   	 * compat32 code. Don't repeat this mistake.
>   	 */
>   	int __user *value;
> -} drm_i915_getparam_t;
> +};
> +
> +/**
> + * typedef drm_i915_getparam_t - Driver parameter query structure.
> + * See struct drm_i915_getparam.
> + */
> +typedef struct drm_i915_getparam drm_i915_getparam_t;
>   
>   /* Ioctl to set kernel params:
>    */
> @@ -1239,76 +1252,119 @@ struct drm_i915_gem_exec_object2 {
>   	__u64 rsvd2;
>   };
>   
> +/**
> + * struct drm_i915_gem_exec_fence - An input or output fence for the execbuf
> + * ioctl.
> + *
> + * The request will wait for input fence to signal before submission.
> + *
> + * The returned output fence will be signaled after the completion of the
> + * request.
> + */
>   struct drm_i915_gem_exec_fence {
> -	/**
> -	 * User's handle for a drm_syncobj to wait on or signal.
> -	 */
> +	/** @handle: User's handle for a drm_syncobj to wait on or signal. */
>   	__u32 handle;
>   
> +	/**
> +	 * @flags: Supported flags are:
> +	 *
> +	 * I915_EXEC_FENCE_WAIT:
> +	 * Wait for the input fence before request submission.
> +	 *
> +	 * I915_EXEC_FENCE_SIGNAL:
> +	 * Return request completion fence as output
> +	 */
> +	__u32 flags;
>   #define I915_EXEC_FENCE_WAIT            (1<<0)
>   #define I915_EXEC_FENCE_SIGNAL          (1<<1)
>   #define __I915_EXEC_FENCE_UNKNOWN_FLAGS (-(I915_EXEC_FENCE_SIGNAL << 1))
> -	__u32 flags;
>   };
>   
> -/*
> - * See drm_i915_gem_execbuffer_ext_timeline_fences.
> - */
> -#define DRM_I915_GEM_EXECBUFFER_EXT_TIMELINE_FENCES 0
> -
> -/*
> +/**
> + * struct drm_i915_gem_execbuffer_ext_timeline_fences - Timeline fences
> + * for execbuf ioctl.
> + *
>    * This structure describes an array of drm_syncobj and associated points for
>    * timeline variants of drm_syncobj. It is invalid to append this structure to
>    * the execbuf if I915_EXEC_FENCE_ARRAY is set.
>    */
>   struct drm_i915_gem_execbuffer_ext_timeline_fences {
> +#define DRM_I915_GEM_EXECBUFFER_EXT_TIMELINE_FENCES 0
> +	/** @base: Extension link. See struct i915_user_extension. */
>   	struct i915_user_extension base;
>   
>   	/**
> -	 * Number of element in the handles_ptr & value_ptr arrays.
> +	 * @fence_count: Number of elements in the @handles_ptr & @value_ptr
> +	 * arrays.
>   	 */
>   	__u64 fence_count;
>   
>   	/**
> -	 * Pointer to an array of struct drm_i915_gem_exec_fence of length
> -	 * fence_count.
> +	 * @handles_ptr: Pointer to an array of struct drm_i915_gem_exec_fence
> +	 * of length @fence_count.
>   	 */
>   	__u64 handles_ptr;
>   
>   	/**
> -	 * Pointer to an array of u64 values of length fence_count. Values
> -	 * must be 0 for a binary drm_syncobj. A Value of 0 for a timeline
> -	 * drm_syncobj is invalid as it turns a drm_syncobj into a binary one.
> +	 * @values_ptr: Pointer to an array of u64 values of length
> +	 * @fence_count.
> +	 * Values must be 0 for a binary drm_syncobj. A Value of 0 for a
> +	 * timeline drm_syncobj is invalid as it turns a drm_syncobj into a
> +	 * binary one.
>   	 */
>   	__u64 values_ptr;
>   };
>   
> +/**
> + * struct drm_i915_gem_execbuffer2 - Structure for DRM_I915_GEM_EXECBUFFER2
> + * ioctl.
> + */
>   struct drm_i915_gem_execbuffer2 {
> -	/**
> -	 * List of gem_exec_object2 structs
> -	 */
> +	/** @buffers_ptr: Pointer to a list of gem_exec_object2 structs */
>   	__u64 buffers_ptr;
> +
> +	/** @buffer_count: Number of elements in @buffers_ptr array */
>   	__u32 buffer_count;
>   
> -	/** Offset in the batchbuffer to start execution from. */
> +	/**
> +	 * @batch_start_offset: Offset in the batchbuffer to start execution
> +	 * from.
> +	 */
>   	__u32 batch_start_offset;
> -	/** Bytes used in batchbuffer from batch_start_offset */
> +
> +	/**
> +	 * @batch_len: Length in bytes of the batch buffer, starting from the
> +	 * @batch_start_offset. If 0, length is assumed to be the batch buffer
> +	 * object size.
> +	 */
>   	__u32 batch_len;
> +
> +	/** @DR1: deprecated */
>   	__u32 DR1;
> +
> +	/** @DR4: deprecated */
>   	__u32 DR4;
> +
> +	/** @num_cliprects: See @cliprects_ptr */
>   	__u32 num_cliprects;
> +
>   	/**
> -	 * This is a struct drm_clip_rect *cliprects if I915_EXEC_FENCE_ARRAY
> -	 * & I915_EXEC_USE_EXTENSIONS are not set.
> +	 * @cliprects_ptr: Kernel clipping was a DRI1 misfeature.
> +	 *
> +	 * It is invalid to use this field if I915_EXEC_FENCE_ARRAY or
> +	 * I915_EXEC_USE_EXTENSIONS flags are not set.
>   	 *
>   	 * If I915_EXEC_FENCE_ARRAY is set, then this is a pointer to an array
> -	 * of struct drm_i915_gem_exec_fence and num_cliprects is the length
> -	 * of the array.
> +	 * of &drm_i915_gem_exec_fence and @num_cliprects is the length of the
> +	 * array.
>   	 *
>   	 * If I915_EXEC_USE_EXTENSIONS is set, then this is a pointer to a
> -	 * single struct i915_user_extension and num_cliprects is 0.
> +	 * single &i915_user_extension and num_cliprects is 0.
>   	 */
>   	__u64 cliprects_ptr;
> +
> +	/** @flags: Execbuf flags */
> +	__u64 flags;
>   #define I915_EXEC_RING_MASK              (0x3f)
>   #define I915_EXEC_DEFAULT                (0<<0)
>   #define I915_EXEC_RENDER                 (1<<0)
> @@ -1326,10 +1382,6 @@ struct drm_i915_gem_execbuffer2 {
>   #define I915_EXEC_CONSTANTS_REL_GENERAL (0<<6) /* default */
>   #define I915_EXEC_CONSTANTS_ABSOLUTE 	(1<<6)
>   #define I915_EXEC_CONSTANTS_REL_SURFACE (2<<6) /* gen4/5 only */
> -	__u64 flags;
> -	__u64 rsvd1; /* now used for context info */
> -	__u64 rsvd2;
> -};
>   
>   /** Resets the SO write offset registers for transform feedback on gen7. */
>   #define I915_EXEC_GEN7_SOL_RESET	(1<<8)
> @@ -1432,9 +1484,23 @@ struct drm_i915_gem_execbuffer2 {
>    * drm_i915_gem_execbuffer_ext enum.
>    */
>   #define I915_EXEC_USE_EXTENSIONS	(1 << 21)
> -
>   #define __I915_EXEC_UNKNOWN_FLAGS (-(I915_EXEC_USE_EXTENSIONS << 1))
>   
> +	/** @rsvd1: Context id */
> +	__u64 rsvd1;
> +
> +	/**
> +	 * @rsvd2: in and out sync_file file descriptors.
> +	 *
> +	 * When I915_EXEC_FENCE_IN or I915_EXEC_FENCE_SUBMIT flag is set, the
> +	 * lower 32 bits of this field will have the in sync_file fd (input).
> +	 *
> +	 * When I915_EXEC_FENCE_OUT flag is set, the upper 32 bits of this
> +	 * field will have the out sync_file fd (output).
> +	 */
> +	__u64 rsvd2;
> +};
> +
>   #define I915_EXEC_CONTEXT_ID_MASK	(0xffffffff)
>   #define i915_execbuffer2_set_context_id(eb2, context) \
>   	(eb2).rsvd1 = context & I915_EXEC_CONTEXT_ID_MASK
> @@ -1814,19 +1880,56 @@ struct drm_i915_gem_context_create {
>   	__u32 pad;
>   };
>   
> +/**
> + * struct drm_i915_gem_context_create_ext - Structure for creating contexts.
> + */
>   struct drm_i915_gem_context_create_ext {
> -	__u32 ctx_id; /* output: id of new context*/
> +	/** @ctx_id: Id of the created context (output) */
> +	__u32 ctx_id;
> +
> +	/**
> +	 * @flags: Supported flags are:
> +	 *
> +	 * I915_CONTEXT_CREATE_FLAGS_USE_EXTENSIONS:
> +	 * Extensions may be appended to this structure and driver must check
> +	 * for those. See @extensions.
> +	 *
> +	 * I915_CONTEXT_CREATE_FLAGS_SINGLE_TIMELINE:
> +	 * Created context will have single timeline.
> +	 */
>   	__u32 flags;
>   #define I915_CONTEXT_CREATE_FLAGS_USE_EXTENSIONS	(1u << 0)
>   #define I915_CONTEXT_CREATE_FLAGS_SINGLE_TIMELINE	(1u << 1)
>   #define I915_CONTEXT_CREATE_FLAGS_UNKNOWN \
>   	(-(I915_CONTEXT_CREATE_FLAGS_SINGLE_TIMELINE << 1))
> +
> +	/**
> +	 * @extensions: Zero-terminated chain of extensions.
> +	 *
> +	 * I915_CONTEXT_CREATE_EXT_SETPARAM:
> +	 * Context parameter to set or query during context creation.
> +	 * See struct drm_i915_gem_context_create_ext_setparam.
> +	 *
> +	 * I915_CONTEXT_CREATE_EXT_CLONE:
> +	 * This extension has been removed. On the off chance someone somewhere
> +	 * has attempted to use it, never re-use this extension number.
> +	 */
>   	__u64 extensions;
> +#define I915_CONTEXT_CREATE_EXT_SETPARAM 0
> +#define I915_CONTEXT_CREATE_EXT_CLONE 1
>   };
>   
> +/**
> + * struct drm_i915_gem_context_param - Context parameter to set or query.
> + */
>   struct drm_i915_gem_context_param {
> +	/** @ctx_id: Context id */
>   	__u32 ctx_id;
> +
> +	/** @size: Size of the parameter @value
>   	__u32 size;
> +
> +	/** @param: Parameter to set or query */
>   	__u64 param;
>   #define I915_CONTEXT_PARAM_BAN_PERIOD	0x1
>   /* I915_CONTEXT_PARAM_NO_ZEROMAP has been removed.  On the off chance
> @@ -1973,6 +2076,7 @@ struct drm_i915_gem_context_param {
>   #define I915_CONTEXT_PARAM_PROTECTED_CONTENT    0xd
>   /* Must be kept compact -- no holes and well documented */
>   
> +	/** @value: Context parameter value to be set or queried */
>   	__u64 value;
>   };
>   
> @@ -2371,23 +2475,29 @@ struct i915_context_param_engines {
>   	struct i915_engine_class_instance engines[N__]; \
>   } __attribute__((packed)) name__
>   
> +/**
> + * struct drm_i915_gem_context_create_ext_setparam - Context parameter
> + * to set or query during context creation.
> + */
>   struct drm_i915_gem_context_create_ext_setparam {
> -#define I915_CONTEXT_CREATE_EXT_SETPARAM 0
> +	/** @base: Extension link. See struct i915_user_extension. */
>   	struct i915_user_extension base;
> +
> +	/**
> +	 * @param: Context parameter to set or query.
> +	 * See struct drm_i915_gem_context_param.
> +	 */
>   	struct drm_i915_gem_context_param param;
>   };
>   
> -/* This API has been removed.  On the off chance someone somewhere has
> - * attempted to use it, never re-use this extension number.
> - */
> -#define I915_CONTEXT_CREATE_EXT_CLONE 1
> -
>   struct drm_i915_gem_context_destroy {
>   	__u32 ctx_id;
>   	__u32 pad;
>   };
>   
> -/*
> +/**
> + * struct drm_i915_gem_vm_control - Structure to create or destroy VM.
> + *
>    * DRM_I915_GEM_VM_CREATE -
>    *
>    * Create a new virtual memory address space (ppGTT) for use within a context
> @@ -2397,20 +2507,23 @@ struct drm_i915_gem_context_destroy {
>    * The id of new VM (bound to the fd) for use with I915_CONTEXT_PARAM_VM is
>    * returned in the outparam @id.
>    *
> - * No flags are defined, with all bits reserved and must be zero.
> - *
>    * An extension chain maybe provided, starting with @extensions, and terminated
>    * by the @next_extension being 0. Currently, no extensions are defined.
>    *
>    * DRM_I915_GEM_VM_DESTROY -
>    *
> - * Destroys a previously created VM id, specified in @id.
> + * Destroys a previously created VM id, specified in @vm_id.
>    *
>    * No extensions or flags are allowed currently, and so must be zero.
>    */
>   struct drm_i915_gem_vm_control {
> +	/** @extensions: Zero-terminated chain of extensions. */
>   	__u64 extensions;
> +
> +	/** @flags: reserved for future usage, currently MBZ */
>   	__u32 flags;
> +
> +	/** @vm_id: Id of the VM created or to be destroyed */
>   	__u32 vm_id;
>   };
>   

Re: [Intel-gfx] [PATCH 2/3] drm/i915: Update i915 uapi documentation

[Niranjana Vishwanathapura]

On Fri, Jun 10, 2022 at 12:01:24PM +0100, Matthew Auld wrote:
>On 10/06/2022 08:07, Niranjana Vishwanathapura wrote:
>>Add some missing i915 upai documentation which the new
>>i915 VM_BIND feature documentation will be refer to.
>>
>>Signed-off-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>
>Reviewed-by: Matthew Auld <matthew.auld@intel.com>
>
>This one looks to be standalone. If no objections should we go ahead 
>and merge this one?

Yah, I agree.

Niranjana

>
>>---
>>  include/uapi/drm/i915_drm.h | 203 ++++++++++++++++++++++++++++--------
>>  1 file changed, 158 insertions(+), 45 deletions(-)
>>
>>diff --git a/include/uapi/drm/i915_drm.h b/include/uapi/drm/i915_drm.h
>>index de49b68b4fc8..9cf3bf47c7f2 100644
>>--- a/include/uapi/drm/i915_drm.h
>>+++ b/include/uapi/drm/i915_drm.h
>>@@ -751,14 +751,27 @@ typedef struct drm_i915_irq_wait {
>>  /* Must be kept compact -- no holes and well documented */
>>-typedef struct drm_i915_getparam {
>>+/**
>>+ * struct drm_i915_getparam - Driver parameter query structure.
>>+ */
>>+struct drm_i915_getparam {
>>+	/** @param: Driver parameter to query. */
>>  	__s32 param;
>>-	/*
>>+
>>+	/**
>>+	 * @value: Address of memory where queried value should be put.
>>+	 *
>>  	 * WARNING: Using pointers instead of fixed-size u64 means we need to write
>>  	 * compat32 code. Don't repeat this mistake.
>>  	 */
>>  	int __user *value;
>>-} drm_i915_getparam_t;
>>+};
>>+
>>+/**
>>+ * typedef drm_i915_getparam_t - Driver parameter query structure.
>>+ * See struct drm_i915_getparam.
>>+ */
>>+typedef struct drm_i915_getparam drm_i915_getparam_t;
>>  /* Ioctl to set kernel params:
>>   */
>>@@ -1239,76 +1252,119 @@ struct drm_i915_gem_exec_object2 {
>>  	__u64 rsvd2;
>>  };
>>+/**
>>+ * struct drm_i915_gem_exec_fence - An input or output fence for the execbuf
>>+ * ioctl.
>>+ *
>>+ * The request will wait for input fence to signal before submission.
>>+ *
>>+ * The returned output fence will be signaled after the completion of the
>>+ * request.
>>+ */
>>  struct drm_i915_gem_exec_fence {
>>-	/**
>>-	 * User's handle for a drm_syncobj to wait on or signal.
>>-	 */
>>+	/** @handle: User's handle for a drm_syncobj to wait on or signal. */
>>  	__u32 handle;
>>+	/**
>>+	 * @flags: Supported flags are:
>>+	 *
>>+	 * I915_EXEC_FENCE_WAIT:
>>+	 * Wait for the input fence before request submission.
>>+	 *
>>+	 * I915_EXEC_FENCE_SIGNAL:
>>+	 * Return request completion fence as output
>>+	 */
>>+	__u32 flags;
>>  #define I915_EXEC_FENCE_WAIT            (1<<0)
>>  #define I915_EXEC_FENCE_SIGNAL          (1<<1)
>>  #define __I915_EXEC_FENCE_UNKNOWN_FLAGS (-(I915_EXEC_FENCE_SIGNAL << 1))
>>-	__u32 flags;
>>  };
>>-/*
>>- * See drm_i915_gem_execbuffer_ext_timeline_fences.
>>- */
>>-#define DRM_I915_GEM_EXECBUFFER_EXT_TIMELINE_FENCES 0
>>-
>>-/*
>>+/**
>>+ * struct drm_i915_gem_execbuffer_ext_timeline_fences - Timeline fences
>>+ * for execbuf ioctl.
>>+ *
>>   * This structure describes an array of drm_syncobj and associated points for
>>   * timeline variants of drm_syncobj. It is invalid to append this structure to
>>   * the execbuf if I915_EXEC_FENCE_ARRAY is set.
>>   */
>>  struct drm_i915_gem_execbuffer_ext_timeline_fences {
>>+#define DRM_I915_GEM_EXECBUFFER_EXT_TIMELINE_FENCES 0
>>+	/** @base: Extension link. See struct i915_user_extension. */
>>  	struct i915_user_extension base;
>>  	/**
>>-	 * Number of element in the handles_ptr & value_ptr arrays.
>>+	 * @fence_count: Number of elements in the @handles_ptr & @value_ptr
>>+	 * arrays.
>>  	 */
>>  	__u64 fence_count;
>>  	/**
>>-	 * Pointer to an array of struct drm_i915_gem_exec_fence of length
>>-	 * fence_count.
>>+	 * @handles_ptr: Pointer to an array of struct drm_i915_gem_exec_fence
>>+	 * of length @fence_count.
>>  	 */
>>  	__u64 handles_ptr;
>>  	/**
>>-	 * Pointer to an array of u64 values of length fence_count. Values
>>-	 * must be 0 for a binary drm_syncobj. A Value of 0 for a timeline
>>-	 * drm_syncobj is invalid as it turns a drm_syncobj into a binary one.
>>+	 * @values_ptr: Pointer to an array of u64 values of length
>>+	 * @fence_count.
>>+	 * Values must be 0 for a binary drm_syncobj. A Value of 0 for a
>>+	 * timeline drm_syncobj is invalid as it turns a drm_syncobj into a
>>+	 * binary one.
>>  	 */
>>  	__u64 values_ptr;
>>  };
>>+/**
>>+ * struct drm_i915_gem_execbuffer2 - Structure for DRM_I915_GEM_EXECBUFFER2
>>+ * ioctl.
>>+ */
>>  struct drm_i915_gem_execbuffer2 {
>>-	/**
>>-	 * List of gem_exec_object2 structs
>>-	 */
>>+	/** @buffers_ptr: Pointer to a list of gem_exec_object2 structs */
>>  	__u64 buffers_ptr;
>>+
>>+	/** @buffer_count: Number of elements in @buffers_ptr array */
>>  	__u32 buffer_count;
>>-	/** Offset in the batchbuffer to start execution from. */
>>+	/**
>>+	 * @batch_start_offset: Offset in the batchbuffer to start execution
>>+	 * from.
>>+	 */
>>  	__u32 batch_start_offset;
>>-	/** Bytes used in batchbuffer from batch_start_offset */
>>+
>>+	/**
>>+	 * @batch_len: Length in bytes of the batch buffer, starting from the
>>+	 * @batch_start_offset. If 0, length is assumed to be the batch buffer
>>+	 * object size.
>>+	 */
>>  	__u32 batch_len;
>>+
>>+	/** @DR1: deprecated */
>>  	__u32 DR1;
>>+
>>+	/** @DR4: deprecated */
>>  	__u32 DR4;
>>+
>>+	/** @num_cliprects: See @cliprects_ptr */
>>  	__u32 num_cliprects;
>>+
>>  	/**
>>-	 * This is a struct drm_clip_rect *cliprects if I915_EXEC_FENCE_ARRAY
>>-	 * & I915_EXEC_USE_EXTENSIONS are not set.
>>+	 * @cliprects_ptr: Kernel clipping was a DRI1 misfeature.
>>+	 *
>>+	 * It is invalid to use this field if I915_EXEC_FENCE_ARRAY or
>>+	 * I915_EXEC_USE_EXTENSIONS flags are not set.
>>  	 *
>>  	 * If I915_EXEC_FENCE_ARRAY is set, then this is a pointer to an array
>>-	 * of struct drm_i915_gem_exec_fence and num_cliprects is the length
>>-	 * of the array.
>>+	 * of &drm_i915_gem_exec_fence and @num_cliprects is the length of the
>>+	 * array.
>>  	 *
>>  	 * If I915_EXEC_USE_EXTENSIONS is set, then this is a pointer to a
>>-	 * single struct i915_user_extension and num_cliprects is 0.
>>+	 * single &i915_user_extension and num_cliprects is 0.
>>  	 */
>>  	__u64 cliprects_ptr;
>>+
>>+	/** @flags: Execbuf flags */
>>+	__u64 flags;
>>  #define I915_EXEC_RING_MASK              (0x3f)
>>  #define I915_EXEC_DEFAULT                (0<<0)
>>  #define I915_EXEC_RENDER                 (1<<0)
>>@@ -1326,10 +1382,6 @@ struct drm_i915_gem_execbuffer2 {
>>  #define I915_EXEC_CONSTANTS_REL_GENERAL (0<<6) /* default */
>>  #define I915_EXEC_CONSTANTS_ABSOLUTE 	(1<<6)
>>  #define I915_EXEC_CONSTANTS_REL_SURFACE (2<<6) /* gen4/5 only */
>>-	__u64 flags;
>>-	__u64 rsvd1; /* now used for context info */
>>-	__u64 rsvd2;
>>-};
>>  /** Resets the SO write offset registers for transform feedback on gen7. */
>>  #define I915_EXEC_GEN7_SOL_RESET	(1<<8)
>>@@ -1432,9 +1484,23 @@ struct drm_i915_gem_execbuffer2 {
>>   * drm_i915_gem_execbuffer_ext enum.
>>   */
>>  #define I915_EXEC_USE_EXTENSIONS	(1 << 21)
>>-
>>  #define __I915_EXEC_UNKNOWN_FLAGS (-(I915_EXEC_USE_EXTENSIONS << 1))
>>+	/** @rsvd1: Context id */
>>+	__u64 rsvd1;
>>+
>>+	/**
>>+	 * @rsvd2: in and out sync_file file descriptors.
>>+	 *
>>+	 * When I915_EXEC_FENCE_IN or I915_EXEC_FENCE_SUBMIT flag is set, the
>>+	 * lower 32 bits of this field will have the in sync_file fd (input).
>>+	 *
>>+	 * When I915_EXEC_FENCE_OUT flag is set, the upper 32 bits of this
>>+	 * field will have the out sync_file fd (output).
>>+	 */
>>+	__u64 rsvd2;
>>+};
>>+
>>  #define I915_EXEC_CONTEXT_ID_MASK	(0xffffffff)
>>  #define i915_execbuffer2_set_context_id(eb2, context) \
>>  	(eb2).rsvd1 = context & I915_EXEC_CONTEXT_ID_MASK
>>@@ -1814,19 +1880,56 @@ struct drm_i915_gem_context_create {
>>  	__u32 pad;
>>  };
>>+/**
>>+ * struct drm_i915_gem_context_create_ext - Structure for creating contexts.
>>+ */
>>  struct drm_i915_gem_context_create_ext {
>>-	__u32 ctx_id; /* output: id of new context*/
>>+	/** @ctx_id: Id of the created context (output) */
>>+	__u32 ctx_id;
>>+
>>+	/**
>>+	 * @flags: Supported flags are:
>>+	 *
>>+	 * I915_CONTEXT_CREATE_FLAGS_USE_EXTENSIONS:
>>+	 * Extensions may be appended to this structure and driver must check
>>+	 * for those. See @extensions.
>>+	 *
>>+	 * I915_CONTEXT_CREATE_FLAGS_SINGLE_TIMELINE:
>>+	 * Created context will have single timeline.
>>+	 */
>>  	__u32 flags;
>>  #define I915_CONTEXT_CREATE_FLAGS_USE_EXTENSIONS	(1u << 0)
>>  #define I915_CONTEXT_CREATE_FLAGS_SINGLE_TIMELINE	(1u << 1)
>>  #define I915_CONTEXT_CREATE_FLAGS_UNKNOWN \
>>  	(-(I915_CONTEXT_CREATE_FLAGS_SINGLE_TIMELINE << 1))
>>+
>>+	/**
>>+	 * @extensions: Zero-terminated chain of extensions.
>>+	 *
>>+	 * I915_CONTEXT_CREATE_EXT_SETPARAM:
>>+	 * Context parameter to set or query during context creation.
>>+	 * See struct drm_i915_gem_context_create_ext_setparam.
>>+	 *
>>+	 * I915_CONTEXT_CREATE_EXT_CLONE:
>>+	 * This extension has been removed. On the off chance someone somewhere
>>+	 * has attempted to use it, never re-use this extension number.
>>+	 */
>>  	__u64 extensions;
>>+#define I915_CONTEXT_CREATE_EXT_SETPARAM 0
>>+#define I915_CONTEXT_CREATE_EXT_CLONE 1
>>  };
>>+/**
>>+ * struct drm_i915_gem_context_param - Context parameter to set or query.
>>+ */
>>  struct drm_i915_gem_context_param {
>>+	/** @ctx_id: Context id */
>>  	__u32 ctx_id;
>>+
>>+	/** @size: Size of the parameter @value
>>  	__u32 size;
>>+
>>+	/** @param: Parameter to set or query */
>>  	__u64 param;
>>  #define I915_CONTEXT_PARAM_BAN_PERIOD	0x1
>>  /* I915_CONTEXT_PARAM_NO_ZEROMAP has been removed.  On the off chance
>>@@ -1973,6 +2076,7 @@ struct drm_i915_gem_context_param {
>>  #define I915_CONTEXT_PARAM_PROTECTED_CONTENT    0xd
>>  /* Must be kept compact -- no holes and well documented */
>>+	/** @value: Context parameter value to be set or queried */
>>  	__u64 value;
>>  };
>>@@ -2371,23 +2475,29 @@ struct i915_context_param_engines {
>>  	struct i915_engine_class_instance engines[N__]; \
>>  } __attribute__((packed)) name__
>>+/**
>>+ * struct drm_i915_gem_context_create_ext_setparam - Context parameter
>>+ * to set or query during context creation.
>>+ */
>>  struct drm_i915_gem_context_create_ext_setparam {
>>-#define I915_CONTEXT_CREATE_EXT_SETPARAM 0
>>+	/** @base: Extension link. See struct i915_user_extension. */
>>  	struct i915_user_extension base;
>>+
>>+	/**
>>+	 * @param: Context parameter to set or query.
>>+	 * See struct drm_i915_gem_context_param.
>>+	 */
>>  	struct drm_i915_gem_context_param param;
>>  };
>>-/* This API has been removed.  On the off chance someone somewhere has
>>- * attempted to use it, never re-use this extension number.
>>- */
>>-#define I915_CONTEXT_CREATE_EXT_CLONE 1
>>-
>>  struct drm_i915_gem_context_destroy {
>>  	__u32 ctx_id;
>>  	__u32 pad;
>>  };
>>-/*
>>+/**
>>+ * struct drm_i915_gem_vm_control - Structure to create or destroy VM.
>>+ *
>>   * DRM_I915_GEM_VM_CREATE -
>>   *
>>   * Create a new virtual memory address space (ppGTT) for use within a context
>>@@ -2397,20 +2507,23 @@ struct drm_i915_gem_context_destroy {
>>   * The id of new VM (bound to the fd) for use with I915_CONTEXT_PARAM_VM is
>>   * returned in the outparam @id.
>>   *
>>- * No flags are defined, with all bits reserved and must be zero.
>>- *
>>   * An extension chain maybe provided, starting with @extensions, and terminated
>>   * by the @next_extension being 0. Currently, no extensions are defined.
>>   *
>>   * DRM_I915_GEM_VM_DESTROY -
>>   *
>>- * Destroys a previously created VM id, specified in @id.
>>+ * Destroys a previously created VM id, specified in @vm_id.
>>   *
>>   * No extensions or flags are allowed currently, and so must be zero.
>>   */
>>  struct drm_i915_gem_vm_control {
>>+	/** @extensions: Zero-terminated chain of extensions. */
>>  	__u64 extensions;
>>+
>>+	/** @flags: reserved for future usage, currently MBZ */
>>  	__u32 flags;
>>+
>>+	/** @vm_id: Id of the VM created or to be destroyed */
>>  	__u32 vm_id;
>>  };

[Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Niranjana Vishwanathapura]

VM_BIND and related uapi definitions

Signed-off-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>
---
 Documentation/gpu/rfc/i915_vm_bind.h | 490 +++++++++++++++++++++++++++
 1 file changed, 490 insertions(+)
 create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h

diff --git a/Documentation/gpu/rfc/i915_vm_bind.h b/Documentation/gpu/rfc/i915_vm_bind.h
new file mode 100644
index 000000000000..9fc854969cfb
--- /dev/null
+++ b/Documentation/gpu/rfc/i915_vm_bind.h
@@ -0,0 +1,490 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2022 Intel Corporation
+ */
+
+/**
+ * DOC: I915_PARAM_HAS_VM_BIND
+ *
+ * VM_BIND feature availability.
+ * See typedef drm_i915_getparam_t param.
+ * bit[0]: If set, VM_BIND is supported, otherwise not.
+ * bits[8-15]: VM_BIND implementation version.
+ * version 0 will not have VM_BIND/UNBIND timeline fence array support.
+ */
+#define I915_PARAM_HAS_VM_BIND		57
+
+/**
+ * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
+ *
+ * Flag to opt-in for VM_BIND mode of binding during VM creation.
+ * See struct drm_i915_gem_vm_control flags.
+ *
+ * The older execbuf2 ioctl will not support VM_BIND mode of operation.
+ * For VM_BIND mode, we have new execbuf3 ioctl which will not accept any
+ * execlist (See struct drm_i915_gem_execbuffer3 for more details).
+ *
+ */
+#define I915_VM_CREATE_FLAGS_USE_VM_BIND	(1 << 0)
+
+/**
+ * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
+ *
+ * Flag to declare context as long running.
+ * See struct drm_i915_gem_context_create_ext flags.
+ *
+ * Usage of dma-fence expects that they complete in reasonable amount of time.
+ * Compute on the other hand can be long running. Hence it is not appropriate
+ * for compute contexts to export request completion dma-fence to user.
+ * The dma-fence usage will be limited to in-kernel consumption only.
+ * Compute contexts need to use user/memory fence.
+ *
+ * So, long running contexts do not support output fences. Hence,
+ * I915_EXEC_FENCE_SIGNAL (See &drm_i915_gem_exec_fence.flags) is expected
+ * to be not used. DRM_I915_GEM_WAIT ioctl call is also not supported for
+ * objects mapped to long running contexts.
+ */
+#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u << 2)
+
+/* VM_BIND related ioctls */
+#define DRM_I915_GEM_VM_BIND		0x3d
+#define DRM_I915_GEM_VM_UNBIND		0x3e
+#define DRM_I915_GEM_EXECBUFFER3	0x3f
+#define DRM_I915_GEM_WAIT_USER_FENCE	0x40
+
+#define DRM_IOCTL_I915_GEM_VM_BIND		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_VM_BIND, struct drm_i915_gem_vm_bind)
+#define DRM_IOCTL_I915_GEM_VM_UNBIND		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_VM_UNBIND, struct drm_i915_gem_vm_bind)
+#define DRM_IOCTL_I915_GEM_EXECBUFFER3		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_EXECBUFFER3, struct drm_i915_gem_execbuffer3)
+#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE	DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_WAIT_USER_FENCE, struct drm_i915_gem_wait_user_fence)
+
+/**
+ * struct drm_i915_gem_vm_bind - VA to object mapping to bind.
+ *
+ * This structure is passed to VM_BIND ioctl and specifies the mapping of GPU
+ * virtual address (VA) range to the section of an object that should be bound
+ * in the device page table of the specified address space (VM).
+ * The VA range specified must be unique (ie., not currently bound) and can
+ * be mapped to whole object or a section of the object (partial binding).
+ * Multiple VA mappings can be created to the same section of the object
+ * (aliasing).
+ *
+ * The @queue_idx specifies the queue to use for binding. Same queue can be
+ * used for both VM_BIND and VM_UNBIND calls. All submitted bind and unbind
+ * operations in a queue are performed in the order of submission.
+ *
+ * The @start, @offset and @length should be 4K page aligned. However the DG2
+ * and XEHPSDV has 64K page size for device local-memory and has compact page
+ * table. On those platforms, for binding device local-memory objects, the
+ * @start should be 2M aligned, @offset and @length should be 64K aligned.
+ * Also, on those platforms, it is not allowed to bind an device local-memory
+ * object and a system memory object in a single 2M section of VA range.
+ */
+struct drm_i915_gem_vm_bind {
+	/** @vm_id: VM (address space) id to bind */
+	__u32 vm_id;
+
+	/** @queue_idx: Index of queue for binding */
+	__u32 queue_idx;
+
+	/** @rsvd: Reserved, MBZ */
+	__u32 rsvd;
+
+	/** @handle: Object handle */
+	__u32 handle;
+
+	/** @start: Virtual Address start to bind */
+	__u64 start;
+
+	/** @offset: Offset in object to bind */
+	__u64 offset;
+
+	/** @length: Length of mapping to bind */
+	__u64 length;
+
+	/**
+	 * @flags: Supported flags are:
+	 *
+	 * I915_GEM_VM_BIND_READONLY:
+	 * Mapping is read-only.
+	 *
+	 * I915_GEM_VM_BIND_CAPTURE:
+	 * Capture this mapping in the dump upon GPU error.
+	 */
+	__u64 flags;
+#define I915_GEM_VM_BIND_READONLY    (1 << 0)
+#define I915_GEM_VM_BIND_CAPTURE     (1 << 1)
+
+	/**
+	 * @extensions: 0-terminated chain of extensions for this operation.
+	 *
+	 * I915_VM_BIND_EXT_TIMELINE_FENCES:
+	 * Specifies an array of input or output timeline fences for this
+	 * binding operation. See struct drm_i915_vm_bind_ext_timeline_fences.
+	 *
+	 * I915_VM_BIND_EXT_USER_FENCES:
+	 * Specifies an array of input or output user fences for this
+	 * binding operation. See struct drm_i915_vm_bind_ext_user_fence.
+	 * This is required for compute contexts.
+	 */
+	__u64 extensions;
+#define I915_VM_BIND_EXT_TIMELINE_FENCES	0
+#define I915_VM_BIND_EXT_USER_FENCES		1
+};
+
+/**
+ * struct drm_i915_gem_vm_unbind - VA to object mapping to unbind.
+ *
+ * This structure is passed to VM_UNBIND ioctl and specifies the GPU virtual
+ * address (VA) range that should be unbound from the device page table of the
+ * specified address space (VM). The specified VA range must match one of the
+ * mappings created with the VM_BIND ioctl. TLB is flushed upon unbind
+ * completion.
+ *
+ * The @queue_idx specifies the queue to use for unbinding.
+ * See struct drm_i915_gem_vm_unbind for more information on @queue_idx.
+ *
+ * The @start and @length musy specify a unique mapping bound with VM_BIND
+ * ioctl.
+ */
+struct drm_i915_gem_vm_unbind {
+	/** @vm_id: VM (address space) id to bind */
+	__u32 vm_id;
+
+	/** @queue_idx: Index of queue for unbinding */
+	__u32 queue_idx;
+
+	/** @start: Virtual Address start to unbind */
+	__u64 start;
+
+	/** @length: Length of mapping to unbind */
+	__u64 length;
+
+	/** @flags: Reserved for future usage, currently MBZ */
+	__u64 flags;
+
+	/**
+	 * @extensions: 0-terminated chain of extensions for this operation.
+	 *
+	 * I915_VM_UNBIND_EXT_TIMELINE_FENCES:
+	 * Specifies an array of input or output timeline fences for this
+	 * unbind operation.
+	 * It has same format as struct drm_i915_vm_bind_ext_timeline_fences.
+	 *
+	 * I915_VM_UNBIND_EXT_USER_FENCES:
+	 * Specifies an array of input or output user fences for this
+	 * unbind operation. This is required for compute contexts.
+	 * It has same format as struct drm_i915_vm_bind_ext_user_fence.
+	 */
+	__u64 extensions;
+#define I915_VM_UNBIND_EXT_TIMELINE_FENCES	0
+#define I915_VM_UNBIND_EXT_USER_FENCES		1
+};
+
+/**
+ * struct drm_i915_vm_bind_fence - An input or output fence for the vm_bind
+ * or the vm_unbind work.
+ *
+ * The vm_bind or vm_unbind aync worker will wait for input fence to signal
+ * before starting the binding or unbinding.
+ *
+ * The vm_bind or vm_unbind async worker will signal the returned output fence
+ * after the completion of binding or unbinding.
+ */
+struct drm_i915_vm_bind_fence {
+	/** @handle: User's handle for a drm_syncobj to wait on or signal. */
+	__u32 handle;
+
+	/**
+	 * @flags: Supported flags are:
+	 *
+	 * I915_VM_BIND_FENCE_WAIT:
+	 * Wait for the input fence before binding/unbinding
+	 *
+	 * I915_VM_BIND_FENCE_SIGNAL:
+	 * Return bind/unbind completion fence as output
+	 */
+	__u32 flags;
+#define I915_VM_BIND_FENCE_WAIT            (1<<0)
+#define I915_VM_BIND_FENCE_SIGNAL          (1<<1)
+#define __I915_VM_BIND_FENCE_UNKNOWN_FLAGS (-(I915_VM_BIND_FENCE_SIGNAL << 1))
+};
+
+/**
+ * struct drm_i915_vm_bind_ext_timeline_fences - Timeline fences for vm_bind
+ * and vm_unbind.
+ *
+ * This structure describes an array of timeline drm_syncobj and associated
+ * points for timeline variants of drm_syncobj. These timeline 'drm_syncobj's
+ * can be input or output fences (See struct drm_i915_vm_bind_fence).
+ */
+struct drm_i915_vm_bind_ext_timeline_fences {
+	/** @base: Extension link. See struct i915_user_extension. */
+	struct i915_user_extension base;
+
+	/**
+	 * @fence_count: Number of elements in the @handles_ptr & @value_ptr
+	 * arrays.
+	 */
+	__u64 fence_count;
+
+	/**
+	 * @handles_ptr: Pointer to an array of struct drm_i915_vm_bind_fence
+	 * of length @fence_count.
+	 */
+	__u64 handles_ptr;
+
+	/**
+	 * @values_ptr: Pointer to an array of u64 values of length
+	 * @fence_count.
+	 * Values must be 0 for a binary drm_syncobj. A Value of 0 for a
+	 * timeline drm_syncobj is invalid as it turns a drm_syncobj into a
+	 * binary one.
+	 */
+	__u64 values_ptr;
+};
+
+/**
+ * struct drm_i915_vm_bind_user_fence - An input or output user fence for the
+ * vm_bind or the vm_unbind work.
+ *
+ * The vm_bind or vm_unbind aync worker will wait for the input fence (value at
+ * @addr to become equal to @val) before starting the binding or unbinding.
+ *
+ * The vm_bind or vm_unbind async worker will signal the output fence after
+ * the completion of binding or unbinding by writing @val to memory location at
+ * @addr
+ */
+struct drm_i915_vm_bind_user_fence {
+	/** @addr: User/Memory fence qword aligned process virtual address */
+	__u64 addr;
+
+	/** @val: User/Memory fence value to be written after bind completion */
+	__u64 val;
+
+	/**
+	 * @flags: Supported flags are:
+	 *
+	 * I915_VM_BIND_USER_FENCE_WAIT:
+	 * Wait for the input fence before binding/unbinding
+	 *
+	 * I915_VM_BIND_USER_FENCE_SIGNAL:
+	 * Return bind/unbind completion fence as output
+	 */
+	__u32 flags;
+#define I915_VM_BIND_USER_FENCE_WAIT            (1<<0)
+#define I915_VM_BIND_USER_FENCE_SIGNAL          (1<<1)
+#define __I915_VM_BIND_USER_FENCE_UNKNOWN_FLAGS \
+	(-(I915_VM_BIND_USER_FENCE_SIGNAL << 1))
+};
+
+/**
+ * struct drm_i915_vm_bind_ext_user_fence - User/memory fences for vm_bind
+ * and vm_unbind.
+ *
+ * These user fences can be input or output fences
+ * (See struct drm_i915_vm_bind_user_fence).
+ */
+struct drm_i915_vm_bind_ext_user_fence {
+	/** @base: Extension link. See struct i915_user_extension. */
+	struct i915_user_extension base;
+
+	/** @fence_count: Number of elements in the @user_fence_ptr array. */
+	__u64 fence_count;
+
+	/**
+	 * @user_fence_ptr: Pointer to an array of
+	 * struct drm_i915_vm_bind_user_fence of length @fence_count.
+	 */
+	__u64 user_fence_ptr;
+};
+
+/**
+ * struct drm_i915_gem_execbuffer3 - Structure for DRM_I915_GEM_EXECBUFFER3
+ * ioctl.
+ *
+ * DRM_I915_GEM_EXECBUFFER3 ioctl only works in VM_BIND mode and VM_BIND mode
+ * only works with this ioctl for submission.
+ * See I915_VM_CREATE_FLAGS_USE_VM_BIND.
+ */
+struct drm_i915_gem_execbuffer3 {
+	/**
+	 * @ctx_id: Context id
+	 *
+	 * Only contexts with user engine map are allowed.
+	 */
+	__u32 ctx_id;
+
+	/**
+	 * @engine_idx: Engine index
+	 *
+	 * An index in the user engine map of the context specified by @ctx_id.
+	 */
+	__u32 engine_idx;
+
+	/** @rsvd1: Reserved, MBZ */
+	__u32 rsvd1;
+
+	/**
+	 * @batch_count: Number of batches in @batch_address array.
+	 *
+	 * 0 is invalid. For parallel submission, it should be equal to the
+	 * number of (parallel) engines involved in that submission.
+	 */
+	__u32 batch_count;
+
+	/**
+	 * @batch_address: Array of batch gpu virtual addresses.
+	 *
+	 * If @batch_count is 1, then it is the gpu virtual address of the
+	 * batch buffer. If @batch_count > 1, then it is a pointer to an array
+	 * of batch buffer gpu virtual addresses.
+	 */
+	__u64 batch_address;
+
+	/**
+	 * @flags: Supported flags are:
+	 *
+	 * I915_EXEC3_SECURE:
+	 * Request a privileged ("secure") batch buffer/s.
+	 * It is only available for DRM_ROOT_ONLY | DRM_MASTER processes.
+	 */
+	__u64 flags;
+#define I915_EXEC3_SECURE	(1<<0)
+
+	/** @rsvd2: Reserved, MBZ */
+	__u64 rsvd2;
+
+	/**
+	 * @extensions: Zero-terminated chain of extensions.
+	 *
+	 * DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES:
+	 * It has same format as DRM_I915_GEM_EXECBUFFER_EXT_TIMELINE_FENCES.
+	 * See struct drm_i915_gem_execbuffer_ext_timeline_fences.
+	 * 
+	 * DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE:
+	 * First level batch completion signaling extension.
+	 * See struct drm_i915_gem_execbuffer3_ext_user_fence.
+	 */
+	__u64 extensions;
+#define DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES	0
+#define DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE		1
+};
+
+/**
+ * struct drm_i915_gem_execbuffer3_ext_user_fence - First level batch completion
+ * signaling extension.
+ *
+ * This extension allows user to attach a user fence (@addr, @value pair) to
+ * execbuf3, to be signaled by the command streamer after the completion of first
+ * level batch, by writing the @value at specified @addr and triggering an
+ * interrupt.
+ * User can either poll for this user fence to signal or can also wait on it
+ * with i915_gem_wait_user_fence ioctl.
+ * This is very much usefaul for long running contexts where waiting on dma-fence
+ * by user (like i915_gem_wait ioctl) is not supported.
+ */
+struct drm_i915_gem_execbuffer3_ext_user_fence {
+	/** @base: Extension link. See struct i915_user_extension. */
+	struct i915_user_extension base;
+
+	/**
+	 * @addr: User/Memory fence qword aligned GPU virtual address.
+	 *
+	 * Address has to be a valid GPU virtual address at the time of
+	 * first level batch completion.
+	 */
+	__u64 addr;
+
+	/**
+	 * @value: User/Memory fence Value to be written to above address
+	 * after first level batch completes.
+	 */
+	__u64 value;
+
+	/** @rsvd: Reserved, MBZ */
+	__u64 rsvd;
+};
+
+/**
+ * struct drm_i915_gem_create_ext_vm_private - Extension to make the object
+ * private to the specified VM.
+ *
+ * See struct drm_i915_gem_create_ext.
+ */
+struct drm_i915_gem_create_ext_vm_private {
+#define I915_GEM_CREATE_EXT_VM_PRIVATE		2
+	/** @base: Extension link. See struct i915_user_extension. */
+	struct i915_user_extension base;
+
+	/** @vm_id: Id of the VM to which the object is private */
+	__u32 vm_id;
+};
+
+/**
+ * struct drm_i915_gem_wait_user_fence - Wait on user/memory fence.
+ *
+ * User/Memory fence can be woken up either by:
+ *
+ * 1. GPU context indicated by @ctx_id, or,
+ * 2. Kerrnel driver async worker upon I915_UFENCE_WAIT_SOFT.
+ *    @ctx_id is ignored when this flag is set.
+ *
+ * Wakeup condition is,
+ * ``((*addr & mask) op (value & mask))``
+ *
+ * See :ref:`Documentation/driver-api/dma-buf.rst <indefinite_dma_fences>`
+ */
+struct drm_i915_gem_wait_user_fence {
+	/** @extensions: Zero-terminated chain of extensions. */
+	__u64 extensions;
+
+	/** @addr: User/Memory fence address */
+	__u64 addr;
+
+	/** @ctx_id: Id of the Context which will signal the fence. */
+	__u32 ctx_id;
+
+	/** @op: Wakeup condition operator */
+	__u16 op;
+#define I915_UFENCE_WAIT_EQ      0
+#define I915_UFENCE_WAIT_NEQ     1
+#define I915_UFENCE_WAIT_GT      2
+#define I915_UFENCE_WAIT_GTE     3
+#define I915_UFENCE_WAIT_LT      4
+#define I915_UFENCE_WAIT_LTE     5
+#define I915_UFENCE_WAIT_BEFORE  6
+#define I915_UFENCE_WAIT_AFTER   7
+
+	/**
+	 * @flags: Supported flags are:
+	 *
+	 * I915_UFENCE_WAIT_SOFT:
+	 *
+	 * To be woken up by i915 driver async worker (not by GPU).
+	 *
+	 * I915_UFENCE_WAIT_ABSTIME:
+	 *
+	 * Wait timeout specified as absolute time.
+	 */
+	__u16 flags;
+#define I915_UFENCE_WAIT_SOFT    0x1
+#define I915_UFENCE_WAIT_ABSTIME 0x2
+
+	/** @value: Wakeup value */
+	__u64 value;
+
+	/** @mask: Wakeup mask */
+	__u64 mask;
+#define I915_UFENCE_WAIT_U8     0xffu
+#define I915_UFENCE_WAIT_U16    0xffffu
+#define I915_UFENCE_WAIT_U32    0xfffffffful
+#define I915_UFENCE_WAIT_U64    0xffffffffffffffffull
+
+	/**
+	 * @timeout: Wait timeout in nanoseconds.
+	 *
+	 * If I915_UFENCE_WAIT_ABSTIME flag is set, then time timeout is the
+	 * absolute time in nsec.
+	 */
+	__s64 timeout;
+};
-- 
2.21.0.rc0.32.g243a4c7e27

Re: [Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Matthew Brost]

On Fri, Jun 10, 2022 at 12:07:11AM -0700, Niranjana Vishwanathapura wrote:
> VM_BIND and related uapi definitions
> 
> Signed-off-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>
> ---
>  Documentation/gpu/rfc/i915_vm_bind.h | 490 +++++++++++++++++++++++++++
>  1 file changed, 490 insertions(+)
>  create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
> 
> diff --git a/Documentation/gpu/rfc/i915_vm_bind.h b/Documentation/gpu/rfc/i915_vm_bind.h
> new file mode 100644
> index 000000000000..9fc854969cfb
> --- /dev/null
> +++ b/Documentation/gpu/rfc/i915_vm_bind.h
> @@ -0,0 +1,490 @@
> +/* SPDX-License-Identifier: MIT */
> +/*
> + * Copyright © 2022 Intel Corporation
> + */
> +
> +/**
> + * DOC: I915_PARAM_HAS_VM_BIND
> + *
> + * VM_BIND feature availability.
> + * See typedef drm_i915_getparam_t param.
> + * bit[0]: If set, VM_BIND is supported, otherwise not.
> + * bits[8-15]: VM_BIND implementation version.
> + * version 0 will not have VM_BIND/UNBIND timeline fence array support.
> + */
> +#define I915_PARAM_HAS_VM_BIND		57
> +
> +/**
> + * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
> + *
> + * Flag to opt-in for VM_BIND mode of binding during VM creation.
> + * See struct drm_i915_gem_vm_control flags.
> + *
> + * The older execbuf2 ioctl will not support VM_BIND mode of operation.
> + * For VM_BIND mode, we have new execbuf3 ioctl which will not accept any
> + * execlist (See struct drm_i915_gem_execbuffer3 for more details).
> + *
> + */
> +#define I915_VM_CREATE_FLAGS_USE_VM_BIND	(1 << 0)
> +
> +/**
> + * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
> + *
> + * Flag to declare context as long running.
> + * See struct drm_i915_gem_context_create_ext flags.
> + *
> + * Usage of dma-fence expects that they complete in reasonable amount of time.
> + * Compute on the other hand can be long running. Hence it is not appropriate
> + * for compute contexts to export request completion dma-fence to user.
> + * The dma-fence usage will be limited to in-kernel consumption only.
> + * Compute contexts need to use user/memory fence.
> + *
> + * So, long running contexts do not support output fences. Hence,
> + * I915_EXEC_FENCE_SIGNAL (See &drm_i915_gem_exec_fence.flags) is expected
> + * to be not used. DRM_I915_GEM_WAIT ioctl call is also not supported for
> + * objects mapped to long running contexts.
> + */
> +#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u << 2)
> +
> +/* VM_BIND related ioctls */
> +#define DRM_I915_GEM_VM_BIND		0x3d
> +#define DRM_I915_GEM_VM_UNBIND		0x3e
> +#define DRM_I915_GEM_EXECBUFFER3	0x3f
> +#define DRM_I915_GEM_WAIT_USER_FENCE	0x40
> +
> +#define DRM_IOCTL_I915_GEM_VM_BIND		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_VM_BIND, struct drm_i915_gem_vm_bind)
> +#define DRM_IOCTL_I915_GEM_VM_UNBIND		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_VM_UNBIND, struct drm_i915_gem_vm_bind)
> +#define DRM_IOCTL_I915_GEM_EXECBUFFER3		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_EXECBUFFER3, struct drm_i915_gem_execbuffer3)
> +#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE	DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_WAIT_USER_FENCE, struct drm_i915_gem_wait_user_fence)
> +
> +/**
> + * struct drm_i915_gem_vm_bind - VA to object mapping to bind.
> + *
> + * This structure is passed to VM_BIND ioctl and specifies the mapping of GPU
> + * virtual address (VA) range to the section of an object that should be bound
> + * in the device page table of the specified address space (VM).
> + * The VA range specified must be unique (ie., not currently bound) and can
> + * be mapped to whole object or a section of the object (partial binding).
> + * Multiple VA mappings can be created to the same section of the object
> + * (aliasing).
> + *
> + * The @queue_idx specifies the queue to use for binding. Same queue can be
> + * used for both VM_BIND and VM_UNBIND calls. All submitted bind and unbind
> + * operations in a queue are performed in the order of submission.
> + *
> + * The @start, @offset and @length should be 4K page aligned. However the DG2
> + * and XEHPSDV has 64K page size for device local-memory and has compact page
> + * table. On those platforms, for binding device local-memory objects, the
> + * @start should be 2M aligned, @offset and @length should be 64K aligned.
> + * Also, on those platforms, it is not allowed to bind an device local-memory
> + * object and a system memory object in a single 2M section of VA range.
> + */
> +struct drm_i915_gem_vm_bind {
> +	/** @vm_id: VM (address space) id to bind */
> +	__u32 vm_id;
> +
> +	/** @queue_idx: Index of queue for binding */
> +	__u32 queue_idx;
> +
> +	/** @rsvd: Reserved, MBZ */
> +	__u32 rsvd;
> +
> +	/** @handle: Object handle */
> +	__u32 handle;
> +
> +	/** @start: Virtual Address start to bind */
> +	__u64 start;
> +
> +	/** @offset: Offset in object to bind */
> +	__u64 offset;
> +
> +	/** @length: Length of mapping to bind */
> +	__u64 length;

This probably isn't needed. We are never going to unbind a subset of a
VMA are we? That being said it can't hurt as a sanity check (e.g.
internal vma->length == user unbind length).

> +
> +	/**
> +	 * @flags: Supported flags are:
> +	 *
> +	 * I915_GEM_VM_BIND_READONLY:
> +	 * Mapping is read-only.
> +	 *
> +	 * I915_GEM_VM_BIND_CAPTURE:
> +	 * Capture this mapping in the dump upon GPU error.
> +	 */
> +	__u64 flags;
> +#define I915_GEM_VM_BIND_READONLY    (1 << 0)
> +#define I915_GEM_VM_BIND_CAPTURE     (1 << 1)
> +
> +	/**
> +	 * @extensions: 0-terminated chain of extensions for this operation.
> +	 *
> +	 * I915_VM_BIND_EXT_TIMELINE_FENCES:
> +	 * Specifies an array of input or output timeline fences for this
> +	 * binding operation. See struct drm_i915_vm_bind_ext_timeline_fences.
> +	 *
> +	 * I915_VM_BIND_EXT_USER_FENCES:
> +	 * Specifies an array of input or output user fences for this
> +	 * binding operation. See struct drm_i915_vm_bind_ext_user_fence.
> +	 * This is required for compute contexts.
> +	 */
> +	__u64 extensions;
> +#define I915_VM_BIND_EXT_TIMELINE_FENCES	0
> +#define I915_VM_BIND_EXT_USER_FENCES		1
> +};
> +
> +/**
> + * struct drm_i915_gem_vm_unbind - VA to object mapping to unbind.
> + *
> + * This structure is passed to VM_UNBIND ioctl and specifies the GPU virtual
> + * address (VA) range that should be unbound from the device page table of the
> + * specified address space (VM). The specified VA range must match one of the
> + * mappings created with the VM_BIND ioctl. TLB is flushed upon unbind
> + * completion.
> + *
> + * The @queue_idx specifies the queue to use for unbinding.
> + * See struct drm_i915_gem_vm_unbind for more information on @queue_idx.
> + *
> + * The @start and @length musy specify a unique mapping bound with VM_BIND
> + * ioctl.
> + */
> +struct drm_i915_gem_vm_unbind {
> +	/** @vm_id: VM (address space) id to bind */
> +	__u32 vm_id;
> +
> +	/** @queue_idx: Index of queue for unbinding */
> +	__u32 queue_idx;
> +
> +	/** @start: Virtual Address start to unbind */
> +	__u64 start;
> +
> +	/** @length: Length of mapping to unbind */
> +	__u64 length;
> +
> +	/** @flags: Reserved for future usage, currently MBZ */
> +	__u64 flags;
> +
> +	/**
> +	 * @extensions: 0-terminated chain of extensions for this operation.
> +	 *
> +	 * I915_VM_UNBIND_EXT_TIMELINE_FENCES:
> +	 * Specifies an array of input or output timeline fences for this
> +	 * unbind operation.
> +	 * It has same format as struct drm_i915_vm_bind_ext_timeline_fences.
> +	 *
> +	 * I915_VM_UNBIND_EXT_USER_FENCES:
> +	 * Specifies an array of input or output user fences for this
> +	 * unbind operation. This is required for compute contexts.
> +	 * It has same format as struct drm_i915_vm_bind_ext_user_fence.
> +	 */
> +	__u64 extensions;
> +#define I915_VM_UNBIND_EXT_TIMELINE_FENCES	0
> +#define I915_VM_UNBIND_EXT_USER_FENCES		1
> +};
> +
> +/**
> + * struct drm_i915_vm_bind_fence - An input or output fence for the vm_bind
> + * or the vm_unbind work.
> + *
> + * The vm_bind or vm_unbind aync worker will wait for input fence to signal
> + * before starting the binding or unbinding.
> + *
> + * The vm_bind or vm_unbind async worker will signal the returned output fence
> + * after the completion of binding or unbinding.
> + */
> +struct drm_i915_vm_bind_fence {
> +	/** @handle: User's handle for a drm_syncobj to wait on or signal. */
> +	__u32 handle;
> +
> +	/**
> +	 * @flags: Supported flags are:
> +	 *
> +	 * I915_VM_BIND_FENCE_WAIT:
> +	 * Wait for the input fence before binding/unbinding
> +	 *
> +	 * I915_VM_BIND_FENCE_SIGNAL:
> +	 * Return bind/unbind completion fence as output
> +	 */
> +	__u32 flags;
> +#define I915_VM_BIND_FENCE_WAIT            (1<<0)
> +#define I915_VM_BIND_FENCE_SIGNAL          (1<<1)
> +#define __I915_VM_BIND_FENCE_UNKNOWN_FLAGS (-(I915_VM_BIND_FENCE_SIGNAL << 1))
> +};
> +
> +/**
> + * struct drm_i915_vm_bind_ext_timeline_fences - Timeline fences for vm_bind
> + * and vm_unbind.
> + *
> + * This structure describes an array of timeline drm_syncobj and associated
> + * points for timeline variants of drm_syncobj. These timeline 'drm_syncobj's
> + * can be input or output fences (See struct drm_i915_vm_bind_fence).
> + */
> +struct drm_i915_vm_bind_ext_timeline_fences {
> +	/** @base: Extension link. See struct i915_user_extension. */
> +	struct i915_user_extension base;
> +
> +	/**
> +	 * @fence_count: Number of elements in the @handles_ptr & @value_ptr
> +	 * arrays.
> +	 */
> +	__u64 fence_count;
> +
> +	/**
> +	 * @handles_ptr: Pointer to an array of struct drm_i915_vm_bind_fence
> +	 * of length @fence_count.
> +	 */
> +	__u64 handles_ptr;
> +
> +	/**
> +	 * @values_ptr: Pointer to an array of u64 values of length
> +	 * @fence_count.
> +	 * Values must be 0 for a binary drm_syncobj. A Value of 0 for a
> +	 * timeline drm_syncobj is invalid as it turns a drm_syncobj into a
> +	 * binary one.
> +	 */
> +	__u64 values_ptr;
> +};
> +
> +/**
> + * struct drm_i915_vm_bind_user_fence - An input or output user fence for the
> + * vm_bind or the vm_unbind work.
> + *
> + * The vm_bind or vm_unbind aync worker will wait for the input fence (value at
> + * @addr to become equal to @val) before starting the binding or unbinding.
> + *
> + * The vm_bind or vm_unbind async worker will signal the output fence after
> + * the completion of binding or unbinding by writing @val to memory location at
> + * @addr
> + */
> +struct drm_i915_vm_bind_user_fence {
> +	/** @addr: User/Memory fence qword aligned process virtual address */
> +	__u64 addr;
> +
> +	/** @val: User/Memory fence value to be written after bind completion */
> +	__u64 val;
> +
> +	/**
> +	 * @flags: Supported flags are:
> +	 *
> +	 * I915_VM_BIND_USER_FENCE_WAIT:
> +	 * Wait for the input fence before binding/unbinding
> +	 *
> +	 * I915_VM_BIND_USER_FENCE_SIGNAL:
> +	 * Return bind/unbind completion fence as output
> +	 */
> +	__u32 flags;
> +#define I915_VM_BIND_USER_FENCE_WAIT            (1<<0)
> +#define I915_VM_BIND_USER_FENCE_SIGNAL          (1<<1)
> +#define __I915_VM_BIND_USER_FENCE_UNKNOWN_FLAGS \
> +	(-(I915_VM_BIND_USER_FENCE_SIGNAL << 1))
> +};
> +
> +/**
> + * struct drm_i915_vm_bind_ext_user_fence - User/memory fences for vm_bind
> + * and vm_unbind.
> + *
> + * These user fences can be input or output fences
> + * (See struct drm_i915_vm_bind_user_fence).
> + */
> +struct drm_i915_vm_bind_ext_user_fence {
> +	/** @base: Extension link. See struct i915_user_extension. */
> +	struct i915_user_extension base;
> +
> +	/** @fence_count: Number of elements in the @user_fence_ptr array. */
> +	__u64 fence_count;
> +
> +	/**
> +	 * @user_fence_ptr: Pointer to an array of
> +	 * struct drm_i915_vm_bind_user_fence of length @fence_count.
> +	 */
> +	__u64 user_fence_ptr;
> +};
> +
> +/**
> + * struct drm_i915_gem_execbuffer3 - Structure for DRM_I915_GEM_EXECBUFFER3
> + * ioctl.
> + *
> + * DRM_I915_GEM_EXECBUFFER3 ioctl only works in VM_BIND mode and VM_BIND mode
> + * only works with this ioctl for submission.
> + * See I915_VM_CREATE_FLAGS_USE_VM_BIND.
> + */
> +struct drm_i915_gem_execbuffer3 {
> +	/**
> +	 * @ctx_id: Context id
> +	 *
> +	 * Only contexts with user engine map are allowed.
> +	 */
> +	__u32 ctx_id;
> +
> +	/**
> +	 * @engine_idx: Engine index
> +	 *
> +	 * An index in the user engine map of the context specified by @ctx_id.
> +	 */
> +	__u32 engine_idx;
> +
> +	/** @rsvd1: Reserved, MBZ */
> +	__u32 rsvd1;
> +
> +	/**
> +	 * @batch_count: Number of batches in @batch_address array.
> +	 *
> +	 * 0 is invalid. For parallel submission, it should be equal to the
> +	 * number of (parallel) engines involved in that submission.
> +	 */
> +	__u32 batch_count;
> +
> +	/**
> +	 * @batch_address: Array of batch gpu virtual addresses.
> +	 *
> +	 * If @batch_count is 1, then it is the gpu virtual address of the
> +	 * batch buffer. If @batch_count > 1, then it is a pointer to an array
> +	 * of batch buffer gpu virtual addresses.
> +	 */
> +	__u64 batch_address;
> +
> +	/**
> +	 * @flags: Supported flags are:
> +	 *
> +	 * I915_EXEC3_SECURE:
> +	 * Request a privileged ("secure") batch buffer/s.
> +	 * It is only available for DRM_ROOT_ONLY | DRM_MASTER processes.
> +	 */
> +	__u64 flags;
> +#define I915_EXEC3_SECURE	(1<<0)
> +
> +	/** @rsvd2: Reserved, MBZ */
> +	__u64 rsvd2;
> +
> +	/**
> +	 * @extensions: Zero-terminated chain of extensions.
> +	 *
> +	 * DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES:
> +	 * It has same format as DRM_I915_GEM_EXECBUFFER_EXT_TIMELINE_FENCES.
> +	 * See struct drm_i915_gem_execbuffer_ext_timeline_fences.
> +	 * 
> +	 * DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE:
> +	 * First level batch completion signaling extension.
> +	 * See struct drm_i915_gem_execbuffer3_ext_user_fence.
> +	 */
> +	__u64 extensions;
> +#define DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES	0
> +#define DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE		1
> +};
> +
> +/**
> + * struct drm_i915_gem_execbuffer3_ext_user_fence - First level batch completion
> + * signaling extension.
> + *
> + * This extension allows user to attach a user fence (@addr, @value pair) to
> + * execbuf3, to be signaled by the command streamer after the completion of first
> + * level batch, by writing the @value at specified @addr and triggering an
> + * interrupt.
> + * User can either poll for this user fence to signal or can also wait on it
> + * with i915_gem_wait_user_fence ioctl.
> + * This is very much usefaul for long running contexts where waiting on dma-fence
> + * by user (like i915_gem_wait ioctl) is not supported.
> + */
> +struct drm_i915_gem_execbuffer3_ext_user_fence {
> +	/** @base: Extension link. See struct i915_user_extension. */
> +	struct i915_user_extension base;
> +
> +	/**
> +	 * @addr: User/Memory fence qword aligned GPU virtual address.
> +	 *
> +	 * Address has to be a valid GPU virtual address at the time of
> +	 * first level batch completion.
> +	 */
> +	__u64 addr;
> +
> +	/**
> +	 * @value: User/Memory fence Value to be written to above address
> +	 * after first level batch completes.
> +	 */
> +	__u64 value;
> +
> +	/** @rsvd: Reserved, MBZ */
> +	__u64 rsvd;
> +};
> +

IMO all of these fence structs should be a generic sync interface shared
between both vm bind and exec3 rather than unique extenisons.

Both vm bind and exec3 should have something like this:

__64 syncs;	/* userptr to an array of generic syncs */
__64 n_syncs;

Having an array of syncs lets the kernel do one user copy for all the
syncs rather than reading them in a a chain.

A generic sync object encapsulates all possible syncs (in / out -
syncobj, syncobj timeline, ufence, future sync concepts).

e.g.

struct {
	__u32 user_ext;
	__u32 flag;	/* in / out, type, whatever else info we need */
	union {
		__u32 handle; 	/* to syncobj */
		__u64 addr; 	/* ufence address */
	};
	__64 seqno;	/* syncobj timeline, ufence write value */
	...reserve enough bits for future...
}

This unifies binds and execs by using the same sync interface
instilling the concept that binds and execs are the same op (queue'd
operation /w in/out fences).

Matt

> +/**
> + * struct drm_i915_gem_create_ext_vm_private - Extension to make the object
> + * private to the specified VM.
> + *
> + * See struct drm_i915_gem_create_ext.
> + */
> +struct drm_i915_gem_create_ext_vm_private {
> +#define I915_GEM_CREATE_EXT_VM_PRIVATE		2
> +	/** @base: Extension link. See struct i915_user_extension. */
> +	struct i915_user_extension base;
> +
> +	/** @vm_id: Id of the VM to which the object is private */
> +	__u32 vm_id;
> +};
> +
> +/**
> + * struct drm_i915_gem_wait_user_fence - Wait on user/memory fence.
> + *
> + * User/Memory fence can be woken up either by:
> + *
> + * 1. GPU context indicated by @ctx_id, or,
> + * 2. Kerrnel driver async worker upon I915_UFENCE_WAIT_SOFT.
> + *    @ctx_id is ignored when this flag is set.
> + *
> + * Wakeup condition is,
> + * ``((*addr & mask) op (value & mask))``
> + *
> + * See :ref:`Documentation/driver-api/dma-buf.rst <indefinite_dma_fences>`
> + */
> +struct drm_i915_gem_wait_user_fence {
> +	/** @extensions: Zero-terminated chain of extensions. */
> +	__u64 extensions;
> +
> +	/** @addr: User/Memory fence address */
> +	__u64 addr;
> +
> +	/** @ctx_id: Id of the Context which will signal the fence. */
> +	__u32 ctx_id;
> +
> +	/** @op: Wakeup condition operator */
> +	__u16 op;
> +#define I915_UFENCE_WAIT_EQ      0
> +#define I915_UFENCE_WAIT_NEQ     1
> +#define I915_UFENCE_WAIT_GT      2
> +#define I915_UFENCE_WAIT_GTE     3
> +#define I915_UFENCE_WAIT_LT      4
> +#define I915_UFENCE_WAIT_LTE     5
> +#define I915_UFENCE_WAIT_BEFORE  6
> +#define I915_UFENCE_WAIT_AFTER   7
> +
> +	/**
> +	 * @flags: Supported flags are:
> +	 *
> +	 * I915_UFENCE_WAIT_SOFT:
> +	 *
> +	 * To be woken up by i915 driver async worker (not by GPU).
> +	 *
> +	 * I915_UFENCE_WAIT_ABSTIME:
> +	 *
> +	 * Wait timeout specified as absolute time.
> +	 */
> +	__u16 flags;
> +#define I915_UFENCE_WAIT_SOFT    0x1
> +#define I915_UFENCE_WAIT_ABSTIME 0x2
> +
> +	/** @value: Wakeup value */
> +	__u64 value;
> +
> +	/** @mask: Wakeup mask */
> +	__u64 mask;
> +#define I915_UFENCE_WAIT_U8     0xffu
> +#define I915_UFENCE_WAIT_U16    0xffffu
> +#define I915_UFENCE_WAIT_U32    0xfffffffful
> +#define I915_UFENCE_WAIT_U64    0xffffffffffffffffull
> +
> +	/**
> +	 * @timeout: Wait timeout in nanoseconds.
> +	 *
> +	 * If I915_UFENCE_WAIT_ABSTIME flag is set, then time timeout is the
> +	 * absolute time in nsec.
> +	 */
> +	__s64 timeout;
> +};
> -- 
> 2.21.0.rc0.32.g243a4c7e27
> 

Re: [Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Matthew Brost]

On Fri, Jun 10, 2022 at 01:53:40AM -0700, Matthew Brost wrote:
> On Fri, Jun 10, 2022 at 12:07:11AM -0700, Niranjana Vishwanathapura wrote:
> > VM_BIND and related uapi definitions
> > 
> > Signed-off-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>
> > ---
> >  Documentation/gpu/rfc/i915_vm_bind.h | 490 +++++++++++++++++++++++++++
> >  1 file changed, 490 insertions(+)
> >  create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
> > 
> > diff --git a/Documentation/gpu/rfc/i915_vm_bind.h b/Documentation/gpu/rfc/i915_vm_bind.h
> > new file mode 100644
> > index 000000000000..9fc854969cfb
> > --- /dev/null
> > +++ b/Documentation/gpu/rfc/i915_vm_bind.h
> > @@ -0,0 +1,490 @@
> > +/* SPDX-License-Identifier: MIT */
> > +/*
> > + * Copyright © 2022 Intel Corporation
> > + */
> > +
> > +/**
> > + * DOC: I915_PARAM_HAS_VM_BIND
> > + *
> > + * VM_BIND feature availability.
> > + * See typedef drm_i915_getparam_t param.
> > + * bit[0]: If set, VM_BIND is supported, otherwise not.
> > + * bits[8-15]: VM_BIND implementation version.
> > + * version 0 will not have VM_BIND/UNBIND timeline fence array support.
> > + */
> > +#define I915_PARAM_HAS_VM_BIND		57
> > +
> > +/**
> > + * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
> > + *
> > + * Flag to opt-in for VM_BIND mode of binding during VM creation.
> > + * See struct drm_i915_gem_vm_control flags.
> > + *
> > + * The older execbuf2 ioctl will not support VM_BIND mode of operation.
> > + * For VM_BIND mode, we have new execbuf3 ioctl which will not accept any
> > + * execlist (See struct drm_i915_gem_execbuffer3 for more details).
> > + *
> > + */
> > +#define I915_VM_CREATE_FLAGS_USE_VM_BIND	(1 << 0)
> > +
> > +/**
> > + * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
> > + *
> > + * Flag to declare context as long running.
> > + * See struct drm_i915_gem_context_create_ext flags.
> > + *
> > + * Usage of dma-fence expects that they complete in reasonable amount of time.
> > + * Compute on the other hand can be long running. Hence it is not appropriate
> > + * for compute contexts to export request completion dma-fence to user.
> > + * The dma-fence usage will be limited to in-kernel consumption only.
> > + * Compute contexts need to use user/memory fence.
> > + *
> > + * So, long running contexts do not support output fences. Hence,
> > + * I915_EXEC_FENCE_SIGNAL (See &drm_i915_gem_exec_fence.flags) is expected
> > + * to be not used. DRM_I915_GEM_WAIT ioctl call is also not supported for
> > + * objects mapped to long running contexts.
> > + */
> > +#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u << 2)
> > +
> > +/* VM_BIND related ioctls */
> > +#define DRM_I915_GEM_VM_BIND		0x3d
> > +#define DRM_I915_GEM_VM_UNBIND		0x3e
> > +#define DRM_I915_GEM_EXECBUFFER3	0x3f
> > +#define DRM_I915_GEM_WAIT_USER_FENCE	0x40
> > +
> > +#define DRM_IOCTL_I915_GEM_VM_BIND		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_VM_BIND, struct drm_i915_gem_vm_bind)
> > +#define DRM_IOCTL_I915_GEM_VM_UNBIND		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_VM_UNBIND, struct drm_i915_gem_vm_bind)
> > +#define DRM_IOCTL_I915_GEM_EXECBUFFER3		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_EXECBUFFER3, struct drm_i915_gem_execbuffer3)
> > +#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE	DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_WAIT_USER_FENCE, struct drm_i915_gem_wait_user_fence)
> > +
> > +/**
> > + * struct drm_i915_gem_vm_bind - VA to object mapping to bind.
> > + *
> > + * This structure is passed to VM_BIND ioctl and specifies the mapping of GPU
> > + * virtual address (VA) range to the section of an object that should be bound
> > + * in the device page table of the specified address space (VM).
> > + * The VA range specified must be unique (ie., not currently bound) and can
> > + * be mapped to whole object or a section of the object (partial binding).
> > + * Multiple VA mappings can be created to the same section of the object
> > + * (aliasing).
> > + *
> > + * The @queue_idx specifies the queue to use for binding. Same queue can be
> > + * used for both VM_BIND and VM_UNBIND calls. All submitted bind and unbind
> > + * operations in a queue are performed in the order of submission.
> > + *
> > + * The @start, @offset and @length should be 4K page aligned. However the DG2
> > + * and XEHPSDV has 64K page size for device local-memory and has compact page
> > + * table. On those platforms, for binding device local-memory objects, the
> > + * @start should be 2M aligned, @offset and @length should be 64K aligned.
> > + * Also, on those platforms, it is not allowed to bind an device local-memory
> > + * object and a system memory object in a single 2M section of VA range.
> > + */
> > +struct drm_i915_gem_vm_bind {
> > +	/** @vm_id: VM (address space) id to bind */
> > +	__u32 vm_id;
> > +
> > +	/** @queue_idx: Index of queue for binding */
> > +	__u32 queue_idx;
> > +
> > +	/** @rsvd: Reserved, MBZ */
> > +	__u32 rsvd;
> > +
> > +	/** @handle: Object handle */
> > +	__u32 handle;
> > +
> > +	/** @start: Virtual Address start to bind */
> > +	__u64 start;
> > +
> > +	/** @offset: Offset in object to bind */
> > +	__u64 offset;
> > +
> > +	/** @length: Length of mapping to bind */
> > +	__u64 length;
> 
> This probably isn't needed. We are never going to unbind a subset of a
> VMA are we? That being said it can't hurt as a sanity check (e.g.
> internal vma->length == user unbind length).
> 

Ugh, I c/p this into the wrong place. This should be in the unbind struct.

> > +
> > +	/**
> > +	 * @flags: Supported flags are:
> > +	 *
> > +	 * I915_GEM_VM_BIND_READONLY:
> > +	 * Mapping is read-only.
> > +	 *
> > +	 * I915_GEM_VM_BIND_CAPTURE:
> > +	 * Capture this mapping in the dump upon GPU error.
> > +	 */
> > +	__u64 flags;
> > +#define I915_GEM_VM_BIND_READONLY    (1 << 0)
> > +#define I915_GEM_VM_BIND_CAPTURE     (1 << 1)
> > +
> > +	/**
> > +	 * @extensions: 0-terminated chain of extensions for this operation.
> > +	 *
> > +	 * I915_VM_BIND_EXT_TIMELINE_FENCES:
> > +	 * Specifies an array of input or output timeline fences for this
> > +	 * binding operation. See struct drm_i915_vm_bind_ext_timeline_fences.
> > +	 *
> > +	 * I915_VM_BIND_EXT_USER_FENCES:
> > +	 * Specifies an array of input or output user fences for this
> > +	 * binding operation. See struct drm_i915_vm_bind_ext_user_fence.
> > +	 * This is required for compute contexts.
> > +	 */
> > +	__u64 extensions;
> > +#define I915_VM_BIND_EXT_TIMELINE_FENCES	0
> > +#define I915_VM_BIND_EXT_USER_FENCES		1
> > +};
> > +
> > +/**
> > + * struct drm_i915_gem_vm_unbind - VA to object mapping to unbind.
> > + *
> > + * This structure is passed to VM_UNBIND ioctl and specifies the GPU virtual
> > + * address (VA) range that should be unbound from the device page table of the
> > + * specified address space (VM). The specified VA range must match one of the
> > + * mappings created with the VM_BIND ioctl. TLB is flushed upon unbind
> > + * completion.
> > + *
> > + * The @queue_idx specifies the queue to use for unbinding.
> > + * See struct drm_i915_gem_vm_unbind for more information on @queue_idx.
> > + *
> > + * The @start and @length musy specify a unique mapping bound with VM_BIND
> > + * ioctl.
> > + */
> > +struct drm_i915_gem_vm_unbind {
> > +	/** @vm_id: VM (address space) id to bind */
> > +	__u32 vm_id;
> > +
> > +	/** @queue_idx: Index of queue for unbinding */
> > +	__u32 queue_idx;
> > +
> > +	/** @start: Virtual Address start to unbind */
> > +	__u64 start;
> > +
> > +	/** @length: Length of mapping to unbind */
> > +	__u64 length;
> > +
> > +	/** @flags: Reserved for future usage, currently MBZ */
> > +	__u64 flags;
> > +
> > +	/**
> > +	 * @extensions: 0-terminated chain of extensions for this operation.
> > +	 *
> > +	 * I915_VM_UNBIND_EXT_TIMELINE_FENCES:
> > +	 * Specifies an array of input or output timeline fences for this
> > +	 * unbind operation.
> > +	 * It has same format as struct drm_i915_vm_bind_ext_timeline_fences.
> > +	 *
> > +	 * I915_VM_UNBIND_EXT_USER_FENCES:
> > +	 * Specifies an array of input or output user fences for this
> > +	 * unbind operation. This is required for compute contexts.
> > +	 * It has same format as struct drm_i915_vm_bind_ext_user_fence.
> > +	 */
> > +	__u64 extensions;
> > +#define I915_VM_UNBIND_EXT_TIMELINE_FENCES	0
> > +#define I915_VM_UNBIND_EXT_USER_FENCES		1
> > +};
> > +
> > +/**
> > + * struct drm_i915_vm_bind_fence - An input or output fence for the vm_bind
> > + * or the vm_unbind work.
> > + *
> > + * The vm_bind or vm_unbind aync worker will wait for input fence to signal
> > + * before starting the binding or unbinding.
> > + *
> > + * The vm_bind or vm_unbind async worker will signal the returned output fence
> > + * after the completion of binding or unbinding.
> > + */
> > +struct drm_i915_vm_bind_fence {
> > +	/** @handle: User's handle for a drm_syncobj to wait on or signal. */
> > +	__u32 handle;
> > +
> > +	/**
> > +	 * @flags: Supported flags are:
> > +	 *
> > +	 * I915_VM_BIND_FENCE_WAIT:
> > +	 * Wait for the input fence before binding/unbinding
> > +	 *
> > +	 * I915_VM_BIND_FENCE_SIGNAL:
> > +	 * Return bind/unbind completion fence as output
> > +	 */
> > +	__u32 flags;
> > +#define I915_VM_BIND_FENCE_WAIT            (1<<0)
> > +#define I915_VM_BIND_FENCE_SIGNAL          (1<<1)
> > +#define __I915_VM_BIND_FENCE_UNKNOWN_FLAGS (-(I915_VM_BIND_FENCE_SIGNAL << 1))
> > +};
> > +
> > +/**
> > + * struct drm_i915_vm_bind_ext_timeline_fences - Timeline fences for vm_bind
> > + * and vm_unbind.
> > + *
> > + * This structure describes an array of timeline drm_syncobj and associated
> > + * points for timeline variants of drm_syncobj. These timeline 'drm_syncobj's
> > + * can be input or output fences (See struct drm_i915_vm_bind_fence).
> > + */
> > +struct drm_i915_vm_bind_ext_timeline_fences {
> > +	/** @base: Extension link. See struct i915_user_extension. */
> > +	struct i915_user_extension base;
> > +
> > +	/**
> > +	 * @fence_count: Number of elements in the @handles_ptr & @value_ptr
> > +	 * arrays.
> > +	 */
> > +	__u64 fence_count;
> > +
> > +	/**
> > +	 * @handles_ptr: Pointer to an array of struct drm_i915_vm_bind_fence
> > +	 * of length @fence_count.
> > +	 */
> > +	__u64 handles_ptr;
> > +
> > +	/**
> > +	 * @values_ptr: Pointer to an array of u64 values of length
> > +	 * @fence_count.
> > +	 * Values must be 0 for a binary drm_syncobj. A Value of 0 for a
> > +	 * timeline drm_syncobj is invalid as it turns a drm_syncobj into a
> > +	 * binary one.
> > +	 */
> > +	__u64 values_ptr;
> > +};
> > +
> > +/**
> > + * struct drm_i915_vm_bind_user_fence - An input or output user fence for the
> > + * vm_bind or the vm_unbind work.
> > + *
> > + * The vm_bind or vm_unbind aync worker will wait for the input fence (value at
> > + * @addr to become equal to @val) before starting the binding or unbinding.
> > + *
> > + * The vm_bind or vm_unbind async worker will signal the output fence after
> > + * the completion of binding or unbinding by writing @val to memory location at
> > + * @addr
> > + */
> > +struct drm_i915_vm_bind_user_fence {
> > +	/** @addr: User/Memory fence qword aligned process virtual address */
> > +	__u64 addr;
> > +
> > +	/** @val: User/Memory fence value to be written after bind completion */
> > +	__u64 val;
> > +
> > +	/**
> > +	 * @flags: Supported flags are:
> > +	 *
> > +	 * I915_VM_BIND_USER_FENCE_WAIT:
> > +	 * Wait for the input fence before binding/unbinding
> > +	 *
> > +	 * I915_VM_BIND_USER_FENCE_SIGNAL:
> > +	 * Return bind/unbind completion fence as output
> > +	 */
> > +	__u32 flags;
> > +#define I915_VM_BIND_USER_FENCE_WAIT            (1<<0)
> > +#define I915_VM_BIND_USER_FENCE_SIGNAL          (1<<1)
> > +#define __I915_VM_BIND_USER_FENCE_UNKNOWN_FLAGS \
> > +	(-(I915_VM_BIND_USER_FENCE_SIGNAL << 1))
> > +};
> > +
> > +/**
> > + * struct drm_i915_vm_bind_ext_user_fence - User/memory fences for vm_bind
> > + * and vm_unbind.
> > + *
> > + * These user fences can be input or output fences
> > + * (See struct drm_i915_vm_bind_user_fence).
> > + */
> > +struct drm_i915_vm_bind_ext_user_fence {
> > +	/** @base: Extension link. See struct i915_user_extension. */
> > +	struct i915_user_extension base;
> > +
> > +	/** @fence_count: Number of elements in the @user_fence_ptr array. */
> > +	__u64 fence_count;
> > +
> > +	/**
> > +	 * @user_fence_ptr: Pointer to an array of
> > +	 * struct drm_i915_vm_bind_user_fence of length @fence_count.
> > +	 */
> > +	__u64 user_fence_ptr;
> > +};
> > +
> > +/**
> > + * struct drm_i915_gem_execbuffer3 - Structure for DRM_I915_GEM_EXECBUFFER3
> > + * ioctl.
> > + *
> > + * DRM_I915_GEM_EXECBUFFER3 ioctl only works in VM_BIND mode and VM_BIND mode
> > + * only works with this ioctl for submission.
> > + * See I915_VM_CREATE_FLAGS_USE_VM_BIND.
> > + */
> > +struct drm_i915_gem_execbuffer3 {
> > +	/**
> > +	 * @ctx_id: Context id
> > +	 *
> > +	 * Only contexts with user engine map are allowed.
> > +	 */
> > +	__u32 ctx_id;
> > +
> > +	/**
> > +	 * @engine_idx: Engine index
> > +	 *
> > +	 * An index in the user engine map of the context specified by @ctx_id.
> > +	 */
> > +	__u32 engine_idx;
> > +
> > +	/** @rsvd1: Reserved, MBZ */
> > +	__u32 rsvd1;
> > +
> > +	/**
> > +	 * @batch_count: Number of batches in @batch_address array.
> > +	 *
> > +	 * 0 is invalid. For parallel submission, it should be equal to the
> > +	 * number of (parallel) engines involved in that submission.
> > +	 */
> > +	__u32 batch_count;
> > +
> > +	/**
> > +	 * @batch_address: Array of batch gpu virtual addresses.
> > +	 *
> > +	 * If @batch_count is 1, then it is the gpu virtual address of the
> > +	 * batch buffer. If @batch_count > 1, then it is a pointer to an array
> > +	 * of batch buffer gpu virtual addresses.
> > +	 */
> > +	__u64 batch_address;
> > +
> > +	/**
> > +	 * @flags: Supported flags are:
> > +	 *
> > +	 * I915_EXEC3_SECURE:
> > +	 * Request a privileged ("secure") batch buffer/s.
> > +	 * It is only available for DRM_ROOT_ONLY | DRM_MASTER processes.
> > +	 */
> > +	__u64 flags;
> > +#define I915_EXEC3_SECURE	(1<<0)
> > +
> > +	/** @rsvd2: Reserved, MBZ */
> > +	__u64 rsvd2;
> > +
> > +	/**
> > +	 * @extensions: Zero-terminated chain of extensions.
> > +	 *
> > +	 * DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES:
> > +	 * It has same format as DRM_I915_GEM_EXECBUFFER_EXT_TIMELINE_FENCES.
> > +	 * See struct drm_i915_gem_execbuffer_ext_timeline_fences.
> > +	 * 
> > +	 * DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE:
> > +	 * First level batch completion signaling extension.
> > +	 * See struct drm_i915_gem_execbuffer3_ext_user_fence.
> > +	 */
> > +	__u64 extensions;
> > +#define DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES	0
> > +#define DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE		1
> > +};
> > +
> > +/**
> > + * struct drm_i915_gem_execbuffer3_ext_user_fence - First level batch completion
> > + * signaling extension.
> > + *
> > + * This extension allows user to attach a user fence (@addr, @value pair) to
> > + * execbuf3, to be signaled by the command streamer after the completion of first
> > + * level batch, by writing the @value at specified @addr and triggering an
> > + * interrupt.
> > + * User can either poll for this user fence to signal or can also wait on it
> > + * with i915_gem_wait_user_fence ioctl.
> > + * This is very much usefaul for long running contexts where waiting on dma-fence
> > + * by user (like i915_gem_wait ioctl) is not supported.
> > + */
> > +struct drm_i915_gem_execbuffer3_ext_user_fence {
> > +	/** @base: Extension link. See struct i915_user_extension. */
> > +	struct i915_user_extension base;
> > +
> > +	/**
> > +	 * @addr: User/Memory fence qword aligned GPU virtual address.
> > +	 *
> > +	 * Address has to be a valid GPU virtual address at the time of
> > +	 * first level batch completion.
> > +	 */
> > +	__u64 addr;
> > +
> > +	/**
> > +	 * @value: User/Memory fence Value to be written to above address
> > +	 * after first level batch completes.
> > +	 */
> > +	__u64 value;
> > +
> > +	/** @rsvd: Reserved, MBZ */
> > +	__u64 rsvd;
> > +};
> > +
> 
> IMO all of these fence structs should be a generic sync interface shared
> between both vm bind and exec3 rather than unique extenisons.
> 
> Both vm bind and exec3 should have something like this:
> 
> __64 syncs;	/* userptr to an array of generic syncs */
> __64 n_syncs;
> 
> Having an array of syncs lets the kernel do one user copy for all the
> syncs rather than reading them in a a chain.
> 
> A generic sync object encapsulates all possible syncs (in / out -
> syncobj, syncobj timeline, ufence, future sync concepts).
> 
> e.g.
> 
> struct {
> 	__u32 user_ext;
> 	__u32 flag;	/* in / out, type, whatever else info we need */
> 	union {
> 		__u32 handle; 	/* to syncobj */
> 		__u64 addr; 	/* ufence address */
> 	};
> 	__64 seqno;	/* syncobj timeline, ufence write value */
> 	...reserve enough bits for future...
> }
> 
> This unifies binds and execs by using the same sync interface
> instilling the concept that binds and execs are the same op (queue'd
> operation /w in/out fences).
> 
> Matt
> 
> > +/**
> > + * struct drm_i915_gem_create_ext_vm_private - Extension to make the object
> > + * private to the specified VM.
> > + *
> > + * See struct drm_i915_gem_create_ext.
> > + */
> > +struct drm_i915_gem_create_ext_vm_private {
> > +#define I915_GEM_CREATE_EXT_VM_PRIVATE		2
> > +	/** @base: Extension link. See struct i915_user_extension. */
> > +	struct i915_user_extension base;
> > +
> > +	/** @vm_id: Id of the VM to which the object is private */
> > +	__u32 vm_id;
> > +};
> > +
> > +/**
> > + * struct drm_i915_gem_wait_user_fence - Wait on user/memory fence.
> > + *
> > + * User/Memory fence can be woken up either by:
> > + *
> > + * 1. GPU context indicated by @ctx_id, or,
> > + * 2. Kerrnel driver async worker upon I915_UFENCE_WAIT_SOFT.
> > + *    @ctx_id is ignored when this flag is set.
> > + *
> > + * Wakeup condition is,
> > + * ``((*addr & mask) op (value & mask))``
> > + *
> > + * See :ref:`Documentation/driver-api/dma-buf.rst <indefinite_dma_fences>`
> > + */
> > +struct drm_i915_gem_wait_user_fence {
> > +	/** @extensions: Zero-terminated chain of extensions. */
> > +	__u64 extensions;
> > +
> > +	/** @addr: User/Memory fence address */
> > +	__u64 addr;
> > +
> > +	/** @ctx_id: Id of the Context which will signal the fence. */
> > +	__u32 ctx_id;
> > +
> > +	/** @op: Wakeup condition operator */
> > +	__u16 op;
> > +#define I915_UFENCE_WAIT_EQ      0
> > +#define I915_UFENCE_WAIT_NEQ     1
> > +#define I915_UFENCE_WAIT_GT      2
> > +#define I915_UFENCE_WAIT_GTE     3
> > +#define I915_UFENCE_WAIT_LT      4
> > +#define I915_UFENCE_WAIT_LTE     5
> > +#define I915_UFENCE_WAIT_BEFORE  6
> > +#define I915_UFENCE_WAIT_AFTER   7
> > +
> > +	/**
> > +	 * @flags: Supported flags are:
> > +	 *
> > +	 * I915_UFENCE_WAIT_SOFT:
> > +	 *
> > +	 * To be woken up by i915 driver async worker (not by GPU).
> > +	 *
> > +	 * I915_UFENCE_WAIT_ABSTIME:
> > +	 *
> > +	 * Wait timeout specified as absolute time.
> > +	 */
> > +	__u16 flags;
> > +#define I915_UFENCE_WAIT_SOFT    0x1
> > +#define I915_UFENCE_WAIT_ABSTIME 0x2
> > +
> > +	/** @value: Wakeup value */
> > +	__u64 value;
> > +
> > +	/** @mask: Wakeup mask */
> > +	__u64 mask;
> > +#define I915_UFENCE_WAIT_U8     0xffu
> > +#define I915_UFENCE_WAIT_U16    0xffffu
> > +#define I915_UFENCE_WAIT_U32    0xfffffffful
> > +#define I915_UFENCE_WAIT_U64    0xffffffffffffffffull
> > +
> > +	/**
> > +	 * @timeout: Wait timeout in nanoseconds.
> > +	 *
> > +	 * If I915_UFENCE_WAIT_ABSTIME flag is set, then time timeout is the
> > +	 * absolute time in nsec.
> > +	 */
> > +	__s64 timeout;
> > +};
> > -- 
> > 2.21.0.rc0.32.g243a4c7e27
> > 

Re: [Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Niranjana Vishwanathapura]

On Fri, Jun 10, 2022 at 01:56:58AM -0700, Matthew Brost wrote:
>On Fri, Jun 10, 2022 at 01:53:40AM -0700, Matthew Brost wrote:
>> On Fri, Jun 10, 2022 at 12:07:11AM -0700, Niranjana Vishwanathapura wrote:
>> > VM_BIND and related uapi definitions
>> >
>> > Signed-off-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>
>> > ---
>> >  Documentation/gpu/rfc/i915_vm_bind.h | 490 +++++++++++++++++++++++++++
>> >  1 file changed, 490 insertions(+)
>> >  create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
>> >
>> > diff --git a/Documentation/gpu/rfc/i915_vm_bind.h b/Documentation/gpu/rfc/i915_vm_bind.h
>> > new file mode 100644
>> > index 000000000000..9fc854969cfb
>> > --- /dev/null
>> > +++ b/Documentation/gpu/rfc/i915_vm_bind.h
>> > @@ -0,0 +1,490 @@
>> > +/* SPDX-License-Identifier: MIT */
>> > +/*
>> > + * Copyright © 2022 Intel Corporation
>> > + */
>> > +
>> > +/**
>> > + * DOC: I915_PARAM_HAS_VM_BIND
>> > + *
>> > + * VM_BIND feature availability.
>> > + * See typedef drm_i915_getparam_t param.
>> > + * bit[0]: If set, VM_BIND is supported, otherwise not.
>> > + * bits[8-15]: VM_BIND implementation version.
>> > + * version 0 will not have VM_BIND/UNBIND timeline fence array support.
>> > + */
>> > +#define I915_PARAM_HAS_VM_BIND		57
>> > +
>> > +/**
>> > + * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
>> > + *
>> > + * Flag to opt-in for VM_BIND mode of binding during VM creation.
>> > + * See struct drm_i915_gem_vm_control flags.
>> > + *
>> > + * The older execbuf2 ioctl will not support VM_BIND mode of operation.
>> > + * For VM_BIND mode, we have new execbuf3 ioctl which will not accept any
>> > + * execlist (See struct drm_i915_gem_execbuffer3 for more details).
>> > + *
>> > + */
>> > +#define I915_VM_CREATE_FLAGS_USE_VM_BIND	(1 << 0)
>> > +
>> > +/**
>> > + * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
>> > + *
>> > + * Flag to declare context as long running.
>> > + * See struct drm_i915_gem_context_create_ext flags.
>> > + *
>> > + * Usage of dma-fence expects that they complete in reasonable amount of time.
>> > + * Compute on the other hand can be long running. Hence it is not appropriate
>> > + * for compute contexts to export request completion dma-fence to user.
>> > + * The dma-fence usage will be limited to in-kernel consumption only.
>> > + * Compute contexts need to use user/memory fence.
>> > + *
>> > + * So, long running contexts do not support output fences. Hence,
>> > + * I915_EXEC_FENCE_SIGNAL (See &drm_i915_gem_exec_fence.flags) is expected
>> > + * to be not used. DRM_I915_GEM_WAIT ioctl call is also not supported for
>> > + * objects mapped to long running contexts.
>> > + */
>> > +#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u << 2)
>> > +
>> > +/* VM_BIND related ioctls */
>> > +#define DRM_I915_GEM_VM_BIND		0x3d
>> > +#define DRM_I915_GEM_VM_UNBIND		0x3e
>> > +#define DRM_I915_GEM_EXECBUFFER3	0x3f
>> > +#define DRM_I915_GEM_WAIT_USER_FENCE	0x40
>> > +
>> > +#define DRM_IOCTL_I915_GEM_VM_BIND		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_VM_BIND, struct drm_i915_gem_vm_bind)
>> > +#define DRM_IOCTL_I915_GEM_VM_UNBIND		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_VM_UNBIND, struct drm_i915_gem_vm_bind)
>> > +#define DRM_IOCTL_I915_GEM_EXECBUFFER3		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_EXECBUFFER3, struct drm_i915_gem_execbuffer3)
>> > +#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE	DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_WAIT_USER_FENCE, struct drm_i915_gem_wait_user_fence)
>> > +
>> > +/**
>> > + * struct drm_i915_gem_vm_bind - VA to object mapping to bind.
>> > + *
>> > + * This structure is passed to VM_BIND ioctl and specifies the mapping of GPU
>> > + * virtual address (VA) range to the section of an object that should be bound
>> > + * in the device page table of the specified address space (VM).
>> > + * The VA range specified must be unique (ie., not currently bound) and can
>> > + * be mapped to whole object or a section of the object (partial binding).
>> > + * Multiple VA mappings can be created to the same section of the object
>> > + * (aliasing).
>> > + *
>> > + * The @queue_idx specifies the queue to use for binding. Same queue can be
>> > + * used for both VM_BIND and VM_UNBIND calls. All submitted bind and unbind
>> > + * operations in a queue are performed in the order of submission.
>> > + *
>> > + * The @start, @offset and @length should be 4K page aligned. However the DG2
>> > + * and XEHPSDV has 64K page size for device local-memory and has compact page
>> > + * table. On those platforms, for binding device local-memory objects, the
>> > + * @start should be 2M aligned, @offset and @length should be 64K aligned.
>> > + * Also, on those platforms, it is not allowed to bind an device local-memory
>> > + * object and a system memory object in a single 2M section of VA range.
>> > + */
>> > +struct drm_i915_gem_vm_bind {
>> > +	/** @vm_id: VM (address space) id to bind */
>> > +	__u32 vm_id;
>> > +
>> > +	/** @queue_idx: Index of queue for binding */
>> > +	__u32 queue_idx;
>> > +
>> > +	/** @rsvd: Reserved, MBZ */
>> > +	__u32 rsvd;
>> > +
>> > +	/** @handle: Object handle */
>> > +	__u32 handle;
>> > +
>> > +	/** @start: Virtual Address start to bind */
>> > +	__u64 start;
>> > +
>> > +	/** @offset: Offset in object to bind */
>> > +	__u64 offset;
>> > +
>> > +	/** @length: Length of mapping to bind */
>> > +	__u64 length;
>>
>> This probably isn't needed. We are never going to unbind a subset of a
>> VMA are we? That being said it can't hurt as a sanity check (e.g.
>> internal vma->length == user unbind length).
>>
>
>Ugh, I c/p this into the wrong place. This should be in the unbind struct.

Having the 'length' field for unbind helps if in future we want to allow
unbinding of multiple mappings (vmas) in a single ioctl call. ie., all
mappings that falls in the 'start' - 'start+length-1' range can be
unmapped. We don't support it today as it is somewhat tied to operations
like vma split/merge etc which we are not supporting with initial vm_bind
support.
So yah, currently, it helps in sanity check and ensure user is correctly
unbinding a mapping.

>
>> > +
>> > +	/**
>> > +	 * @flags: Supported flags are:
>> > +	 *
>> > +	 * I915_GEM_VM_BIND_READONLY:
>> > +	 * Mapping is read-only.
>> > +	 *
>> > +	 * I915_GEM_VM_BIND_CAPTURE:
>> > +	 * Capture this mapping in the dump upon GPU error.
>> > +	 */
>> > +	__u64 flags;
>> > +#define I915_GEM_VM_BIND_READONLY    (1 << 0)
>> > +#define I915_GEM_VM_BIND_CAPTURE     (1 << 1)
>> > +
>> > +	/**
>> > +	 * @extensions: 0-terminated chain of extensions for this operation.
>> > +	 *
>> > +	 * I915_VM_BIND_EXT_TIMELINE_FENCES:
>> > +	 * Specifies an array of input or output timeline fences for this
>> > +	 * binding operation. See struct drm_i915_vm_bind_ext_timeline_fences.
>> > +	 *
>> > +	 * I915_VM_BIND_EXT_USER_FENCES:
>> > +	 * Specifies an array of input or output user fences for this
>> > +	 * binding operation. See struct drm_i915_vm_bind_ext_user_fence.
>> > +	 * This is required for compute contexts.
>> > +	 */
>> > +	__u64 extensions;
>> > +#define I915_VM_BIND_EXT_TIMELINE_FENCES	0
>> > +#define I915_VM_BIND_EXT_USER_FENCES		1
>> > +};
>> > +
>> > +/**
>> > + * struct drm_i915_gem_vm_unbind - VA to object mapping to unbind.
>> > + *
>> > + * This structure is passed to VM_UNBIND ioctl and specifies the GPU virtual
>> > + * address (VA) range that should be unbound from the device page table of the
>> > + * specified address space (VM). The specified VA range must match one of the
>> > + * mappings created with the VM_BIND ioctl. TLB is flushed upon unbind
>> > + * completion.
>> > + *
>> > + * The @queue_idx specifies the queue to use for unbinding.
>> > + * See struct drm_i915_gem_vm_unbind for more information on @queue_idx.
>> > + *
>> > + * The @start and @length musy specify a unique mapping bound with VM_BIND
>> > + * ioctl.
>> > + */
>> > +struct drm_i915_gem_vm_unbind {
>> > +	/** @vm_id: VM (address space) id to bind */
>> > +	__u32 vm_id;
>> > +
>> > +	/** @queue_idx: Index of queue for unbinding */
>> > +	__u32 queue_idx;
>> > +
>> > +	/** @start: Virtual Address start to unbind */
>> > +	__u64 start;
>> > +
>> > +	/** @length: Length of mapping to unbind */
>> > +	__u64 length;
>> > +
>> > +	/** @flags: Reserved for future usage, currently MBZ */
>> > +	__u64 flags;
>> > +
>> > +	/**
>> > +	 * @extensions: 0-terminated chain of extensions for this operation.
>> > +	 *
>> > +	 * I915_VM_UNBIND_EXT_TIMELINE_FENCES:
>> > +	 * Specifies an array of input or output timeline fences for this
>> > +	 * unbind operation.
>> > +	 * It has same format as struct drm_i915_vm_bind_ext_timeline_fences.
>> > +	 *
>> > +	 * I915_VM_UNBIND_EXT_USER_FENCES:
>> > +	 * Specifies an array of input or output user fences for this
>> > +	 * unbind operation. This is required for compute contexts.
>> > +	 * It has same format as struct drm_i915_vm_bind_ext_user_fence.
>> > +	 */
>> > +	__u64 extensions;
>> > +#define I915_VM_UNBIND_EXT_TIMELINE_FENCES	0
>> > +#define I915_VM_UNBIND_EXT_USER_FENCES		1
>> > +};
>> > +
>> > +/**
>> > + * struct drm_i915_vm_bind_fence - An input or output fence for the vm_bind
>> > + * or the vm_unbind work.
>> > + *
>> > + * The vm_bind or vm_unbind aync worker will wait for input fence to signal
>> > + * before starting the binding or unbinding.
>> > + *
>> > + * The vm_bind or vm_unbind async worker will signal the returned output fence
>> > + * after the completion of binding or unbinding.
>> > + */
>> > +struct drm_i915_vm_bind_fence {
>> > +	/** @handle: User's handle for a drm_syncobj to wait on or signal. */
>> > +	__u32 handle;
>> > +
>> > +	/**
>> > +	 * @flags: Supported flags are:
>> > +	 *
>> > +	 * I915_VM_BIND_FENCE_WAIT:
>> > +	 * Wait for the input fence before binding/unbinding
>> > +	 *
>> > +	 * I915_VM_BIND_FENCE_SIGNAL:
>> > +	 * Return bind/unbind completion fence as output
>> > +	 */
>> > +	__u32 flags;
>> > +#define I915_VM_BIND_FENCE_WAIT            (1<<0)
>> > +#define I915_VM_BIND_FENCE_SIGNAL          (1<<1)
>> > +#define __I915_VM_BIND_FENCE_UNKNOWN_FLAGS (-(I915_VM_BIND_FENCE_SIGNAL << 1))
>> > +};
>> > +
>> > +/**
>> > + * struct drm_i915_vm_bind_ext_timeline_fences - Timeline fences for vm_bind
>> > + * and vm_unbind.
>> > + *
>> > + * This structure describes an array of timeline drm_syncobj and associated
>> > + * points for timeline variants of drm_syncobj. These timeline 'drm_syncobj's
>> > + * can be input or output fences (See struct drm_i915_vm_bind_fence).
>> > + */
>> > +struct drm_i915_vm_bind_ext_timeline_fences {
>> > +	/** @base: Extension link. See struct i915_user_extension. */
>> > +	struct i915_user_extension base;
>> > +
>> > +	/**
>> > +	 * @fence_count: Number of elements in the @handles_ptr & @value_ptr
>> > +	 * arrays.
>> > +	 */
>> > +	__u64 fence_count;
>> > +
>> > +	/**
>> > +	 * @handles_ptr: Pointer to an array of struct drm_i915_vm_bind_fence
>> > +	 * of length @fence_count.
>> > +	 */
>> > +	__u64 handles_ptr;
>> > +
>> > +	/**
>> > +	 * @values_ptr: Pointer to an array of u64 values of length
>> > +	 * @fence_count.
>> > +	 * Values must be 0 for a binary drm_syncobj. A Value of 0 for a
>> > +	 * timeline drm_syncobj is invalid as it turns a drm_syncobj into a
>> > +	 * binary one.
>> > +	 */
>> > +	__u64 values_ptr;
>> > +};
>> > +
>> > +/**
>> > + * struct drm_i915_vm_bind_user_fence - An input or output user fence for the
>> > + * vm_bind or the vm_unbind work.
>> > + *
>> > + * The vm_bind or vm_unbind aync worker will wait for the input fence (value at
>> > + * @addr to become equal to @val) before starting the binding or unbinding.
>> > + *
>> > + * The vm_bind or vm_unbind async worker will signal the output fence after
>> > + * the completion of binding or unbinding by writing @val to memory location at
>> > + * @addr
>> > + */
>> > +struct drm_i915_vm_bind_user_fence {
>> > +	/** @addr: User/Memory fence qword aligned process virtual address */
>> > +	__u64 addr;
>> > +
>> > +	/** @val: User/Memory fence value to be written after bind completion */
>> > +	__u64 val;
>> > +
>> > +	/**
>> > +	 * @flags: Supported flags are:
>> > +	 *
>> > +	 * I915_VM_BIND_USER_FENCE_WAIT:
>> > +	 * Wait for the input fence before binding/unbinding
>> > +	 *
>> > +	 * I915_VM_BIND_USER_FENCE_SIGNAL:
>> > +	 * Return bind/unbind completion fence as output
>> > +	 */
>> > +	__u32 flags;
>> > +#define I915_VM_BIND_USER_FENCE_WAIT            (1<<0)
>> > +#define I915_VM_BIND_USER_FENCE_SIGNAL          (1<<1)
>> > +#define __I915_VM_BIND_USER_FENCE_UNKNOWN_FLAGS \
>> > +	(-(I915_VM_BIND_USER_FENCE_SIGNAL << 1))
>> > +};
>> > +
>> > +/**
>> > + * struct drm_i915_vm_bind_ext_user_fence - User/memory fences for vm_bind
>> > + * and vm_unbind.
>> > + *
>> > + * These user fences can be input or output fences
>> > + * (See struct drm_i915_vm_bind_user_fence).
>> > + */
>> > +struct drm_i915_vm_bind_ext_user_fence {
>> > +	/** @base: Extension link. See struct i915_user_extension. */
>> > +	struct i915_user_extension base;
>> > +
>> > +	/** @fence_count: Number of elements in the @user_fence_ptr array. */
>> > +	__u64 fence_count;
>> > +
>> > +	/**
>> > +	 * @user_fence_ptr: Pointer to an array of
>> > +	 * struct drm_i915_vm_bind_user_fence of length @fence_count.
>> > +	 */
>> > +	__u64 user_fence_ptr;
>> > +};
>> > +
>> > +/**
>> > + * struct drm_i915_gem_execbuffer3 - Structure for DRM_I915_GEM_EXECBUFFER3
>> > + * ioctl.
>> > + *
>> > + * DRM_I915_GEM_EXECBUFFER3 ioctl only works in VM_BIND mode and VM_BIND mode
>> > + * only works with this ioctl for submission.
>> > + * See I915_VM_CREATE_FLAGS_USE_VM_BIND.
>> > + */
>> > +struct drm_i915_gem_execbuffer3 {
>> > +	/**
>> > +	 * @ctx_id: Context id
>> > +	 *
>> > +	 * Only contexts with user engine map are allowed.
>> > +	 */
>> > +	__u32 ctx_id;
>> > +
>> > +	/**
>> > +	 * @engine_idx: Engine index
>> > +	 *
>> > +	 * An index in the user engine map of the context specified by @ctx_id.
>> > +	 */
>> > +	__u32 engine_idx;
>> > +
>> > +	/** @rsvd1: Reserved, MBZ */
>> > +	__u32 rsvd1;
>> > +
>> > +	/**
>> > +	 * @batch_count: Number of batches in @batch_address array.
>> > +	 *
>> > +	 * 0 is invalid. For parallel submission, it should be equal to the
>> > +	 * number of (parallel) engines involved in that submission.
>> > +	 */
>> > +	__u32 batch_count;
>> > +
>> > +	/**
>> > +	 * @batch_address: Array of batch gpu virtual addresses.
>> > +	 *
>> > +	 * If @batch_count is 1, then it is the gpu virtual address of the
>> > +	 * batch buffer. If @batch_count > 1, then it is a pointer to an array
>> > +	 * of batch buffer gpu virtual addresses.
>> > +	 */
>> > +	__u64 batch_address;
>> > +
>> > +	/**
>> > +	 * @flags: Supported flags are:
>> > +	 *
>> > +	 * I915_EXEC3_SECURE:
>> > +	 * Request a privileged ("secure") batch buffer/s.
>> > +	 * It is only available for DRM_ROOT_ONLY | DRM_MASTER processes.
>> > +	 */
>> > +	__u64 flags;
>> > +#define I915_EXEC3_SECURE	(1<<0)
>> > +
>> > +	/** @rsvd2: Reserved, MBZ */
>> > +	__u64 rsvd2;
>> > +
>> > +	/**
>> > +	 * @extensions: Zero-terminated chain of extensions.
>> > +	 *
>> > +	 * DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES:
>> > +	 * It has same format as DRM_I915_GEM_EXECBUFFER_EXT_TIMELINE_FENCES.
>> > +	 * See struct drm_i915_gem_execbuffer_ext_timeline_fences.
>> > +	 *
>> > +	 * DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE:
>> > +	 * First level batch completion signaling extension.
>> > +	 * See struct drm_i915_gem_execbuffer3_ext_user_fence.
>> > +	 */
>> > +	__u64 extensions;
>> > +#define DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES	0
>> > +#define DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE		1
>> > +};
>> > +
>> > +/**
>> > + * struct drm_i915_gem_execbuffer3_ext_user_fence - First level batch completion
>> > + * signaling extension.
>> > + *
>> > + * This extension allows user to attach a user fence (@addr, @value pair) to
>> > + * execbuf3, to be signaled by the command streamer after the completion of first
>> > + * level batch, by writing the @value at specified @addr and triggering an
>> > + * interrupt.
>> > + * User can either poll for this user fence to signal or can also wait on it
>> > + * with i915_gem_wait_user_fence ioctl.
>> > + * This is very much usefaul for long running contexts where waiting on dma-fence
>> > + * by user (like i915_gem_wait ioctl) is not supported.
>> > + */
>> > +struct drm_i915_gem_execbuffer3_ext_user_fence {
>> > +	/** @base: Extension link. See struct i915_user_extension. */
>> > +	struct i915_user_extension base;
>> > +
>> > +	/**
>> > +	 * @addr: User/Memory fence qword aligned GPU virtual address.
>> > +	 *
>> > +	 * Address has to be a valid GPU virtual address at the time of
>> > +	 * first level batch completion.
>> > +	 */
>> > +	__u64 addr;
>> > +
>> > +	/**
>> > +	 * @value: User/Memory fence Value to be written to above address
>> > +	 * after first level batch completes.
>> > +	 */
>> > +	__u64 value;
>> > +
>> > +	/** @rsvd: Reserved, MBZ */
>> > +	__u64 rsvd;
>> > +};
>> > +
>>
>> IMO all of these fence structs should be a generic sync interface shared
>> between both vm bind and exec3 rather than unique extenisons.
>>
>> Both vm bind and exec3 should have something like this:
>>
>> __64 syncs;	/* userptr to an array of generic syncs */
>> __64 n_syncs;
>>
>> Having an array of syncs lets the kernel do one user copy for all the
>> syncs rather than reading them in a a chain.
>>
>> A generic sync object encapsulates all possible syncs (in / out -
>> syncobj, syncobj timeline, ufence, future sync concepts).
>>
>> e.g.
>>
>> struct {
>> 	__u32 user_ext;
>> 	__u32 flag;	/* in / out, type, whatever else info we need */
>> 	union {
>> 		__u32 handle; 	/* to syncobj */
>> 		__u64 addr; 	/* ufence address */
>> 	};
>> 	__64 seqno;	/* syncobj timeline, ufence write value */
>> 	...reserve enough bits for future...
>> }
>>
>> This unifies binds and execs by using the same sync interface
>> instilling the concept that binds and execs are the same op (queue'd
>> operation /w in/out fences).
>>

Hmm...I am not sure whether we really need to merge all fence types
together in a single structure.

Yes, timeline fence array struct is same between vm_bind and execbuf.
But user fence array is not. For compute use case, VM_BIND/UNBIND can
have user fence array. But execbuf will only have one user fence as
out fence. Execbuf can have timeline fence array as 'in' fences for
compute use case.

Besides, for VM_BIND/UNBIND, user will use either the timeline fence
array (mesa) or the user fence array (compute) and will never mix
them (ie., having some fences as timeline fences and some as user
fences is not allowed). Hence, no multiple copy_from_user concern.
So, not sure if having a generic fence is a good idea. It will be
also less confusing for the user if we define them separately.
But internally i915 can maintain them in a generic sturcture if needed.

Niranjana

>> Matt
>>
>> > +/**
>> > + * struct drm_i915_gem_create_ext_vm_private - Extension to make the object
>> > + * private to the specified VM.
>> > + *
>> > + * See struct drm_i915_gem_create_ext.
>> > + */
>> > +struct drm_i915_gem_create_ext_vm_private {
>> > +#define I915_GEM_CREATE_EXT_VM_PRIVATE		2
>> > +	/** @base: Extension link. See struct i915_user_extension. */
>> > +	struct i915_user_extension base;
>> > +
>> > +	/** @vm_id: Id of the VM to which the object is private */
>> > +	__u32 vm_id;
>> > +};
>> > +
>> > +/**
>> > + * struct drm_i915_gem_wait_user_fence - Wait on user/memory fence.
>> > + *
>> > + * User/Memory fence can be woken up either by:
>> > + *
>> > + * 1. GPU context indicated by @ctx_id, or,
>> > + * 2. Kerrnel driver async worker upon I915_UFENCE_WAIT_SOFT.
>> > + *    @ctx_id is ignored when this flag is set.
>> > + *
>> > + * Wakeup condition is,
>> > + * ``((*addr & mask) op (value & mask))``
>> > + *
>> > + * See :ref:`Documentation/driver-api/dma-buf.rst <indefinite_dma_fences>`
>> > + */
>> > +struct drm_i915_gem_wait_user_fence {
>> > +	/** @extensions: Zero-terminated chain of extensions. */
>> > +	__u64 extensions;
>> > +
>> > +	/** @addr: User/Memory fence address */
>> > +	__u64 addr;
>> > +
>> > +	/** @ctx_id: Id of the Context which will signal the fence. */
>> > +	__u32 ctx_id;
>> > +
>> > +	/** @op: Wakeup condition operator */
>> > +	__u16 op;
>> > +#define I915_UFENCE_WAIT_EQ      0
>> > +#define I915_UFENCE_WAIT_NEQ     1
>> > +#define I915_UFENCE_WAIT_GT      2
>> > +#define I915_UFENCE_WAIT_GTE     3
>> > +#define I915_UFENCE_WAIT_LT      4
>> > +#define I915_UFENCE_WAIT_LTE     5
>> > +#define I915_UFENCE_WAIT_BEFORE  6
>> > +#define I915_UFENCE_WAIT_AFTER   7
>> > +
>> > +	/**
>> > +	 * @flags: Supported flags are:
>> > +	 *
>> > +	 * I915_UFENCE_WAIT_SOFT:
>> > +	 *
>> > +	 * To be woken up by i915 driver async worker (not by GPU).
>> > +	 *
>> > +	 * I915_UFENCE_WAIT_ABSTIME:
>> > +	 *
>> > +	 * Wait timeout specified as absolute time.
>> > +	 */
>> > +	__u16 flags;
>> > +#define I915_UFENCE_WAIT_SOFT    0x1
>> > +#define I915_UFENCE_WAIT_ABSTIME 0x2
>> > +
>> > +	/** @value: Wakeup value */
>> > +	__u64 value;
>> > +
>> > +	/** @mask: Wakeup mask */
>> > +	__u64 mask;
>> > +#define I915_UFENCE_WAIT_U8     0xffu
>> > +#define I915_UFENCE_WAIT_U16    0xffffu
>> > +#define I915_UFENCE_WAIT_U32    0xfffffffful
>> > +#define I915_UFENCE_WAIT_U64    0xffffffffffffffffull
>> > +
>> > +	/**
>> > +	 * @timeout: Wait timeout in nanoseconds.
>> > +	 *
>> > +	 * If I915_UFENCE_WAIT_ABSTIME flag is set, then time timeout is the
>> > +	 * absolute time in nsec.
>> > +	 */
>> > +	__s64 timeout;
>> > +};
>> > --
>> > 2.21.0.rc0.32.g243a4c7e27
>> >

Re: [Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Niranjana Vishwanathapura]

On Fri, Jun 10, 2022 at 09:35:14AM -0700, Niranjana Vishwanathapura wrote:
>On Fri, Jun 10, 2022 at 01:56:58AM -0700, Matthew Brost wrote:
>>On Fri, Jun 10, 2022 at 01:53:40AM -0700, Matthew Brost wrote:
>>>On Fri, Jun 10, 2022 at 12:07:11AM -0700, Niranjana Vishwanathapura wrote:
>>>> VM_BIND and related uapi definitions
>>>>
>>>> Signed-off-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>
>>>> ---
>>>>  Documentation/gpu/rfc/i915_vm_bind.h | 490 +++++++++++++++++++++++++++
>>>>  1 file changed, 490 insertions(+)
>>>>  create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
>>>>
>>>> diff --git a/Documentation/gpu/rfc/i915_vm_bind.h b/Documentation/gpu/rfc/i915_vm_bind.h
>>>> new file mode 100644
>>>> index 000000000000..9fc854969cfb
>>>> --- /dev/null
>>>> +++ b/Documentation/gpu/rfc/i915_vm_bind.h
>>>> @@ -0,0 +1,490 @@
>>>> +/* SPDX-License-Identifier: MIT */
>>>> +/*
>>>> + * Copyright © 2022 Intel Corporation
>>>> + */
>>>> +
>>>> +/**
>>>> + * DOC: I915_PARAM_HAS_VM_BIND
>>>> + *
>>>> + * VM_BIND feature availability.
>>>> + * See typedef drm_i915_getparam_t param.
>>>> + * bit[0]: If set, VM_BIND is supported, otherwise not.
>>>> + * bits[8-15]: VM_BIND implementation version.
>>>> + * version 0 will not have VM_BIND/UNBIND timeline fence array support.
>>>> + */
>>>> +#define I915_PARAM_HAS_VM_BIND		57
>>>> +
>>>> +/**
>>>> + * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
>>>> + *
>>>> + * Flag to opt-in for VM_BIND mode of binding during VM creation.
>>>> + * See struct drm_i915_gem_vm_control flags.
>>>> + *
>>>> + * The older execbuf2 ioctl will not support VM_BIND mode of operation.
>>>> + * For VM_BIND mode, we have new execbuf3 ioctl which will not accept any
>>>> + * execlist (See struct drm_i915_gem_execbuffer3 for more details).
>>>> + *
>>>> + */
>>>> +#define I915_VM_CREATE_FLAGS_USE_VM_BIND	(1 << 0)
>>>> +
>>>> +/**
>>>> + * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
>>>> + *
>>>> + * Flag to declare context as long running.
>>>> + * See struct drm_i915_gem_context_create_ext flags.
>>>> + *
>>>> + * Usage of dma-fence expects that they complete in reasonable amount of time.
>>>> + * Compute on the other hand can be long running. Hence it is not appropriate
>>>> + * for compute contexts to export request completion dma-fence to user.
>>>> + * The dma-fence usage will be limited to in-kernel consumption only.
>>>> + * Compute contexts need to use user/memory fence.
>>>> + *
>>>> + * So, long running contexts do not support output fences. Hence,
>>>> + * I915_EXEC_FENCE_SIGNAL (See &drm_i915_gem_exec_fence.flags) is expected
>>>> + * to be not used. DRM_I915_GEM_WAIT ioctl call is also not supported for
>>>> + * objects mapped to long running contexts.
>>>> + */
>>>> +#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u << 2)
>>>> +
>>>> +/* VM_BIND related ioctls */
>>>> +#define DRM_I915_GEM_VM_BIND		0x3d
>>>> +#define DRM_I915_GEM_VM_UNBIND		0x3e
>>>> +#define DRM_I915_GEM_EXECBUFFER3	0x3f
>>>> +#define DRM_I915_GEM_WAIT_USER_FENCE	0x40
>>>> +
>>>> +#define DRM_IOCTL_I915_GEM_VM_BIND		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_VM_BIND, struct drm_i915_gem_vm_bind)
>>>> +#define DRM_IOCTL_I915_GEM_VM_UNBIND		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_VM_UNBIND, struct drm_i915_gem_vm_bind)
>>>> +#define DRM_IOCTL_I915_GEM_EXECBUFFER3		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_EXECBUFFER3, struct drm_i915_gem_execbuffer3)
>>>> +#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE	DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_WAIT_USER_FENCE, struct drm_i915_gem_wait_user_fence)
>>>> +
>>>> +/**
>>>> + * struct drm_i915_gem_vm_bind - VA to object mapping to bind.
>>>> + *
>>>> + * This structure is passed to VM_BIND ioctl and specifies the mapping of GPU
>>>> + * virtual address (VA) range to the section of an object that should be bound
>>>> + * in the device page table of the specified address space (VM).
>>>> + * The VA range specified must be unique (ie., not currently bound) and can
>>>> + * be mapped to whole object or a section of the object (partial binding).
>>>> + * Multiple VA mappings can be created to the same section of the object
>>>> + * (aliasing).
>>>> + *
>>>> + * The @queue_idx specifies the queue to use for binding. Same queue can be
>>>> + * used for both VM_BIND and VM_UNBIND calls. All submitted bind and unbind
>>>> + * operations in a queue are performed in the order of submission.
>>>> + *
>>>> + * The @start, @offset and @length should be 4K page aligned. However the DG2
>>>> + * and XEHPSDV has 64K page size for device local-memory and has compact page
>>>> + * table. On those platforms, for binding device local-memory objects, the
>>>> + * @start should be 2M aligned, @offset and @length should be 64K aligned.
>>>> + * Also, on those platforms, it is not allowed to bind an device local-memory
>>>> + * object and a system memory object in a single 2M section of VA range.
>>>> + */
>>>> +struct drm_i915_gem_vm_bind {
>>>> +	/** @vm_id: VM (address space) id to bind */
>>>> +	__u32 vm_id;
>>>> +
>>>> +	/** @queue_idx: Index of queue for binding */
>>>> +	__u32 queue_idx;
>>>> +
>>>> +	/** @rsvd: Reserved, MBZ */
>>>> +	__u32 rsvd;
>>>> +
>>>> +	/** @handle: Object handle */
>>>> +	__u32 handle;
>>>> +
>>>> +	/** @start: Virtual Address start to bind */
>>>> +	__u64 start;
>>>> +
>>>> +	/** @offset: Offset in object to bind */
>>>> +	__u64 offset;
>>>> +
>>>> +	/** @length: Length of mapping to bind */
>>>> +	__u64 length;
>>>
>>>This probably isn't needed. We are never going to unbind a subset of a
>>>VMA are we? That being said it can't hurt as a sanity check (e.g.
>>>internal vma->length == user unbind length).
>>>
>>
>>Ugh, I c/p this into the wrong place. This should be in the unbind struct.
>
>Having the 'length' field for unbind helps if in future we want to allow
>unbinding of multiple mappings (vmas) in a single ioctl call. ie., all
>mappings that falls in the 'start' - 'start+length-1' range can be
>unmapped. We don't support it today as it is somewhat tied to operations
>like vma split/merge etc which we are not supporting with initial vm_bind
>support.
>So yah, currently, it helps in sanity check and ensure user is correctly
>unbinding a mapping.
>
>>
>>>> +
>>>> +	/**
>>>> +	 * @flags: Supported flags are:
>>>> +	 *
>>>> +	 * I915_GEM_VM_BIND_READONLY:
>>>> +	 * Mapping is read-only.
>>>> +	 *
>>>> +	 * I915_GEM_VM_BIND_CAPTURE:
>>>> +	 * Capture this mapping in the dump upon GPU error.
>>>> +	 */
>>>> +	__u64 flags;
>>>> +#define I915_GEM_VM_BIND_READONLY    (1 << 0)
>>>> +#define I915_GEM_VM_BIND_CAPTURE     (1 << 1)
>>>> +
>>>> +	/**
>>>> +	 * @extensions: 0-terminated chain of extensions for this operation.
>>>> +	 *
>>>> +	 * I915_VM_BIND_EXT_TIMELINE_FENCES:
>>>> +	 * Specifies an array of input or output timeline fences for this
>>>> +	 * binding operation. See struct drm_i915_vm_bind_ext_timeline_fences.
>>>> +	 *
>>>> +	 * I915_VM_BIND_EXT_USER_FENCES:
>>>> +	 * Specifies an array of input or output user fences for this
>>>> +	 * binding operation. See struct drm_i915_vm_bind_ext_user_fence.
>>>> +	 * This is required for compute contexts.
>>>> +	 */
>>>> +	__u64 extensions;
>>>> +#define I915_VM_BIND_EXT_TIMELINE_FENCES	0
>>>> +#define I915_VM_BIND_EXT_USER_FENCES		1
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct drm_i915_gem_vm_unbind - VA to object mapping to unbind.
>>>> + *
>>>> + * This structure is passed to VM_UNBIND ioctl and specifies the GPU virtual
>>>> + * address (VA) range that should be unbound from the device page table of the
>>>> + * specified address space (VM). The specified VA range must match one of the
>>>> + * mappings created with the VM_BIND ioctl. TLB is flushed upon unbind
>>>> + * completion.
>>>> + *
>>>> + * The @queue_idx specifies the queue to use for unbinding.
>>>> + * See struct drm_i915_gem_vm_unbind for more information on @queue_idx.
>>>> + *
>>>> + * The @start and @length musy specify a unique mapping bound with VM_BIND
>>>> + * ioctl.
>>>> + */
>>>> +struct drm_i915_gem_vm_unbind {
>>>> +	/** @vm_id: VM (address space) id to bind */
>>>> +	__u32 vm_id;
>>>> +
>>>> +	/** @queue_idx: Index of queue for unbinding */
>>>> +	__u32 queue_idx;
>>>> +
>>>> +	/** @start: Virtual Address start to unbind */
>>>> +	__u64 start;
>>>> +
>>>> +	/** @length: Length of mapping to unbind */
>>>> +	__u64 length;
>>>> +
>>>> +	/** @flags: Reserved for future usage, currently MBZ */
>>>> +	__u64 flags;
>>>> +
>>>> +	/**
>>>> +	 * @extensions: 0-terminated chain of extensions for this operation.
>>>> +	 *
>>>> +	 * I915_VM_UNBIND_EXT_TIMELINE_FENCES:
>>>> +	 * Specifies an array of input or output timeline fences for this
>>>> +	 * unbind operation.
>>>> +	 * It has same format as struct drm_i915_vm_bind_ext_timeline_fences.
>>>> +	 *
>>>> +	 * I915_VM_UNBIND_EXT_USER_FENCES:
>>>> +	 * Specifies an array of input or output user fences for this
>>>> +	 * unbind operation. This is required for compute contexts.
>>>> +	 * It has same format as struct drm_i915_vm_bind_ext_user_fence.
>>>> +	 */
>>>> +	__u64 extensions;
>>>> +#define I915_VM_UNBIND_EXT_TIMELINE_FENCES	0
>>>> +#define I915_VM_UNBIND_EXT_USER_FENCES		1
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct drm_i915_vm_bind_fence - An input or output fence for the vm_bind
>>>> + * or the vm_unbind work.
>>>> + *
>>>> + * The vm_bind or vm_unbind aync worker will wait for input fence to signal
>>>> + * before starting the binding or unbinding.
>>>> + *
>>>> + * The vm_bind or vm_unbind async worker will signal the returned output fence
>>>> + * after the completion of binding or unbinding.
>>>> + */
>>>> +struct drm_i915_vm_bind_fence {
>>>> +	/** @handle: User's handle for a drm_syncobj to wait on or signal. */
>>>> +	__u32 handle;
>>>> +
>>>> +	/**
>>>> +	 * @flags: Supported flags are:
>>>> +	 *
>>>> +	 * I915_VM_BIND_FENCE_WAIT:
>>>> +	 * Wait for the input fence before binding/unbinding
>>>> +	 *
>>>> +	 * I915_VM_BIND_FENCE_SIGNAL:
>>>> +	 * Return bind/unbind completion fence as output
>>>> +	 */
>>>> +	__u32 flags;
>>>> +#define I915_VM_BIND_FENCE_WAIT            (1<<0)
>>>> +#define I915_VM_BIND_FENCE_SIGNAL          (1<<1)
>>>> +#define __I915_VM_BIND_FENCE_UNKNOWN_FLAGS (-(I915_VM_BIND_FENCE_SIGNAL << 1))
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct drm_i915_vm_bind_ext_timeline_fences - Timeline fences for vm_bind
>>>> + * and vm_unbind.
>>>> + *
>>>> + * This structure describes an array of timeline drm_syncobj and associated
>>>> + * points for timeline variants of drm_syncobj. These timeline 'drm_syncobj's
>>>> + * can be input or output fences (See struct drm_i915_vm_bind_fence).
>>>> + */
>>>> +struct drm_i915_vm_bind_ext_timeline_fences {
>>>> +	/** @base: Extension link. See struct i915_user_extension. */
>>>> +	struct i915_user_extension base;
>>>> +
>>>> +	/**
>>>> +	 * @fence_count: Number of elements in the @handles_ptr & @value_ptr
>>>> +	 * arrays.
>>>> +	 */
>>>> +	__u64 fence_count;
>>>> +
>>>> +	/**
>>>> +	 * @handles_ptr: Pointer to an array of struct drm_i915_vm_bind_fence
>>>> +	 * of length @fence_count.
>>>> +	 */
>>>> +	__u64 handles_ptr;
>>>> +
>>>> +	/**
>>>> +	 * @values_ptr: Pointer to an array of u64 values of length
>>>> +	 * @fence_count.
>>>> +	 * Values must be 0 for a binary drm_syncobj. A Value of 0 for a
>>>> +	 * timeline drm_syncobj is invalid as it turns a drm_syncobj into a
>>>> +	 * binary one.
>>>> +	 */
>>>> +	__u64 values_ptr;
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct drm_i915_vm_bind_user_fence - An input or output user fence for the
>>>> + * vm_bind or the vm_unbind work.
>>>> + *
>>>> + * The vm_bind or vm_unbind aync worker will wait for the input fence (value at
>>>> + * @addr to become equal to @val) before starting the binding or unbinding.
>>>> + *
>>>> + * The vm_bind or vm_unbind async worker will signal the output fence after
>>>> + * the completion of binding or unbinding by writing @val to memory location at
>>>> + * @addr
>>>> + */
>>>> +struct drm_i915_vm_bind_user_fence {
>>>> +	/** @addr: User/Memory fence qword aligned process virtual address */
>>>> +	__u64 addr;
>>>> +
>>>> +	/** @val: User/Memory fence value to be written after bind completion */
>>>> +	__u64 val;
>>>> +
>>>> +	/**
>>>> +	 * @flags: Supported flags are:
>>>> +	 *
>>>> +	 * I915_VM_BIND_USER_FENCE_WAIT:
>>>> +	 * Wait for the input fence before binding/unbinding
>>>> +	 *
>>>> +	 * I915_VM_BIND_USER_FENCE_SIGNAL:
>>>> +	 * Return bind/unbind completion fence as output
>>>> +	 */
>>>> +	__u32 flags;
>>>> +#define I915_VM_BIND_USER_FENCE_WAIT            (1<<0)
>>>> +#define I915_VM_BIND_USER_FENCE_SIGNAL          (1<<1)
>>>> +#define __I915_VM_BIND_USER_FENCE_UNKNOWN_FLAGS \
>>>> +	(-(I915_VM_BIND_USER_FENCE_SIGNAL << 1))
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct drm_i915_vm_bind_ext_user_fence - User/memory fences for vm_bind
>>>> + * and vm_unbind.
>>>> + *
>>>> + * These user fences can be input or output fences
>>>> + * (See struct drm_i915_vm_bind_user_fence).
>>>> + */
>>>> +struct drm_i915_vm_bind_ext_user_fence {
>>>> +	/** @base: Extension link. See struct i915_user_extension. */
>>>> +	struct i915_user_extension base;
>>>> +
>>>> +	/** @fence_count: Number of elements in the @user_fence_ptr array. */
>>>> +	__u64 fence_count;
>>>> +
>>>> +	/**
>>>> +	 * @user_fence_ptr: Pointer to an array of
>>>> +	 * struct drm_i915_vm_bind_user_fence of length @fence_count.
>>>> +	 */
>>>> +	__u64 user_fence_ptr;
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct drm_i915_gem_execbuffer3 - Structure for DRM_I915_GEM_EXECBUFFER3
>>>> + * ioctl.
>>>> + *
>>>> + * DRM_I915_GEM_EXECBUFFER3 ioctl only works in VM_BIND mode and VM_BIND mode
>>>> + * only works with this ioctl for submission.
>>>> + * See I915_VM_CREATE_FLAGS_USE_VM_BIND.
>>>> + */
>>>> +struct drm_i915_gem_execbuffer3 {
>>>> +	/**
>>>> +	 * @ctx_id: Context id
>>>> +	 *
>>>> +	 * Only contexts with user engine map are allowed.
>>>> +	 */
>>>> +	__u32 ctx_id;
>>>> +
>>>> +	/**
>>>> +	 * @engine_idx: Engine index
>>>> +	 *
>>>> +	 * An index in the user engine map of the context specified by @ctx_id.
>>>> +	 */
>>>> +	__u32 engine_idx;
>>>> +
>>>> +	/** @rsvd1: Reserved, MBZ */
>>>> +	__u32 rsvd1;
>>>> +
>>>> +	/**
>>>> +	 * @batch_count: Number of batches in @batch_address array.
>>>> +	 *
>>>> +	 * 0 is invalid. For parallel submission, it should be equal to the
>>>> +	 * number of (parallel) engines involved in that submission.
>>>> +	 */
>>>> +	__u32 batch_count;
>>>> +
>>>> +	/**
>>>> +	 * @batch_address: Array of batch gpu virtual addresses.
>>>> +	 *
>>>> +	 * If @batch_count is 1, then it is the gpu virtual address of the
>>>> +	 * batch buffer. If @batch_count > 1, then it is a pointer to an array
>>>> +	 * of batch buffer gpu virtual addresses.
>>>> +	 */
>>>> +	__u64 batch_address;
>>>> +
>>>> +	/**
>>>> +	 * @flags: Supported flags are:
>>>> +	 *
>>>> +	 * I915_EXEC3_SECURE:
>>>> +	 * Request a privileged ("secure") batch buffer/s.
>>>> +	 * It is only available for DRM_ROOT_ONLY | DRM_MASTER processes.
>>>> +	 */
>>>> +	__u64 flags;
>>>> +#define I915_EXEC3_SECURE	(1<<0)
>>>> +
>>>> +	/** @rsvd2: Reserved, MBZ */
>>>> +	__u64 rsvd2;
>>>> +
>>>> +	/**
>>>> +	 * @extensions: Zero-terminated chain of extensions.
>>>> +	 *
>>>> +	 * DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES:
>>>> +	 * It has same format as DRM_I915_GEM_EXECBUFFER_EXT_TIMELINE_FENCES.
>>>> +	 * See struct drm_i915_gem_execbuffer_ext_timeline_fences.
>>>> +	 *
>>>> +	 * DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE:
>>>> +	 * First level batch completion signaling extension.
>>>> +	 * See struct drm_i915_gem_execbuffer3_ext_user_fence.
>>>> +	 */
>>>> +	__u64 extensions;
>>>> +#define DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES	0
>>>> +#define DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE		1
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct drm_i915_gem_execbuffer3_ext_user_fence - First level batch completion
>>>> + * signaling extension.
>>>> + *
>>>> + * This extension allows user to attach a user fence (@addr, @value pair) to
>>>> + * execbuf3, to be signaled by the command streamer after the completion of first
>>>> + * level batch, by writing the @value at specified @addr and triggering an
>>>> + * interrupt.
>>>> + * User can either poll for this user fence to signal or can also wait on it
>>>> + * with i915_gem_wait_user_fence ioctl.
>>>> + * This is very much usefaul for long running contexts where waiting on dma-fence
>>>> + * by user (like i915_gem_wait ioctl) is not supported.
>>>> + */
>>>> +struct drm_i915_gem_execbuffer3_ext_user_fence {
>>>> +	/** @base: Extension link. See struct i915_user_extension. */
>>>> +	struct i915_user_extension base;
>>>> +
>>>> +	/**
>>>> +	 * @addr: User/Memory fence qword aligned GPU virtual address.
>>>> +	 *
>>>> +	 * Address has to be a valid GPU virtual address at the time of
>>>> +	 * first level batch completion.
>>>> +	 */
>>>> +	__u64 addr;
>>>> +
>>>> +	/**
>>>> +	 * @value: User/Memory fence Value to be written to above address
>>>> +	 * after first level batch completes.
>>>> +	 */
>>>> +	__u64 value;
>>>> +
>>>> +	/** @rsvd: Reserved, MBZ */
>>>> +	__u64 rsvd;
>>>> +};
>>>> +
>>>
>>>IMO all of these fence structs should be a generic sync interface shared
>>>between both vm bind and exec3 rather than unique extenisons.
>>>
>>>Both vm bind and exec3 should have something like this:
>>>
>>>__64 syncs;	/* userptr to an array of generic syncs */
>>>__64 n_syncs;
>>>
>>>Having an array of syncs lets the kernel do one user copy for all the
>>>syncs rather than reading them in a a chain.
>>>
>>>A generic sync object encapsulates all possible syncs (in / out -
>>>syncobj, syncobj timeline, ufence, future sync concepts).
>>>
>>>e.g.
>>>
>>>struct {
>>>	__u32 user_ext;
>>>	__u32 flag;	/* in / out, type, whatever else info we need */
>>>	union {
>>>		__u32 handle; 	/* to syncobj */
>>>		__u64 addr; 	/* ufence address */
>>>	};
>>>	__64 seqno;	/* syncobj timeline, ufence write value */
>>>	...reserve enough bits for future...
>>>}
>>>
>>>This unifies binds and execs by using the same sync interface
>>>instilling the concept that binds and execs are the same op (queue'd
>>>operation /w in/out fences).
>>>
>
>Hmm...I am not sure whether we really need to merge all fence types
>together in a single structure.
>
>Yes, timeline fence array struct is same between vm_bind and execbuf.
>But user fence array is not. For compute use case, VM_BIND/UNBIND can
>have user fence array. But execbuf will only have one user fence as
>out fence. Execbuf can have timeline fence array as 'in' fences for
>compute use case.
>
>Besides, for VM_BIND/UNBIND, user will use either the timeline fence
>array (mesa) or the user fence array (compute) and will never mix
>them (ie., having some fences as timeline fences and some as user
>fences is not allowed). Hence, no multiple copy_from_user concern.
>So, not sure if having a generic fence is a good idea. It will be
>also less confusing for the user if we define them separately.
>But internally i915 can maintain them in a generic sturcture if needed.
>

Also, I don't think we will have a case where an array of fences
passed by user will have a mix of timeline and user fences.
So, we don't need a 'type' in each element of the array. The
extension (timeline or user fence) will be the type.

Niranjana

>Niranjana
>
>>>Matt
>>>
>>>> +/**
>>>> + * struct drm_i915_gem_create_ext_vm_private - Extension to make the object
>>>> + * private to the specified VM.
>>>> + *
>>>> + * See struct drm_i915_gem_create_ext.
>>>> + */
>>>> +struct drm_i915_gem_create_ext_vm_private {
>>>> +#define I915_GEM_CREATE_EXT_VM_PRIVATE		2
>>>> +	/** @base: Extension link. See struct i915_user_extension. */
>>>> +	struct i915_user_extension base;
>>>> +
>>>> +	/** @vm_id: Id of the VM to which the object is private */
>>>> +	__u32 vm_id;
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct drm_i915_gem_wait_user_fence - Wait on user/memory fence.
>>>> + *
>>>> + * User/Memory fence can be woken up either by:
>>>> + *
>>>> + * 1. GPU context indicated by @ctx_id, or,
>>>> + * 2. Kerrnel driver async worker upon I915_UFENCE_WAIT_SOFT.
>>>> + *    @ctx_id is ignored when this flag is set.
>>>> + *
>>>> + * Wakeup condition is,
>>>> + * ``((*addr & mask) op (value & mask))``
>>>> + *
>>>> + * See :ref:`Documentation/driver-api/dma-buf.rst <indefinite_dma_fences>`
>>>> + */
>>>> +struct drm_i915_gem_wait_user_fence {
>>>> +	/** @extensions: Zero-terminated chain of extensions. */
>>>> +	__u64 extensions;
>>>> +
>>>> +	/** @addr: User/Memory fence address */
>>>> +	__u64 addr;
>>>> +
>>>> +	/** @ctx_id: Id of the Context which will signal the fence. */
>>>> +	__u32 ctx_id;
>>>> +
>>>> +	/** @op: Wakeup condition operator */
>>>> +	__u16 op;
>>>> +#define I915_UFENCE_WAIT_EQ      0
>>>> +#define I915_UFENCE_WAIT_NEQ     1
>>>> +#define I915_UFENCE_WAIT_GT      2
>>>> +#define I915_UFENCE_WAIT_GTE     3
>>>> +#define I915_UFENCE_WAIT_LT      4
>>>> +#define I915_UFENCE_WAIT_LTE     5
>>>> +#define I915_UFENCE_WAIT_BEFORE  6
>>>> +#define I915_UFENCE_WAIT_AFTER   7
>>>> +
>>>> +	/**
>>>> +	 * @flags: Supported flags are:
>>>> +	 *
>>>> +	 * I915_UFENCE_WAIT_SOFT:
>>>> +	 *
>>>> +	 * To be woken up by i915 driver async worker (not by GPU).
>>>> +	 *
>>>> +	 * I915_UFENCE_WAIT_ABSTIME:
>>>> +	 *
>>>> +	 * Wait timeout specified as absolute time.
>>>> +	 */
>>>> +	__u16 flags;
>>>> +#define I915_UFENCE_WAIT_SOFT    0x1
>>>> +#define I915_UFENCE_WAIT_ABSTIME 0x2
>>>> +
>>>> +	/** @value: Wakeup value */
>>>> +	__u64 value;
>>>> +
>>>> +	/** @mask: Wakeup mask */
>>>> +	__u64 mask;
>>>> +#define I915_UFENCE_WAIT_U8     0xffu
>>>> +#define I915_UFENCE_WAIT_U16    0xffffu
>>>> +#define I915_UFENCE_WAIT_U32    0xfffffffful
>>>> +#define I915_UFENCE_WAIT_U64    0xffffffffffffffffull
>>>> +
>>>> +	/**
>>>> +	 * @timeout: Wait timeout in nanoseconds.
>>>> +	 *
>>>> +	 * If I915_UFENCE_WAIT_ABSTIME flag is set, then time timeout is the
>>>> +	 * absolute time in nsec.
>>>> +	 */
>>>> +	__s64 timeout;
>>>> +};
>>>> --
>>>> 2.21.0.rc0.32.g243a4c7e27
>>>>

Re: [Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Lionel Landwerlin]

On 10/06/2022 11:53, Matthew Brost wrote:
> On Fri, Jun 10, 2022 at 12:07:11AM -0700, Niranjana Vishwanathapura wrote:
>> VM_BIND and related uapi definitions
>>
>> Signed-off-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>
>> ---
>>   Documentation/gpu/rfc/i915_vm_bind.h | 490 +++++++++++++++++++++++++++
>>   1 file changed, 490 insertions(+)
>>   create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
>>
>> diff --git a/Documentation/gpu/rfc/i915_vm_bind.h b/Documentation/gpu/rfc/i915_vm_bind.h
>> new file mode 100644
>> index 000000000000..9fc854969cfb
>> --- /dev/null
>> +++ b/Documentation/gpu/rfc/i915_vm_bind.h
>> @@ -0,0 +1,490 @@
>> +/* SPDX-License-Identifier: MIT */
>> +/*
>> + * Copyright © 2022 Intel Corporation
>> + */
>> +
>> +/**
>> + * DOC: I915_PARAM_HAS_VM_BIND
>> + *
>> + * VM_BIND feature availability.
>> + * See typedef drm_i915_getparam_t param.
>> + * bit[0]: If set, VM_BIND is supported, otherwise not.
>> + * bits[8-15]: VM_BIND implementation version.
>> + * version 0 will not have VM_BIND/UNBIND timeline fence array support.
>> + */
>> +#define I915_PARAM_HAS_VM_BIND		57
>> +
>> +/**
>> + * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
>> + *
>> + * Flag to opt-in for VM_BIND mode of binding during VM creation.
>> + * See struct drm_i915_gem_vm_control flags.
>> + *
>> + * The older execbuf2 ioctl will not support VM_BIND mode of operation.
>> + * For VM_BIND mode, we have new execbuf3 ioctl which will not accept any
>> + * execlist (See struct drm_i915_gem_execbuffer3 for more details).
>> + *
>> + */
>> +#define I915_VM_CREATE_FLAGS_USE_VM_BIND	(1 << 0)
>> +
>> +/**
>> + * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
>> + *
>> + * Flag to declare context as long running.
>> + * See struct drm_i915_gem_context_create_ext flags.
>> + *
>> + * Usage of dma-fence expects that they complete in reasonable amount of time.
>> + * Compute on the other hand can be long running. Hence it is not appropriate
>> + * for compute contexts to export request completion dma-fence to user.
>> + * The dma-fence usage will be limited to in-kernel consumption only.
>> + * Compute contexts need to use user/memory fence.
>> + *
>> + * So, long running contexts do not support output fences. Hence,
>> + * I915_EXEC_FENCE_SIGNAL (See &drm_i915_gem_exec_fence.flags) is expected
>> + * to be not used. DRM_I915_GEM_WAIT ioctl call is also not supported for
>> + * objects mapped to long running contexts.
>> + */
>> +#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u << 2)
>> +
>> +/* VM_BIND related ioctls */
>> +#define DRM_I915_GEM_VM_BIND		0x3d
>> +#define DRM_I915_GEM_VM_UNBIND		0x3e
>> +#define DRM_I915_GEM_EXECBUFFER3	0x3f
>> +#define DRM_I915_GEM_WAIT_USER_FENCE	0x40
>> +
>> +#define DRM_IOCTL_I915_GEM_VM_BIND		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_VM_BIND, struct drm_i915_gem_vm_bind)
>> +#define DRM_IOCTL_I915_GEM_VM_UNBIND		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_VM_UNBIND, struct drm_i915_gem_vm_bind)
>> +#define DRM_IOCTL_I915_GEM_EXECBUFFER3		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_EXECBUFFER3, struct drm_i915_gem_execbuffer3)
>> +#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE	DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_WAIT_USER_FENCE, struct drm_i915_gem_wait_user_fence)
>> +
>> +/**
>> + * struct drm_i915_gem_vm_bind - VA to object mapping to bind.
>> + *
>> + * This structure is passed to VM_BIND ioctl and specifies the mapping of GPU
>> + * virtual address (VA) range to the section of an object that should be bound
>> + * in the device page table of the specified address space (VM).
>> + * The VA range specified must be unique (ie., not currently bound) and can
>> + * be mapped to whole object or a section of the object (partial binding).
>> + * Multiple VA mappings can be created to the same section of the object
>> + * (aliasing).
>> + *
>> + * The @queue_idx specifies the queue to use for binding. Same queue can be
>> + * used for both VM_BIND and VM_UNBIND calls. All submitted bind and unbind
>> + * operations in a queue are performed in the order of submission.
>> + *
>> + * The @start, @offset and @length should be 4K page aligned. However the DG2
>> + * and XEHPSDV has 64K page size for device local-memory and has compact page
>> + * table. On those platforms, for binding device local-memory objects, the
>> + * @start should be 2M aligned, @offset and @length should be 64K aligned.
>> + * Also, on those platforms, it is not allowed to bind an device local-memory
>> + * object and a system memory object in a single 2M section of VA range.
>> + */
>> +struct drm_i915_gem_vm_bind {
>> +	/** @vm_id: VM (address space) id to bind */
>> +	__u32 vm_id;
>> +
>> +	/** @queue_idx: Index of queue for binding */
>> +	__u32 queue_idx;
>> +
>> +	/** @rsvd: Reserved, MBZ */
>> +	__u32 rsvd;
>> +
>> +	/** @handle: Object handle */
>> +	__u32 handle;
>> +
>> +	/** @start: Virtual Address start to bind */
>> +	__u64 start;
>> +
>> +	/** @offset: Offset in object to bind */
>> +	__u64 offset;
>> +
>> +	/** @length: Length of mapping to bind */
>> +	__u64 length;
> This probably isn't needed. We are never going to unbind a subset of a
> VMA are we? That being said it can't hurt as a sanity check (e.g.
> internal vma->length == user unbind length).


Not sure what you mean by that.


Vulkan can unbind a whole range of addresses and it seems like there is 
no restriction on doing something like this :


bind vma=0x1000000 GEMBO=3 offset=0 range=8192

unbind vma=0x1001000 range=4096


You would be left with a single 4k page of GEMBO=3 bound at vma=0x1000000


Or :

bind vma=0x1000000 GEMBO=3 offset=0 range=4096

bind vma=0x1001000 GEMBO=4 offset=0 range=4096

unbind vma=0x1000000 range=8192


You're unbinding 2 bindings with a single operation.


-Lionel


>
>> +
>> +	/**
>> +	 * @flags: Supported flags are:
>> +	 *
>> +	 * I915_GEM_VM_BIND_READONLY:
>> +	 * Mapping is read-only.
>> +	 *
>> +	 * I915_GEM_VM_BIND_CAPTURE:
>> +	 * Capture this mapping in the dump upon GPU error.
>> +	 */
>> +	__u64 flags;
>> +#define I915_GEM_VM_BIND_READONLY    (1 << 0)
>> +#define I915_GEM_VM_BIND_CAPTURE     (1 << 1)
>> +
>> +	/**
>> +	 * @extensions: 0-terminated chain of extensions for this operation.
>> +	 *
>> +	 * I915_VM_BIND_EXT_TIMELINE_FENCES:
>> +	 * Specifies an array of input or output timeline fences for this
>> +	 * binding operation. See struct drm_i915_vm_bind_ext_timeline_fences.
>> +	 *
>> +	 * I915_VM_BIND_EXT_USER_FENCES:
>> +	 * Specifies an array of input or output user fences for this
>> +	 * binding operation. See struct drm_i915_vm_bind_ext_user_fence.
>> +	 * This is required for compute contexts.
>> +	 */
>> +	__u64 extensions;
>> +#define I915_VM_BIND_EXT_TIMELINE_FENCES	0
>> +#define I915_VM_BIND_EXT_USER_FENCES		1
>> +};
>> +
>> +/**
>> + * struct drm_i915_gem_vm_unbind - VA to object mapping to unbind.
>> + *
>> + * This structure is passed to VM_UNBIND ioctl and specifies the GPU virtual
>> + * address (VA) range that should be unbound from the device page table of the
>> + * specified address space (VM). The specified VA range must match one of the
>> + * mappings created with the VM_BIND ioctl. TLB is flushed upon unbind
>> + * completion.
>> + *
>> + * The @queue_idx specifies the queue to use for unbinding.
>> + * See struct drm_i915_gem_vm_unbind for more information on @queue_idx.
>> + *
>> + * The @start and @length musy specify a unique mapping bound with VM_BIND
>> + * ioctl.
>> + */
>> +struct drm_i915_gem_vm_unbind {
>> +	/** @vm_id: VM (address space) id to bind */
>> +	__u32 vm_id;
>> +
>> +	/** @queue_idx: Index of queue for unbinding */
>> +	__u32 queue_idx;
>> +
>> +	/** @start: Virtual Address start to unbind */
>> +	__u64 start;
>> +
>> +	/** @length: Length of mapping to unbind */
>> +	__u64 length;
>> +
>> +	/** @flags: Reserved for future usage, currently MBZ */
>> +	__u64 flags;
>> +
>> +	/**
>> +	 * @extensions: 0-terminated chain of extensions for this operation.
>> +	 *
>> +	 * I915_VM_UNBIND_EXT_TIMELINE_FENCES:
>> +	 * Specifies an array of input or output timeline fences for this
>> +	 * unbind operation.
>> +	 * It has same format as struct drm_i915_vm_bind_ext_timeline_fences.
>> +	 *
>> +	 * I915_VM_UNBIND_EXT_USER_FENCES:
>> +	 * Specifies an array of input or output user fences for this
>> +	 * unbind operation. This is required for compute contexts.
>> +	 * It has same format as struct drm_i915_vm_bind_ext_user_fence.
>> +	 */
>> +	__u64 extensions;
>> +#define I915_VM_UNBIND_EXT_TIMELINE_FENCES	0
>> +#define I915_VM_UNBIND_EXT_USER_FENCES		1
>> +};
>> +
>> +/**
>> + * struct drm_i915_vm_bind_fence - An input or output fence for the vm_bind
>> + * or the vm_unbind work.
>> + *
>> + * The vm_bind or vm_unbind aync worker will wait for input fence to signal
>> + * before starting the binding or unbinding.
>> + *
>> + * The vm_bind or vm_unbind async worker will signal the returned output fence
>> + * after the completion of binding or unbinding.
>> + */
>> +struct drm_i915_vm_bind_fence {
>> +	/** @handle: User's handle for a drm_syncobj to wait on or signal. */
>> +	__u32 handle;
>> +
>> +	/**
>> +	 * @flags: Supported flags are:
>> +	 *
>> +	 * I915_VM_BIND_FENCE_WAIT:
>> +	 * Wait for the input fence before binding/unbinding
>> +	 *
>> +	 * I915_VM_BIND_FENCE_SIGNAL:
>> +	 * Return bind/unbind completion fence as output
>> +	 */
>> +	__u32 flags;
>> +#define I915_VM_BIND_FENCE_WAIT            (1<<0)
>> +#define I915_VM_BIND_FENCE_SIGNAL          (1<<1)
>> +#define __I915_VM_BIND_FENCE_UNKNOWN_FLAGS (-(I915_VM_BIND_FENCE_SIGNAL << 1))
>> +};
>> +
>> +/**
>> + * struct drm_i915_vm_bind_ext_timeline_fences - Timeline fences for vm_bind
>> + * and vm_unbind.
>> + *
>> + * This structure describes an array of timeline drm_syncobj and associated
>> + * points for timeline variants of drm_syncobj. These timeline 'drm_syncobj's
>> + * can be input or output fences (See struct drm_i915_vm_bind_fence).
>> + */
>> +struct drm_i915_vm_bind_ext_timeline_fences {
>> +	/** @base: Extension link. See struct i915_user_extension. */
>> +	struct i915_user_extension base;
>> +
>> +	/**
>> +	 * @fence_count: Number of elements in the @handles_ptr & @value_ptr
>> +	 * arrays.
>> +	 */
>> +	__u64 fence_count;
>> +
>> +	/**
>> +	 * @handles_ptr: Pointer to an array of struct drm_i915_vm_bind_fence
>> +	 * of length @fence_count.
>> +	 */
>> +	__u64 handles_ptr;
>> +
>> +	/**
>> +	 * @values_ptr: Pointer to an array of u64 values of length
>> +	 * @fence_count.
>> +	 * Values must be 0 for a binary drm_syncobj. A Value of 0 for a
>> +	 * timeline drm_syncobj is invalid as it turns a drm_syncobj into a
>> +	 * binary one.
>> +	 */
>> +	__u64 values_ptr;
>> +};
>> +
>> +/**
>> + * struct drm_i915_vm_bind_user_fence - An input or output user fence for the
>> + * vm_bind or the vm_unbind work.
>> + *
>> + * The vm_bind or vm_unbind aync worker will wait for the input fence (value at
>> + * @addr to become equal to @val) before starting the binding or unbinding.
>> + *
>> + * The vm_bind or vm_unbind async worker will signal the output fence after
>> + * the completion of binding or unbinding by writing @val to memory location at
>> + * @addr
>> + */
>> +struct drm_i915_vm_bind_user_fence {
>> +	/** @addr: User/Memory fence qword aligned process virtual address */
>> +	__u64 addr;
>> +
>> +	/** @val: User/Memory fence value to be written after bind completion */
>> +	__u64 val;
>> +
>> +	/**
>> +	 * @flags: Supported flags are:
>> +	 *
>> +	 * I915_VM_BIND_USER_FENCE_WAIT:
>> +	 * Wait for the input fence before binding/unbinding
>> +	 *
>> +	 * I915_VM_BIND_USER_FENCE_SIGNAL:
>> +	 * Return bind/unbind completion fence as output
>> +	 */
>> +	__u32 flags;
>> +#define I915_VM_BIND_USER_FENCE_WAIT            (1<<0)
>> +#define I915_VM_BIND_USER_FENCE_SIGNAL          (1<<1)
>> +#define __I915_VM_BIND_USER_FENCE_UNKNOWN_FLAGS \
>> +	(-(I915_VM_BIND_USER_FENCE_SIGNAL << 1))
>> +};
>> +
>> +/**
>> + * struct drm_i915_vm_bind_ext_user_fence - User/memory fences for vm_bind
>> + * and vm_unbind.
>> + *
>> + * These user fences can be input or output fences
>> + * (See struct drm_i915_vm_bind_user_fence).
>> + */
>> +struct drm_i915_vm_bind_ext_user_fence {
>> +	/** @base: Extension link. See struct i915_user_extension. */
>> +	struct i915_user_extension base;
>> +
>> +	/** @fence_count: Number of elements in the @user_fence_ptr array. */
>> +	__u64 fence_count;
>> +
>> +	/**
>> +	 * @user_fence_ptr: Pointer to an array of
>> +	 * struct drm_i915_vm_bind_user_fence of length @fence_count.
>> +	 */
>> +	__u64 user_fence_ptr;
>> +};
>> +
>> +/**
>> + * struct drm_i915_gem_execbuffer3 - Structure for DRM_I915_GEM_EXECBUFFER3
>> + * ioctl.
>> + *
>> + * DRM_I915_GEM_EXECBUFFER3 ioctl only works in VM_BIND mode and VM_BIND mode
>> + * only works with this ioctl for submission.
>> + * See I915_VM_CREATE_FLAGS_USE_VM_BIND.
>> + */
>> +struct drm_i915_gem_execbuffer3 {
>> +	/**
>> +	 * @ctx_id: Context id
>> +	 *
>> +	 * Only contexts with user engine map are allowed.
>> +	 */
>> +	__u32 ctx_id;
>> +
>> +	/**
>> +	 * @engine_idx: Engine index
>> +	 *
>> +	 * An index in the user engine map of the context specified by @ctx_id.
>> +	 */
>> +	__u32 engine_idx;
>> +
>> +	/** @rsvd1: Reserved, MBZ */
>> +	__u32 rsvd1;
>> +
>> +	/**
>> +	 * @batch_count: Number of batches in @batch_address array.
>> +	 *
>> +	 * 0 is invalid. For parallel submission, it should be equal to the
>> +	 * number of (parallel) engines involved in that submission.
>> +	 */
>> +	__u32 batch_count;
>> +
>> +	/**
>> +	 * @batch_address: Array of batch gpu virtual addresses.
>> +	 *
>> +	 * If @batch_count is 1, then it is the gpu virtual address of the
>> +	 * batch buffer. If @batch_count > 1, then it is a pointer to an array
>> +	 * of batch buffer gpu virtual addresses.
>> +	 */
>> +	__u64 batch_address;
>> +
>> +	/**
>> +	 * @flags: Supported flags are:
>> +	 *
>> +	 * I915_EXEC3_SECURE:
>> +	 * Request a privileged ("secure") batch buffer/s.
>> +	 * It is only available for DRM_ROOT_ONLY | DRM_MASTER processes.
>> +	 */
>> +	__u64 flags;
>> +#define I915_EXEC3_SECURE	(1<<0)
>> +
>> +	/** @rsvd2: Reserved, MBZ */
>> +	__u64 rsvd2;
>> +
>> +	/**
>> +	 * @extensions: Zero-terminated chain of extensions.
>> +	 *
>> +	 * DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES:
>> +	 * It has same format as DRM_I915_GEM_EXECBUFFER_EXT_TIMELINE_FENCES.
>> +	 * See struct drm_i915_gem_execbuffer_ext_timeline_fences.
>> +	 *
>> +	 * DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE:
>> +	 * First level batch completion signaling extension.
>> +	 * See struct drm_i915_gem_execbuffer3_ext_user_fence.
>> +	 */
>> +	__u64 extensions;
>> +#define DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES	0
>> +#define DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE		1
>> +};
>> +
>> +/**
>> + * struct drm_i915_gem_execbuffer3_ext_user_fence - First level batch completion
>> + * signaling extension.
>> + *
>> + * This extension allows user to attach a user fence (@addr, @value pair) to
>> + * execbuf3, to be signaled by the command streamer after the completion of first
>> + * level batch, by writing the @value at specified @addr and triggering an
>> + * interrupt.
>> + * User can either poll for this user fence to signal or can also wait on it
>> + * with i915_gem_wait_user_fence ioctl.
>> + * This is very much usefaul for long running contexts where waiting on dma-fence
>> + * by user (like i915_gem_wait ioctl) is not supported.
>> + */
>> +struct drm_i915_gem_execbuffer3_ext_user_fence {
>> +	/** @base: Extension link. See struct i915_user_extension. */
>> +	struct i915_user_extension base;
>> +
>> +	/**
>> +	 * @addr: User/Memory fence qword aligned GPU virtual address.
>> +	 *
>> +	 * Address has to be a valid GPU virtual address at the time of
>> +	 * first level batch completion.
>> +	 */
>> +	__u64 addr;
>> +
>> +	/**
>> +	 * @value: User/Memory fence Value to be written to above address
>> +	 * after first level batch completes.
>> +	 */
>> +	__u64 value;
>> +
>> +	/** @rsvd: Reserved, MBZ */
>> +	__u64 rsvd;
>> +};
>> +
> IMO all of these fence structs should be a generic sync interface shared
> between both vm bind and exec3 rather than unique extenisons.
>
> Both vm bind and exec3 should have something like this:
>
> __64 syncs;	/* userptr to an array of generic syncs */
> __64 n_syncs;
>
> Having an array of syncs lets the kernel do one user copy for all the
> syncs rather than reading them in a a chain.
>
> A generic sync object encapsulates all possible syncs (in / out -
> syncobj, syncobj timeline, ufence, future sync concepts).
>
> e.g.
>
> struct {
> 	__u32 user_ext;
> 	__u32 flag;	/* in / out, type, whatever else info we need */
> 	union {
> 		__u32 handle; 	/* to syncobj */
> 		__u64 addr; 	/* ufence address */
> 	};
> 	__64 seqno;	/* syncobj timeline, ufence write value */
> 	...reserve enough bits for future...
> }
>
> This unifies binds and execs by using the same sync interface
> instilling the concept that binds and execs are the same op (queue'd
> operation /w in/out fences).
>
> Matt
>
>> +/**
>> + * struct drm_i915_gem_create_ext_vm_private - Extension to make the object
>> + * private to the specified VM.
>> + *
>> + * See struct drm_i915_gem_create_ext.
>> + */
>> +struct drm_i915_gem_create_ext_vm_private {
>> +#define I915_GEM_CREATE_EXT_VM_PRIVATE		2
>> +	/** @base: Extension link. See struct i915_user_extension. */
>> +	struct i915_user_extension base;
>> +
>> +	/** @vm_id: Id of the VM to which the object is private */
>> +	__u32 vm_id;
>> +};
>> +
>> +/**
>> + * struct drm_i915_gem_wait_user_fence - Wait on user/memory fence.
>> + *
>> + * User/Memory fence can be woken up either by:
>> + *
>> + * 1. GPU context indicated by @ctx_id, or,
>> + * 2. Kerrnel driver async worker upon I915_UFENCE_WAIT_SOFT.
>> + *    @ctx_id is ignored when this flag is set.
>> + *
>> + * Wakeup condition is,
>> + * ``((*addr & mask) op (value & mask))``
>> + *
>> + * See :ref:`Documentation/driver-api/dma-buf.rst <indefinite_dma_fences>`
>> + */
>> +struct drm_i915_gem_wait_user_fence {
>> +	/** @extensions: Zero-terminated chain of extensions. */
>> +	__u64 extensions;
>> +
>> +	/** @addr: User/Memory fence address */
>> +	__u64 addr;
>> +
>> +	/** @ctx_id: Id of the Context which will signal the fence. */
>> +	__u32 ctx_id;
>> +
>> +	/** @op: Wakeup condition operator */
>> +	__u16 op;
>> +#define I915_UFENCE_WAIT_EQ      0
>> +#define I915_UFENCE_WAIT_NEQ     1
>> +#define I915_UFENCE_WAIT_GT      2
>> +#define I915_UFENCE_WAIT_GTE     3
>> +#define I915_UFENCE_WAIT_LT      4
>> +#define I915_UFENCE_WAIT_LTE     5
>> +#define I915_UFENCE_WAIT_BEFORE  6
>> +#define I915_UFENCE_WAIT_AFTER   7
>> +
>> +	/**
>> +	 * @flags: Supported flags are:
>> +	 *
>> +	 * I915_UFENCE_WAIT_SOFT:
>> +	 *
>> +	 * To be woken up by i915 driver async worker (not by GPU).
>> +	 *
>> +	 * I915_UFENCE_WAIT_ABSTIME:
>> +	 *
>> +	 * Wait timeout specified as absolute time.
>> +	 */
>> +	__u16 flags;
>> +#define I915_UFENCE_WAIT_SOFT    0x1
>> +#define I915_UFENCE_WAIT_ABSTIME 0x2
>> +
>> +	/** @value: Wakeup value */
>> +	__u64 value;
>> +
>> +	/** @mask: Wakeup mask */
>> +	__u64 mask;
>> +#define I915_UFENCE_WAIT_U8     0xffu
>> +#define I915_UFENCE_WAIT_U16    0xffffu
>> +#define I915_UFENCE_WAIT_U32    0xfffffffful
>> +#define I915_UFENCE_WAIT_U64    0xffffffffffffffffull
>> +
>> +	/**
>> +	 * @timeout: Wait timeout in nanoseconds.
>> +	 *
>> +	 * If I915_UFENCE_WAIT_ABSTIME flag is set, then time timeout is the
>> +	 * absolute time in nsec.
>> +	 */
>> +	__s64 timeout;
>> +};
>> -- 
>> 2.21.0.rc0.32.g243a4c7e27
>>

Re: [Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Niranjana Vishwanathapura]

On Tue, Jun 14, 2022 at 09:27:05AM +0300, Lionel Landwerlin wrote:
>On 10/06/2022 11:53, Matthew Brost wrote:
>>On Fri, Jun 10, 2022 at 12:07:11AM -0700, Niranjana Vishwanathapura wrote:
>>>VM_BIND and related uapi definitions
>>>
>>>Signed-off-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>
>>>---
>>>  Documentation/gpu/rfc/i915_vm_bind.h | 490 +++++++++++++++++++++++++++
>>>  1 file changed, 490 insertions(+)
>>>  create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
>>>
>>>diff --git a/Documentation/gpu/rfc/i915_vm_bind.h b/Documentation/gpu/rfc/i915_vm_bind.h
>>>new file mode 100644
>>>index 000000000000..9fc854969cfb
>>>--- /dev/null
>>>+++ b/Documentation/gpu/rfc/i915_vm_bind.h
>>>@@ -0,0 +1,490 @@
>>>+/* SPDX-License-Identifier: MIT */
>>>+/*
>>>+ * Copyright © 2022 Intel Corporation
>>>+ */
>>>+
>>>+/**
>>>+ * DOC: I915_PARAM_HAS_VM_BIND
>>>+ *
>>>+ * VM_BIND feature availability.
>>>+ * See typedef drm_i915_getparam_t param.
>>>+ * bit[0]: If set, VM_BIND is supported, otherwise not.
>>>+ * bits[8-15]: VM_BIND implementation version.
>>>+ * version 0 will not have VM_BIND/UNBIND timeline fence array support.
>>>+ */
>>>+#define I915_PARAM_HAS_VM_BIND		57
>>>+
>>>+/**
>>>+ * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
>>>+ *
>>>+ * Flag to opt-in for VM_BIND mode of binding during VM creation.
>>>+ * See struct drm_i915_gem_vm_control flags.
>>>+ *
>>>+ * The older execbuf2 ioctl will not support VM_BIND mode of operation.
>>>+ * For VM_BIND mode, we have new execbuf3 ioctl which will not accept any
>>>+ * execlist (See struct drm_i915_gem_execbuffer3 for more details).
>>>+ *
>>>+ */
>>>+#define I915_VM_CREATE_FLAGS_USE_VM_BIND	(1 << 0)
>>>+
>>>+/**
>>>+ * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
>>>+ *
>>>+ * Flag to declare context as long running.
>>>+ * See struct drm_i915_gem_context_create_ext flags.
>>>+ *
>>>+ * Usage of dma-fence expects that they complete in reasonable amount of time.
>>>+ * Compute on the other hand can be long running. Hence it is not appropriate
>>>+ * for compute contexts to export request completion dma-fence to user.
>>>+ * The dma-fence usage will be limited to in-kernel consumption only.
>>>+ * Compute contexts need to use user/memory fence.
>>>+ *
>>>+ * So, long running contexts do not support output fences. Hence,
>>>+ * I915_EXEC_FENCE_SIGNAL (See &drm_i915_gem_exec_fence.flags) is expected
>>>+ * to be not used. DRM_I915_GEM_WAIT ioctl call is also not supported for
>>>+ * objects mapped to long running contexts.
>>>+ */
>>>+#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u << 2)
>>>+
>>>+/* VM_BIND related ioctls */
>>>+#define DRM_I915_GEM_VM_BIND		0x3d
>>>+#define DRM_I915_GEM_VM_UNBIND		0x3e
>>>+#define DRM_I915_GEM_EXECBUFFER3	0x3f
>>>+#define DRM_I915_GEM_WAIT_USER_FENCE	0x40
>>>+
>>>+#define DRM_IOCTL_I915_GEM_VM_BIND		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_VM_BIND, struct drm_i915_gem_vm_bind)
>>>+#define DRM_IOCTL_I915_GEM_VM_UNBIND		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_VM_UNBIND, struct drm_i915_gem_vm_bind)
>>>+#define DRM_IOCTL_I915_GEM_EXECBUFFER3		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_EXECBUFFER3, struct drm_i915_gem_execbuffer3)
>>>+#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE	DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_WAIT_USER_FENCE, struct drm_i915_gem_wait_user_fence)
>>>+
>>>+/**
>>>+ * struct drm_i915_gem_vm_bind - VA to object mapping to bind.
>>>+ *
>>>+ * This structure is passed to VM_BIND ioctl and specifies the mapping of GPU
>>>+ * virtual address (VA) range to the section of an object that should be bound
>>>+ * in the device page table of the specified address space (VM).
>>>+ * The VA range specified must be unique (ie., not currently bound) and can
>>>+ * be mapped to whole object or a section of the object (partial binding).
>>>+ * Multiple VA mappings can be created to the same section of the object
>>>+ * (aliasing).
>>>+ *
>>>+ * The @queue_idx specifies the queue to use for binding. Same queue can be
>>>+ * used for both VM_BIND and VM_UNBIND calls. All submitted bind and unbind
>>>+ * operations in a queue are performed in the order of submission.
>>>+ *
>>>+ * The @start, @offset and @length should be 4K page aligned. However the DG2
>>>+ * and XEHPSDV has 64K page size for device local-memory and has compact page
>>>+ * table. On those platforms, for binding device local-memory objects, the
>>>+ * @start should be 2M aligned, @offset and @length should be 64K aligned.
>>>+ * Also, on those platforms, it is not allowed to bind an device local-memory
>>>+ * object and a system memory object in a single 2M section of VA range.
>>>+ */
>>>+struct drm_i915_gem_vm_bind {
>>>+	/** @vm_id: VM (address space) id to bind */
>>>+	__u32 vm_id;
>>>+
>>>+	/** @queue_idx: Index of queue for binding */
>>>+	__u32 queue_idx;
>>>+
>>>+	/** @rsvd: Reserved, MBZ */
>>>+	__u32 rsvd;
>>>+
>>>+	/** @handle: Object handle */
>>>+	__u32 handle;
>>>+
>>>+	/** @start: Virtual Address start to bind */
>>>+	__u64 start;
>>>+
>>>+	/** @offset: Offset in object to bind */
>>>+	__u64 offset;
>>>+
>>>+	/** @length: Length of mapping to bind */
>>>+	__u64 length;
>>This probably isn't needed. We are never going to unbind a subset of a
>>VMA are we? That being said it can't hurt as a sanity check (e.g.
>>internal vma->length == user unbind length).
>
>
>Not sure what you mean by that.
>
>
>Vulkan can unbind a whole range of addresses and it seems like there 
>is no restriction on doing something like this :
>
>
>bind vma=0x1000000 GEMBO=3 offset=0 range=8192
>
>unbind vma=0x1001000 range=4096
>

VM_BIND interface proposed here do not support it, specifically
the vma merge and split operations.
In the VM_UNBIND documentation I have mentioned that the specified
mapping should uniqely identify the mapping bound through VM_BIND call.
UMDs should handle it.

>
>You would be left with a single 4k page of GEMBO=3 bound at vma=0x1000000
>
>
>Or :
>
>bind vma=0x1000000 GEMBO=3 offset=0 range=4096
>
>bind vma=0x1001000 GEMBO=4 offset=0 range=4096
>
>unbind vma=0x1000000 range=8192
>
>
>You're unbinding 2 bindings with a single operation.
>

This was also considered, but this also is part of vma merge and split
story, hence is not supported currently.

May be in future KMD can support it. But for now UMDs should handle it.

Niranjana

>
>-Lionel
>
>
>>
>>>+
>>>+	/**
>>>+	 * @flags: Supported flags are:
>>>+	 *
>>>+	 * I915_GEM_VM_BIND_READONLY:
>>>+	 * Mapping is read-only.
>>>+	 *
>>>+	 * I915_GEM_VM_BIND_CAPTURE:
>>>+	 * Capture this mapping in the dump upon GPU error.
>>>+	 */
>>>+	__u64 flags;
>>>+#define I915_GEM_VM_BIND_READONLY    (1 << 0)
>>>+#define I915_GEM_VM_BIND_CAPTURE     (1 << 1)
>>>+
>>>+	/**
>>>+	 * @extensions: 0-terminated chain of extensions for this operation.
>>>+	 *
>>>+	 * I915_VM_BIND_EXT_TIMELINE_FENCES:
>>>+	 * Specifies an array of input or output timeline fences for this
>>>+	 * binding operation. See struct drm_i915_vm_bind_ext_timeline_fences.
>>>+	 *
>>>+	 * I915_VM_BIND_EXT_USER_FENCES:
>>>+	 * Specifies an array of input or output user fences for this
>>>+	 * binding operation. See struct drm_i915_vm_bind_ext_user_fence.
>>>+	 * This is required for compute contexts.
>>>+	 */
>>>+	__u64 extensions;
>>>+#define I915_VM_BIND_EXT_TIMELINE_FENCES	0
>>>+#define I915_VM_BIND_EXT_USER_FENCES		1
>>>+};
>>>+
>>>+/**
>>>+ * struct drm_i915_gem_vm_unbind - VA to object mapping to unbind.
>>>+ *
>>>+ * This structure is passed to VM_UNBIND ioctl and specifies the GPU virtual
>>>+ * address (VA) range that should be unbound from the device page table of the
>>>+ * specified address space (VM). The specified VA range must match one of the
>>>+ * mappings created with the VM_BIND ioctl. TLB is flushed upon unbind
>>>+ * completion.
>>>+ *
>>>+ * The @queue_idx specifies the queue to use for unbinding.
>>>+ * See struct drm_i915_gem_vm_unbind for more information on @queue_idx.
>>>+ *
>>>+ * The @start and @length musy specify a unique mapping bound with VM_BIND
>>>+ * ioctl.
>>>+ */
>>>+struct drm_i915_gem_vm_unbind {
>>>+	/** @vm_id: VM (address space) id to bind */
>>>+	__u32 vm_id;
>>>+
>>>+	/** @queue_idx: Index of queue for unbinding */
>>>+	__u32 queue_idx;
>>>+
>>>+	/** @start: Virtual Address start to unbind */
>>>+	__u64 start;
>>>+
>>>+	/** @length: Length of mapping to unbind */
>>>+	__u64 length;
>>>+
>>>+	/** @flags: Reserved for future usage, currently MBZ */
>>>+	__u64 flags;
>>>+
>>>+	/**
>>>+	 * @extensions: 0-terminated chain of extensions for this operation.
>>>+	 *
>>>+	 * I915_VM_UNBIND_EXT_TIMELINE_FENCES:
>>>+	 * Specifies an array of input or output timeline fences for this
>>>+	 * unbind operation.
>>>+	 * It has same format as struct drm_i915_vm_bind_ext_timeline_fences.
>>>+	 *
>>>+	 * I915_VM_UNBIND_EXT_USER_FENCES:
>>>+	 * Specifies an array of input or output user fences for this
>>>+	 * unbind operation. This is required for compute contexts.
>>>+	 * It has same format as struct drm_i915_vm_bind_ext_user_fence.
>>>+	 */
>>>+	__u64 extensions;
>>>+#define I915_VM_UNBIND_EXT_TIMELINE_FENCES	0
>>>+#define I915_VM_UNBIND_EXT_USER_FENCES		1
>>>+};
>>>+
>>>+/**
>>>+ * struct drm_i915_vm_bind_fence - An input or output fence for the vm_bind
>>>+ * or the vm_unbind work.
>>>+ *
>>>+ * The vm_bind or vm_unbind aync worker will wait for input fence to signal
>>>+ * before starting the binding or unbinding.
>>>+ *
>>>+ * The vm_bind or vm_unbind async worker will signal the returned output fence
>>>+ * after the completion of binding or unbinding.
>>>+ */
>>>+struct drm_i915_vm_bind_fence {
>>>+	/** @handle: User's handle for a drm_syncobj to wait on or signal. */
>>>+	__u32 handle;
>>>+
>>>+	/**
>>>+	 * @flags: Supported flags are:
>>>+	 *
>>>+	 * I915_VM_BIND_FENCE_WAIT:
>>>+	 * Wait for the input fence before binding/unbinding
>>>+	 *
>>>+	 * I915_VM_BIND_FENCE_SIGNAL:
>>>+	 * Return bind/unbind completion fence as output
>>>+	 */
>>>+	__u32 flags;
>>>+#define I915_VM_BIND_FENCE_WAIT            (1<<0)
>>>+#define I915_VM_BIND_FENCE_SIGNAL          (1<<1)
>>>+#define __I915_VM_BIND_FENCE_UNKNOWN_FLAGS (-(I915_VM_BIND_FENCE_SIGNAL << 1))
>>>+};
>>>+
>>>+/**
>>>+ * struct drm_i915_vm_bind_ext_timeline_fences - Timeline fences for vm_bind
>>>+ * and vm_unbind.
>>>+ *
>>>+ * This structure describes an array of timeline drm_syncobj and associated
>>>+ * points for timeline variants of drm_syncobj. These timeline 'drm_syncobj's
>>>+ * can be input or output fences (See struct drm_i915_vm_bind_fence).
>>>+ */
>>>+struct drm_i915_vm_bind_ext_timeline_fences {
>>>+	/** @base: Extension link. See struct i915_user_extension. */
>>>+	struct i915_user_extension base;
>>>+
>>>+	/**
>>>+	 * @fence_count: Number of elements in the @handles_ptr & @value_ptr
>>>+	 * arrays.
>>>+	 */
>>>+	__u64 fence_count;
>>>+
>>>+	/**
>>>+	 * @handles_ptr: Pointer to an array of struct drm_i915_vm_bind_fence
>>>+	 * of length @fence_count.
>>>+	 */
>>>+	__u64 handles_ptr;
>>>+
>>>+	/**
>>>+	 * @values_ptr: Pointer to an array of u64 values of length
>>>+	 * @fence_count.
>>>+	 * Values must be 0 for a binary drm_syncobj. A Value of 0 for a
>>>+	 * timeline drm_syncobj is invalid as it turns a drm_syncobj into a
>>>+	 * binary one.
>>>+	 */
>>>+	__u64 values_ptr;
>>>+};
>>>+
>>>+/**
>>>+ * struct drm_i915_vm_bind_user_fence - An input or output user fence for the
>>>+ * vm_bind or the vm_unbind work.
>>>+ *
>>>+ * The vm_bind or vm_unbind aync worker will wait for the input fence (value at
>>>+ * @addr to become equal to @val) before starting the binding or unbinding.
>>>+ *
>>>+ * The vm_bind or vm_unbind async worker will signal the output fence after
>>>+ * the completion of binding or unbinding by writing @val to memory location at
>>>+ * @addr
>>>+ */
>>>+struct drm_i915_vm_bind_user_fence {
>>>+	/** @addr: User/Memory fence qword aligned process virtual address */
>>>+	__u64 addr;
>>>+
>>>+	/** @val: User/Memory fence value to be written after bind completion */
>>>+	__u64 val;
>>>+
>>>+	/**
>>>+	 * @flags: Supported flags are:
>>>+	 *
>>>+	 * I915_VM_BIND_USER_FENCE_WAIT:
>>>+	 * Wait for the input fence before binding/unbinding
>>>+	 *
>>>+	 * I915_VM_BIND_USER_FENCE_SIGNAL:
>>>+	 * Return bind/unbind completion fence as output
>>>+	 */
>>>+	__u32 flags;
>>>+#define I915_VM_BIND_USER_FENCE_WAIT            (1<<0)
>>>+#define I915_VM_BIND_USER_FENCE_SIGNAL          (1<<1)
>>>+#define __I915_VM_BIND_USER_FENCE_UNKNOWN_FLAGS \
>>>+	(-(I915_VM_BIND_USER_FENCE_SIGNAL << 1))
>>>+};
>>>+
>>>+/**
>>>+ * struct drm_i915_vm_bind_ext_user_fence - User/memory fences for vm_bind
>>>+ * and vm_unbind.
>>>+ *
>>>+ * These user fences can be input or output fences
>>>+ * (See struct drm_i915_vm_bind_user_fence).
>>>+ */
>>>+struct drm_i915_vm_bind_ext_user_fence {
>>>+	/** @base: Extension link. See struct i915_user_extension. */
>>>+	struct i915_user_extension base;
>>>+
>>>+	/** @fence_count: Number of elements in the @user_fence_ptr array. */
>>>+	__u64 fence_count;
>>>+
>>>+	/**
>>>+	 * @user_fence_ptr: Pointer to an array of
>>>+	 * struct drm_i915_vm_bind_user_fence of length @fence_count.
>>>+	 */
>>>+	__u64 user_fence_ptr;
>>>+};
>>>+
>>>+/**
>>>+ * struct drm_i915_gem_execbuffer3 - Structure for DRM_I915_GEM_EXECBUFFER3
>>>+ * ioctl.
>>>+ *
>>>+ * DRM_I915_GEM_EXECBUFFER3 ioctl only works in VM_BIND mode and VM_BIND mode
>>>+ * only works with this ioctl for submission.
>>>+ * See I915_VM_CREATE_FLAGS_USE_VM_BIND.
>>>+ */
>>>+struct drm_i915_gem_execbuffer3 {
>>>+	/**
>>>+	 * @ctx_id: Context id
>>>+	 *
>>>+	 * Only contexts with user engine map are allowed.
>>>+	 */
>>>+	__u32 ctx_id;
>>>+
>>>+	/**
>>>+	 * @engine_idx: Engine index
>>>+	 *
>>>+	 * An index in the user engine map of the context specified by @ctx_id.
>>>+	 */
>>>+	__u32 engine_idx;
>>>+
>>>+	/** @rsvd1: Reserved, MBZ */
>>>+	__u32 rsvd1;
>>>+
>>>+	/**
>>>+	 * @batch_count: Number of batches in @batch_address array.
>>>+	 *
>>>+	 * 0 is invalid. For parallel submission, it should be equal to the
>>>+	 * number of (parallel) engines involved in that submission.
>>>+	 */
>>>+	__u32 batch_count;
>>>+
>>>+	/**
>>>+	 * @batch_address: Array of batch gpu virtual addresses.
>>>+	 *
>>>+	 * If @batch_count is 1, then it is the gpu virtual address of the
>>>+	 * batch buffer. If @batch_count > 1, then it is a pointer to an array
>>>+	 * of batch buffer gpu virtual addresses.
>>>+	 */
>>>+	__u64 batch_address;
>>>+
>>>+	/**
>>>+	 * @flags: Supported flags are:
>>>+	 *
>>>+	 * I915_EXEC3_SECURE:
>>>+	 * Request a privileged ("secure") batch buffer/s.
>>>+	 * It is only available for DRM_ROOT_ONLY | DRM_MASTER processes.
>>>+	 */
>>>+	__u64 flags;
>>>+#define I915_EXEC3_SECURE	(1<<0)
>>>+
>>>+	/** @rsvd2: Reserved, MBZ */
>>>+	__u64 rsvd2;
>>>+
>>>+	/**
>>>+	 * @extensions: Zero-terminated chain of extensions.
>>>+	 *
>>>+	 * DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES:
>>>+	 * It has same format as DRM_I915_GEM_EXECBUFFER_EXT_TIMELINE_FENCES.
>>>+	 * See struct drm_i915_gem_execbuffer_ext_timeline_fences.
>>>+	 *
>>>+	 * DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE:
>>>+	 * First level batch completion signaling extension.
>>>+	 * See struct drm_i915_gem_execbuffer3_ext_user_fence.
>>>+	 */
>>>+	__u64 extensions;
>>>+#define DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES	0
>>>+#define DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE		1
>>>+};
>>>+
>>>+/**
>>>+ * struct drm_i915_gem_execbuffer3_ext_user_fence - First level batch completion
>>>+ * signaling extension.
>>>+ *
>>>+ * This extension allows user to attach a user fence (@addr, @value pair) to
>>>+ * execbuf3, to be signaled by the command streamer after the completion of first
>>>+ * level batch, by writing the @value at specified @addr and triggering an
>>>+ * interrupt.
>>>+ * User can either poll for this user fence to signal or can also wait on it
>>>+ * with i915_gem_wait_user_fence ioctl.
>>>+ * This is very much usefaul for long running contexts where waiting on dma-fence
>>>+ * by user (like i915_gem_wait ioctl) is not supported.
>>>+ */
>>>+struct drm_i915_gem_execbuffer3_ext_user_fence {
>>>+	/** @base: Extension link. See struct i915_user_extension. */
>>>+	struct i915_user_extension base;
>>>+
>>>+	/**
>>>+	 * @addr: User/Memory fence qword aligned GPU virtual address.
>>>+	 *
>>>+	 * Address has to be a valid GPU virtual address at the time of
>>>+	 * first level batch completion.
>>>+	 */
>>>+	__u64 addr;
>>>+
>>>+	/**
>>>+	 * @value: User/Memory fence Value to be written to above address
>>>+	 * after first level batch completes.
>>>+	 */
>>>+	__u64 value;
>>>+
>>>+	/** @rsvd: Reserved, MBZ */
>>>+	__u64 rsvd;
>>>+};
>>>+
>>IMO all of these fence structs should be a generic sync interface shared
>>between both vm bind and exec3 rather than unique extenisons.
>>
>>Both vm bind and exec3 should have something like this:
>>
>>__64 syncs;	/* userptr to an array of generic syncs */
>>__64 n_syncs;
>>
>>Having an array of syncs lets the kernel do one user copy for all the
>>syncs rather than reading them in a a chain.
>>
>>A generic sync object encapsulates all possible syncs (in / out -
>>syncobj, syncobj timeline, ufence, future sync concepts).
>>
>>e.g.
>>
>>struct {
>>	__u32 user_ext;
>>	__u32 flag;	/* in / out, type, whatever else info we need */
>>	union {
>>		__u32 handle; 	/* to syncobj */
>>		__u64 addr; 	/* ufence address */
>>	};
>>	__64 seqno;	/* syncobj timeline, ufence write value */
>>	...reserve enough bits for future...
>>}
>>
>>This unifies binds and execs by using the same sync interface
>>instilling the concept that binds and execs are the same op (queue'd
>>operation /w in/out fences).
>>
>>Matt
>>
>>>+/**
>>>+ * struct drm_i915_gem_create_ext_vm_private - Extension to make the object
>>>+ * private to the specified VM.
>>>+ *
>>>+ * See struct drm_i915_gem_create_ext.
>>>+ */
>>>+struct drm_i915_gem_create_ext_vm_private {
>>>+#define I915_GEM_CREATE_EXT_VM_PRIVATE		2
>>>+	/** @base: Extension link. See struct i915_user_extension. */
>>>+	struct i915_user_extension base;
>>>+
>>>+	/** @vm_id: Id of the VM to which the object is private */
>>>+	__u32 vm_id;
>>>+};
>>>+
>>>+/**
>>>+ * struct drm_i915_gem_wait_user_fence - Wait on user/memory fence.
>>>+ *
>>>+ * User/Memory fence can be woken up either by:
>>>+ *
>>>+ * 1. GPU context indicated by @ctx_id, or,
>>>+ * 2. Kerrnel driver async worker upon I915_UFENCE_WAIT_SOFT.
>>>+ *    @ctx_id is ignored when this flag is set.
>>>+ *
>>>+ * Wakeup condition is,
>>>+ * ``((*addr & mask) op (value & mask))``
>>>+ *
>>>+ * See :ref:`Documentation/driver-api/dma-buf.rst <indefinite_dma_fences>`
>>>+ */
>>>+struct drm_i915_gem_wait_user_fence {
>>>+	/** @extensions: Zero-terminated chain of extensions. */
>>>+	__u64 extensions;
>>>+
>>>+	/** @addr: User/Memory fence address */
>>>+	__u64 addr;
>>>+
>>>+	/** @ctx_id: Id of the Context which will signal the fence. */
>>>+	__u32 ctx_id;
>>>+
>>>+	/** @op: Wakeup condition operator */
>>>+	__u16 op;
>>>+#define I915_UFENCE_WAIT_EQ      0
>>>+#define I915_UFENCE_WAIT_NEQ     1
>>>+#define I915_UFENCE_WAIT_GT      2
>>>+#define I915_UFENCE_WAIT_GTE     3
>>>+#define I915_UFENCE_WAIT_LT      4
>>>+#define I915_UFENCE_WAIT_LTE     5
>>>+#define I915_UFENCE_WAIT_BEFORE  6
>>>+#define I915_UFENCE_WAIT_AFTER   7
>>>+
>>>+	/**
>>>+	 * @flags: Supported flags are:
>>>+	 *
>>>+	 * I915_UFENCE_WAIT_SOFT:
>>>+	 *
>>>+	 * To be woken up by i915 driver async worker (not by GPU).
>>>+	 *
>>>+	 * I915_UFENCE_WAIT_ABSTIME:
>>>+	 *
>>>+	 * Wait timeout specified as absolute time.
>>>+	 */
>>>+	__u16 flags;
>>>+#define I915_UFENCE_WAIT_SOFT    0x1
>>>+#define I915_UFENCE_WAIT_ABSTIME 0x2
>>>+
>>>+	/** @value: Wakeup value */
>>>+	__u64 value;
>>>+
>>>+	/** @mask: Wakeup mask */
>>>+	__u64 mask;
>>>+#define I915_UFENCE_WAIT_U8     0xffu
>>>+#define I915_UFENCE_WAIT_U16    0xffffu
>>>+#define I915_UFENCE_WAIT_U32    0xfffffffful
>>>+#define I915_UFENCE_WAIT_U64    0xffffffffffffffffull
>>>+
>>>+	/**
>>>+	 * @timeout: Wait timeout in nanoseconds.
>>>+	 *
>>>+	 * If I915_UFENCE_WAIT_ABSTIME flag is set, then time timeout is the
>>>+	 * absolute time in nsec.
>>>+	 */
>>>+	__s64 timeout;
>>>+};
>>>-- 
>>>2.21.0.rc0.32.g243a4c7e27
>>>
>

Re: [Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Tvrtko Ursulin]

On 10/06/2022 08:07, Niranjana Vishwanathapura wrote:
> VM_BIND and related uapi definitions
> 
> Signed-off-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>
> ---
>   Documentation/gpu/rfc/i915_vm_bind.h | 490 +++++++++++++++++++++++++++
>   1 file changed, 490 insertions(+)
>   create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
> 
> diff --git a/Documentation/gpu/rfc/i915_vm_bind.h b/Documentation/gpu/rfc/i915_vm_bind.h
> new file mode 100644
> index 000000000000..9fc854969cfb
> --- /dev/null
> +++ b/Documentation/gpu/rfc/i915_vm_bind.h
> @@ -0,0 +1,490 @@
> +/* SPDX-License-Identifier: MIT */
> +/*
> + * Copyright © 2022 Intel Corporation
> + */
> +
> +/**
> + * DOC: I915_PARAM_HAS_VM_BIND
> + *
> + * VM_BIND feature availability.
> + * See typedef drm_i915_getparam_t param.
> + * bit[0]: If set, VM_BIND is supported, otherwise not.
> + * bits[8-15]: VM_BIND implementation version.
> + * version 0 will not have VM_BIND/UNBIND timeline fence array support.
> + */
> +#define I915_PARAM_HAS_VM_BIND		57
> +
> +/**
> + * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
> + *
> + * Flag to opt-in for VM_BIND mode of binding during VM creation.
> + * See struct drm_i915_gem_vm_control flags.
> + *
> + * The older execbuf2 ioctl will not support VM_BIND mode of operation.
> + * For VM_BIND mode, we have new execbuf3 ioctl which will not accept any
> + * execlist (See struct drm_i915_gem_execbuffer3 for more details).
> + *
> + */
> +#define I915_VM_CREATE_FLAGS_USE_VM_BIND	(1 << 0)
> +
> +/**
> + * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
> + *
> + * Flag to declare context as long running.
> + * See struct drm_i915_gem_context_create_ext flags.
> + *
> + * Usage of dma-fence expects that they complete in reasonable amount of time.
> + * Compute on the other hand can be long running. Hence it is not appropriate
> + * for compute contexts to export request completion dma-fence to user.
> + * The dma-fence usage will be limited to in-kernel consumption only.
> + * Compute contexts need to use user/memory fence.
> + *
> + * So, long running contexts do not support output fences. Hence,
> + * I915_EXEC_FENCE_SIGNAL (See &drm_i915_gem_exec_fence.flags) is expected
> + * to be not used. DRM_I915_GEM_WAIT ioctl call is also not supported for
> + * objects mapped to long running contexts.
> + */
> +#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u << 2)
> +
> +/* VM_BIND related ioctls */
> +#define DRM_I915_GEM_VM_BIND		0x3d
> +#define DRM_I915_GEM_VM_UNBIND		0x3e
> +#define DRM_I915_GEM_EXECBUFFER3	0x3f
> +#define DRM_I915_GEM_WAIT_USER_FENCE	0x40
> +
> +#define DRM_IOCTL_I915_GEM_VM_BIND		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_VM_BIND, struct drm_i915_gem_vm_bind)
> +#define DRM_IOCTL_I915_GEM_VM_UNBIND		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_VM_UNBIND, struct drm_i915_gem_vm_bind)
> +#define DRM_IOCTL_I915_GEM_EXECBUFFER3		DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_EXECBUFFER3, struct drm_i915_gem_execbuffer3)
> +#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE	DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_WAIT_USER_FENCE, struct drm_i915_gem_wait_user_fence)
> +
> +/**
> + * struct drm_i915_gem_vm_bind - VA to object mapping to bind.
> + *
> + * This structure is passed to VM_BIND ioctl and specifies the mapping of GPU
> + * virtual address (VA) range to the section of an object that should be bound
> + * in the device page table of the specified address space (VM).
> + * The VA range specified must be unique (ie., not currently bound) and can
> + * be mapped to whole object or a section of the object (partial binding).
> + * Multiple VA mappings can be created to the same section of the object
> + * (aliasing).
> + *
> + * The @queue_idx specifies the queue to use for binding. Same queue can be
> + * used for both VM_BIND and VM_UNBIND calls. All submitted bind and unbind
> + * operations in a queue are performed in the order of submission.
> + *
> + * The @start, @offset and @length should be 4K page aligned. However the DG2
> + * and XEHPSDV has 64K page size for device local-memory and has compact page
> + * table. On those platforms, for binding device local-memory objects, the
> + * @start should be 2M aligned, @offset and @length should be 64K aligned.
> + * Also, on those platforms, it is not allowed to bind an device local-memory
> + * object and a system memory object in a single 2M section of VA range.
> + */
> +struct drm_i915_gem_vm_bind {
> +	/** @vm_id: VM (address space) id to bind */
> +	__u32 vm_id;
> +
> +	/** @queue_idx: Index of queue for binding */
> +	__u32 queue_idx;

I have a question here to which I did not find an answer by browsing the 
old threads.

Queue index appears to be an implicit synchronisation mechanism, right? 
Operations on the same index are executed/complete in order of ioctl 
submission?

Do we _have_ to implement this on the kernel side and could just allow 
in/out fence and let userspace deal with it?

Arbitrary/on-demand number of queues will add the complexity on the 
kernel side which should be avoided if possible.

Regards,

Tvrtko

> +
> +	/** @rsvd: Reserved, MBZ */
> +	__u32 rsvd;
> +
> +	/** @handle: Object handle */
> +	__u32 handle;
> +
> +	/** @start: Virtual Address start to bind */
> +	__u64 start;
> +
> +	/** @offset: Offset in object to bind */
> +	__u64 offset;
> +
> +	/** @length: Length of mapping to bind */
> +	__u64 length;
> +
> +	/**
> +	 * @flags: Supported flags are:
> +	 *
> +	 * I915_GEM_VM_BIND_READONLY:
> +	 * Mapping is read-only.
> +	 *
> +	 * I915_GEM_VM_BIND_CAPTURE:
> +	 * Capture this mapping in the dump upon GPU error.
> +	 */
> +	__u64 flags;
> +#define I915_GEM_VM_BIND_READONLY    (1 << 0)
> +#define I915_GEM_VM_BIND_CAPTURE     (1 << 1)
> +
> +	/**
> +	 * @extensions: 0-terminated chain of extensions for this operation.
> +	 *
> +	 * I915_VM_BIND_EXT_TIMELINE_FENCES:
> +	 * Specifies an array of input or output timeline fences for this
> +	 * binding operation. See struct drm_i915_vm_bind_ext_timeline_fences.
> +	 *
> +	 * I915_VM_BIND_EXT_USER_FENCES:
> +	 * Specifies an array of input or output user fences for this
> +	 * binding operation. See struct drm_i915_vm_bind_ext_user_fence.
> +	 * This is required for compute contexts.
> +	 */
> +	__u64 extensions;
> +#define I915_VM_BIND_EXT_TIMELINE_FENCES	0
> +#define I915_VM_BIND_EXT_USER_FENCES		1
> +};
> +
> +/**
> + * struct drm_i915_gem_vm_unbind - VA to object mapping to unbind.
> + *
> + * This structure is passed to VM_UNBIND ioctl and specifies the GPU virtual
> + * address (VA) range that should be unbound from the device page table of the
> + * specified address space (VM). The specified VA range must match one of the
> + * mappings created with the VM_BIND ioctl. TLB is flushed upon unbind
> + * completion.
> + *
> + * The @queue_idx specifies the queue to use for unbinding.
> + * See struct drm_i915_gem_vm_unbind for more information on @queue_idx.
> + *
> + * The @start and @length musy specify a unique mapping bound with VM_BIND
> + * ioctl.
> + */
> +struct drm_i915_gem_vm_unbind {
> +	/** @vm_id: VM (address space) id to bind */
> +	__u32 vm_id;
> +
> +	/** @queue_idx: Index of queue for unbinding */
> +	__u32 queue_idx;
> +
> +	/** @start: Virtual Address start to unbind */
> +	__u64 start;
> +
> +	/** @length: Length of mapping to unbind */
> +	__u64 length;
> +
> +	/** @flags: Reserved for future usage, currently MBZ */
> +	__u64 flags;
> +
> +	/**
> +	 * @extensions: 0-terminated chain of extensions for this operation.
> +	 *
> +	 * I915_VM_UNBIND_EXT_TIMELINE_FENCES:
> +	 * Specifies an array of input or output timeline fences for this
> +	 * unbind operation.
> +	 * It has same format as struct drm_i915_vm_bind_ext_timeline_fences.
> +	 *
> +	 * I915_VM_UNBIND_EXT_USER_FENCES:
> +	 * Specifies an array of input or output user fences for this
> +	 * unbind operation. This is required for compute contexts.
> +	 * It has same format as struct drm_i915_vm_bind_ext_user_fence.
> +	 */
> +	__u64 extensions;
> +#define I915_VM_UNBIND_EXT_TIMELINE_FENCES	0
> +#define I915_VM_UNBIND_EXT_USER_FENCES		1
> +};
> +
> +/**
> + * struct drm_i915_vm_bind_fence - An input or output fence for the vm_bind
> + * or the vm_unbind work.
> + *
> + * The vm_bind or vm_unbind aync worker will wait for input fence to signal
> + * before starting the binding or unbinding.
> + *
> + * The vm_bind or vm_unbind async worker will signal the returned output fence
> + * after the completion of binding or unbinding.
> + */
> +struct drm_i915_vm_bind_fence {
> +	/** @handle: User's handle for a drm_syncobj to wait on or signal. */
> +	__u32 handle;
> +
> +	/**
> +	 * @flags: Supported flags are:
> +	 *
> +	 * I915_VM_BIND_FENCE_WAIT:
> +	 * Wait for the input fence before binding/unbinding
> +	 *
> +	 * I915_VM_BIND_FENCE_SIGNAL:
> +	 * Return bind/unbind completion fence as output
> +	 */
> +	__u32 flags;
> +#define I915_VM_BIND_FENCE_WAIT            (1<<0)
> +#define I915_VM_BIND_FENCE_SIGNAL          (1<<1)
> +#define __I915_VM_BIND_FENCE_UNKNOWN_FLAGS (-(I915_VM_BIND_FENCE_SIGNAL << 1))
> +};
> +
> +/**
> + * struct drm_i915_vm_bind_ext_timeline_fences - Timeline fences for vm_bind
> + * and vm_unbind.
> + *
> + * This structure describes an array of timeline drm_syncobj and associated
> + * points for timeline variants of drm_syncobj. These timeline 'drm_syncobj's
> + * can be input or output fences (See struct drm_i915_vm_bind_fence).
> + */
> +struct drm_i915_vm_bind_ext_timeline_fences {
> +	/** @base: Extension link. See struct i915_user_extension. */
> +	struct i915_user_extension base;
> +
> +	/**
> +	 * @fence_count: Number of elements in the @handles_ptr & @value_ptr
> +	 * arrays.
> +	 */
> +	__u64 fence_count;
> +
> +	/**
> +	 * @handles_ptr: Pointer to an array of struct drm_i915_vm_bind_fence
> +	 * of length @fence_count.
> +	 */
> +	__u64 handles_ptr;
> +
> +	/**
> +	 * @values_ptr: Pointer to an array of u64 values of length
> +	 * @fence_count.
> +	 * Values must be 0 for a binary drm_syncobj. A Value of 0 for a
> +	 * timeline drm_syncobj is invalid as it turns a drm_syncobj into a
> +	 * binary one.
> +	 */
> +	__u64 values_ptr;
> +};
> +
> +/**
> + * struct drm_i915_vm_bind_user_fence - An input or output user fence for the
> + * vm_bind or the vm_unbind work.
> + *
> + * The vm_bind or vm_unbind aync worker will wait for the input fence (value at
> + * @addr to become equal to @val) before starting the binding or unbinding.
> + *
> + * The vm_bind or vm_unbind async worker will signal the output fence after
> + * the completion of binding or unbinding by writing @val to memory location at
> + * @addr
> + */
> +struct drm_i915_vm_bind_user_fence {
> +	/** @addr: User/Memory fence qword aligned process virtual address */
> +	__u64 addr;
> +
> +	/** @val: User/Memory fence value to be written after bind completion */
> +	__u64 val;
> +
> +	/**
> +	 * @flags: Supported flags are:
> +	 *
> +	 * I915_VM_BIND_USER_FENCE_WAIT:
> +	 * Wait for the input fence before binding/unbinding
> +	 *
> +	 * I915_VM_BIND_USER_FENCE_SIGNAL:
> +	 * Return bind/unbind completion fence as output
> +	 */
> +	__u32 flags;
> +#define I915_VM_BIND_USER_FENCE_WAIT            (1<<0)
> +#define I915_VM_BIND_USER_FENCE_SIGNAL          (1<<1)
> +#define __I915_VM_BIND_USER_FENCE_UNKNOWN_FLAGS \
> +	(-(I915_VM_BIND_USER_FENCE_SIGNAL << 1))
> +};
> +
> +/**
> + * struct drm_i915_vm_bind_ext_user_fence - User/memory fences for vm_bind
> + * and vm_unbind.
> + *
> + * These user fences can be input or output fences
> + * (See struct drm_i915_vm_bind_user_fence).
> + */
> +struct drm_i915_vm_bind_ext_user_fence {
> +	/** @base: Extension link. See struct i915_user_extension. */
> +	struct i915_user_extension base;
> +
> +	/** @fence_count: Number of elements in the @user_fence_ptr array. */
> +	__u64 fence_count;
> +
> +	/**
> +	 * @user_fence_ptr: Pointer to an array of
> +	 * struct drm_i915_vm_bind_user_fence of length @fence_count.
> +	 */
> +	__u64 user_fence_ptr;
> +};
> +
> +/**
> + * struct drm_i915_gem_execbuffer3 - Structure for DRM_I915_GEM_EXECBUFFER3
> + * ioctl.
> + *
> + * DRM_I915_GEM_EXECBUFFER3 ioctl only works in VM_BIND mode and VM_BIND mode
> + * only works with this ioctl for submission.
> + * See I915_VM_CREATE_FLAGS_USE_VM_BIND.
> + */
> +struct drm_i915_gem_execbuffer3 {
> +	/**
> +	 * @ctx_id: Context id
> +	 *
> +	 * Only contexts with user engine map are allowed.
> +	 */
> +	__u32 ctx_id;
> +
> +	/**
> +	 * @engine_idx: Engine index
> +	 *
> +	 * An index in the user engine map of the context specified by @ctx_id.
> +	 */
> +	__u32 engine_idx;
> +
> +	/** @rsvd1: Reserved, MBZ */
> +	__u32 rsvd1;
> +
> +	/**
> +	 * @batch_count: Number of batches in @batch_address array.
> +	 *
> +	 * 0 is invalid. For parallel submission, it should be equal to the
> +	 * number of (parallel) engines involved in that submission.
> +	 */
> +	__u32 batch_count;
> +
> +	/**
> +	 * @batch_address: Array of batch gpu virtual addresses.
> +	 *
> +	 * If @batch_count is 1, then it is the gpu virtual address of the
> +	 * batch buffer. If @batch_count > 1, then it is a pointer to an array
> +	 * of batch buffer gpu virtual addresses.
> +	 */
> +	__u64 batch_address;
> +
> +	/**
> +	 * @flags: Supported flags are:
> +	 *
> +	 * I915_EXEC3_SECURE:
> +	 * Request a privileged ("secure") batch buffer/s.
> +	 * It is only available for DRM_ROOT_ONLY | DRM_MASTER processes.
> +	 */
> +	__u64 flags;
> +#define I915_EXEC3_SECURE	(1<<0)
> +
> +	/** @rsvd2: Reserved, MBZ */
> +	__u64 rsvd2;
> +
> +	/**
> +	 * @extensions: Zero-terminated chain of extensions.
> +	 *
> +	 * DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES:
> +	 * It has same format as DRM_I915_GEM_EXECBUFFER_EXT_TIMELINE_FENCES.
> +	 * See struct drm_i915_gem_execbuffer_ext_timeline_fences.
> +	 *
> +	 * DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE:
> +	 * First level batch completion signaling extension.
> +	 * See struct drm_i915_gem_execbuffer3_ext_user_fence.
> +	 */
> +	__u64 extensions;
> +#define DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES	0
> +#define DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE		1
> +};
> +
> +/**
> + * struct drm_i915_gem_execbuffer3_ext_user_fence - First level batch completion
> + * signaling extension.
> + *
> + * This extension allows user to attach a user fence (@addr, @value pair) to
> + * execbuf3, to be signaled by the command streamer after the completion of first
> + * level batch, by writing the @value at specified @addr and triggering an
> + * interrupt.
> + * User can either poll for this user fence to signal or can also wait on it
> + * with i915_gem_wait_user_fence ioctl.
> + * This is very much usefaul for long running contexts where waiting on dma-fence
> + * by user (like i915_gem_wait ioctl) is not supported.
> + */
> +struct drm_i915_gem_execbuffer3_ext_user_fence {
> +	/** @base: Extension link. See struct i915_user_extension. */
> +	struct i915_user_extension base;
> +
> +	/**
> +	 * @addr: User/Memory fence qword aligned GPU virtual address.
> +	 *
> +	 * Address has to be a valid GPU virtual address at the time of
> +	 * first level batch completion.
> +	 */
> +	__u64 addr;
> +
> +	/**
> +	 * @value: User/Memory fence Value to be written to above address
> +	 * after first level batch completes.
> +	 */
> +	__u64 value;
> +
> +	/** @rsvd: Reserved, MBZ */
> +	__u64 rsvd;
> +};
> +
> +/**
> + * struct drm_i915_gem_create_ext_vm_private - Extension to make the object
> + * private to the specified VM.
> + *
> + * See struct drm_i915_gem_create_ext.
> + */
> +struct drm_i915_gem_create_ext_vm_private {
> +#define I915_GEM_CREATE_EXT_VM_PRIVATE		2
> +	/** @base: Extension link. See struct i915_user_extension. */
> +	struct i915_user_extension base;
> +
> +	/** @vm_id: Id of the VM to which the object is private */
> +	__u32 vm_id;
> +};
> +
> +/**
> + * struct drm_i915_gem_wait_user_fence - Wait on user/memory fence.
> + *
> + * User/Memory fence can be woken up either by:
> + *
> + * 1. GPU context indicated by @ctx_id, or,
> + * 2. Kerrnel driver async worker upon I915_UFENCE_WAIT_SOFT.
> + *    @ctx_id is ignored when this flag is set.
> + *
> + * Wakeup condition is,
> + * ``((*addr & mask) op (value & mask))``
> + *
> + * See :ref:`Documentation/driver-api/dma-buf.rst <indefinite_dma_fences>`
> + */
> +struct drm_i915_gem_wait_user_fence {
> +	/** @extensions: Zero-terminated chain of extensions. */
> +	__u64 extensions;
> +
> +	/** @addr: User/Memory fence address */
> +	__u64 addr;
> +
> +	/** @ctx_id: Id of the Context which will signal the fence. */
> +	__u32 ctx_id;
> +
> +	/** @op: Wakeup condition operator */
> +	__u16 op;
> +#define I915_UFENCE_WAIT_EQ      0
> +#define I915_UFENCE_WAIT_NEQ     1
> +#define I915_UFENCE_WAIT_GT      2
> +#define I915_UFENCE_WAIT_GTE     3
> +#define I915_UFENCE_WAIT_LT      4
> +#define I915_UFENCE_WAIT_LTE     5
> +#define I915_UFENCE_WAIT_BEFORE  6
> +#define I915_UFENCE_WAIT_AFTER   7
> +
> +	/**
> +	 * @flags: Supported flags are:
> +	 *
> +	 * I915_UFENCE_WAIT_SOFT:
> +	 *
> +	 * To be woken up by i915 driver async worker (not by GPU).
> +	 *
> +	 * I915_UFENCE_WAIT_ABSTIME:
> +	 *
> +	 * Wait timeout specified as absolute time.
> +	 */
> +	__u16 flags;
> +#define I915_UFENCE_WAIT_SOFT    0x1
> +#define I915_UFENCE_WAIT_ABSTIME 0x2
> +
> +	/** @value: Wakeup value */
> +	__u64 value;
> +
> +	/** @mask: Wakeup mask */
> +	__u64 mask;
> +#define I915_UFENCE_WAIT_U8     0xffu
> +#define I915_UFENCE_WAIT_U16    0xffffu
> +#define I915_UFENCE_WAIT_U32    0xfffffffful
> +#define I915_UFENCE_WAIT_U64    0xffffffffffffffffull
> +
> +	/**
> +	 * @timeout: Wait timeout in nanoseconds.
> +	 *
> +	 * If I915_UFENCE_WAIT_ABSTIME flag is set, then time timeout is the
> +	 * absolute time in nsec.
> +	 */
> +	__s64 timeout;
> +};

Re: [Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Lionel Landwerlin]

On 10/06/2022 13:37, Tvrtko Ursulin wrote:
>
> On 10/06/2022 08:07, Niranjana Vishwanathapura wrote:
>> VM_BIND and related uapi definitions
>>
>> Signed-off-by: Niranjana Vishwanathapura 
>> <niranjana.vishwanathapura@intel.com>
>> ---
>>   Documentation/gpu/rfc/i915_vm_bind.h | 490 +++++++++++++++++++++++++++
>>   1 file changed, 490 insertions(+)
>>   create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
>>
>> diff --git a/Documentation/gpu/rfc/i915_vm_bind.h 
>> b/Documentation/gpu/rfc/i915_vm_bind.h
>> new file mode 100644
>> index 000000000000..9fc854969cfb
>> --- /dev/null
>> +++ b/Documentation/gpu/rfc/i915_vm_bind.h
>> @@ -0,0 +1,490 @@
>> +/* SPDX-License-Identifier: MIT */
>> +/*
>> + * Copyright © 2022 Intel Corporation
>> + */
>> +
>> +/**
>> + * DOC: I915_PARAM_HAS_VM_BIND
>> + *
>> + * VM_BIND feature availability.
>> + * See typedef drm_i915_getparam_t param.
>> + * bit[0]: If set, VM_BIND is supported, otherwise not.
>> + * bits[8-15]: VM_BIND implementation version.
>> + * version 0 will not have VM_BIND/UNBIND timeline fence array support.
>> + */
>> +#define I915_PARAM_HAS_VM_BIND        57
>> +
>> +/**
>> + * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
>> + *
>> + * Flag to opt-in for VM_BIND mode of binding during VM creation.
>> + * See struct drm_i915_gem_vm_control flags.
>> + *
>> + * The older execbuf2 ioctl will not support VM_BIND mode of operation.
>> + * For VM_BIND mode, we have new execbuf3 ioctl which will not 
>> accept any
>> + * execlist (See struct drm_i915_gem_execbuffer3 for more details).
>> + *
>> + */
>> +#define I915_VM_CREATE_FLAGS_USE_VM_BIND    (1 << 0)
>> +
>> +/**
>> + * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
>> + *
>> + * Flag to declare context as long running.
>> + * See struct drm_i915_gem_context_create_ext flags.
>> + *
>> + * Usage of dma-fence expects that they complete in reasonable 
>> amount of time.
>> + * Compute on the other hand can be long running. Hence it is not 
>> appropriate
>> + * for compute contexts to export request completion dma-fence to user.
>> + * The dma-fence usage will be limited to in-kernel consumption only.
>> + * Compute contexts need to use user/memory fence.
>> + *
>> + * So, long running contexts do not support output fences. Hence,
>> + * I915_EXEC_FENCE_SIGNAL (See &drm_i915_gem_exec_fence.flags) is 
>> expected
>> + * to be not used. DRM_I915_GEM_WAIT ioctl call is also not 
>> supported for
>> + * objects mapped to long running contexts.
>> + */
>> +#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u << 2)
>> +
>> +/* VM_BIND related ioctls */
>> +#define DRM_I915_GEM_VM_BIND        0x3d
>> +#define DRM_I915_GEM_VM_UNBIND        0x3e
>> +#define DRM_I915_GEM_EXECBUFFER3    0x3f
>> +#define DRM_I915_GEM_WAIT_USER_FENCE    0x40
>> +
>> +#define DRM_IOCTL_I915_GEM_VM_BIND DRM_IOWR(DRM_COMMAND_BASE + 
>> DRM_I915_GEM_VM_BIND, struct drm_i915_gem_vm_bind)
>> +#define DRM_IOCTL_I915_GEM_VM_UNBIND DRM_IOWR(DRM_COMMAND_BASE + 
>> DRM_I915_GEM_VM_UNBIND, struct drm_i915_gem_vm_bind)
>> +#define DRM_IOCTL_I915_GEM_EXECBUFFER3 DRM_IOWR(DRM_COMMAND_BASE + 
>> DRM_I915_GEM_EXECBUFFER3, struct drm_i915_gem_execbuffer3)
>> +#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE DRM_IOWR(DRM_COMMAND_BASE 
>> + DRM_I915_GEM_WAIT_USER_FENCE, struct drm_i915_gem_wait_user_fence)
>> +
>> +/**
>> + * struct drm_i915_gem_vm_bind - VA to object mapping to bind.
>> + *
>> + * This structure is passed to VM_BIND ioctl and specifies the 
>> mapping of GPU
>> + * virtual address (VA) range to the section of an object that 
>> should be bound
>> + * in the device page table of the specified address space (VM).
>> + * The VA range specified must be unique (ie., not currently bound) 
>> and can
>> + * be mapped to whole object or a section of the object (partial 
>> binding).
>> + * Multiple VA mappings can be created to the same section of the 
>> object
>> + * (aliasing).
>> + *
>> + * The @queue_idx specifies the queue to use for binding. Same queue 
>> can be
>> + * used for both VM_BIND and VM_UNBIND calls. All submitted bind and 
>> unbind
>> + * operations in a queue are performed in the order of submission.
>> + *
>> + * The @start, @offset and @length should be 4K page aligned. 
>> However the DG2
>> + * and XEHPSDV has 64K page size for device local-memory and has 
>> compact page
>> + * table. On those platforms, for binding device local-memory 
>> objects, the
>> + * @start should be 2M aligned, @offset and @length should be 64K 
>> aligned.
>> + * Also, on those platforms, it is not allowed to bind an device 
>> local-memory
>> + * object and a system memory object in a single 2M section of VA 
>> range.
>> + */
>> +struct drm_i915_gem_vm_bind {
>> +    /** @vm_id: VM (address space) id to bind */
>> +    __u32 vm_id;
>> +
>> +    /** @queue_idx: Index of queue for binding */
>> +    __u32 queue_idx;
>
> I have a question here to which I did not find an answer by browsing 
> the old threads.
>
> Queue index appears to be an implicit synchronisation mechanism, 
> right? Operations on the same index are executed/complete in order of 
> ioctl submission?
>
> Do we _have_ to implement this on the kernel side and could just allow 
> in/out fence and let userspace deal with it?


It orders operations like in a queue. Which is kind of what happens with 
existing queues/engines.

If I understood correctly, it's going to be a kthread + a linked list right?


-Lionel


>
> Arbitrary/on-demand number of queues will add the complexity on the 
> kernel side which should be avoided if possible.
>
> Regards,
>
> Tvrtko
>
>> +
>> +    /** @rsvd: Reserved, MBZ */
>> +    __u32 rsvd;
>> +
>> +    /** @handle: Object handle */
>> +    __u32 handle;
>> +
>> +    /** @start: Virtual Address start to bind */
>> +    __u64 start;
>> +
>> +    /** @offset: Offset in object to bind */
>> +    __u64 offset;
>> +
>> +    /** @length: Length of mapping to bind */
>> +    __u64 length;
>> +
>> +    /**
>> +     * @flags: Supported flags are:
>> +     *
>> +     * I915_GEM_VM_BIND_READONLY:
>> +     * Mapping is read-only.
>> +     *
>> +     * I915_GEM_VM_BIND_CAPTURE:
>> +     * Capture this mapping in the dump upon GPU error.
>> +     */
>> +    __u64 flags;
>> +#define I915_GEM_VM_BIND_READONLY    (1 << 0)
>> +#define I915_GEM_VM_BIND_CAPTURE     (1 << 1)
>> +
>> +    /**
>> +     * @extensions: 0-terminated chain of extensions for this 
>> operation.
>> +     *
>> +     * I915_VM_BIND_EXT_TIMELINE_FENCES:
>> +     * Specifies an array of input or output timeline fences for this
>> +     * binding operation. See struct 
>> drm_i915_vm_bind_ext_timeline_fences.
>> +     *
>> +     * I915_VM_BIND_EXT_USER_FENCES:
>> +     * Specifies an array of input or output user fences for this
>> +     * binding operation. See struct drm_i915_vm_bind_ext_user_fence.
>> +     * This is required for compute contexts.
>> +     */
>> +    __u64 extensions;
>> +#define I915_VM_BIND_EXT_TIMELINE_FENCES    0
>> +#define I915_VM_BIND_EXT_USER_FENCES        1
>> +};
>> +
>> +/**
>> + * struct drm_i915_gem_vm_unbind - VA to object mapping to unbind.
>> + *
>> + * This structure is passed to VM_UNBIND ioctl and specifies the GPU 
>> virtual
>> + * address (VA) range that should be unbound from the device page 
>> table of the
>> + * specified address space (VM). The specified VA range must match 
>> one of the
>> + * mappings created with the VM_BIND ioctl. TLB is flushed upon unbind
>> + * completion.
>> + *
>> + * The @queue_idx specifies the queue to use for unbinding.
>> + * See struct drm_i915_gem_vm_unbind for more information on 
>> @queue_idx.
>> + *
>> + * The @start and @length musy specify a unique mapping bound with 
>> VM_BIND
>> + * ioctl.
>> + */
>> +struct drm_i915_gem_vm_unbind {
>> +    /** @vm_id: VM (address space) id to bind */
>> +    __u32 vm_id;
>> +
>> +    /** @queue_idx: Index of queue for unbinding */
>> +    __u32 queue_idx;
>> +
>> +    /** @start: Virtual Address start to unbind */
>> +    __u64 start;
>> +
>> +    /** @length: Length of mapping to unbind */
>> +    __u64 length;
>> +
>> +    /** @flags: Reserved for future usage, currently MBZ */
>> +    __u64 flags;
>> +
>> +    /**
>> +     * @extensions: 0-terminated chain of extensions for this 
>> operation.
>> +     *
>> +     * I915_VM_UNBIND_EXT_TIMELINE_FENCES:
>> +     * Specifies an array of input or output timeline fences for this
>> +     * unbind operation.
>> +     * It has same format as struct 
>> drm_i915_vm_bind_ext_timeline_fences.
>> +     *
>> +     * I915_VM_UNBIND_EXT_USER_FENCES:
>> +     * Specifies an array of input or output user fences for this
>> +     * unbind operation. This is required for compute contexts.
>> +     * It has same format as struct drm_i915_vm_bind_ext_user_fence.
>> +     */
>> +    __u64 extensions;
>> +#define I915_VM_UNBIND_EXT_TIMELINE_FENCES    0
>> +#define I915_VM_UNBIND_EXT_USER_FENCES        1
>> +};
>> +
>> +/**
>> + * struct drm_i915_vm_bind_fence - An input or output fence for the 
>> vm_bind
>> + * or the vm_unbind work.
>> + *
>> + * The vm_bind or vm_unbind aync worker will wait for input fence to 
>> signal
>> + * before starting the binding or unbinding.
>> + *
>> + * The vm_bind or vm_unbind async worker will signal the returned 
>> output fence
>> + * after the completion of binding or unbinding.
>> + */
>> +struct drm_i915_vm_bind_fence {
>> +    /** @handle: User's handle for a drm_syncobj to wait on or 
>> signal. */
>> +    __u32 handle;
>> +
>> +    /**
>> +     * @flags: Supported flags are:
>> +     *
>> +     * I915_VM_BIND_FENCE_WAIT:
>> +     * Wait for the input fence before binding/unbinding
>> +     *
>> +     * I915_VM_BIND_FENCE_SIGNAL:
>> +     * Return bind/unbind completion fence as output
>> +     */
>> +    __u32 flags;
>> +#define I915_VM_BIND_FENCE_WAIT            (1<<0)
>> +#define I915_VM_BIND_FENCE_SIGNAL          (1<<1)
>> +#define __I915_VM_BIND_FENCE_UNKNOWN_FLAGS 
>> (-(I915_VM_BIND_FENCE_SIGNAL << 1))
>> +};
>> +
>> +/**
>> + * struct drm_i915_vm_bind_ext_timeline_fences - Timeline fences for 
>> vm_bind
>> + * and vm_unbind.
>> + *
>> + * This structure describes an array of timeline drm_syncobj and 
>> associated
>> + * points for timeline variants of drm_syncobj. These timeline 
>> 'drm_syncobj's
>> + * can be input or output fences (See struct drm_i915_vm_bind_fence).
>> + */
>> +struct drm_i915_vm_bind_ext_timeline_fences {
>> +    /** @base: Extension link. See struct i915_user_extension. */
>> +    struct i915_user_extension base;
>> +
>> +    /**
>> +     * @fence_count: Number of elements in the @handles_ptr & 
>> @value_ptr
>> +     * arrays.
>> +     */
>> +    __u64 fence_count;
>> +
>> +    /**
>> +     * @handles_ptr: Pointer to an array of struct 
>> drm_i915_vm_bind_fence
>> +     * of length @fence_count.
>> +     */
>> +    __u64 handles_ptr;
>> +
>> +    /**
>> +     * @values_ptr: Pointer to an array of u64 values of length
>> +     * @fence_count.
>> +     * Values must be 0 for a binary drm_syncobj. A Value of 0 for a
>> +     * timeline drm_syncobj is invalid as it turns a drm_syncobj into a
>> +     * binary one.
>> +     */
>> +    __u64 values_ptr;
>> +};
>> +
>> +/**
>> + * struct drm_i915_vm_bind_user_fence - An input or output user 
>> fence for the
>> + * vm_bind or the vm_unbind work.
>> + *
>> + * The vm_bind or vm_unbind aync worker will wait for the input 
>> fence (value at
>> + * @addr to become equal to @val) before starting the binding or 
>> unbinding.
>> + *
>> + * The vm_bind or vm_unbind async worker will signal the output 
>> fence after
>> + * the completion of binding or unbinding by writing @val to memory 
>> location at
>> + * @addr
>> + */
>> +struct drm_i915_vm_bind_user_fence {
>> +    /** @addr: User/Memory fence qword aligned process virtual 
>> address */
>> +    __u64 addr;
>> +
>> +    /** @val: User/Memory fence value to be written after bind 
>> completion */
>> +    __u64 val;
>> +
>> +    /**
>> +     * @flags: Supported flags are:
>> +     *
>> +     * I915_VM_BIND_USER_FENCE_WAIT:
>> +     * Wait for the input fence before binding/unbinding
>> +     *
>> +     * I915_VM_BIND_USER_FENCE_SIGNAL:
>> +     * Return bind/unbind completion fence as output
>> +     */
>> +    __u32 flags;
>> +#define I915_VM_BIND_USER_FENCE_WAIT            (1<<0)
>> +#define I915_VM_BIND_USER_FENCE_SIGNAL          (1<<1)
>> +#define __I915_VM_BIND_USER_FENCE_UNKNOWN_FLAGS \
>> +    (-(I915_VM_BIND_USER_FENCE_SIGNAL << 1))
>> +};
>> +
>> +/**
>> + * struct drm_i915_vm_bind_ext_user_fence - User/memory fences for 
>> vm_bind
>> + * and vm_unbind.
>> + *
>> + * These user fences can be input or output fences
>> + * (See struct drm_i915_vm_bind_user_fence).
>> + */
>> +struct drm_i915_vm_bind_ext_user_fence {
>> +    /** @base: Extension link. See struct i915_user_extension. */
>> +    struct i915_user_extension base;
>> +
>> +    /** @fence_count: Number of elements in the @user_fence_ptr 
>> array. */
>> +    __u64 fence_count;
>> +
>> +    /**
>> +     * @user_fence_ptr: Pointer to an array of
>> +     * struct drm_i915_vm_bind_user_fence of length @fence_count.
>> +     */
>> +    __u64 user_fence_ptr;
>> +};
>> +
>> +/**
>> + * struct drm_i915_gem_execbuffer3 - Structure for 
>> DRM_I915_GEM_EXECBUFFER3
>> + * ioctl.
>> + *
>> + * DRM_I915_GEM_EXECBUFFER3 ioctl only works in VM_BIND mode and 
>> VM_BIND mode
>> + * only works with this ioctl for submission.
>> + * See I915_VM_CREATE_FLAGS_USE_VM_BIND.
>> + */
>> +struct drm_i915_gem_execbuffer3 {
>> +    /**
>> +     * @ctx_id: Context id
>> +     *
>> +     * Only contexts with user engine map are allowed.
>> +     */
>> +    __u32 ctx_id;
>> +
>> +    /**
>> +     * @engine_idx: Engine index
>> +     *
>> +     * An index in the user engine map of the context specified by 
>> @ctx_id.
>> +     */
>> +    __u32 engine_idx;
>> +
>> +    /** @rsvd1: Reserved, MBZ */
>> +    __u32 rsvd1;
>> +
>> +    /**
>> +     * @batch_count: Number of batches in @batch_address array.
>> +     *
>> +     * 0 is invalid. For parallel submission, it should be equal to the
>> +     * number of (parallel) engines involved in that submission.
>> +     */
>> +    __u32 batch_count;
>> +
>> +    /**
>> +     * @batch_address: Array of batch gpu virtual addresses.
>> +     *
>> +     * If @batch_count is 1, then it is the gpu virtual address of the
>> +     * batch buffer. If @batch_count > 1, then it is a pointer to an 
>> array
>> +     * of batch buffer gpu virtual addresses.
>> +     */
>> +    __u64 batch_address;
>> +
>> +    /**
>> +     * @flags: Supported flags are:
>> +     *
>> +     * I915_EXEC3_SECURE:
>> +     * Request a privileged ("secure") batch buffer/s.
>> +     * It is only available for DRM_ROOT_ONLY | DRM_MASTER processes.
>> +     */
>> +    __u64 flags;
>> +#define I915_EXEC3_SECURE    (1<<0)
>> +
>> +    /** @rsvd2: Reserved, MBZ */
>> +    __u64 rsvd2;
>> +
>> +    /**
>> +     * @extensions: Zero-terminated chain of extensions.
>> +     *
>> +     * DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES:
>> +     * It has same format as 
>> DRM_I915_GEM_EXECBUFFER_EXT_TIMELINE_FENCES.
>> +     * See struct drm_i915_gem_execbuffer_ext_timeline_fences.
>> +     *
>> +     * DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE:
>> +     * First level batch completion signaling extension.
>> +     * See struct drm_i915_gem_execbuffer3_ext_user_fence.
>> +     */
>> +    __u64 extensions;
>> +#define DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES    0
>> +#define DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE        1
>> +};
>> +
>> +/**
>> + * struct drm_i915_gem_execbuffer3_ext_user_fence - First level 
>> batch completion
>> + * signaling extension.
>> + *
>> + * This extension allows user to attach a user fence (@addr, @value 
>> pair) to
>> + * execbuf3, to be signaled by the command streamer after the 
>> completion of first
>> + * level batch, by writing the @value at specified @addr and 
>> triggering an
>> + * interrupt.
>> + * User can either poll for this user fence to signal or can also 
>> wait on it
>> + * with i915_gem_wait_user_fence ioctl.
>> + * This is very much usefaul for long running contexts where waiting 
>> on dma-fence
>> + * by user (like i915_gem_wait ioctl) is not supported.
>> + */
>> +struct drm_i915_gem_execbuffer3_ext_user_fence {
>> +    /** @base: Extension link. See struct i915_user_extension. */
>> +    struct i915_user_extension base;
>> +
>> +    /**
>> +     * @addr: User/Memory fence qword aligned GPU virtual address.
>> +     *
>> +     * Address has to be a valid GPU virtual address at the time of
>> +     * first level batch completion.
>> +     */
>> +    __u64 addr;
>> +
>> +    /**
>> +     * @value: User/Memory fence Value to be written to above address
>> +     * after first level batch completes.
>> +     */
>> +    __u64 value;
>> +
>> +    /** @rsvd: Reserved, MBZ */
>> +    __u64 rsvd;
>> +};
>> +
>> +/**
>> + * struct drm_i915_gem_create_ext_vm_private - Extension to make the 
>> object
>> + * private to the specified VM.
>> + *
>> + * See struct drm_i915_gem_create_ext.
>> + */
>> +struct drm_i915_gem_create_ext_vm_private {
>> +#define I915_GEM_CREATE_EXT_VM_PRIVATE        2
>> +    /** @base: Extension link. See struct i915_user_extension. */
>> +    struct i915_user_extension base;
>> +
>> +    /** @vm_id: Id of the VM to which the object is private */
>> +    __u32 vm_id;
>> +};
>> +
>> +/**
>> + * struct drm_i915_gem_wait_user_fence - Wait on user/memory fence.
>> + *
>> + * User/Memory fence can be woken up either by:
>> + *
>> + * 1. GPU context indicated by @ctx_id, or,
>> + * 2. Kerrnel driver async worker upon I915_UFENCE_WAIT_SOFT.
>> + *    @ctx_id is ignored when this flag is set.
>> + *
>> + * Wakeup condition is,
>> + * ``((*addr & mask) op (value & mask))``
>> + *
>> + * See :ref:`Documentation/driver-api/dma-buf.rst 
>> <indefinite_dma_fences>`
>> + */
>> +struct drm_i915_gem_wait_user_fence {
>> +    /** @extensions: Zero-terminated chain of extensions. */
>> +    __u64 extensions;
>> +
>> +    /** @addr: User/Memory fence address */
>> +    __u64 addr;
>> +
>> +    /** @ctx_id: Id of the Context which will signal the fence. */
>> +    __u32 ctx_id;
>> +
>> +    /** @op: Wakeup condition operator */
>> +    __u16 op;
>> +#define I915_UFENCE_WAIT_EQ      0
>> +#define I915_UFENCE_WAIT_NEQ     1
>> +#define I915_UFENCE_WAIT_GT      2
>> +#define I915_UFENCE_WAIT_GTE     3
>> +#define I915_UFENCE_WAIT_LT      4
>> +#define I915_UFENCE_WAIT_LTE     5
>> +#define I915_UFENCE_WAIT_BEFORE  6
>> +#define I915_UFENCE_WAIT_AFTER   7
>> +
>> +    /**
>> +     * @flags: Supported flags are:
>> +     *
>> +     * I915_UFENCE_WAIT_SOFT:
>> +     *
>> +     * To be woken up by i915 driver async worker (not by GPU).
>> +     *
>> +     * I915_UFENCE_WAIT_ABSTIME:
>> +     *
>> +     * Wait timeout specified as absolute time.
>> +     */
>> +    __u16 flags;
>> +#define I915_UFENCE_WAIT_SOFT    0x1
>> +#define I915_UFENCE_WAIT_ABSTIME 0x2
>> +
>> +    /** @value: Wakeup value */
>> +    __u64 value;
>> +
>> +    /** @mask: Wakeup mask */
>> +    __u64 mask;
>> +#define I915_UFENCE_WAIT_U8     0xffu
>> +#define I915_UFENCE_WAIT_U16    0xffffu
>> +#define I915_UFENCE_WAIT_U32    0xfffffffful
>> +#define I915_UFENCE_WAIT_U64    0xffffffffffffffffull
>> +
>> +    /**
>> +     * @timeout: Wait timeout in nanoseconds.
>> +     *
>> +     * If I915_UFENCE_WAIT_ABSTIME flag is set, then time timeout is 
>> the
>> +     * absolute time in nsec.
>> +     */
>> +    __s64 timeout;
>> +};

Re: [Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Niranjana Vishwanathapura]

On Fri, Jun 10, 2022 at 05:48:39PM +0300, Lionel Landwerlin wrote:
>On 10/06/2022 13:37, Tvrtko Ursulin wrote:
>>
>>On 10/06/2022 08:07, Niranjana Vishwanathapura wrote:
>>>VM_BIND and related uapi definitions
>>>
>>>Signed-off-by: Niranjana Vishwanathapura 
>>><niranjana.vishwanathapura@intel.com>
>>>---
>>>  Documentation/gpu/rfc/i915_vm_bind.h | 490 +++++++++++++++++++++++++++
>>>  1 file changed, 490 insertions(+)
>>>  create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
>>>
>>>diff --git a/Documentation/gpu/rfc/i915_vm_bind.h 
>>>b/Documentation/gpu/rfc/i915_vm_bind.h
>>>new file mode 100644
>>>index 000000000000..9fc854969cfb
>>>--- /dev/null
>>>+++ b/Documentation/gpu/rfc/i915_vm_bind.h
>>>@@ -0,0 +1,490 @@
>>>+/* SPDX-License-Identifier: MIT */
>>>+/*
>>>+ * Copyright © 2022 Intel Corporation
>>>+ */
>>>+
>>>+/**
>>>+ * DOC: I915_PARAM_HAS_VM_BIND
>>>+ *
>>>+ * VM_BIND feature availability.
>>>+ * See typedef drm_i915_getparam_t param.
>>>+ * bit[0]: If set, VM_BIND is supported, otherwise not.
>>>+ * bits[8-15]: VM_BIND implementation version.
>>>+ * version 0 will not have VM_BIND/UNBIND timeline fence array support.
>>>+ */
>>>+#define I915_PARAM_HAS_VM_BIND        57
>>>+
>>>+/**
>>>+ * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
>>>+ *
>>>+ * Flag to opt-in for VM_BIND mode of binding during VM creation.
>>>+ * See struct drm_i915_gem_vm_control flags.
>>>+ *
>>>+ * The older execbuf2 ioctl will not support VM_BIND mode of operation.
>>>+ * For VM_BIND mode, we have new execbuf3 ioctl which will not 
>>>accept any
>>>+ * execlist (See struct drm_i915_gem_execbuffer3 for more details).
>>>+ *
>>>+ */
>>>+#define I915_VM_CREATE_FLAGS_USE_VM_BIND    (1 << 0)
>>>+
>>>+/**
>>>+ * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
>>>+ *
>>>+ * Flag to declare context as long running.
>>>+ * See struct drm_i915_gem_context_create_ext flags.
>>>+ *
>>>+ * Usage of dma-fence expects that they complete in reasonable 
>>>amount of time.
>>>+ * Compute on the other hand can be long running. Hence it is not 
>>>appropriate
>>>+ * for compute contexts to export request completion dma-fence to user.
>>>+ * The dma-fence usage will be limited to in-kernel consumption only.
>>>+ * Compute contexts need to use user/memory fence.
>>>+ *
>>>+ * So, long running contexts do not support output fences. Hence,
>>>+ * I915_EXEC_FENCE_SIGNAL (See &drm_i915_gem_exec_fence.flags) is 
>>>expected
>>>+ * to be not used. DRM_I915_GEM_WAIT ioctl call is also not 
>>>supported for
>>>+ * objects mapped to long running contexts.
>>>+ */
>>>+#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u << 2)
>>>+
>>>+/* VM_BIND related ioctls */
>>>+#define DRM_I915_GEM_VM_BIND        0x3d
>>>+#define DRM_I915_GEM_VM_UNBIND        0x3e
>>>+#define DRM_I915_GEM_EXECBUFFER3    0x3f
>>>+#define DRM_I915_GEM_WAIT_USER_FENCE    0x40
>>>+
>>>+#define DRM_IOCTL_I915_GEM_VM_BIND DRM_IOWR(DRM_COMMAND_BASE + 
>>>DRM_I915_GEM_VM_BIND, struct drm_i915_gem_vm_bind)
>>>+#define DRM_IOCTL_I915_GEM_VM_UNBIND DRM_IOWR(DRM_COMMAND_BASE + 
>>>DRM_I915_GEM_VM_UNBIND, struct drm_i915_gem_vm_bind)
>>>+#define DRM_IOCTL_I915_GEM_EXECBUFFER3 DRM_IOWR(DRM_COMMAND_BASE 
>>>+ DRM_I915_GEM_EXECBUFFER3, struct drm_i915_gem_execbuffer3)
>>>+#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE 
>>>DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_WAIT_USER_FENCE, struct 
>>>drm_i915_gem_wait_user_fence)
>>>+
>>>+/**
>>>+ * struct drm_i915_gem_vm_bind - VA to object mapping to bind.
>>>+ *
>>>+ * This structure is passed to VM_BIND ioctl and specifies the 
>>>mapping of GPU
>>>+ * virtual address (VA) range to the section of an object that 
>>>should be bound
>>>+ * in the device page table of the specified address space (VM).
>>>+ * The VA range specified must be unique (ie., not currently 
>>>bound) and can
>>>+ * be mapped to whole object or a section of the object (partial 
>>>binding).
>>>+ * Multiple VA mappings can be created to the same section of the 
>>>object
>>>+ * (aliasing).
>>>+ *
>>>+ * The @queue_idx specifies the queue to use for binding. Same 
>>>queue can be
>>>+ * used for both VM_BIND and VM_UNBIND calls. All submitted bind 
>>>and unbind
>>>+ * operations in a queue are performed in the order of submission.
>>>+ *
>>>+ * The @start, @offset and @length should be 4K page aligned. 
>>>However the DG2
>>>+ * and XEHPSDV has 64K page size for device local-memory and has 
>>>compact page
>>>+ * table. On those platforms, for binding device local-memory 
>>>objects, the
>>>+ * @start should be 2M aligned, @offset and @length should be 64K 
>>>aligned.
>>>+ * Also, on those platforms, it is not allowed to bind an device 
>>>local-memory
>>>+ * object and a system memory object in a single 2M section of VA 
>>>range.
>>>+ */
>>>+struct drm_i915_gem_vm_bind {
>>>+    /** @vm_id: VM (address space) id to bind */
>>>+    __u32 vm_id;
>>>+
>>>+    /** @queue_idx: Index of queue for binding */
>>>+    __u32 queue_idx;
>>
>>I have a question here to which I did not find an answer by browsing 
>>the old threads.
>>
>>Queue index appears to be an implicit synchronisation mechanism, 
>>right? Operations on the same index are executed/complete in order 
>>of ioctl submission?
>>
>>Do we _have_ to implement this on the kernel side and could just 
>>allow in/out fence and let userspace deal with it?
>
>
>It orders operations like in a queue. Which is kind of what happens 
>with existing queues/engines.
>
>If I understood correctly, it's going to be a kthread + a linked list right?
>

Yes, that is correct.

>
>-Lionel
>
>
>>
>>Arbitrary/on-demand number of queues will add the complexity on the 
>>kernel side which should be avoided if possible.
>>

It was discussed in the other thread. Jason prefers this over putting
an artificial limit on number of queues (as user can anyway can exhaust
the memory). I think complexity in the driver is manageable.

The other option being discussed in to have the user create those
queues (like creating engine map) before hand and use that in vm_bind
and vm_unbind ioctls. This puts a limit on the number of queues.
But it is not clean either and not sure it is worth making the interface
more complex.
https://www.spinics.net/lists/dri-devel/msg350448.html

Niranjana

>>Regards,
>>
>>Tvrtko
>>
>>>+
>>>+    /** @rsvd: Reserved, MBZ */
>>>+    __u32 rsvd;
>>>+
>>>+    /** @handle: Object handle */
>>>+    __u32 handle;
>>>+
>>>+    /** @start: Virtual Address start to bind */
>>>+    __u64 start;
>>>+
>>>+    /** @offset: Offset in object to bind */
>>>+    __u64 offset;
>>>+
>>>+    /** @length: Length of mapping to bind */
>>>+    __u64 length;
>>>+
>>>+    /**
>>>+     * @flags: Supported flags are:
>>>+     *
>>>+     * I915_GEM_VM_BIND_READONLY:
>>>+     * Mapping is read-only.
>>>+     *
>>>+     * I915_GEM_VM_BIND_CAPTURE:
>>>+     * Capture this mapping in the dump upon GPU error.
>>>+     */
>>>+    __u64 flags;
>>>+#define I915_GEM_VM_BIND_READONLY    (1 << 0)
>>>+#define I915_GEM_VM_BIND_CAPTURE     (1 << 1)
>>>+
>>>+    /**
>>>+     * @extensions: 0-terminated chain of extensions for this 
>>>operation.
>>>+     *
>>>+     * I915_VM_BIND_EXT_TIMELINE_FENCES:
>>>+     * Specifies an array of input or output timeline fences for this
>>>+     * binding operation. See struct 
>>>drm_i915_vm_bind_ext_timeline_fences.
>>>+     *
>>>+     * I915_VM_BIND_EXT_USER_FENCES:
>>>+     * Specifies an array of input or output user fences for this
>>>+     * binding operation. See struct drm_i915_vm_bind_ext_user_fence.
>>>+     * This is required for compute contexts.
>>>+     */
>>>+    __u64 extensions;
>>>+#define I915_VM_BIND_EXT_TIMELINE_FENCES    0
>>>+#define I915_VM_BIND_EXT_USER_FENCES        1
>>>+};
>>>+
>>>+/**
>>>+ * struct drm_i915_gem_vm_unbind - VA to object mapping to unbind.
>>>+ *
>>>+ * This structure is passed to VM_UNBIND ioctl and specifies the 
>>>GPU virtual
>>>+ * address (VA) range that should be unbound from the device page 
>>>table of the
>>>+ * specified address space (VM). The specified VA range must 
>>>match one of the
>>>+ * mappings created with the VM_BIND ioctl. TLB is flushed upon unbind
>>>+ * completion.
>>>+ *
>>>+ * The @queue_idx specifies the queue to use for unbinding.
>>>+ * See struct drm_i915_gem_vm_unbind for more information on 
>>>@queue_idx.
>>>+ *
>>>+ * The @start and @length musy specify a unique mapping bound 
>>>with VM_BIND
>>>+ * ioctl.
>>>+ */
>>>+struct drm_i915_gem_vm_unbind {
>>>+    /** @vm_id: VM (address space) id to bind */
>>>+    __u32 vm_id;
>>>+
>>>+    /** @queue_idx: Index of queue for unbinding */
>>>+    __u32 queue_idx;
>>>+
>>>+    /** @start: Virtual Address start to unbind */
>>>+    __u64 start;
>>>+
>>>+    /** @length: Length of mapping to unbind */
>>>+    __u64 length;
>>>+
>>>+    /** @flags: Reserved for future usage, currently MBZ */
>>>+    __u64 flags;
>>>+
>>>+    /**
>>>+     * @extensions: 0-terminated chain of extensions for this 
>>>operation.
>>>+     *
>>>+     * I915_VM_UNBIND_EXT_TIMELINE_FENCES:
>>>+     * Specifies an array of input or output timeline fences for this
>>>+     * unbind operation.
>>>+     * It has same format as struct 
>>>drm_i915_vm_bind_ext_timeline_fences.
>>>+     *
>>>+     * I915_VM_UNBIND_EXT_USER_FENCES:
>>>+     * Specifies an array of input or output user fences for this
>>>+     * unbind operation. This is required for compute contexts.
>>>+     * It has same format as struct drm_i915_vm_bind_ext_user_fence.
>>>+     */
>>>+    __u64 extensions;
>>>+#define I915_VM_UNBIND_EXT_TIMELINE_FENCES    0
>>>+#define I915_VM_UNBIND_EXT_USER_FENCES        1
>>>+};
>>>+
>>>+/**
>>>+ * struct drm_i915_vm_bind_fence - An input or output fence for 
>>>the vm_bind
>>>+ * or the vm_unbind work.
>>>+ *
>>>+ * The vm_bind or vm_unbind aync worker will wait for input fence 
>>>to signal
>>>+ * before starting the binding or unbinding.
>>>+ *
>>>+ * The vm_bind or vm_unbind async worker will signal the returned 
>>>output fence
>>>+ * after the completion of binding or unbinding.
>>>+ */
>>>+struct drm_i915_vm_bind_fence {
>>>+    /** @handle: User's handle for a drm_syncobj to wait on or 
>>>signal. */
>>>+    __u32 handle;
>>>+
>>>+    /**
>>>+     * @flags: Supported flags are:
>>>+     *
>>>+     * I915_VM_BIND_FENCE_WAIT:
>>>+     * Wait for the input fence before binding/unbinding
>>>+     *
>>>+     * I915_VM_BIND_FENCE_SIGNAL:
>>>+     * Return bind/unbind completion fence as output
>>>+     */
>>>+    __u32 flags;
>>>+#define I915_VM_BIND_FENCE_WAIT            (1<<0)
>>>+#define I915_VM_BIND_FENCE_SIGNAL          (1<<1)
>>>+#define __I915_VM_BIND_FENCE_UNKNOWN_FLAGS 
>>>(-(I915_VM_BIND_FENCE_SIGNAL << 1))
>>>+};
>>>+
>>>+/**
>>>+ * struct drm_i915_vm_bind_ext_timeline_fences - Timeline fences 
>>>for vm_bind
>>>+ * and vm_unbind.
>>>+ *
>>>+ * This structure describes an array of timeline drm_syncobj and 
>>>associated
>>>+ * points for timeline variants of drm_syncobj. These timeline 
>>>'drm_syncobj's
>>>+ * can be input or output fences (See struct drm_i915_vm_bind_fence).
>>>+ */
>>>+struct drm_i915_vm_bind_ext_timeline_fences {
>>>+    /** @base: Extension link. See struct i915_user_extension. */
>>>+    struct i915_user_extension base;
>>>+
>>>+    /**
>>>+     * @fence_count: Number of elements in the @handles_ptr & 
>>>@value_ptr
>>>+     * arrays.
>>>+     */
>>>+    __u64 fence_count;
>>>+
>>>+    /**
>>>+     * @handles_ptr: Pointer to an array of struct 
>>>drm_i915_vm_bind_fence
>>>+     * of length @fence_count.
>>>+     */
>>>+    __u64 handles_ptr;
>>>+
>>>+    /**
>>>+     * @values_ptr: Pointer to an array of u64 values of length
>>>+     * @fence_count.
>>>+     * Values must be 0 for a binary drm_syncobj. A Value of 0 for a
>>>+     * timeline drm_syncobj is invalid as it turns a drm_syncobj into a
>>>+     * binary one.
>>>+     */
>>>+    __u64 values_ptr;
>>>+};
>>>+
>>>+/**
>>>+ * struct drm_i915_vm_bind_user_fence - An input or output user 
>>>fence for the
>>>+ * vm_bind or the vm_unbind work.
>>>+ *
>>>+ * The vm_bind or vm_unbind aync worker will wait for the input 
>>>fence (value at
>>>+ * @addr to become equal to @val) before starting the binding or 
>>>unbinding.
>>>+ *
>>>+ * The vm_bind or vm_unbind async worker will signal the output 
>>>fence after
>>>+ * the completion of binding or unbinding by writing @val to 
>>>memory location at
>>>+ * @addr
>>>+ */
>>>+struct drm_i915_vm_bind_user_fence {
>>>+    /** @addr: User/Memory fence qword aligned process virtual 
>>>address */
>>>+    __u64 addr;
>>>+
>>>+    /** @val: User/Memory fence value to be written after bind 
>>>completion */
>>>+    __u64 val;
>>>+
>>>+    /**
>>>+     * @flags: Supported flags are:
>>>+     *
>>>+     * I915_VM_BIND_USER_FENCE_WAIT:
>>>+     * Wait for the input fence before binding/unbinding
>>>+     *
>>>+     * I915_VM_BIND_USER_FENCE_SIGNAL:
>>>+     * Return bind/unbind completion fence as output
>>>+     */
>>>+    __u32 flags;
>>>+#define I915_VM_BIND_USER_FENCE_WAIT            (1<<0)
>>>+#define I915_VM_BIND_USER_FENCE_SIGNAL          (1<<1)
>>>+#define __I915_VM_BIND_USER_FENCE_UNKNOWN_FLAGS \
>>>+    (-(I915_VM_BIND_USER_FENCE_SIGNAL << 1))
>>>+};
>>>+
>>>+/**
>>>+ * struct drm_i915_vm_bind_ext_user_fence - User/memory fences 
>>>for vm_bind
>>>+ * and vm_unbind.
>>>+ *
>>>+ * These user fences can be input or output fences
>>>+ * (See struct drm_i915_vm_bind_user_fence).
>>>+ */
>>>+struct drm_i915_vm_bind_ext_user_fence {
>>>+    /** @base: Extension link. See struct i915_user_extension. */
>>>+    struct i915_user_extension base;
>>>+
>>>+    /** @fence_count: Number of elements in the @user_fence_ptr 
>>>array. */
>>>+    __u64 fence_count;
>>>+
>>>+    /**
>>>+     * @user_fence_ptr: Pointer to an array of
>>>+     * struct drm_i915_vm_bind_user_fence of length @fence_count.
>>>+     */
>>>+    __u64 user_fence_ptr;
>>>+};
>>>+
>>>+/**
>>>+ * struct drm_i915_gem_execbuffer3 - Structure for 
>>>DRM_I915_GEM_EXECBUFFER3
>>>+ * ioctl.
>>>+ *
>>>+ * DRM_I915_GEM_EXECBUFFER3 ioctl only works in VM_BIND mode and 
>>>VM_BIND mode
>>>+ * only works with this ioctl for submission.
>>>+ * See I915_VM_CREATE_FLAGS_USE_VM_BIND.
>>>+ */
>>>+struct drm_i915_gem_execbuffer3 {
>>>+    /**
>>>+     * @ctx_id: Context id
>>>+     *
>>>+     * Only contexts with user engine map are allowed.
>>>+     */
>>>+    __u32 ctx_id;
>>>+
>>>+    /**
>>>+     * @engine_idx: Engine index
>>>+     *
>>>+     * An index in the user engine map of the context specified 
>>>by @ctx_id.
>>>+     */
>>>+    __u32 engine_idx;
>>>+
>>>+    /** @rsvd1: Reserved, MBZ */
>>>+    __u32 rsvd1;
>>>+
>>>+    /**
>>>+     * @batch_count: Number of batches in @batch_address array.
>>>+     *
>>>+     * 0 is invalid. For parallel submission, it should be equal to the
>>>+     * number of (parallel) engines involved in that submission.
>>>+     */
>>>+    __u32 batch_count;
>>>+
>>>+    /**
>>>+     * @batch_address: Array of batch gpu virtual addresses.
>>>+     *
>>>+     * If @batch_count is 1, then it is the gpu virtual address of the
>>>+     * batch buffer. If @batch_count > 1, then it is a pointer to 
>>>an array
>>>+     * of batch buffer gpu virtual addresses.
>>>+     */
>>>+    __u64 batch_address;
>>>+
>>>+    /**
>>>+     * @flags: Supported flags are:
>>>+     *
>>>+     * I915_EXEC3_SECURE:
>>>+     * Request a privileged ("secure") batch buffer/s.
>>>+     * It is only available for DRM_ROOT_ONLY | DRM_MASTER processes.
>>>+     */
>>>+    __u64 flags;
>>>+#define I915_EXEC3_SECURE    (1<<0)
>>>+
>>>+    /** @rsvd2: Reserved, MBZ */
>>>+    __u64 rsvd2;
>>>+
>>>+    /**
>>>+     * @extensions: Zero-terminated chain of extensions.
>>>+     *
>>>+     * DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES:
>>>+     * It has same format as 
>>>DRM_I915_GEM_EXECBUFFER_EXT_TIMELINE_FENCES.
>>>+     * See struct drm_i915_gem_execbuffer_ext_timeline_fences.
>>>+     *
>>>+     * DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE:
>>>+     * First level batch completion signaling extension.
>>>+     * See struct drm_i915_gem_execbuffer3_ext_user_fence.
>>>+     */
>>>+    __u64 extensions;
>>>+#define DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES    0
>>>+#define DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE        1
>>>+};
>>>+
>>>+/**
>>>+ * struct drm_i915_gem_execbuffer3_ext_user_fence - First level 
>>>batch completion
>>>+ * signaling extension.
>>>+ *
>>>+ * This extension allows user to attach a user fence (@addr, 
>>>@value pair) to
>>>+ * execbuf3, to be signaled by the command streamer after the 
>>>completion of first
>>>+ * level batch, by writing the @value at specified @addr and 
>>>triggering an
>>>+ * interrupt.
>>>+ * User can either poll for this user fence to signal or can also 
>>>wait on it
>>>+ * with i915_gem_wait_user_fence ioctl.
>>>+ * This is very much usefaul for long running contexts where 
>>>waiting on dma-fence
>>>+ * by user (like i915_gem_wait ioctl) is not supported.
>>>+ */
>>>+struct drm_i915_gem_execbuffer3_ext_user_fence {
>>>+    /** @base: Extension link. See struct i915_user_extension. */
>>>+    struct i915_user_extension base;
>>>+
>>>+    /**
>>>+     * @addr: User/Memory fence qword aligned GPU virtual address.
>>>+     *
>>>+     * Address has to be a valid GPU virtual address at the time of
>>>+     * first level batch completion.
>>>+     */
>>>+    __u64 addr;
>>>+
>>>+    /**
>>>+     * @value: User/Memory fence Value to be written to above address
>>>+     * after first level batch completes.
>>>+     */
>>>+    __u64 value;
>>>+
>>>+    /** @rsvd: Reserved, MBZ */
>>>+    __u64 rsvd;
>>>+};
>>>+
>>>+/**
>>>+ * struct drm_i915_gem_create_ext_vm_private - Extension to make 
>>>the object
>>>+ * private to the specified VM.
>>>+ *
>>>+ * See struct drm_i915_gem_create_ext.
>>>+ */
>>>+struct drm_i915_gem_create_ext_vm_private {
>>>+#define I915_GEM_CREATE_EXT_VM_PRIVATE        2
>>>+    /** @base: Extension link. See struct i915_user_extension. */
>>>+    struct i915_user_extension base;
>>>+
>>>+    /** @vm_id: Id of the VM to which the object is private */
>>>+    __u32 vm_id;
>>>+};
>>>+
>>>+/**
>>>+ * struct drm_i915_gem_wait_user_fence - Wait on user/memory fence.
>>>+ *
>>>+ * User/Memory fence can be woken up either by:
>>>+ *
>>>+ * 1. GPU context indicated by @ctx_id, or,
>>>+ * 2. Kerrnel driver async worker upon I915_UFENCE_WAIT_SOFT.
>>>+ *    @ctx_id is ignored when this flag is set.
>>>+ *
>>>+ * Wakeup condition is,
>>>+ * ``((*addr & mask) op (value & mask))``
>>>+ *
>>>+ * See :ref:`Documentation/driver-api/dma-buf.rst 
>>><indefinite_dma_fences>`
>>>+ */
>>>+struct drm_i915_gem_wait_user_fence {
>>>+    /** @extensions: Zero-terminated chain of extensions. */
>>>+    __u64 extensions;
>>>+
>>>+    /** @addr: User/Memory fence address */
>>>+    __u64 addr;
>>>+
>>>+    /** @ctx_id: Id of the Context which will signal the fence. */
>>>+    __u32 ctx_id;
>>>+
>>>+    /** @op: Wakeup condition operator */
>>>+    __u16 op;
>>>+#define I915_UFENCE_WAIT_EQ      0
>>>+#define I915_UFENCE_WAIT_NEQ     1
>>>+#define I915_UFENCE_WAIT_GT      2
>>>+#define I915_UFENCE_WAIT_GTE     3
>>>+#define I915_UFENCE_WAIT_LT      4
>>>+#define I915_UFENCE_WAIT_LTE     5
>>>+#define I915_UFENCE_WAIT_BEFORE  6
>>>+#define I915_UFENCE_WAIT_AFTER   7
>>>+
>>>+    /**
>>>+     * @flags: Supported flags are:
>>>+     *
>>>+     * I915_UFENCE_WAIT_SOFT:
>>>+     *
>>>+     * To be woken up by i915 driver async worker (not by GPU).
>>>+     *
>>>+     * I915_UFENCE_WAIT_ABSTIME:
>>>+     *
>>>+     * Wait timeout specified as absolute time.
>>>+     */
>>>+    __u16 flags;
>>>+#define I915_UFENCE_WAIT_SOFT    0x1
>>>+#define I915_UFENCE_WAIT_ABSTIME 0x2
>>>+
>>>+    /** @value: Wakeup value */
>>>+    __u64 value;
>>>+
>>>+    /** @mask: Wakeup mask */
>>>+    __u64 mask;
>>>+#define I915_UFENCE_WAIT_U8     0xffu
>>>+#define I915_UFENCE_WAIT_U16    0xffffu
>>>+#define I915_UFENCE_WAIT_U32    0xfffffffful
>>>+#define I915_UFENCE_WAIT_U64    0xffffffffffffffffull
>>>+
>>>+    /**
>>>+     * @timeout: Wait timeout in nanoseconds.
>>>+     *
>>>+     * If I915_UFENCE_WAIT_ABSTIME flag is set, then time timeout 
>>>is the
>>>+     * absolute time in nsec.
>>>+     */
>>>+    __s64 timeout;
>>>+};
>
>

Re: [Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Tvrtko Ursulin]

On 10/06/2022 17:14, Niranjana Vishwanathapura wrote:
> On Fri, Jun 10, 2022 at 05:48:39PM +0300, Lionel Landwerlin wrote:
>> On 10/06/2022 13:37, Tvrtko Ursulin wrote:
>>>
>>> On 10/06/2022 08:07, Niranjana Vishwanathapura wrote:
>>>> VM_BIND and related uapi definitions
>>>>
>>>> Signed-off-by: Niranjana Vishwanathapura 
>>>> <niranjana.vishwanathapura@intel.com>
>>>> ---
>>>>   Documentation/gpu/rfc/i915_vm_bind.h | 490 
>>>> +++++++++++++++++++++++++++
>>>>   1 file changed, 490 insertions(+)
>>>>   create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
>>>>
>>>> diff --git a/Documentation/gpu/rfc/i915_vm_bind.h 
>>>> b/Documentation/gpu/rfc/i915_vm_bind.h
>>>> new file mode 100644
>>>> index 000000000000..9fc854969cfb
>>>> --- /dev/null
>>>> +++ b/Documentation/gpu/rfc/i915_vm_bind.h
>>>> @@ -0,0 +1,490 @@
>>>> +/* SPDX-License-Identifier: MIT */
>>>> +/*
>>>> + * Copyright © 2022 Intel Corporation
>>>> + */
>>>> +
>>>> +/**
>>>> + * DOC: I915_PARAM_HAS_VM_BIND
>>>> + *
>>>> + * VM_BIND feature availability.
>>>> + * See typedef drm_i915_getparam_t param.
>>>> + * bit[0]: If set, VM_BIND is supported, otherwise not.
>>>> + * bits[8-15]: VM_BIND implementation version.
>>>> + * version 0 will not have VM_BIND/UNBIND timeline fence array 
>>>> support.
>>>> + */
>>>> +#define I915_PARAM_HAS_VM_BIND        57
>>>> +
>>>> +/**
>>>> + * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
>>>> + *
>>>> + * Flag to opt-in for VM_BIND mode of binding during VM creation.
>>>> + * See struct drm_i915_gem_vm_control flags.
>>>> + *
>>>> + * The older execbuf2 ioctl will not support VM_BIND mode of 
>>>> operation.
>>>> + * For VM_BIND mode, we have new execbuf3 ioctl which will not 
>>>> accept any
>>>> + * execlist (See struct drm_i915_gem_execbuffer3 for more details).
>>>> + *
>>>> + */
>>>> +#define I915_VM_CREATE_FLAGS_USE_VM_BIND    (1 << 0)
>>>> +
>>>> +/**
>>>> + * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
>>>> + *
>>>> + * Flag to declare context as long running.
>>>> + * See struct drm_i915_gem_context_create_ext flags.
>>>> + *
>>>> + * Usage of dma-fence expects that they complete in reasonable 
>>>> amount of time.
>>>> + * Compute on the other hand can be long running. Hence it is not 
>>>> appropriate
>>>> + * for compute contexts to export request completion dma-fence to 
>>>> user.
>>>> + * The dma-fence usage will be limited to in-kernel consumption only.
>>>> + * Compute contexts need to use user/memory fence.
>>>> + *
>>>> + * So, long running contexts do not support output fences. Hence,
>>>> + * I915_EXEC_FENCE_SIGNAL (See &drm_i915_gem_exec_fence.flags) is 
>>>> expected
>>>> + * to be not used. DRM_I915_GEM_WAIT ioctl call is also not 
>>>> supported for
>>>> + * objects mapped to long running contexts.
>>>> + */
>>>> +#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u << 2)
>>>> +
>>>> +/* VM_BIND related ioctls */
>>>> +#define DRM_I915_GEM_VM_BIND        0x3d
>>>> +#define DRM_I915_GEM_VM_UNBIND        0x3e
>>>> +#define DRM_I915_GEM_EXECBUFFER3    0x3f
>>>> +#define DRM_I915_GEM_WAIT_USER_FENCE    0x40
>>>> +
>>>> +#define DRM_IOCTL_I915_GEM_VM_BIND DRM_IOWR(DRM_COMMAND_BASE + 
>>>> DRM_I915_GEM_VM_BIND, struct drm_i915_gem_vm_bind)
>>>> +#define DRM_IOCTL_I915_GEM_VM_UNBIND DRM_IOWR(DRM_COMMAND_BASE + 
>>>> DRM_I915_GEM_VM_UNBIND, struct drm_i915_gem_vm_bind)
>>>> +#define DRM_IOCTL_I915_GEM_EXECBUFFER3 DRM_IOWR(DRM_COMMAND_BASE + 
>>>> DRM_I915_GEM_EXECBUFFER3, struct drm_i915_gem_execbuffer3)
>>>> +#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE 
>>>> DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_WAIT_USER_FENCE, struct 
>>>> drm_i915_gem_wait_user_fence)
>>>> +
>>>> +/**
>>>> + * struct drm_i915_gem_vm_bind - VA to object mapping to bind.
>>>> + *
>>>> + * This structure is passed to VM_BIND ioctl and specifies the 
>>>> mapping of GPU
>>>> + * virtual address (VA) range to the section of an object that 
>>>> should be bound
>>>> + * in the device page table of the specified address space (VM).
>>>> + * The VA range specified must be unique (ie., not currently bound) 
>>>> and can
>>>> + * be mapped to whole object or a section of the object (partial 
>>>> binding).
>>>> + * Multiple VA mappings can be created to the same section of the 
>>>> object
>>>> + * (aliasing).
>>>> + *
>>>> + * The @queue_idx specifies the queue to use for binding. Same 
>>>> queue can be
>>>> + * used for both VM_BIND and VM_UNBIND calls. All submitted bind 
>>>> and unbind
>>>> + * operations in a queue are performed in the order of submission.
>>>> + *
>>>> + * The @start, @offset and @length should be 4K page aligned. 
>>>> However the DG2
>>>> + * and XEHPSDV has 64K page size for device local-memory and has 
>>>> compact page
>>>> + * table. On those platforms, for binding device local-memory 
>>>> objects, the
>>>> + * @start should be 2M aligned, @offset and @length should be 64K 
>>>> aligned.
>>>> + * Also, on those platforms, it is not allowed to bind an device 
>>>> local-memory
>>>> + * object and a system memory object in a single 2M section of VA 
>>>> range.
>>>> + */
>>>> +struct drm_i915_gem_vm_bind {
>>>> +    /** @vm_id: VM (address space) id to bind */
>>>> +    __u32 vm_id;
>>>> +
>>>> +    /** @queue_idx: Index of queue for binding */
>>>> +    __u32 queue_idx;
>>>
>>> I have a question here to which I did not find an answer by browsing 
>>> the old threads.
>>>
>>> Queue index appears to be an implicit synchronisation mechanism, 
>>> right? Operations on the same index are executed/complete in order of 
>>> ioctl submission?
>>>
>>> Do we _have_ to implement this on the kernel side and could just 
>>> allow in/out fence and let userspace deal with it?
>>
>>
>> It orders operations like in a queue. Which is kind of what happens 
>> with existing queues/engines.
>>
>> If I understood correctly, it's going to be a kthread + a linked list 
>> right?
>>
> 
> Yes, that is correct.
> 
>>
>> -Lionel
>>
>>
>>>
>>> Arbitrary/on-demand number of queues will add the complexity on the 
>>> kernel side which should be avoided if possible.
>>>
> 
> It was discussed in the other thread. Jason prefers this over putting
> an artificial limit on number of queues (as user can anyway can exhaust
> the memory). I think complexity in the driver is manageable.

You'll need to create tracking structures on demand, with atomic replace 
of last fence, ref counting and locking of some sort, more or less?

> The other option being discussed in to have the user create those
> queues (like creating engine map) before hand and use that in vm_bind
> and vm_unbind ioctls. This puts a limit on the number of queues.
> But it is not clean either and not sure it is worth making the interface
> more complex.
> https://www.spinics.net/lists/dri-devel/msg350448.html

What about the third option of a flag to return a fence (of some sort) 
and pass in a fence? That way userspace can imagine zero or N queues 
with very little effort on the kernel side. Was this considered?

Regards,

Tvrtko

>>> Regards,
>>>
>>> Tvrtko
>>>
>>>> +
>>>> +    /** @rsvd: Reserved, MBZ */
>>>> +    __u32 rsvd;
>>>> +
>>>> +    /** @handle: Object handle */
>>>> +    __u32 handle;
>>>> +
>>>> +    /** @start: Virtual Address start to bind */
>>>> +    __u64 start;
>>>> +
>>>> +    /** @offset: Offset in object to bind */
>>>> +    __u64 offset;
>>>> +
>>>> +    /** @length: Length of mapping to bind */
>>>> +    __u64 length;
>>>> +
>>>> +    /**
>>>> +     * @flags: Supported flags are:
>>>> +     *
>>>> +     * I915_GEM_VM_BIND_READONLY:
>>>> +     * Mapping is read-only.
>>>> +     *
>>>> +     * I915_GEM_VM_BIND_CAPTURE:
>>>> +     * Capture this mapping in the dump upon GPU error.
>>>> +     */
>>>> +    __u64 flags;
>>>> +#define I915_GEM_VM_BIND_READONLY    (1 << 0)
>>>> +#define I915_GEM_VM_BIND_CAPTURE     (1 << 1)
>>>> +
>>>> +    /**
>>>> +     * @extensions: 0-terminated chain of extensions for this 
>>>> operation.
>>>> +     *
>>>> +     * I915_VM_BIND_EXT_TIMELINE_FENCES:
>>>> +     * Specifies an array of input or output timeline fences for this
>>>> +     * binding operation. See struct 
>>>> drm_i915_vm_bind_ext_timeline_fences.
>>>> +     *
>>>> +     * I915_VM_BIND_EXT_USER_FENCES:
>>>> +     * Specifies an array of input or output user fences for this
>>>> +     * binding operation. See struct drm_i915_vm_bind_ext_user_fence.
>>>> +     * This is required for compute contexts.
>>>> +     */
>>>> +    __u64 extensions;
>>>> +#define I915_VM_BIND_EXT_TIMELINE_FENCES    0
>>>> +#define I915_VM_BIND_EXT_USER_FENCES        1
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct drm_i915_gem_vm_unbind - VA to object mapping to unbind.
>>>> + *
>>>> + * This structure is passed to VM_UNBIND ioctl and specifies the 
>>>> GPU virtual
>>>> + * address (VA) range that should be unbound from the device page 
>>>> table of the
>>>> + * specified address space (VM). The specified VA range must match 
>>>> one of the
>>>> + * mappings created with the VM_BIND ioctl. TLB is flushed upon unbind
>>>> + * completion.
>>>> + *
>>>> + * The @queue_idx specifies the queue to use for unbinding.
>>>> + * See struct drm_i915_gem_vm_unbind for more information on 
>>>> @queue_idx.
>>>> + *
>>>> + * The @start and @length musy specify a unique mapping bound with 
>>>> VM_BIND
>>>> + * ioctl.
>>>> + */
>>>> +struct drm_i915_gem_vm_unbind {
>>>> +    /** @vm_id: VM (address space) id to bind */
>>>> +    __u32 vm_id;
>>>> +
>>>> +    /** @queue_idx: Index of queue for unbinding */
>>>> +    __u32 queue_idx;
>>>> +
>>>> +    /** @start: Virtual Address start to unbind */
>>>> +    __u64 start;
>>>> +
>>>> +    /** @length: Length of mapping to unbind */
>>>> +    __u64 length;
>>>> +
>>>> +    /** @flags: Reserved for future usage, currently MBZ */
>>>> +    __u64 flags;
>>>> +
>>>> +    /**
>>>> +     * @extensions: 0-terminated chain of extensions for this 
>>>> operation.
>>>> +     *
>>>> +     * I915_VM_UNBIND_EXT_TIMELINE_FENCES:
>>>> +     * Specifies an array of input or output timeline fences for this
>>>> +     * unbind operation.
>>>> +     * It has same format as struct 
>>>> drm_i915_vm_bind_ext_timeline_fences.
>>>> +     *
>>>> +     * I915_VM_UNBIND_EXT_USER_FENCES:
>>>> +     * Specifies an array of input or output user fences for this
>>>> +     * unbind operation. This is required for compute contexts.
>>>> +     * It has same format as struct drm_i915_vm_bind_ext_user_fence.
>>>> +     */
>>>> +    __u64 extensions;
>>>> +#define I915_VM_UNBIND_EXT_TIMELINE_FENCES    0
>>>> +#define I915_VM_UNBIND_EXT_USER_FENCES        1
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct drm_i915_vm_bind_fence - An input or output fence for the 
>>>> vm_bind
>>>> + * or the vm_unbind work.
>>>> + *
>>>> + * The vm_bind or vm_unbind aync worker will wait for input fence 
>>>> to signal
>>>> + * before starting the binding or unbinding.
>>>> + *
>>>> + * The vm_bind or vm_unbind async worker will signal the returned 
>>>> output fence
>>>> + * after the completion of binding or unbinding.
>>>> + */
>>>> +struct drm_i915_vm_bind_fence {
>>>> +    /** @handle: User's handle for a drm_syncobj to wait on or 
>>>> signal. */
>>>> +    __u32 handle;
>>>> +
>>>> +    /**
>>>> +     * @flags: Supported flags are:
>>>> +     *
>>>> +     * I915_VM_BIND_FENCE_WAIT:
>>>> +     * Wait for the input fence before binding/unbinding
>>>> +     *
>>>> +     * I915_VM_BIND_FENCE_SIGNAL:
>>>> +     * Return bind/unbind completion fence as output
>>>> +     */
>>>> +    __u32 flags;
>>>> +#define I915_VM_BIND_FENCE_WAIT            (1<<0)
>>>> +#define I915_VM_BIND_FENCE_SIGNAL          (1<<1)
>>>> +#define __I915_VM_BIND_FENCE_UNKNOWN_FLAGS 
>>>> (-(I915_VM_BIND_FENCE_SIGNAL << 1))
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct drm_i915_vm_bind_ext_timeline_fences - Timeline fences 
>>>> for vm_bind
>>>> + * and vm_unbind.
>>>> + *
>>>> + * This structure describes an array of timeline drm_syncobj and 
>>>> associated
>>>> + * points for timeline variants of drm_syncobj. These timeline 
>>>> 'drm_syncobj's
>>>> + * can be input or output fences (See struct drm_i915_vm_bind_fence).
>>>> + */
>>>> +struct drm_i915_vm_bind_ext_timeline_fences {
>>>> +    /** @base: Extension link. See struct i915_user_extension. */
>>>> +    struct i915_user_extension base;
>>>> +
>>>> +    /**
>>>> +     * @fence_count: Number of elements in the @handles_ptr & 
>>>> @value_ptr
>>>> +     * arrays.
>>>> +     */
>>>> +    __u64 fence_count;
>>>> +
>>>> +    /**
>>>> +     * @handles_ptr: Pointer to an array of struct 
>>>> drm_i915_vm_bind_fence
>>>> +     * of length @fence_count.
>>>> +     */
>>>> +    __u64 handles_ptr;
>>>> +
>>>> +    /**
>>>> +     * @values_ptr: Pointer to an array of u64 values of length
>>>> +     * @fence_count.
>>>> +     * Values must be 0 for a binary drm_syncobj. A Value of 0 for a
>>>> +     * timeline drm_syncobj is invalid as it turns a drm_syncobj 
>>>> into a
>>>> +     * binary one.
>>>> +     */
>>>> +    __u64 values_ptr;
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct drm_i915_vm_bind_user_fence - An input or output user 
>>>> fence for the
>>>> + * vm_bind or the vm_unbind work.
>>>> + *
>>>> + * The vm_bind or vm_unbind aync worker will wait for the input 
>>>> fence (value at
>>>> + * @addr to become equal to @val) before starting the binding or 
>>>> unbinding.
>>>> + *
>>>> + * The vm_bind or vm_unbind async worker will signal the output 
>>>> fence after
>>>> + * the completion of binding or unbinding by writing @val to memory 
>>>> location at
>>>> + * @addr
>>>> + */
>>>> +struct drm_i915_vm_bind_user_fence {
>>>> +    /** @addr: User/Memory fence qword aligned process virtual 
>>>> address */
>>>> +    __u64 addr;
>>>> +
>>>> +    /** @val: User/Memory fence value to be written after bind 
>>>> completion */
>>>> +    __u64 val;
>>>> +
>>>> +    /**
>>>> +     * @flags: Supported flags are:
>>>> +     *
>>>> +     * I915_VM_BIND_USER_FENCE_WAIT:
>>>> +     * Wait for the input fence before binding/unbinding
>>>> +     *
>>>> +     * I915_VM_BIND_USER_FENCE_SIGNAL:
>>>> +     * Return bind/unbind completion fence as output
>>>> +     */
>>>> +    __u32 flags;
>>>> +#define I915_VM_BIND_USER_FENCE_WAIT            (1<<0)
>>>> +#define I915_VM_BIND_USER_FENCE_SIGNAL          (1<<1)
>>>> +#define __I915_VM_BIND_USER_FENCE_UNKNOWN_FLAGS \
>>>> +    (-(I915_VM_BIND_USER_FENCE_SIGNAL << 1))
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct drm_i915_vm_bind_ext_user_fence - User/memory fences for 
>>>> vm_bind
>>>> + * and vm_unbind.
>>>> + *
>>>> + * These user fences can be input or output fences
>>>> + * (See struct drm_i915_vm_bind_user_fence).
>>>> + */
>>>> +struct drm_i915_vm_bind_ext_user_fence {
>>>> +    /** @base: Extension link. See struct i915_user_extension. */
>>>> +    struct i915_user_extension base;
>>>> +
>>>> +    /** @fence_count: Number of elements in the @user_fence_ptr 
>>>> array. */
>>>> +    __u64 fence_count;
>>>> +
>>>> +    /**
>>>> +     * @user_fence_ptr: Pointer to an array of
>>>> +     * struct drm_i915_vm_bind_user_fence of length @fence_count.
>>>> +     */
>>>> +    __u64 user_fence_ptr;
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct drm_i915_gem_execbuffer3 - Structure for 
>>>> DRM_I915_GEM_EXECBUFFER3
>>>> + * ioctl.
>>>> + *
>>>> + * DRM_I915_GEM_EXECBUFFER3 ioctl only works in VM_BIND mode and 
>>>> VM_BIND mode
>>>> + * only works with this ioctl for submission.
>>>> + * See I915_VM_CREATE_FLAGS_USE_VM_BIND.
>>>> + */
>>>> +struct drm_i915_gem_execbuffer3 {
>>>> +    /**
>>>> +     * @ctx_id: Context id
>>>> +     *
>>>> +     * Only contexts with user engine map are allowed.
>>>> +     */
>>>> +    __u32 ctx_id;
>>>> +
>>>> +    /**
>>>> +     * @engine_idx: Engine index
>>>> +     *
>>>> +     * An index in the user engine map of the context specified by 
>>>> @ctx_id.
>>>> +     */
>>>> +    __u32 engine_idx;
>>>> +
>>>> +    /** @rsvd1: Reserved, MBZ */
>>>> +    __u32 rsvd1;
>>>> +
>>>> +    /**
>>>> +     * @batch_count: Number of batches in @batch_address array.
>>>> +     *
>>>> +     * 0 is invalid. For parallel submission, it should be equal to 
>>>> the
>>>> +     * number of (parallel) engines involved in that submission.
>>>> +     */
>>>> +    __u32 batch_count;
>>>> +
>>>> +    /**
>>>> +     * @batch_address: Array of batch gpu virtual addresses.
>>>> +     *
>>>> +     * If @batch_count is 1, then it is the gpu virtual address of the
>>>> +     * batch buffer. If @batch_count > 1, then it is a pointer to 
>>>> an array
>>>> +     * of batch buffer gpu virtual addresses.
>>>> +     */
>>>> +    __u64 batch_address;
>>>> +
>>>> +    /**
>>>> +     * @flags: Supported flags are:
>>>> +     *
>>>> +     * I915_EXEC3_SECURE:
>>>> +     * Request a privileged ("secure") batch buffer/s.
>>>> +     * It is only available for DRM_ROOT_ONLY | DRM_MASTER processes.
>>>> +     */
>>>> +    __u64 flags;
>>>> +#define I915_EXEC3_SECURE    (1<<0)
>>>> +
>>>> +    /** @rsvd2: Reserved, MBZ */
>>>> +    __u64 rsvd2;
>>>> +
>>>> +    /**
>>>> +     * @extensions: Zero-terminated chain of extensions.
>>>> +     *
>>>> +     * DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES:
>>>> +     * It has same format as 
>>>> DRM_I915_GEM_EXECBUFFER_EXT_TIMELINE_FENCES.
>>>> +     * See struct drm_i915_gem_execbuffer_ext_timeline_fences.
>>>> +     *
>>>> +     * DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE:
>>>> +     * First level batch completion signaling extension.
>>>> +     * See struct drm_i915_gem_execbuffer3_ext_user_fence.
>>>> +     */
>>>> +    __u64 extensions;
>>>> +#define DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES    0
>>>> +#define DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE        1
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct drm_i915_gem_execbuffer3_ext_user_fence - First level 
>>>> batch completion
>>>> + * signaling extension.
>>>> + *
>>>> + * This extension allows user to attach a user fence (@addr, @value 
>>>> pair) to
>>>> + * execbuf3, to be signaled by the command streamer after the 
>>>> completion of first
>>>> + * level batch, by writing the @value at specified @addr and 
>>>> triggering an
>>>> + * interrupt.
>>>> + * User can either poll for this user fence to signal or can also 
>>>> wait on it
>>>> + * with i915_gem_wait_user_fence ioctl.
>>>> + * This is very much usefaul for long running contexts where 
>>>> waiting on dma-fence
>>>> + * by user (like i915_gem_wait ioctl) is not supported.
>>>> + */
>>>> +struct drm_i915_gem_execbuffer3_ext_user_fence {
>>>> +    /** @base: Extension link. See struct i915_user_extension. */
>>>> +    struct i915_user_extension base;
>>>> +
>>>> +    /**
>>>> +     * @addr: User/Memory fence qword aligned GPU virtual address.
>>>> +     *
>>>> +     * Address has to be a valid GPU virtual address at the time of
>>>> +     * first level batch completion.
>>>> +     */
>>>> +    __u64 addr;
>>>> +
>>>> +    /**
>>>> +     * @value: User/Memory fence Value to be written to above address
>>>> +     * after first level batch completes.
>>>> +     */
>>>> +    __u64 value;
>>>> +
>>>> +    /** @rsvd: Reserved, MBZ */
>>>> +    __u64 rsvd;
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct drm_i915_gem_create_ext_vm_private - Extension to make 
>>>> the object
>>>> + * private to the specified VM.
>>>> + *
>>>> + * See struct drm_i915_gem_create_ext.
>>>> + */
>>>> +struct drm_i915_gem_create_ext_vm_private {
>>>> +#define I915_GEM_CREATE_EXT_VM_PRIVATE        2
>>>> +    /** @base: Extension link. See struct i915_user_extension. */
>>>> +    struct i915_user_extension base;
>>>> +
>>>> +    /** @vm_id: Id of the VM to which the object is private */
>>>> +    __u32 vm_id;
>>>> +};
>>>> +
>>>> +/**
>>>> + * struct drm_i915_gem_wait_user_fence - Wait on user/memory fence.
>>>> + *
>>>> + * User/Memory fence can be woken up either by:
>>>> + *
>>>> + * 1. GPU context indicated by @ctx_id, or,
>>>> + * 2. Kerrnel driver async worker upon I915_UFENCE_WAIT_SOFT.
>>>> + *    @ctx_id is ignored when this flag is set.
>>>> + *
>>>> + * Wakeup condition is,
>>>> + * ``((*addr & mask) op (value & mask))``
>>>> + *
>>>> + * See :ref:`Documentation/driver-api/dma-buf.rst 
>>>> <indefinite_dma_fences>`
>>>> + */
>>>> +struct drm_i915_gem_wait_user_fence {
>>>> +    /** @extensions: Zero-terminated chain of extensions. */
>>>> +    __u64 extensions;
>>>> +
>>>> +    /** @addr: User/Memory fence address */
>>>> +    __u64 addr;
>>>> +
>>>> +    /** @ctx_id: Id of the Context which will signal the fence. */
>>>> +    __u32 ctx_id;
>>>> +
>>>> +    /** @op: Wakeup condition operator */
>>>> +    __u16 op;
>>>> +#define I915_UFENCE_WAIT_EQ      0
>>>> +#define I915_UFENCE_WAIT_NEQ     1
>>>> +#define I915_UFENCE_WAIT_GT      2
>>>> +#define I915_UFENCE_WAIT_GTE     3
>>>> +#define I915_UFENCE_WAIT_LT      4
>>>> +#define I915_UFENCE_WAIT_LTE     5
>>>> +#define I915_UFENCE_WAIT_BEFORE  6
>>>> +#define I915_UFENCE_WAIT_AFTER   7
>>>> +
>>>> +    /**
>>>> +     * @flags: Supported flags are:
>>>> +     *
>>>> +     * I915_UFENCE_WAIT_SOFT:
>>>> +     *
>>>> +     * To be woken up by i915 driver async worker (not by GPU).
>>>> +     *
>>>> +     * I915_UFENCE_WAIT_ABSTIME:
>>>> +     *
>>>> +     * Wait timeout specified as absolute time.
>>>> +     */
>>>> +    __u16 flags;
>>>> +#define I915_UFENCE_WAIT_SOFT    0x1
>>>> +#define I915_UFENCE_WAIT_ABSTIME 0x2
>>>> +
>>>> +    /** @value: Wakeup value */
>>>> +    __u64 value;
>>>> +
>>>> +    /** @mask: Wakeup mask */
>>>> +    __u64 mask;
>>>> +#define I915_UFENCE_WAIT_U8     0xffu
>>>> +#define I915_UFENCE_WAIT_U16    0xffffu
>>>> +#define I915_UFENCE_WAIT_U32    0xfffffffful
>>>> +#define I915_UFENCE_WAIT_U64    0xffffffffffffffffull
>>>> +
>>>> +    /**
>>>> +     * @timeout: Wait timeout in nanoseconds.
>>>> +     *
>>>> +     * If I915_UFENCE_WAIT_ABSTIME flag is set, then time timeout 
>>>> is the
>>>> +     * absolute time in nsec.
>>>> +     */
>>>> +    __s64 timeout;
>>>> +};
>>
>>

Re: [Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Niranjana Vishwanathapura]

On Mon, Jun 13, 2022 at 09:24:18AM +0100, Tvrtko Ursulin wrote:
>
>On 10/06/2022 17:14, Niranjana Vishwanathapura wrote:
>>On Fri, Jun 10, 2022 at 05:48:39PM +0300, Lionel Landwerlin wrote:
>>>On 10/06/2022 13:37, Tvrtko Ursulin wrote:
>>>>
>>>>On 10/06/2022 08:07, Niranjana Vishwanathapura wrote:
>>>>>VM_BIND and related uapi definitions
>>>>>
>>>>>Signed-off-by: Niranjana Vishwanathapura 
>>>>><niranjana.vishwanathapura@intel.com>
>>>>>---
>>>>>  Documentation/gpu/rfc/i915_vm_bind.h | 490 
>>>>>+++++++++++++++++++++++++++
>>>>>  1 file changed, 490 insertions(+)
>>>>>  create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
>>>>>
>>>>>diff --git a/Documentation/gpu/rfc/i915_vm_bind.h 
>>>>>b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>new file mode 100644
>>>>>index 000000000000..9fc854969cfb
>>>>>--- /dev/null
>>>>>+++ b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>@@ -0,0 +1,490 @@
>>>>>+/* SPDX-License-Identifier: MIT */
>>>>>+/*
>>>>>+ * Copyright © 2022 Intel Corporation
>>>>>+ */
>>>>>+
>>>>>+/**
>>>>>+ * DOC: I915_PARAM_HAS_VM_BIND
>>>>>+ *
>>>>>+ * VM_BIND feature availability.
>>>>>+ * See typedef drm_i915_getparam_t param.
>>>>>+ * bit[0]: If set, VM_BIND is supported, otherwise not.
>>>>>+ * bits[8-15]: VM_BIND implementation version.
>>>>>+ * version 0 will not have VM_BIND/UNBIND timeline fence 
>>>>>array support.
>>>>>+ */
>>>>>+#define I915_PARAM_HAS_VM_BIND        57
>>>>>+
>>>>>+/**
>>>>>+ * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
>>>>>+ *
>>>>>+ * Flag to opt-in for VM_BIND mode of binding during VM creation.
>>>>>+ * See struct drm_i915_gem_vm_control flags.
>>>>>+ *
>>>>>+ * The older execbuf2 ioctl will not support VM_BIND mode of 
>>>>>operation.
>>>>>+ * For VM_BIND mode, we have new execbuf3 ioctl which will 
>>>>>not accept any
>>>>>+ * execlist (See struct drm_i915_gem_execbuffer3 for more details).
>>>>>+ *
>>>>>+ */
>>>>>+#define I915_VM_CREATE_FLAGS_USE_VM_BIND    (1 << 0)
>>>>>+
>>>>>+/**
>>>>>+ * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
>>>>>+ *
>>>>>+ * Flag to declare context as long running.
>>>>>+ * See struct drm_i915_gem_context_create_ext flags.
>>>>>+ *
>>>>>+ * Usage of dma-fence expects that they complete in 
>>>>>reasonable amount of time.
>>>>>+ * Compute on the other hand can be long running. Hence it is 
>>>>>not appropriate
>>>>>+ * for compute contexts to export request completion 
>>>>>dma-fence to user.
>>>>>+ * The dma-fence usage will be limited to in-kernel consumption only.
>>>>>+ * Compute contexts need to use user/memory fence.
>>>>>+ *
>>>>>+ * So, long running contexts do not support output fences. Hence,
>>>>>+ * I915_EXEC_FENCE_SIGNAL (See 
>>>>>&drm_i915_gem_exec_fence.flags) is expected
>>>>>+ * to be not used. DRM_I915_GEM_WAIT ioctl call is also not 
>>>>>supported for
>>>>>+ * objects mapped to long running contexts.
>>>>>+ */
>>>>>+#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u << 2)
>>>>>+
>>>>>+/* VM_BIND related ioctls */
>>>>>+#define DRM_I915_GEM_VM_BIND        0x3d
>>>>>+#define DRM_I915_GEM_VM_UNBIND        0x3e
>>>>>+#define DRM_I915_GEM_EXECBUFFER3    0x3f
>>>>>+#define DRM_I915_GEM_WAIT_USER_FENCE    0x40
>>>>>+
>>>>>+#define DRM_IOCTL_I915_GEM_VM_BIND DRM_IOWR(DRM_COMMAND_BASE 
>>>>>+ DRM_I915_GEM_VM_BIND, struct drm_i915_gem_vm_bind)
>>>>>+#define DRM_IOCTL_I915_GEM_VM_UNBIND 
>>>>>DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_VM_UNBIND, struct 
>>>>>drm_i915_gem_vm_bind)
>>>>>+#define DRM_IOCTL_I915_GEM_EXECBUFFER3 
>>>>>DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_EXECBUFFER3, struct 
>>>>>drm_i915_gem_execbuffer3)
>>>>>+#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE 
>>>>>DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_WAIT_USER_FENCE, 
>>>>>struct drm_i915_gem_wait_user_fence)
>>>>>+
>>>>>+/**
>>>>>+ * struct drm_i915_gem_vm_bind - VA to object mapping to bind.
>>>>>+ *
>>>>>+ * This structure is passed to VM_BIND ioctl and specifies 
>>>>>the mapping of GPU
>>>>>+ * virtual address (VA) range to the section of an object 
>>>>>that should be bound
>>>>>+ * in the device page table of the specified address space (VM).
>>>>>+ * The VA range specified must be unique (ie., not currently 
>>>>>bound) and can
>>>>>+ * be mapped to whole object or a section of the object 
>>>>>(partial binding).
>>>>>+ * Multiple VA mappings can be created to the same section of 
>>>>>the object
>>>>>+ * (aliasing).
>>>>>+ *
>>>>>+ * The @queue_idx specifies the queue to use for binding. 
>>>>>Same queue can be
>>>>>+ * used for both VM_BIND and VM_UNBIND calls. All submitted 
>>>>>bind and unbind
>>>>>+ * operations in a queue are performed in the order of submission.
>>>>>+ *
>>>>>+ * The @start, @offset and @length should be 4K page aligned. 
>>>>>However the DG2
>>>>>+ * and XEHPSDV has 64K page size for device local-memory and 
>>>>>has compact page
>>>>>+ * table. On those platforms, for binding device local-memory 
>>>>>objects, the
>>>>>+ * @start should be 2M aligned, @offset and @length should be 
>>>>>64K aligned.
>>>>>+ * Also, on those platforms, it is not allowed to bind an 
>>>>>device local-memory
>>>>>+ * object and a system memory object in a single 2M section 
>>>>>of VA range.
>>>>>+ */
>>>>>+struct drm_i915_gem_vm_bind {
>>>>>+    /** @vm_id: VM (address space) id to bind */
>>>>>+    __u32 vm_id;
>>>>>+
>>>>>+    /** @queue_idx: Index of queue for binding */
>>>>>+    __u32 queue_idx;
>>>>
>>>>I have a question here to which I did not find an answer by 
>>>>browsing the old threads.
>>>>
>>>>Queue index appears to be an implicit synchronisation mechanism, 
>>>>right? Operations on the same index are executed/complete in 
>>>>order of ioctl submission?
>>>>
>>>>Do we _have_ to implement this on the kernel side and could just 
>>>>allow in/out fence and let userspace deal with it?
>>>
>>>
>>>It orders operations like in a queue. Which is kind of what 
>>>happens with existing queues/engines.
>>>
>>>If I understood correctly, it's going to be a kthread + a linked 
>>>list right?
>>>
>>
>>Yes, that is correct.
>>
>>>
>>>-Lionel
>>>
>>>
>>>>
>>>>Arbitrary/on-demand number of queues will add the complexity on 
>>>>the kernel side which should be avoided if possible.
>>>>
>>
>>It was discussed in the other thread. Jason prefers this over putting
>>an artificial limit on number of queues (as user can anyway can exhaust
>>the memory). I think complexity in the driver is manageable.
>
>You'll need to create tracking structures on demand, with atomic 
>replace of last fence, ref counting and locking of some sort, more or 
>less?
>

We will have a workqueue, an work item and a linked list per queue.
VM_BIND/UNBIND call will add the mapping request to the specified queue's
linked list and schedule the work item on the workqueue of that queue.
I am not sure what you mean by last fence and replacing it.

>>The other option being discussed in to have the user create those
>>queues (like creating engine map) before hand and use that in vm_bind
>>and vm_unbind ioctls. This puts a limit on the number of queues.
>>But it is not clean either and not sure it is worth making the interface
>>more complex.
>>https://www.spinics.net/lists/dri-devel/msg350448.html
>
>What about the third option of a flag to return a fence (of some sort) 
>and pass in a fence? That way userspace can imagine zero or N queues 
>with very little effort on the kernel side. Was this considered?
>

I am not clear what fence you are talking about here and how does that
help with the number of vm_bind queues. Can you eloborate?

Niranjana

>Regards,
>
>Tvrtko
>
>>>>Regards,
>>>>
>>>>Tvrtko
>>>>
>>>>>+
>>>>>+    /** @rsvd: Reserved, MBZ */
>>>>>+    __u32 rsvd;
>>>>>+
>>>>>+    /** @handle: Object handle */
>>>>>+    __u32 handle;
>>>>>+
>>>>>+    /** @start: Virtual Address start to bind */
>>>>>+    __u64 start;
>>>>>+
>>>>>+    /** @offset: Offset in object to bind */
>>>>>+    __u64 offset;
>>>>>+
>>>>>+    /** @length: Length of mapping to bind */
>>>>>+    __u64 length;
>>>>>+
>>>>>+    /**
>>>>>+     * @flags: Supported flags are:
>>>>>+     *
>>>>>+     * I915_GEM_VM_BIND_READONLY:
>>>>>+     * Mapping is read-only.
>>>>>+     *
>>>>>+     * I915_GEM_VM_BIND_CAPTURE:
>>>>>+     * Capture this mapping in the dump upon GPU error.
>>>>>+     */
>>>>>+    __u64 flags;
>>>>>+#define I915_GEM_VM_BIND_READONLY    (1 << 0)
>>>>>+#define I915_GEM_VM_BIND_CAPTURE     (1 << 1)
>>>>>+
>>>>>+    /**
>>>>>+     * @extensions: 0-terminated chain of extensions for this 
>>>>>operation.
>>>>>+     *
>>>>>+     * I915_VM_BIND_EXT_TIMELINE_FENCES:
>>>>>+     * Specifies an array of input or output timeline fences for this
>>>>>+     * binding operation. See struct 
>>>>>drm_i915_vm_bind_ext_timeline_fences.
>>>>>+     *
>>>>>+     * I915_VM_BIND_EXT_USER_FENCES:
>>>>>+     * Specifies an array of input or output user fences for this
>>>>>+     * binding operation. See struct drm_i915_vm_bind_ext_user_fence.
>>>>>+     * This is required for compute contexts.
>>>>>+     */
>>>>>+    __u64 extensions;
>>>>>+#define I915_VM_BIND_EXT_TIMELINE_FENCES    0
>>>>>+#define I915_VM_BIND_EXT_USER_FENCES        1
>>>>>+};
>>>>>+
>>>>>+/**
>>>>>+ * struct drm_i915_gem_vm_unbind - VA to object mapping to unbind.
>>>>>+ *
>>>>>+ * This structure is passed to VM_UNBIND ioctl and specifies 
>>>>>the GPU virtual
>>>>>+ * address (VA) range that should be unbound from the device 
>>>>>page table of the
>>>>>+ * specified address space (VM). The specified VA range must 
>>>>>match one of the
>>>>>+ * mappings created with the VM_BIND ioctl. TLB is flushed upon unbind
>>>>>+ * completion.
>>>>>+ *
>>>>>+ * The @queue_idx specifies the queue to use for unbinding.
>>>>>+ * See struct drm_i915_gem_vm_unbind for more information on 
>>>>>@queue_idx.
>>>>>+ *
>>>>>+ * The @start and @length musy specify a unique mapping bound 
>>>>>with VM_BIND
>>>>>+ * ioctl.
>>>>>+ */
>>>>>+struct drm_i915_gem_vm_unbind {
>>>>>+    /** @vm_id: VM (address space) id to bind */
>>>>>+    __u32 vm_id;
>>>>>+
>>>>>+    /** @queue_idx: Index of queue for unbinding */
>>>>>+    __u32 queue_idx;
>>>>>+
>>>>>+    /** @start: Virtual Address start to unbind */
>>>>>+    __u64 start;
>>>>>+
>>>>>+    /** @length: Length of mapping to unbind */
>>>>>+    __u64 length;
>>>>>+
>>>>>+    /** @flags: Reserved for future usage, currently MBZ */
>>>>>+    __u64 flags;
>>>>>+
>>>>>+    /**
>>>>>+     * @extensions: 0-terminated chain of extensions for this 
>>>>>operation.
>>>>>+     *
>>>>>+     * I915_VM_UNBIND_EXT_TIMELINE_FENCES:
>>>>>+     * Specifies an array of input or output timeline fences for this
>>>>>+     * unbind operation.
>>>>>+     * It has same format as struct 
>>>>>drm_i915_vm_bind_ext_timeline_fences.
>>>>>+     *
>>>>>+     * I915_VM_UNBIND_EXT_USER_FENCES:
>>>>>+     * Specifies an array of input or output user fences for this
>>>>>+     * unbind operation. This is required for compute contexts.
>>>>>+     * It has same format as struct drm_i915_vm_bind_ext_user_fence.
>>>>>+     */
>>>>>+    __u64 extensions;
>>>>>+#define I915_VM_UNBIND_EXT_TIMELINE_FENCES    0
>>>>>+#define I915_VM_UNBIND_EXT_USER_FENCES        1
>>>>>+};
>>>>>+
>>>>>+/**
>>>>>+ * struct drm_i915_vm_bind_fence - An input or output fence 
>>>>>for the vm_bind
>>>>>+ * or the vm_unbind work.
>>>>>+ *
>>>>>+ * The vm_bind or vm_unbind aync worker will wait for input 
>>>>>fence to signal
>>>>>+ * before starting the binding or unbinding.
>>>>>+ *
>>>>>+ * The vm_bind or vm_unbind async worker will signal the 
>>>>>returned output fence
>>>>>+ * after the completion of binding or unbinding.
>>>>>+ */
>>>>>+struct drm_i915_vm_bind_fence {
>>>>>+    /** @handle: User's handle for a drm_syncobj to wait on 
>>>>>or signal. */
>>>>>+    __u32 handle;
>>>>>+
>>>>>+    /**
>>>>>+     * @flags: Supported flags are:
>>>>>+     *
>>>>>+     * I915_VM_BIND_FENCE_WAIT:
>>>>>+     * Wait for the input fence before binding/unbinding
>>>>>+     *
>>>>>+     * I915_VM_BIND_FENCE_SIGNAL:
>>>>>+     * Return bind/unbind completion fence as output
>>>>>+     */
>>>>>+    __u32 flags;
>>>>>+#define I915_VM_BIND_FENCE_WAIT            (1<<0)
>>>>>+#define I915_VM_BIND_FENCE_SIGNAL          (1<<1)
>>>>>+#define __I915_VM_BIND_FENCE_UNKNOWN_FLAGS 
>>>>>(-(I915_VM_BIND_FENCE_SIGNAL << 1))
>>>>>+};
>>>>>+
>>>>>+/**
>>>>>+ * struct drm_i915_vm_bind_ext_timeline_fences - Timeline 
>>>>>fences for vm_bind
>>>>>+ * and vm_unbind.
>>>>>+ *
>>>>>+ * This structure describes an array of timeline drm_syncobj 
>>>>>and associated
>>>>>+ * points for timeline variants of drm_syncobj. These 
>>>>>timeline 'drm_syncobj's
>>>>>+ * can be input or output fences (See struct drm_i915_vm_bind_fence).
>>>>>+ */
>>>>>+struct drm_i915_vm_bind_ext_timeline_fences {
>>>>>+    /** @base: Extension link. See struct i915_user_extension. */
>>>>>+    struct i915_user_extension base;
>>>>>+
>>>>>+    /**
>>>>>+     * @fence_count: Number of elements in the @handles_ptr & 
>>>>>@value_ptr
>>>>>+     * arrays.
>>>>>+     */
>>>>>+    __u64 fence_count;
>>>>>+
>>>>>+    /**
>>>>>+     * @handles_ptr: Pointer to an array of struct 
>>>>>drm_i915_vm_bind_fence
>>>>>+     * of length @fence_count.
>>>>>+     */
>>>>>+    __u64 handles_ptr;
>>>>>+
>>>>>+    /**
>>>>>+     * @values_ptr: Pointer to an array of u64 values of length
>>>>>+     * @fence_count.
>>>>>+     * Values must be 0 for a binary drm_syncobj. A Value of 0 for a
>>>>>+     * timeline drm_syncobj is invalid as it turns a 
>>>>>drm_syncobj into a
>>>>>+     * binary one.
>>>>>+     */
>>>>>+    __u64 values_ptr;
>>>>>+};
>>>>>+
>>>>>+/**
>>>>>+ * struct drm_i915_vm_bind_user_fence - An input or output 
>>>>>user fence for the
>>>>>+ * vm_bind or the vm_unbind work.
>>>>>+ *
>>>>>+ * The vm_bind or vm_unbind aync worker will wait for the 
>>>>>input fence (value at
>>>>>+ * @addr to become equal to @val) before starting the binding 
>>>>>or unbinding.
>>>>>+ *
>>>>>+ * The vm_bind or vm_unbind async worker will signal the 
>>>>>output fence after
>>>>>+ * the completion of binding or unbinding by writing @val to 
>>>>>memory location at
>>>>>+ * @addr
>>>>>+ */
>>>>>+struct drm_i915_vm_bind_user_fence {
>>>>>+    /** @addr: User/Memory fence qword aligned process 
>>>>>virtual address */
>>>>>+    __u64 addr;
>>>>>+
>>>>>+    /** @val: User/Memory fence value to be written after 
>>>>>bind completion */
>>>>>+    __u64 val;
>>>>>+
>>>>>+    /**
>>>>>+     * @flags: Supported flags are:
>>>>>+     *
>>>>>+     * I915_VM_BIND_USER_FENCE_WAIT:
>>>>>+     * Wait for the input fence before binding/unbinding
>>>>>+     *
>>>>>+     * I915_VM_BIND_USER_FENCE_SIGNAL:
>>>>>+     * Return bind/unbind completion fence as output
>>>>>+     */
>>>>>+    __u32 flags;
>>>>>+#define I915_VM_BIND_USER_FENCE_WAIT            (1<<0)
>>>>>+#define I915_VM_BIND_USER_FENCE_SIGNAL          (1<<1)
>>>>>+#define __I915_VM_BIND_USER_FENCE_UNKNOWN_FLAGS \
>>>>>+    (-(I915_VM_BIND_USER_FENCE_SIGNAL << 1))
>>>>>+};
>>>>>+
>>>>>+/**
>>>>>+ * struct drm_i915_vm_bind_ext_user_fence - User/memory 
>>>>>fences for vm_bind
>>>>>+ * and vm_unbind.
>>>>>+ *
>>>>>+ * These user fences can be input or output fences
>>>>>+ * (See struct drm_i915_vm_bind_user_fence).
>>>>>+ */
>>>>>+struct drm_i915_vm_bind_ext_user_fence {
>>>>>+    /** @base: Extension link. See struct i915_user_extension. */
>>>>>+    struct i915_user_extension base;
>>>>>+
>>>>>+    /** @fence_count: Number of elements in the 
>>>>>@user_fence_ptr array. */
>>>>>+    __u64 fence_count;
>>>>>+
>>>>>+    /**
>>>>>+     * @user_fence_ptr: Pointer to an array of
>>>>>+     * struct drm_i915_vm_bind_user_fence of length @fence_count.
>>>>>+     */
>>>>>+    __u64 user_fence_ptr;
>>>>>+};
>>>>>+
>>>>>+/**
>>>>>+ * struct drm_i915_gem_execbuffer3 - Structure for 
>>>>>DRM_I915_GEM_EXECBUFFER3
>>>>>+ * ioctl.
>>>>>+ *
>>>>>+ * DRM_I915_GEM_EXECBUFFER3 ioctl only works in VM_BIND mode 
>>>>>and VM_BIND mode
>>>>>+ * only works with this ioctl for submission.
>>>>>+ * See I915_VM_CREATE_FLAGS_USE_VM_BIND.
>>>>>+ */
>>>>>+struct drm_i915_gem_execbuffer3 {
>>>>>+    /**
>>>>>+     * @ctx_id: Context id
>>>>>+     *
>>>>>+     * Only contexts with user engine map are allowed.
>>>>>+     */
>>>>>+    __u32 ctx_id;
>>>>>+
>>>>>+    /**
>>>>>+     * @engine_idx: Engine index
>>>>>+     *
>>>>>+     * An index in the user engine map of the context 
>>>>>specified by @ctx_id.
>>>>>+     */
>>>>>+    __u32 engine_idx;
>>>>>+
>>>>>+    /** @rsvd1: Reserved, MBZ */
>>>>>+    __u32 rsvd1;
>>>>>+
>>>>>+    /**
>>>>>+     * @batch_count: Number of batches in @batch_address array.
>>>>>+     *
>>>>>+     * 0 is invalid. For parallel submission, it should be 
>>>>>equal to the
>>>>>+     * number of (parallel) engines involved in that submission.
>>>>>+     */
>>>>>+    __u32 batch_count;
>>>>>+
>>>>>+    /**
>>>>>+     * @batch_address: Array of batch gpu virtual addresses.
>>>>>+     *
>>>>>+     * If @batch_count is 1, then it is the gpu virtual address of the
>>>>>+     * batch buffer. If @batch_count > 1, then it is a 
>>>>>pointer to an array
>>>>>+     * of batch buffer gpu virtual addresses.
>>>>>+     */
>>>>>+    __u64 batch_address;
>>>>>+
>>>>>+    /**
>>>>>+     * @flags: Supported flags are:
>>>>>+     *
>>>>>+     * I915_EXEC3_SECURE:
>>>>>+     * Request a privileged ("secure") batch buffer/s.
>>>>>+     * It is only available for DRM_ROOT_ONLY | DRM_MASTER processes.
>>>>>+     */
>>>>>+    __u64 flags;
>>>>>+#define I915_EXEC3_SECURE    (1<<0)
>>>>>+
>>>>>+    /** @rsvd2: Reserved, MBZ */
>>>>>+    __u64 rsvd2;
>>>>>+
>>>>>+    /**
>>>>>+     * @extensions: Zero-terminated chain of extensions.
>>>>>+     *
>>>>>+     * DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES:
>>>>>+     * It has same format as 
>>>>>DRM_I915_GEM_EXECBUFFER_EXT_TIMELINE_FENCES.
>>>>>+     * See struct drm_i915_gem_execbuffer_ext_timeline_fences.
>>>>>+     *
>>>>>+     * DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE:
>>>>>+     * First level batch completion signaling extension.
>>>>>+     * See struct drm_i915_gem_execbuffer3_ext_user_fence.
>>>>>+     */
>>>>>+    __u64 extensions;
>>>>>+#define DRM_I915_GEM_EXECBUFFER3_EXT_TIMELINE_FENCES    0
>>>>>+#define DRM_I915_GEM_EXECBUFFER3_EXT_USER_FENCE        1
>>>>>+};
>>>>>+
>>>>>+/**
>>>>>+ * struct drm_i915_gem_execbuffer3_ext_user_fence - First 
>>>>>level batch completion
>>>>>+ * signaling extension.
>>>>>+ *
>>>>>+ * This extension allows user to attach a user fence (@addr, 
>>>>>@value pair) to
>>>>>+ * execbuf3, to be signaled by the command streamer after the 
>>>>>completion of first
>>>>>+ * level batch, by writing the @value at specified @addr and 
>>>>>triggering an
>>>>>+ * interrupt.
>>>>>+ * User can either poll for this user fence to signal or can 
>>>>>also wait on it
>>>>>+ * with i915_gem_wait_user_fence ioctl.
>>>>>+ * This is very much usefaul for long running contexts where 
>>>>>waiting on dma-fence
>>>>>+ * by user (like i915_gem_wait ioctl) is not supported.
>>>>>+ */
>>>>>+struct drm_i915_gem_execbuffer3_ext_user_fence {
>>>>>+    /** @base: Extension link. See struct i915_user_extension. */
>>>>>+    struct i915_user_extension base;
>>>>>+
>>>>>+    /**
>>>>>+     * @addr: User/Memory fence qword aligned GPU virtual address.
>>>>>+     *
>>>>>+     * Address has to be a valid GPU virtual address at the time of
>>>>>+     * first level batch completion.
>>>>>+     */
>>>>>+    __u64 addr;
>>>>>+
>>>>>+    /**
>>>>>+     * @value: User/Memory fence Value to be written to above address
>>>>>+     * after first level batch completes.
>>>>>+     */
>>>>>+    __u64 value;
>>>>>+
>>>>>+    /** @rsvd: Reserved, MBZ */
>>>>>+    __u64 rsvd;
>>>>>+};
>>>>>+
>>>>>+/**
>>>>>+ * struct drm_i915_gem_create_ext_vm_private - Extension to 
>>>>>make the object
>>>>>+ * private to the specified VM.
>>>>>+ *
>>>>>+ * See struct drm_i915_gem_create_ext.
>>>>>+ */
>>>>>+struct drm_i915_gem_create_ext_vm_private {
>>>>>+#define I915_GEM_CREATE_EXT_VM_PRIVATE        2
>>>>>+    /** @base: Extension link. See struct i915_user_extension. */
>>>>>+    struct i915_user_extension base;
>>>>>+
>>>>>+    /** @vm_id: Id of the VM to which the object is private */
>>>>>+    __u32 vm_id;
>>>>>+};
>>>>>+
>>>>>+/**
>>>>>+ * struct drm_i915_gem_wait_user_fence - Wait on user/memory fence.
>>>>>+ *
>>>>>+ * User/Memory fence can be woken up either by:
>>>>>+ *
>>>>>+ * 1. GPU context indicated by @ctx_id, or,
>>>>>+ * 2. Kerrnel driver async worker upon I915_UFENCE_WAIT_SOFT.
>>>>>+ *    @ctx_id is ignored when this flag is set.
>>>>>+ *
>>>>>+ * Wakeup condition is,
>>>>>+ * ``((*addr & mask) op (value & mask))``
>>>>>+ *
>>>>>+ * See :ref:`Documentation/driver-api/dma-buf.rst 
>>>>><indefinite_dma_fences>`
>>>>>+ */
>>>>>+struct drm_i915_gem_wait_user_fence {
>>>>>+    /** @extensions: Zero-terminated chain of extensions. */
>>>>>+    __u64 extensions;
>>>>>+
>>>>>+    /** @addr: User/Memory fence address */
>>>>>+    __u64 addr;
>>>>>+
>>>>>+    /** @ctx_id: Id of the Context which will signal the fence. */
>>>>>+    __u32 ctx_id;
>>>>>+
>>>>>+    /** @op: Wakeup condition operator */
>>>>>+    __u16 op;
>>>>>+#define I915_UFENCE_WAIT_EQ      0
>>>>>+#define I915_UFENCE_WAIT_NEQ     1
>>>>>+#define I915_UFENCE_WAIT_GT      2
>>>>>+#define I915_UFENCE_WAIT_GTE     3
>>>>>+#define I915_UFENCE_WAIT_LT      4
>>>>>+#define I915_UFENCE_WAIT_LTE     5
>>>>>+#define I915_UFENCE_WAIT_BEFORE  6
>>>>>+#define I915_UFENCE_WAIT_AFTER   7
>>>>>+
>>>>>+    /**
>>>>>+     * @flags: Supported flags are:
>>>>>+     *
>>>>>+     * I915_UFENCE_WAIT_SOFT:
>>>>>+     *
>>>>>+     * To be woken up by i915 driver async worker (not by GPU).
>>>>>+     *
>>>>>+     * I915_UFENCE_WAIT_ABSTIME:
>>>>>+     *
>>>>>+     * Wait timeout specified as absolute time.
>>>>>+     */
>>>>>+    __u16 flags;
>>>>>+#define I915_UFENCE_WAIT_SOFT    0x1
>>>>>+#define I915_UFENCE_WAIT_ABSTIME 0x2
>>>>>+
>>>>>+    /** @value: Wakeup value */
>>>>>+    __u64 value;
>>>>>+
>>>>>+    /** @mask: Wakeup mask */
>>>>>+    __u64 mask;
>>>>>+#define I915_UFENCE_WAIT_U8     0xffu
>>>>>+#define I915_UFENCE_WAIT_U16    0xffffu
>>>>>+#define I915_UFENCE_WAIT_U32    0xfffffffful
>>>>>+#define I915_UFENCE_WAIT_U64    0xffffffffffffffffull
>>>>>+
>>>>>+    /**
>>>>>+     * @timeout: Wait timeout in nanoseconds.
>>>>>+     *
>>>>>+     * If I915_UFENCE_WAIT_ABSTIME flag is set, then time 
>>>>>timeout is the
>>>>>+     * absolute time in nsec.
>>>>>+     */
>>>>>+    __s64 timeout;
>>>>>+};
>>>
>>>

Re: [Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Tvrtko Ursulin]

On 13/06/2022 16:05, Niranjana Vishwanathapura wrote:
> On Mon, Jun 13, 2022 at 09:24:18AM +0100, Tvrtko Ursulin wrote:
>>
>> On 10/06/2022 17:14, Niranjana Vishwanathapura wrote:
>>> On Fri, Jun 10, 2022 at 05:48:39PM +0300, Lionel Landwerlin wrote:
>>>> On 10/06/2022 13:37, Tvrtko Ursulin wrote:
>>>>>
>>>>> On 10/06/2022 08:07, Niranjana Vishwanathapura wrote:
>>>>>> VM_BIND and related uapi definitions
>>>>>>
>>>>>> Signed-off-by: Niranjana Vishwanathapura 
>>>>>> <niranjana.vishwanathapura@intel.com>
>>>>>> ---
>>>>>>   Documentation/gpu/rfc/i915_vm_bind.h | 490 
>>>>>> +++++++++++++++++++++++++++
>>>>>>   1 file changed, 490 insertions(+)
>>>>>>   create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>
>>>>>> diff --git a/Documentation/gpu/rfc/i915_vm_bind.h 
>>>>>> b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>> new file mode 100644
>>>>>> index 000000000000..9fc854969cfb
>>>>>> --- /dev/null
>>>>>> +++ b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>> @@ -0,0 +1,490 @@
>>>>>> +/* SPDX-License-Identifier: MIT */
>>>>>> +/*
>>>>>> + * Copyright © 2022 Intel Corporation
>>>>>> + */
>>>>>> +
>>>>>> +/**
>>>>>> + * DOC: I915_PARAM_HAS_VM_BIND
>>>>>> + *
>>>>>> + * VM_BIND feature availability.
>>>>>> + * See typedef drm_i915_getparam_t param.
>>>>>> + * bit[0]: If set, VM_BIND is supported, otherwise not.
>>>>>> + * bits[8-15]: VM_BIND implementation version.
>>>>>> + * version 0 will not have VM_BIND/UNBIND timeline fence array 
>>>>>> support.
>>>>>> + */
>>>>>> +#define I915_PARAM_HAS_VM_BIND        57
>>>>>> +
>>>>>> +/**
>>>>>> + * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
>>>>>> + *
>>>>>> + * Flag to opt-in for VM_BIND mode of binding during VM creation.
>>>>>> + * See struct drm_i915_gem_vm_control flags.
>>>>>> + *
>>>>>> + * The older execbuf2 ioctl will not support VM_BIND mode of 
>>>>>> operation.
>>>>>> + * For VM_BIND mode, we have new execbuf3 ioctl which will not 
>>>>>> accept any
>>>>>> + * execlist (See struct drm_i915_gem_execbuffer3 for more details).
>>>>>> + *
>>>>>> + */
>>>>>> +#define I915_VM_CREATE_FLAGS_USE_VM_BIND    (1 << 0)
>>>>>> +
>>>>>> +/**
>>>>>> + * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
>>>>>> + *
>>>>>> + * Flag to declare context as long running.
>>>>>> + * See struct drm_i915_gem_context_create_ext flags.
>>>>>> + *
>>>>>> + * Usage of dma-fence expects that they complete in reasonable 
>>>>>> amount of time.
>>>>>> + * Compute on the other hand can be long running. Hence it is not 
>>>>>> appropriate
>>>>>> + * for compute contexts to export request completion dma-fence to 
>>>>>> user.
>>>>>> + * The dma-fence usage will be limited to in-kernel consumption 
>>>>>> only.
>>>>>> + * Compute contexts need to use user/memory fence.
>>>>>> + *
>>>>>> + * So, long running contexts do not support output fences. Hence,
>>>>>> + * I915_EXEC_FENCE_SIGNAL (See &drm_i915_gem_exec_fence.flags) is 
>>>>>> expected
>>>>>> + * to be not used. DRM_I915_GEM_WAIT ioctl call is also not 
>>>>>> supported for
>>>>>> + * objects mapped to long running contexts.
>>>>>> + */
>>>>>> +#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u << 2)
>>>>>> +
>>>>>> +/* VM_BIND related ioctls */
>>>>>> +#define DRM_I915_GEM_VM_BIND        0x3d
>>>>>> +#define DRM_I915_GEM_VM_UNBIND        0x3e
>>>>>> +#define DRM_I915_GEM_EXECBUFFER3    0x3f
>>>>>> +#define DRM_I915_GEM_WAIT_USER_FENCE    0x40
>>>>>> +
>>>>>> +#define DRM_IOCTL_I915_GEM_VM_BIND DRM_IOWR(DRM_COMMAND_BASE + 
>>>>>> DRM_I915_GEM_VM_BIND, struct drm_i915_gem_vm_bind)
>>>>>> +#define DRM_IOCTL_I915_GEM_VM_UNBIND DRM_IOWR(DRM_COMMAND_BASE + 
>>>>>> DRM_I915_GEM_VM_UNBIND, struct drm_i915_gem_vm_bind)
>>>>>> +#define DRM_IOCTL_I915_GEM_EXECBUFFER3 DRM_IOWR(DRM_COMMAND_BASE 
>>>>>> + DRM_I915_GEM_EXECBUFFER3, struct drm_i915_gem_execbuffer3)
>>>>>> +#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE 
>>>>>> DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_WAIT_USER_FENCE, struct 
>>>>>> drm_i915_gem_wait_user_fence)
>>>>>> +
>>>>>> +/**
>>>>>> + * struct drm_i915_gem_vm_bind - VA to object mapping to bind.
>>>>>> + *
>>>>>> + * This structure is passed to VM_BIND ioctl and specifies the 
>>>>>> mapping of GPU
>>>>>> + * virtual address (VA) range to the section of an object that 
>>>>>> should be bound
>>>>>> + * in the device page table of the specified address space (VM).
>>>>>> + * The VA range specified must be unique (ie., not currently 
>>>>>> bound) and can
>>>>>> + * be mapped to whole object or a section of the object (partial 
>>>>>> binding).
>>>>>> + * Multiple VA mappings can be created to the same section of the 
>>>>>> object
>>>>>> + * (aliasing).
>>>>>> + *
>>>>>> + * The @queue_idx specifies the queue to use for binding. Same 
>>>>>> queue can be
>>>>>> + * used for both VM_BIND and VM_UNBIND calls. All submitted bind 
>>>>>> and unbind
>>>>>> + * operations in a queue are performed in the order of submission.
>>>>>> + *
>>>>>> + * The @start, @offset and @length should be 4K page aligned. 
>>>>>> However the DG2
>>>>>> + * and XEHPSDV has 64K page size for device local-memory and has 
>>>>>> compact page
>>>>>> + * table. On those platforms, for binding device local-memory 
>>>>>> objects, the
>>>>>> + * @start should be 2M aligned, @offset and @length should be 64K 
>>>>>> aligned.
>>>>>> + * Also, on those platforms, it is not allowed to bind an device 
>>>>>> local-memory
>>>>>> + * object and a system memory object in a single 2M section of VA 
>>>>>> range.
>>>>>> + */
>>>>>> +struct drm_i915_gem_vm_bind {
>>>>>> +    /** @vm_id: VM (address space) id to bind */
>>>>>> +    __u32 vm_id;
>>>>>> +
>>>>>> +    /** @queue_idx: Index of queue for binding */
>>>>>> +    __u32 queue_idx;
>>>>>
>>>>> I have a question here to which I did not find an answer by 
>>>>> browsing the old threads.
>>>>>
>>>>> Queue index appears to be an implicit synchronisation mechanism, 
>>>>> right? Operations on the same index are executed/complete in order 
>>>>> of ioctl submission?
>>>>>
>>>>> Do we _have_ to implement this on the kernel side and could just 
>>>>> allow in/out fence and let userspace deal with it?
>>>>
>>>>
>>>> It orders operations like in a queue. Which is kind of what happens 
>>>> with existing queues/engines.
>>>>
>>>> If I understood correctly, it's going to be a kthread + a linked 
>>>> list right?
>>>>
>>>
>>> Yes, that is correct.
>>>
>>>>
>>>> -Lionel
>>>>
>>>>
>>>>>
>>>>> Arbitrary/on-demand number of queues will add the complexity on the 
>>>>> kernel side which should be avoided if possible.
>>>>>
>>>
>>> It was discussed in the other thread. Jason prefers this over putting
>>> an artificial limit on number of queues (as user can anyway can exhaust
>>> the memory). I think complexity in the driver is manageable.
>>
>> You'll need to create tracking structures on demand, with atomic 
>> replace of last fence, ref counting and locking of some sort, more or 
>> less?
>>
> 
> We will have a workqueue, an work item and a linked list per queue.
> VM_BIND/UNBIND call will add the mapping request to the specified queue's
> linked list and schedule the work item on the workqueue of that queue.
> I am not sure what you mean by last fence and replacing it.
> 
>>> The other option being discussed in to have the user create those
>>> queues (like creating engine map) before hand and use that in vm_bind
>>> and vm_unbind ioctls. This puts a limit on the number of queues.
>>> But it is not clean either and not sure it is worth making the interface
>>> more complex.
>>> https://www.spinics.net/lists/dri-devel/msg350448.html
>>
>> What about the third option of a flag to return a fence (of some sort) 
>> and pass in a fence? That way userspace can imagine zero or N queues 
>> with very little effort on the kernel side. Was this considered?
>>
> 
> I am not clear what fence you are talking about here and how does that
> help with the number of vm_bind queues. Can you eloborate?

It is actually already documented that bind/unbind will support input 
and output fences - so what are these queues on top of what userspace 
can already achieve by using them? Purely a convenience or there is more 
to it?

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Niranjana Vishwanathapura]

On Mon, Jun 13, 2022 at 05:22:02PM +0100, Tvrtko Ursulin wrote:
>
>On 13/06/2022 16:05, Niranjana Vishwanathapura wrote:
>>On Mon, Jun 13, 2022 at 09:24:18AM +0100, Tvrtko Ursulin wrote:
>>>
>>>On 10/06/2022 17:14, Niranjana Vishwanathapura wrote:
>>>>On Fri, Jun 10, 2022 at 05:48:39PM +0300, Lionel Landwerlin wrote:
>>>>>On 10/06/2022 13:37, Tvrtko Ursulin wrote:
>>>>>>
>>>>>>On 10/06/2022 08:07, Niranjana Vishwanathapura wrote:
>>>>>>>VM_BIND and related uapi definitions
>>>>>>>
>>>>>>>Signed-off-by: Niranjana Vishwanathapura 
>>>>>>><niranjana.vishwanathapura@intel.com>
>>>>>>>---
>>>>>>>  Documentation/gpu/rfc/i915_vm_bind.h | 490 
>>>>>>>+++++++++++++++++++++++++++
>>>>>>>  1 file changed, 490 insertions(+)
>>>>>>>  create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>
>>>>>>>diff --git a/Documentation/gpu/rfc/i915_vm_bind.h 
>>>>>>>b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>new file mode 100644
>>>>>>>index 000000000000..9fc854969cfb
>>>>>>>--- /dev/null
>>>>>>>+++ b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>@@ -0,0 +1,490 @@
>>>>>>>+/* SPDX-License-Identifier: MIT */
>>>>>>>+/*
>>>>>>>+ * Copyright © 2022 Intel Corporation
>>>>>>>+ */
>>>>>>>+
>>>>>>>+/**
>>>>>>>+ * DOC: I915_PARAM_HAS_VM_BIND
>>>>>>>+ *
>>>>>>>+ * VM_BIND feature availability.
>>>>>>>+ * See typedef drm_i915_getparam_t param.
>>>>>>>+ * bit[0]: If set, VM_BIND is supported, otherwise not.
>>>>>>>+ * bits[8-15]: VM_BIND implementation version.
>>>>>>>+ * version 0 will not have VM_BIND/UNBIND timeline fence 
>>>>>>>array support.
>>>>>>>+ */
>>>>>>>+#define I915_PARAM_HAS_VM_BIND        57
>>>>>>>+
>>>>>>>+/**
>>>>>>>+ * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
>>>>>>>+ *
>>>>>>>+ * Flag to opt-in for VM_BIND mode of binding during VM creation.
>>>>>>>+ * See struct drm_i915_gem_vm_control flags.
>>>>>>>+ *
>>>>>>>+ * The older execbuf2 ioctl will not support VM_BIND mode 
>>>>>>>of operation.
>>>>>>>+ * For VM_BIND mode, we have new execbuf3 ioctl which 
>>>>>>>will not accept any
>>>>>>>+ * execlist (See struct drm_i915_gem_execbuffer3 for more details).
>>>>>>>+ *
>>>>>>>+ */
>>>>>>>+#define I915_VM_CREATE_FLAGS_USE_VM_BIND    (1 << 0)
>>>>>>>+
>>>>>>>+/**
>>>>>>>+ * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
>>>>>>>+ *
>>>>>>>+ * Flag to declare context as long running.
>>>>>>>+ * See struct drm_i915_gem_context_create_ext flags.
>>>>>>>+ *
>>>>>>>+ * Usage of dma-fence expects that they complete in 
>>>>>>>reasonable amount of time.
>>>>>>>+ * Compute on the other hand can be long running. Hence 
>>>>>>>it is not appropriate
>>>>>>>+ * for compute contexts to export request completion 
>>>>>>>dma-fence to user.
>>>>>>>+ * The dma-fence usage will be limited to in-kernel 
>>>>>>>consumption only.
>>>>>>>+ * Compute contexts need to use user/memory fence.
>>>>>>>+ *
>>>>>>>+ * So, long running contexts do not support output fences. Hence,
>>>>>>>+ * I915_EXEC_FENCE_SIGNAL (See 
>>>>>>>&drm_i915_gem_exec_fence.flags) is expected
>>>>>>>+ * to be not used. DRM_I915_GEM_WAIT ioctl call is also 
>>>>>>>not supported for
>>>>>>>+ * objects mapped to long running contexts.
>>>>>>>+ */
>>>>>>>+#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u << 2)
>>>>>>>+
>>>>>>>+/* VM_BIND related ioctls */
>>>>>>>+#define DRM_I915_GEM_VM_BIND        0x3d
>>>>>>>+#define DRM_I915_GEM_VM_UNBIND        0x3e
>>>>>>>+#define DRM_I915_GEM_EXECBUFFER3    0x3f
>>>>>>>+#define DRM_I915_GEM_WAIT_USER_FENCE    0x40
>>>>>>>+
>>>>>>>+#define DRM_IOCTL_I915_GEM_VM_BIND 
>>>>>>>DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_VM_BIND, struct 
>>>>>>>drm_i915_gem_vm_bind)
>>>>>>>+#define DRM_IOCTL_I915_GEM_VM_UNBIND 
>>>>>>>DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_VM_UNBIND, struct 
>>>>>>>drm_i915_gem_vm_bind)
>>>>>>>+#define DRM_IOCTL_I915_GEM_EXECBUFFER3 
>>>>>>>DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_EXECBUFFER3, 
>>>>>>>struct drm_i915_gem_execbuffer3)
>>>>>>>+#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE 
>>>>>>>DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_WAIT_USER_FENCE, 
>>>>>>>struct drm_i915_gem_wait_user_fence)
>>>>>>>+
>>>>>>>+/**
>>>>>>>+ * struct drm_i915_gem_vm_bind - VA to object mapping to bind.
>>>>>>>+ *
>>>>>>>+ * This structure is passed to VM_BIND ioctl and 
>>>>>>>specifies the mapping of GPU
>>>>>>>+ * virtual address (VA) range to the section of an object 
>>>>>>>that should be bound
>>>>>>>+ * in the device page table of the specified address space (VM).
>>>>>>>+ * The VA range specified must be unique (ie., not 
>>>>>>>currently bound) and can
>>>>>>>+ * be mapped to whole object or a section of the object 
>>>>>>>(partial binding).
>>>>>>>+ * Multiple VA mappings can be created to the same 
>>>>>>>section of the object
>>>>>>>+ * (aliasing).
>>>>>>>+ *
>>>>>>>+ * The @queue_idx specifies the queue to use for binding. 
>>>>>>>Same queue can be
>>>>>>>+ * used for both VM_BIND and VM_UNBIND calls. All 
>>>>>>>submitted bind and unbind
>>>>>>>+ * operations in a queue are performed in the order of submission.
>>>>>>>+ *
>>>>>>>+ * The @start, @offset and @length should be 4K page 
>>>>>>>aligned. However the DG2
>>>>>>>+ * and XEHPSDV has 64K page size for device local-memory 
>>>>>>>and has compact page
>>>>>>>+ * table. On those platforms, for binding device 
>>>>>>>local-memory objects, the
>>>>>>>+ * @start should be 2M aligned, @offset and @length 
>>>>>>>should be 64K aligned.
>>>>>>>+ * Also, on those platforms, it is not allowed to bind an 
>>>>>>>device local-memory
>>>>>>>+ * object and a system memory object in a single 2M 
>>>>>>>section of VA range.
>>>>>>>+ */
>>>>>>>+struct drm_i915_gem_vm_bind {
>>>>>>>+    /** @vm_id: VM (address space) id to bind */
>>>>>>>+    __u32 vm_id;
>>>>>>>+
>>>>>>>+    /** @queue_idx: Index of queue for binding */
>>>>>>>+    __u32 queue_idx;
>>>>>>
>>>>>>I have a question here to which I did not find an answer by 
>>>>>>browsing the old threads.
>>>>>>
>>>>>>Queue index appears to be an implicit synchronisation 
>>>>>>mechanism, right? Operations on the same index are 
>>>>>>executed/complete in order of ioctl submission?
>>>>>>
>>>>>>Do we _have_ to implement this on the kernel side and could 
>>>>>>just allow in/out fence and let userspace deal with it?
>>>>>
>>>>>
>>>>>It orders operations like in a queue. Which is kind of what 
>>>>>happens with existing queues/engines.
>>>>>
>>>>>If I understood correctly, it's going to be a kthread + a 
>>>>>linked list right?
>>>>>
>>>>
>>>>Yes, that is correct.
>>>>
>>>>>
>>>>>-Lionel
>>>>>
>>>>>
>>>>>>
>>>>>>Arbitrary/on-demand number of queues will add the complexity 
>>>>>>on the kernel side which should be avoided if possible.
>>>>>>
>>>>
>>>>It was discussed in the other thread. Jason prefers this over putting
>>>>an artificial limit on number of queues (as user can anyway can exhaust
>>>>the memory). I think complexity in the driver is manageable.
>>>
>>>You'll need to create tracking structures on demand, with atomic 
>>>replace of last fence, ref counting and locking of some sort, more 
>>>or less?
>>>
>>
>>We will have a workqueue, an work item and a linked list per queue.
>>VM_BIND/UNBIND call will add the mapping request to the specified queue's
>>linked list and schedule the work item on the workqueue of that queue.
>>I am not sure what you mean by last fence and replacing it.
>>
>>>>The other option being discussed in to have the user create those
>>>>queues (like creating engine map) before hand and use that in vm_bind
>>>>and vm_unbind ioctls. This puts a limit on the number of queues.
>>>>But it is not clean either and not sure it is worth making the interface
>>>>more complex.
>>>>https://www.spinics.net/lists/dri-devel/msg350448.html
>>>
>>>What about the third option of a flag to return a fence (of some 
>>>sort) and pass in a fence? That way userspace can imagine zero or 
>>>N queues with very little effort on the kernel side. Was this 
>>>considered?
>>>
>>
>>I am not clear what fence you are talking about here and how does that
>>help with the number of vm_bind queues. Can you eloborate?
>
>It is actually already documented that bind/unbind will support input 
>and output fences - so what are these queues on top of what userspace 
>can already achieve by using them? Purely a convenience or there is 
>more to it?
>

Oh, the vm_bind queues are discussed in this thread.
https://lists.freedesktop.org/archives/intel-gfx/2022-June/299217.html

Apparently Vulkan has requirement for multiple queues, each queue
processing vm_bind/unbind calls in the order of submission.

Niranjana

>Regards,
>
>Tvrtko

Re: [Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Tvrtko Ursulin]

On 13/06/2022 18:49, Niranjana Vishwanathapura wrote:
> On Mon, Jun 13, 2022 at 05:22:02PM +0100, Tvrtko Ursulin wrote:
>>
>> On 13/06/2022 16:05, Niranjana Vishwanathapura wrote:
>>> On Mon, Jun 13, 2022 at 09:24:18AM +0100, Tvrtko Ursulin wrote:
>>>>
>>>> On 10/06/2022 17:14, Niranjana Vishwanathapura wrote:
>>>>> On Fri, Jun 10, 2022 at 05:48:39PM +0300, Lionel Landwerlin wrote:
>>>>>> On 10/06/2022 13:37, Tvrtko Ursulin wrote:
>>>>>>>
>>>>>>> On 10/06/2022 08:07, Niranjana Vishwanathapura wrote:
>>>>>>>> VM_BIND and related uapi definitions
>>>>>>>>
>>>>>>>> Signed-off-by: Niranjana Vishwanathapura 
>>>>>>>> <niranjana.vishwanathapura@intel.com>
>>>>>>>> ---
>>>>>>>>   Documentation/gpu/rfc/i915_vm_bind.h | 490 
>>>>>>>> +++++++++++++++++++++++++++
>>>>>>>>   1 file changed, 490 insertions(+)
>>>>>>>>   create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>
>>>>>>>> diff --git a/Documentation/gpu/rfc/i915_vm_bind.h 
>>>>>>>> b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>> new file mode 100644
>>>>>>>> index 000000000000..9fc854969cfb
>>>>>>>> --- /dev/null
>>>>>>>> +++ b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>> @@ -0,0 +1,490 @@
>>>>>>>> +/* SPDX-License-Identifier: MIT */
>>>>>>>> +/*
>>>>>>>> + * Copyright © 2022 Intel Corporation
>>>>>>>> + */
>>>>>>>> +
>>>>>>>> +/**
>>>>>>>> + * DOC: I915_PARAM_HAS_VM_BIND
>>>>>>>> + *
>>>>>>>> + * VM_BIND feature availability.
>>>>>>>> + * See typedef drm_i915_getparam_t param.
>>>>>>>> + * bit[0]: If set, VM_BIND is supported, otherwise not.
>>>>>>>> + * bits[8-15]: VM_BIND implementation version.
>>>>>>>> + * version 0 will not have VM_BIND/UNBIND timeline fence array 
>>>>>>>> support.
>>>>>>>> + */
>>>>>>>> +#define I915_PARAM_HAS_VM_BIND        57
>>>>>>>> +
>>>>>>>> +/**
>>>>>>>> + * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
>>>>>>>> + *
>>>>>>>> + * Flag to opt-in for VM_BIND mode of binding during VM creation.
>>>>>>>> + * See struct drm_i915_gem_vm_control flags.
>>>>>>>> + *
>>>>>>>> + * The older execbuf2 ioctl will not support VM_BIND mode of 
>>>>>>>> operation.
>>>>>>>> + * For VM_BIND mode, we have new execbuf3 ioctl which will not 
>>>>>>>> accept any
>>>>>>>> + * execlist (See struct drm_i915_gem_execbuffer3 for more 
>>>>>>>> details).
>>>>>>>> + *
>>>>>>>> + */
>>>>>>>> +#define I915_VM_CREATE_FLAGS_USE_VM_BIND    (1 << 0)
>>>>>>>> +
>>>>>>>> +/**
>>>>>>>> + * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
>>>>>>>> + *
>>>>>>>> + * Flag to declare context as long running.
>>>>>>>> + * See struct drm_i915_gem_context_create_ext flags.
>>>>>>>> + *
>>>>>>>> + * Usage of dma-fence expects that they complete in reasonable 
>>>>>>>> amount of time.
>>>>>>>> + * Compute on the other hand can be long running. Hence it is 
>>>>>>>> not appropriate
>>>>>>>> + * for compute contexts to export request completion dma-fence 
>>>>>>>> to user.
>>>>>>>> + * The dma-fence usage will be limited to in-kernel consumption 
>>>>>>>> only.
>>>>>>>> + * Compute contexts need to use user/memory fence.
>>>>>>>> + *
>>>>>>>> + * So, long running contexts do not support output fences. Hence,
>>>>>>>> + * I915_EXEC_FENCE_SIGNAL (See &drm_i915_gem_exec_fence.flags) 
>>>>>>>> is expected
>>>>>>>> + * to be not used. DRM_I915_GEM_WAIT ioctl call is also not 
>>>>>>>> supported for
>>>>>>>> + * objects mapped to long running contexts.
>>>>>>>> + */
>>>>>>>> +#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u << 2)
>>>>>>>> +
>>>>>>>> +/* VM_BIND related ioctls */
>>>>>>>> +#define DRM_I915_GEM_VM_BIND        0x3d
>>>>>>>> +#define DRM_I915_GEM_VM_UNBIND        0x3e
>>>>>>>> +#define DRM_I915_GEM_EXECBUFFER3    0x3f
>>>>>>>> +#define DRM_I915_GEM_WAIT_USER_FENCE    0x40
>>>>>>>> +
>>>>>>>> +#define DRM_IOCTL_I915_GEM_VM_BIND DRM_IOWR(DRM_COMMAND_BASE + 
>>>>>>>> DRM_I915_GEM_VM_BIND, struct drm_i915_gem_vm_bind)
>>>>>>>> +#define DRM_IOCTL_I915_GEM_VM_UNBIND DRM_IOWR(DRM_COMMAND_BASE 
>>>>>>>> + DRM_I915_GEM_VM_UNBIND, struct drm_i915_gem_vm_bind)
>>>>>>>> +#define DRM_IOCTL_I915_GEM_EXECBUFFER3 
>>>>>>>> DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_EXECBUFFER3, struct 
>>>>>>>> drm_i915_gem_execbuffer3)
>>>>>>>> +#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE 
>>>>>>>> DRM_IOWR(DRM_COMMAND_BASE + DRM_I915_GEM_WAIT_USER_FENCE, struct 
>>>>>>>> drm_i915_gem_wait_user_fence)
>>>>>>>> +
>>>>>>>> +/**
>>>>>>>> + * struct drm_i915_gem_vm_bind - VA to object mapping to bind.
>>>>>>>> + *
>>>>>>>> + * This structure is passed to VM_BIND ioctl and specifies the 
>>>>>>>> mapping of GPU
>>>>>>>> + * virtual address (VA) range to the section of an object that 
>>>>>>>> should be bound
>>>>>>>> + * in the device page table of the specified address space (VM).
>>>>>>>> + * The VA range specified must be unique (ie., not currently 
>>>>>>>> bound) and can
>>>>>>>> + * be mapped to whole object or a section of the object 
>>>>>>>> (partial binding).
>>>>>>>> + * Multiple VA mappings can be created to the same section of 
>>>>>>>> the object
>>>>>>>> + * (aliasing).
>>>>>>>> + *
>>>>>>>> + * The @queue_idx specifies the queue to use for binding. Same 
>>>>>>>> queue can be
>>>>>>>> + * used for both VM_BIND and VM_UNBIND calls. All submitted 
>>>>>>>> bind and unbind
>>>>>>>> + * operations in a queue are performed in the order of submission.
>>>>>>>> + *
>>>>>>>> + * The @start, @offset and @length should be 4K page aligned. 
>>>>>>>> However the DG2
>>>>>>>> + * and XEHPSDV has 64K page size for device local-memory and 
>>>>>>>> has compact page
>>>>>>>> + * table. On those platforms, for binding device local-memory 
>>>>>>>> objects, the
>>>>>>>> + * @start should be 2M aligned, @offset and @length should be 
>>>>>>>> 64K aligned.
>>>>>>>> + * Also, on those platforms, it is not allowed to bind an 
>>>>>>>> device local-memory
>>>>>>>> + * object and a system memory object in a single 2M section of 
>>>>>>>> VA range.
>>>>>>>> + */
>>>>>>>> +struct drm_i915_gem_vm_bind {
>>>>>>>> +    /** @vm_id: VM (address space) id to bind */
>>>>>>>> +    __u32 vm_id;
>>>>>>>> +
>>>>>>>> +    /** @queue_idx: Index of queue for binding */
>>>>>>>> +    __u32 queue_idx;
>>>>>>>
>>>>>>> I have a question here to which I did not find an answer by 
>>>>>>> browsing the old threads.
>>>>>>>
>>>>>>> Queue index appears to be an implicit synchronisation mechanism, 
>>>>>>> right? Operations on the same index are executed/complete in 
>>>>>>> order of ioctl submission?
>>>>>>>
>>>>>>> Do we _have_ to implement this on the kernel side and could just 
>>>>>>> allow in/out fence and let userspace deal with it?
>>>>>>
>>>>>>
>>>>>> It orders operations like in a queue. Which is kind of what 
>>>>>> happens with existing queues/engines.
>>>>>>
>>>>>> If I understood correctly, it's going to be a kthread + a linked 
>>>>>> list right?
>>>>>>
>>>>>
>>>>> Yes, that is correct.
>>>>>
>>>>>>
>>>>>> -Lionel
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Arbitrary/on-demand number of queues will add the complexity on 
>>>>>>> the kernel side which should be avoided if possible.
>>>>>>>
>>>>>
>>>>> It was discussed in the other thread. Jason prefers this over putting
>>>>> an artificial limit on number of queues (as user can anyway can 
>>>>> exhaust
>>>>> the memory). I think complexity in the driver is manageable.
>>>>
>>>> You'll need to create tracking structures on demand, with atomic 
>>>> replace of last fence, ref counting and locking of some sort, more 
>>>> or less?
>>>>
>>>
>>> We will have a workqueue, an work item and a linked list per queue.
>>> VM_BIND/UNBIND call will add the mapping request to the specified 
>>> queue's
>>> linked list and schedule the work item on the workqueue of that queue.
>>> I am not sure what you mean by last fence and replacing it.
>>>
>>>>> The other option being discussed in to have the user create those
>>>>> queues (like creating engine map) before hand and use that in vm_bind
>>>>> and vm_unbind ioctls. This puts a limit on the number of queues.
>>>>> But it is not clean either and not sure it is worth making the 
>>>>> interface
>>>>> more complex.
>>>>> https://www.spinics.net/lists/dri-devel/msg350448.html
>>>>
>>>> What about the third option of a flag to return a fence (of some 
>>>> sort) and pass in a fence? That way userspace can imagine zero or N 
>>>> queues with very little effort on the kernel side. Was this considered?
>>>>
>>>
>>> I am not clear what fence you are talking about here and how does that
>>> help with the number of vm_bind queues. Can you eloborate?
>>
>> It is actually already documented that bind/unbind will support input 
>> and output fences - so what are these queues on top of what userspace 
>> can already achieve by using them? Purely a convenience or there is 
>> more to it?
>>
> 
> Oh, the vm_bind queues are discussed in this thread.
> https://lists.freedesktop.org/archives/intel-gfx/2022-June/299217.html
> 
> Apparently Vulkan has requirement for multiple queues, each queue
> processing vm_bind/unbind calls in the order of submission.

I don't see how that answers my question so I will take the freedom to 
repeat it. What are these queues on top of what userspace can already 
achieve by using in-out fences? Purely a convenience or there is more to it?

Queue1:

out_fence_A = vm_bind A
out_fence_B = vm_bind B, in_fence=out_fence_A
execbuf(in_fence = out_fence_B)

Queue2:

out_fence_C = vm_bind C
out_fence_D = vm_bind D, in_fence=out_fence_C
execbuf(in_fence = out_fence_D)

Parallel bind:
out_fence_E = vm_bind E
out_fence_F = vm_bind F
merged_fence = fence_merge(out_fence_E, out_fence_F)
execbuf(in_fence = merged_fence)

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Matthew Brost]

On Mon, Jun 13, 2022 at 07:09:06PM +0100, Tvrtko Ursulin wrote:
> 
> On 13/06/2022 18:49, Niranjana Vishwanathapura wrote:
> > On Mon, Jun 13, 2022 at 05:22:02PM +0100, Tvrtko Ursulin wrote:
> > > 
> > > On 13/06/2022 16:05, Niranjana Vishwanathapura wrote:
> > > > On Mon, Jun 13, 2022 at 09:24:18AM +0100, Tvrtko Ursulin wrote:
> > > > > 
> > > > > On 10/06/2022 17:14, Niranjana Vishwanathapura wrote:
> > > > > > On Fri, Jun 10, 2022 at 05:48:39PM +0300, Lionel Landwerlin wrote:
> > > > > > > On 10/06/2022 13:37, Tvrtko Ursulin wrote:
> > > > > > > > 
> > > > > > > > On 10/06/2022 08:07, Niranjana Vishwanathapura wrote:
> > > > > > > > > VM_BIND and related uapi definitions
> > > > > > > > > 
> > > > > > > > > Signed-off-by: Niranjana Vishwanathapura
> > > > > > > > > <niranjana.vishwanathapura@intel.com>
> > > > > > > > > ---
> > > > > > > > >   Documentation/gpu/rfc/i915_vm_bind.h | 490
> > > > > > > > > +++++++++++++++++++++++++++
> > > > > > > > >   1 file changed, 490 insertions(+)
> > > > > > > > >   create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
> > > > > > > > > 
> > > > > > > > > diff --git
> > > > > > > > > a/Documentation/gpu/rfc/i915_vm_bind.h
> > > > > > > > > b/Documentation/gpu/rfc/i915_vm_bind.h
> > > > > > > > > new file mode 100644
> > > > > > > > > index 000000000000..9fc854969cfb
> > > > > > > > > --- /dev/null
> > > > > > > > > +++ b/Documentation/gpu/rfc/i915_vm_bind.h
> > > > > > > > > @@ -0,0 +1,490 @@
> > > > > > > > > +/* SPDX-License-Identifier: MIT */
> > > > > > > > > +/*
> > > > > > > > > + * Copyright © 2022 Intel Corporation
> > > > > > > > > + */
> > > > > > > > > +
> > > > > > > > > +/**
> > > > > > > > > + * DOC: I915_PARAM_HAS_VM_BIND
> > > > > > > > > + *
> > > > > > > > > + * VM_BIND feature availability.
> > > > > > > > > + * See typedef drm_i915_getparam_t param.
> > > > > > > > > + * bit[0]: If set, VM_BIND is supported, otherwise not.
> > > > > > > > > + * bits[8-15]: VM_BIND implementation version.
> > > > > > > > > + * version 0 will not have VM_BIND/UNBIND
> > > > > > > > > timeline fence array support.
> > > > > > > > > + */
> > > > > > > > > +#define I915_PARAM_HAS_VM_BIND        57
> > > > > > > > > +
> > > > > > > > > +/**
> > > > > > > > > + * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
> > > > > > > > > + *
> > > > > > > > > + * Flag to opt-in for VM_BIND mode of binding during VM creation.
> > > > > > > > > + * See struct drm_i915_gem_vm_control flags.
> > > > > > > > > + *
> > > > > > > > > + * The older execbuf2 ioctl will not
> > > > > > > > > support VM_BIND mode of operation.
> > > > > > > > > + * For VM_BIND mode, we have new execbuf3
> > > > > > > > > ioctl which will not accept any
> > > > > > > > > + * execlist (See struct
> > > > > > > > > drm_i915_gem_execbuffer3 for more details).
> > > > > > > > > + *
> > > > > > > > > + */
> > > > > > > > > +#define I915_VM_CREATE_FLAGS_USE_VM_BIND    (1 << 0)
> > > > > > > > > +
> > > > > > > > > +/**
> > > > > > > > > + * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
> > > > > > > > > + *
> > > > > > > > > + * Flag to declare context as long running.
> > > > > > > > > + * See struct drm_i915_gem_context_create_ext flags.
> > > > > > > > > + *
> > > > > > > > > + * Usage of dma-fence expects that they
> > > > > > > > > complete in reasonable amount of time.
> > > > > > > > > + * Compute on the other hand can be long
> > > > > > > > > running. Hence it is not appropriate
> > > > > > > > > + * for compute contexts to export request
> > > > > > > > > completion dma-fence to user.
> > > > > > > > > + * The dma-fence usage will be limited to
> > > > > > > > > in-kernel consumption only.
> > > > > > > > > + * Compute contexts need to use user/memory fence.
> > > > > > > > > + *
> > > > > > > > > + * So, long running contexts do not support output fences. Hence,
> > > > > > > > > + * I915_EXEC_FENCE_SIGNAL (See
> > > > > > > > > &drm_i915_gem_exec_fence.flags) is expected
> > > > > > > > > + * to be not used. DRM_I915_GEM_WAIT ioctl
> > > > > > > > > call is also not supported for
> > > > > > > > > + * objects mapped to long running contexts.
> > > > > > > > > + */
> > > > > > > > > +#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u << 2)
> > > > > > > > > +
> > > > > > > > > +/* VM_BIND related ioctls */
> > > > > > > > > +#define DRM_I915_GEM_VM_BIND        0x3d
> > > > > > > > > +#define DRM_I915_GEM_VM_UNBIND        0x3e
> > > > > > > > > +#define DRM_I915_GEM_EXECBUFFER3    0x3f
> > > > > > > > > +#define DRM_I915_GEM_WAIT_USER_FENCE    0x40
> > > > > > > > > +
> > > > > > > > > +#define DRM_IOCTL_I915_GEM_VM_BIND
> > > > > > > > > DRM_IOWR(DRM_COMMAND_BASE +
> > > > > > > > > DRM_I915_GEM_VM_BIND, struct
> > > > > > > > > drm_i915_gem_vm_bind)
> > > > > > > > > +#define DRM_IOCTL_I915_GEM_VM_UNBIND
> > > > > > > > > DRM_IOWR(DRM_COMMAND_BASE +
> > > > > > > > > DRM_I915_GEM_VM_UNBIND, struct
> > > > > > > > > drm_i915_gem_vm_bind)
> > > > > > > > > +#define DRM_IOCTL_I915_GEM_EXECBUFFER3
> > > > > > > > > DRM_IOWR(DRM_COMMAND_BASE +
> > > > > > > > > DRM_I915_GEM_EXECBUFFER3, struct
> > > > > > > > > drm_i915_gem_execbuffer3)
> > > > > > > > > +#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE
> > > > > > > > > DRM_IOWR(DRM_COMMAND_BASE +
> > > > > > > > > DRM_I915_GEM_WAIT_USER_FENCE, struct
> > > > > > > > > drm_i915_gem_wait_user_fence)
> > > > > > > > > +
> > > > > > > > > +/**
> > > > > > > > > + * struct drm_i915_gem_vm_bind - VA to object mapping to bind.
> > > > > > > > > + *
> > > > > > > > > + * This structure is passed to VM_BIND
> > > > > > > > > ioctl and specifies the mapping of GPU
> > > > > > > > > + * virtual address (VA) range to the
> > > > > > > > > section of an object that should be bound
> > > > > > > > > + * in the device page table of the specified address space (VM).
> > > > > > > > > + * The VA range specified must be unique
> > > > > > > > > (ie., not currently bound) and can
> > > > > > > > > + * be mapped to whole object or a section
> > > > > > > > > of the object (partial binding).
> > > > > > > > > + * Multiple VA mappings can be created to
> > > > > > > > > the same section of the object
> > > > > > > > > + * (aliasing).
> > > > > > > > > + *
> > > > > > > > > + * The @queue_idx specifies the queue to
> > > > > > > > > use for binding. Same queue can be
> > > > > > > > > + * used for both VM_BIND and VM_UNBIND
> > > > > > > > > calls. All submitted bind and unbind
> > > > > > > > > + * operations in a queue are performed in the order of submission.
> > > > > > > > > + *
> > > > > > > > > + * The @start, @offset and @length should
> > > > > > > > > be 4K page aligned. However the DG2
> > > > > > > > > + * and XEHPSDV has 64K page size for device
> > > > > > > > > local-memory and has compact page
> > > > > > > > > + * table. On those platforms, for binding
> > > > > > > > > device local-memory objects, the
> > > > > > > > > + * @start should be 2M aligned, @offset and
> > > > > > > > > @length should be 64K aligned.
> > > > > > > > > + * Also, on those platforms, it is not
> > > > > > > > > allowed to bind an device local-memory
> > > > > > > > > + * object and a system memory object in a
> > > > > > > > > single 2M section of VA range.
> > > > > > > > > + */
> > > > > > > > > +struct drm_i915_gem_vm_bind {
> > > > > > > > > +    /** @vm_id: VM (address space) id to bind */
> > > > > > > > > +    __u32 vm_id;
> > > > > > > > > +
> > > > > > > > > +    /** @queue_idx: Index of queue for binding */
> > > > > > > > > +    __u32 queue_idx;
> > > > > > > > 
> > > > > > > > I have a question here to which I did not find
> > > > > > > > an answer by browsing the old threads.
> > > > > > > > 
> > > > > > > > Queue index appears to be an implicit
> > > > > > > > synchronisation mechanism, right? Operations on
> > > > > > > > the same index are executed/complete in order of
> > > > > > > > ioctl submission?
> > > > > > > > 
> > > > > > > > Do we _have_ to implement this on the kernel
> > > > > > > > side and could just allow in/out fence and let
> > > > > > > > userspace deal with it?
> > > > > > > 
> > > > > > > 
> > > > > > > It orders operations like in a queue. Which is kind
> > > > > > > of what happens with existing queues/engines.
> > > > > > > 
> > > > > > > If I understood correctly, it's going to be a
> > > > > > > kthread + a linked list right?
> > > > > > > 
> > > > > > 
> > > > > > Yes, that is correct.
> > > > > > 
> > > > > > > 
> > > > > > > -Lionel
> > > > > > > 
> > > > > > > 
> > > > > > > > 
> > > > > > > > Arbitrary/on-demand number of queues will add
> > > > > > > > the complexity on the kernel side which should
> > > > > > > > be avoided if possible.
> > > > > > > > 
> > > > > > 
> > > > > > It was discussed in the other thread. Jason prefers this over putting
> > > > > > an artificial limit on number of queues (as user can
> > > > > > anyway can exhaust
> > > > > > the memory). I think complexity in the driver is manageable.
> > > > > 
> > > > > You'll need to create tracking structures on demand, with
> > > > > atomic replace of last fence, ref counting and locking of
> > > > > some sort, more or less?
> > > > > 
> > > > 
> > > > We will have a workqueue, an work item and a linked list per queue.
> > > > VM_BIND/UNBIND call will add the mapping request to the
> > > > specified queue's
> > > > linked list and schedule the work item on the workqueue of that queue.
> > > > I am not sure what you mean by last fence and replacing it.
> > > > 
> > > > > > The other option being discussed in to have the user create those
> > > > > > queues (like creating engine map) before hand and use that in vm_bind
> > > > > > and vm_unbind ioctls. This puts a limit on the number of queues.
> > > > > > But it is not clean either and not sure it is worth
> > > > > > making the interface
> > > > > > more complex.
> > > > > > https://www.spinics.net/lists/dri-devel/msg350448.html
> > > > > 
> > > > > What about the third option of a flag to return a fence (of
> > > > > some sort) and pass in a fence? That way userspace can
> > > > > imagine zero or N queues with very little effort on the
> > > > > kernel side. Was this considered?
> > > > > 
> > > > 
> > > > I am not clear what fence you are talking about here and how does that
> > > > help with the number of vm_bind queues. Can you eloborate?
> > > 
> > > It is actually already documented that bind/unbind will support
> > > input and output fences - so what are these queues on top of what
> > > userspace can already achieve by using them? Purely a convenience or
> > > there is more to it?
> > > 
> > 
> > Oh, the vm_bind queues are discussed in this thread.
> > https://lists.freedesktop.org/archives/intel-gfx/2022-June/299217.html
> > 
> > Apparently Vulkan has requirement for multiple queues, each queue
> > processing vm_bind/unbind calls in the order of submission.
> 
> I don't see how that answers my question so I will take the freedom to
> repeat it. What are these queues on top of what userspace can already
> achieve by using in-out fences? Purely a convenience or there is more to it?
> 
> Queue1:
> 
> out_fence_A = vm_bind A
> out_fence_B = vm_bind B, in_fence=out_fence_A
> execbuf(in_fence = out_fence_B)
> 
> Queue2:
> 
> out_fence_C = vm_bind C
> out_fence_D = vm_bind D, in_fence=out_fence_C
> execbuf(in_fence = out_fence_D)
> 
> Parallel bind:
> out_fence_E = vm_bind E
> out_fence_F = vm_bind F
> merged_fence = fence_merge(out_fence_E, out_fence_F)
> execbuf(in_fence = merged_fence)
> 

Let's say you do this and only 1 queue:

VM_BIND_A (in_fence=fence_A)
VM_BIND_B (in_fence=NULL)

With 1 queue VM_BIND_B in blocked on fence_A, hence the need for than 1
queue.

e.g.
 
VM_BIND_A (queue_id=0, in_fence=fence_A)
VM_BIND_B (queue_id=1, in_fence=NULL)

Now VM_BIND_B can immediately be executed regardless of fence_A status.

Matt

> Regards,
> 
> Tvrtko

Re: [Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Tvrtko Ursulin]

On 14/06/2022 00:39, Matthew Brost wrote:
> On Mon, Jun 13, 2022 at 07:09:06PM +0100, Tvrtko Ursulin wrote:
>>
>> On 13/06/2022 18:49, Niranjana Vishwanathapura wrote:
>>> On Mon, Jun 13, 2022 at 05:22:02PM +0100, Tvrtko Ursulin wrote:
>>>>
>>>> On 13/06/2022 16:05, Niranjana Vishwanathapura wrote:
>>>>> On Mon, Jun 13, 2022 at 09:24:18AM +0100, Tvrtko Ursulin wrote:
>>>>>>
>>>>>> On 10/06/2022 17:14, Niranjana Vishwanathapura wrote:
>>>>>>> On Fri, Jun 10, 2022 at 05:48:39PM +0300, Lionel Landwerlin wrote:
>>>>>>>> On 10/06/2022 13:37, Tvrtko Ursulin wrote:
>>>>>>>>>
>>>>>>>>> On 10/06/2022 08:07, Niranjana Vishwanathapura wrote:
>>>>>>>>>> VM_BIND and related uapi definitions
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Niranjana Vishwanathapura
>>>>>>>>>> <niranjana.vishwanathapura@intel.com>
>>>>>>>>>> ---
>>>>>>>>>>    Documentation/gpu/rfc/i915_vm_bind.h | 490
>>>>>>>>>> +++++++++++++++++++++++++++
>>>>>>>>>>    1 file changed, 490 insertions(+)
>>>>>>>>>>    create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>
>>>>>>>>>> diff --git
>>>>>>>>>> a/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>> b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>> new file mode 100644
>>>>>>>>>> index 000000000000..9fc854969cfb
>>>>>>>>>> --- /dev/null
>>>>>>>>>> +++ b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>> @@ -0,0 +1,490 @@
>>>>>>>>>> +/* SPDX-License-Identifier: MIT */
>>>>>>>>>> +/*
>>>>>>>>>> + * Copyright © 2022 Intel Corporation
>>>>>>>>>> + */
>>>>>>>>>> +
>>>>>>>>>> +/**
>>>>>>>>>> + * DOC: I915_PARAM_HAS_VM_BIND
>>>>>>>>>> + *
>>>>>>>>>> + * VM_BIND feature availability.
>>>>>>>>>> + * See typedef drm_i915_getparam_t param.
>>>>>>>>>> + * bit[0]: If set, VM_BIND is supported, otherwise not.
>>>>>>>>>> + * bits[8-15]: VM_BIND implementation version.
>>>>>>>>>> + * version 0 will not have VM_BIND/UNBIND
>>>>>>>>>> timeline fence array support.
>>>>>>>>>> + */
>>>>>>>>>> +#define I915_PARAM_HAS_VM_BIND        57
>>>>>>>>>> +
>>>>>>>>>> +/**
>>>>>>>>>> + * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
>>>>>>>>>> + *
>>>>>>>>>> + * Flag to opt-in for VM_BIND mode of binding during VM creation.
>>>>>>>>>> + * See struct drm_i915_gem_vm_control flags.
>>>>>>>>>> + *
>>>>>>>>>> + * The older execbuf2 ioctl will not
>>>>>>>>>> support VM_BIND mode of operation.
>>>>>>>>>> + * For VM_BIND mode, we have new execbuf3
>>>>>>>>>> ioctl which will not accept any
>>>>>>>>>> + * execlist (See struct
>>>>>>>>>> drm_i915_gem_execbuffer3 for more details).
>>>>>>>>>> + *
>>>>>>>>>> + */
>>>>>>>>>> +#define I915_VM_CREATE_FLAGS_USE_VM_BIND    (1 << 0)
>>>>>>>>>> +
>>>>>>>>>> +/**
>>>>>>>>>> + * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
>>>>>>>>>> + *
>>>>>>>>>> + * Flag to declare context as long running.
>>>>>>>>>> + * See struct drm_i915_gem_context_create_ext flags.
>>>>>>>>>> + *
>>>>>>>>>> + * Usage of dma-fence expects that they
>>>>>>>>>> complete in reasonable amount of time.
>>>>>>>>>> + * Compute on the other hand can be long
>>>>>>>>>> running. Hence it is not appropriate
>>>>>>>>>> + * for compute contexts to export request
>>>>>>>>>> completion dma-fence to user.
>>>>>>>>>> + * The dma-fence usage will be limited to
>>>>>>>>>> in-kernel consumption only.
>>>>>>>>>> + * Compute contexts need to use user/memory fence.
>>>>>>>>>> + *
>>>>>>>>>> + * So, long running contexts do not support output fences. Hence,
>>>>>>>>>> + * I915_EXEC_FENCE_SIGNAL (See
>>>>>>>>>> &drm_i915_gem_exec_fence.flags) is expected
>>>>>>>>>> + * to be not used. DRM_I915_GEM_WAIT ioctl
>>>>>>>>>> call is also not supported for
>>>>>>>>>> + * objects mapped to long running contexts.
>>>>>>>>>> + */
>>>>>>>>>> +#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u << 2)
>>>>>>>>>> +
>>>>>>>>>> +/* VM_BIND related ioctls */
>>>>>>>>>> +#define DRM_I915_GEM_VM_BIND        0x3d
>>>>>>>>>> +#define DRM_I915_GEM_VM_UNBIND        0x3e
>>>>>>>>>> +#define DRM_I915_GEM_EXECBUFFER3    0x3f
>>>>>>>>>> +#define DRM_I915_GEM_WAIT_USER_FENCE    0x40
>>>>>>>>>> +
>>>>>>>>>> +#define DRM_IOCTL_I915_GEM_VM_BIND
>>>>>>>>>> DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>> DRM_I915_GEM_VM_BIND, struct
>>>>>>>>>> drm_i915_gem_vm_bind)
>>>>>>>>>> +#define DRM_IOCTL_I915_GEM_VM_UNBIND
>>>>>>>>>> DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>> DRM_I915_GEM_VM_UNBIND, struct
>>>>>>>>>> drm_i915_gem_vm_bind)
>>>>>>>>>> +#define DRM_IOCTL_I915_GEM_EXECBUFFER3
>>>>>>>>>> DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>> DRM_I915_GEM_EXECBUFFER3, struct
>>>>>>>>>> drm_i915_gem_execbuffer3)
>>>>>>>>>> +#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE
>>>>>>>>>> DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>> DRM_I915_GEM_WAIT_USER_FENCE, struct
>>>>>>>>>> drm_i915_gem_wait_user_fence)
>>>>>>>>>> +
>>>>>>>>>> +/**
>>>>>>>>>> + * struct drm_i915_gem_vm_bind - VA to object mapping to bind.
>>>>>>>>>> + *
>>>>>>>>>> + * This structure is passed to VM_BIND
>>>>>>>>>> ioctl and specifies the mapping of GPU
>>>>>>>>>> + * virtual address (VA) range to the
>>>>>>>>>> section of an object that should be bound
>>>>>>>>>> + * in the device page table of the specified address space (VM).
>>>>>>>>>> + * The VA range specified must be unique
>>>>>>>>>> (ie., not currently bound) and can
>>>>>>>>>> + * be mapped to whole object or a section
>>>>>>>>>> of the object (partial binding).
>>>>>>>>>> + * Multiple VA mappings can be created to
>>>>>>>>>> the same section of the object
>>>>>>>>>> + * (aliasing).
>>>>>>>>>> + *
>>>>>>>>>> + * The @queue_idx specifies the queue to
>>>>>>>>>> use for binding. Same queue can be
>>>>>>>>>> + * used for both VM_BIND and VM_UNBIND
>>>>>>>>>> calls. All submitted bind and unbind
>>>>>>>>>> + * operations in a queue are performed in the order of submission.
>>>>>>>>>> + *
>>>>>>>>>> + * The @start, @offset and @length should
>>>>>>>>>> be 4K page aligned. However the DG2
>>>>>>>>>> + * and XEHPSDV has 64K page size for device
>>>>>>>>>> local-memory and has compact page
>>>>>>>>>> + * table. On those platforms, for binding
>>>>>>>>>> device local-memory objects, the
>>>>>>>>>> + * @start should be 2M aligned, @offset and
>>>>>>>>>> @length should be 64K aligned.
>>>>>>>>>> + * Also, on those platforms, it is not
>>>>>>>>>> allowed to bind an device local-memory
>>>>>>>>>> + * object and a system memory object in a
>>>>>>>>>> single 2M section of VA range.
>>>>>>>>>> + */
>>>>>>>>>> +struct drm_i915_gem_vm_bind {
>>>>>>>>>> +    /** @vm_id: VM (address space) id to bind */
>>>>>>>>>> +    __u32 vm_id;
>>>>>>>>>> +
>>>>>>>>>> +    /** @queue_idx: Index of queue for binding */
>>>>>>>>>> +    __u32 queue_idx;
>>>>>>>>>
>>>>>>>>> I have a question here to which I did not find
>>>>>>>>> an answer by browsing the old threads.
>>>>>>>>>
>>>>>>>>> Queue index appears to be an implicit
>>>>>>>>> synchronisation mechanism, right? Operations on
>>>>>>>>> the same index are executed/complete in order of
>>>>>>>>> ioctl submission?
>>>>>>>>>
>>>>>>>>> Do we _have_ to implement this on the kernel
>>>>>>>>> side and could just allow in/out fence and let
>>>>>>>>> userspace deal with it?
>>>>>>>>
>>>>>>>>
>>>>>>>> It orders operations like in a queue. Which is kind
>>>>>>>> of what happens with existing queues/engines.
>>>>>>>>
>>>>>>>> If I understood correctly, it's going to be a
>>>>>>>> kthread + a linked list right?
>>>>>>>>
>>>>>>>
>>>>>>> Yes, that is correct.
>>>>>>>
>>>>>>>>
>>>>>>>> -Lionel
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Arbitrary/on-demand number of queues will add
>>>>>>>>> the complexity on the kernel side which should
>>>>>>>>> be avoided if possible.
>>>>>>>>>
>>>>>>>
>>>>>>> It was discussed in the other thread. Jason prefers this over putting
>>>>>>> an artificial limit on number of queues (as user can
>>>>>>> anyway can exhaust
>>>>>>> the memory). I think complexity in the driver is manageable.
>>>>>>
>>>>>> You'll need to create tracking structures on demand, with
>>>>>> atomic replace of last fence, ref counting and locking of
>>>>>> some sort, more or less?
>>>>>>
>>>>>
>>>>> We will have a workqueue, an work item and a linked list per queue.
>>>>> VM_BIND/UNBIND call will add the mapping request to the
>>>>> specified queue's
>>>>> linked list and schedule the work item on the workqueue of that queue.
>>>>> I am not sure what you mean by last fence and replacing it.
>>>>>
>>>>>>> The other option being discussed in to have the user create those
>>>>>>> queues (like creating engine map) before hand and use that in vm_bind
>>>>>>> and vm_unbind ioctls. This puts a limit on the number of queues.
>>>>>>> But it is not clean either and not sure it is worth
>>>>>>> making the interface
>>>>>>> more complex.
>>>>>>> https://www.spinics.net/lists/dri-devel/msg350448.html
>>>>>>
>>>>>> What about the third option of a flag to return a fence (of
>>>>>> some sort) and pass in a fence? That way userspace can
>>>>>> imagine zero or N queues with very little effort on the
>>>>>> kernel side. Was this considered?
>>>>>>
>>>>>
>>>>> I am not clear what fence you are talking about here and how does that
>>>>> help with the number of vm_bind queues. Can you eloborate?
>>>>
>>>> It is actually already documented that bind/unbind will support
>>>> input and output fences - so what are these queues on top of what
>>>> userspace can already achieve by using them? Purely a convenience or
>>>> there is more to it?
>>>>
>>>
>>> Oh, the vm_bind queues are discussed in this thread.
>>> https://lists.freedesktop.org/archives/intel-gfx/2022-June/299217.html
>>>
>>> Apparently Vulkan has requirement for multiple queues, each queue
>>> processing vm_bind/unbind calls in the order of submission.
>>
>> I don't see how that answers my question so I will take the freedom to
>> repeat it. What are these queues on top of what userspace can already
>> achieve by using in-out fences? Purely a convenience or there is more to it?
>>
>> Queue1:
>>
>> out_fence_A = vm_bind A
>> out_fence_B = vm_bind B, in_fence=out_fence_A
>> execbuf(in_fence = out_fence_B)
>>
>> Queue2:
>>
>> out_fence_C = vm_bind C
>> out_fence_D = vm_bind D, in_fence=out_fence_C
>> execbuf(in_fence = out_fence_D)
>>
>> Parallel bind:
>> out_fence_E = vm_bind E
>> out_fence_F = vm_bind F
>> merged_fence = fence_merge(out_fence_E, out_fence_F)
>> execbuf(in_fence = merged_fence)
>>
> 
> Let's say you do this and only 1 queue:
> 
> VM_BIND_A (in_fence=fence_A)
> VM_BIND_B (in_fence=NULL)
> 
> With 1 queue VM_BIND_B in blocked on fence_A, hence the need for than 1
> queue.

I don't follow - there isn't a concept of a queue exposed in uapi in 
what I have described so the above two run in parallel there, if we 
ignore fence_A in your example doesn't even exist before you pass it to 
bind A so something is not right.

> e.g.
>   
> VM_BIND_A (queue_id=0, in_fence=fence_A)
> VM_BIND_B (queue_id=1, in_fence=NULL)
> 
> Now VM_BIND_B can immediately be executed regardless of fence_A status.

In my examples userspace can serialise or not as it sees fit using 
fences. The "parallel bind" examples two binds run in parallel. 
Userspace can create multiple such parallel "queues" if it wanted.

Parallel bind 1 and 2 interleaved:
out_fence_A = vm_bind A
out_fence_B = vm_bind B
out_fence_C = vm_bind C
out_fence_D = vm_bind D
// all binds can run in parallel
merged_fence_1 = fence_merge(out_fence_A, out_fence_B)
merged_fence_2 = fence_merge(out_fence_C, out_fence_D)
execbuf(in_fence = merged_fence_1) // after A&B to finish
execbuf(in_fence = merged_fence_2) // after C&D finish

There is a huge disconnect somewhere but I don't know where.

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Niranjana Vishwanathapura]

On Tue, Jun 14, 2022 at 08:16:41AM +0100, Tvrtko Ursulin wrote:
>
>On 14/06/2022 00:39, Matthew Brost wrote:
>>On Mon, Jun 13, 2022 at 07:09:06PM +0100, Tvrtko Ursulin wrote:
>>>
>>>On 13/06/2022 18:49, Niranjana Vishwanathapura wrote:
>>>>On Mon, Jun 13, 2022 at 05:22:02PM +0100, Tvrtko Ursulin wrote:
>>>>>
>>>>>On 13/06/2022 16:05, Niranjana Vishwanathapura wrote:
>>>>>>On Mon, Jun 13, 2022 at 09:24:18AM +0100, Tvrtko Ursulin wrote:
>>>>>>>
>>>>>>>On 10/06/2022 17:14, Niranjana Vishwanathapura wrote:
>>>>>>>>On Fri, Jun 10, 2022 at 05:48:39PM +0300, Lionel Landwerlin wrote:
>>>>>>>>>On 10/06/2022 13:37, Tvrtko Ursulin wrote:
>>>>>>>>>>
>>>>>>>>>>On 10/06/2022 08:07, Niranjana Vishwanathapura wrote:
>>>>>>>>>>>VM_BIND and related uapi definitions
>>>>>>>>>>>
>>>>>>>>>>>Signed-off-by: Niranjana Vishwanathapura
>>>>>>>>>>><niranjana.vishwanathapura@intel.com>
>>>>>>>>>>>---
>>>>>>>>>>>   Documentation/gpu/rfc/i915_vm_bind.h | 490
>>>>>>>>>>>+++++++++++++++++++++++++++
>>>>>>>>>>>   1 file changed, 490 insertions(+)
>>>>>>>>>>>   create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>
>>>>>>>>>>>diff --git
>>>>>>>>>>>a/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>new file mode 100644
>>>>>>>>>>>index 000000000000..9fc854969cfb
>>>>>>>>>>>--- /dev/null
>>>>>>>>>>>+++ b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>@@ -0,0 +1,490 @@
>>>>>>>>>>>+/* SPDX-License-Identifier: MIT */
>>>>>>>>>>>+/*
>>>>>>>>>>>+ * Copyright © 2022 Intel Corporation
>>>>>>>>>>>+ */
>>>>>>>>>>>+
>>>>>>>>>>>+/**
>>>>>>>>>>>+ * DOC: I915_PARAM_HAS_VM_BIND
>>>>>>>>>>>+ *
>>>>>>>>>>>+ * VM_BIND feature availability.
>>>>>>>>>>>+ * See typedef drm_i915_getparam_t param.
>>>>>>>>>>>+ * bit[0]: If set, VM_BIND is supported, otherwise not.
>>>>>>>>>>>+ * bits[8-15]: VM_BIND implementation version.
>>>>>>>>>>>+ * version 0 will not have VM_BIND/UNBIND
>>>>>>>>>>>timeline fence array support.
>>>>>>>>>>>+ */
>>>>>>>>>>>+#define I915_PARAM_HAS_VM_BIND        57
>>>>>>>>>>>+
>>>>>>>>>>>+/**
>>>>>>>>>>>+ * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
>>>>>>>>>>>+ *
>>>>>>>>>>>+ * Flag to opt-in for VM_BIND mode of binding during VM creation.
>>>>>>>>>>>+ * See struct drm_i915_gem_vm_control flags.
>>>>>>>>>>>+ *
>>>>>>>>>>>+ * The older execbuf2 ioctl will not
>>>>>>>>>>>support VM_BIND mode of operation.
>>>>>>>>>>>+ * For VM_BIND mode, we have new execbuf3
>>>>>>>>>>>ioctl which will not accept any
>>>>>>>>>>>+ * execlist (See struct
>>>>>>>>>>>drm_i915_gem_execbuffer3 for more details).
>>>>>>>>>>>+ *
>>>>>>>>>>>+ */
>>>>>>>>>>>+#define I915_VM_CREATE_FLAGS_USE_VM_BIND    (1 << 0)
>>>>>>>>>>>+
>>>>>>>>>>>+/**
>>>>>>>>>>>+ * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
>>>>>>>>>>>+ *
>>>>>>>>>>>+ * Flag to declare context as long running.
>>>>>>>>>>>+ * See struct drm_i915_gem_context_create_ext flags.
>>>>>>>>>>>+ *
>>>>>>>>>>>+ * Usage of dma-fence expects that they
>>>>>>>>>>>complete in reasonable amount of time.
>>>>>>>>>>>+ * Compute on the other hand can be long
>>>>>>>>>>>running. Hence it is not appropriate
>>>>>>>>>>>+ * for compute contexts to export request
>>>>>>>>>>>completion dma-fence to user.
>>>>>>>>>>>+ * The dma-fence usage will be limited to
>>>>>>>>>>>in-kernel consumption only.
>>>>>>>>>>>+ * Compute contexts need to use user/memory fence.
>>>>>>>>>>>+ *
>>>>>>>>>>>+ * So, long running contexts do not support output fences. Hence,
>>>>>>>>>>>+ * I915_EXEC_FENCE_SIGNAL (See
>>>>>>>>>>>&drm_i915_gem_exec_fence.flags) is expected
>>>>>>>>>>>+ * to be not used. DRM_I915_GEM_WAIT ioctl
>>>>>>>>>>>call is also not supported for
>>>>>>>>>>>+ * objects mapped to long running contexts.
>>>>>>>>>>>+ */
>>>>>>>>>>>+#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u << 2)
>>>>>>>>>>>+
>>>>>>>>>>>+/* VM_BIND related ioctls */
>>>>>>>>>>>+#define DRM_I915_GEM_VM_BIND        0x3d
>>>>>>>>>>>+#define DRM_I915_GEM_VM_UNBIND        0x3e
>>>>>>>>>>>+#define DRM_I915_GEM_EXECBUFFER3    0x3f
>>>>>>>>>>>+#define DRM_I915_GEM_WAIT_USER_FENCE    0x40
>>>>>>>>>>>+
>>>>>>>>>>>+#define DRM_IOCTL_I915_GEM_VM_BIND
>>>>>>>>>>>DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>DRM_I915_GEM_VM_BIND, struct
>>>>>>>>>>>drm_i915_gem_vm_bind)
>>>>>>>>>>>+#define DRM_IOCTL_I915_GEM_VM_UNBIND
>>>>>>>>>>>DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>DRM_I915_GEM_VM_UNBIND, struct
>>>>>>>>>>>drm_i915_gem_vm_bind)
>>>>>>>>>>>+#define DRM_IOCTL_I915_GEM_EXECBUFFER3
>>>>>>>>>>>DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>DRM_I915_GEM_EXECBUFFER3, struct
>>>>>>>>>>>drm_i915_gem_execbuffer3)
>>>>>>>>>>>+#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE
>>>>>>>>>>>DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>DRM_I915_GEM_WAIT_USER_FENCE, struct
>>>>>>>>>>>drm_i915_gem_wait_user_fence)
>>>>>>>>>>>+
>>>>>>>>>>>+/**
>>>>>>>>>>>+ * struct drm_i915_gem_vm_bind - VA to object mapping to bind.
>>>>>>>>>>>+ *
>>>>>>>>>>>+ * This structure is passed to VM_BIND
>>>>>>>>>>>ioctl and specifies the mapping of GPU
>>>>>>>>>>>+ * virtual address (VA) range to the
>>>>>>>>>>>section of an object that should be bound
>>>>>>>>>>>+ * in the device page table of the specified address space (VM).
>>>>>>>>>>>+ * The VA range specified must be unique
>>>>>>>>>>>(ie., not currently bound) and can
>>>>>>>>>>>+ * be mapped to whole object or a section
>>>>>>>>>>>of the object (partial binding).
>>>>>>>>>>>+ * Multiple VA mappings can be created to
>>>>>>>>>>>the same section of the object
>>>>>>>>>>>+ * (aliasing).
>>>>>>>>>>>+ *
>>>>>>>>>>>+ * The @queue_idx specifies the queue to
>>>>>>>>>>>use for binding. Same queue can be
>>>>>>>>>>>+ * used for both VM_BIND and VM_UNBIND
>>>>>>>>>>>calls. All submitted bind and unbind
>>>>>>>>>>>+ * operations in a queue are performed in the order of submission.
>>>>>>>>>>>+ *
>>>>>>>>>>>+ * The @start, @offset and @length should
>>>>>>>>>>>be 4K page aligned. However the DG2
>>>>>>>>>>>+ * and XEHPSDV has 64K page size for device
>>>>>>>>>>>local-memory and has compact page
>>>>>>>>>>>+ * table. On those platforms, for binding
>>>>>>>>>>>device local-memory objects, the
>>>>>>>>>>>+ * @start should be 2M aligned, @offset and
>>>>>>>>>>>@length should be 64K aligned.
>>>>>>>>>>>+ * Also, on those platforms, it is not
>>>>>>>>>>>allowed to bind an device local-memory
>>>>>>>>>>>+ * object and a system memory object in a
>>>>>>>>>>>single 2M section of VA range.
>>>>>>>>>>>+ */
>>>>>>>>>>>+struct drm_i915_gem_vm_bind {
>>>>>>>>>>>+    /** @vm_id: VM (address space) id to bind */
>>>>>>>>>>>+    __u32 vm_id;
>>>>>>>>>>>+
>>>>>>>>>>>+    /** @queue_idx: Index of queue for binding */
>>>>>>>>>>>+    __u32 queue_idx;
>>>>>>>>>>
>>>>>>>>>>I have a question here to which I did not find
>>>>>>>>>>an answer by browsing the old threads.
>>>>>>>>>>
>>>>>>>>>>Queue index appears to be an implicit
>>>>>>>>>>synchronisation mechanism, right? Operations on
>>>>>>>>>>the same index are executed/complete in order of
>>>>>>>>>>ioctl submission?
>>>>>>>>>>
>>>>>>>>>>Do we _have_ to implement this on the kernel
>>>>>>>>>>side and could just allow in/out fence and let
>>>>>>>>>>userspace deal with it?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>It orders operations like in a queue. Which is kind
>>>>>>>>>of what happens with existing queues/engines.
>>>>>>>>>
>>>>>>>>>If I understood correctly, it's going to be a
>>>>>>>>>kthread + a linked list right?
>>>>>>>>>
>>>>>>>>
>>>>>>>>Yes, that is correct.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>-Lionel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>Arbitrary/on-demand number of queues will add
>>>>>>>>>>the complexity on the kernel side which should
>>>>>>>>>>be avoided if possible.
>>>>>>>>>>
>>>>>>>>
>>>>>>>>It was discussed in the other thread. Jason prefers this over putting
>>>>>>>>an artificial limit on number of queues (as user can
>>>>>>>>anyway can exhaust
>>>>>>>>the memory). I think complexity in the driver is manageable.
>>>>>>>
>>>>>>>You'll need to create tracking structures on demand, with
>>>>>>>atomic replace of last fence, ref counting and locking of
>>>>>>>some sort, more or less?
>>>>>>>
>>>>>>
>>>>>>We will have a workqueue, an work item and a linked list per queue.
>>>>>>VM_BIND/UNBIND call will add the mapping request to the
>>>>>>specified queue's
>>>>>>linked list and schedule the work item on the workqueue of that queue.
>>>>>>I am not sure what you mean by last fence and replacing it.
>>>>>>
>>>>>>>>The other option being discussed in to have the user create those
>>>>>>>>queues (like creating engine map) before hand and use that in vm_bind
>>>>>>>>and vm_unbind ioctls. This puts a limit on the number of queues.
>>>>>>>>But it is not clean either and not sure it is worth
>>>>>>>>making the interface
>>>>>>>>more complex.
>>>>>>>>https://www.spinics.net/lists/dri-devel/msg350448.html
>>>>>>>
>>>>>>>What about the third option of a flag to return a fence (of
>>>>>>>some sort) and pass in a fence? That way userspace can
>>>>>>>imagine zero or N queues with very little effort on the
>>>>>>>kernel side. Was this considered?
>>>>>>>
>>>>>>
>>>>>>I am not clear what fence you are talking about here and how does that
>>>>>>help with the number of vm_bind queues. Can you eloborate?
>>>>>
>>>>>It is actually already documented that bind/unbind will support
>>>>>input and output fences - so what are these queues on top of what
>>>>>userspace can already achieve by using them? Purely a convenience or
>>>>>there is more to it?
>>>>>
>>>>
>>>>Oh, the vm_bind queues are discussed in this thread.
>>>>https://lists.freedesktop.org/archives/intel-gfx/2022-June/299217.html
>>>>
>>>>Apparently Vulkan has requirement for multiple queues, each queue
>>>>processing vm_bind/unbind calls in the order of submission.
>>>
>>>I don't see how that answers my question so I will take the freedom to
>>>repeat it. What are these queues on top of what userspace can already
>>>achieve by using in-out fences? Purely a convenience or there is more to it?
>>>
>>>Queue1:
>>>
>>>out_fence_A = vm_bind A
>>>out_fence_B = vm_bind B, in_fence=out_fence_A
>>>execbuf(in_fence = out_fence_B)
>>>
>>>Queue2:
>>>
>>>out_fence_C = vm_bind C
>>>out_fence_D = vm_bind D, in_fence=out_fence_C
>>>execbuf(in_fence = out_fence_D)
>>>
>>>Parallel bind:
>>>out_fence_E = vm_bind E
>>>out_fence_F = vm_bind F
>>>merged_fence = fence_merge(out_fence_E, out_fence_F)
>>>execbuf(in_fence = merged_fence)
>>>
>>
>>Let's say you do this and only 1 queue:
>>
>>VM_BIND_A (in_fence=fence_A)
>>VM_BIND_B (in_fence=NULL)
>>
>>With 1 queue VM_BIND_B in blocked on fence_A, hence the need for than 1
>>queue.
>
>I don't follow - there isn't a concept of a queue exposed in uapi in 
>what I have described so the above two run in parallel there, if we 
>ignore fence_A in your example doesn't even exist before you pass it 
>to bind A so something is not right.
>
>>e.g.
>>VM_BIND_A (queue_id=0, in_fence=fence_A)
>>VM_BIND_B (queue_id=1, in_fence=NULL)
>>
>>Now VM_BIND_B can immediately be executed regardless of fence_A status.
>
>In my examples userspace can serialise or not as it sees fit using 
>fences. The "parallel bind" examples two binds run in parallel. 
>Userspace can create multiple such parallel "queues" if it wanted.
>
>Parallel bind 1 and 2 interleaved:
>out_fence_A = vm_bind A
>out_fence_B = vm_bind B
>out_fence_C = vm_bind C
>out_fence_D = vm_bind D
>// all binds can run in parallel
>merged_fence_1 = fence_merge(out_fence_A, out_fence_B)
>merged_fence_2 = fence_merge(out_fence_C, out_fence_D)
>execbuf(in_fence = merged_fence_1) // after A&B to finish
>execbuf(in_fence = merged_fence_2) // after C&D finish
>
>There is a huge disconnect somewhere but I don't know where.
>

Note that Vulkan has requirement that VM_BIND and VM_UNBIND
operations will also have 'in' fences associated with them
and not just the 'out' fences (which your example above shows).

Yes, one of the solution discussed was not to have any queue_idx
at all (assume single queue) and let the vm_bind/unbind operations
submitted run and complete out of submission order. That way
a vm_bind/unbind sumitted later will not be blocked by a vm_bind/unbind
submitted earlier.
But removing the ordering here comes at a cost. Having the operations
run in submission order has some benefits. These are discussed in the
other thread.
https://lists.freedesktop.org/archives/intel-gfx/2022-June/299217.html

Hence having multiple queues gives the benefit of both worlds and gives
user more options.

Niranjana

>Regards,
>
>Tvrtko

Re: [Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Tvrtko Ursulin]

On 14/06/2022 16:43, Niranjana Vishwanathapura wrote:
> On Tue, Jun 14, 2022 at 08:16:41AM +0100, Tvrtko Ursulin wrote:
>>
>> On 14/06/2022 00:39, Matthew Brost wrote:
>>> On Mon, Jun 13, 2022 at 07:09:06PM +0100, Tvrtko Ursulin wrote:
>>>>
>>>> On 13/06/2022 18:49, Niranjana Vishwanathapura wrote:
>>>>> On Mon, Jun 13, 2022 at 05:22:02PM +0100, Tvrtko Ursulin wrote:
>>>>>>
>>>>>> On 13/06/2022 16:05, Niranjana Vishwanathapura wrote:
>>>>>>> On Mon, Jun 13, 2022 at 09:24:18AM +0100, Tvrtko Ursulin wrote:
>>>>>>>>
>>>>>>>> On 10/06/2022 17:14, Niranjana Vishwanathapura wrote:
>>>>>>>>> On Fri, Jun 10, 2022 at 05:48:39PM +0300, Lionel Landwerlin wrote:
>>>>>>>>>> On 10/06/2022 13:37, Tvrtko Ursulin wrote:
>>>>>>>>>>>
>>>>>>>>>>> On 10/06/2022 08:07, Niranjana Vishwanathapura wrote:
>>>>>>>>>>>> VM_BIND and related uapi definitions
>>>>>>>>>>>>
>>>>>>>>>>>> Signed-off-by: Niranjana Vishwanathapura
>>>>>>>>>>>> <niranjana.vishwanathapura@intel.com>
>>>>>>>>>>>> ---
>>>>>>>>>>>>   Documentation/gpu/rfc/i915_vm_bind.h | 490
>>>>>>>>>>>> +++++++++++++++++++++++++++
>>>>>>>>>>>>   1 file changed, 490 insertions(+)
>>>>>>>>>>>>   create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>>
>>>>>>>>>>>> diff --git
>>>>>>>>>>>> a/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>> b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>> new file mode 100644
>>>>>>>>>>>> index 000000000000..9fc854969cfb
>>>>>>>>>>>> --- /dev/null
>>>>>>>>>>>> +++ b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>> @@ -0,0 +1,490 @@
>>>>>>>>>>>> +/* SPDX-License-Identifier: MIT */
>>>>>>>>>>>> +/*
>>>>>>>>>>>> + * Copyright © 2022 Intel Corporation
>>>>>>>>>>>> + */
>>>>>>>>>>>> +
>>>>>>>>>>>> +/**
>>>>>>>>>>>> + * DOC: I915_PARAM_HAS_VM_BIND
>>>>>>>>>>>> + *
>>>>>>>>>>>> + * VM_BIND feature availability.
>>>>>>>>>>>> + * See typedef drm_i915_getparam_t param.
>>>>>>>>>>>> + * bit[0]: If set, VM_BIND is supported, otherwise not.
>>>>>>>>>>>> + * bits[8-15]: VM_BIND implementation version.
>>>>>>>>>>>> + * version 0 will not have VM_BIND/UNBIND
>>>>>>>>>>>> timeline fence array support.
>>>>>>>>>>>> + */
>>>>>>>>>>>> +#define I915_PARAM_HAS_VM_BIND        57
>>>>>>>>>>>> +
>>>>>>>>>>>> +/**
>>>>>>>>>>>> + * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
>>>>>>>>>>>> + *
>>>>>>>>>>>> + * Flag to opt-in for VM_BIND mode of binding during VM 
>>>>>>>>>>>> creation.
>>>>>>>>>>>> + * See struct drm_i915_gem_vm_control flags.
>>>>>>>>>>>> + *
>>>>>>>>>>>> + * The older execbuf2 ioctl will not
>>>>>>>>>>>> support VM_BIND mode of operation.
>>>>>>>>>>>> + * For VM_BIND mode, we have new execbuf3
>>>>>>>>>>>> ioctl which will not accept any
>>>>>>>>>>>> + * execlist (See struct
>>>>>>>>>>>> drm_i915_gem_execbuffer3 for more details).
>>>>>>>>>>>> + *
>>>>>>>>>>>> + */
>>>>>>>>>>>> +#define I915_VM_CREATE_FLAGS_USE_VM_BIND    (1 << 0)
>>>>>>>>>>>> +
>>>>>>>>>>>> +/**
>>>>>>>>>>>> + * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
>>>>>>>>>>>> + *
>>>>>>>>>>>> + * Flag to declare context as long running.
>>>>>>>>>>>> + * See struct drm_i915_gem_context_create_ext flags.
>>>>>>>>>>>> + *
>>>>>>>>>>>> + * Usage of dma-fence expects that they
>>>>>>>>>>>> complete in reasonable amount of time.
>>>>>>>>>>>> + * Compute on the other hand can be long
>>>>>>>>>>>> running. Hence it is not appropriate
>>>>>>>>>>>> + * for compute contexts to export request
>>>>>>>>>>>> completion dma-fence to user.
>>>>>>>>>>>> + * The dma-fence usage will be limited to
>>>>>>>>>>>> in-kernel consumption only.
>>>>>>>>>>>> + * Compute contexts need to use user/memory fence.
>>>>>>>>>>>> + *
>>>>>>>>>>>> + * So, long running contexts do not support output fences. 
>>>>>>>>>>>> Hence,
>>>>>>>>>>>> + * I915_EXEC_FENCE_SIGNAL (See
>>>>>>>>>>>> &drm_i915_gem_exec_fence.flags) is expected
>>>>>>>>>>>> + * to be not used. DRM_I915_GEM_WAIT ioctl
>>>>>>>>>>>> call is also not supported for
>>>>>>>>>>>> + * objects mapped to long running contexts.
>>>>>>>>>>>> + */
>>>>>>>>>>>> +#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u << 2)
>>>>>>>>>>>> +
>>>>>>>>>>>> +/* VM_BIND related ioctls */
>>>>>>>>>>>> +#define DRM_I915_GEM_VM_BIND        0x3d
>>>>>>>>>>>> +#define DRM_I915_GEM_VM_UNBIND        0x3e
>>>>>>>>>>>> +#define DRM_I915_GEM_EXECBUFFER3    0x3f
>>>>>>>>>>>> +#define DRM_I915_GEM_WAIT_USER_FENCE    0x40
>>>>>>>>>>>> +
>>>>>>>>>>>> +#define DRM_IOCTL_I915_GEM_VM_BIND
>>>>>>>>>>>> DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>> DRM_I915_GEM_VM_BIND, struct
>>>>>>>>>>>> drm_i915_gem_vm_bind)
>>>>>>>>>>>> +#define DRM_IOCTL_I915_GEM_VM_UNBIND
>>>>>>>>>>>> DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>> DRM_I915_GEM_VM_UNBIND, struct
>>>>>>>>>>>> drm_i915_gem_vm_bind)
>>>>>>>>>>>> +#define DRM_IOCTL_I915_GEM_EXECBUFFER3
>>>>>>>>>>>> DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>> DRM_I915_GEM_EXECBUFFER3, struct
>>>>>>>>>>>> drm_i915_gem_execbuffer3)
>>>>>>>>>>>> +#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE
>>>>>>>>>>>> DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>> DRM_I915_GEM_WAIT_USER_FENCE, struct
>>>>>>>>>>>> drm_i915_gem_wait_user_fence)
>>>>>>>>>>>> +
>>>>>>>>>>>> +/**
>>>>>>>>>>>> + * struct drm_i915_gem_vm_bind - VA to object mapping to bind.
>>>>>>>>>>>> + *
>>>>>>>>>>>> + * This structure is passed to VM_BIND
>>>>>>>>>>>> ioctl and specifies the mapping of GPU
>>>>>>>>>>>> + * virtual address (VA) range to the
>>>>>>>>>>>> section of an object that should be bound
>>>>>>>>>>>> + * in the device page table of the specified address space 
>>>>>>>>>>>> (VM).
>>>>>>>>>>>> + * The VA range specified must be unique
>>>>>>>>>>>> (ie., not currently bound) and can
>>>>>>>>>>>> + * be mapped to whole object or a section
>>>>>>>>>>>> of the object (partial binding).
>>>>>>>>>>>> + * Multiple VA mappings can be created to
>>>>>>>>>>>> the same section of the object
>>>>>>>>>>>> + * (aliasing).
>>>>>>>>>>>> + *
>>>>>>>>>>>> + * The @queue_idx specifies the queue to
>>>>>>>>>>>> use for binding. Same queue can be
>>>>>>>>>>>> + * used for both VM_BIND and VM_UNBIND
>>>>>>>>>>>> calls. All submitted bind and unbind
>>>>>>>>>>>> + * operations in a queue are performed in the order of 
>>>>>>>>>>>> submission.
>>>>>>>>>>>> + *
>>>>>>>>>>>> + * The @start, @offset and @length should
>>>>>>>>>>>> be 4K page aligned. However the DG2
>>>>>>>>>>>> + * and XEHPSDV has 64K page size for device
>>>>>>>>>>>> local-memory and has compact page
>>>>>>>>>>>> + * table. On those platforms, for binding
>>>>>>>>>>>> device local-memory objects, the
>>>>>>>>>>>> + * @start should be 2M aligned, @offset and
>>>>>>>>>>>> @length should be 64K aligned.
>>>>>>>>>>>> + * Also, on those platforms, it is not
>>>>>>>>>>>> allowed to bind an device local-memory
>>>>>>>>>>>> + * object and a system memory object in a
>>>>>>>>>>>> single 2M section of VA range.
>>>>>>>>>>>> + */
>>>>>>>>>>>> +struct drm_i915_gem_vm_bind {
>>>>>>>>>>>> +    /** @vm_id: VM (address space) id to bind */
>>>>>>>>>>>> +    __u32 vm_id;
>>>>>>>>>>>> +
>>>>>>>>>>>> +    /** @queue_idx: Index of queue for binding */
>>>>>>>>>>>> +    __u32 queue_idx;
>>>>>>>>>>>
>>>>>>>>>>> I have a question here to which I did not find
>>>>>>>>>>> an answer by browsing the old threads.
>>>>>>>>>>>
>>>>>>>>>>> Queue index appears to be an implicit
>>>>>>>>>>> synchronisation mechanism, right? Operations on
>>>>>>>>>>> the same index are executed/complete in order of
>>>>>>>>>>> ioctl submission?
>>>>>>>>>>>
>>>>>>>>>>> Do we _have_ to implement this on the kernel
>>>>>>>>>>> side and could just allow in/out fence and let
>>>>>>>>>>> userspace deal with it?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> It orders operations like in a queue. Which is kind
>>>>>>>>>> of what happens with existing queues/engines.
>>>>>>>>>>
>>>>>>>>>> If I understood correctly, it's going to be a
>>>>>>>>>> kthread + a linked list right?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yes, that is correct.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -Lionel
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Arbitrary/on-demand number of queues will add
>>>>>>>>>>> the complexity on the kernel side which should
>>>>>>>>>>> be avoided if possible.
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> It was discussed in the other thread. Jason prefers this over 
>>>>>>>>> putting
>>>>>>>>> an artificial limit on number of queues (as user can
>>>>>>>>> anyway can exhaust
>>>>>>>>> the memory). I think complexity in the driver is manageable.
>>>>>>>>
>>>>>>>> You'll need to create tracking structures on demand, with
>>>>>>>> atomic replace of last fence, ref counting and locking of
>>>>>>>> some sort, more or less?
>>>>>>>>
>>>>>>>
>>>>>>> We will have a workqueue, an work item and a linked list per queue.
>>>>>>> VM_BIND/UNBIND call will add the mapping request to the
>>>>>>> specified queue's
>>>>>>> linked list and schedule the work item on the workqueue of that 
>>>>>>> queue.
>>>>>>> I am not sure what you mean by last fence and replacing it.
>>>>>>>
>>>>>>>>> The other option being discussed in to have the user create those
>>>>>>>>> queues (like creating engine map) before hand and use that in 
>>>>>>>>> vm_bind
>>>>>>>>> and vm_unbind ioctls. This puts a limit on the number of queues.
>>>>>>>>> But it is not clean either and not sure it is worth
>>>>>>>>> making the interface
>>>>>>>>> more complex.
>>>>>>>>> https://www.spinics.net/lists/dri-devel/msg350448.html
>>>>>>>>
>>>>>>>> What about the third option of a flag to return a fence (of
>>>>>>>> some sort) and pass in a fence? That way userspace can
>>>>>>>> imagine zero or N queues with very little effort on the
>>>>>>>> kernel side. Was this considered?
>>>>>>>>
>>>>>>>
>>>>>>> I am not clear what fence you are talking about here and how does 
>>>>>>> that
>>>>>>> help with the number of vm_bind queues. Can you eloborate?
>>>>>>
>>>>>> It is actually already documented that bind/unbind will support
>>>>>> input and output fences - so what are these queues on top of what
>>>>>> userspace can already achieve by using them? Purely a convenience or
>>>>>> there is more to it?
>>>>>>
>>>>>
>>>>> Oh, the vm_bind queues are discussed in this thread.
>>>>> https://lists.freedesktop.org/archives/intel-gfx/2022-June/299217.html
>>>>>
>>>>> Apparently Vulkan has requirement for multiple queues, each queue
>>>>> processing vm_bind/unbind calls in the order of submission.
>>>>
>>>> I don't see how that answers my question so I will take the freedom to
>>>> repeat it. What are these queues on top of what userspace can already
>>>> achieve by using in-out fences? Purely a convenience or there is 
>>>> more to it?
>>>>
>>>> Queue1:
>>>>
>>>> out_fence_A = vm_bind A
>>>> out_fence_B = vm_bind B, in_fence=out_fence_A
>>>> execbuf(in_fence = out_fence_B)
>>>>
>>>> Queue2:
>>>>
>>>> out_fence_C = vm_bind C
>>>> out_fence_D = vm_bind D, in_fence=out_fence_C
>>>> execbuf(in_fence = out_fence_D)
>>>>
>>>> Parallel bind:
>>>> out_fence_E = vm_bind E
>>>> out_fence_F = vm_bind F
>>>> merged_fence = fence_merge(out_fence_E, out_fence_F)
>>>> execbuf(in_fence = merged_fence)
>>>>
>>>
>>> Let's say you do this and only 1 queue:
>>>
>>> VM_BIND_A (in_fence=fence_A)
>>> VM_BIND_B (in_fence=NULL)
>>>
>>> With 1 queue VM_BIND_B in blocked on fence_A, hence the need for than 1
>>> queue.
>>
>> I don't follow - there isn't a concept of a queue exposed in uapi in 
>> what I have described so the above two run in parallel there, if we 
>> ignore fence_A in your example doesn't even exist before you pass it 
>> to bind A so something is not right.
>>
>>> e.g.
>>> VM_BIND_A (queue_id=0, in_fence=fence_A)
>>> VM_BIND_B (queue_id=1, in_fence=NULL)
>>>
>>> Now VM_BIND_B can immediately be executed regardless of fence_A status.
>>
>> In my examples userspace can serialise or not as it sees fit using 
>> fences. The "parallel bind" examples two binds run in parallel. 
>> Userspace can create multiple such parallel "queues" if it wanted.
>>
>> Parallel bind 1 and 2 interleaved:
>> out_fence_A = vm_bind A
>> out_fence_B = vm_bind B
>> out_fence_C = vm_bind C
>> out_fence_D = vm_bind D
>> // all binds can run in parallel
>> merged_fence_1 = fence_merge(out_fence_A, out_fence_B)
>> merged_fence_2 = fence_merge(out_fence_C, out_fence_D)
>> execbuf(in_fence = merged_fence_1) // after A&B to finish
>> execbuf(in_fence = merged_fence_2) // after C&D finish
>>
>> There is a huge disconnect somewhere but I don't know where.
>>
> 
> Note that Vulkan has requirement that VM_BIND and VM_UNBIND
> operations will also have 'in' fences associated with them
> and not just the 'out' fences (which your example above shows).

I gave more examples earlier:

"""
Queue1:

out_fence_A = vm_bind A
out_fence_B = vm_bind B, in_fence=out_fence_A
execbuf(in_fence = out_fence_B)
"""

Clearly I showed both in and out fence.

> Yes, one of the solution discussed was not to have any queue_idx
> at all (assume single queue) and let the vm_bind/unbind operations
> submitted run and complete out of submission order. That way
> a vm_bind/unbind sumitted later will not be blocked by a vm_bind/unbind
> submitted earlier.
> But removing the ordering here comes at a cost. Having the operations
> run in submission order has some benefits. These are discussed in the
> other thread.
> https://lists.freedesktop.org/archives/intel-gfx/2022-June/299217.html

That is some messed up deep quoting in that link. Could you please 
summarize the cost which queues in the uapi intended to avoid?

In any case it is not just for me. A significant addition is proposed 
for the driver so there should be a clear summary of cost vs benefit 
rather than a messy thread.

> Hence having multiple queues gives the benefit of both worlds and gives
> user more options.

Maybe, but lets be specific.

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Tvrtko Ursulin]

On 14/06/2022 17:02, Tvrtko Ursulin wrote:
> 
> On 14/06/2022 16:43, Niranjana Vishwanathapura wrote:
>> On Tue, Jun 14, 2022 at 08:16:41AM +0100, Tvrtko Ursulin wrote:
>>>
>>> On 14/06/2022 00:39, Matthew Brost wrote:
>>>> On Mon, Jun 13, 2022 at 07:09:06PM +0100, Tvrtko Ursulin wrote:
>>>>>
>>>>> On 13/06/2022 18:49, Niranjana Vishwanathapura wrote:
>>>>>> On Mon, Jun 13, 2022 at 05:22:02PM +0100, Tvrtko Ursulin wrote:
>>>>>>>
>>>>>>> On 13/06/2022 16:05, Niranjana Vishwanathapura wrote:
>>>>>>>> On Mon, Jun 13, 2022 at 09:24:18AM +0100, Tvrtko Ursulin wrote:
>>>>>>>>>
>>>>>>>>> On 10/06/2022 17:14, Niranjana Vishwanathapura wrote:
>>>>>>>>>> On Fri, Jun 10, 2022 at 05:48:39PM +0300, Lionel Landwerlin 
>>>>>>>>>> wrote:
>>>>>>>>>>> On 10/06/2022 13:37, Tvrtko Ursulin wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> On 10/06/2022 08:07, Niranjana Vishwanathapura wrote:
>>>>>>>>>>>>> VM_BIND and related uapi definitions
>>>>>>>>>>>>>
>>>>>>>>>>>>> Signed-off-by: Niranjana Vishwanathapura
>>>>>>>>>>>>> <niranjana.vishwanathapura@intel.com>
>>>>>>>>>>>>> ---
>>>>>>>>>>>>>   Documentation/gpu/rfc/i915_vm_bind.h | 490
>>>>>>>>>>>>> +++++++++++++++++++++++++++
>>>>>>>>>>>>>   1 file changed, 490 insertions(+)
>>>>>>>>>>>>>   create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>>>
>>>>>>>>>>>>> diff --git
>>>>>>>>>>>>> a/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>>> b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>>> new file mode 100644
>>>>>>>>>>>>> index 000000000000..9fc854969cfb
>>>>>>>>>>>>> --- /dev/null
>>>>>>>>>>>>> +++ b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>>> @@ -0,0 +1,490 @@
>>>>>>>>>>>>> +/* SPDX-License-Identifier: MIT */
>>>>>>>>>>>>> +/*
>>>>>>>>>>>>> + * Copyright © 2022 Intel Corporation
>>>>>>>>>>>>> + */
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +/**
>>>>>>>>>>>>> + * DOC: I915_PARAM_HAS_VM_BIND
>>>>>>>>>>>>> + *
>>>>>>>>>>>>> + * VM_BIND feature availability.
>>>>>>>>>>>>> + * See typedef drm_i915_getparam_t param.
>>>>>>>>>>>>> + * bit[0]: If set, VM_BIND is supported, otherwise not.
>>>>>>>>>>>>> + * bits[8-15]: VM_BIND implementation version.
>>>>>>>>>>>>> + * version 0 will not have VM_BIND/UNBIND
>>>>>>>>>>>>> timeline fence array support.
>>>>>>>>>>>>> + */
>>>>>>>>>>>>> +#define I915_PARAM_HAS_VM_BIND        57
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +/**
>>>>>>>>>>>>> + * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
>>>>>>>>>>>>> + *
>>>>>>>>>>>>> + * Flag to opt-in for VM_BIND mode of binding during VM 
>>>>>>>>>>>>> creation.
>>>>>>>>>>>>> + * See struct drm_i915_gem_vm_control flags.
>>>>>>>>>>>>> + *
>>>>>>>>>>>>> + * The older execbuf2 ioctl will not
>>>>>>>>>>>>> support VM_BIND mode of operation.
>>>>>>>>>>>>> + * For VM_BIND mode, we have new execbuf3
>>>>>>>>>>>>> ioctl which will not accept any
>>>>>>>>>>>>> + * execlist (See struct
>>>>>>>>>>>>> drm_i915_gem_execbuffer3 for more details).
>>>>>>>>>>>>> + *
>>>>>>>>>>>>> + */
>>>>>>>>>>>>> +#define I915_VM_CREATE_FLAGS_USE_VM_BIND    (1 << 0)
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +/**
>>>>>>>>>>>>> + * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
>>>>>>>>>>>>> + *
>>>>>>>>>>>>> + * Flag to declare context as long running.
>>>>>>>>>>>>> + * See struct drm_i915_gem_context_create_ext flags.
>>>>>>>>>>>>> + *
>>>>>>>>>>>>> + * Usage of dma-fence expects that they
>>>>>>>>>>>>> complete in reasonable amount of time.
>>>>>>>>>>>>> + * Compute on the other hand can be long
>>>>>>>>>>>>> running. Hence it is not appropriate
>>>>>>>>>>>>> + * for compute contexts to export request
>>>>>>>>>>>>> completion dma-fence to user.
>>>>>>>>>>>>> + * The dma-fence usage will be limited to
>>>>>>>>>>>>> in-kernel consumption only.
>>>>>>>>>>>>> + * Compute contexts need to use user/memory fence.
>>>>>>>>>>>>> + *
>>>>>>>>>>>>> + * So, long running contexts do not support output fences. 
>>>>>>>>>>>>> Hence,
>>>>>>>>>>>>> + * I915_EXEC_FENCE_SIGNAL (See
>>>>>>>>>>>>> &drm_i915_gem_exec_fence.flags) is expected
>>>>>>>>>>>>> + * to be not used. DRM_I915_GEM_WAIT ioctl
>>>>>>>>>>>>> call is also not supported for
>>>>>>>>>>>>> + * objects mapped to long running contexts.
>>>>>>>>>>>>> + */
>>>>>>>>>>>>> +#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u << 2)
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +/* VM_BIND related ioctls */
>>>>>>>>>>>>> +#define DRM_I915_GEM_VM_BIND        0x3d
>>>>>>>>>>>>> +#define DRM_I915_GEM_VM_UNBIND        0x3e
>>>>>>>>>>>>> +#define DRM_I915_GEM_EXECBUFFER3    0x3f
>>>>>>>>>>>>> +#define DRM_I915_GEM_WAIT_USER_FENCE    0x40
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +#define DRM_IOCTL_I915_GEM_VM_BIND
>>>>>>>>>>>>> DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>>> DRM_I915_GEM_VM_BIND, struct
>>>>>>>>>>>>> drm_i915_gem_vm_bind)
>>>>>>>>>>>>> +#define DRM_IOCTL_I915_GEM_VM_UNBIND
>>>>>>>>>>>>> DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>>> DRM_I915_GEM_VM_UNBIND, struct
>>>>>>>>>>>>> drm_i915_gem_vm_bind)
>>>>>>>>>>>>> +#define DRM_IOCTL_I915_GEM_EXECBUFFER3
>>>>>>>>>>>>> DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>>> DRM_I915_GEM_EXECBUFFER3, struct
>>>>>>>>>>>>> drm_i915_gem_execbuffer3)
>>>>>>>>>>>>> +#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE
>>>>>>>>>>>>> DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>>> DRM_I915_GEM_WAIT_USER_FENCE, struct
>>>>>>>>>>>>> drm_i915_gem_wait_user_fence)
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +/**
>>>>>>>>>>>>> + * struct drm_i915_gem_vm_bind - VA to object mapping to 
>>>>>>>>>>>>> bind.
>>>>>>>>>>>>> + *
>>>>>>>>>>>>> + * This structure is passed to VM_BIND
>>>>>>>>>>>>> ioctl and specifies the mapping of GPU
>>>>>>>>>>>>> + * virtual address (VA) range to the
>>>>>>>>>>>>> section of an object that should be bound
>>>>>>>>>>>>> + * in the device page table of the specified address space 
>>>>>>>>>>>>> (VM).
>>>>>>>>>>>>> + * The VA range specified must be unique
>>>>>>>>>>>>> (ie., not currently bound) and can
>>>>>>>>>>>>> + * be mapped to whole object or a section
>>>>>>>>>>>>> of the object (partial binding).
>>>>>>>>>>>>> + * Multiple VA mappings can be created to
>>>>>>>>>>>>> the same section of the object
>>>>>>>>>>>>> + * (aliasing).
>>>>>>>>>>>>> + *
>>>>>>>>>>>>> + * The @queue_idx specifies the queue to
>>>>>>>>>>>>> use for binding. Same queue can be
>>>>>>>>>>>>> + * used for both VM_BIND and VM_UNBIND
>>>>>>>>>>>>> calls. All submitted bind and unbind
>>>>>>>>>>>>> + * operations in a queue are performed in the order of 
>>>>>>>>>>>>> submission.
>>>>>>>>>>>>> + *
>>>>>>>>>>>>> + * The @start, @offset and @length should
>>>>>>>>>>>>> be 4K page aligned. However the DG2
>>>>>>>>>>>>> + * and XEHPSDV has 64K page size for device
>>>>>>>>>>>>> local-memory and has compact page
>>>>>>>>>>>>> + * table. On those platforms, for binding
>>>>>>>>>>>>> device local-memory objects, the
>>>>>>>>>>>>> + * @start should be 2M aligned, @offset and
>>>>>>>>>>>>> @length should be 64K aligned.
>>>>>>>>>>>>> + * Also, on those platforms, it is not
>>>>>>>>>>>>> allowed to bind an device local-memory
>>>>>>>>>>>>> + * object and a system memory object in a
>>>>>>>>>>>>> single 2M section of VA range.
>>>>>>>>>>>>> + */
>>>>>>>>>>>>> +struct drm_i915_gem_vm_bind {
>>>>>>>>>>>>> +    /** @vm_id: VM (address space) id to bind */
>>>>>>>>>>>>> +    __u32 vm_id;
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +    /** @queue_idx: Index of queue for binding */
>>>>>>>>>>>>> +    __u32 queue_idx;
>>>>>>>>>>>>
>>>>>>>>>>>> I have a question here to which I did not find
>>>>>>>>>>>> an answer by browsing the old threads.
>>>>>>>>>>>>
>>>>>>>>>>>> Queue index appears to be an implicit
>>>>>>>>>>>> synchronisation mechanism, right? Operations on
>>>>>>>>>>>> the same index are executed/complete in order of
>>>>>>>>>>>> ioctl submission?
>>>>>>>>>>>>
>>>>>>>>>>>> Do we _have_ to implement this on the kernel
>>>>>>>>>>>> side and could just allow in/out fence and let
>>>>>>>>>>>> userspace deal with it?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> It orders operations like in a queue. Which is kind
>>>>>>>>>>> of what happens with existing queues/engines.
>>>>>>>>>>>
>>>>>>>>>>> If I understood correctly, it's going to be a
>>>>>>>>>>> kthread + a linked list right?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Yes, that is correct.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -Lionel
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Arbitrary/on-demand number of queues will add
>>>>>>>>>>>> the complexity on the kernel side which should
>>>>>>>>>>>> be avoided if possible.
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> It was discussed in the other thread. Jason prefers this over 
>>>>>>>>>> putting
>>>>>>>>>> an artificial limit on number of queues (as user can
>>>>>>>>>> anyway can exhaust
>>>>>>>>>> the memory). I think complexity in the driver is manageable.
>>>>>>>>>
>>>>>>>>> You'll need to create tracking structures on demand, with
>>>>>>>>> atomic replace of last fence, ref counting and locking of
>>>>>>>>> some sort, more or less?
>>>>>>>>>
>>>>>>>>
>>>>>>>> We will have a workqueue, an work item and a linked list per queue.
>>>>>>>> VM_BIND/UNBIND call will add the mapping request to the
>>>>>>>> specified queue's
>>>>>>>> linked list and schedule the work item on the workqueue of that 
>>>>>>>> queue.
>>>>>>>> I am not sure what you mean by last fence and replacing it.
>>>>>>>>
>>>>>>>>>> The other option being discussed in to have the user create those
>>>>>>>>>> queues (like creating engine map) before hand and use that in 
>>>>>>>>>> vm_bind
>>>>>>>>>> and vm_unbind ioctls. This puts a limit on the number of queues.
>>>>>>>>>> But it is not clean either and not sure it is worth
>>>>>>>>>> making the interface
>>>>>>>>>> more complex.
>>>>>>>>>> https://www.spinics.net/lists/dri-devel/msg350448.html
>>>>>>>>>
>>>>>>>>> What about the third option of a flag to return a fence (of
>>>>>>>>> some sort) and pass in a fence? That way userspace can
>>>>>>>>> imagine zero or N queues with very little effort on the
>>>>>>>>> kernel side. Was this considered?
>>>>>>>>>
>>>>>>>>
>>>>>>>> I am not clear what fence you are talking about here and how 
>>>>>>>> does that
>>>>>>>> help with the number of vm_bind queues. Can you eloborate?
>>>>>>>
>>>>>>> It is actually already documented that bind/unbind will support
>>>>>>> input and output fences - so what are these queues on top of what
>>>>>>> userspace can already achieve by using them? Purely a convenience or
>>>>>>> there is more to it?
>>>>>>>
>>>>>>
>>>>>> Oh, the vm_bind queues are discussed in this thread.
>>>>>> https://lists.freedesktop.org/archives/intel-gfx/2022-June/299217.html 
>>>>>>
>>>>>>
>>>>>> Apparently Vulkan has requirement for multiple queues, each queue
>>>>>> processing vm_bind/unbind calls in the order of submission.
>>>>>
>>>>> I don't see how that answers my question so I will take the freedom to
>>>>> repeat it. What are these queues on top of what userspace can already
>>>>> achieve by using in-out fences? Purely a convenience or there is 
>>>>> more to it?
>>>>>
>>>>> Queue1:
>>>>>
>>>>> out_fence_A = vm_bind A
>>>>> out_fence_B = vm_bind B, in_fence=out_fence_A
>>>>> execbuf(in_fence = out_fence_B)
>>>>>
>>>>> Queue2:
>>>>>
>>>>> out_fence_C = vm_bind C
>>>>> out_fence_D = vm_bind D, in_fence=out_fence_C
>>>>> execbuf(in_fence = out_fence_D)
>>>>>
>>>>> Parallel bind:
>>>>> out_fence_E = vm_bind E
>>>>> out_fence_F = vm_bind F
>>>>> merged_fence = fence_merge(out_fence_E, out_fence_F)
>>>>> execbuf(in_fence = merged_fence)
>>>>>
>>>>
>>>> Let's say you do this and only 1 queue:
>>>>
>>>> VM_BIND_A (in_fence=fence_A)
>>>> VM_BIND_B (in_fence=NULL)
>>>>
>>>> With 1 queue VM_BIND_B in blocked on fence_A, hence the need for than 1
>>>> queue.
>>>
>>> I don't follow - there isn't a concept of a queue exposed in uapi in 
>>> what I have described so the above two run in parallel there, if we 
>>> ignore fence_A in your example doesn't even exist before you pass it 
>>> to bind A so something is not right.
>>>
>>>> e.g.
>>>> VM_BIND_A (queue_id=0, in_fence=fence_A)
>>>> VM_BIND_B (queue_id=1, in_fence=NULL)
>>>>
>>>> Now VM_BIND_B can immediately be executed regardless of fence_A status.
>>>
>>> In my examples userspace can serialise or not as it sees fit using 
>>> fences. The "parallel bind" examples two binds run in parallel. 
>>> Userspace can create multiple such parallel "queues" if it wanted.
>>>
>>> Parallel bind 1 and 2 interleaved:
>>> out_fence_A = vm_bind A
>>> out_fence_B = vm_bind B
>>> out_fence_C = vm_bind C
>>> out_fence_D = vm_bind D
>>> // all binds can run in parallel
>>> merged_fence_1 = fence_merge(out_fence_A, out_fence_B)
>>> merged_fence_2 = fence_merge(out_fence_C, out_fence_D)
>>> execbuf(in_fence = merged_fence_1) // after A&B to finish
>>> execbuf(in_fence = merged_fence_2) // after C&D finish
>>>
>>> There is a huge disconnect somewhere but I don't know where.
>>>
>>
>> Note that Vulkan has requirement that VM_BIND and VM_UNBIND
>> operations will also have 'in' fences associated with them
>> and not just the 'out' fences (which your example above shows).
> 
> I gave more examples earlier:
> 
> """
> Queue1:
> 
> out_fence_A = vm_bind A
> out_fence_B = vm_bind B, in_fence=out_fence_A
> execbuf(in_fence = out_fence_B)
> """
> 
> Clearly I showed both in and out fence.
> 
>> Yes, one of the solution discussed was not to have any queue_idx
>> at all (assume single queue) and let the vm_bind/unbind operations
>> submitted run and complete out of submission order. That way
>> a vm_bind/unbind sumitted later will not be blocked by a vm_bind/unbind
>> submitted earlier.
>> But removing the ordering here comes at a cost. Having the operations
>> run in submission order has some benefits. These are discussed in the
>> other thread.
>> https://lists.freedesktop.org/archives/intel-gfx/2022-June/299217.html
> 
> That is some messed up deep quoting in that link. Could you please 
> summarize the cost which queues in the uapi intended to avoid?
> 
> In any case it is not just for me. A significant addition is proposed 
> for the driver so there should be a clear summary of cost vs benefit 
> rather than a messy thread.
> 
>> Hence having multiple queues gives the benefit of both worlds and gives
>> user more options.
> 
> Maybe, but lets be specific.

Also, it can't really give the benefit of both worlds. You can't go 
fully async with the queue_idx scheme without i915 having to create N 
internal queues (where N = number of bind operations submitted).

Therefore I suspect it's a trade-off between cost and convenience but 
I'd like things clearly summarized so decision can be made.

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Niranjana Vishwanathapura]

On Tue, Jun 14, 2022 at 05:07:37PM +0100, Tvrtko Ursulin wrote:
>
>On 14/06/2022 17:02, Tvrtko Ursulin wrote:
>>
>>On 14/06/2022 16:43, Niranjana Vishwanathapura wrote:
>>>On Tue, Jun 14, 2022 at 08:16:41AM +0100, Tvrtko Ursulin wrote:
>>>>
>>>>On 14/06/2022 00:39, Matthew Brost wrote:
>>>>>On Mon, Jun 13, 2022 at 07:09:06PM +0100, Tvrtko Ursulin wrote:
>>>>>>
>>>>>>On 13/06/2022 18:49, Niranjana Vishwanathapura wrote:
>>>>>>>On Mon, Jun 13, 2022 at 05:22:02PM +0100, Tvrtko Ursulin wrote:
>>>>>>>>
>>>>>>>>On 13/06/2022 16:05, Niranjana Vishwanathapura wrote:
>>>>>>>>>On Mon, Jun 13, 2022 at 09:24:18AM +0100, Tvrtko Ursulin wrote:
>>>>>>>>>>
>>>>>>>>>>On 10/06/2022 17:14, Niranjana Vishwanathapura wrote:
>>>>>>>>>>>On Fri, Jun 10, 2022 at 05:48:39PM +0300, Lionel 
>>>>>>>>>>>Landwerlin wrote:
>>>>>>>>>>>>On 10/06/2022 13:37, Tvrtko Ursulin wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>On 10/06/2022 08:07, Niranjana Vishwanathapura wrote:
>>>>>>>>>>>>>>VM_BIND and related uapi definitions
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>Signed-off-by: Niranjana Vishwanathapura
>>>>>>>>>>>>>><niranjana.vishwanathapura@intel.com>
>>>>>>>>>>>>>>---
>>>>>>>>>>>>>>  Documentation/gpu/rfc/i915_vm_bind.h | 490
>>>>>>>>>>>>>>+++++++++++++++++++++++++++
>>>>>>>>>>>>>>  1 file changed, 490 insertions(+)
>>>>>>>>>>>>>>  create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>diff --git
>>>>>>>>>>>>>>a/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>>>>b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>>>>new file mode 100644
>>>>>>>>>>>>>>index 000000000000..9fc854969cfb
>>>>>>>>>>>>>>--- /dev/null
>>>>>>>>>>>>>>+++ b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>>>>@@ -0,0 +1,490 @@
>>>>>>>>>>>>>>+/* SPDX-License-Identifier: MIT */
>>>>>>>>>>>>>>+/*
>>>>>>>>>>>>>>+ * Copyright © 2022 Intel Corporation
>>>>>>>>>>>>>>+ */
>>>>>>>>>>>>>>+
>>>>>>>>>>>>>>+/**
>>>>>>>>>>>>>>+ * DOC: I915_PARAM_HAS_VM_BIND
>>>>>>>>>>>>>>+ *
>>>>>>>>>>>>>>+ * VM_BIND feature availability.
>>>>>>>>>>>>>>+ * See typedef drm_i915_getparam_t param.
>>>>>>>>>>>>>>+ * bit[0]: If set, VM_BIND is supported, otherwise not.
>>>>>>>>>>>>>>+ * bits[8-15]: VM_BIND implementation version.
>>>>>>>>>>>>>>+ * version 0 will not have VM_BIND/UNBIND
>>>>>>>>>>>>>>timeline fence array support.
>>>>>>>>>>>>>>+ */
>>>>>>>>>>>>>>+#define I915_PARAM_HAS_VM_BIND        57
>>>>>>>>>>>>>>+
>>>>>>>>>>>>>>+/**
>>>>>>>>>>>>>>+ * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
>>>>>>>>>>>>>>+ *
>>>>>>>>>>>>>>+ * Flag to opt-in for VM_BIND mode of 
>>>>>>>>>>>>>>binding during VM creation.
>>>>>>>>>>>>>>+ * See struct drm_i915_gem_vm_control flags.
>>>>>>>>>>>>>>+ *
>>>>>>>>>>>>>>+ * The older execbuf2 ioctl will not
>>>>>>>>>>>>>>support VM_BIND mode of operation.
>>>>>>>>>>>>>>+ * For VM_BIND mode, we have new execbuf3
>>>>>>>>>>>>>>ioctl which will not accept any
>>>>>>>>>>>>>>+ * execlist (See struct
>>>>>>>>>>>>>>drm_i915_gem_execbuffer3 for more details).
>>>>>>>>>>>>>>+ *
>>>>>>>>>>>>>>+ */
>>>>>>>>>>>>>>+#define I915_VM_CREATE_FLAGS_USE_VM_BIND    (1 << 0)
>>>>>>>>>>>>>>+
>>>>>>>>>>>>>>+/**
>>>>>>>>>>>>>>+ * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
>>>>>>>>>>>>>>+ *
>>>>>>>>>>>>>>+ * Flag to declare context as long running.
>>>>>>>>>>>>>>+ * See struct drm_i915_gem_context_create_ext flags.
>>>>>>>>>>>>>>+ *
>>>>>>>>>>>>>>+ * Usage of dma-fence expects that they
>>>>>>>>>>>>>>complete in reasonable amount of time.
>>>>>>>>>>>>>>+ * Compute on the other hand can be long
>>>>>>>>>>>>>>running. Hence it is not appropriate
>>>>>>>>>>>>>>+ * for compute contexts to export request
>>>>>>>>>>>>>>completion dma-fence to user.
>>>>>>>>>>>>>>+ * The dma-fence usage will be limited to
>>>>>>>>>>>>>>in-kernel consumption only.
>>>>>>>>>>>>>>+ * Compute contexts need to use user/memory fence.
>>>>>>>>>>>>>>+ *
>>>>>>>>>>>>>>+ * So, long running contexts do not support 
>>>>>>>>>>>>>>output fences. Hence,
>>>>>>>>>>>>>>+ * I915_EXEC_FENCE_SIGNAL (See
>>>>>>>>>>>>>>&drm_i915_gem_exec_fence.flags) is expected
>>>>>>>>>>>>>>+ * to be not used. DRM_I915_GEM_WAIT ioctl
>>>>>>>>>>>>>>call is also not supported for
>>>>>>>>>>>>>>+ * objects mapped to long running contexts.
>>>>>>>>>>>>>>+ */
>>>>>>>>>>>>>>+#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u << 2)
>>>>>>>>>>>>>>+
>>>>>>>>>>>>>>+/* VM_BIND related ioctls */
>>>>>>>>>>>>>>+#define DRM_I915_GEM_VM_BIND        0x3d
>>>>>>>>>>>>>>+#define DRM_I915_GEM_VM_UNBIND        0x3e
>>>>>>>>>>>>>>+#define DRM_I915_GEM_EXECBUFFER3    0x3f
>>>>>>>>>>>>>>+#define DRM_I915_GEM_WAIT_USER_FENCE    0x40
>>>>>>>>>>>>>>+
>>>>>>>>>>>>>>+#define DRM_IOCTL_I915_GEM_VM_BIND
>>>>>>>>>>>>>>DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>>>>DRM_I915_GEM_VM_BIND, struct
>>>>>>>>>>>>>>drm_i915_gem_vm_bind)
>>>>>>>>>>>>>>+#define DRM_IOCTL_I915_GEM_VM_UNBIND
>>>>>>>>>>>>>>DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>>>>DRM_I915_GEM_VM_UNBIND, struct
>>>>>>>>>>>>>>drm_i915_gem_vm_bind)
>>>>>>>>>>>>>>+#define DRM_IOCTL_I915_GEM_EXECBUFFER3
>>>>>>>>>>>>>>DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>>>>DRM_I915_GEM_EXECBUFFER3, struct
>>>>>>>>>>>>>>drm_i915_gem_execbuffer3)
>>>>>>>>>>>>>>+#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE
>>>>>>>>>>>>>>DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>>>>DRM_I915_GEM_WAIT_USER_FENCE, struct
>>>>>>>>>>>>>>drm_i915_gem_wait_user_fence)
>>>>>>>>>>>>>>+
>>>>>>>>>>>>>>+/**
>>>>>>>>>>>>>>+ * struct drm_i915_gem_vm_bind - VA to 
>>>>>>>>>>>>>>object mapping to bind.
>>>>>>>>>>>>>>+ *
>>>>>>>>>>>>>>+ * This structure is passed to VM_BIND
>>>>>>>>>>>>>>ioctl and specifies the mapping of GPU
>>>>>>>>>>>>>>+ * virtual address (VA) range to the
>>>>>>>>>>>>>>section of an object that should be bound
>>>>>>>>>>>>>>+ * in the device page table of the 
>>>>>>>>>>>>>>specified address space (VM).
>>>>>>>>>>>>>>+ * The VA range specified must be unique
>>>>>>>>>>>>>>(ie., not currently bound) and can
>>>>>>>>>>>>>>+ * be mapped to whole object or a section
>>>>>>>>>>>>>>of the object (partial binding).
>>>>>>>>>>>>>>+ * Multiple VA mappings can be created to
>>>>>>>>>>>>>>the same section of the object
>>>>>>>>>>>>>>+ * (aliasing).
>>>>>>>>>>>>>>+ *
>>>>>>>>>>>>>>+ * The @queue_idx specifies the queue to
>>>>>>>>>>>>>>use for binding. Same queue can be
>>>>>>>>>>>>>>+ * used for both VM_BIND and VM_UNBIND
>>>>>>>>>>>>>>calls. All submitted bind and unbind
>>>>>>>>>>>>>>+ * operations in a queue are performed in 
>>>>>>>>>>>>>>the order of submission.
>>>>>>>>>>>>>>+ *
>>>>>>>>>>>>>>+ * The @start, @offset and @length should
>>>>>>>>>>>>>>be 4K page aligned. However the DG2
>>>>>>>>>>>>>>+ * and XEHPSDV has 64K page size for device
>>>>>>>>>>>>>>local-memory and has compact page
>>>>>>>>>>>>>>+ * table. On those platforms, for binding
>>>>>>>>>>>>>>device local-memory objects, the
>>>>>>>>>>>>>>+ * @start should be 2M aligned, @offset and
>>>>>>>>>>>>>>@length should be 64K aligned.
>>>>>>>>>>>>>>+ * Also, on those platforms, it is not
>>>>>>>>>>>>>>allowed to bind an device local-memory
>>>>>>>>>>>>>>+ * object and a system memory object in a
>>>>>>>>>>>>>>single 2M section of VA range.
>>>>>>>>>>>>>>+ */
>>>>>>>>>>>>>>+struct drm_i915_gem_vm_bind {
>>>>>>>>>>>>>>+    /** @vm_id: VM (address space) id to bind */
>>>>>>>>>>>>>>+    __u32 vm_id;
>>>>>>>>>>>>>>+
>>>>>>>>>>>>>>+    /** @queue_idx: Index of queue for binding */
>>>>>>>>>>>>>>+    __u32 queue_idx;
>>>>>>>>>>>>>
>>>>>>>>>>>>>I have a question here to which I did not find
>>>>>>>>>>>>>an answer by browsing the old threads.
>>>>>>>>>>>>>
>>>>>>>>>>>>>Queue index appears to be an implicit
>>>>>>>>>>>>>synchronisation mechanism, right? Operations on
>>>>>>>>>>>>>the same index are executed/complete in order of
>>>>>>>>>>>>>ioctl submission?
>>>>>>>>>>>>>
>>>>>>>>>>>>>Do we _have_ to implement this on the kernel
>>>>>>>>>>>>>side and could just allow in/out fence and let
>>>>>>>>>>>>>userspace deal with it?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>It orders operations like in a queue. Which is kind
>>>>>>>>>>>>of what happens with existing queues/engines.
>>>>>>>>>>>>
>>>>>>>>>>>>If I understood correctly, it's going to be a
>>>>>>>>>>>>kthread + a linked list right?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>Yes, that is correct.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>-Lionel
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>Arbitrary/on-demand number of queues will add
>>>>>>>>>>>>>the complexity on the kernel side which should
>>>>>>>>>>>>>be avoided if possible.
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>It was discussed in the other thread. Jason 
>>>>>>>>>>>prefers this over putting
>>>>>>>>>>>an artificial limit on number of queues (as user can
>>>>>>>>>>>anyway can exhaust
>>>>>>>>>>>the memory). I think complexity in the driver is manageable.
>>>>>>>>>>
>>>>>>>>>>You'll need to create tracking structures on demand, with
>>>>>>>>>>atomic replace of last fence, ref counting and locking of
>>>>>>>>>>some sort, more or less?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>We will have a workqueue, an work item and a linked list per queue.
>>>>>>>>>VM_BIND/UNBIND call will add the mapping request to the
>>>>>>>>>specified queue's
>>>>>>>>>linked list and schedule the work item on the 
>>>>>>>>>workqueue of that queue.
>>>>>>>>>I am not sure what you mean by last fence and replacing it.
>>>>>>>>>
>>>>>>>>>>>The other option being discussed in to have the user create those
>>>>>>>>>>>queues (like creating engine map) before hand and 
>>>>>>>>>>>use that in vm_bind
>>>>>>>>>>>and vm_unbind ioctls. This puts a limit on the number of queues.
>>>>>>>>>>>But it is not clean either and not sure it is worth
>>>>>>>>>>>making the interface
>>>>>>>>>>>more complex.
>>>>>>>>>>>https://www.spinics.net/lists/dri-devel/msg350448.html
>>>>>>>>>>
>>>>>>>>>>What about the third option of a flag to return a fence (of
>>>>>>>>>>some sort) and pass in a fence? That way userspace can
>>>>>>>>>>imagine zero or N queues with very little effort on the
>>>>>>>>>>kernel side. Was this considered?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>I am not clear what fence you are talking about here 
>>>>>>>>>and how does that
>>>>>>>>>help with the number of vm_bind queues. Can you eloborate?
>>>>>>>>
>>>>>>>>It is actually already documented that bind/unbind will support
>>>>>>>>input and output fences - so what are these queues on top of what
>>>>>>>>userspace can already achieve by using them? Purely a convenience or
>>>>>>>>there is more to it?
>>>>>>>>
>>>>>>>
>>>>>>>Oh, the vm_bind queues are discussed in this thread.
>>>>>>>https://lists.freedesktop.org/archives/intel-gfx/2022-June/299217.html
>>>>>>>
>>>>>>>
>>>>>>>Apparently Vulkan has requirement for multiple queues, each queue
>>>>>>>processing vm_bind/unbind calls in the order of submission.
>>>>>>
>>>>>>I don't see how that answers my question so I will take the freedom to
>>>>>>repeat it. What are these queues on top of what userspace can already
>>>>>>achieve by using in-out fences? Purely a convenience or 
>>>>>>there is more to it?
>>>>>>
>>>>>>Queue1:
>>>>>>
>>>>>>out_fence_A = vm_bind A
>>>>>>out_fence_B = vm_bind B, in_fence=out_fence_A
>>>>>>execbuf(in_fence = out_fence_B)
>>>>>>
>>>>>>Queue2:
>>>>>>
>>>>>>out_fence_C = vm_bind C
>>>>>>out_fence_D = vm_bind D, in_fence=out_fence_C
>>>>>>execbuf(in_fence = out_fence_D)
>>>>>>
>>>>>>Parallel bind:
>>>>>>out_fence_E = vm_bind E
>>>>>>out_fence_F = vm_bind F
>>>>>>merged_fence = fence_merge(out_fence_E, out_fence_F)
>>>>>>execbuf(in_fence = merged_fence)
>>>>>>
>>>>>
>>>>>Let's say you do this and only 1 queue:
>>>>>
>>>>>VM_BIND_A (in_fence=fence_A)
>>>>>VM_BIND_B (in_fence=NULL)
>>>>>
>>>>>With 1 queue VM_BIND_B in blocked on fence_A, hence the need for than 1
>>>>>queue.
>>>>
>>>>I don't follow - there isn't a concept of a queue exposed in 
>>>>uapi in what I have described so the above two run in parallel 
>>>>there, if we ignore fence_A in your example doesn't even exist 
>>>>before you pass it to bind A so something is not right.
>>>>
>>>>>e.g.
>>>>>VM_BIND_A (queue_id=0, in_fence=fence_A)
>>>>>VM_BIND_B (queue_id=1, in_fence=NULL)
>>>>>
>>>>>Now VM_BIND_B can immediately be executed regardless of fence_A status.
>>>>
>>>>In my examples userspace can serialise or not as it sees fit 
>>>>using fences. The "parallel bind" examples two binds run in 
>>>>parallel. Userspace can create multiple such parallel "queues" 
>>>>if it wanted.
>>>>
>>>>Parallel bind 1 and 2 interleaved:
>>>>out_fence_A = vm_bind A
>>>>out_fence_B = vm_bind B
>>>>out_fence_C = vm_bind C
>>>>out_fence_D = vm_bind D
>>>>// all binds can run in parallel
>>>>merged_fence_1 = fence_merge(out_fence_A, out_fence_B)
>>>>merged_fence_2 = fence_merge(out_fence_C, out_fence_D)
>>>>execbuf(in_fence = merged_fence_1) // after A&B to finish
>>>>execbuf(in_fence = merged_fence_2) // after C&D finish
>>>>
>>>>There is a huge disconnect somewhere but I don't know where.
>>>>
>>>
>>>Note that Vulkan has requirement that VM_BIND and VM_UNBIND
>>>operations will also have 'in' fences associated with them
>>>and not just the 'out' fences (which your example above shows).
>>
>>I gave more examples earlier:
>>
>>"""
>>Queue1:
>>
>>out_fence_A = vm_bind A
>>out_fence_B = vm_bind B, in_fence=out_fence_A
>>execbuf(in_fence = out_fence_B)
>>"""
>>
>>Clearly I showed both in and out fence.
>>

Ok, guess I missed that.

>>>Yes, one of the solution discussed was not to have any queue_idx
>>>at all (assume single queue) and let the vm_bind/unbind operations
>>>submitted run and complete out of submission order. That way
>>>a vm_bind/unbind sumitted later will not be blocked by a vm_bind/unbind
>>>submitted earlier.
>>>But removing the ordering here comes at a cost. Having the operations
>>>run in submission order has some benefits. These are discussed in the
>>>other thread.
>>>https://lists.freedesktop.org/archives/intel-gfx/2022-June/299217.html
>>
>>That is some messed up deep quoting in that link. Could you please 
>>summarize the cost which queues in the uapi intended to avoid?
>>
>>In any case it is not just for me. A significant addition is 
>>proposed for the driver so there should be a clear summary of cost 
>>vs benefit rather than a messy thread.
>>

Say, user has a bunch of mappings to bind or unbind which must be done
in the submission order. If we have only one queue which runs the
operations out of submission order, then user has to insert in and out
fences for each of the operation in the bunch. But by having a in order
processing queues, user needs to insert 'in' fence only for the first
submission and 'out' fence only for the last submission in that bunch.

Also, having in order processing queues allows user to unbind a VA
mapping and re-use the same VA in a subsequent bind operation without
having any dependency (dependency is met by the fact that they are
process in the submission order).

>>>Hence having multiple queues gives the benefit of both worlds and gives
>>>user more options.
>>
>>Maybe, but lets be specific.
>
>Also, it can't really give the benefit of both worlds. You can't go 
>fully async with the queue_idx scheme without i915 having to create N 
>internal queues (where N = number of bind operations submitted).
>

We will have 1 internal queue for each queue_idx used by the user as
each queue is an ordered queue (processed in the submission order).

Niranjana

>Therefore I suspect it's a trade-off between cost and convenience but 
>I'd like things clearly summarized so decision can be made.
>
>Regards,
>
>Tvrtko

Re: [Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Tvrtko Ursulin]

On 14/06/2022 17:42, Niranjana Vishwanathapura wrote:
> On Tue, Jun 14, 2022 at 05:07:37PM +0100, Tvrtko Ursulin wrote:
>>
>> On 14/06/2022 17:02, Tvrtko Ursulin wrote:
>>>
>>> On 14/06/2022 16:43, Niranjana Vishwanathapura wrote:
>>>> On Tue, Jun 14, 2022 at 08:16:41AM +0100, Tvrtko Ursulin wrote:
>>>>>
>>>>> On 14/06/2022 00:39, Matthew Brost wrote:
>>>>>> On Mon, Jun 13, 2022 at 07:09:06PM +0100, Tvrtko Ursulin wrote:
>>>>>>>
>>>>>>> On 13/06/2022 18:49, Niranjana Vishwanathapura wrote:
>>>>>>>> On Mon, Jun 13, 2022 at 05:22:02PM +0100, Tvrtko Ursulin wrote:
>>>>>>>>>
>>>>>>>>> On 13/06/2022 16:05, Niranjana Vishwanathapura wrote:
>>>>>>>>>> On Mon, Jun 13, 2022 at 09:24:18AM +0100, Tvrtko Ursulin wrote:
>>>>>>>>>>>
>>>>>>>>>>> On 10/06/2022 17:14, Niranjana Vishwanathapura wrote:
>>>>>>>>>>>> On Fri, Jun 10, 2022 at 05:48:39PM +0300, Lionel Landwerlin 
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> On 10/06/2022 13:37, Tvrtko Ursulin wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 10/06/2022 08:07, Niranjana Vishwanathapura wrote:
>>>>>>>>>>>>>>> VM_BIND and related uapi definitions
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Signed-off-by: Niranjana Vishwanathapura
>>>>>>>>>>>>>>> <niranjana.vishwanathapura@intel.com>
>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>   Documentation/gpu/rfc/i915_vm_bind.h | 490
>>>>>>>>>>>>>>> +++++++++++++++++++++++++++
>>>>>>>>>>>>>>>   1 file changed, 490 insertions(+)
>>>>>>>>>>>>>>>   create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> diff --git
>>>>>>>>>>>>>>> a/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>>>>> b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>>>>> new file mode 100644
>>>>>>>>>>>>>>> index 000000000000..9fc854969cfb
>>>>>>>>>>>>>>> --- /dev/null
>>>>>>>>>>>>>>> +++ b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>>>>> @@ -0,0 +1,490 @@
>>>>>>>>>>>>>>> +/* SPDX-License-Identifier: MIT */
>>>>>>>>>>>>>>> +/*
>>>>>>>>>>>>>>> + * Copyright © 2022 Intel Corporation
>>>>>>>>>>>>>>> + */
>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>> +/**
>>>>>>>>>>>>>>> + * DOC: I915_PARAM_HAS_VM_BIND
>>>>>>>>>>>>>>> + *
>>>>>>>>>>>>>>> + * VM_BIND feature availability.
>>>>>>>>>>>>>>> + * See typedef drm_i915_getparam_t param.
>>>>>>>>>>>>>>> + * bit[0]: If set, VM_BIND is supported, otherwise not.
>>>>>>>>>>>>>>> + * bits[8-15]: VM_BIND implementation version.
>>>>>>>>>>>>>>> + * version 0 will not have VM_BIND/UNBIND
>>>>>>>>>>>>>>> timeline fence array support.
>>>>>>>>>>>>>>> + */
>>>>>>>>>>>>>>> +#define I915_PARAM_HAS_VM_BIND        57
>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>> +/**
>>>>>>>>>>>>>>> + * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
>>>>>>>>>>>>>>> + *
>>>>>>>>>>>>>>> + * Flag to opt-in for VM_BIND mode of binding during VM 
>>>>>>>>>>>>>>> creation.
>>>>>>>>>>>>>>> + * See struct drm_i915_gem_vm_control flags.
>>>>>>>>>>>>>>> + *
>>>>>>>>>>>>>>> + * The older execbuf2 ioctl will not
>>>>>>>>>>>>>>> support VM_BIND mode of operation.
>>>>>>>>>>>>>>> + * For VM_BIND mode, we have new execbuf3
>>>>>>>>>>>>>>> ioctl which will not accept any
>>>>>>>>>>>>>>> + * execlist (See struct
>>>>>>>>>>>>>>> drm_i915_gem_execbuffer3 for more details).
>>>>>>>>>>>>>>> + *
>>>>>>>>>>>>>>> + */
>>>>>>>>>>>>>>> +#define I915_VM_CREATE_FLAGS_USE_VM_BIND    (1 << 0)
>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>> +/**
>>>>>>>>>>>>>>> + * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
>>>>>>>>>>>>>>> + *
>>>>>>>>>>>>>>> + * Flag to declare context as long running.
>>>>>>>>>>>>>>> + * See struct drm_i915_gem_context_create_ext flags.
>>>>>>>>>>>>>>> + *
>>>>>>>>>>>>>>> + * Usage of dma-fence expects that they
>>>>>>>>>>>>>>> complete in reasonable amount of time.
>>>>>>>>>>>>>>> + * Compute on the other hand can be long
>>>>>>>>>>>>>>> running. Hence it is not appropriate
>>>>>>>>>>>>>>> + * for compute contexts to export request
>>>>>>>>>>>>>>> completion dma-fence to user.
>>>>>>>>>>>>>>> + * The dma-fence usage will be limited to
>>>>>>>>>>>>>>> in-kernel consumption only.
>>>>>>>>>>>>>>> + * Compute contexts need to use user/memory fence.
>>>>>>>>>>>>>>> + *
>>>>>>>>>>>>>>> + * So, long running contexts do not support output 
>>>>>>>>>>>>>>> fences. Hence,
>>>>>>>>>>>>>>> + * I915_EXEC_FENCE_SIGNAL (See
>>>>>>>>>>>>>>> &drm_i915_gem_exec_fence.flags) is expected
>>>>>>>>>>>>>>> + * to be not used. DRM_I915_GEM_WAIT ioctl
>>>>>>>>>>>>>>> call is also not supported for
>>>>>>>>>>>>>>> + * objects mapped to long running contexts.
>>>>>>>>>>>>>>> + */
>>>>>>>>>>>>>>> +#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u << 2)
>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>> +/* VM_BIND related ioctls */
>>>>>>>>>>>>>>> +#define DRM_I915_GEM_VM_BIND        0x3d
>>>>>>>>>>>>>>> +#define DRM_I915_GEM_VM_UNBIND        0x3e
>>>>>>>>>>>>>>> +#define DRM_I915_GEM_EXECBUFFER3    0x3f
>>>>>>>>>>>>>>> +#define DRM_I915_GEM_WAIT_USER_FENCE    0x40
>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>> +#define DRM_IOCTL_I915_GEM_VM_BIND
>>>>>>>>>>>>>>> DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>>>>> DRM_I915_GEM_VM_BIND, struct
>>>>>>>>>>>>>>> drm_i915_gem_vm_bind)
>>>>>>>>>>>>>>> +#define DRM_IOCTL_I915_GEM_VM_UNBIND
>>>>>>>>>>>>>>> DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>>>>> DRM_I915_GEM_VM_UNBIND, struct
>>>>>>>>>>>>>>> drm_i915_gem_vm_bind)
>>>>>>>>>>>>>>> +#define DRM_IOCTL_I915_GEM_EXECBUFFER3
>>>>>>>>>>>>>>> DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>>>>> DRM_I915_GEM_EXECBUFFER3, struct
>>>>>>>>>>>>>>> drm_i915_gem_execbuffer3)
>>>>>>>>>>>>>>> +#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE
>>>>>>>>>>>>>>> DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>>>>> DRM_I915_GEM_WAIT_USER_FENCE, struct
>>>>>>>>>>>>>>> drm_i915_gem_wait_user_fence)
>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>> +/**
>>>>>>>>>>>>>>> + * struct drm_i915_gem_vm_bind - VA to object mapping to 
>>>>>>>>>>>>>>> bind.
>>>>>>>>>>>>>>> + *
>>>>>>>>>>>>>>> + * This structure is passed to VM_BIND
>>>>>>>>>>>>>>> ioctl and specifies the mapping of GPU
>>>>>>>>>>>>>>> + * virtual address (VA) range to the
>>>>>>>>>>>>>>> section of an object that should be bound
>>>>>>>>>>>>>>> + * in the device page table of the specified address 
>>>>>>>>>>>>>>> space (VM).
>>>>>>>>>>>>>>> + * The VA range specified must be unique
>>>>>>>>>>>>>>> (ie., not currently bound) and can
>>>>>>>>>>>>>>> + * be mapped to whole object or a section
>>>>>>>>>>>>>>> of the object (partial binding).
>>>>>>>>>>>>>>> + * Multiple VA mappings can be created to
>>>>>>>>>>>>>>> the same section of the object
>>>>>>>>>>>>>>> + * (aliasing).
>>>>>>>>>>>>>>> + *
>>>>>>>>>>>>>>> + * The @queue_idx specifies the queue to
>>>>>>>>>>>>>>> use for binding. Same queue can be
>>>>>>>>>>>>>>> + * used for both VM_BIND and VM_UNBIND
>>>>>>>>>>>>>>> calls. All submitted bind and unbind
>>>>>>>>>>>>>>> + * operations in a queue are performed in the order of 
>>>>>>>>>>>>>>> submission.
>>>>>>>>>>>>>>> + *
>>>>>>>>>>>>>>> + * The @start, @offset and @length should
>>>>>>>>>>>>>>> be 4K page aligned. However the DG2
>>>>>>>>>>>>>>> + * and XEHPSDV has 64K page size for device
>>>>>>>>>>>>>>> local-memory and has compact page
>>>>>>>>>>>>>>> + * table. On those platforms, for binding
>>>>>>>>>>>>>>> device local-memory objects, the
>>>>>>>>>>>>>>> + * @start should be 2M aligned, @offset and
>>>>>>>>>>>>>>> @length should be 64K aligned.
>>>>>>>>>>>>>>> + * Also, on those platforms, it is not
>>>>>>>>>>>>>>> allowed to bind an device local-memory
>>>>>>>>>>>>>>> + * object and a system memory object in a
>>>>>>>>>>>>>>> single 2M section of VA range.
>>>>>>>>>>>>>>> + */
>>>>>>>>>>>>>>> +struct drm_i915_gem_vm_bind {
>>>>>>>>>>>>>>> +    /** @vm_id: VM (address space) id to bind */
>>>>>>>>>>>>>>> +    __u32 vm_id;
>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>> +    /** @queue_idx: Index of queue for binding */
>>>>>>>>>>>>>>> +    __u32 queue_idx;
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have a question here to which I did not find
>>>>>>>>>>>>>> an answer by browsing the old threads.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Queue index appears to be an implicit
>>>>>>>>>>>>>> synchronisation mechanism, right? Operations on
>>>>>>>>>>>>>> the same index are executed/complete in order of
>>>>>>>>>>>>>> ioctl submission?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Do we _have_ to implement this on the kernel
>>>>>>>>>>>>>> side and could just allow in/out fence and let
>>>>>>>>>>>>>> userspace deal with it?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> It orders operations like in a queue. Which is kind
>>>>>>>>>>>>> of what happens with existing queues/engines.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If I understood correctly, it's going to be a
>>>>>>>>>>>>> kthread + a linked list right?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, that is correct.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Lionel
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Arbitrary/on-demand number of queues will add
>>>>>>>>>>>>>> the complexity on the kernel side which should
>>>>>>>>>>>>>> be avoided if possible.
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> It was discussed in the other thread. Jason prefers this 
>>>>>>>>>>>> over putting
>>>>>>>>>>>> an artificial limit on number of queues (as user can
>>>>>>>>>>>> anyway can exhaust
>>>>>>>>>>>> the memory). I think complexity in the driver is manageable.
>>>>>>>>>>>
>>>>>>>>>>> You'll need to create tracking structures on demand, with
>>>>>>>>>>> atomic replace of last fence, ref counting and locking of
>>>>>>>>>>> some sort, more or less?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> We will have a workqueue, an work item and a linked list per 
>>>>>>>>>> queue.
>>>>>>>>>> VM_BIND/UNBIND call will add the mapping request to the
>>>>>>>>>> specified queue's
>>>>>>>>>> linked list and schedule the work item on the workqueue of 
>>>>>>>>>> that queue.
>>>>>>>>>> I am not sure what you mean by last fence and replacing it.
>>>>>>>>>>
>>>>>>>>>>>> The other option being discussed in to have the user create 
>>>>>>>>>>>> those
>>>>>>>>>>>> queues (like creating engine map) before hand and use that 
>>>>>>>>>>>> in vm_bind
>>>>>>>>>>>> and vm_unbind ioctls. This puts a limit on the number of 
>>>>>>>>>>>> queues.
>>>>>>>>>>>> But it is not clean either and not sure it is worth
>>>>>>>>>>>> making the interface
>>>>>>>>>>>> more complex.
>>>>>>>>>>>> https://www.spinics.net/lists/dri-devel/msg350448.html
>>>>>>>>>>>
>>>>>>>>>>> What about the third option of a flag to return a fence (of
>>>>>>>>>>> some sort) and pass in a fence? That way userspace can
>>>>>>>>>>> imagine zero or N queues with very little effort on the
>>>>>>>>>>> kernel side. Was this considered?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I am not clear what fence you are talking about here and how 
>>>>>>>>>> does that
>>>>>>>>>> help with the number of vm_bind queues. Can you eloborate?
>>>>>>>>>
>>>>>>>>> It is actually already documented that bind/unbind will support
>>>>>>>>> input and output fences - so what are these queues on top of what
>>>>>>>>> userspace can already achieve by using them? Purely a 
>>>>>>>>> convenience or
>>>>>>>>> there is more to it?
>>>>>>>>>
>>>>>>>>
>>>>>>>> Oh, the vm_bind queues are discussed in this thread.
>>>>>>>> https://lists.freedesktop.org/archives/intel-gfx/2022-June/299217.html 
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Apparently Vulkan has requirement for multiple queues, each queue
>>>>>>>> processing vm_bind/unbind calls in the order of submission.
>>>>>>>
>>>>>>> I don't see how that answers my question so I will take the 
>>>>>>> freedom to
>>>>>>> repeat it. What are these queues on top of what userspace can 
>>>>>>> already
>>>>>>> achieve by using in-out fences? Purely a convenience or there is 
>>>>>>> more to it?
>>>>>>>
>>>>>>> Queue1:
>>>>>>>
>>>>>>> out_fence_A = vm_bind A
>>>>>>> out_fence_B = vm_bind B, in_fence=out_fence_A
>>>>>>> execbuf(in_fence = out_fence_B)
>>>>>>>
>>>>>>> Queue2:
>>>>>>>
>>>>>>> out_fence_C = vm_bind C
>>>>>>> out_fence_D = vm_bind D, in_fence=out_fence_C
>>>>>>> execbuf(in_fence = out_fence_D)
>>>>>>>
>>>>>>> Parallel bind:
>>>>>>> out_fence_E = vm_bind E
>>>>>>> out_fence_F = vm_bind F
>>>>>>> merged_fence = fence_merge(out_fence_E, out_fence_F)
>>>>>>> execbuf(in_fence = merged_fence)
>>>>>>>
>>>>>>
>>>>>> Let's say you do this and only 1 queue:
>>>>>>
>>>>>> VM_BIND_A (in_fence=fence_A)
>>>>>> VM_BIND_B (in_fence=NULL)
>>>>>>
>>>>>> With 1 queue VM_BIND_B in blocked on fence_A, hence the need for 
>>>>>> than 1
>>>>>> queue.
>>>>>
>>>>> I don't follow - there isn't a concept of a queue exposed in uapi 
>>>>> in what I have described so the above two run in parallel there, if 
>>>>> we ignore fence_A in your example doesn't even exist before you 
>>>>> pass it to bind A so something is not right.
>>>>>
>>>>>> e.g.
>>>>>> VM_BIND_A (queue_id=0, in_fence=fence_A)
>>>>>> VM_BIND_B (queue_id=1, in_fence=NULL)
>>>>>>
>>>>>> Now VM_BIND_B can immediately be executed regardless of fence_A 
>>>>>> status.
>>>>>
>>>>> In my examples userspace can serialise or not as it sees fit using 
>>>>> fences. The "parallel bind" examples two binds run in parallel. 
>>>>> Userspace can create multiple such parallel "queues" if it wanted.
>>>>>
>>>>> Parallel bind 1 and 2 interleaved:
>>>>> out_fence_A = vm_bind A
>>>>> out_fence_B = vm_bind B
>>>>> out_fence_C = vm_bind C
>>>>> out_fence_D = vm_bind D
>>>>> // all binds can run in parallel
>>>>> merged_fence_1 = fence_merge(out_fence_A, out_fence_B)
>>>>> merged_fence_2 = fence_merge(out_fence_C, out_fence_D)
>>>>> execbuf(in_fence = merged_fence_1) // after A&B to finish
>>>>> execbuf(in_fence = merged_fence_2) // after C&D finish
>>>>>
>>>>> There is a huge disconnect somewhere but I don't know where.
>>>>>
>>>>
>>>> Note that Vulkan has requirement that VM_BIND and VM_UNBIND
>>>> operations will also have 'in' fences associated with them
>>>> and not just the 'out' fences (which your example above shows).
>>>
>>> I gave more examples earlier:
>>>
>>> """
>>> Queue1:
>>>
>>> out_fence_A = vm_bind A
>>> out_fence_B = vm_bind B, in_fence=out_fence_A
>>> execbuf(in_fence = out_fence_B)
>>> """
>>>
>>> Clearly I showed both in and out fence.
>>>
> 
> Ok, guess I missed that.
> 
>>>> Yes, one of the solution discussed was not to have any queue_idx
>>>> at all (assume single queue) and let the vm_bind/unbind operations
>>>> submitted run and complete out of submission order. That way
>>>> a vm_bind/unbind sumitted later will not be blocked by a vm_bind/unbind
>>>> submitted earlier.
>>>> But removing the ordering here comes at a cost. Having the operations
>>>> run in submission order has some benefits. These are discussed in the
>>>> other thread.
>>>> https://lists.freedesktop.org/archives/intel-gfx/2022-June/299217.html
>>>
>>> That is some messed up deep quoting in that link. Could you please 
>>> summarize the cost which queues in the uapi intended to avoid?
>>>
>>> In any case it is not just for me. A significant addition is proposed 
>>> for the driver so there should be a clear summary of cost vs benefit 
>>> rather than a messy thread.
>>>
> 
> Say, user has a bunch of mappings to bind or unbind which must be done
> in the submission order. If we have only one queue which runs the
> operations out of submission order, then user has to insert in and out
> fences for each of the operation in the bunch. But by having a in order
> processing queues, user needs to insert 'in' fence only for the first
> submission and 'out' fence only for the last submission in that bunch.
> 
> Also, having in order processing queues allows user to unbind a VA
> mapping and re-use the same VA in a subsequent bind operation without
> having any dependency (dependency is met by the fact that they are
> process in the submission order).

Okay so it is a convenience thing and maybe more performance efficient.

Has a) the performance impact of requiring fences with every bind/unbind 
been looked at, so we know if it is worth adding code to the driver to 
handle queues and b) do you have the queued implementation sketched out 
so amount of kernel code required can be judged?

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Niranjana Vishwanathapura]

On Wed, Jun 15, 2022 at 08:22:23AM +0100, Tvrtko Ursulin wrote:
>
>On 14/06/2022 17:42, Niranjana Vishwanathapura wrote:
>>On Tue, Jun 14, 2022 at 05:07:37PM +0100, Tvrtko Ursulin wrote:
>>>
>>>On 14/06/2022 17:02, Tvrtko Ursulin wrote:
>>>>
>>>>On 14/06/2022 16:43, Niranjana Vishwanathapura wrote:
>>>>>On Tue, Jun 14, 2022 at 08:16:41AM +0100, Tvrtko Ursulin wrote:
>>>>>>
>>>>>>On 14/06/2022 00:39, Matthew Brost wrote:
>>>>>>>On Mon, Jun 13, 2022 at 07:09:06PM +0100, Tvrtko Ursulin wrote:
>>>>>>>>
>>>>>>>>On 13/06/2022 18:49, Niranjana Vishwanathapura wrote:
>>>>>>>>>On Mon, Jun 13, 2022 at 05:22:02PM +0100, Tvrtko Ursulin wrote:
>>>>>>>>>>
>>>>>>>>>>On 13/06/2022 16:05, Niranjana Vishwanathapura wrote:
>>>>>>>>>>>On Mon, Jun 13, 2022 at 09:24:18AM +0100, Tvrtko Ursulin wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>On 10/06/2022 17:14, Niranjana Vishwanathapura wrote:
>>>>>>>>>>>>>On Fri, Jun 10, 2022 at 05:48:39PM +0300, 
>>>>>>>>>>>>>Lionel Landwerlin wrote:
>>>>>>>>>>>>>>On 10/06/2022 13:37, Tvrtko Ursulin wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>On 10/06/2022 08:07, Niranjana Vishwanathapura wrote:
>>>>>>>>>>>>>>>>VM_BIND and related uapi definitions
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>Signed-off-by: Niranjana Vishwanathapura
>>>>>>>>>>>>>>>><niranjana.vishwanathapura@intel.com>
>>>>>>>>>>>>>>>>---
>>>>>>>>>>>>>>>>  Documentation/gpu/rfc/i915_vm_bind.h | 490
>>>>>>>>>>>>>>>>+++++++++++++++++++++++++++
>>>>>>>>>>>>>>>>  1 file changed, 490 insertions(+)
>>>>>>>>>>>>>>>>  create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>diff --git
>>>>>>>>>>>>>>>>a/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>>>>>>b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>>>>>>new file mode 100644
>>>>>>>>>>>>>>>>index 000000000000..9fc854969cfb
>>>>>>>>>>>>>>>>--- /dev/null
>>>>>>>>>>>>>>>>+++ b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>>>>>>@@ -0,0 +1,490 @@
>>>>>>>>>>>>>>>>+/* SPDX-License-Identifier: MIT */
>>>>>>>>>>>>>>>>+/*
>>>>>>>>>>>>>>>>+ * Copyright © 2022 Intel Corporation
>>>>>>>>>>>>>>>>+ */
>>>>>>>>>>>>>>>>+
>>>>>>>>>>>>>>>>+/**
>>>>>>>>>>>>>>>>+ * DOC: I915_PARAM_HAS_VM_BIND
>>>>>>>>>>>>>>>>+ *
>>>>>>>>>>>>>>>>+ * VM_BIND feature availability.
>>>>>>>>>>>>>>>>+ * See typedef drm_i915_getparam_t param.
>>>>>>>>>>>>>>>>+ * bit[0]: If set, VM_BIND is supported, otherwise not.
>>>>>>>>>>>>>>>>+ * bits[8-15]: VM_BIND implementation version.
>>>>>>>>>>>>>>>>+ * version 0 will not have VM_BIND/UNBIND
>>>>>>>>>>>>>>>>timeline fence array support.
>>>>>>>>>>>>>>>>+ */
>>>>>>>>>>>>>>>>+#define I915_PARAM_HAS_VM_BIND        57
>>>>>>>>>>>>>>>>+
>>>>>>>>>>>>>>>>+/**
>>>>>>>>>>>>>>>>+ * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
>>>>>>>>>>>>>>>>+ *
>>>>>>>>>>>>>>>>+ * Flag to opt-in for VM_BIND mode of 
>>>>>>>>>>>>>>>>binding during VM creation.
>>>>>>>>>>>>>>>>+ * See struct drm_i915_gem_vm_control flags.
>>>>>>>>>>>>>>>>+ *
>>>>>>>>>>>>>>>>+ * The older execbuf2 ioctl will not
>>>>>>>>>>>>>>>>support VM_BIND mode of operation.
>>>>>>>>>>>>>>>>+ * For VM_BIND mode, we have new execbuf3
>>>>>>>>>>>>>>>>ioctl which will not accept any
>>>>>>>>>>>>>>>>+ * execlist (See struct
>>>>>>>>>>>>>>>>drm_i915_gem_execbuffer3 for more details).
>>>>>>>>>>>>>>>>+ *
>>>>>>>>>>>>>>>>+ */
>>>>>>>>>>>>>>>>+#define I915_VM_CREATE_FLAGS_USE_VM_BIND    (1 << 0)
>>>>>>>>>>>>>>>>+
>>>>>>>>>>>>>>>>+/**
>>>>>>>>>>>>>>>>+ * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
>>>>>>>>>>>>>>>>+ *
>>>>>>>>>>>>>>>>+ * Flag to declare context as long running.
>>>>>>>>>>>>>>>>+ * See struct drm_i915_gem_context_create_ext flags.
>>>>>>>>>>>>>>>>+ *
>>>>>>>>>>>>>>>>+ * Usage of dma-fence expects that they
>>>>>>>>>>>>>>>>complete in reasonable amount of time.
>>>>>>>>>>>>>>>>+ * Compute on the other hand can be long
>>>>>>>>>>>>>>>>running. Hence it is not appropriate
>>>>>>>>>>>>>>>>+ * for compute contexts to export request
>>>>>>>>>>>>>>>>completion dma-fence to user.
>>>>>>>>>>>>>>>>+ * The dma-fence usage will be limited to
>>>>>>>>>>>>>>>>in-kernel consumption only.
>>>>>>>>>>>>>>>>+ * Compute contexts need to use user/memory fence.
>>>>>>>>>>>>>>>>+ *
>>>>>>>>>>>>>>>>+ * So, long running contexts do not 
>>>>>>>>>>>>>>>>support output fences. Hence,
>>>>>>>>>>>>>>>>+ * I915_EXEC_FENCE_SIGNAL (See
>>>>>>>>>>>>>>>>&drm_i915_gem_exec_fence.flags) is expected
>>>>>>>>>>>>>>>>+ * to be not used. DRM_I915_GEM_WAIT ioctl
>>>>>>>>>>>>>>>>call is also not supported for
>>>>>>>>>>>>>>>>+ * objects mapped to long running contexts.
>>>>>>>>>>>>>>>>+ */
>>>>>>>>>>>>>>>>+#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u << 2)
>>>>>>>>>>>>>>>>+
>>>>>>>>>>>>>>>>+/* VM_BIND related ioctls */
>>>>>>>>>>>>>>>>+#define DRM_I915_GEM_VM_BIND        0x3d
>>>>>>>>>>>>>>>>+#define DRM_I915_GEM_VM_UNBIND        0x3e
>>>>>>>>>>>>>>>>+#define DRM_I915_GEM_EXECBUFFER3    0x3f
>>>>>>>>>>>>>>>>+#define DRM_I915_GEM_WAIT_USER_FENCE    0x40
>>>>>>>>>>>>>>>>+
>>>>>>>>>>>>>>>>+#define DRM_IOCTL_I915_GEM_VM_BIND
>>>>>>>>>>>>>>>>DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>>>>>>DRM_I915_GEM_VM_BIND, struct
>>>>>>>>>>>>>>>>drm_i915_gem_vm_bind)
>>>>>>>>>>>>>>>>+#define DRM_IOCTL_I915_GEM_VM_UNBIND
>>>>>>>>>>>>>>>>DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>>>>>>DRM_I915_GEM_VM_UNBIND, struct
>>>>>>>>>>>>>>>>drm_i915_gem_vm_bind)
>>>>>>>>>>>>>>>>+#define DRM_IOCTL_I915_GEM_EXECBUFFER3
>>>>>>>>>>>>>>>>DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>>>>>>DRM_I915_GEM_EXECBUFFER3, struct
>>>>>>>>>>>>>>>>drm_i915_gem_execbuffer3)
>>>>>>>>>>>>>>>>+#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE
>>>>>>>>>>>>>>>>DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>>>>>>DRM_I915_GEM_WAIT_USER_FENCE, struct
>>>>>>>>>>>>>>>>drm_i915_gem_wait_user_fence)
>>>>>>>>>>>>>>>>+
>>>>>>>>>>>>>>>>+/**
>>>>>>>>>>>>>>>>+ * struct drm_i915_gem_vm_bind - VA to 
>>>>>>>>>>>>>>>>object mapping to bind.
>>>>>>>>>>>>>>>>+ *
>>>>>>>>>>>>>>>>+ * This structure is passed to VM_BIND
>>>>>>>>>>>>>>>>ioctl and specifies the mapping of GPU
>>>>>>>>>>>>>>>>+ * virtual address (VA) range to the
>>>>>>>>>>>>>>>>section of an object that should be bound
>>>>>>>>>>>>>>>>+ * in the device page table of the 
>>>>>>>>>>>>>>>>specified address space (VM).
>>>>>>>>>>>>>>>>+ * The VA range specified must be unique
>>>>>>>>>>>>>>>>(ie., not currently bound) and can
>>>>>>>>>>>>>>>>+ * be mapped to whole object or a section
>>>>>>>>>>>>>>>>of the object (partial binding).
>>>>>>>>>>>>>>>>+ * Multiple VA mappings can be created to
>>>>>>>>>>>>>>>>the same section of the object
>>>>>>>>>>>>>>>>+ * (aliasing).
>>>>>>>>>>>>>>>>+ *
>>>>>>>>>>>>>>>>+ * The @queue_idx specifies the queue to
>>>>>>>>>>>>>>>>use for binding. Same queue can be
>>>>>>>>>>>>>>>>+ * used for both VM_BIND and VM_UNBIND
>>>>>>>>>>>>>>>>calls. All submitted bind and unbind
>>>>>>>>>>>>>>>>+ * operations in a queue are performed 
>>>>>>>>>>>>>>>>in the order of submission.
>>>>>>>>>>>>>>>>+ *
>>>>>>>>>>>>>>>>+ * The @start, @offset and @length should
>>>>>>>>>>>>>>>>be 4K page aligned. However the DG2
>>>>>>>>>>>>>>>>+ * and XEHPSDV has 64K page size for device
>>>>>>>>>>>>>>>>local-memory and has compact page
>>>>>>>>>>>>>>>>+ * table. On those platforms, for binding
>>>>>>>>>>>>>>>>device local-memory objects, the
>>>>>>>>>>>>>>>>+ * @start should be 2M aligned, @offset and
>>>>>>>>>>>>>>>>@length should be 64K aligned.
>>>>>>>>>>>>>>>>+ * Also, on those platforms, it is not
>>>>>>>>>>>>>>>>allowed to bind an device local-memory
>>>>>>>>>>>>>>>>+ * object and a system memory object in a
>>>>>>>>>>>>>>>>single 2M section of VA range.
>>>>>>>>>>>>>>>>+ */
>>>>>>>>>>>>>>>>+struct drm_i915_gem_vm_bind {
>>>>>>>>>>>>>>>>+    /** @vm_id: VM (address space) id to bind */
>>>>>>>>>>>>>>>>+    __u32 vm_id;
>>>>>>>>>>>>>>>>+
>>>>>>>>>>>>>>>>+    /** @queue_idx: Index of queue for binding */
>>>>>>>>>>>>>>>>+    __u32 queue_idx;
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>I have a question here to which I did not find
>>>>>>>>>>>>>>>an answer by browsing the old threads.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Queue index appears to be an implicit
>>>>>>>>>>>>>>>synchronisation mechanism, right? Operations on
>>>>>>>>>>>>>>>the same index are executed/complete in order of
>>>>>>>>>>>>>>>ioctl submission?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Do we _have_ to implement this on the kernel
>>>>>>>>>>>>>>>side and could just allow in/out fence and let
>>>>>>>>>>>>>>>userspace deal with it?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>It orders operations like in a queue. Which is kind
>>>>>>>>>>>>>>of what happens with existing queues/engines.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>If I understood correctly, it's going to be a
>>>>>>>>>>>>>>kthread + a linked list right?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>Yes, that is correct.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>-Lionel
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>Arbitrary/on-demand number of queues will add
>>>>>>>>>>>>>>>the complexity on the kernel side which should
>>>>>>>>>>>>>>>be avoided if possible.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>It was discussed in the other thread. Jason 
>>>>>>>>>>>>>prefers this over putting
>>>>>>>>>>>>>an artificial limit on number of queues (as user can
>>>>>>>>>>>>>anyway can exhaust
>>>>>>>>>>>>>the memory). I think complexity in the driver is manageable.
>>>>>>>>>>>>
>>>>>>>>>>>>You'll need to create tracking structures on demand, with
>>>>>>>>>>>>atomic replace of last fence, ref counting and locking of
>>>>>>>>>>>>some sort, more or less?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>We will have a workqueue, an work item and a 
>>>>>>>>>>>linked list per queue.
>>>>>>>>>>>VM_BIND/UNBIND call will add the mapping request to the
>>>>>>>>>>>specified queue's
>>>>>>>>>>>linked list and schedule the work item on the 
>>>>>>>>>>>workqueue of that queue.
>>>>>>>>>>>I am not sure what you mean by last fence and replacing it.
>>>>>>>>>>>
>>>>>>>>>>>>>The other option being discussed in to have 
>>>>>>>>>>>>>the user create those
>>>>>>>>>>>>>queues (like creating engine map) before hand 
>>>>>>>>>>>>>and use that in vm_bind
>>>>>>>>>>>>>and vm_unbind ioctls. This puts a limit on the 
>>>>>>>>>>>>>number of queues.
>>>>>>>>>>>>>But it is not clean either and not sure it is worth
>>>>>>>>>>>>>making the interface
>>>>>>>>>>>>>more complex.
>>>>>>>>>>>>>https://www.spinics.net/lists/dri-devel/msg350448.html
>>>>>>>>>>>>
>>>>>>>>>>>>What about the third option of a flag to return a fence (of
>>>>>>>>>>>>some sort) and pass in a fence? That way userspace can
>>>>>>>>>>>>imagine zero or N queues with very little effort on the
>>>>>>>>>>>>kernel side. Was this considered?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>I am not clear what fence you are talking about 
>>>>>>>>>>>here and how does that
>>>>>>>>>>>help with the number of vm_bind queues. Can you eloborate?
>>>>>>>>>>
>>>>>>>>>>It is actually already documented that bind/unbind will support
>>>>>>>>>>input and output fences - so what are these queues on top of what
>>>>>>>>>>userspace can already achieve by using them? Purely 
>>>>>>>>>>a convenience or
>>>>>>>>>>there is more to it?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>Oh, the vm_bind queues are discussed in this thread.
>>>>>>>>>https://lists.freedesktop.org/archives/intel-gfx/2022-June/299217.html
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>Apparently Vulkan has requirement for multiple queues, each queue
>>>>>>>>>processing vm_bind/unbind calls in the order of submission.
>>>>>>>>
>>>>>>>>I don't see how that answers my question so I will take 
>>>>>>>>the freedom to
>>>>>>>>repeat it. What are these queues on top of what 
>>>>>>>>userspace can already
>>>>>>>>achieve by using in-out fences? Purely a convenience or 
>>>>>>>>there is more to it?
>>>>>>>>
>>>>>>>>Queue1:
>>>>>>>>
>>>>>>>>out_fence_A = vm_bind A
>>>>>>>>out_fence_B = vm_bind B, in_fence=out_fence_A
>>>>>>>>execbuf(in_fence = out_fence_B)
>>>>>>>>
>>>>>>>>Queue2:
>>>>>>>>
>>>>>>>>out_fence_C = vm_bind C
>>>>>>>>out_fence_D = vm_bind D, in_fence=out_fence_C
>>>>>>>>execbuf(in_fence = out_fence_D)
>>>>>>>>
>>>>>>>>Parallel bind:
>>>>>>>>out_fence_E = vm_bind E
>>>>>>>>out_fence_F = vm_bind F
>>>>>>>>merged_fence = fence_merge(out_fence_E, out_fence_F)
>>>>>>>>execbuf(in_fence = merged_fence)
>>>>>>>>
>>>>>>>
>>>>>>>Let's say you do this and only 1 queue:
>>>>>>>
>>>>>>>VM_BIND_A (in_fence=fence_A)
>>>>>>>VM_BIND_B (in_fence=NULL)
>>>>>>>
>>>>>>>With 1 queue VM_BIND_B in blocked on fence_A, hence the 
>>>>>>>need for than 1
>>>>>>>queue.
>>>>>>
>>>>>>I don't follow - there isn't a concept of a queue exposed in 
>>>>>>uapi in what I have described so the above two run in 
>>>>>>parallel there, if we ignore fence_A in your example doesn't 
>>>>>>even exist before you pass it to bind A so something is not 
>>>>>>right.
>>>>>>
>>>>>>>e.g.
>>>>>>>VM_BIND_A (queue_id=0, in_fence=fence_A)
>>>>>>>VM_BIND_B (queue_id=1, in_fence=NULL)
>>>>>>>
>>>>>>>Now VM_BIND_B can immediately be executed regardless of 
>>>>>>>fence_A status.
>>>>>>
>>>>>>In my examples userspace can serialise or not as it sees fit 
>>>>>>using fences. The "parallel bind" examples two binds run in 
>>>>>>parallel. Userspace can create multiple such parallel 
>>>>>>"queues" if it wanted.
>>>>>>
>>>>>>Parallel bind 1 and 2 interleaved:
>>>>>>out_fence_A = vm_bind A
>>>>>>out_fence_B = vm_bind B
>>>>>>out_fence_C = vm_bind C
>>>>>>out_fence_D = vm_bind D
>>>>>>// all binds can run in parallel
>>>>>>merged_fence_1 = fence_merge(out_fence_A, out_fence_B)
>>>>>>merged_fence_2 = fence_merge(out_fence_C, out_fence_D)
>>>>>>execbuf(in_fence = merged_fence_1) // after A&B to finish
>>>>>>execbuf(in_fence = merged_fence_2) // after C&D finish
>>>>>>
>>>>>>There is a huge disconnect somewhere but I don't know where.
>>>>>>
>>>>>
>>>>>Note that Vulkan has requirement that VM_BIND and VM_UNBIND
>>>>>operations will also have 'in' fences associated with them
>>>>>and not just the 'out' fences (which your example above shows).
>>>>
>>>>I gave more examples earlier:
>>>>
>>>>"""
>>>>Queue1:
>>>>
>>>>out_fence_A = vm_bind A
>>>>out_fence_B = vm_bind B, in_fence=out_fence_A
>>>>execbuf(in_fence = out_fence_B)
>>>>"""
>>>>
>>>>Clearly I showed both in and out fence.
>>>>
>>
>>Ok, guess I missed that.
>>
>>>>>Yes, one of the solution discussed was not to have any queue_idx
>>>>>at all (assume single queue) and let the vm_bind/unbind operations
>>>>>submitted run and complete out of submission order. That way
>>>>>a vm_bind/unbind sumitted later will not be blocked by a vm_bind/unbind
>>>>>submitted earlier.
>>>>>But removing the ordering here comes at a cost. Having the operations
>>>>>run in submission order has some benefits. These are discussed in the
>>>>>other thread.
>>>>>https://lists.freedesktop.org/archives/intel-gfx/2022-June/299217.html
>>>>
>>>>That is some messed up deep quoting in that link. Could you 
>>>>please summarize the cost which queues in the uapi intended to 
>>>>avoid?
>>>>
>>>>In any case it is not just for me. A significant addition is 
>>>>proposed for the driver so there should be a clear summary of 
>>>>cost vs benefit rather than a messy thread.
>>>>
>>
>>Say, user has a bunch of mappings to bind or unbind which must be done
>>in the submission order. If we have only one queue which runs the
>>operations out of submission order, then user has to insert in and out
>>fences for each of the operation in the bunch. But by having a in order
>>processing queues, user needs to insert 'in' fence only for the first
>>submission and 'out' fence only for the last submission in that bunch.
>>
>>Also, having in order processing queues allows user to unbind a VA
>>mapping and re-use the same VA in a subsequent bind operation without
>>having any dependency (dependency is met by the fact that they are
>>process in the submission order).
>
>Okay so it is a convenience thing and maybe more performance efficient.
>
>Has a) the performance impact of requiring fences with every 
>bind/unbind been looked at, so we know if it is worth adding code to 
>the driver to handle queues and b) do you have the queued 
>implementation sketched out so amount of kernel code required can be 
>judged?

No, this sparse requirement (in/out fences) is fairely new and the
queue request came up during this review. I will prototype this
once I post the existing set of vm_bind features in i915 for review.

Niranjana

>
>Regards,
>
>Tvrtko

Re: [Intel-gfx] [PATCH 3/3] drm/doc/rfc: VM_BIND uapi definition

[Tvrtko Ursulin]

On 15/06/2022 16:20, Niranjana Vishwanathapura wrote:
> On Wed, Jun 15, 2022 at 08:22:23AM +0100, Tvrtko Ursulin wrote:
>>
>> On 14/06/2022 17:42, Niranjana Vishwanathapura wrote:
>>> On Tue, Jun 14, 2022 at 05:07:37PM +0100, Tvrtko Ursulin wrote:
>>>>
>>>> On 14/06/2022 17:02, Tvrtko Ursulin wrote:
>>>>>
>>>>> On 14/06/2022 16:43, Niranjana Vishwanathapura wrote:
>>>>>> On Tue, Jun 14, 2022 at 08:16:41AM +0100, Tvrtko Ursulin wrote:
>>>>>>>
>>>>>>> On 14/06/2022 00:39, Matthew Brost wrote:
>>>>>>>> On Mon, Jun 13, 2022 at 07:09:06PM +0100, Tvrtko Ursulin wrote:
>>>>>>>>>
>>>>>>>>> On 13/06/2022 18:49, Niranjana Vishwanathapura wrote:
>>>>>>>>>> On Mon, Jun 13, 2022 at 05:22:02PM +0100, Tvrtko Ursulin wrote:
>>>>>>>>>>>
>>>>>>>>>>> On 13/06/2022 16:05, Niranjana Vishwanathapura wrote:
>>>>>>>>>>>> On Mon, Jun 13, 2022 at 09:24:18AM +0100, Tvrtko Ursulin wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 10/06/2022 17:14, Niranjana Vishwanathapura wrote:
>>>>>>>>>>>>>> On Fri, Jun 10, 2022 at 05:48:39PM +0300, Lionel 
>>>>>>>>>>>>>> Landwerlin wrote:
>>>>>>>>>>>>>>> On 10/06/2022 13:37, Tvrtko Ursulin wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 10/06/2022 08:07, Niranjana Vishwanathapura wrote:
>>>>>>>>>>>>>>>>> VM_BIND and related uapi definitions
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Signed-off-by: Niranjana Vishwanathapura
>>>>>>>>>>>>>>>>> <niranjana.vishwanathapura@intel.com>
>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>   Documentation/gpu/rfc/i915_vm_bind.h | 490
>>>>>>>>>>>>>>>>> +++++++++++++++++++++++++++
>>>>>>>>>>>>>>>>>   1 file changed, 490 insertions(+)
>>>>>>>>>>>>>>>>>   create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> diff --git
>>>>>>>>>>>>>>>>> a/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>>>>>>> b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>>>>>>> new file mode 100644
>>>>>>>>>>>>>>>>> index 000000000000..9fc854969cfb
>>>>>>>>>>>>>>>>> --- /dev/null
>>>>>>>>>>>>>>>>> +++ b/Documentation/gpu/rfc/i915_vm_bind.h
>>>>>>>>>>>>>>>>> @@ -0,0 +1,490 @@
>>>>>>>>>>>>>>>>> +/* SPDX-License-Identifier: MIT */
>>>>>>>>>>>>>>>>> +/*
>>>>>>>>>>>>>>>>> + * Copyright © 2022 Intel Corporation
>>>>>>>>>>>>>>>>> + */
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> +/**
>>>>>>>>>>>>>>>>> + * DOC: I915_PARAM_HAS_VM_BIND
>>>>>>>>>>>>>>>>> + *
>>>>>>>>>>>>>>>>> + * VM_BIND feature availability.
>>>>>>>>>>>>>>>>> + * See typedef drm_i915_getparam_t param.
>>>>>>>>>>>>>>>>> + * bit[0]: If set, VM_BIND is supported, otherwise not.
>>>>>>>>>>>>>>>>> + * bits[8-15]: VM_BIND implementation version.
>>>>>>>>>>>>>>>>> + * version 0 will not have VM_BIND/UNBIND
>>>>>>>>>>>>>>>>> timeline fence array support.
>>>>>>>>>>>>>>>>> + */
>>>>>>>>>>>>>>>>> +#define I915_PARAM_HAS_VM_BIND        57
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> +/**
>>>>>>>>>>>>>>>>> + * DOC: I915_VM_CREATE_FLAGS_USE_VM_BIND
>>>>>>>>>>>>>>>>> + *
>>>>>>>>>>>>>>>>> + * Flag to opt-in for VM_BIND mode of binding during 
>>>>>>>>>>>>>>>>> VM creation.
>>>>>>>>>>>>>>>>> + * See struct drm_i915_gem_vm_control flags.
>>>>>>>>>>>>>>>>> + *
>>>>>>>>>>>>>>>>> + * The older execbuf2 ioctl will not
>>>>>>>>>>>>>>>>> support VM_BIND mode of operation.
>>>>>>>>>>>>>>>>> + * For VM_BIND mode, we have new execbuf3
>>>>>>>>>>>>>>>>> ioctl which will not accept any
>>>>>>>>>>>>>>>>> + * execlist (See struct
>>>>>>>>>>>>>>>>> drm_i915_gem_execbuffer3 for more details).
>>>>>>>>>>>>>>>>> + *
>>>>>>>>>>>>>>>>> + */
>>>>>>>>>>>>>>>>> +#define I915_VM_CREATE_FLAGS_USE_VM_BIND    (1 << 0)
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> +/**
>>>>>>>>>>>>>>>>> + * DOC: I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING
>>>>>>>>>>>>>>>>> + *
>>>>>>>>>>>>>>>>> + * Flag to declare context as long running.
>>>>>>>>>>>>>>>>> + * See struct drm_i915_gem_context_create_ext flags.
>>>>>>>>>>>>>>>>> + *
>>>>>>>>>>>>>>>>> + * Usage of dma-fence expects that they
>>>>>>>>>>>>>>>>> complete in reasonable amount of time.
>>>>>>>>>>>>>>>>> + * Compute on the other hand can be long
>>>>>>>>>>>>>>>>> running. Hence it is not appropriate
>>>>>>>>>>>>>>>>> + * for compute contexts to export request
>>>>>>>>>>>>>>>>> completion dma-fence to user.
>>>>>>>>>>>>>>>>> + * The dma-fence usage will be limited to
>>>>>>>>>>>>>>>>> in-kernel consumption only.
>>>>>>>>>>>>>>>>> + * Compute contexts need to use user/memory fence.
>>>>>>>>>>>>>>>>> + *
>>>>>>>>>>>>>>>>> + * So, long running contexts do not support output 
>>>>>>>>>>>>>>>>> fences. Hence,
>>>>>>>>>>>>>>>>> + * I915_EXEC_FENCE_SIGNAL (See
>>>>>>>>>>>>>>>>> &drm_i915_gem_exec_fence.flags) is expected
>>>>>>>>>>>>>>>>> + * to be not used. DRM_I915_GEM_WAIT ioctl
>>>>>>>>>>>>>>>>> call is also not supported for
>>>>>>>>>>>>>>>>> + * objects mapped to long running contexts.
>>>>>>>>>>>>>>>>> + */
>>>>>>>>>>>>>>>>> +#define I915_CONTEXT_CREATE_FLAGS_LONG_RUNNING   (1u 
>>>>>>>>>>>>>>>>> << 2)
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> +/* VM_BIND related ioctls */
>>>>>>>>>>>>>>>>> +#define DRM_I915_GEM_VM_BIND        0x3d
>>>>>>>>>>>>>>>>> +#define DRM_I915_GEM_VM_UNBIND        0x3e
>>>>>>>>>>>>>>>>> +#define DRM_I915_GEM_EXECBUFFER3    0x3f
>>>>>>>>>>>>>>>>> +#define DRM_I915_GEM_WAIT_USER_FENCE    0x40
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> +#define DRM_IOCTL_I915_GEM_VM_BIND
>>>>>>>>>>>>>>>>> DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>>>>>>> DRM_I915_GEM_VM_BIND, struct
>>>>>>>>>>>>>>>>> drm_i915_gem_vm_bind)
>>>>>>>>>>>>>>>>> +#define DRM_IOCTL_I915_GEM_VM_UNBIND
>>>>>>>>>>>>>>>>> DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>>>>>>> DRM_I915_GEM_VM_UNBIND, struct
>>>>>>>>>>>>>>>>> drm_i915_gem_vm_bind)
>>>>>>>>>>>>>>>>> +#define DRM_IOCTL_I915_GEM_EXECBUFFER3
>>>>>>>>>>>>>>>>> DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>>>>>>> DRM_I915_GEM_EXECBUFFER3, struct
>>>>>>>>>>>>>>>>> drm_i915_gem_execbuffer3)
>>>>>>>>>>>>>>>>> +#define DRM_IOCTL_I915_GEM_WAIT_USER_FENCE
>>>>>>>>>>>>>>>>> DRM_IOWR(DRM_COMMAND_BASE +
>>>>>>>>>>>>>>>>> DRM_I915_GEM_WAIT_USER_FENCE, struct
>>>>>>>>>>>>>>>>> drm_i915_gem_wait_user_fence)
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> +/**
>>>>>>>>>>>>>>>>> + * struct drm_i915_gem_vm_bind - VA to object mapping 
>>>>>>>>>>>>>>>>> to bind.
>>>>>>>>>>>>>>>>> + *
>>>>>>>>>>>>>>>>> + * This structure is passed to VM_BIND
>>>>>>>>>>>>>>>>> ioctl and specifies the mapping of GPU
>>>>>>>>>>>>>>>>> + * virtual address (VA) range to the
>>>>>>>>>>>>>>>>> section of an object that should be bound
>>>>>>>>>>>>>>>>> + * in the device page table of the specified address 
>>>>>>>>>>>>>>>>> space (VM).
>>>>>>>>>>>>>>>>> + * The VA range specified must be unique
>>>>>>>>>>>>>>>>> (ie., not currently bound) and can
>>>>>>>>>>>>>>>>> + * be mapped to whole object or a section
>>>>>>>>>>>>>>>>> of the object (partial binding).
>>>>>>>>>>>>>>>>> + * Multiple VA mappings can be created to
>>>>>>>>>>>>>>>>> the same section of the object
>>>>>>>>>>>>>>>>> + * (aliasing).
>>>>>>>>>>>>>>>>> + *
>>>>>>>>>>>>>>>>> + * The @queue_idx specifies the queue to
>>>>>>>>>>>>>>>>> use for binding. Same queue can be
>>>>>>>>>>>>>>>>> + * used for both VM_BIND and VM_UNBIND
>>>>>>>>>>>>>>>>> calls. All submitted bind and unbind
>>>>>>>>>>>>>>>>> + * operations in a queue are performed in the order of 
>>>>>>>>>>>>>>>>> submission.
>>>>>>>>>>>>>>>>> + *
>>>>>>>>>>>>>>>>> + * The @start, @offset and @length should
>>>>>>>>>>>>>>>>> be 4K page aligned. However the DG2
>>>>>>>>>>>>>>>>> + * and XEHPSDV has 64K page size for device
>>>>>>>>>>>>>>>>> local-memory and has compact page
>>>>>>>>>>>>>>>>> + * table. On those platforms, for binding
>>>>>>>>>>>>>>>>> device local-memory objects, the
>>>>>>>>>>>>>>>>> + * @start should be 2M aligned, @offset and
>>>>>>>>>>>>>>>>> @length should be 64K aligned.
>>>>>>>>>>>>>>>>> + * Also, on those platforms, it is not
>>>>>>>>>>>>>>>>> allowed to bind an device local-memory
>>>>>>>>>>>>>>>>> + * object and a system memory object in a
>>>>>>>>>>>>>>>>> single 2M section of VA range.
>>>>>>>>>>>>>>>>> + */
>>>>>>>>>>>>>>>>> +struct drm_i915_gem_vm_bind {
>>>>>>>>>>>>>>>>> +    /** @vm_id: VM (address space) id to bind */
>>>>>>>>>>>>>>>>> +    __u32 vm_id;
>>>>>>>>>>>>>>>>> +
>>>>>>>>>>>>>>>>> +    /** @queue_idx: Index of queue for binding */
>>>>>>>>>>>>>>>>> +    __u32 queue_idx;
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have a question here to which I did not find
>>>>>>>>>>>>>>>> an answer by browsing the old threads.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Queue index appears to be an implicit
>>>>>>>>>>>>>>>> synchronisation mechanism, right? Operations on
>>>>>>>>>>>>>>>> the same index are executed/complete in order of
>>>>>>>>>>>>>>>> ioctl submission?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Do we _have_ to implement this on the kernel
>>>>>>>>>>>>>>>> side and could just allow in/out fence and let
>>>>>>>>>>>>>>>> userspace deal with it?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It orders operations like in a queue. Which is kind
>>>>>>>>>>>>>>> of what happens with existing queues/engines.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If I understood correctly, it's going to be a
>>>>>>>>>>>>>>> kthread + a linked list right?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, that is correct.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -Lionel
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Arbitrary/on-demand number of queues will add
>>>>>>>>>>>>>>>> the complexity on the kernel side which should
>>>>>>>>>>>>>>>> be avoided if possible.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It was discussed in the other thread. Jason prefers this 
>>>>>>>>>>>>>> over putting
>>>>>>>>>>>>>> an artificial limit on number of queues (as user can
>>>>>>>>>>>>>> anyway can exhaust
>>>>>>>>>>>>>> the memory). I think complexity in the driver is manageable.
>>>>>>>>>>>>>
>>>>>>>>>>>>> You'll need to create tracking structures on demand, with
>>>>>>>>>>>>> atomic replace of last fence, ref counting and locking of
>>>>>>>>>>>>> some sort, more or less?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> We will have a workqueue, an work item and a linked list per 
>>>>>>>>>>>> queue.
>>>>>>>>>>>> VM_BIND/UNBIND call will add the mapping request to the
>>>>>>>>>>>> specified queue's
>>>>>>>>>>>> linked list and schedule the work item on the workqueue of 
>>>>>>>>>>>> that queue.
>>>>>>>>>>>> I am not sure what you mean by last fence and replacing it.
>>>>>>>>>>>>
>>>>>>>>>>>>>> The other option being discussed in to have the user 
>>>>>>>>>>>>>> create those
>>>>>>>>>>>>>> queues (like creating engine map) before hand and use that 
>>>>>>>>>>>>>> in vm_bind
>>>>>>>>>>>>>> and vm_unbind ioctls. This puts a limit on the number of 
>>>>>>>>>>>>>> queues.
>>>>>>>>>>>>>> But it is not clean either and not sure it is worth
>>>>>>>>>>>>>> making the interface
>>>>>>>>>>>>>> more complex.
>>>>>>>>>>>>>> https://www.spinics.net/lists/dri-devel/msg350448.html
>>>>>>>>>>>>>
>>>>>>>>>>>>> What about the third option of a flag to return a fence (of
>>>>>>>>>>>>> some sort) and pass in a fence? That way userspace can
>>>>>>>>>>>>> imagine zero or N queues with very little effort on the
>>>>>>>>>>>>> kernel side. Was this considered?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I am not clear what fence you are talking about here and how 
>>>>>>>>>>>> does that
>>>>>>>>>>>> help with the number of vm_bind queues. Can you eloborate?
>>>>>>>>>>>
>>>>>>>>>>> It is actually already documented that bind/unbind will support
>>>>>>>>>>> input and output fences - so what are these queues on top of 
>>>>>>>>>>> what
>>>>>>>>>>> userspace can already achieve by using them? Purely a 
>>>>>>>>>>> convenience or
>>>>>>>>>>> there is more to it?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Oh, the vm_bind queues are discussed in this thread.
>>>>>>>>>> https://lists.freedesktop.org/archives/intel-gfx/2022-June/299217.html 
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Apparently Vulkan has requirement for multiple queues, each queue
>>>>>>>>>> processing vm_bind/unbind calls in the order of submission.
>>>>>>>>>
>>>>>>>>> I don't see how that answers my question so I will take the 
>>>>>>>>> freedom to
>>>>>>>>> repeat it. What are these queues on top of what userspace can 
>>>>>>>>> already
>>>>>>>>> achieve by using in-out fences? Purely a convenience or there 
>>>>>>>>> is more to it?
>>>>>>>>>
>>>>>>>>> Queue1:
>>>>>>>>>
>>>>>>>>> out_fence_A = vm_bind A
>>>>>>>>> out_fence_B = vm_bind B, in_fence=out_fence_A
>>>>>>>>> execbuf(in_fence = out_fence_B)
>>>>>>>>>
>>>>>>>>> Queue2:
>>>>>>>>>
>>>>>>>>> out_fence_C = vm_bind C
>>>>>>>>> out_fence_D = vm_bind D, in_fence=out_fence_C
>>>>>>>>> execbuf(in_fence = out_fence_D)
>>>>>>>>>
>>>>>>>>> Parallel bind:
>>>>>>>>> out_fence_E = vm_bind E
>>>>>>>>> out_fence_F = vm_bind F
>>>>>>>>> merged_fence = fence_merge(out_fence_E, out_fence_F)
>>>>>>>>> execbuf(in_fence = merged_fence)
>>>>>>>>>
>>>>>>>>
>>>>>>>> Let's say you do this and only 1 queue:
>>>>>>>>
>>>>>>>> VM_BIND_A (in_fence=fence_A)
>>>>>>>> VM_BIND_B (in_fence=NULL)
>>>>>>>>
>>>>>>>> With 1 queue VM_BIND_B in blocked on fence_A, hence the need for 
>>>>>>>> than 1
>>>>>>>> queue.
>>>>>>>
>>>>>>> I don't follow - there isn't a concept of a queue exposed in uapi 
>>>>>>> in what I have described so the above two run in parallel there, 
>>>>>>> if we ignore fence_A in your example doesn't even exist before 
>>>>>>> you pass it to bind A so something is not right.
>>>>>>>
>>>>>>>> e.g.
>>>>>>>> VM_BIND_A (queue_id=0, in_fence=fence_A)
>>>>>>>> VM_BIND_B (queue_id=1, in_fence=NULL)
>>>>>>>>
>>>>>>>> Now VM_BIND_B can immediately be executed regardless of fence_A 
>>>>>>>> status.
>>>>>>>
>>>>>>> In my examples userspace can serialise or not as it sees fit 
>>>>>>> using fences. The "parallel bind" examples two binds run in 
>>>>>>> parallel. Userspace can create multiple such parallel "queues" if 
>>>>>>> it wanted.
>>>>>>>
>>>>>>> Parallel bind 1 and 2 interleaved:
>>>>>>> out_fence_A = vm_bind A
>>>>>>> out_fence_B = vm_bind B
>>>>>>> out_fence_C = vm_bind C
>>>>>>> out_fence_D = vm_bind D
>>>>>>> // all binds can run in parallel
>>>>>>> merged_fence_1 = fence_merge(out_fence_A, out_fence_B)
>>>>>>> merged_fence_2 = fence_merge(out_fence_C, out_fence_D)
>>>>>>> execbuf(in_fence = merged_fence_1) // after A&B to finish
>>>>>>> execbuf(in_fence = merged_fence_2) // after C&D finish
>>>>>>>
>>>>>>> There is a huge disconnect somewhere but I don't know where.
>>>>>>>
>>>>>>
>>>>>> Note that Vulkan has requirement that VM_BIND and VM_UNBIND
>>>>>> operations will also have 'in' fences associated with them
>>>>>> and not just the 'out' fences (which your example above shows).
>>>>>
>>>>> I gave more examples earlier:
>>>>>
>>>>> """
>>>>> Queue1:
>>>>>
>>>>> out_fence_A = vm_bind A
>>>>> out_fence_B = vm_bind B, in_fence=out_fence_A
>>>>> execbuf(in_fence = out_fence_B)
>>>>> """
>>>>>
>>>>> Clearly I showed both in and out fence.
>>>>>
>>>
>>> Ok, guess I missed that.
>>>
>>>>>> Yes, one of the solution discussed was not to have any queue_idx
>>>>>> at all (assume single queue) and let the vm_bind/unbind operations
>>>>>> submitted run and complete out of submission order. That way
>>>>>> a vm_bind/unbind sumitted later will not be blocked by a 
>>>>>> vm_bind/unbind
>>>>>> submitted earlier.
>>>>>> But removing the ordering here comes at a cost. Having the operations
>>>>>> run in submission order has some benefits. These are discussed in the
>>>>>> other thread.
>>>>>> https://lists.freedesktop.org/archives/intel-gfx/2022-June/299217.html 
>>>>>>
>>>>>
>>>>> That is some messed up deep quoting in that link. Could you please 
>>>>> summarize the cost which queues in the uapi intended to avoid?
>>>>>
>>>>> In any case it is not just for me. A significant addition is 
>>>>> proposed for the driver so there should be a clear summary of cost 
>>>>> vs benefit rather than a messy thread.
>>>>>
>>>
>>> Say, user has a bunch of mappings to bind or unbind which must be done
>>> in the submission order. If we have only one queue which runs the
>>> operations out of submission order, then user has to insert in and out
>>> fences for each of the operation in the bunch. But by having a in order
>>> processing queues, user needs to insert 'in' fence only for the first
>>> submission and 'out' fence only for the last submission in that bunch.
>>>
>>> Also, having in order processing queues allows user to unbind a VA
>>> mapping and re-use the same VA in a subsequent bind operation without
>>> having any dependency (dependency is met by the fact that they are
>>> process in the submission order).
>>
>> Okay so it is a convenience thing and maybe more performance efficient.
>>
>> Has a) the performance impact of requiring fences with every 
>> bind/unbind been looked at, so we know if it is worth adding code to 
>> the driver to handle queues and b) do you have the queued 
>> implementation sketched out so amount of kernel code required can be 
>> judged?
> 
> No, this sparse requirement (in/out fences) is fairely new and the
> queue request came up during this review. I will prototype this
> once I post the existing set of vm_bind features in i915 for review.

If I understand right there isn't an implementation for neither in/out 
fences nor queues? I am thinking, if at least you had in/out fences you 
could easily measure the overhead with tight (un)bind in a loop with and 
without in/out fences.

Or maybe even execbuf2 path could be used. It's a lot heavier path in 
general, but w/ and wo/ fences might be representative of an absolute 
cost of dealing with them (fences). And evaluate that number together 
with plain (un)bind ioctl cost from the implementation you have.

If it shows that chaining in-out fences adds a significant overhead then 
queue in the uapi would be acceptable.

Regards,

Tvrtko

[Intel-gfx] ✗ Fi.CI.BUILD: failure for drm/doc/rfc: i915 VM_BIND feature design + uapi

[Patchwork]

== Series Details ==

Series: drm/doc/rfc: i915 VM_BIND feature design + uapi
URL   : https://patchwork.freedesktop.org/series/104962/
State : failure

== Summary ==

Error: make failed
  CALL    scripts/checksyscalls.sh
  CALL    scripts/atomic/check-atomics.sh
  DESCEND objtool
  CHK     include/generated/compile.h
  CC [M]  drivers/gpu/drm/i915/i915_driver.o
In file included from ./drivers/gpu/drm/i915/i915_pmu.h:13,
                 from ./drivers/gpu/drm/i915/gt/intel_engine_types.h:21,
                 from ./drivers/gpu/drm/i915/gt/intel_context_types.h:18,
                 from ./drivers/gpu/drm/i915/gem/i915_gem_context_types.h:20,
                 from ./drivers/gpu/drm/i915/i915_request.h:34,
                 from ./drivers/gpu/drm/i915/i915_active.h:13,
                 from ./drivers/gpu/drm/i915/gt/intel_ggtt_fencing.h:12,
                 from ./drivers/gpu/drm/i915/i915_vma.h:33,
                 from drivers/gpu/drm/i915/display/intel_display_types.h:48,
                 from drivers/gpu/drm/i915/i915_driver.c:52:
./include/uapi/drm/i915_drm.h:1932:2: error: "/*" within comment [-Werror=comment]
  /** @param: Parameter to set or query */
   
cc1: all warnings being treated as errors
scripts/Makefile.build:249: recipe for target 'drivers/gpu/drm/i915/i915_driver.o' failed
make[4]: *** [drivers/gpu/drm/i915/i915_driver.o] Error 1
scripts/Makefile.build:466: recipe for target 'drivers/gpu/drm/i915' failed
make[3]: *** [drivers/gpu/drm/i915] Error 2
scripts/Makefile.build:466: recipe for target 'drivers/gpu/drm' failed
make[2]: *** [drivers/gpu/drm] Error 2
scripts/Makefile.build:466: recipe for target 'drivers/gpu' failed
make[1]: *** [drivers/gpu] Error 2
Makefile:1839: recipe for target 'drivers' failed
make: *** [drivers] Error 2

Re: [Intel-gfx] [PATCH 0/3] drm/doc/rfc: i915 VM_BIND feature design + uapi

[Niranjana Vishwanathapura]

On Fri, Jun 10, 2022 at 12:07:08AM -0700, Niranjana Vishwanathapura wrote:
>This is the i915 driver VM_BIND feature design RFC patch series along
>with the required uapi definition and description of intended use cases.
>

Some of us had an offline dicussion on this.
Based on that,

1) The scope of this work (VM_BIND support in i915) will reduced to
   support simple Mesa use case. So, I will remove all compute related
   uapi for now.
2) VM_BIND/UNBIND will only support an 'out' fence. ie., it won't
   support 'in' fences and hence no timeline fence array as well.
   UMDs are expected to handle any 'in' fence requirement.
3) We will not support any VM_BIND/UNBIND queues. The binding and
   unbinding operations can get completed out of submission order.
   Normally, they will get completed synchronously, but if the object
   is being moved, the binding will happen once that is complete
   and out fence will be signaled after binding is complete.
4) We will still have execbuf3 for VM_BIND mode.

I will update the patch series and send out.

Thanks,
Niranjana

>This series is an updated version of the below RFC series. It address
>the review feedback by adding execbuf3 ioctl for vm_bind, adding
>multiple queues support for vm_bind/unbind ioctls and some formatting
>and documentation updates.
>https://www.spinics.net/lists/dri-devel/msg347731.html
>
>Signed-off-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>
>
>Niranjana Vishwanathapura (3):
>  drm/doc/rfc: VM_BIND feature design document
>  drm/i915: Update i915 uapi documentation
>  drm/doc/rfc: VM_BIND uapi definition
>
> Documentation/driver-api/dma-buf.rst   |   2 +
> Documentation/gpu/rfc/i915_vm_bind.h   | 490 +++++++++++++++++++++++++
> Documentation/gpu/rfc/i915_vm_bind.rst | 309 ++++++++++++++++
> Documentation/gpu/rfc/index.rst        |   4 +
> include/uapi/drm/i915_drm.h            | 203 +++++++---
> 5 files changed, 963 insertions(+), 45 deletions(-)
> create mode 100644 Documentation/gpu/rfc/i915_vm_bind.h
> create mode 100644 Documentation/gpu/rfc/i915_vm_bind.rst
>
>-- 
>2.21.0.rc0.32.g243a4c7e27
>