[PATCH 0/3] [PATCH 0/3] dmaengine: polled completion status for out-of-order providers

[PATCH 0/3] [PATCH 0/3] dmaengine: polled completion status for out-of-order providers

[Daniel Byrne]

A client that polls transaction status on a device whose work queues
complete out of order, as DSA's do when several engines serve one
queue, cannot use the cookie-ordered dma_async_is_tx_complete(): the
completed-cookie watermark moves past descriptors still in flight.
These three patches add dmaengine_async_is_tx_complete() as the polling
API, routing the query to device_tx_status() without the cookie-window
outputs, implement it in idxd from a generation-numbered cookie that
reads the descriptor's own completion record, and drop
DMA_COMPLETION_NO_ORDER and DMA_OUT_OF_ORDER, whose only setter was
idxd. dmatest's polled mode is the in-tree user; the io_uring DMA copy
offload, to be posted separately, is the one that needs the exact answers.

This applies on top of "dmaengine: idxd: fixes found by an in-kernel
polling client", posted alongside, which the status implementation's
reset-epoch handling depends on.

Daniel Byrne (3):
  dmaengine: add dmaengine_async_is_tx_complete() for channel status
    queries
  dmaengine: idxd: generation-number transaction status
  dmaengine: drop DMA_COMPLETION_NO_ORDER and DMA_OUT_OF_ORDER

 Documentation/driver-api/dmaengine/client.rst |  24 +--
 .../driver-api/dmaengine/provider.rst         |  64 ++++---
 drivers/dma/amba-pl08x.c                      |   2 +-
 drivers/dma/at_hdmac.c                        |   5 +-
 drivers/dma/dmaengine.c                       |   2 +-
 drivers/dma/dmaengine.h                       |  21 ++-
 drivers/dma/dmatest.c                         |  14 +-
 drivers/dma/idxd/device.c                     |   1 +
 drivers/dma/idxd/dma.c                        | 159 +++++++++++++++++-
 drivers/dma/idxd/idxd.h                       |   3 +
 drivers/dma/idxd/irq.c                        |   8 +
 include/linux/dmaengine.h                     |  55 +++---
 12 files changed, 268 insertions(+), 90 deletions(-)


base-commit: c03c1a14507dfaf8b856c2608f9ac216021fc222
prerequisite-patch-id: b50615a6696ce4ffa7e20911f77b9f7f0481f14f
prerequisite-patch-id: 36d833fd556bce04054a7e373ee46575186bb58c
prerequisite-patch-id: 3aeecac097eafb6812e10b9896a242c61c4ca5da
prerequisite-patch-id: 99e0468cdabf56a287c6a7f2d7ef06aa6d5a573e
prerequisite-patch-id: b1dd90551e6b671e751aae8d2a688d9db8f300f5
prerequisite-patch-id: c99a06be81e6a56d84a14bf9e00d2b48455c3a89
prerequisite-patch-id: 3e39b306130b75fedae70a328072dc9577818b9f
prerequisite-patch-id: b1d8fb5b5f7f478adca95f4c8b632480164d18b4
prerequisite-patch-id: b6d48ab83002f113833ba874be299c241788dd30
prerequisite-patch-id: 53bb73bdfd98cd78ca5d86b00ad7e96fcebf7ee6
prerequisite-patch-id: de44e2d44df13e8e07ae00df842d67c9d5c874d2
prerequisite-patch-id: 8191bc0ef2c849d2d9d55f73fc2169f695ea8594
prerequisite-patch-id: bfc55b104522278ca1fba36d3f4adb3a306267bc
prerequisite-patch-id: 12c1f4cba95509b8e7e87ce5c2f9c4265ec0c2df
prerequisite-patch-id: 6e859bfd95e53386bce02ac6585f09a067d0bb8e
prerequisite-patch-id: 2874936c0a5b3863db0309e06ee93008890a714e
prerequisite-patch-id: 85ef17aa68aecd398d3b47ad8df435ce715572e8
-- 
2.43.0

[PATCH 1/3] dmaengine: add dmaengine_async_is_tx_complete() for channel status queries

[Daniel Byrne]

Add dmaengine_async_is_tx_complete() as the client API for polling the
completion of one transaction. Like dma_async_is_tx_complete(), which
it deprecates, it routes the query to the channel's device_tx_status()
callback; the change is that it drops the @last and @used outputs,
which only make sense for providers that complete in cookie order.
A provider that tracks per-descriptor state can answer exactly from
that state, and a high-rate in-kernel polling user, the io_uring DSA
copy offload to be posted separately, depends on such answers.

The kernel-doc says how far such an answer goes. It is exact for a
transaction submitted without DMA_PREP_INTERRUPT, on the first query
that reports it finished. An interrupt-armed transaction reports its
result through callback_result, and a cookie whose descriptor has been
retired reads DMA_COMPLETE. The client guide's advice to pause or stop
the channel before a query points at that text.

dma_async_is_complete() was only used through dma_cookie_status(), so
delete it from the client header and open-code the cookie-window
comparison there. Move dma_set_tx_state(), which only providers call,
from the client header into drivers/dma/dmaengine.h; every in-tree
user already includes it. Convert dma_sync_wait() and dmatest to the
new helper, fix the two comments that still name the deleted helper,
and document the cookie assignment helpers in the provider guide.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Daniel Byrne <daniel1.byrne@intel.com>
---
 Documentation/driver-api/dmaengine/client.rst | 24 ++++-----
 .../driver-api/dmaengine/provider.rst         | 45 ++++++++++++++--
 drivers/dma/amba-pl08x.c                      |  2 +-
 drivers/dma/at_hdmac.c                        |  5 +-
 drivers/dma/dmaengine.c                       |  2 +-
 drivers/dma/dmaengine.h                       | 21 +++++++-
 drivers/dma/dmatest.c                         |  3 +-
 include/linux/dmaengine.h                     | 54 +++++++++----------
 8 files changed, 103 insertions(+), 53 deletions(-)

diff --git a/Documentation/driver-api/dmaengine/client.rst b/Documentation/driver-api/dmaengine/client.rst
index 5ee5d4a3596d..255aaad33339 100644
--- a/Documentation/driver-api/dmaengine/client.rst
+++ b/Documentation/driver-api/dmaengine/client.rst
@@ -277,8 +277,8 @@ The details of these operations are:
 
       dma_cookie_t dmaengine_submit(struct dma_async_tx_descriptor *desc)
 
-   This returns a cookie can be used to check the progress of DMA engine
-   activity via other DMA engine calls not covered in this document.
+   This returns a cookie that can be used to check the progress of a transaction
+   via dmaengine_async_is_tx_complete().
 
    dmaengine_submit() will not start the DMA operation, it merely adds
    it to the pending queue. For this, see step 5, dma_async_issue_pending.
@@ -357,23 +357,23 @@ Further APIs
 
    .. code-block:: c
 
-      enum dma_status dma_async_is_tx_complete(struct dma_chan *chan,
-		dma_cookie_t cookie, dma_cookie_t *last, dma_cookie_t *used)
+      enum dma_status dmaengine_async_is_tx_complete(struct dma_chan *chan,
+		dma_cookie_t cookie)
 
-   This can be used to check the status of the channel. Please see
-   the documentation in include/linux/dmaengine.h for a more complete
-   description of this API.
-
-   This can be used in conjunction with dma_async_is_complete() and
-   the cookie returned from dmaengine_submit() to check for
-   completion of a specific DMA transaction.
+   This can be used with the cookie returned from dmaengine_submit()
+   to check for completion of a specific DMA transaction.
 
    .. note::
 
       Not all DMA engine drivers can return reliable information for
       a running DMA channel. It is recommended that DMA engine users
       pause or stop (via dmaengine_terminate_all()) the channel before
-      using this API.
+      using this API, unless the provider tracks per-descriptor state.
+      Even such a provider answers exactly only within the limits that
+      the kernel-doc of dmaengine_async_is_tx_complete() in
+      include/linux/dmaengine.h describes: for a transaction submitted
+      without DMA_PREP_INTERRUPT, and on the first query that reports it
+      finished.
 
 5. Synchronize termination API
 
diff --git a/Documentation/driver-api/dmaengine/provider.rst b/Documentation/driver-api/dmaengine/provider.rst
index f4ed98f701c9..ef342886af96 100644
--- a/Documentation/driver-api/dmaengine/provider.rst
+++ b/Documentation/driver-api/dmaengine/provider.rst
@@ -419,7 +419,9 @@ supported.
 
     - tx_submit: A pointer to a function you have to implement,
       that is supposed to push the current transaction descriptor to a
-      pending queue, waiting for issue_pending to be called.
+      pending queue, waiting for issue_pending to be called. Each
+      descriptor is given a cookie to identify it. See the section
+      "Cookie Management" below.
 
     - phys: Physical address of the descriptor which is used later by
       the dma engine to read the descriptor and initiate transfer.
@@ -533,6 +535,41 @@ supported.
 
   - May sleep.
 
+Cookie Management
+------------------
+
+When a transaction is queued for submission via tx_submit(), the provider
+must assign that transaction a cookie (dma_cookie_t) to uniquely identify it.
+The provider is allowed to perform this assignment however it wants, but for
+convenience the following utility functions are available to create
+monotonically increasing cookies:
+
+  .. code-block:: c
+
+    void dma_cookie_init(struct dma_chan *chan);
+
+  Called once at channel creation.
+
+  .. code-block:: c
+
+    dma_cookie_t dma_cookie_assign(struct dma_async_tx_descriptor *tx);
+
+  Assign a cookie to the given descriptor.
+
+  .. code-block:: c
+
+    void dma_cookie_complete(struct dma_async_tx_descriptor *tx);
+
+  Mark the descriptor as complete and invalidate the cookie.
+
+  .. code-block:: c
+
+    enum dma_status dma_cookie_status(struct dma_chan *chan,
+      dma_cookie_t cookie, struct dma_tx_state *state);
+
+  Report the status of the cookie, filling in the state structure if
+  it is not NULL.
+
 
 Misc notes
 ==========
@@ -550,10 +587,10 @@ where to put them)
 
 dma_cookie_t
 
-- it's a DMA transaction ID that will increment over time.
+- it's a DMA transaction ID.
 
-- Not really relevant any more since the introduction of ``virt-dma``
-  that abstracts it away.
+- The value can be chosen by the provider, or use the helper APIs
+  such as dma_cookie_assign() and dma_cookie_complete().
 
 dma_vec
 
diff --git a/drivers/dma/amba-pl08x.c b/drivers/dma/amba-pl08x.c
index 5e88fd44812d..8c235a603bb0 100644
--- a/drivers/dma/amba-pl08x.c
+++ b/drivers/dma/amba-pl08x.c
@@ -1536,7 +1536,7 @@ static void pl08x_free_chan_resources(struct dma_chan *chan)
 }
 
 /*
- * Code accessing dma_async_is_complete() in a tight loop may give problems.
+ * Code polling the transaction status in a tight loop may give problems.
  * If slaves are relying on interrupts to signal completion this function
  * must not be called with interrupts disabled.
  */
diff --git a/drivers/dma/at_hdmac.c b/drivers/dma/at_hdmac.c
index e5b30a57c477..7ea0822a3900 100644
--- a/drivers/dma/at_hdmac.c
+++ b/drivers/dma/at_hdmac.c
@@ -1666,9 +1666,8 @@ static int atc_terminate_all(struct dma_chan *chan)
  * @cookie: transaction identifier to check status of
  * @txstate: if not %NULL updated with transaction state
  *
- * If @txstate is passed in, upon return it reflect the driver
- * internal state and can be used with dma_async_is_complete() to check
- * the status of multiple cookies without re-checking hardware state.
+ * If @txstate is passed in, upon return it reflects the driver
+ * internal state: the last completed and the last issued cookie.
  */
 static enum dma_status
 atc_tx_status(struct dma_chan *chan,
diff --git a/drivers/dma/dmaengine.c b/drivers/dma/dmaengine.c
index ced78c985d1c..d7febef65a2d 100644
--- a/drivers/dma/dmaengine.c
+++ b/drivers/dma/dmaengine.c
@@ -537,7 +537,7 @@ enum dma_status dma_sync_wait(struct dma_chan *chan, dma_cookie_t cookie)
 
 	dma_async_issue_pending(chan);
 	do {
-		status = dma_async_is_tx_complete(chan, cookie, NULL, NULL);
+		status = dmaengine_async_is_tx_complete(chan, cookie);
 		if (time_after_eq(jiffies, dma_sync_wait_timeout)) {
 			dev_err(chan->device->dev, "%s: timeout!\n", __func__);
 			return DMA_ERROR;
diff --git a/drivers/dma/dmaengine.h b/drivers/dma/dmaengine.h
index 53f16d3f0029..e72876a512a3 100644
--- a/drivers/dma/dmaengine.h
+++ b/drivers/dma/dmaengine.h
@@ -79,7 +79,26 @@ static inline enum dma_status dma_cookie_status(struct dma_chan *chan,
 		state->residue = 0;
 		state->in_flight_bytes = 0;
 	}
-	return dma_async_is_complete(cookie, complete, used);
+
+	if (complete <= used) {
+		if ((cookie <= complete) || (cookie > used))
+			return DMA_COMPLETE;
+	} else {
+		if ((cookie <= complete) && (cookie > used))
+			return DMA_COMPLETE;
+	}
+	return DMA_IN_PROGRESS;
+}
+
+static inline void dma_set_tx_state(struct dma_tx_state *st,
+	dma_cookie_t last, dma_cookie_t used, u32 residue)
+{
+	if (!st)
+		return;
+
+	st->last = last;
+	st->used = used;
+	st->residue = residue;
 }
 
 static inline void dma_set_residue(struct dma_tx_state *state, u32 residue)
diff --git a/drivers/dma/dmatest.c b/drivers/dma/dmatest.c
index 2ae3469397f3..3d0d54a97add 100644
--- a/drivers/dma/dmatest.c
+++ b/drivers/dma/dmatest.c
@@ -847,8 +847,7 @@ static int dmatest_func(void *data)
 					done->done,
 					msecs_to_jiffies(params->timeout));
 
-			status = dma_async_is_tx_complete(chan, cookie, NULL,
-							  NULL);
+			status = dmaengine_async_is_tx_complete(chan, cookie);
 		}
 
 		if (!done->done) {
diff --git a/include/linux/dmaengine.h b/include/linux/dmaengine.h
index 4178248611d9..83bef7a70b99 100644
--- a/include/linux/dmaengine.h
+++ b/include/linux/dmaengine.h
@@ -1596,9 +1596,12 @@ static inline void dma_async_issue_pending(struct dma_chan *chan)
  * @last: returns last completed cookie, can be NULL
  * @used: returns last issued cookie, can be NULL
  *
- * If @last and @used are passed in, upon return they reflect the driver
- * internal state and can be used with dma_async_is_complete() to check
- * the status of multiple cookies without re-checking hardware state.
+ * This function is deprecated. New code should use
+ * dmaengine_async_is_tx_complete() instead.
+ *
+ * If @last and @used are passed in, upon return they reflect the most
+ * recently submitted (used) cookie and the most recently completed
+ * cookie.
  */
 static inline enum dma_status dma_async_is_tx_complete(struct dma_chan *chan,
 	dma_cookie_t cookie, dma_cookie_t *last, dma_cookie_t *used)
@@ -1615,36 +1618,29 @@ static inline enum dma_status dma_async_is_tx_complete(struct dma_chan *chan,
 }
 
 /**
- * dma_async_is_complete - test a cookie against chan state
- * @cookie: transaction identifier to test status of
- * @last_complete: last know completed transaction
- * @last_used: last cookie value handed out
+ * dmaengine_async_is_tx_complete - poll for transaction completion
+ * @chan: DMA channel
+ * @cookie: transaction identifier to check status of
  *
- * dma_async_is_complete() is used in dma_async_is_tx_complete()
- * the test logic is separated for lightweight testing of multiple cookies
+ * Query the provider's device_tx_status() callback for the state of a
+ * single transaction. Providers that rely on cookie ordering answer with
+ * the cookie-window approximation. Providers that track per-descriptor
+ * state can answer exactly, but only for a transaction submitted without
+ * DMA_PREP_INTERRUPT and only on the first query that reports it
+ * finished. A transaction submitted with DMA_PREP_INTERRUPT reports its
+ * result through callback_result, and the query only tells whether it
+ * has finished. Once the first terminal answer has retired the
+ * descriptor, a later query for the same cookie reads DMA_COMPLETE.
+ *
+ * Return: DMA_COMPLETE, DMA_IN_PROGRESS, DMA_PAUSED, DMA_ERROR or
+ * DMA_OUT_OF_ORDER.
  */
-static inline enum dma_status dma_async_is_complete(dma_cookie_t cookie,
-			dma_cookie_t last_complete, dma_cookie_t last_used)
-{
-	if (last_complete <= last_used) {
-		if ((cookie <= last_complete) || (cookie > last_used))
-			return DMA_COMPLETE;
-	} else {
-		if ((cookie <= last_complete) && (cookie > last_used))
-			return DMA_COMPLETE;
-	}
-	return DMA_IN_PROGRESS;
-}
-
-static inline void
-dma_set_tx_state(struct dma_tx_state *st, dma_cookie_t last, dma_cookie_t used, u32 residue)
+static inline enum dma_status dmaengine_async_is_tx_complete(struct dma_chan *chan,
+	dma_cookie_t cookie)
 {
-	if (!st)
-		return;
+	struct dma_tx_state state;
 
-	st->last = last;
-	st->used = used;
-	st->residue = residue;
+	return chan->device->device_tx_status(chan, cookie, &state);
 }
 
 #ifdef CONFIG_DMA_ENGINE
-- 
2.43.0

[PATCH 2/3] dmaengine: idxd: generation-number transaction status

[Daniel Byrne]

Implement device_tx_status() by reading the addressed descriptor's
completion record. The cookie encodes the descriptor id in its lower
16 bits and a generation number in the 14 bits above, so a query
against a recycled descriptor sees a stale cookie and reads
DMA_COMPLETE. The generation wraps after 16383 submissions of the same
slot; the contract is that a client polls a cookie to its terminal
answer, or abandons it, before its slot has been reused 16383 times,
which every in-tree polling client satisfies by construction.

Descriptors with a completion interrupt armed stay with the IRQ
handler: a generic client such as dmatest may request an interrupt,
and reading such a descriptor's completion record here would race with
the handler recycling it. tx_status therefore reports DMA_IN_PROGRESS
while such a cookie is live and the stale-cookie check answers once the
handler has completed it. Whether a descriptor is interrupt-armed
travels in the cookie, in the bit below the sign bit, since the
descriptor's own flags are rewritten when the handler recycles it.

A polled descriptor has two terminal answers. Its completion record
gives the first; the query retires the descriptor and reports
DMA_COMPLETE or DMA_ERROR from the status. The second is DMA_ERROR for
a descriptor outstanding on a work queue or a device that is no longer
enabled, since nothing will complete it; the slot is retired only when
the device halted, which kills it for good. Descriptors carry the
device's reset epoch from submit time for that, and the halt stores its
state before it bumps the epoch, so a poll that read the old epoch and
the halted state reads the epoch again and retires on a mismatch. The
same epoch lets device_synchronize() treat a drain that a halt ended
as complete, since the halt killed everything the drain covered.

Switch the prep callbacks' descriptor allocation to IDXD_OP_NONBLOCK:
prep callbacks may not sleep, and on an exhausted pool they return NULL
so the caller falls back. Direct kernel clients such as iaa_crypto keep
the blocking allocator path. This gives dmaengine_async_is_tx_complete()
exact per-descriptor results for polling clients on a device whose work
queues complete out of order across a group's engines.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Daniel Byrne <daniel1.byrne@intel.com>
---
 drivers/dma/idxd/device.c |   1 +
 drivers/dma/idxd/dma.c    | 158 ++++++++++++++++++++++++++++++++++++--
 drivers/dma/idxd/idxd.h   |   3 +
 drivers/dma/idxd/irq.c    |   8 ++
 4 files changed, 165 insertions(+), 5 deletions(-)

diff --git a/drivers/dma/idxd/device.c b/drivers/dma/idxd/device.c
index 5ccdc5ff8d53..68a7de74f9e6 100644
--- a/drivers/dma/idxd/device.c
+++ b/drivers/dma/idxd/device.c
@@ -155,6 +155,7 @@ int idxd_wq_alloc_resources(struct idxd_wq *wq)
 			desc->iax_completion = &wq->iax_compls[i];
 		desc->compl_dma = wq->compls_addr + idxd->data->compl_size * i;
 		desc->id = i;
+		desc->gen = 1;
 		desc->wq = wq;
 		desc->cpu = -1;
 	}
diff --git a/drivers/dma/idxd/dma.c b/drivers/dma/idxd/dma.c
index 0a7517cb51f2..dd5a9d9274ce 100644
--- a/drivers/dma/idxd/dma.c
+++ b/drivers/dma/idxd/dma.c
@@ -12,6 +12,20 @@
 #include "registers.h"
 #include "idxd.h"
 
+#define DMA_COOKIE_BITS (sizeof(dma_cookie_t) * 8)
+/*
+ * The descriptor id takes the lower 16 bits of the cookie.
+ */
+#define DESC_ID_BITS 16
+#define DESC_ID_MASK ((1 << DESC_ID_BITS) - 1)
+/*
+ * The generation takes the 14 bits above the id. Since dma_cookie_t is
+ * signed, the upper-most bit is left for the sign, and the bit below it
+ * records that the descriptor completes by interrupt.
+ */
+#define IDXD_COOKIE_RCI BIT(DMA_COOKIE_BITS - 2)
+#define DESC_GEN_MAX ((1 << (DMA_COOKIE_BITS - DESC_ID_BITS - 2)) - 1)
+
 static inline struct idxd_wq *to_idxd_wq(struct dma_chan *c)
 {
 	struct idxd_dma_chan *idxd_chan;
@@ -93,7 +107,7 @@ idxd_dma_prep_interrupt(struct dma_chan *c, unsigned long flags)
 		return NULL;
 
 	op_flag_setup(flags, &desc_flags);
-	desc = idxd_alloc_desc(wq, IDXD_OP_BLOCK);
+	desc = idxd_alloc_desc(wq, IDXD_OP_NONBLOCK);
 	if (IS_ERR(desc))
 		return NULL;
 
@@ -120,7 +134,7 @@ idxd_dma_submit_memcpy(struct dma_chan *c, dma_addr_t dma_dest,
 		return NULL;
 
 	op_flag_setup(flags, &desc_flags);
-	desc = idxd_alloc_desc(wq, IDXD_OP_BLOCK);
+	desc = idxd_alloc_desc(wq, IDXD_OP_NONBLOCK);
 	if (IS_ERR(desc))
 		return NULL;
 
@@ -154,11 +168,128 @@ static void idxd_dma_free_chan_resources(struct dma_chan *chan)
 		idxd_wq_refcount(wq));
 }
 
+/*
+ * A cookie has exactly one polling owner, the client that submitted the
+ * transaction. Observing a terminal completion record here retires and
+ * recycles the descriptor, since a polled descriptor has no interrupt
+ * and nobody else will. Two contexts polling the same live cookie
+ * would race that recycle, which every in-tree dmaengine polling
+ * client already avoids by construction.
+ */
 static enum dma_status idxd_dma_tx_status(struct dma_chan *dma_chan,
 					  dma_cookie_t cookie,
 					  struct dma_tx_state *txstate)
 {
-	return DMA_OUT_OF_ORDER;
+	struct idxd_wq *wq;
+	struct idxd_desc *desc;
+	u32 idx;
+	u8 status;
+
+	if (txstate)
+		memset(txstate, 0, sizeof(*txstate));
+
+	if (dma_submit_error(cookie))
+		return DMA_ERROR;
+
+	wq = to_idxd_wq(dma_chan);
+
+	idx = cookie & DESC_ID_MASK;
+	if (idx >= wq->num_descs)
+		return DMA_ERROR;
+
+	desc = wq->descs[idx];
+
+	/*
+	 * Descriptors with a completion interrupt armed (RCI) are
+	 * completed and recycled by the IRQ handler. Reading their
+	 * completion record here would race with the handler, and so
+	 * would reading the recycled descriptor's flags, which is why
+	 * the property travels in the cookie the caller owns. The handler
+	 * invalidates the cookie on completion, so a matching cookie means
+	 * the command is still in progress.
+	 */
+	if (cookie & IDXD_COOKIE_RCI)
+		return READ_ONCE(desc->txd.cookie) == cookie ?
+			DMA_IN_PROGRESS : DMA_COMPLETE;
+
+	if (desc->txd.cookie != cookie) {
+		/*
+		 * The cookie belongs to an old transaction that has
+		 * already completed and been recycled.
+		 */
+		return DMA_COMPLETE;
+	}
+
+	status = desc->completion->status & DSA_COMP_STATUS_MASK;
+
+	if (status) {
+		/* Capture completion record fields before desc is freed below. */
+		u8 fault_info = desc->completion->fault_info;
+		u64 fault_addr = desc->completion->fault_addr;
+
+		/*
+		 * Check against the original status because ABORT is
+		 * software defined as 0xff, which DSA_COMP_STATUS_MASK
+		 * can mask out.
+		 */
+		if (unlikely(desc->completion->status == IDXD_COMP_DESC_ABORT))
+			idxd_dma_complete_txd(desc, IDXD_COMPLETE_ABORT, true, NULL, NULL);
+		else
+			idxd_dma_complete_txd(desc, IDXD_COMPLETE_NORMAL, true, NULL, NULL);
+
+		dev_dbg(&wq->idxd->pdev->dev,
+			"dsa completion: status=0x%02x fault_info=0x%02x fault_addr=0x%llx\n",
+			status, fault_info, fault_addr);
+
+		if (status == DSA_COMP_SUCCESS)
+			return DMA_COMPLETE;
+
+		return DMA_ERROR;
+	}
+
+	/*
+	 * The descriptor is genuinely outstanding. A polled descriptor
+	 * carries no interrupt, so nothing but this function will ever
+	 * retire it, and a work queue or device that is no longer enabled
+	 * will not complete it either. Report the failure instead of
+	 * leaving the caller polling forever.
+	 *
+	 * A descriptor submitted before a device halt is dead for good:
+	 * the halt ended the engines and the reset that follows starts
+	 * them empty, so its record will never be written even once the
+	 * device is enabled again. Only in that case can the descriptor
+	 * be retired here safely; otherwise the pool would shrink by one
+	 * for every descriptor caught by a halt.
+	 */
+	if (desc->epoch != READ_ONCE(wq->idxd->reset_epoch)) {
+		idxd_dma_complete_txd(desc, IDXD_COMPLETE_ABORT, true, NULL, NULL);
+		return DMA_ERROR;
+	}
+	/*
+	 * A device that is no longer enabled runs nothing it holds, so an
+	 * outstanding descriptor on it has failed. A halt stores its state
+	 * before it bumps the epoch, and this poll may have read the epoch
+	 * before the halt and the state after it, so the epoch is read
+	 * again: only the retire frees the slot of a descriptor the halt
+	 * caught. A halted device whose epoch still matches is between the
+	 * two stores, and the next poll retires the descriptor. The submit
+	 * reads the state after the stamp, so it refuses a descriptor
+	 * stamped with the bumped epoch.
+	 */
+	if (wq->idxd->state != IDXD_DEV_ENABLED) {
+		if (desc->epoch != READ_ONCE(wq->idxd->reset_epoch)) {
+			idxd_dma_complete_txd(desc, IDXD_COMPLETE_ABORT, true,
+					      NULL, NULL);
+			return DMA_ERROR;
+		}
+		if (wq->idxd->state == IDXD_DEV_HALTED)
+			return DMA_IN_PROGRESS;
+		return DMA_ERROR;
+	}
+	if (wq->state != IDXD_WQ_ENABLED)
+		return DMA_ERROR;
+
+	return DMA_IN_PROGRESS;
 }
 
 /*
@@ -177,7 +308,17 @@ static dma_cookie_t idxd_dma_tx_submit(struct dma_async_tx_descriptor *tx)
 	int rc;
 	struct idxd_desc *desc = container_of(tx, struct idxd_desc, txd);
 
-	cookie = dma_cookie_assign(tx);
+	cookie = (desc->gen << DESC_ID_BITS) | (desc->id & DESC_ID_MASK);
+	if (desc->hw->flags & IDXD_OP_FLAG_RCI)
+		cookie |= IDXD_COOKIE_RCI;
+
+	if (desc->gen == DESC_GEN_MAX)
+		desc->gen = 1;
+	else
+		desc->gen++;
+
+	tx->cookie = cookie;
+	desc->epoch = READ_ONCE(wq->idxd->reset_epoch);
 
 	rc = idxd_submit_desc(wq, desc);
 	if (rc < 0) {
@@ -219,6 +360,7 @@ static void idxd_dma_synchronize(struct dma_chan *c)
 {
 	struct idxd_wq *wq = to_idxd_wq(c);
 	LIST_HEAD(quarantined);
+	u32 epoch;
 
 	/*
 	 * dmaengine_synchronize() promises the client that its memory may
@@ -240,7 +382,13 @@ static void idxd_dma_synchronize(struct dma_chan *c)
 	 * Only a device reset ends the exposure of the client's memory.
 	 */
 	idxd_wq_quarantine_take(wq, &quarantined);
-	if (!idxd_wq_drain(wq)) {
+	/*
+	 * A halt during the drain kills every descriptor submitted before
+	 * it, which is all of them, and a reset that followed may have
+	 * ended the drain itself. Such a drain is as good as complete.
+	 */
+	epoch = READ_ONCE(wq->idxd->reset_epoch);
+	if (!idxd_wq_drain(wq) || READ_ONCE(wq->idxd->reset_epoch) != epoch) {
 		idxd_wq_quarantine_free(wq, &quarantined);
 		WRITE_ONCE(wq->stuck, false);
 		return;
diff --git a/drivers/dma/idxd/idxd.h b/drivers/dma/idxd/idxd.h
index b0cfd7101174..7fe5ec6c3bf3 100644
--- a/drivers/dma/idxd/idxd.h
+++ b/drivers/dma/idxd/idxd.h
@@ -349,6 +349,7 @@ struct idxd_device {
 	spinlock_t cmd_lock;	/* spinlock for device commands */
 	struct completion *cmd_done;
 	u32 cmd_seq;		/* bumps when a command starts running */
+	u32 reset_epoch;	/* bumps on every device halt */
 	struct idxd_group **groups;
 	struct idxd_wq **wqs;
 	struct idxd_engine **engines;
@@ -447,9 +448,11 @@ struct idxd_desc {
 	struct list_head list;
 	int id;
 	int cpu;
+	u16 gen;
 	u16 terminate_gen;	/* the queue's terminate_gen at an
 				 * interrupt-armed submit
 				 */
+	u32 epoch;		/* the device reset_epoch at submit */
 	struct idxd_wq *wq;
 };
 
diff --git a/drivers/dma/idxd/irq.c b/drivers/dma/idxd/irq.c
index 8217698056bb..5e34790359ee 100644
--- a/drivers/dma/idxd/irq.c
+++ b/drivers/dma/idxd/irq.c
@@ -508,6 +508,14 @@ static irqreturn_t idxd_halt(struct idxd_device *idxd)
 	gensts.bits = ioread32(idxd->reg_base + IDXD_GENSTATS_OFFSET);
 	if (gensts.state == IDXD_DEVICE_STATE_HALT) {
 		idxd->state = IDXD_DEV_HALTED;
+		/*
+		 * Every descriptor submitted before this point is dead: the
+		 * device will never write its completion record. Pollers
+		 * compare their descriptor's epoch against this to retire
+		 * such descriptors instead of waiting for a record that never
+		 * comes.
+		 */
+		WRITE_ONCE(idxd->reset_epoch, idxd->reset_epoch + 1);
 		if (gensts.reset_type == IDXD_DEVICE_RESET_SOFTWARE) {
 			/* The reinit below re-enables the queues, so revive them. */
 			idxd_wqs_halt_flush(idxd, true);
-- 
2.43.0

[PATCH 3/3] dmaengine: drop DMA_COMPLETION_NO_ORDER and DMA_OUT_OF_ORDER

[Daniel Byrne]

DMA_COMPLETION_NO_ORDER (commit 47ec7f09bc10 ("dmaengine: cookie bypass
for out of order completion")) flagged devices whose completions can
overtake submission order. On DSA this happens when a work queue (one
dmaengine channel) belongs to a group that contains several engines.
Per-channel cookie tracking gives wrong answers in this case, since
dma_cookie_complete() moves the completed-cookie watermark past
still-in-flight earlier descriptors, so cookie-order status can report
DMA_COMPLETE for a running transfer. Clients were told to use
callbacks only, dmatest refused polled mode, and DMA_OUT_OF_ORDER
existed as the tx_status answer for such devices.

idxd, the only in-tree setter, now answers
dmaengine_async_is_tx_complete() from per-descriptor generation
numbers, exactly within the limits that helper's kernel-doc states: for
a transaction submitted without DMA_PREP_INTERRUPT, on the first query
that reports it finished. Polling an out-of-order device is therefore
safe and neither the capability nor the status code has any remaining
meaning. Drop them.

Whether reordering occurs at all is group configuration rather than a
device property. A group with a single engine completes its work
queues' descriptors in order. However, the driver cannot assume how
the administrator grouped engines, and the all-engines-in-one-group
layout is the device's throughput configuration.

Assisted-by: Claude:claude-opus-5-5
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Daniel Byrne <daniel1.byrne@intel.com>
---
 .../driver-api/dmaengine/provider.rst         | 19 -------------------
 drivers/dma/dmatest.c                         | 11 +----------
 drivers/dma/idxd/dma.c                        |  1 -
 include/linux/dmaengine.h                     |  5 +----
 4 files changed, 2 insertions(+), 34 deletions(-)

diff --git a/Documentation/driver-api/dmaengine/provider.rst b/Documentation/driver-api/dmaengine/provider.rst
index ef342886af96..435ccd3bedac 100644
--- a/Documentation/driver-api/dmaengine/provider.rst
+++ b/Documentation/driver-api/dmaengine/provider.rst
@@ -260,22 +260,6 @@ Currently, the types available are:
     want to transfer a portion of uncompressed data directly to the
     display to print it
 
-- DMA_COMPLETION_NO_ORDER
-
-  - The device does not support in order completion.
-
-  - The driver should return DMA_OUT_OF_ORDER for device_tx_status if
-    the device is setting this capability.
-
-  - All cookie tracking and checking API should be treated as invalid if
-    the device exports this capability.
-
-  - At this point, this is incompatible with polling option for dmatest.
-
-  - If this cap is set, the user is recommended to provide an unique
-    identifier for each descriptor sent to the DMA device in order to
-    properly track the completion.
-
 - DMA_REPEAT
 
   - The device supports repeated transfers. A repeated transfer, indicated by
@@ -468,9 +452,6 @@ supported.
   - In the case of a cyclic transfer, it should only take into
     account the total size of the cyclic buffer.
 
-  - Should return DMA_OUT_OF_ORDER if the device does not support in order
-    completion and is completing the operation out of order.
-
   - This function can be called in an interrupt context.
 
 - device_config
diff --git a/drivers/dma/dmatest.c b/drivers/dma/dmatest.c
index 3d0d54a97add..fbfa2f1d6f29 100644
--- a/drivers/dma/dmatest.c
+++ b/drivers/dma/dmatest.c
@@ -854,10 +854,7 @@ static int dmatest_func(void *data)
 			result("test timed out", total_tests, src->off, dst->off,
 			       len, 0);
 			goto error_unmap_continue;
-		} else if (status != DMA_COMPLETE &&
-			   !(dma_has_cap(DMA_COMPLETION_NO_ORDER,
-					 dev->cap_mask) &&
-			     status == DMA_OUT_OF_ORDER)) {
+		} else if (status != DMA_COMPLETE) {
 			result(status == DMA_ERROR ?
 			       "completion error status" :
 			       "completion busy status", total_tests, src->off,
@@ -1035,12 +1032,6 @@ static int dmatest_add_channel(struct dmatest_info *info,
 	dtc->chan = chan;
 	INIT_LIST_HEAD(&dtc->threads);
 
-	if (dma_has_cap(DMA_COMPLETION_NO_ORDER, dma_dev->cap_mask) &&
-	    info->params.polled) {
-		info->params.polled = false;
-		pr_warn("DMA_COMPLETION_NO_ORDER, polled disabled\n");
-	}
-
 	if (dma_has_cap(DMA_MEMCPY, dma_dev->cap_mask)) {
 		if (dmatest == 0) {
 			cnt = dmatest_add_threads(info, dtc, DMA_MEMCPY);
diff --git a/drivers/dma/idxd/dma.c b/drivers/dma/idxd/dma.c
index dd5a9d9274ce..19340c7ae2d3 100644
--- a/drivers/dma/idxd/dma.c
+++ b/drivers/dma/idxd/dma.c
@@ -417,7 +417,6 @@ int idxd_register_dma_device(struct idxd_device *idxd)
 
 	dma_cap_set(DMA_INTERRUPT, dma->cap_mask);
 	dma_cap_set(DMA_PRIVATE, dma->cap_mask);
-	dma_cap_set(DMA_COMPLETION_NO_ORDER, dma->cap_mask);
 	dma->device_release = idxd_dma_release;
 
 	dma->device_prep_dma_interrupt = idxd_dma_prep_interrupt;
diff --git a/include/linux/dmaengine.h b/include/linux/dmaengine.h
index 83bef7a70b99..9f73470759bb 100644
--- a/include/linux/dmaengine.h
+++ b/include/linux/dmaengine.h
@@ -39,7 +39,6 @@ enum dma_status {
 	DMA_IN_PROGRESS,
 	DMA_PAUSED,
 	DMA_ERROR,
-	DMA_OUT_OF_ORDER,
 };
 
 /**
@@ -62,7 +61,6 @@ enum dma_transaction_type {
 	DMA_SLAVE,
 	DMA_CYCLIC,
 	DMA_INTERLEAVE,
-	DMA_COMPLETION_NO_ORDER,
 	DMA_REPEAT,
 	DMA_LOAD_EOT,
 /* last transaction type for creation of the capabilities mask */
@@ -1632,8 +1630,7 @@ static inline enum dma_status dma_async_is_tx_complete(struct dma_chan *chan,
  * has finished. Once the first terminal answer has retired the
  * descriptor, a later query for the same cookie reads DMA_COMPLETE.
  *
- * Return: DMA_COMPLETE, DMA_IN_PROGRESS, DMA_PAUSED, DMA_ERROR or
- * DMA_OUT_OF_ORDER.
+ * Return: DMA_COMPLETE, DMA_IN_PROGRESS, DMA_PAUSED or DMA_ERROR.
  */
 static inline enum dma_status dmaengine_async_is_tx_complete(struct dma_chan *chan,
 	dma_cookie_t cookie)
-- 
2.43.0

Re: [PATCH 1/3] dmaengine: add dmaengine_async_is_tx_complete() for channel status queries

[Frank Li]

On Mon, Oct 19, 2026 at 08:06:30AM -0700, Daniel Byrne wrote:
> [You don't often get email from daniel1.byrne@intel.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>
> Add dmaengine_async_is_tx_complete() as the client API for polling the
> completion of one transaction. Like dma_async_is_tx_complete(), which
> it deprecates, it routes the query to the channel's device_tx_status()
> callback; the change is that it drops the @last and @used outputs,
> which only make sense for providers that complete in cookie order.
> A provider that tracks per-descriptor state can answer exactly from
> that state, and a high-rate in-kernel polling user, the io_uring DSA
> copy offload to be posted separately, depends on such answers.

Does it should wrap dmaengine_tx_status()?  from naming conversion

dmaengine_async_is_tx_complete() should return bool type.

>
> The kernel-doc says how far such an answer goes. It is exact for a
> transaction submitted without DMA_PREP_INTERRUPT, on the first query
> that reports it finished. An interrupt-armed transaction reports its
> result through callback_result, and a cookie whose descriptor has been
> retired reads DMA_COMPLETE. The client guide's advice to pause or stop
> the channel before a query points at that text.

I don't that it is true. Supposed user can always poll state regardless if
set DMA_PREP_INTERRUPT.

>
> dma_async_is_complete() was only used through dma_cookie_status(), so
> delete it from the client header and open-code the cookie-window
> comparison there.

> Move dma_set_tx_state(), which only providers call,
> from the client header into drivers/dma/dmaengine.h; every in-tree

Use sperate patch for such movement.

Frank

Re: [PATCH 2/3] dmaengine: idxd: generation-number transaction status

[Dave Jiang]

On 10/19/26 8:06 AM, Daniel Byrne wrote:
> Implement device_tx_status() by reading the addressed descriptor's
> completion record. The cookie encodes the descriptor id in its lower
> 16 bits and a generation number in the 14 bits above, so a query
> against a recycled descriptor sees a stale cookie and reads
> DMA_COMPLETE. The generation wraps after 16383 submissions of the same
> slot; the contract is that a client polls a cookie to its terminal
> answer, or abandons it, before its slot has been reused 16383 times,
> which every in-tree polling client satisfies by construction.
> 
> Descriptors with a completion interrupt armed stay with the IRQ
> handler: a generic client such as dmatest may request an interrupt,
> and reading such a descriptor's completion record here would race with
> the handler recycling it. tx_status therefore reports DMA_IN_PROGRESS
> while such a cookie is live and the stale-cookie check answers once the
> handler has completed it. Whether a descriptor is interrupt-armed
> travels in the cookie, in the bit below the sign bit, since the
> descriptor's own flags are rewritten when the handler recycles it.
> 
> A polled descriptor has two terminal answers. Its completion record
> gives the first; the query retires the descriptor and reports
> DMA_COMPLETE or DMA_ERROR from the status. The second is DMA_ERROR for
> a descriptor outstanding on a work queue or a device that is no longer
> enabled, since nothing will complete it; the slot is retired only when
> the device halted, which kills it for good. Descriptors carry the
> device's reset epoch from submit time for that, and the halt stores its
> state before it bumps the epoch, so a poll that read the old epoch and
> the halted state reads the epoch again and retires on a mismatch. The
> same epoch lets device_synchronize() treat a drain that a halt ended
> as complete, since the halt killed everything the drain covered.
> 
> Switch the prep callbacks' descriptor allocation to IDXD_OP_NONBLOCK:
> prep callbacks may not sleep, and on an exhausted pool they return NULL
> so the caller falls back. Direct kernel clients such as iaa_crypto keep
> the blocking allocator path. This gives dmaengine_async_is_tx_complete()
> exact per-descriptor results for polling clients on a device whose work
> queues complete out of order across a group's engines.
> 
> Assisted-by: Claude:claude-opus-5-5
> Assisted-by: Claude:claude-fable-5-1
> Signed-off-by: Daniel Byrne <daniel1.byrne@intel.com>
> ---
>  drivers/dma/idxd/device.c |   1 +
>  drivers/dma/idxd/dma.c    | 158 ++++++++++++++++++++++++++++++++++++--
>  drivers/dma/idxd/idxd.h   |   3 +
>  drivers/dma/idxd/irq.c    |   8 ++
>  4 files changed, 165 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/dma/idxd/device.c b/drivers/dma/idxd/device.c
> index 5ccdc5ff8d53..68a7de74f9e6 100644
> --- a/drivers/dma/idxd/device.c
> +++ b/drivers/dma/idxd/device.c
> @@ -155,6 +155,7 @@ int idxd_wq_alloc_resources(struct idxd_wq *wq)
>  			desc->iax_completion = &wq->iax_compls[i];
>  		desc->compl_dma = wq->compls_addr + idxd->data->compl_size * i;
>  		desc->id = i;
> +		desc->gen = 1;
>  		desc->wq = wq;
>  		desc->cpu = -1;
>  	}
> diff --git a/drivers/dma/idxd/dma.c b/drivers/dma/idxd/dma.c
> index 0a7517cb51f2..dd5a9d9274ce 100644
> --- a/drivers/dma/idxd/dma.c
> +++ b/drivers/dma/idxd/dma.c
> @@ -12,6 +12,20 @@
>  #include "registers.h"
>  #include "idxd.h"
>  
> +#define DMA_COOKIE_BITS (sizeof(dma_cookie_t) * 8)
> +/*
> + * The descriptor id takes the lower 16 bits of the cookie.
> + */
> +#define DESC_ID_BITS 16
> +#define DESC_ID_MASK ((1 << DESC_ID_BITS) - 1)
> +/*
> + * The generation takes the 14 bits above the id. Since dma_cookie_t is
> + * signed, the upper-most bit is left for the sign, and the bit below it
> + * records that the descriptor completes by interrupt.
> + */
> +#define IDXD_COOKIE_RCI BIT(DMA_COOKIE_BITS - 2)
> +#define DESC_GEN_MAX ((1 << (DMA_COOKIE_BITS - DESC_ID_BITS - 2)) - 1)
> +
>  static inline struct idxd_wq *to_idxd_wq(struct dma_chan *c)
>  {
>  	struct idxd_dma_chan *idxd_chan;
> @@ -93,7 +107,7 @@ idxd_dma_prep_interrupt(struct dma_chan *c, unsigned long flags)
>  		return NULL;
>  
>  	op_flag_setup(flags, &desc_flags);
> -	desc = idxd_alloc_desc(wq, IDXD_OP_BLOCK);
> +	desc = idxd_alloc_desc(wq, IDXD_OP_NONBLOCK);
>  	if (IS_ERR(desc))
>  		return NULL;
>  
> @@ -120,7 +134,7 @@ idxd_dma_submit_memcpy(struct dma_chan *c, dma_addr_t dma_dest,
>  		return NULL;
>  
>  	op_flag_setup(flags, &desc_flags);
> -	desc = idxd_alloc_desc(wq, IDXD_OP_BLOCK);
> +	desc = idxd_alloc_desc(wq, IDXD_OP_NONBLOCK);
>  	if (IS_ERR(desc))
>  		return NULL;
>  
> @@ -154,11 +168,128 @@ static void idxd_dma_free_chan_resources(struct dma_chan *chan)
>  		idxd_wq_refcount(wq));
>  }
>  
> +/*
> + * A cookie has exactly one polling owner, the client that submitted the
> + * transaction. Observing a terminal completion record here retires and
> + * recycles the descriptor, since a polled descriptor has no interrupt
> + * and nobody else will. Two contexts polling the same live cookie
> + * would race that recycle, which every in-tree dmaengine polling
> + * client already avoids by construction.
> + */
>  static enum dma_status idxd_dma_tx_status(struct dma_chan *dma_chan,
>  					  dma_cookie_t cookie,
>  					  struct dma_tx_state *txstate)
>  {
> -	return DMA_OUT_OF_ORDER;
> +	struct idxd_wq *wq;
> +	struct idxd_desc *desc;
> +	u32 idx;
> +	u8 status;
> +
> +	if (txstate)
> +		memset(txstate, 0, sizeof(*txstate));
> +
> +	if (dma_submit_error(cookie))
> +		return DMA_ERROR;
> +
> +	wq = to_idxd_wq(dma_chan);
> +
> +	idx = cookie & DESC_ID_MASK;
> +	if (idx >= wq->num_descs)
> +		return DMA_ERROR;
> +
> +	desc = wq->descs[idx];
> +
> +	/*
> +	 * Descriptors with a completion interrupt armed (RCI) are
> +	 * completed and recycled by the IRQ handler. Reading their
> +	 * completion record here would race with the handler, and so
> +	 * would reading the recycled descriptor's flags, which is why
> +	 * the property travels in the cookie the caller owns. The handler
> +	 * invalidates the cookie on completion, so a matching cookie means
> +	 * the command is still in progress.
> +	 */
> +	if (cookie & IDXD_COOKIE_RCI)
> +		return READ_ONCE(desc->txd.cookie) == cookie ?
> +			DMA_IN_PROGRESS : DMA_COMPLETE;
> +
> +	if (desc->txd.cookie != cookie) {
> +		/*
> +		 * The cookie belongs to an old transaction that has
> +		 * already completed and been recycled.
> +		 */
> +		return DMA_COMPLETE;
> +	}
> +
> +	status = desc->completion->status & DSA_COMP_STATUS_MASK;
> +
> +	if (status) {
> +		/* Capture completion record fields before desc is freed below. */
> +		u8 fault_info = desc->completion->fault_info;
> +		u64 fault_addr = desc->completion->fault_addr;
> +
> +		/*
> +		 * Check against the original status because ABORT is
> +		 * software defined as 0xff, which DSA_COMP_STATUS_MASK
> +		 * can mask out.
> +		 */
> +		if (unlikely(desc->completion->status == IDXD_COMP_DESC_ABORT))
> +			idxd_dma_complete_txd(desc, IDXD_COMPLETE_ABORT, true, NULL, NULL);
> +		else
> +			idxd_dma_complete_txd(desc, IDXD_COMPLETE_NORMAL, true, NULL, NULL);
> +
> +		dev_dbg(&wq->idxd->pdev->dev,
> +			"dsa completion: status=0x%02x fault_info=0x%02x fault_addr=0x%llx\n",
> +			status, fault_info, fault_addr);
> +
> +		if (status == DSA_COMP_SUCCESS)
> +			return DMA_COMPLETE;
> +
> +		return DMA_ERROR;
> +	}
> +
> +	/*
> +	 * The descriptor is genuinely outstanding. A polled descriptor
> +	 * carries no interrupt, so nothing but this function will ever
> +	 * retire it, and a work queue or device that is no longer enabled
> +	 * will not complete it either. Report the failure instead of
> +	 * leaving the caller polling forever.
> +	 *
> +	 * A descriptor submitted before a device halt is dead for good:
> +	 * the halt ended the engines and the reset that follows starts
> +	 * them empty, so its record will never be written even once the
> +	 * device is enabled again. Only in that case can the descriptor
> +	 * be retired here safely; otherwise the pool would shrink by one
> +	 * for every descriptor caught by a halt.
> +	 */
> +	if (desc->epoch != READ_ONCE(wq->idxd->reset_epoch)) {
> +		idxd_dma_complete_txd(desc, IDXD_COMPLETE_ABORT, true, NULL, NULL);
> +		return DMA_ERROR;
> +	}
> +	/*
> +	 * A device that is no longer enabled runs nothing it holds, so an
> +	 * outstanding descriptor on it has failed. A halt stores its state
> +	 * before it bumps the epoch, and this poll may have read the epoch
> +	 * before the halt and the state after it, so the epoch is read
> +	 * again: only the retire frees the slot of a descriptor the halt
> +	 * caught. A halted device whose epoch still matches is between the
> +	 * two stores, and the next poll retires the descriptor. The submit
> +	 * reads the state after the stamp, so it refuses a descriptor
> +	 * stamped with the bumped epoch.
> +	 */
> +	if (wq->idxd->state != IDXD_DEV_ENABLED) {
> +		if (desc->epoch != READ_ONCE(wq->idxd->reset_epoch)) {
> +			idxd_dma_complete_txd(desc, IDXD_COMPLETE_ABORT, true,
> +					      NULL, NULL);
> +			return DMA_ERROR;
> +		}
> +		if (wq->idxd->state == IDXD_DEV_HALTED)
> +			return DMA_IN_PROGRESS;
> +		return DMA_ERROR;
> +	}
> +	if (wq->state != IDXD_WQ_ENABLED)
> +		return DMA_ERROR;
> +
> +	return DMA_IN_PROGRESS;
>  }
>  
>  /*
> @@ -177,7 +308,17 @@ static dma_cookie_t idxd_dma_tx_submit(struct dma_async_tx_descriptor *tx)
>  	int rc;
>  	struct idxd_desc *desc = container_of(tx, struct idxd_desc, txd);
>  
> -	cookie = dma_cookie_assign(tx);
> +	cookie = (desc->gen << DESC_ID_BITS) | (desc->id & DESC_ID_MASK);
> +	if (desc->hw->flags & IDXD_OP_FLAG_RCI)
> +		cookie |= IDXD_COOKIE_RCI;
> +
> +	if (desc->gen == DESC_GEN_MAX)
> +		desc->gen = 1;
> +	else
> +		desc->gen++;
> +
> +	tx->cookie = cookie;
> +	desc->epoch = READ_ONCE(wq->idxd->reset_epoch);
>  
>  	rc = idxd_submit_desc(wq, desc);
>  	if (rc < 0) {
> @@ -219,6 +360,7 @@ static void idxd_dma_synchronize(struct dma_chan *c)
>  {
>  	struct idxd_wq *wq = to_idxd_wq(c);
>  	LIST_HEAD(quarantined);
> +	u32 epoch;
>  
>  	/*
>  	 * dmaengine_synchronize() promises the client that its memory may
> @@ -240,7 +382,13 @@ static void idxd_dma_synchronize(struct dma_chan *c)
>  	 * Only a device reset ends the exposure of the client's memory.
>  	 */
>  	idxd_wq_quarantine_take(wq, &quarantined);
> -	if (!idxd_wq_drain(wq)) {
> +	/*
> +	 * A halt during the drain kills every descriptor submitted before
> +	 * it, which is all of them, and a reset that followed may have
> +	 * ended the drain itself. Such a drain is as good as complete.
> +	 */
> +	epoch = READ_ONCE(wq->idxd->reset_epoch);
> +	if (!idxd_wq_drain(wq) || READ_ONCE(wq->idxd->reset_epoch) != epoch) {
>  		idxd_wq_quarantine_free(wq, &quarantined);
>  		WRITE_ONCE(wq->stuck, false);
>  		return;
> diff --git a/drivers/dma/idxd/idxd.h b/drivers/dma/idxd/idxd.h
> index b0cfd7101174..7fe5ec6c3bf3 100644
> --- a/drivers/dma/idxd/idxd.h
> +++ b/drivers/dma/idxd/idxd.h
> @@ -349,6 +349,7 @@ struct idxd_device {
>  	spinlock_t cmd_lock;	/* spinlock for device commands */
>  	struct completion *cmd_done;
>  	u32 cmd_seq;		/* bumps when a command starts running */
> +	u32 reset_epoch;	/* bumps on every device halt */
>  	struct idxd_group **groups;
>  	struct idxd_wq **wqs;
>  	struct idxd_engine **engines;
> @@ -447,9 +448,11 @@ struct idxd_desc {
>  	struct list_head list;
>  	int id;
>  	int cpu;
> +	u16 gen;
>  	u16 terminate_gen;	/* the queue's terminate_gen at an
>  				 * interrupt-armed submit
>  				 */
> +	u32 epoch;		/* the device reset_epoch at submit */
>  	struct idxd_wq *wq;
>  };
>  
> diff --git a/drivers/dma/idxd/irq.c b/drivers/dma/idxd/irq.c
> index 8217698056bb..5e34790359ee 100644
> --- a/drivers/dma/idxd/irq.c
> +++ b/drivers/dma/idxd/irq.c
> @@ -508,6 +508,14 @@ static irqreturn_t idxd_halt(struct idxd_device *idxd)
>  	gensts.bits = ioread32(idxd->reg_base + IDXD_GENSTATS_OFFSET);
>  	if (gensts.state == IDXD_DEVICE_STATE_HALT) {
>  		idxd->state = IDXD_DEV_HALTED;
> +		/*
> +		 * Every descriptor submitted before this point is dead: the
> +		 * device will never write its completion record. Pollers
> +		 * compare their descriptor's epoch against this to retire
> +		 * such descriptors instead of waiting for a record that never
> +		 * comes.
> +		 */
> +		WRITE_ONCE(idxd->reset_epoch, idxd->reset_epoch + 1);
>  		if (gensts.reset_type == IDXD_DEVICE_RESET_SOFTWARE) {
>  			/* The reinit below re-enables the queues, so revive them. */
>  			idxd_wqs_halt_flush(idxd, true);


LLM suggests possible race:
1. Recording the count too early. idxd_dma_tx_submit() saves the reset count into desc->epoch (dma.c:321) before calling idxd_submit_desc(). Until __idxd_submit_desc() checks idxd->state (submit.c:230) and takes wq_active, the submitter holds no lock and no reference.

2. The halt doesn't wait for this submitter. idxd_halt() (irq.c:509-529) sets HALTED and increments reset_epoch, then calls idxd_wqs_halt_flush(idxd, true). That flush's idxd_wq_kill() waits only for submitters holding wq_active, so it never waits on ours. It then revives wq_active and queues idxd_device_reinit().

3. Recovery finishes. idxd_device_reinit() (irq.c:32-70) resets, configures and enables the device, which sets ENABLED (device.c:810). It then re-enables the queue (device.c:217). Nothing in this path touches the descriptors or desc->epoch.

4. The descriptor runs on the new device. The submitter resumes and passes both state checks and the tryget. It then posts the descriptor, which still carries the old count, and the reset device runs it.

5. The poll frees it early. The first poll after the post sees status 0, and the descriptor matches the cookie. The desc->epoch != reset_epoch branch (dma.c:264-267) then calls idxd_dma_complete_txd(..., true, ...), which frees the slot. It returns DMA_ERROR while the copy is still running.

6. The late write corrupts the next user. __get_desc() clears the completion record when the slot is reused (submit.c:19). The device then writes its record to the same dener can see a result for a transfer that hasn't run.

LLM suggested fix:
Move the reset-count read (desc->epoch) out of idxd_dma_tx_submit() and into __idxd_submit_desc(), after the submitter holds wq_active and before the second state check. On the halt side, order the state store before the count store.

--- a/drivers/dma/idxd/dma.c
+++ b/drivers/dma/idxd/dma.c
@@ static dma_cookie_t idxd_dma_tx_submit(struct dma_async_tx_descriptor *tx)
        tx->cookie = cookie;
-       desc->epoch = READ_ONCE(wq->idxd->reset_epoch);

        rc = idxd_submit_desc(wq, desc);
--- a/drivers/dma/idxd/submit.c
+++ b/drivers/dma/idxd/submit.c
@@ static int __idxd_submit_desc(struct idxd_wq *wq, struct idxd_desc *desc,
        smp_rmb();
+       /*
+        * Stamp under wq_active: a halt that bumps the epoch after this
+        * read waits in idxd_wq_kill() for the reference, so an old stamp
+        * proves the post came before that halt's flush. A halt stores its
+        * state before its bump, so a bumped stamp fails the check below.
+        */
+       desc->epoch = READ_ONCE(idxd->reset_epoch);
+       smp_rmb();
        if (idxd->state != IDXD_DEV_ENABLED) {
                percpu_ref_put(&wq->wq_active);
                return -EIO;
        }
--- a/drivers/dma/idxd/irq.c
+++ b/drivers/dma/idxd/irq.c
@@ static irqreturn_t idxd_halt(struct idxd_device *idxd)
        if (gensts.state == IDXD_DEVICE_STATE_HALT) {
                idxd->state = IDXD_DEV_HALTED;
+               /* Pairs with the smp_rmb() after the submit's epoch stamp. */
+               smp_wmb();
                /*
                 * Every descriptor submitted before this point is dead: the

Re: [PATCH 3/3] dmaengine: drop DMA_COMPLETION_NO_ORDER and DMA_OUT_OF_ORDER

[Frank Li]

On Mon, Oct 19, 2026 at 08:06:32AM -0700, Daniel Byrne wrote:
> [You don't often get email from daniel1.byrne@intel.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>
> DMA_COMPLETION_NO_ORDER (commit 47ec7f09bc10 ("dmaengine: cookie bypass
> for out of order completion")) flagged devices whose completions can
> overtake submission order. On DSA this happens when a work queue (one
> dmaengine channel) belongs to a group that contains several engines.
> Per-channel cookie tracking gives wrong answers in this case, since
> dma_cookie_complete() moves the completed-cookie watermark past
> still-in-flight earlier descriptors, so cookie-order status can report
> DMA_COMPLETE for a running transfer. Clients were told to use
> callbacks only, dmatest refused polled mode, and DMA_OUT_OF_ORDER
> existed as the tx_status answer for such devices.

Thank you for such cleanup. But I have not understood what means. Does
DMA_OUT_OF_ORDER will never happen now?

Frank