Linux USB
 help / color / mirror / Atom feed
* [PATCH v5 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC)
@ 2026-11-16 19:06 Petar Stepanovic
  2026-11-16 19:06 ` [PATCH v5 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC Petar Stepanovic
                   ` (5 more replies)
  0 siblings, 6 replies; 13+ messages in thread
From: Petar Stepanovic @ 2026-11-16 19:06 UTC (permalink / raw)
  To: Prasad Bolisetty, Greg Kroah-Hartman, Rob Herring,
	Krzysztof Kozlowski, Conor Dooley, Tzu-Hao Wei,
	Karthikeyan Mitran, Philipp Zabel, Neill Kapron
  Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
	Petar Stepanovic, Krutik Shah, Naveen Kumar Rajgiri Bassappa,
	Krzysztof Kozlowski

Dear maintainers,

This series continues the work of adding support for the USB device
controller (UDC) integrated into the Axiado AX3000 and AX3005 SoCs.

The controller is based on the Corigine USB 3.1 IP core. The core
supports device and host roles, and this series adds support for device
mode on the Axiado AX3000 and AX3005 SoCs.

The controller uses MMIO registers for commands and TRB rings for
transfers and events. The driver manages the SETUP, DATA, and STATUS
stages of EP0 control transfers explicitly.

The driver follows the cdns2 source layout where applicable, with
separate files for core, gadget, EP0, and ring handling, and tracepoints
for controller and transfer events.

The devicetree binding also picks up one item left over from the v2
review: the description now notes the underlying core is dual-role
capable (device and host), used here in device role.

Tested on an Axiado AX3005 EVK with a mass-storage + composite HID
(keyboard/mouse) gadget, including repeated bind/unbind/rebind cycles
and host-driven bus resets.

Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
Changes in v5:
- Updated the CRG USB UDC maintainer list in MAINTAINERS.
- Fixed out-of-bounds TRB access, ring wrap handling, dequeue pointer
  updates, and transfer length calculations.
- Improved multi-TD request tracking and transfer-ring capacity
  management.
- Reworked endpoint dequeue, halt, and request completion handling.
- Added controller recovery for endpoint stop and disable timeouts,
  dequeue failures, and EP0 abort or reinitialization failures.
- Waited for the controller to halt before issuing a software reset and
  added an external reset fallback where available.
- Used private DMA bounce buffers for gadget requests and rejected
  pre-mapped requests. Retained the private DMA mappings when recovery
  could not confirm that the controller had stopped accessing them.
- Preallocated two transfer-ring banks per endpoint to avoid allocation
  in atomic context. Switched banks when rebuilding after a running
  session so late completions for the previous bank cannot match new
  requests.
- Fixed EP0 STALL, status-stage, and SETUP cancellation handling.
- Added controller recovery guards and generation checks to reject
  stale events and callbacks.
- Fixed port reset, disconnect, and reconnect handling. Separated
  controller recovery from gadget reset notification and deferred link
  restart until after the gadget reset callback.
- Improved synchronization between interrupts, resets, and endpoint
  operations. Fixed endpoint cleanup and resource release during
  teardown, and improved suspend and resume handling.
- Link to v4: https://patch.msgid.link/20261030-axiado-ax3000-usb-device-controller-v4-0-c5267b36a777@axiado.com

Changes in v4:

Ring and transfer management:
- Prevent stale event-ring dequeue updates after concurrent ring
  reinitialization by tracking ring generations.
- Continue scheduling pending requests while transfer-ring space is
  available.
- Avoid unsafe request-list iteration when completion callbacks drop
  udc_lock.
- Clear stale request bookkeeping before rebuilding cancelled transfer
  rings.
- Preserve transfer-ring cycle bits and Link TRBs when rewinding the
  ring.

Endpoint 0 handling:
- Prevent NULL dereferences and queue corruption when reusing the shared
  EP0 status request.
- Remove duplicate status-stage scheduling from GET_STATUS completion.
- Serialize shared status-request buffer access and completion
  callbacks.
- Correct SET_FEATURE and CLEAR_FEATURE validation, recipient handling,
  and function suspend options.
- Accept SET_ADDRESS(0) in the Default state and fix error handling.
- Correct SET_SEL payload parsing using struct usb_set_sel_req.
- Set the chain bit on non-final control data-stage TRBs.
- Stop EP0 and wait for hardware completion before cancelling requests
  and reinitializing its transfer ring.

Gadget and endpoint operations:
- Prevent TRB interleaving between requests, premature ZLP generation,
  and stalled transfers spanning multiple TDs.
- Fix transfer-ring allocation, resizing, and error cleanup to avoid
  dangling pointers and concurrent DMA buffer access.
- Synchronize reset work with gadget callback disabling and unbinding.
- Correct last_trb tracking across ring wraparound and multi-TD
  transfers.
- Use the correct dequeue cycle state when updating endpoint contexts.
- Wait for hardware endpoint disable and finalize endpoint state before
  completing pending requests.
- Correct High-Speed interrupt and isochronous endpoint maxburst
  handling.
- Fix port link-state resume handling and disconnect notifications.
- Revalidate endpoint state and EP0 setup identity after DMA mapping.
- Protect gadget start/stop state and register updates with udc_lock.

Core driver:
- Notify gadget drivers of disconnect before clearing endpoint state.
- Initialize endpoint structures, event rings, and EP0 before enabling
  IRQs.
- Move gadget removal ahead of controller state changes during teardown.
- Serialize shutdown register updates and cancel pending reset work.
- Fix 32-bit build compatibility when reconstructing 64-bit DMA
  addresses.
- Prevent overwriting pending controller commands while RUN is cleared.
- Replace SIMPLE_DEV_PM_OPS with DEFINE_SIMPLE_DEV_PM_OPS and
  pm_sleep_ptr().

- Link to v3: https://patch.msgid.link/20261020-axiado-ax3000-usb-device-controller-v3-0-3012ecfc0a93@axiado.com

Changes in v3:

Devicetree binding:
- Note that the underlying Corigine core is dual-role capable (device
  and host), used here in device role.
- Drop the explicit maximum-speed property declaration. Krzysztof noted
  in the v2 review that compatible already implies SuperSpeed
capability,
  and testing confirms the property is not required.
- Reference the generic usb.yaml schema and replace additionalProperties
  with unevaluatedProperties to permit standard USB properties while
  rejecting unsupported properties.
- Validated the updated binding with dt_binding_check.

Driver structure (matching drivers/usb/gadget/udc/cdns2):
- Split crg_udc.c/crg_udc.h into crg-udc-core.c, crg-udc-gadget.c,
  crg-udc-ep0.c, crg-udc-ring.c and crg-udc.h, one concern per file.
- Add ftrace tracepoint support (crg-udc-trace.h/.c) covering pullup,
  doorbell, TRB queueing, event processing, control requests, and
  request alloc/free/giveback, plus a decode-helper header
  (crg-udc-debug.h) for the PORTSC/completion-code/TRB-type strings
  used by the trace TP_printk()s.
- Make internal-only functions static and normalize the remaining
  exported ones to a consistent crg_udc_ prefix.
- Reorganize #include lists so each file pulls in only the headers it
  actually uses, instead of funneling every kernel header the driver
  needs through the shared crg-udc.h.
- Match the SPDX/license comment style used elsewhere in the tree:
  a line comment for the SPDX tag, block comment for the description
  and copyright.

Bug fixes:
- Restore the VBUS-detect vendor-register write on udc_start(),
  udc_stop(), remove() and shutdown(). Without it the controller never
  considers itself powered and enumeration is unreliable.
- Restore the DATA_STAGE_XFER/DATA_STAGE_RECV cases dropped from the
  EP0 status-stage completion handler. Without them, any control read
  (e.g. GET_DESCRIPTOR) never arms its STATUS stage and the host times
  out waiting for it.
- Fix the transfer-ring TRB-deferral check to defer building a TRB
  only when the ring is actually full, not merely because another
  request is already queued on the same endpoint. The stricter check
  silently broke back-to-back completions on one endpoint, such as
  mass storage's CSW immediately following its data stage.
- Restore the RUN-gated completion check in the hardware command
  helper, so a command issued after the controller has already been
  stopped (e.g. during udc_stop()) does not spin waiting on a
  completion the hardware will never post.
- Call usb_gadget_udc_reset() on a bus reset so the composite/function
  drivers correctly re-run set_alt() and re-enable their endpoints
  after a reset, instead of composite silently treating a repeated
  SET_CONFIGURATION as a no-op. This is deferred to a workqueue rather
  than called directly from the port-status interrupt path, since the
  synchronous call re-enters the function drivers and would otherwise
  require dropping the driver lock in the middle of event processing.
- Clear the generic usb_ep.enabled flag and endpoint descriptors during
  internal endpoint teardown and controller reset, preventing stale
  endpoint state across sessions and allowing endpoints to be correctly
  re-enabled after a new bind or reset.

Additional fixes from Sashiko review:
- Use dma_addr_t for DMA offset calculations and validate DMA address
  translation failures before programming hardware registers.
- Reserve transfer-ring capacity and publish multi-TRB descriptors
  only after all TRBs have been initialized.
- Track remaining TRBs and buffer length for requests spanning
  multiple transfer descriptors.
- Advance the hardware dequeue pointer before completing canceled
  requests, and restore the producer cycle state when rewinding rings.
- Detach pending requests before completing them during endpoint
  teardown, and complete requests that fail during deferred queueing.
- Reduce the hardware reset timeout from 500 ms to 10 ms and release
  the spinlock while polling for completion.
- Protect pullup register updates with the controller spinlock.
- Update the event-ring dequeue pointer before yielding the CPU.
- Unregister the interrupt handler before freeing DMA resources.

- Link to v2: https://patch.msgid.link/20260817-axiado-ax3000-usb-device-controller-v2-0-0ef033dd0a68@axiado.com

Changes in v2:
- Drop the dual (GPL-2.0+ OR MIT) license; the driver and header are now
  GPL-2.0+ only.
- Register via module_platform_driver() and add MODULE_AUTHOR/MODULE_ALIAS
  instead of the hand-written module_init/module_exit pair.
- Acquire clock, reset and PHY through the driver model
  (devm_clk_get_optional, devm_reset_control_get_optional_exclusive,
  devm_phy_optional_get), with teardown wired up via
  devm_add_action_or_reset.
- Switch to devm_request_irq() and drop the manual free_irq() paths.
- Support multiple instances using an IDA (DEFINE_IDA/ida_alloc/ida_free)
  instead of a file-static device counter.
- Read match data with device_get_match_data() and move the former
  compile-time feature switches to DT properties (dis-u1u2-entry-quirk,
  dis-u1u2-initiation-quirk, reinit-ep0-on-bus-reset).
- Set the DMA mask via dma_set_mask_and_coherent() with a 32-bit fallback,
  and convert PM to SIMPLE_DEV_PM_OPS.
- Fix endianness handling for TRBs, events and endpoint contexts using
  cpu_to_le32()/le32_to_cpu(), and decode setup packets without casting.
- Locking: pass the caller's IRQ flags through the completion paths so the
  interrupt state is correctly restored across the unlock/lock window around
  gadget callbacks.
- Properly free all coherent DMA allocations and add
  dma_mapping_error() checks after dma_map_single()
- Fix DMA directions on the GET_STATUS/SET_SEL control transfers.
- Replace open-coded register busy-wait loops with
  readl_poll_timeout_atomic() so they time out instead of spinning forever,
  and return proper errnos (-EBUSY/-ETIMEDOUT/-EIO) instead of -1.
- Replace the custom SETF_VAR/GETF/MAKEF_VAR shift-and-mask macros with
  FIELD_PREP/FIELD_GET (include linux/bitfield.h), and use BIT()/GENMASK()
  and dma_wmb()/dma_rmb().
- Use DMA_MAPPING_ERROR instead of a private DMA_ADDR_INVALID.
- Remove the U-Boot-derived VBUS cable-detect routine, the VBUS-detect
  kthread and its wait queue, dead defines and struct fields, the unused
  interrupt-register-array macros, the mdelay()-based reset delays and the
  bulk of the dev_dbg() tracing.
- Implement the pullup(is_on == 0) path and route suspend/shutdown through
  pullup(0); mark crg_udc_ep_ops const and return IRQ_NONE from the shared
  IRQ handler when the interrupt is not ours.
- Use dev_err_probe() with structured goto-based unwinding in probe.
- Reword the Kconfig help text for USB_CRG_UDC to name the crg_udc module.
- Link to v1: https://lore.kernel.org/r/20260202-axiado-ax3000-usb-device-controller-v1-0-45ce0a8b014f@axiado.com

To: Prasad Bolisetty <pbolisetty@axiado.com>
To: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: Rob Herring <robh@kernel.org>
To: Krzysztof Kozlowski <krzk+dt@kernel.org>
To: Conor Dooley <conor+dt@kernel.org>
To: Tzu-Hao Wei <twei@axiado.com>
To: Karthikeyan Mitran <kmitran@axiado.com>
To: Philipp Zabel <p.zabel@pengutronix.de>
To: Petar Stepanovic <pstepanovic@axiado.com>
To: Neill Kapron <nkapron@google.com>
Cc: linux-usb@vger.kernel.org
Cc: devicetree@vger.kernel.org
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org

---
Petar Stepanovic (6):
      dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC
      usb: gadget: udc: crg: Add hardware definitions and structs
      usb: gadget: udc: crg: Add ring and transfer management
      usb: gadget: udc: crg: Add endpoint 0 handling
      usb: gadget: udc: crg: Add gadget and endpoint operations
      usb: gadget: udc: crg: Add core driver and build integration

 .../devicetree/bindings/usb/axiado,ax3000-udc.yaml |   69 +
 MAINTAINERS                                        |    9 +
 drivers/usb/gadget/udc/Kconfig                     |    2 +
 drivers/usb/gadget/udc/Makefile                    |    1 +
 drivers/usb/gadget/udc/crg/Kconfig                 |   16 +
 drivers/usb/gadget/udc/crg/Makefile                |    9 +
 drivers/usb/gadget/udc/crg/crg-udc-core.c          | 1376 +++++++++
 drivers/usb/gadget/udc/crg/crg-udc-debug.h         |  143 +
 drivers/usb/gadget/udc/crg/crg-udc-ep0.c           | 1264 ++++++++
 drivers/usb/gadget/udc/crg/crg-udc-gadget.c        | 3255 ++++++++++++++++++++
 drivers/usb/gadget/udc/crg/crg-udc-ring.c          |  696 +++++
 drivers/usb/gadget/udc/crg/crg-udc-trace.c         |    9 +
 drivers/usb/gadget/udc/crg/crg-udc-trace.h         |  164 +
 drivers/usb/gadget/udc/crg/crg-udc.h               |  910 ++++++
 14 files changed, 7923 insertions(+)
---
base-commit: b3dfbde62cf1981c4638939c159dfd087b7a72e9
change-id: 20260128-axiado-ax3000-usb-device-controller-156d3f1840c9

Best regards,
--  
Petar Stepanovic <pstepanovic@axiado.com>


^ permalink raw reply	[flat|nested] 13+ messages in thread

* [PATCH v5 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC
  2026-11-16 19:06 [PATCH v5 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
@ 2026-11-16 19:06 ` Petar Stepanovic
  2026-11-16 19:12   ` sashiko-bot
  2026-11-16 19:06 ` [PATCH v5 2/6] usb: gadget: udc: crg: Add hardware definitions and structs Petar Stepanovic
                   ` (4 subsequent siblings)
  5 siblings, 1 reply; 13+ messages in thread
From: Petar Stepanovic @ 2026-11-16 19:06 UTC (permalink / raw)
  To: Prasad Bolisetty, Greg Kroah-Hartman, Rob Herring,
	Krzysztof Kozlowski, Conor Dooley, Tzu-Hao Wei,
	Karthikeyan Mitran, Philipp Zabel, Neill Kapron
  Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
	Petar Stepanovic, Krutik Shah, Naveen Kumar Rajgiri Bassappa,
	Krzysztof Kozlowski

Add a Devicetree binding for the USB device controller integrated into
the Axiado AX3000 and AX3005 SoCs.

The controller is based on a Corigine USB 3.1 IP core that is
dual-role capable (device and host), used here in device role. It
provides SuperSpeed (5 Gb/s) or High-Speed (480 Mb/s) operation, and
supports control, bulk, interrupt, and isochronous transfer types
across multiple configurable endpoints.

Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Tzu-Hao Wei <twei@axiado.com>
Signed-off-by: Tzu-Hao Wei <twei@axiado.com>
Reviewed-by: Krzysztof Kozlowski <krzysztof.kozlowski@oss.qualcomm.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
 .../devicetree/bindings/usb/axiado,ax3000-udc.yaml | 69 ++++++++++++++++++++++
 MAINTAINERS                                        |  8 +++
 2 files changed, 77 insertions(+)

diff --git a/Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml b/Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
new file mode 100644
index 000000000000..8b9acdb81907
--- /dev/null
+++ b/Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
@@ -0,0 +1,69 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/usb/axiado,ax3000-udc.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Axiado AX3000/AX3005 USB Device Controller
+
+maintainers:
+  - Petar Stepanovic <pstepanovic@axiado.com>
+  - Tzu-Hao Wei <twei@axiado.com>
+  - Prasad Bolisetty <pbolisetty@axiado.com>
+
+description:
+  Axiado USB dual-role controller based on the Corigine USB 3.1 IP core.
+  Supports USB 3.1 SuperSpeed and USB 2.0 High-Speed device operation
+  with control, bulk, interrupt, and isochronous transfers.
+
+allOf:
+  - $ref: usb.yaml#
+
+properties:
+  compatible:
+    oneOf:
+      - const: axiado,ax3000-udc
+      - items:
+          - const: axiado,ax3005-udc
+          - const: axiado,ax3000-udc
+
+  reg:
+    maxItems: 1
+
+  interrupts:
+    maxItems: 1
+
+  clocks:
+    maxItems: 1
+
+  resets:
+    maxItems: 1
+
+  phys:
+    maxItems: 1
+
+required:
+  - compatible
+  - reg
+  - interrupts
+
+unevaluatedProperties: false
+
+examples:
+  - |
+    #include <dt-bindings/interrupt-controller/arm-gic.h>
+    #include <dt-bindings/interrupt-controller/irq.h>
+
+    soc {
+        #address-cells = <2>;
+        #size-cells = <2>;
+
+        usb@81200000 {
+            compatible = "axiado,ax3000-udc";
+            reg = <0x0 0x81200000 0x0 0x100000>;
+            interrupt-parent = <&gic500>;
+            interrupts = <GIC_SPI 21 IRQ_TYPE_LEVEL_HIGH>;
+            resets = <&reset 0>;
+            phys = <&usb_phy>;
+        };
+    };
diff --git a/MAINTAINERS b/MAINTAINERS
index 222b7fc2c3aa..1bf0408e00f3 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4662,6 +4662,14 @@ F:	Documentation/devicetree/bindings/hwmon/axiado,ax3000-tsadc.yaml
 F:	Documentation/hwmon/axiado-tsadc.rst
 F:	drivers/hwmon/axiado-tsadc.c
 
+AXIADO USB UDC DRIVER
+M:	Petar Stepanovic <pstepanovic@axiado.com>
+M:	Tzu-Hao Wei <twei@axiado.com>
+M:	Prasad Bolisetty <pbolisetty@axiado.com>
+L:	linux-usb@vger.kernel.org
+S:	Supported
+F:	Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
+
 AYANEO PLATFORM EC DRIVER
 M:	Antheas Kapenekakis <lkml@antheas.dev>
 L:	platform-driver-x86@vger.kernel.org

-- 
2.34.1


^ permalink raw reply related	[flat|nested] 13+ messages in thread

* [PATCH v5 2/6] usb: gadget: udc: crg: Add hardware definitions and structs
  2026-11-16 19:06 [PATCH v5 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
  2026-11-16 19:06 ` [PATCH v5 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC Petar Stepanovic
@ 2026-11-16 19:06 ` Petar Stepanovic
  2026-11-16 19:24   ` sashiko-bot
  2026-11-16 19:06 ` [PATCH v5 3/6] usb: gadget: udc: crg: Add ring and transfer management Petar Stepanovic
                   ` (3 subsequent siblings)
  5 siblings, 1 reply; 13+ messages in thread
From: Petar Stepanovic @ 2026-11-16 19:06 UTC (permalink / raw)
  To: Prasad Bolisetty, Greg Kroah-Hartman, Rob Herring,
	Krzysztof Kozlowski, Conor Dooley, Tzu-Hao Wei,
	Karthikeyan Mitran, Philipp Zabel, Neill Kapron
  Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
	Petar Stepanovic, Krutik Shah, Naveen Kumar Rajgiri Bassappa

This patch introduces the hardware definitions, register offsets, and
fundamental data structures for the Axiado AX3000/AX3005 USB Device
Controller driver.

Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Tzu-Hao Wei <twei@axiado.com>
Signed-off-by: Tzu-Hao Wei <twei@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
 MAINTAINERS                                |   1 +
 drivers/usb/gadget/udc/crg/crg-udc-debug.h | 143 +++++
 drivers/usb/gadget/udc/crg/crg-udc-trace.h | 164 ++++++
 drivers/usb/gadget/udc/crg/crg-udc.h       | 910 +++++++++++++++++++++++++++++
 4 files changed, 1218 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 1bf0408e00f3..a3bbf6a752a2 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4669,6 +4669,7 @@ M:	Prasad Bolisetty <pbolisetty@axiado.com>
 L:	linux-usb@vger.kernel.org
 S:	Supported
 F:	Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
+F:	drivers/usb/gadget/udc/crg/
 
 AYANEO PLATFORM EC DRIVER
 M:	Antheas Kapenekakis <lkml@antheas.dev>
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-debug.h b/drivers/usb/gadget/udc/crg/crg-udc-debug.h
new file mode 100644
index 000000000000..f73c60a79ade
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-debug.h
@@ -0,0 +1,143 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ * Debug/trace decode helpers.
+ *
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+
+#ifndef __CRG_UDC_DEBUG_H
+#define __CRG_UDC_DEBUG_H
+
+#include "crg-udc.h"
+
+static inline const char *crg_udc_trb_type_string(u32 type)
+{
+	switch (type) {
+	case TRB_TYPE_XFER_NORMAL:
+		return "Normal";
+	case TRB_TYPE_XFER_DATA_STAGE:
+		return "Data Stage";
+	case TRB_TYPE_XFER_STATUS_STAGE:
+		return "Status Stage";
+	case TRB_TYPE_XFER_DATA_ISOCH:
+		return "Isoch";
+	case TRB_TYPE_LINK:
+		return "Link";
+	case TRB_TYPE_NO_OP:
+		return "No-Op";
+	case TRB_TYPE_EVT_TRANSFER:
+		return "Transfer Event";
+	case TRB_TYPE_EVT_CMD_COMPLETION:
+		return "Command Completion Event";
+	case TRB_TYPE_EVT_PORT_STATUS_CHANGE:
+		return "Port Status Change Event";
+	case TRB_TYPE_EVT_MFINDEX_WRAP:
+		return "MFINDEX Wrap Event";
+	case TRB_TYPE_EVT_SETUP_PKT:
+		return "Setup Packet Event";
+	default:
+		return "Reserved";
+	}
+}
+
+static inline const char *crg_udc_cmpl_code_string(u32 code)
+{
+	switch (code) {
+	case CMPL_CODE_INVALID:
+		return "Invalid";
+	case CMPL_CODE_SUCCESS:
+		return "Success";
+	case CMPL_CODE_DATA_BUFFER_ERR:
+		return "Data Buffer Error";
+	case CMPL_CODE_BABBLE_DETECTED_ERR:
+		return "Babble Detected";
+	case CMPL_CODE_USB_TRANS_ERR:
+		return "USB Transaction Error";
+	case CMPL_CODE_TRB_ERR:
+		return "TRB Error";
+	case CMPL_CODE_TRB_STALL:
+		return "Stall";
+	case CMPL_CODE_INVALID_STREAM_TYPE_ERR:
+		return "Invalid Stream Type";
+	case CMPL_CODE_SHORT_PKT:
+		return "Short Packet";
+	case CMPL_CODE_RING_UNDERRUN:
+		return "Ring Underrun";
+	case CMPL_CODE_RING_OVERRUN:
+		return "Ring Overrun";
+	case CMPL_CODE_EVENT_RING_FULL_ERR:
+		return "Event Ring Full";
+	case CMPL_CODE_STOPPED:
+		return "Stopped";
+	case CMPL_CODE_STOPPED_LENGTH_INVALID:
+		return "Stopped - Length Invalid";
+	case CMPL_CODE_ISOCH_BUFFER_OVERRUN:
+		return "Isoch Buffer Overrun";
+	case CMPL_CODE_PROTOCOL_STALL:
+		return "Protocol Stall";
+	case CMPL_CODE_SETUP_TAG_MISMATCH:
+		return "Setup Tag Mismatch";
+	case CMPL_CODE_HALTED:
+		return "Halted";
+	case CMPL_CODE_HALTED_LENGTH_INVALID:
+		return "Halted - Length Invalid";
+	case CMPL_CODE_DISABLED:
+		return "Disabled";
+	case CMPL_CODE_DISABLED_LENGTH_INVALID:
+		return "Disabled - Length Invalid";
+	default:
+		return "Unknown";
+	}
+}
+
+static inline const char *crg_udc_decode_portsc(char *str, size_t size,
+						u32 portsc)
+{
+	scnprintf(str, size, "0x%08x, PLS: %lu, Speed: %lu, %s%s%s%s",
+		 portsc, FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, portsc),
+		 FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, portsc),
+		 (portsc & CRG_U3DC_PORTSC_CCS) ? "CCS " : "",
+		 (portsc & CRG_U3DC_PORTSC_PED) ? "PED " : "",
+		 (portsc & CRG_U3DC_PORTSC_PR) ? "PR " : "",
+		 (portsc & CRG_U3DC_PORTSC_PP) ? "PP " : "");
+
+	return str;
+}
+
+static inline const char *crg_udc_decode_trb(char *str, size_t size,
+					     struct crg_transfer_trb *trb)
+{
+	u32 dw2 = le32_to_cpu(trb->dw2);
+	u32 dw3 = le32_to_cpu(trb->dw3);
+	u32 type = FIELD_GET(TRB_TYPE_MASK, dw3);
+
+	scnprintf(str, size,
+		  "type: %s, len: %lu, cycle: %lu, chain: %lu, ioc: %lu",
+		  crg_udc_trb_type_string(type),
+		  FIELD_GET(TRB_TRANSFER_LEN_MASK, dw2),
+		  dw3 & TRB_CYCLE_BIT_MASK ? 1UL : 0UL,
+		  dw3 & TRB_CHAIN_BIT_MASK ? 1UL : 0UL,
+		  dw3 & TRB_INTR_ON_COMPLETION_MASK ? 1UL : 0UL);
+
+	return str;
+}
+
+static inline const char *crg_udc_decode_event_trb(char *str, size_t size,
+						   struct crg_event_trb *trb)
+{
+	u32 dw2 = le32_to_cpu(trb->dw2);
+	u32 dw3 = le32_to_cpu(trb->dw3);
+	u32 type = FIELD_GET(EVE_TRB_TYPE_MASK, dw3);
+
+	scnprintf(str, size, "type: %s, cmpl: %s, len: %lu, dci: %lu",
+		  crg_udc_trb_type_string(type),
+		  crg_udc_cmpl_code_string(
+			  FIELD_GET(EVE_TRB_COMPL_CODE_MASK, dw2)),
+		  FIELD_GET(EVE_TRB_TRAN_LEN_MASK, dw2),
+		  FIELD_GET(EVE_TRB_ENDPOINT_ID_MASK, dw3));
+
+	return str;
+}
+
+#endif /* __CRG_UDC_DEBUG_H */
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-trace.h b/drivers/usb/gadget/udc/crg/crg-udc-trace.h
new file mode 100644
index 000000000000..672a719a041b
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-trace.h
@@ -0,0 +1,164 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ * Trace support header file.
+ *
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM crg-udc
+
+#undef TRACE_SYSTEM_VAR
+#define TRACE_SYSTEM_VAR crg_udc
+
+#if !defined(__CRG_UDC_TRACE_H) || defined(TRACE_HEADER_MULTI_READ)
+#define __CRG_UDC_TRACE_H
+
+#include <linux/types.h>
+#include <linux/tracepoint.h>
+#include <linux/usb/ch9.h>
+#include "crg-udc.h"
+#include "crg-udc-debug.h"
+
+#define CRG_MSG_MAX 200
+
+DECLARE_EVENT_CLASS(crg_udc_log_enable_disable, TP_PROTO(int set), TP_ARGS(set),
+		    TP_STRUCT__entry(__field(int, set)),
+		    TP_fast_assign(__entry->set = set;),
+		    TP_printk("%s", __entry->set ? "enabled" : "disabled"));
+
+DEFINE_EVENT(crg_udc_log_enable_disable, crg_udc_pullup, TP_PROTO(int set),
+	     TP_ARGS(set));
+
+TRACE_EVENT(crg_udc_no_room_on_ring, TP_PROTO(char *msg), TP_ARGS(msg),
+	    TP_STRUCT__entry(__string(text, msg)),
+	    TP_fast_assign(__assign_str(text);),
+	    TP_printk("%s", __get_str(text)));
+
+TRACE_EVENT(crg_udc_ep_halt,
+	    TP_PROTO(struct crg_udc_ep *udc_ep, u8 halt, u8 wedge),
+	    TP_ARGS(udc_ep, halt, wedge),
+	    TP_STRUCT__entry(__string(name, udc_ep->usb_ep.name)
+				     __field(u8, halt) __field(u8, wedge)),
+	    TP_fast_assign(__assign_str(name); __entry->halt = halt;
+			   __entry->wedge = wedge;),
+	    TP_printk("%s: %s%s", __get_str(name),
+		      __entry->halt ? "halted" : "cleared",
+		      __entry->wedge ? " (wedged)" : ""));
+
+TRACE_EVENT(crg_udc_portsc, TP_PROTO(u32 portsc), TP_ARGS(portsc),
+	    TP_STRUCT__entry(__field(u32, portsc)),
+	    TP_fast_assign(__entry->portsc = portsc;),
+	    TP_printk("%s",
+		      crg_udc_decode_portsc(__get_buf(CRG_MSG_MAX), CRG_MSG_MAX,
+					    __entry->portsc)));
+
+TRACE_EVENT(crg_udc_doorbell, TP_PROTO(struct crg_gadget_dev *crg_udc, int dci),
+	    TP_ARGS(crg_udc, dci),
+	    TP_STRUCT__entry(__string(name, crg_udc->udc_ep[dci].usb_ep.name)
+				     __field(int, dci)),
+	    TP_fast_assign(__assign_str(name); __entry->dci = dci;),
+	    TP_printk("%s, dci %d", __get_str(name), __entry->dci));
+
+DECLARE_EVENT_CLASS(
+	crg_udc_log_ctrl, TP_PROTO(struct usb_ctrlrequest *ctrl), TP_ARGS(ctrl),
+	TP_STRUCT__entry(__field(u8, bRequestType) __field(u8, bRequest)
+				 __field(u16, wValue) __field(u16, wIndex)
+					 __field(u16, wLength)),
+	TP_fast_assign(__entry->bRequestType = ctrl->bRequestType;
+		       __entry->bRequest = ctrl->bRequest;
+		       __entry->wValue = le16_to_cpu(ctrl->wValue);
+		       __entry->wIndex = le16_to_cpu(ctrl->wIndex);
+		       __entry->wLength = le16_to_cpu(ctrl->wLength);),
+	TP_printk("%s", usb_decode_ctrl(__get_buf(CRG_MSG_MAX), CRG_MSG_MAX,
+					__entry->bRequestType,
+					__entry->bRequest, __entry->wValue,
+					__entry->wIndex, __entry->wLength)));
+
+DEFINE_EVENT(crg_udc_log_ctrl, crg_udc_ctrl_req,
+	     TP_PROTO(struct usb_ctrlrequest *ctrl), TP_ARGS(ctrl));
+
+DECLARE_EVENT_CLASS(
+	crg_udc_log_trb,
+	TP_PROTO(struct crg_udc_ep *udc_ep, struct crg_transfer_trb *trb),
+	TP_ARGS(udc_ep, trb),
+	TP_STRUCT__entry(__string(name, udc_ep->usb_ep.name)
+				 __field(struct crg_transfer_trb *, trb)
+					 __dynamic_array(char, decoded,
+							 CRG_MSG_MAX)),
+	TP_fast_assign(__assign_str(name); __entry->trb = trb;
+		       crg_udc_decode_trb(__get_dynamic_array(decoded),
+					  CRG_MSG_MAX, trb);),
+	TP_printk("%s: trb %p, %s", __get_str(name), __entry->trb,
+		  __get_str(decoded)));
+
+DEFINE_EVENT(crg_udc_log_trb, crg_udc_queue_trb,
+	     TP_PROTO(struct crg_udc_ep *udc_ep, struct crg_transfer_trb *trb),
+	     TP_ARGS(udc_ep, trb));
+
+TRACE_EVENT(
+	crg_udc_event_trb, TP_PROTO(struct crg_event_trb *trb), TP_ARGS(trb),
+	TP_STRUCT__entry(__field(struct crg_event_trb *, trb)
+				 __dynamic_array(char, decoded, CRG_MSG_MAX)),
+	TP_fast_assign(__entry->trb = trb;
+		       crg_udc_decode_event_trb(__get_dynamic_array(decoded),
+						CRG_MSG_MAX, trb);),
+	TP_printk("trb %p, %s", __entry->trb, __get_str(decoded)));
+
+DECLARE_EVENT_CLASS(
+	crg_udc_log_request,
+	TP_PROTO(struct crg_udc_ep *udc_ep, struct crg_udc_request *udc_req),
+	TP_ARGS(udc_ep, udc_req),
+	TP_STRUCT__entry(
+		__string(name, udc_ep->usb_ep.name) __field(
+			struct usb_request *, request) __field(void *, buf)
+			__field(unsigned int, actual) __field(unsigned int,
+							      length)
+				__field(int, status) __field(dma_addr_t, dma)
+					__field(int, zero) __field(int,
+								   short_not_ok)
+						__field(int, no_interrupt)),
+	TP_fast_assign(__assign_str(name); __entry->request = &udc_req->usb_req;
+		       __entry->buf = udc_req->usb_req.buf;
+		       __entry->actual = udc_req->usb_req.actual;
+		       __entry->length = udc_req->usb_req.length;
+		       __entry->status = udc_req->usb_req.status;
+		       __entry->dma = udc_req->usb_req.dma;
+		       __entry->zero = udc_req->usb_req.zero;
+		       __entry->short_not_ok = udc_req->usb_req.short_not_ok;
+		       __entry->no_interrupt = udc_req->usb_req.no_interrupt;),
+	TP_printk(
+		"%s: req: %p, buf: %p, length: %u/%u, status: %d, dma: %pad, %s%s%s",
+		__get_str(name), __entry->request, __entry->buf,
+		__entry->actual, __entry->length, __entry->status,
+		&__entry->dma, __entry->zero ? "Z" : "z",
+		__entry->short_not_ok ? "S" : "s",
+		__entry->no_interrupt ? "I" : "i"));
+
+DEFINE_EVENT(crg_udc_log_request, crg_udc_alloc_request,
+	     TP_PROTO(struct crg_udc_ep *udc_ep,
+		      struct crg_udc_request *udc_req),
+	     TP_ARGS(udc_ep, udc_req));
+
+DEFINE_EVENT(crg_udc_log_request, crg_udc_free_request,
+	     TP_PROTO(struct crg_udc_ep *udc_ep,
+		      struct crg_udc_request *udc_req),
+	     TP_ARGS(udc_ep, udc_req));
+
+DEFINE_EVENT(crg_udc_log_request, crg_udc_request_giveback,
+	     TP_PROTO(struct crg_udc_ep *udc_ep,
+		      struct crg_udc_request *udc_req),
+	     TP_ARGS(udc_ep, udc_req));
+
+#endif /* __CRG_UDC_TRACE_H */
+
+/* This part must be outside header guard. */
+
+#undef TRACE_INCLUDE_PATH
+#define TRACE_INCLUDE_PATH .
+
+#undef TRACE_INCLUDE_FILE
+#define TRACE_INCLUDE_FILE crg-udc-trace
+
+#include <trace/define_trace.h>
diff --git a/drivers/usb/gadget/udc/crg/crg-udc.h b/drivers/usb/gadget/udc/crg/crg-udc.h
new file mode 100644
index 000000000000..afc81a303f07
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc.h
@@ -0,0 +1,910 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+
+#ifndef CRG_UDC_H
+#define CRG_UDC_H
+
+#include <linux/bitfield.h>
+#include <linux/bits.h>
+#include <linux/dma-mapping.h>
+#include <linux/io.h>
+#include <linux/kernel.h>
+#include <linux/scatterlist.h>
+#include <linux/types.h>
+#include <linux/wait.h>
+#include <linux/workqueue.h>
+
+#include <linux/usb/ch9.h>
+#include <linux/usb/gadget.h>
+
+#define CRG_UCCR_OFFSET 0x2400
+#define CRG_UICR_OFFSET 0x2500
+#define CRG_UICR_STRIDE 0x20
+
+/* Corigine USB 3.1 device core register layout. */
+struct crg_uccr {
+	u32 capability;
+	u32 resv0[3];
+
+	u32 config0;
+	u32 config1;
+	u32 resv1[2];
+
+	u32 control;
+	u32 status;
+	u32 dcbaplo;
+	u32 dcbaphi;
+	u32 portsc;
+	u32 u3portpmsc;
+	u32 u2portpmsc;
+	u32 u3portli;
+
+	u32 doorbell;
+	u32 mfindex;
+	u32 ptmctrl;
+	u32 resv3[5];
+
+	u32 ep_enable;
+	u32 ep_running;
+	u32 resv4[2];
+
+	u32 cmd_param0;
+	u32 cmd_param1;
+	u32 cmd_control;
+	u32 resv5[1];
+
+	u32 odb_capability;
+	u32 resv6[3];
+	u32 odb_config[8];
+
+	u32 debug0;
+};
+
+struct crg_uicr {
+	u32 iman;
+	u32 imod;
+	u32 erstsz;
+	u32 resv0;
+
+	u32 erstbalo;
+	u32 erstbahi;
+	u32 erdplo;
+	u32 erdphi;
+};
+
+/* config0/config1 registers */
+#define CRG_U3DC_CFG0_MAXSPEED_MASK GENMASK(3, 0)
+#define CRG_U3DC_CFG0_MAXSPEED_FS 1
+#define CRG_U3DC_CFG0_MAXSPEED_HS 3
+#define CRG_U3DC_CFG0_MAXSPEED_SS 4
+#define CRG_U3DC_CFG0_MAXSPEED_SSP 5
+
+#define CRG_U3DC_CFG1_CSC_EVENT_EN BIT(0)
+#define CRG_U3DC_CFG1_PEC_EVENT_EN BIT(1)
+#define CRG_U3DC_CFG1_PPC_EVENT_EN BIT(3)
+#define CRG_U3DC_CFG1_PRC_EVENT_EN BIT(4)
+#define CRG_U3DC_CFG1_PLC_EVENT_EN BIT(5)
+#define CRG_U3DC_CFG1_CEC_EVENT_EN BIT(6)
+
+#define CRG_U3DC_CFG1_U3_ENTRY_EN BIT(8)
+#define CRG_U3DC_CFG1_L1_ENTRY_EN BIT(9)
+#define CRG_U3DC_CFG1_U3_RESUME_EN BIT(10)
+#define CRG_U3DC_CFG1_L1_RESUME_EN BIT(11)
+#define CRG_U3DC_CFG1_INACTIVE_PLC_EN BIT(12)
+#define CRG_U3DC_CFG1_U3_RESUME_NORESP_PLC_EN BIT(13)
+#define CRG_U3DC_CFG1_U2_RESUME_NORESP_PLC_EN BIT(14)
+
+#define CRG_U3DC_CFG1_SETUP_EVENT_EN BIT(16)
+
+/* control register */
+#define CRG_U3DC_CTRL_RUN BIT(0)
+#define CRG_U3DC_CTRL_STOP 0
+#define CRG_U3DC_CTRL_SWRST BIT(1)
+#define CRG_U3DC_CTRL_INT_EN BIT(2)
+#define CRG_U3DC_CTRL_SYSERR_EN BIT(3)
+#define CRG_U3DC_CTRL_EWE BIT(10)
+#define CRG_U3DC_CTRL_KP_CNCT BIT(11)
+
+/* status register */
+#define CRG_U3DC_STATUS_DEV_CTRL_HALT BIT(0)
+#define CRG_U3DC_STATUS_SYS_ERR BIT(2)
+#define CRG_U3DC_STATUS_EINT BIT(3)
+
+/* portsc register */
+#define CRG_U3DC_PORTSC_CCS BIT(0)
+#define CRG_U3DC_PORTSC_PED BIT(1)
+#define CRG_U3DC_PORTSC_PP BIT(3)
+#define CRG_U3DC_PORTSC_PR BIT(4)
+#define CRG_U3DC_PORTSC_PLS_MASK GENMASK(8, 5)
+#define CRG_U3DC_PORTSC_PLS_U3 3
+
+#define CRG_U3DC_PORTSC_SPEED_MASK GENMASK(13, 10)
+#define CRG_U3DC_PORTSC_SPEED_FS 0x1
+#define CRG_U3DC_PORTSC_SPEED_LS 0x2
+#define CRG_U3DC_PORTSC_SPEED_HS 0x3
+#define CRG_U3DC_PORTSC_SPEED_SS 0x4
+#define CRG_U3DC_PORTSC_SPEED_SSP 0x5
+
+#define CRG_U3DC_PORTSC_LWS BIT(16)
+#define CRG_U3DC_PORTSC_CSC BIT(17)
+#define CRG_U3DC_PORTSC_PEC BIT(18)
+#define CRG_U3DC_PORTSC_PPC BIT(20)
+#define CRG_U3DC_PORTSC_PRC BIT(21)
+#define CRG_U3DC_PORTSC_PLC BIT(22)
+#define CRG_U3DC_PORTSC_CEC BIT(23)
+#define CRG_U3DC_PORTSC_WCE BIT(25)
+#define CRG_U3DC_PORTSC_WDE BIT(26)
+#define CRG_U3DC_PORTSC_WPR BIT(31)
+
+#define CRG_U3DC_PORTSC_W1C_MASK                                           \
+	(CRG_U3DC_PORTSC_CSC | CRG_U3DC_PORTSC_PEC | CRG_U3DC_PORTSC_PPC | \
+	 CRG_U3DC_PORTSC_PRC | CRG_U3DC_PORTSC_PLC | CRG_U3DC_PORTSC_CEC)
+
+/* mfindex register */
+#define CRG_U3DC_MFINDEX_CURRENT_MASK GENMASK(17, 4)
+#define CRG_U3DC_MFINDEX_FRAME_MASK GENMASK(17, 7)
+
+/* ptmctrl register */
+#define CRG_U3DC_PTMCTRL_ISOCH_DELAY_MASK GENMASK(13, 0)
+
+/* u3portpmsc */
+#define CRG_U3DC_U3PORTPM_U1TMOUT_MASK GENMASK(7, 0)
+#define CRG_U3DC_U3PORTPM_U2TMOUT_MASK GENMASK(15, 8)
+#define CRG_U3DC_U3PORTPM_FLA BIT(16)
+#define CRG_U3DC_U3PORTPM_U1IEN BIT(20)
+#define CRG_U3DC_U3PORTPM_U2IEN BIT(21)
+#define CRG_U3DC_U3PORTPM_U1AEN BIT(22)
+#define CRG_U3DC_U3PORTPM_U2AEN BIT(23)
+
+#define CRG_U3DC_U3PORTPM_U1U2TMOUT_MASK GENMASK(31, 24)
+
+/* u2portpmsc */
+#define CRG_U3DC_U2PORTPM_RJ_TH_MASK GENMASK(3, 0)
+#define CRG_U3DC_U2PORTPM_DS_TH_MASK GENMASK(7, 4)
+#define CRG_U3DC_U2PORTPM_LPM_EN BIT(8)
+#define CRG_U3DC_U2PORTPM_RJ_TH_EN BIT(9)
+#define CRG_U3DC_U2PORTPM_DS_EN BIT(10)
+#define CRG_U3DC_U2PORTPM_SLP_EN BIT(11)
+#define CRG_U3DC_U2PORTPM_LPM_FACK BIT(12)
+#define CRG_U3DC_U2PORTPM_L1_AEX BIT(13)
+#define CRG_U3DC_U2PORTPM_H_B_MASK GENMASK(19, 16)
+#define CRG_U3DC_U2PORTPM_RWE BIT(20)
+#define CRG_U3DC_U2PORTPM_TM_MASK GENMASK(31, 28)
+
+/* doorbell register */
+#define CRG_U3DC_DB_TARGET_MASK GENMASK(4, 0)
+
+/* odb registers */
+#define CRG_U3DC_ODBCFG_2N_OFFSET_MASK GENMASK(9, 0)
+#define CRG_U3DC_ODBCFG_2N_SIZE_MASK GENMASK(12, 10)
+#define CRG_U3DC_ODBCFG_2N1_OFFSET_MASK GENMASK(25, 16)
+#define CRG_U3DC_ODBCFG_2N1_SIZE_MASK GENMASK(28, 26)
+
+/* command control register */
+#define CRG_U3DC_CMD_CTRL_ACTIVE BIT(0)
+#define CRG_U3DC_CMD_CTRL_IOC_EN BIT(1)
+#define CRG_U3DC_CMD_CTRL_TYPE_MASK GENMASK(7, 4)
+#define CRG_U3DC_CMD_CTRL_STATUS_MASK GENMASK(19, 16)
+
+/* interrupt register / IMAN bits */
+#define CRG_U3DC_IMAN_INT_PEND BIT(0)
+#define CRG_U3DC_IMAN_INT_EN BIT(1)
+
+/* erdp bits */
+#define CRG_U3DC_ERDPLO_EHB BIT(3)
+
+/* command parameters: init EP0 */
+#define CRG_CMD0_0_DQPTRLO_MASK GENMASK(31, 4)
+#define CRG_CMD0_0_DCS_MASK BIT(0)
+
+/* command parameters: update EP0 */
+#define CRG_CMD1_0_MPS_MASK GENMASK(31, 16)
+
+/* command parameters: set address */
+#define CRG_CMD2_0_DEV_ADDR_MASK GENMASK(7, 0)
+
+#define CRG_UDC_MODE_REG 0x20fc
+#define CRG_UDC_ROLE_DEVICE 0x1
+
+#define CRG_UDC_VENDOR_REG 0x210c
+#define CRG_UDC_VBUS_DETECT BIT(26)
+
+/*
+ * Device-visible data structures. The controller reads and writes these
+ * over DMA, so the layouts and field encodings below are fixed by the
+ * hardware in the same way as the register definitions above.
+ */
+
+/* Transfer and event TRB types */
+enum crg_trb_type {
+	TRB_TYPE_RSVD = 0,
+	TRB_TYPE_XFER_NORMAL,
+	TRB_TYPE_RSVD2,
+	TRB_TYPE_XFER_DATA_STAGE,
+	TRB_TYPE_XFER_STATUS_STAGE,
+	TRB_TYPE_XFER_DATA_ISOCH,
+	TRB_TYPE_LINK,
+	TRB_TYPE_RSVD7,
+	TRB_TYPE_NO_OP,
+
+	TRB_TYPE_EVT_TRANSFER = 32,
+	TRB_TYPE_EVT_CMD_COMPLETION = 33,
+	TRB_TYPE_EVT_PORT_STATUS_CHANGE = 34,
+	TRB_TYPE_EVT_MFINDEX_WRAP = 39,
+	TRB_TYPE_EVT_SETUP_PKT = 40,
+};
+
+/* Event TRB completion codes */
+enum crg_trb_cmpl_code {
+	CMPL_CODE_INVALID = 0,
+	CMPL_CODE_SUCCESS,
+	CMPL_CODE_DATA_BUFFER_ERR,
+	CMPL_CODE_BABBLE_DETECTED_ERR,
+	CMPL_CODE_USB_TRANS_ERR,
+	CMPL_CODE_TRB_ERR,
+	CMPL_CODE_TRB_STALL,
+	CMPL_CODE_INVALID_STREAM_TYPE_ERR = 10,
+	CMPL_CODE_SHORT_PKT = 13,
+	CMPL_CODE_RING_UNDERRUN,
+	CMPL_CODE_RING_OVERRUN,
+	CMPL_CODE_EVENT_RING_FULL_ERR = 21,
+	CMPL_CODE_STOPPED = 26,
+	CMPL_CODE_STOPPED_LENGTH_INVALID = 27,
+	CMPL_CODE_ISOCH_BUFFER_OVERRUN = 31,
+
+	CMPL_CODE_PROTOCOL_STALL = 192,
+	CMPL_CODE_SETUP_TAG_MISMATCH = 193,
+	CMPL_CODE_HALTED = 194,
+	CMPL_CODE_HALTED_LENGTH_INVALID = 195,
+	CMPL_CODE_DISABLED = 196,
+	CMPL_CODE_DISABLED_LENGTH_INVALID = 197,
+};
+
+/* Endpoint type encoding in the endpoint context */
+enum crg_ep_type {
+	EP_TYPE_INVALID = 0,
+	EP_TYPE_ISOCH_OUTBOUND,
+	EP_TYPE_BULK_OUTBOUND,
+	EP_TYPE_INTR_OUTBOUND,
+	EP_TYPE_INVALID2,
+	EP_TYPE_ISOCH_INBOUND,
+	EP_TYPE_BULK_INBOUND,
+	EP_TYPE_INTR_INBOUND,
+};
+
+/* crg_transfer_trb dw2 fields */
+#define TRB_TRANSFER_LEN_MASK GENMASK(16, 0)
+#define TRB_TD_SIZE_MASK GENMASK(21, 17)
+#define TRB_INTR_TARGET_MASK GENMASK(31, 22)
+
+/* crg_transfer_trb dw3 fields */
+#define TRB_CYCLE_BIT_MASK BIT(0)
+#define TRB_LINK_TOGGLE_CYCLE_MASK BIT(1)
+#define TRB_INTR_ON_SHORT_PKT_MASK BIT(2)
+#define TRB_NO_SNOOP_MASK BIT(3)
+#define TRB_CHAIN_BIT_MASK BIT(4)
+#define TRB_INTR_ON_COMPLETION_MASK BIT(5)
+#define TRB_APPEND_ZLP_MASK BIT(7)
+#define TRB_BLOCK_EVENT_INT_MASK BIT(9)
+#define TRB_TYPE_MASK GENMASK(15, 10)
+#define DATA_STAGE_TRB_DIR_MASK BIT(16)
+#define TRB_SETUP_TAG_MASK GENMASK(18, 17)
+#define STATUS_STAGE_TRB_STALL_MASK BIT(19)
+#define STATUS_STAGE_TRB_SET_ADDR_MASK BIT(20)
+#define ISOC_TRB_FRAME_ID_MASK GENMASK(30, 20)
+#define ISOC_TRB_SIA_MASK BIT(31)
+
+struct crg_transfer_trb {
+	__le32 dw0;
+	__le32 dw1;
+	__le32 dw2;
+	__le32 dw3;
+};
+
+/* crg_event_trb dw2 fields */
+#define EVE_TRB_TRAN_LEN_MASK GENMASK(16, 0)
+#define EVE_TRB_COMPL_CODE_MASK GENMASK(31, 24)
+
+/* crg_event_trb dw3 fields */
+#define EVE_TRB_CYCLE_BIT_MASK BIT(0)
+#define EVE_TRB_TYPE_MASK GENMASK(15, 10)
+#define EVE_TRB_ENDPOINT_ID_MASK GENMASK(20, 16)
+#define EVE_TRB_SETUP_TAG_MASK GENMASK(22, 21)
+
+struct crg_event_trb {
+	__le32 dw0;
+	__le32 dw1;
+	__le32 dw2;
+	__le32 dw3;
+};
+
+/* crg_ep_cx dw0 fields */
+#define EP_CX_LOGICAL_EP_NUM_MASK GENMASK(6, 3)
+#define EP_CX_INTERVAL_MASK GENMASK(23, 16)
+
+/* crg_ep_cx dw1 fields */
+#define EP_CX_EP_TYPE_MASK GENMASK(5, 3)
+#define EP_CX_MAX_BURST_SIZE_MASK GENMASK(15, 8)
+#define EP_CX_MAX_PACKET_SIZE_MASK GENMASK(31, 16)
+
+/* crg_ep_cx dw2 fields */
+#define EP_CX_DEQ_CYC_STATE_MASK BIT(0)
+#define EP_CX_TR_DQPT_LO_MASK GENMASK(31, 4)
+
+struct crg_ep_cx {
+	__le32 dw0;
+	__le32 dw1;
+	__le32 dw2;
+	__le32 dw3;
+};
+
+struct crg_erst {
+	/* 64-bit event ring segment address */
+	__le32 seg_addr_lo;
+	__le32 seg_addr_hi;
+	__le32 seg_size;
+	/* Set to zero */
+	__le32 rsvd;
+};
+
+/* command type */
+enum crg_cmd_type {
+	CRG_CMD_INIT_EP0 = 0,
+	CRG_CMD_UPDATE_EP0_CFG = 1,
+	CRG_CMD_SET_ADDR = 2,
+	CRG_CMD_SEND_DEV_NOTIFICATION = 3,
+	CRG_CMD_CONFIG_EP = 4,
+	CRG_CMD_SET_HALT = 5,
+	CRG_CMD_CLEAR_HALT = 6,
+	CRG_CMD_RESET_SEQNUM = 7,
+	CRG_CMD_STOP_EP = 8,
+	CRG_CMD_SET_TR_DQPTR = 9,
+	CRG_CMD_FORCE_FLOW_CONTROL = 10,
+	CRG_CMD_REQ_LDM_EXCHANGE = 11,
+};
+
+#define CRG_MAX_PACKET_SIZE 1024
+
+#define CRG_ERST_SIZE 1
+#define CRG_EVENT_RING_SIZE 256
+#define CRG_NUM_EP_CX 32
+
+/*
+ * Cap on events drained per udc_lock acquisition in crg_udc_process_event_ring().
+ * Command polling and crg_udc_reset()'s hardware-reset poll both run from
+ * crg_udc_handle_event() with udc_lock held, so without a cap a busy event
+ * ring could keep the lock, and local IRQs, held for as long as events
+ * keep arriving.
+ */
+#define CRG_EVENT_RING_BATCH 32
+
+#define CRG_UDC_TRB_MAX_BUFFER_SIZE 65536
+#define CRG_UDC_CONTROL_EP_TD_RING_SIZE 16
+#define CRG_UDC_BULK_EP_TD_RING_SIZE 1024
+#define CRG_UDC_ISOC_EP_TD_RING_SIZE 32
+#define CRG_UDC_INT_EP_TD_RING_SIZE 8
+
+#define CRG_UDC_U1_TIMEOUT_VAL 0x70
+#define CRG_UDC_U2_TIMEOUT_VAL 0x70
+
+enum crg_ep_state {
+	EP_STATE_DISABLED = 0,
+	EP_STATE_RUNNING = 1,
+	EP_STATE_HALTED = 2,
+	EP_STATE_STOPPED = 3
+};
+
+/* control-transfer setup state machine */
+enum crg_setup_status {
+	WAIT_FOR_SETUP = 0,
+	SETUP_PKT_PROCESS_IN_PROGRESS,
+	DATA_STAGE_XFER,
+	DATA_STAGE_RECV,
+	STATUS_STAGE_XFER,
+	STATUS_STAGE_RECV,
+};
+
+/* interrupt-on-completion moderation rate (in TRBs) per endpoint type */
+enum {
+	CRG_UDC_BULK_EP_INTERRUPT_RATE = 5,
+	CRG_UDC_ISOC_EP_INTERRUPT_RATE = 1,
+};
+
+struct crg_buffer_info {
+	void *cpu_addr;
+	dma_addr_t dma_addr;
+	u32 len;
+};
+
+struct crg_sel_value {
+	u16 u2_pel_value;
+	u16 u2_sel_value;
+	u8 u1_pel_value;
+	u8 u1_sel_value;
+};
+
+/* May outlive the request if hardware still owns its DMA addresses. */
+struct crg_udc_bounce {
+	struct list_head quarantine;
+	struct list_head pages;
+	struct sg_table sgt;
+	void *buf;
+	dma_addr_t dma;
+	size_t len;
+	enum dma_data_direction dir;
+	bool mapped;
+};
+
+struct crg_udc_request {
+	struct usb_request usb_req;
+	struct list_head queue;
+	struct crg_udc_bounce *bounce;
+	struct scatterlist *sg;
+	unsigned int sg_index;
+	u32 sg_offset;
+	u64 buff_len_left;
+	u32 trbs_needed;
+	struct crg_transfer_trb *first_trb;
+	struct crg_transfer_trb *last_trb;
+	/* Bytes queued in TDs that completed before the current TD. */
+	u32 td_start_offset;
+	bool all_trbs_queued;
+	/* A TD has been published and its completion event is outstanding. */
+	bool td_pending;
+};
+
+struct crg_udc_ep {
+	struct usb_ep usb_ep;
+
+	struct crg_buffer_info tran_ring_alloc;
+	struct crg_buffer_info tran_ring_info;
+	struct crg_transfer_trb *first_trb;
+	struct crg_transfer_trb *last_trb;
+
+	struct crg_transfer_trb *enq_pt;
+	struct crg_transfer_trb *deq_pt;
+	u8 pcs;
+
+	char name[10];
+	u8 dci;
+	struct list_head queue;
+	const struct usb_endpoint_descriptor *desc;
+	const struct usb_ss_ep_comp_descriptor *comp_desc;
+	bool tran_ring_full;
+	bool configuring;
+	/* HALT/STOP did not confirm DMA stopped; keep requests mapped. */
+	bool halt_failed;
+	/* A failed disable must not be recovered by SET_HALT/CLEAR_HALT. */
+	bool disable_failed;
+	/* STOP_EP completed in ep_dequeue(); giveback callbacks may queue,
+	 * but must not publish TRBs until the original dequeue resumes.
+	 */
+	bool dequeue_paused;
+	/* Keep giveback callbacks from repopulating a queue being drained. */
+	bool nuking;
+	bool clear_enabled_on_nuke;
+	u32 generation;
+	struct crg_gadget_dev *crg_udc;
+
+	enum crg_ep_state ep_state;
+
+	bool wedge;
+};
+
+#define CRG_RING_NUM 1
+
+struct crg_udc_event {
+	struct crg_buffer_info erst;
+	struct crg_erst *p_erst;
+	struct crg_buffer_info event_ring;
+	struct crg_event_trb *evt_dq_pt;
+	/* Changed on every ring reinitialization under udc_lock. */
+	u32 generation;
+	u8 ccs;
+	struct crg_event_trb *evt_seg0_last_trb;
+};
+
+enum crg_ep0_state {
+	EP0_STATE_IDLE = 0,
+	EP0_STATE_ABORTING,
+	EP0_STATE_ERROR,
+};
+
+struct crg_gadget_dev {
+	void __iomem *mmio_virt_base;
+	struct crg_uccr __iomem *uccr;
+	struct crg_uicr __iomem *uicr[CRG_RING_NUM];
+
+	/* udc_lock device lock */
+	spinlock_t udc_lock;
+
+	struct device *dev;
+	struct clk *clk;
+	struct phy *phy;
+	struct reset_control *rst;
+	bool hw_reset_asserted;
+	bool dma_quiesce_failed;
+	bool external_reset_pending;
+	/* Endpoint command failure needs SWRST and giveback in process context. */
+	bool endpoint_recovery_pending;
+	/* Set RUN after composite drops the pre-reset configuration. */
+	bool restart_after_reset;
+	/* Set when DMA addresses are programmed; cleared by a confirmed reset. */
+	bool hw_dma_armed;
+	bool transfer_ring_bank;
+	bool transfer_rings_live;
+	/* A failed reset/reconstruction prevents all new controller activity. */
+	bool controller_faulted;
+	bool gadget_stopping;
+	wait_queue_head_t teardown_wait;
+	unsigned int queueing;
+	struct list_head dma_quarantine;
+
+	struct usb_gadget gadget;
+	struct usb_gadget_driver *gadget_driver;
+	/* Speed selected for the currently bound gadget driver. */
+	enum usb_device_speed max_speed;
+
+	int irq;
+
+	struct usb_endpoint_descriptor ep0_desc;
+
+	struct crg_udc_ep udc_ep[CRG_NUM_EP_CX];
+	struct crg_buffer_info ep_cx;
+	struct crg_ep_cx *p_epcx;
+
+	struct crg_udc_event udc_event[CRG_RING_NUM];
+
+	struct crg_udc_request *status_req;
+	/* Protected by udc_lock, remains set during a data-stage giveback. */
+	bool status_req_callback_pending;
+	struct crg_sel_value sel_value;
+	void (*setup_fn_call_back)(struct crg_gadget_dev *a);
+
+	enum crg_setup_status setup_status;
+	/* EP0 rejects new requests while the previous control transfer is
+	 * being aborted and its completion callbacks are being given back.
+	 */
+	enum crg_ep0_state ep0_state;
+
+	u8 device_state;
+	u16 dev_addr;
+	u8 setup_tag;
+	u8 set_tm;
+	/* Latest SETUP event received before post-reset EP0 initialization. */
+	struct usb_ctrlrequest pending_setup;
+	u8 pending_setup_tag;
+	bool pending_setup_valid;
+
+	u32 num_enabled_eps;
+
+	bool connected;
+	bool pullup_on;
+	bool async_callbacks;
+	/*
+	 * Set while crg_udc_reset() has udc_lock dropped and is polling for
+	 * SWRST to clear. Checked by crg_udc_ep_queue() so a request queued
+	 * on another endpoint during that window is rejected instead of
+	 * being built into a ring and doorbell-rung while the controller is
+	 * still mid-reset.
+	 */
+	bool resetting;
+	/* Device lifecycle gates held across IRQ and reset-work callbacks. */
+	bool suspended;
+	bool shutting_down;
+	/* RUN was stopped while an endpoint still had DMA-visible TRBs. */
+	bool stopped_needs_reset;
+	/* Covers SWRST, callback giveback, and context/EP0 reconstruction. */
+	bool controller_recovering;
+	/* Quiesced hardware; composite still needs to disable its endpoints. */
+	bool bus_reset_pending;
+	/* Invalidates an older reset worker after a newer reset/disconnect. */
+	u32 bus_reset_generation;
+	/* Serialize the gadget reset callback against new port recovery. */
+	bool reset_callback_active;
+	/* External gadget callbacks may drop udc_lock and invoke ep_disable(). */
+	unsigned int gadget_callbacks_active;
+	/* Keep core endpoint-enabled state until composite reset runs. */
+	bool preserve_ep_enabled_on_reset;
+	/* System errors are recovered after the gadget reset callback. */
+	bool deferred_system_error;
+	/* W1C port events acknowledged while that callback was running. */
+	u32 deferred_port_events;
+	/* PRC was acknowledged before the port had a valid enabled link. */
+	bool port_reset_pending;
+	/* Retry acknowledged PRC when no further port change IRQ arrives. */
+	struct delayed_work port_retry_work;
+	u8 port_reset_retry_count;
+
+	bool u2_rwe;
+	bool feature_u1_enable;
+	bool feature_u2_enable;
+
+	bool portsc_on_reconnecting;
+	bool u12_forbidden;
+	bool u12_initiate_forbidden;
+	bool reinit_ep0_on_bus_reset;
+
+	/*
+	 * usb_gadget_udc_reset() re-enters the gadget/function drivers
+	 * synchronously. Calling it directly from the port-status IRQ path
+	 * would require dropping udc_lock mid-event-processing, opening a
+	 * window for a second port-status event to run crg_udc_reinit()
+	 * again while the first event's SET_CONFIGURATION/set_alt() work is
+	 * still landing on another thread, tearing the just-enabled
+	 * endpoints back down. Defer the notification to this work item so
+	 * the IRQ-thread's event processing never has to drop the lock.
+	 */
+	struct work_struct udc_reset_work;
+	/* EP0 flushes udc_reset_work, so use separate work to drain the IRQ. */
+	struct work_struct recovery_work;
+};
+
+/*
+ * Caller must hold udc_lock. These helpers describe the current USB session;
+ * they do not replace endpoint, SETUP-tag, or generation validation.
+ */
+static inline bool crg_udc_session_ready(const struct crg_gadget_dev *udc)
+{
+	return !udc->controller_faulted && !udc->resetting &&
+	       !udc->controller_recovering && !udc->endpoint_recovery_pending &&
+	       !udc->shutting_down && !udc->gadget_stopping && !udc->suspended &&
+	       udc->pullup_on && !udc->stopped_needs_reset;
+}
+
+static inline bool crg_udc_gadget_ready(const struct crg_gadget_dev *udc)
+{
+	return crg_udc_session_ready(udc) && udc->gadget_driver &&
+	       udc->async_callbacks;
+}
+
+/* Keep usb_ep_disable() in the driver until the last giveback returns. */
+static inline void crg_udc_ep_clear_enabled(struct crg_udc_ep *ep)
+{
+	if (ep->nuking)
+		ep->clear_enabled_on_nuke = true;
+	else
+		ep->usb_ep.enabled = false;
+}
+
+/*
+ * TD Size is the number of USB packets remaining in this TD after this TRB,
+ * not the number of TRBs left. See Corigine USB 3.1 device spec, 8.2.2.
+ * A partially consumed packet is counted until the last TRB of the TD.
+ */
+static inline u32 crg_udc_td_size(u32 td_bytes, u32 bytes_through_trb,
+				  u32 maxpacket, bool last_trb)
+{
+	u32 packets;
+
+	if (last_trb || !td_bytes)
+		return 0;
+
+	packets = DIV_ROUND_UP(td_bytes, maxpacket) -
+		  bytes_through_trb / maxpacket;
+	return min(packets, 31U);
+}
+
+/*
+ * Controller recovery and bus-reset reinitialization use the same gate.
+ * Call these with udc_lock held at the outermost operation.
+ */
+static inline void crg_udc_recovery_begin(struct crg_gadget_dev *udc)
+{
+	udc->controller_recovering = true;
+}
+
+/*
+ * Called under udc_lock. Never run composite reset concurrently with
+ * ->setup(), ->disconnect() or a request completion callback.
+ */
+static inline void crg_udc_kick_reset_work(struct crg_gadget_dev *udc)
+{
+	if (udc->gadget_callbacks_active || udc->reset_callback_active ||
+	    udc->controller_recovering || udc->shutting_down ||
+	    udc->gadget_stopping || udc->suspended)
+		return;
+
+	if (udc->endpoint_recovery_pending || udc->external_reset_pending)
+		schedule_work(&udc->recovery_work);
+	else if (udc->bus_reset_pending && !udc->resetting &&
+		 udc->async_callbacks && udc->gadget_driver)
+		schedule_work(&udc->udc_reset_work);
+}
+
+static inline void crg_udc_recovery_end(struct crg_gadget_dev *udc)
+{
+	udc->controller_recovering = false;
+	if (udc->gadget_stopping || udc->shutting_down)
+		wake_up(&udc->teardown_wait);
+	if (udc->endpoint_recovery_pending)
+		crg_udc_kick_reset_work(udc);
+}
+
+/* Called with udc_lock held. Leave DMA-visible memory owned by the worker. */
+static inline void crg_udc_schedule_endpoint_recovery(struct crg_gadget_dev *udc)
+{
+	udc->endpoint_recovery_pending = true;
+	crg_udc_kick_reset_work(udc);
+}
+
+static inline void crg_udc_callback_enter(struct crg_gadget_dev *udc)
+{
+	udc->gadget_callbacks_active++;
+}
+
+static inline void crg_udc_callback_exit(struct crg_gadget_dev *udc)
+{
+	udc->gadget_callbacks_active--;
+	if (udc->gadget_stopping && !udc->gadget_callbacks_active)
+		wake_up(&udc->teardown_wait);
+	crg_udc_kick_reset_work(udc);
+}
+
+/* Shared prototypes */
+int crg_udc_build_ep0_status(struct crg_udc_ep *udc_ep_ptr, bool default_value,
+			      u32 status, struct crg_udc_request *udc_req_ptr,
+			      u8 set_addr, u8 stall);
+/*
+ * Abort the current EP0 control transfer when a new SETUP packet arrives.
+ *
+ * A SETUP packet is allowed to abort any in-progress control transfer. Do
+ * not defer the new SETUP until the old transfer completes: doing so leaves
+ * the old request on the EP0 software queue and subsequent gadget requests
+ * are rejected as "queue not empty".
+ *
+ * The controller's EP0 ring is reinitialized at the ring start, making all
+ * TRBs belonging to the aborted control transfer stale. The new SETUP is
+ * then handled immediately with a fresh setup tag.
+ */
+int crg_udc_abort_ep0_for_setup(struct crg_gadget_dev *crg_udc);
+int crg_udc_enable_setup(struct crg_gadget_dev *crg_udc);
+void crg_udc_ep0_req_complete(struct crg_udc_ep *udc_ep_ptr,
+			      unsigned long *flags);
+int crg_udc_prepare_for_setup(struct crg_gadget_dev *crg_udc);
+int crg_udc_set_ep0_halt(struct crg_gadget_dev *crg_udc);
+int crg_udc_update_ep0_maxpacketsize(struct crg_gadget_dev *crg_udc);
+void crg_udc_handle_setup_pkt(struct crg_gadget_dev *crg_udc,
+			      struct usb_ctrlrequest *setup_pkt, u8 setup_tag,
+			      unsigned long *flags);
+int crg_udc_init_ep0(struct crg_gadget_dev *crg_udc);
+int crg_udc_queue_ctrl(struct crg_udc_ep *udc_ep_ptr,
+		       struct crg_udc_request *udc_req_ptr,
+		       u32 num_of_trbs_needed);
+void crg_udc_advance_dequeue_pt(struct crg_udc_ep *udc_ep);
+unsigned int crg_udc_count_trbs_linear(dma_addr_t addr, u64 len);
+unsigned int crg_udc_count_trbs_sg(struct crg_udc_request *udc_req);
+int crg_udc_init_event_ring(struct crg_gadget_dev *crg_udc, int index);
+bool crg_udc_is_event_rings_empty(struct crg_gadget_dev *crg_udc);
+bool crg_udc_is_pointer_less_than(struct crg_transfer_trb *a,
+				  struct crg_transfer_trb *b,
+				  struct crg_udc_ep *udc_ep);
+int crg_udc_process_event_ring(struct crg_gadget_dev *crg_udc, int index,
+			       unsigned long *flags);
+/* This function will go through the list of the USB requests for the
+ * given endpoint and schedule any unscheduled trb's to the xfer ring
+ */
+int crg_udc_queue_pending_trbs(struct crg_udc_ep *udc_ep_ptr);
+/* num_trbs here is the size of the ring. */
+u32 crg_udc_room_on_ring(struct crg_gadget_dev *crg_udc, u32 num_trbs,
+			 struct crg_transfer_trb *p_ring,
+			 struct crg_transfer_trb *enq_pt,
+			 struct crg_transfer_trb *dq_pt);
+void crg_udc_setup_datastage_trb(struct crg_transfer_trb *p_trb, dma_addr_t dma,
+				 u8 pcs, u32 transfer_length, u32 td_size,
+				 bool ioc, bool chain, bool azp, u8 dir, u8 setup_tag);
+void crg_udc_setup_link_trb(struct crg_transfer_trb *link_trb, bool toggle,
+			    dma_addr_t next_trb);
+void crg_udc_setup_status_trb(struct crg_gadget_dev *crg_udc,
+			      struct crg_transfer_trb *p_trb,
+			      struct usb_request *usb_req, u8 pcs, u8 set_addr,
+			      u8 stall);
+void crg_udc_setup_trb(struct crg_transfer_trb *p_trb, u32 xfer_len,
+		       dma_addr_t xfer_buf_addr, u8 td_size, u8 pcs,
+		       u8 trb_type, bool short_pkt, bool chain_bit,
+		       bool intr_on_compl, bool b_setup_stage, u8 usb_dir,
+		       bool b_isoc, u16 frame_id, bool sia, bool azp);
+void crg_udc_squeeze_xfer_ring(struct crg_udc_ep *udc_ep_ptr,
+			       struct crg_udc_request *udc_req_ptr);
+struct crg_transfer_trb *crg_udc_tran_trb_dma_to_virt(struct crg_udc_ep *udc_ep,
+						    dma_addr_t address);
+dma_addr_t crg_udc_tran_trb_virt_to_dma(struct crg_udc_ep *udc_ep,
+					struct crg_transfer_trb *trb);
+void crg_udc_update_dequeue_pt(struct crg_event_trb *event,
+			       struct crg_udc_ep *udc_ep);
+u32 crg_udc_trb_chunk_len(dma_addr_t dma, u64 remaining);
+u32 crg_udc_actual_before(struct crg_udc_ep *udc_ep,
+			  struct crg_udc_request *udc_req,
+			  struct crg_transfer_trb *pause);
+/*
+ * command related ops
+ */
+int crg_udc_issue_command(struct crg_gadget_dev *crg_udc,
+			  enum crg_cmd_type type, u32 param0, u32 param1);
+int crg_udc_stop_gadget(struct crg_gadget_dev *crg_udc);
+int crg_udc_restart_gadget(struct crg_gadget_dev *crg_udc,
+			   unsigned long *flags);
+void crg_udc_release_quarantine(struct crg_gadget_dev *crg_udc);
+int crg_udc_program_speed(struct crg_gadget_dev *crg_udc);
+int crg_udc_reset_data_struct(struct crg_gadget_dev *crg_udc);
+int crg_udc_recover(struct crg_gadget_dev *crg_udc,
+		    unsigned long *flags);
+int crg_udc_recover_link(struct crg_gadget_dev *crg_udc, unsigned long *flags,
+			 bool notify_reset);
+int crg_udc_recover_endpoints(struct crg_gadget_dev *crg_udc,
+			      unsigned long *flags);
+struct usb_request *crg_udc_alloc_request(struct usb_ep *_ep, gfp_t gfp_flags);
+int crg_udc_build_td(struct crg_udc_ep *udc_ep_ptr,
+		     struct crg_udc_request *udc_req_ptr);
+void crg_udc_clear_portpm(struct crg_gadget_dev *crg_udc);
+void crg_udc_ep_nuke(struct crg_udc_ep *udc_ep, int status);
+void crg_udc_free_request(struct usb_ep *_ep, struct usb_request *_req);
+void crg_udc_set_pullup(struct crg_gadget_dev *crg_udc, bool is_on);
+void crg_udc_reset_work(struct work_struct *work);
+void crg_udc_port_retry_work(struct work_struct *work);
+void crg_udc_set_vbus_detect(struct crg_gadget_dev *crg_udc, bool enable);
+int crg_udc_bounce_map_request(struct crg_udc_ep *udc_ep,
+			       struct crg_udc_request *udc_req,
+			       enum dma_data_direction dir, gfp_t gfp);
+/* Unmap and free the private bounce buffer after hardware DMA stops. */
+void crg_udc_unmap_request(struct crg_udc_ep *udc_ep,
+			   struct crg_udc_request *udc_req, size_t received);
+int crg_udc_handle_port_status(struct crg_gadget_dev *crg_udc,
+			       unsigned long *flags);
+int crg_udc_handle_xfer_event(struct crg_gadget_dev *crg_udc,
+			      struct crg_event_trb *event, unsigned long *flags);
+void crg_udc_dispatch_setup(struct crg_gadget_dev *crg_udc,
+			    struct usb_ctrlrequest *setup, u8 setup_tag,
+			    unsigned long *flags);
+int crg_udc_handle_event(struct crg_gadget_dev *crg_udc,
+			 struct crg_event_trb *event, unsigned long *flags);
+/*
+ * Invoke the bound gadget driver's ->disconnect() callback, if any.
+ *
+ * The gadget disconnect callback may be called in interrupt context.
+ * Release only udc_lock while keeping local IRQs disabled.
+ */
+void crg_udc_notify_disconnect(struct crg_gadget_dev *crg_udc,
+			       unsigned long *flags);
+int crg_udc_ep_halt(struct crg_udc_ep *udc_ep_ptr, int halt, int ignore_wedge,
+		    unsigned long *flags);
+enum crg_ep_state crg_udc_get_ep_state(struct crg_gadget_dev *crg_udc, int dci);
+u32 crg_udc_index2dci(u16 index);
+int crg_udc_init_device_context(struct crg_gadget_dev *crg_udc);
+void crg_udc_init_ep_info(struct crg_gadget_dev *crg_udc);
+void crg_udc_knock_doorbell(struct crg_gadget_dev *crg_udc, int dci);
+/*
+ * Complete a request and give it back to the gadget layer via the
+ * standard usb_gadget_giveback_request() helper.
+ *
+ * Must be called with udc_lock held and interrupts disabled.
+ *
+ * The request's ->complete() callback must not sleep. crg_udc_req_done() drops
+ * udc_lock before the giveback and re-acquires it before returning.
+ * Callers therefore cannot assume state protected by udc_lock is
+ * unchanged across this call.
+ */
+void crg_udc_req_done(struct crg_udc_ep *udc_ep,
+		      struct crg_udc_request *udc_req, int status);
+void crg_udc_clear_req_container(struct crg_udc_request *udc_req_ptr);
+
+#define CRG_QUIRK_DIS_U1U2_ENTRY BIT(0)
+#define CRG_QUIRK_DIS_U1U2_INITIATION BIT(1)
+#define CRG_QUIRK_REINIT_EP0_ON_RESET BIT(2)
+
+#define CRG_UDC_QUIRKS_DEFAULT                                      \
+	(CRG_QUIRK_DIS_U1U2_ENTRY | CRG_QUIRK_DIS_U1U2_INITIATION | \
+	 CRG_QUIRK_REINIT_EP0_ON_RESET)
+
+struct crg_udc_priv {
+	u32 quirks;
+};
+
+extern const struct usb_gadget_ops crg_gadget_ops;
+extern const struct usb_ep_ops crg_udc_ep_ops;
+#define gadget_to_udc(g) (container_of(g, struct crg_gadget_dev, gadget))
+
+#endif /* CRG_UDC_H */

-- 
2.34.1


^ permalink raw reply related	[flat|nested] 13+ messages in thread

* [PATCH v5 3/6] usb: gadget: udc: crg: Add ring and transfer management
  2026-11-16 19:06 [PATCH v5 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
  2026-11-16 19:06 ` [PATCH v5 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC Petar Stepanovic
  2026-11-16 19:06 ` [PATCH v5 2/6] usb: gadget: udc: crg: Add hardware definitions and structs Petar Stepanovic
@ 2026-11-16 19:06 ` Petar Stepanovic
  2026-11-16 19:46   ` sashiko-bot
  2026-11-16 19:06 ` [PATCH v5 4/6] usb: gadget: udc: crg: Add endpoint 0 handling Petar Stepanovic
                   ` (2 subsequent siblings)
  5 siblings, 1 reply; 13+ messages in thread
From: Petar Stepanovic @ 2026-11-16 19:06 UTC (permalink / raw)
  To: Prasad Bolisetty, Greg Kroah-Hartman, Rob Herring,
	Krzysztof Kozlowski, Conor Dooley, Tzu-Hao Wei,
	Karthikeyan Mitran, Philipp Zabel, Neill Kapron
  Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
	Petar Stepanovic, Krutik Shah, Naveen Kumar Rajgiri Bassappa

This patch introduces the memory management and ring buffer logic
(TRBs and event rings) required by the Axiado AX3000/AX3005 UDC.

Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Tzu-Hao Wei <twei@axiado.com>
Signed-off-by: Tzu-Hao Wei <twei@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
 drivers/usb/gadget/udc/crg/crg-udc-ring.c | 696 ++++++++++++++++++++++++++++++
 1 file changed, 696 insertions(+)

diff --git a/drivers/usb/gadget/udc/crg/crg-udc-ring.c b/drivers/usb/gadget/udc/crg/crg-udc-ring.c
new file mode 100644
index 000000000000..348d2b611a33
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-ring.c
@@ -0,0 +1,696 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/sched.h>
+#include <linux/scatterlist.h>
+
+#include "crg-udc.h"
+
+void crg_udc_advance_dequeue_pt(struct crg_udc_ep *udc_ep)
+{
+	struct crg_udc_request *udc_req;
+
+	/* The completed request remains at the queue head until giveback. */
+	if (!list_empty(&udc_ep->queue)) {
+		udc_req = list_first_entry(&udc_ep->queue,
+					   struct crg_udc_request, queue);
+		if (!list_is_last(&udc_req->queue, &udc_ep->queue)) {
+			udc_req = list_next_entry(udc_req, queue);
+			if (udc_req->first_trb) {
+				udc_ep->deq_pt = udc_req->first_trb;
+				return;
+			}
+		}
+	}
+
+	udc_ep->deq_pt = udc_ep->enq_pt;
+}
+
+unsigned int crg_udc_count_trbs_linear(dma_addr_t addr, u64 len)
+{
+	unsigned int num_trbs = 0;
+
+	while (len) {
+		u32 chunk = crg_udc_trb_chunk_len(addr, len);
+
+		if (!chunk)
+			return 0;
+		num_trbs++;
+		addr += chunk;
+		len -= chunk;
+	}
+
+	return num_trbs;
+}
+
+unsigned int crg_udc_count_trbs_sg(struct crg_udc_request *udc_req)
+{
+	struct scatterlist *sg;
+	u64 remaining = udc_req->usb_req.length;
+	unsigned int i;
+	unsigned int num_trbs = 0;
+
+	for_each_sgtable_dma_sg(&udc_req->bounce->sgt, sg, i) {
+		dma_addr_t addr;
+		u64 len;
+
+		if (!sg_dma_len(sg))
+			continue;
+		if (!remaining)
+			break;
+
+		len = min_t(u64, sg_dma_len(sg), remaining);
+		addr = sg_dma_address(sg);
+		while (len) {
+			u32 chunk = crg_udc_trb_chunk_len(addr, len);
+
+			if (!chunk)
+				return 0;
+			num_trbs++;
+			addr += chunk;
+			len -= chunk;
+			remaining -= chunk;
+		}
+	}
+
+	return remaining ? 0 : num_trbs;
+}
+
+static dma_addr_t event_trb_virt_to_dma(struct crg_udc_event *udc_event,
+					struct crg_event_trb *event)
+{
+	dma_addr_t dma_addr = 0;
+	unsigned long seg_offset;
+
+	if (!udc_event || !event)
+		return DMA_MAPPING_ERROR;
+
+	if ((void *)event < udc_event->event_ring.cpu_addr ||
+	    (void *)event >=
+		    udc_event->event_ring.cpu_addr + udc_event->event_ring.len)
+		return DMA_MAPPING_ERROR;
+
+	seg_offset = (void *)event - udc_event->event_ring.cpu_addr;
+	dma_addr = udc_event->event_ring.dma_addr + seg_offset;
+
+	return dma_addr;
+}
+
+int crg_udc_init_event_ring(struct crg_gadget_dev *crg_udc, int index)
+{
+	struct crg_uicr __iomem *uicr = crg_udc->uicr[index];
+	struct crg_udc_event *udc_event = &crg_udc->udc_event[index];
+
+	int ret;
+	u32 buff_length;
+	dma_addr_t mapping;
+	bool erst_allocated = false;
+
+	buff_length = CRG_ERST_SIZE * sizeof(struct crg_erst);
+	if (!udc_event->erst.cpu_addr) {
+		udc_event->erst.cpu_addr = dma_alloc_coherent(
+			crg_udc->dev, buff_length, &mapping, GFP_ATOMIC);
+
+		if (!udc_event->erst.cpu_addr)
+			return -ENOMEM;
+		erst_allocated = true;
+	} else {
+		mapping = udc_event->erst.dma_addr;
+	}
+
+	udc_event->erst.len = buff_length;
+	udc_event->erst.dma_addr = mapping;
+	udc_event->p_erst = udc_event->erst.cpu_addr;
+
+	buff_length = CRG_EVENT_RING_SIZE * sizeof(struct crg_event_trb);
+	if (!udc_event->event_ring.cpu_addr) {
+		udc_event->event_ring.cpu_addr = dma_alloc_coherent(
+			crg_udc->dev, buff_length, &mapping, GFP_ATOMIC);
+
+		if (!udc_event->event_ring.cpu_addr) {
+			ret = -ENOMEM;
+			goto err_free_erst;
+		}
+	} else {
+		mapping = udc_event->event_ring.dma_addr;
+	}
+
+	udc_event->event_ring.len = buff_length;
+	udc_event->event_ring.dma_addr = mapping;
+	/* Invalidate event handlers which dropped udc_lock during reset. */
+	udc_event->generation++;
+	udc_event->evt_dq_pt = udc_event->event_ring.cpu_addr;
+
+	udc_event->evt_seg0_last_trb =
+		(struct crg_event_trb *)(udc_event->event_ring.cpu_addr) +
+		(CRG_EVENT_RING_SIZE - 1);
+
+	udc_event->ccs = 1;
+
+	udc_event->p_erst->seg_addr_lo =
+		cpu_to_le32(lower_32_bits(udc_event->event_ring.dma_addr));
+	udc_event->p_erst->seg_addr_hi =
+		cpu_to_le32(upper_32_bits(udc_event->event_ring.dma_addr));
+	udc_event->p_erst->seg_size = cpu_to_le32(CRG_EVENT_RING_SIZE);
+	udc_event->p_erst->rsvd = cpu_to_le32(0);
+	/* Ensure that event ring is updated */
+	dma_wmb();
+	/* clear the event ring, to avoid hw unexpected ops
+	 * because of dirty data
+	 */
+	memset(udc_event->event_ring.cpu_addr, 0, buff_length);
+
+	/* Ensure that event ring is clear before programming
+	 * ERSTBA/ERSTSZ/ERDP
+	 */
+	dma_wmb();
+
+	crg_udc->hw_dma_armed = true;
+	writel(CRG_ERST_SIZE, &uicr->erstsz);
+	writel(lower_32_bits(udc_event->erst.dma_addr), &uicr->erstbalo);
+	writel(upper_32_bits(udc_event->erst.dma_addr), &uicr->erstbahi);
+	writel(lower_32_bits(udc_event->event_ring.dma_addr) |
+		       CRG_U3DC_ERDPLO_EHB,
+	       &uicr->erdplo);
+	writel(upper_32_bits(udc_event->event_ring.dma_addr), &uicr->erdphi);
+
+	writel((CRG_U3DC_IMAN_INT_EN | CRG_U3DC_IMAN_INT_PEND), &uicr->iman);
+	/* Interrupt moderation: 4000 * 250ns = 1ms */
+	writel(4000, &uicr->imod);
+	return 0;
+
+err_free_erst:
+	/* Only free what this call allocated; a pre-existing erst is owned
+	 * by an earlier successful init and released by crg_udc_free_resources().
+	 */
+	if (erst_allocated) {
+		dma_free_coherent(crg_udc->dev, udc_event->erst.len,
+				  udc_event->erst.cpu_addr,
+				  udc_event->erst.dma_addr);
+		udc_event->erst.cpu_addr = NULL;
+	}
+	return ret;
+}
+
+static bool is_event_ring_x_empty(struct crg_gadget_dev *crg_udc, int index)
+{
+	struct crg_event_trb *event;
+	struct crg_udc_event *udc_event;
+
+	udc_event = &crg_udc->udc_event[index];
+	if (udc_event->evt_dq_pt) {
+		event = (struct crg_event_trb *)udc_event->evt_dq_pt;
+
+		if (FIELD_GET(EVE_TRB_CYCLE_BIT_MASK,
+			      le32_to_cpu(event->dw3)) != udc_event->ccs)
+			return true;
+	}
+
+	return false;
+}
+
+bool crg_udc_is_event_rings_empty(struct crg_gadget_dev *crg_udc)
+{
+	int i;
+
+	for (i = 0; i < CRG_RING_NUM; i++) {
+		if (!is_event_ring_x_empty(crg_udc, i)) {
+			dev_err(crg_udc->dev, "event ring not empty\n");
+			return false;
+		}
+	}
+	return true;
+}
+
+bool crg_udc_is_pointer_less_than(struct crg_transfer_trb *a,
+				  struct crg_transfer_trb *b,
+				  struct crg_udc_ep *udc_ep)
+{
+	if (!a || !b || !udc_ep->first_trb || !udc_ep->last_trb)
+		return false;
+	if (b > a && (udc_ep->enq_pt >= b || udc_ep->enq_pt < a))
+		return true;
+	if (b < a && (udc_ep->enq_pt >= b && udc_ep->enq_pt < a))
+		return true;
+	return false;
+}
+
+int crg_udc_process_event_ring(struct crg_gadget_dev *crg_udc, int index,
+			       unsigned long *flags)
+{
+	struct crg_event_trb *event;
+	struct crg_udc_event *udc_event;
+	struct crg_uicr __iomem *uicr = crg_udc->uicr[index];
+	u32 val;
+	dma_addr_t erdp;
+	int ret;
+	u32 generation;
+	unsigned int processed = 0;
+
+	if (!uicr)
+		return -ENODEV;
+
+	/* Do not touch the interrupter while the controller is resetting. */
+	if (unlikely(crg_udc->resetting))
+		return 0;
+
+	val = readl(&uicr->iman);
+	val |= CRG_U3DC_IMAN_INT_PEND;
+	writel(val, &uicr->iman);
+
+	udc_event = &crg_udc->udc_event[index];
+	generation = udc_event->generation;
+	while (udc_event->evt_dq_pt) {
+		event = (struct crg_event_trb *)udc_event->evt_dq_pt;
+
+		if (FIELD_GET(EVE_TRB_CYCLE_BIT_MASK,
+			      le32_to_cpu(event->dw3)) != udc_event->ccs)
+			break;
+		/* Read the payload only after observing device ownership. */
+		dma_rmb();
+
+		ret = crg_udc_handle_event(crg_udc, event, flags);
+		/* The giveback callback may release the lock and reset the ring. */
+		if (unlikely(generation != udc_event->generation ||
+			     crg_udc->resetting))
+			return 0;
+		if (ret == -ECONNRESET) {
+			/* The disconnect path rebuilt and reprogrammed the ring. */
+			return 0;
+		}
+		if (ret < 0)
+			dev_dbg(crg_udc->dev, "failed to process event: %d\n",
+				ret);
+
+		/* Consume malformed events so they cannot pin the ring head. */
+		if (event == udc_event->evt_seg0_last_trb) {
+			udc_event->ccs = udc_event->ccs ? 0 : 1;
+			udc_event->evt_dq_pt = udc_event->event_ring.cpu_addr;
+		} else {
+			udc_event->evt_dq_pt++;
+		}
+
+		/*
+		 * udc_event state is only re-read from crg_udc, never cached
+		 * across this drop, so releasing and reacquiring the lock
+		 * here is safe and lets other udc_lock users, and this CPU's
+		 * other interrupts, make progress during a busy event ring.
+		 */
+		if (++processed % CRG_EVENT_RING_BATCH == 0) {
+			erdp = event_trb_virt_to_dma(udc_event,
+						     udc_event->evt_dq_pt);
+			if (!dma_mapping_error(crg_udc->dev, erdp)) {
+				writel(upper_32_bits(erdp), &uicr->erdphi);
+				writel(lower_32_bits(erdp) |
+					       CRG_U3DC_ERDPLO_EHB,
+				       &uicr->erdplo);
+			}
+			spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+			cond_resched();
+			spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+			if (unlikely(generation != udc_event->generation ||
+				     crg_udc->resetting))
+				return 0;
+		}
+	}
+
+	/* update dequeue pointer */
+	erdp = event_trb_virt_to_dma(udc_event, udc_event->evt_dq_pt);
+	if (dma_mapping_error(crg_udc->dev, erdp))
+		return -EINVAL;
+	writel(upper_32_bits(erdp), &uicr->erdphi);
+	writel(lower_32_bits(erdp) | CRG_U3DC_ERDPLO_EHB, &uicr->erdplo);
+
+	return 0;
+}
+
+/* This function will go through the list of the USB requests for the
+ * given endpoint and schedule any unscheduled trb's to the xfer ring
+ */
+int crg_udc_queue_pending_trbs(struct crg_udc_ep *udc_ep_ptr)
+{
+	struct crg_udc_request *req;
+	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+	u32 generation = crg_udc->udc_event[0].generation;
+	u32 ep_generation = udc_ep_ptr->generation;
+	int ret;
+
+restart:
+	/* A giveback can reset/disable the endpoint while udc_lock is dropped. */
+	if (crg_udc->resetting || crg_udc->port_reset_pending ||
+	    crg_udc->endpoint_recovery_pending ||
+	    !udc_ep_ptr->desc || udc_ep_ptr->halt_failed || udc_ep_ptr->nuking ||
+	    udc_ep_ptr->ep_state != EP_STATE_RUNNING)
+		return 0;
+
+	list_for_each_entry(req, &udc_ep_ptr->queue, queue) {
+		if (req->all_trbs_queued)
+			continue;
+
+		/* A large request must finish its current TD before the next
+		 * TD (or the next USB request) can be submitted: the event
+		 * handler uses all_trbs_queued to complete the USB request.
+		 */
+		if (req->td_pending)
+			return 0;
+
+		ret = crg_udc_build_td(udc_ep_ptr, req);
+		if (ret == -ENOSPC || ret == -EBUSY)
+			return ret;
+		if (ret) {
+			dev_err(crg_udc->dev,
+				"failed to build pending transfer: %d\n", ret);
+			/* Giveback removes req and drops udc_lock: callbacks can
+			 * remove or free every saved request-list iterator.
+			 */
+			crg_udc_req_done(udc_ep_ptr, req, ret);
+			if (generation != crg_udc->udc_event[0].generation ||
+			    ep_generation != udc_ep_ptr->generation)
+				return 0;
+			goto restart;
+		}
+
+		if (udc_ep_ptr->tran_ring_full || !req->all_trbs_queued)
+			return 0;
+	}
+	return 0;
+}
+
+/* num_trbs here is the size of the ring. */
+u32 crg_udc_room_on_ring(struct crg_gadget_dev *crg_udc, u32 num_trbs,
+			 struct crg_transfer_trb *p_ring,
+			 struct crg_transfer_trb *enq_pt,
+			 struct crg_transfer_trb *dq_pt)
+{
+	u32 capacity, enq_idx, dq_idx;
+
+	/* One slot is the Link TRB and one data slot remains unused. */
+	if (num_trbs < 3 || !p_ring || !enq_pt || !dq_pt)
+		return 0;
+
+	capacity = num_trbs - 1;
+	/* Never dereference a pointer derived from an invalid/stale event. */
+	if ((unsigned long)enq_pt < (unsigned long)p_ring ||
+	    (unsigned long)dq_pt < (unsigned long)p_ring ||
+	    (unsigned long)enq_pt >= (unsigned long)(p_ring + capacity) ||
+	    (unsigned long)dq_pt >= (unsigned long)(p_ring + capacity)) {
+		WARN_ON_ONCE(1);
+		return 0;
+	}
+
+	enq_idx = enq_pt - p_ring;
+	dq_idx = dq_pt - p_ring;
+	if (enq_idx == dq_idx)
+		return capacity - 1;
+
+	/* Data slots strictly between enqueue and dequeue, modulo Link. */
+	return (dq_idx + capacity - enq_idx) % capacity - 1;
+}
+
+void crg_udc_setup_datastage_trb(struct crg_transfer_trb *p_trb, dma_addr_t dma,
+				 u8 pcs, u32 transfer_length, u32 td_size,
+				 bool ioc, bool chain, bool azp, u8 dir, u8 setup_tag)
+{
+	u32 tmp;
+
+	p_trb->dw0 = cpu_to_le32(lower_32_bits(dma));
+	p_trb->dw1 = cpu_to_le32(upper_32_bits(dma));
+
+	/* TRB_Transfer_Length
+	 * For USB_DIR_OUT, this field is the number of data bytes expected
+	 * from xhc. For USB_DIR_IN, this field is the number of data bytes
+	 * the device will send.
+	 */
+	tmp = 0;
+	FIELD_MODIFY(TRB_TRANSFER_LEN_MASK, &tmp, transfer_length);
+	FIELD_MODIFY(TRB_TD_SIZE_MASK, &tmp, td_size);
+	FIELD_MODIFY(TRB_INTR_TARGET_MASK, &tmp, 0);
+	p_trb->dw2 = cpu_to_le32(tmp);
+
+	tmp = 0;
+	FIELD_MODIFY(TRB_CYCLE_BIT_MASK, &tmp, pcs);
+	FIELD_MODIFY(TRB_INTR_ON_SHORT_PKT_MASK, &tmp, 1);
+	FIELD_MODIFY(TRB_INTR_ON_COMPLETION_MASK, &tmp, ioc);
+	FIELD_MODIFY(TRB_CHAIN_BIT_MASK, &tmp, chain);
+	FIELD_MODIFY(TRB_TYPE_MASK, &tmp, TRB_TYPE_XFER_DATA_STAGE);
+	FIELD_MODIFY(TRB_APPEND_ZLP_MASK, &tmp, azp);
+	FIELD_MODIFY(DATA_STAGE_TRB_DIR_MASK, &tmp, dir);
+	FIELD_MODIFY(TRB_SETUP_TAG_MASK, &tmp, setup_tag);
+
+	/* Publish ownership only after the TRB payload is visible. */
+	dma_wmb();
+	p_trb->dw3 = cpu_to_le32(tmp);
+}
+
+void crg_udc_setup_link_trb(struct crg_transfer_trb *link_trb, bool toggle,
+			    dma_addr_t next_trb)
+{
+	u32 dw = 0;
+
+	link_trb->dw0 = cpu_to_le32(lower_32_bits(next_trb));
+	link_trb->dw1 = cpu_to_le32(upper_32_bits(next_trb));
+	link_trb->dw2 = cpu_to_le32(0);
+
+	FIELD_MODIFY(TRB_TYPE_MASK, &dw, TRB_TYPE_LINK);
+	if (toggle)
+		FIELD_MODIFY(TRB_LINK_TOGGLE_CYCLE_MASK, &dw, 1);
+	else
+		FIELD_MODIFY(TRB_LINK_TOGGLE_CYCLE_MASK, &dw, 0);
+
+	/* Publish ownership only after the link address is visible. */
+	dma_wmb();
+	link_trb->dw3 = cpu_to_le32(dw);
+}
+
+void crg_udc_setup_status_trb(struct crg_gadget_dev *crg_udc,
+			      struct crg_transfer_trb *p_trb,
+			      struct usb_request *usb_req, u8 pcs, u8 set_addr,
+			      u8 stall)
+{
+	u32 tmp, dir = 0;
+
+	/* There are some cases where crg_udc_setup_status_trb() is called with
+	 * usb_req set to NULL.
+	 */
+	p_trb->dw0 = cpu_to_le32(0);
+	p_trb->dw1 = cpu_to_le32(0);
+
+	tmp = 0;
+	FIELD_MODIFY(TRB_INTR_TARGET_MASK, &tmp, 0);
+	p_trb->dw2 = cpu_to_le32(tmp);
+
+	tmp = 0;
+	FIELD_MODIFY(TRB_CYCLE_BIT_MASK, &tmp, pcs);
+	FIELD_MODIFY(TRB_INTR_ON_COMPLETION_MASK, &tmp, 1);
+	FIELD_MODIFY(TRB_TYPE_MASK, &tmp, TRB_TYPE_XFER_STATUS_STAGE);
+
+	dir = (crg_udc->setup_status == STATUS_STAGE_XFER) ? 0 : 1;
+	FIELD_MODIFY(DATA_STAGE_TRB_DIR_MASK, &tmp, dir);
+
+	FIELD_MODIFY(TRB_SETUP_TAG_MASK, &tmp, crg_udc->setup_tag);
+	FIELD_MODIFY(STATUS_STAGE_TRB_STALL_MASK, &tmp, stall);
+	FIELD_MODIFY(STATUS_STAGE_TRB_SET_ADDR_MASK, &tmp, set_addr);
+
+	/* Publish ownership only after the TRB payload is visible. */
+	dma_wmb();
+	p_trb->dw3 = cpu_to_le32(tmp);
+}
+
+void crg_udc_setup_trb(struct crg_transfer_trb *p_trb, u32 xfer_len,
+		       dma_addr_t xfer_buf_addr, u8 td_size, u8 pcs,
+		       u8 trb_type, bool short_pkt, bool chain_bit,
+		       bool intr_on_compl, bool b_setup_stage, u8 usb_dir,
+		       bool b_isoc, u16 frame_id, bool sia, bool azp)
+{
+	u32 tmp;
+
+	p_trb->dw0 = cpu_to_le32(lower_32_bits(xfer_buf_addr));
+	p_trb->dw1 = cpu_to_le32(upper_32_bits(xfer_buf_addr));
+
+	tmp = 0;
+	FIELD_MODIFY(TRB_TRANSFER_LEN_MASK, &tmp, xfer_len);
+	FIELD_MODIFY(TRB_TD_SIZE_MASK, &tmp, td_size);
+	FIELD_MODIFY(TRB_INTR_TARGET_MASK, &tmp, 0);
+
+	p_trb->dw2 = cpu_to_le32(tmp);
+
+	tmp = 0;
+	FIELD_MODIFY(TRB_CYCLE_BIT_MASK, &tmp, pcs);
+	FIELD_MODIFY(TRB_INTR_ON_SHORT_PKT_MASK, &tmp, short_pkt);
+	FIELD_MODIFY(TRB_CHAIN_BIT_MASK, &tmp, chain_bit);
+	FIELD_MODIFY(TRB_INTR_ON_COMPLETION_MASK, &tmp, intr_on_compl);
+	FIELD_MODIFY(TRB_APPEND_ZLP_MASK, &tmp, azp);
+	FIELD_MODIFY(TRB_TYPE_MASK, &tmp, trb_type);
+
+	if (b_setup_stage)
+		FIELD_MODIFY(DATA_STAGE_TRB_DIR_MASK, &tmp, usb_dir);
+
+	if (b_isoc) {
+		FIELD_MODIFY(ISOC_TRB_FRAME_ID_MASK, &tmp, frame_id);
+		FIELD_MODIFY(ISOC_TRB_SIA_MASK, &tmp, sia);
+	}
+
+	/* Publish ownership only after the TRB payload is visible. */
+	dma_wmb();
+	p_trb->dw3 = cpu_to_le32(tmp);
+}
+
+void crg_udc_squeeze_xfer_ring(struct crg_udc_ep *udc_ep_ptr,
+			       struct crg_udc_request *udc_req_ptr)
+{
+	struct crg_transfer_trb *trb = udc_req_ptr->first_trb;
+	struct crg_udc_request *next_req;
+	u8 new_pcs, clear_pcs;
+
+	if (!trb)
+		return;
+
+	/* Rewind the producer to the cancelled request.  These TRBs and
+	 * everything queued after them will be discarded and rebuilt.
+	 */
+	new_pcs = FIELD_GET(TRB_CYCLE_BIT_MASK, le32_to_cpu(trb->dw3));
+	clear_pcs = new_pcs;
+	while (trb != udc_ep_ptr->enq_pt) {
+		/* Zero is not always invalid: after a wrap the consumer
+		 * expects cycle 0.  Invalidate each slot for its cycle pass.
+		 */
+		le32p_replace_bits(&trb->dw3, clear_pcs ^ 1,
+				   TRB_CYCLE_BIT_MASK);
+		trb++;
+		if (trb == udc_ep_ptr->last_trb) {
+			trb = udc_ep_ptr->first_trb;
+			clear_pcs ^= 1;
+		}
+	}
+
+	udc_ep_ptr->enq_pt = udc_req_ptr->first_trb;
+	udc_ep_ptr->pcs = new_pcs;
+	udc_ep_ptr->tran_ring_full = false;
+	/* Do not overwrite the Link TRB: a previous active TD may still
+	 * require its cycle/chain bits to traverse the ring.
+	 */
+	dma_wmb();
+
+	/* Invalidate the bookkeeping of ALL subsequent requests before
+	 * attempting to rebuild any: rebuilding may stop when space runs out.
+	 * The caller gives back the cancelled request before rebuilding.
+	 */
+	next_req = udc_req_ptr;
+	list_for_each_entry_continue(next_req, &udc_ep_ptr->queue, queue) {
+		next_req->usb_req.status = -EINPROGRESS;
+		next_req->usb_req.actual = 0;
+		crg_udc_clear_req_container(next_req);
+	}
+}
+
+struct crg_transfer_trb *crg_udc_tran_trb_dma_to_virt(struct crg_udc_ep *udc_ep,
+						    dma_addr_t address)
+{
+	dma_addr_t offset;
+	struct crg_transfer_trb *trb_virt;
+
+	if (lower_32_bits(address) & 0xf)
+		return NULL;
+
+	offset = address - udc_ep->tran_ring_info.dma_addr;
+	if (unlikely(offset >= udc_ep->tran_ring_info.len))
+		return NULL;
+	offset = offset / sizeof(struct crg_transfer_trb);
+	trb_virt = udc_ep->first_trb + offset;
+	if (unlikely(trb_virt > udc_ep->last_trb))
+		return NULL;
+	return trb_virt;
+}
+
+dma_addr_t crg_udc_tran_trb_virt_to_dma(struct crg_udc_ep *udc_ep,
+					struct crg_transfer_trb *trb)
+{
+	unsigned long offset;
+	int trb_idx;
+	dma_addr_t dma_addr = 0;
+
+	trb_idx = trb - udc_ep->first_trb;
+	if (unlikely(trb_idx < 0 || trb > udc_ep->last_trb))
+		return DMA_MAPPING_ERROR;
+
+	offset = trb_idx * sizeof(*trb);
+	if (unlikely(offset >= udc_ep->tran_ring_info.len))
+		return DMA_MAPPING_ERROR;
+	dma_addr = udc_ep->tran_ring_info.dma_addr + offset;
+	return dma_addr;
+}
+
+void crg_udc_update_dequeue_pt(struct crg_event_trb *event,
+			       struct crg_udc_ep *udc_ep)
+{
+	u32 deq_pt_lo = le32_to_cpu(event->dw0);
+	u32 deq_pt_hi = le32_to_cpu(event->dw1);
+	u64 dq_pt_addr = (u64)deq_pt_lo | ((u64)deq_pt_hi << 32);
+	struct crg_transfer_trb *deq_pt;
+
+	deq_pt = crg_udc_tran_trb_dma_to_virt(udc_ep, dq_pt_addr);
+	if (!deq_pt)
+		return;
+
+	/*
+	 * The reported dequeue pointer can be the Link TRB itself, so wrap
+	 * before advancing: last_trb is the final entry of the ring buffer
+	 * and last_trb + 1 is outside the coherent allocation.
+	 */
+	if (deq_pt == udc_ep->last_trb)
+		deq_pt = udc_ep->first_trb;
+	else
+		deq_pt++;
+
+	if (deq_pt == udc_ep->last_trb ||
+	    FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(deq_pt->dw3)) == TRB_TYPE_LINK)
+		deq_pt = udc_ep->first_trb;
+
+	udc_ep->deq_pt = deq_pt;
+}
+
+u32 crg_udc_trb_chunk_len(dma_addr_t dma, u64 remaining)
+{
+	u32 boundary = CRG_UDC_TRB_MAX_BUFFER_SIZE -
+		       (lower_32_bits(dma) & (CRG_UDC_TRB_MAX_BUFFER_SIZE - 1));
+
+	return min_t(u64, remaining, boundary);
+}
+
+u32 crg_udc_actual_before(struct crg_udc_ep *udc_ep,
+			  struct crg_udc_request *udc_req,
+			  struct crg_transfer_trb *pause)
+{
+	struct crg_transfer_trb *trb = udc_req->first_trb;
+	u64 actual = udc_req->td_start_offset;
+	u32 guard = 0;
+	u32 max_iter;
+
+	if (!trb || !pause || !udc_ep->last_trb)
+		return 0;
+	max_iter = udc_ep->last_trb - udc_ep->first_trb + 1;
+
+	while (trb != pause && trb != udc_ep->enq_pt && guard++ < max_iter) {
+		u32 len;
+
+		/*
+		 * Wrap here rather than after the increment below: pause can
+		 * be the Link TRB, and jumping straight from last_trb - 1 to
+		 * first_trb would step over it and sum the whole ring.
+		 */
+		if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(trb->dw3)) ==
+		    TRB_TYPE_LINK) {
+			trb = udc_ep->first_trb;
+			continue;
+		}
+		len = FIELD_GET(TRB_TRANSFER_LEN_MASK, le32_to_cpu(trb->dw2));
+		actual += len;
+		trb++;
+	}
+
+	return min_t(u64, actual, udc_req->usb_req.length);
+}

-- 
2.34.1


^ permalink raw reply related	[flat|nested] 13+ messages in thread

* [PATCH v5 4/6] usb: gadget: udc: crg: Add endpoint 0 handling
  2026-11-16 19:06 [PATCH v5 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
                   ` (2 preceding siblings ...)
  2026-11-16 19:06 ` [PATCH v5 3/6] usb: gadget: udc: crg: Add ring and transfer management Petar Stepanovic
@ 2026-11-16 19:06 ` Petar Stepanovic
  2026-11-16 20:16   ` sashiko-bot
  2026-11-16 19:06 ` [PATCH v5 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations Petar Stepanovic
  2026-11-16 19:06 ` [PATCH v5 6/6] usb: gadget: udc: crg: Add core driver and build integration Petar Stepanovic
  5 siblings, 1 reply; 13+ messages in thread
From: Petar Stepanovic @ 2026-11-16 19:06 UTC (permalink / raw)
  To: Prasad Bolisetty, Greg Kroah-Hartman, Rob Herring,
	Krzysztof Kozlowski, Conor Dooley, Tzu-Hao Wei,
	Karthikeyan Mitran, Philipp Zabel, Neill Kapron
  Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
	Petar Stepanovic, Krutik Shah, Naveen Kumar Rajgiri Bassappa

This patch adds the endpoint 0 setup packet, status, and control
transfer handling for the Axiado AX3000/AX3005 UDC.

Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Tzu-Hao Wei <twei@axiado.com>
Signed-off-by: Tzu-Hao Wei <twei@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
 drivers/usb/gadget/udc/crg/crg-udc-ep0.c | 1264 ++++++++++++++++++++++++++++++
 1 file changed, 1264 insertions(+)

diff --git a/drivers/usb/gadget/udc/crg/crg-udc-ep0.c b/drivers/usb/gadget/udc/crg/crg-udc-ep0.c
new file mode 100644
index 000000000000..4bd0dd4da8c6
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-ep0.c
@@ -0,0 +1,1264 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/iopoll.h>
+
+#include "crg-udc.h"
+#include "crg-udc-trace.h"
+
+static void enable_setup_event(struct crg_gadget_dev *crg_udc);
+
+int crg_udc_build_ep0_status(struct crg_udc_ep *udc_ep_ptr, bool default_value,
+			      u32 status, struct crg_udc_request *udc_req_ptr,
+			      u8 set_addr, u8 stall)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+	struct crg_transfer_trb *enq_pt = udc_ep_ptr->enq_pt;
+
+	/* Internal status requests bypass usb_ep_queue(), so they must also
+	 * respect a teardown that is giving back the old EP0 queue.
+	 */
+	if (udc_ep_ptr->nuking) {
+		crg_udc->ep0_state = EP0_STATE_ERROR;
+		return -EBUSY;
+	}
+
+	/* Reserve a free transfer-ring slot before touching request state or
+	 * publishing the STATUS TRB. crg_udc_room_on_ring() treats equal
+	 * pointers as empty, so check tran_ring_full separately.
+	 */
+	if (!udc_ep_ptr->first_trb || !udc_ep_ptr->last_trb ||
+	    !udc_ep_ptr->enq_pt || !udc_ep_ptr->deq_pt)
+		return -ENODEV;
+	if (udc_ep_ptr->ep_state != EP_STATE_RUNNING)
+		return -ESHUTDOWN;
+	if (enq_pt < udc_ep_ptr->first_trb ||
+	    enq_pt >= udc_ep_ptr->last_trb ||
+	    udc_ep_ptr->deq_pt < udc_ep_ptr->first_trb ||
+	    udc_ep_ptr->deq_pt >= udc_ep_ptr->last_trb)
+		return -EIO;
+	if (udc_ep_ptr->tran_ring_full ||
+	    !crg_udc_room_on_ring(crg_udc, CRG_UDC_CONTROL_EP_TD_RING_SIZE,
+				 udc_ep_ptr->first_trb, enq_pt,
+				 udc_ep_ptr->deq_pt))
+		return -ENOSPC;
+
+	if (default_value) {
+		udc_req_ptr = crg_udc->status_req;
+		/* The shared request may still be in its data-stage callback. */
+		if (!udc_req_ptr || crg_udc->status_req_callback_pending ||
+		    !list_empty(&udc_req_ptr->queue)) {
+			dev_err(crg_udc->dev, "EP0 status request unavailable\n");
+			crg_udc->ep0_state = EP0_STATE_ERROR;
+			return udc_req_ptr ? -EBUSY : -ENODEV;
+		}
+		udc_req_ptr->usb_req.length = 0;
+		udc_req_ptr->usb_req.status = status;
+		udc_req_ptr->usb_req.actual = 0;
+		udc_req_ptr->usb_req.complete = NULL;
+	} else {
+		if (!udc_req_ptr) {
+			crg_udc->ep0_state = EP0_STATE_ERROR;
+			return -EINVAL;
+		}
+		udc_req_ptr->usb_req.status = status;
+		udc_req_ptr->usb_req.actual = 0;
+	}
+
+	crg_udc_setup_status_trb(crg_udc, enq_pt, &udc_req_ptr->usb_req,
+				 udc_ep_ptr->pcs, set_addr, stall);
+
+	/* STATUS requests must be matched against their own TRB too. */
+	udc_req_ptr->first_trb = enq_pt;
+	udc_req_ptr->last_trb = enq_pt;
+
+	enq_pt++;
+
+	/* check if we are at end of trb segment.  If so, update
+	 * pcs and enq for next segment
+	 */
+	if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(enq_pt->dw3)) ==
+	    TRB_TYPE_LINK) {
+		if (FIELD_GET(TRB_LINK_TOGGLE_CYCLE_MASK,
+			      le32_to_cpu(enq_pt->dw3))) {
+			dma_wmb();
+			le32p_replace_bits(&enq_pt->dw3, udc_ep_ptr->pcs,
+					   TRB_CYCLE_BIT_MASK);
+			udc_ep_ptr->pcs ^= 0x1;
+		}
+		enq_pt = udc_ep_ptr->first_trb;
+	}
+	udc_ep_ptr->enq_pt = enq_pt;
+
+	crg_udc_knock_doorbell(crg_udc, 0);
+
+	list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+	return 0;
+}
+
+/* Report status-stage publication errors consistently to all callers. */
+static void crg_udc_submit_ep0_status(struct crg_gadget_dev *crg_udc,
+				     int status, u8 set_addr)
+{
+	int ret;
+
+	ret = crg_udc_build_ep0_status(&crg_udc->udc_ep[0], true,
+				       status, NULL, set_addr, 0);
+	if (ret) {
+		/* A new SETUP must stop and reset EP0 before reuse. */
+		crg_udc->ep0_state = EP0_STATE_ERROR;
+		crg_udc->setup_fn_call_back = NULL;
+		dev_err(crg_udc->dev,
+			"failed to queue EP0 status stage: %d\n", ret);
+	}
+}
+
+/*
+ * Abort the current EP0 control transfer when a new SETUP packet arrives.
+ *
+ * A SETUP packet is allowed to abort any in-progress control transfer. Do
+ * not defer the new SETUP until the old transfer completes: doing so leaves
+ * the old request on the EP0 software queue and subsequent gadget requests
+ * are rejected as "queue not empty".
+ *
+ * The controller's EP0 ring is reinitialized at the ring start, making all
+ * TRBs belonging to the aborted control transfer stale. The new SETUP is
+ * then handled immediately with a fresh setup tag.
+ */
+int crg_udc_abort_ep0_for_setup(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_udc_ep *udc_ep0 = &crg_udc->udc_ep[0];
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	dma_addr_t dqptaddr;
+	u32 cmd_param0;
+	u32 cmd_param1;
+	u32 generation;
+	u32 tmp;
+	int ret;
+
+	/* A disconnect has stopped RUN. Reconnection rebuilds EP0 first. */
+	if (!crg_udc_session_ready(crg_udc))
+		return -ESHUTDOWN;
+
+	/*
+	 * A new SETUP is allowed to abort the current control transfer.
+	 * Keep EP0 in ABORTING for the complete stop/nuke/reinitialize
+	 * sequence. crg_udc_req_done() drops udc_lock while invoking the gadget
+	 * callback, so the callback may re-enter the UDC during the nuke.
+	 */
+	udc_ep0->generation++;
+	crg_udc->setup_status = WAIT_FOR_SETUP;
+	crg_udc->ep0_state = EP0_STATE_ABORTING;
+
+	/*
+	 * Stop EP0 before touching the transfer ring. The controller may still
+	 * own the old TRBs even when no EP0 request is running from the gadget's
+	 * point of view. Do not clear the DMA-visible ring until the hardware
+	 * reports that EP0 is no longer running.
+	 */
+	ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP, BIT(0), 0);
+	if (ret) {
+		dev_err(crg_udc->dev,
+			"failed to stop EP0 for SETUP abort: %d\n", ret);
+		goto recover;
+	}
+
+	ret = readl_poll_timeout_atomic(&uccr->ep_running, tmp, !(tmp & BIT(0)),
+					10, 10000);
+	if (ret) {
+		dev_err(crg_udc->dev, "timeout stopping EP0 for SETUP abort\n");
+		goto recover;
+	}
+
+	udc_ep0->ep_state = EP_STATE_STOPPED;
+	/* Complete requests belonging to the aborted control transfer. */
+	generation = udc_ep0->generation;
+	crg_udc_ep_nuke(udc_ep0, -ECONNRESET);
+	/*
+	 * Giveback drops udc_lock. A disconnect can stop RUN without
+	 * changing the endpoint generation, so check the USB session too.
+	 */
+	if (!crg_udc_gadget_ready(crg_udc) ||
+	    generation != udc_ep0->generation)
+		return -ECONNRESET;
+
+	if (!udc_ep0->first_trb || !udc_ep0->tran_ring_info.cpu_addr) {
+		ret = -ENODEV;
+		goto recover;
+	}
+
+	/*
+	 * The controller has stopped consuming EP0 TRBs. It is now safe to
+	 * discard the old transfer descriptors and restart the ring from its
+	 * beginning. No ring memory is freed here.
+	 */
+	memset(udc_ep0->first_trb, 0, udc_ep0->tran_ring_info.len);
+	udc_ep0->enq_pt = udc_ep0->first_trb;
+	udc_ep0->deq_pt = udc_ep0->first_trb;
+	udc_ep0->pcs = 1;
+	udc_ep0->tran_ring_full = false;
+	crg_udc_setup_link_trb(udc_ep0->last_trb, true,
+			       udc_ep0->tran_ring_info.dma_addr);
+
+	/* Make the new ring contents visible before handing the ring to HW. */
+	dma_wmb();
+
+	dqptaddr = crg_udc_tran_trb_virt_to_dma(udc_ep0, udc_ep0->first_trb);
+	if (dma_mapping_error(crg_udc->dev, dqptaddr)) {
+		ret = -EIO;
+		goto recover;
+	}
+
+	cmd_param0 = (lower_32_bits(dqptaddr) & CRG_CMD0_0_DQPTRLO_MASK) |
+		     FIELD_PREP(CRG_CMD0_0_DCS_MASK, udc_ep0->pcs);
+	cmd_param1 = upper_32_bits(dqptaddr);
+
+	ret = crg_udc_issue_command(crg_udc, CRG_CMD_INIT_EP0, cmd_param0,
+				    cmd_param1);
+	if (ret) {
+		dev_err(crg_udc->dev,
+			"failed to reinitialize EP0 for SETUP: %d\n", ret);
+		goto recover;
+	}
+
+	udc_ep0->ep_state = EP_STATE_RUNNING;
+	crg_udc->ep0_state = EP0_STATE_IDLE;
+	return 0;
+
+recover:
+	crg_udc->ep0_state = EP0_STATE_ERROR;
+	crg_udc_schedule_endpoint_recovery(crg_udc);
+	return ret;
+}
+
+int crg_udc_enable_setup(struct crg_gadget_dev *crg_udc)
+{
+	enable_setup_event(crg_udc);
+	crg_udc->device_state = USB_STATE_DEFAULT;
+	usb_gadget_set_state(&crg_udc->gadget, USB_STATE_DEFAULT);
+	crg_udc->setup_status = WAIT_FOR_SETUP;
+
+	return 0;
+}
+
+static void enable_setup_event(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 val;
+
+	val = readl(&uccr->config1);
+	val |= CRG_U3DC_CFG1_SETUP_EVENT_EN;
+	writel(val, &uccr->config1);
+}
+
+void crg_udc_ep0_req_complete(struct crg_udc_ep *udc_ep_ptr,
+			      unsigned long *flags)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+
+	switch (crg_udc->setup_status) {
+	case DATA_STAGE_XFER:
+		crg_udc->setup_status = STATUS_STAGE_RECV;
+		crg_udc_submit_ep0_status(crg_udc, -EINPROGRESS, 0);
+		break;
+	case DATA_STAGE_RECV:
+		crg_udc->setup_status = STATUS_STAGE_XFER;
+		crg_udc_submit_ep0_status(crg_udc, -EINPROGRESS, 0);
+		break;
+	default:
+		if (crg_udc->setup_fn_call_back) {
+			crg_udc->setup_fn_call_back(crg_udc);
+			crg_udc->setup_fn_call_back = NULL;
+		}
+		crg_udc->setup_status = WAIT_FOR_SETUP;
+		break;
+	}
+}
+
+static void get_status_cmpl(struct usb_ep *ep, struct usb_request *req)
+{
+	struct crg_udc_ep *udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+	struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+	unsigned long flags;
+	void *buf;
+
+	/* The data-stage completion handler queues the status stage. */
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	buf = req->buf;
+	req->buf = NULL;
+	crg_udc->status_req_callback_pending = false;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	kfree(buf);
+}
+
+static void getstatusrequest(struct crg_gadget_dev *crg_udc, u8 request_type,
+			     u16 value, u16 index, u16 length)
+{
+	struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+	struct crg_udc_request *udc_req_ptr = crg_udc->status_req;
+	u16 status_val = 0;
+	u32 dci;
+	int ret;
+
+	if (!udc_req_ptr || crg_udc->status_req_callback_pending ||
+	    !list_empty(&udc_req_ptr->queue) || value ||
+	    length != sizeof(u16)) {
+		crg_udc_set_ep0_halt(crg_udc);
+		return;
+	}
+
+	switch (request_type & USB_RECIP_MASK) {
+	case USB_RECIP_DEVICE:
+		if (index)
+			goto invalid;
+		if (crg_udc->gadget.speed == USB_SPEED_HIGH ||
+		    crg_udc->gadget.speed == USB_SPEED_FULL) {
+			if (crg_udc->u2_rwe)
+				status_val |= BIT(USB_DEVICE_REMOTE_WAKEUP);
+		} else if (crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+			if (crg_udc->feature_u1_enable)
+				status_val |= BIT(USB_DEV_STAT_U1_ENABLED);
+			if (crg_udc->feature_u2_enable)
+				status_val |= BIT(USB_DEV_STAT_U2_ENABLED);
+		}
+		if (crg_udc->gadget.is_selfpowered)
+			status_val |= BIT(USB_DEVICE_SELF_POWERED);
+		break;
+
+	case USB_RECIP_INTERFACE:
+		if (crg_udc->device_state != USB_STATE_CONFIGURED ||
+		    (index & 0xff00))
+			goto invalid;
+		break;
+
+	case USB_RECIP_ENDPOINT:
+		if (index & ~(USB_ENDPOINT_NUMBER_MASK | USB_DIR_IN))
+			goto invalid;
+		dci = crg_udc_index2dci(index);
+		if (dci == 1 || dci >= CRG_NUM_EP_CX)
+			goto invalid;
+		if (crg_udc->device_state < USB_STATE_ADDRESS && dci != 0)
+			goto invalid;
+		if (crg_udc->device_state == USB_STATE_ADDRESS && dci != 0)
+			goto invalid;
+		if (crg_udc->device_state == USB_STATE_CONFIGURED &&
+		    crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_DISABLED)
+			goto invalid;
+		if (crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_HALTED)
+			status_val = BIT(USB_ENDPOINT_HALT);
+		break;
+
+	default:
+		goto invalid;
+	}
+
+	kfree(udc_req_ptr->usb_req.buf);
+
+	udc_req_ptr->usb_req.buf = kzalloc_obj(status_val, GFP_ATOMIC);
+	if (!udc_req_ptr->usb_req.buf)
+		goto invalid;
+
+	*(__le16 *)udc_req_ptr->usb_req.buf = cpu_to_le16(status_val);
+	udc_req_ptr->usb_req.length = sizeof(status_val);
+	udc_req_ptr->usb_req.status = -EINPROGRESS;
+	udc_req_ptr->usb_req.actual = 0;
+	udc_req_ptr->usb_req.complete = get_status_cmpl;
+	crg_udc->status_req_callback_pending = true;
+
+	ret = crg_udc_bounce_map_request(udc_ep_ptr, udc_req_ptr,
+					 DMA_TO_DEVICE, GFP_ATOMIC);
+	if (ret) {
+		kfree(udc_req_ptr->usb_req.buf);
+		udc_req_ptr->usb_req.buf = NULL;
+		crg_udc->status_req_callback_pending = false;
+		goto invalid;
+	}
+
+	crg_udc->setup_status = DATA_STAGE_XFER;
+
+	ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+	if (ret) {
+		crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr, 0);
+		kfree(udc_req_ptr->usb_req.buf);
+		udc_req_ptr->usb_req.buf = NULL;
+		crg_udc->status_req_callback_pending = false;
+		crg_udc_set_ep0_halt(crg_udc);
+		return;
+	}
+	list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+	return;
+
+invalid:
+	crg_udc_set_ep0_halt(crg_udc);
+}
+
+int crg_udc_prepare_for_setup(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_udc_ep *udc_ep0_ptr;
+	dma_addr_t dqptaddr;
+	u32 cmd_param0;
+	u32 cmd_param1;
+	int ret;
+
+	if (!crg_udc_gadget_ready(crg_udc))
+		return -ESHUTDOWN;
+
+	if (!crg_udc_is_event_rings_empty(crg_udc) ||
+	    crg_udc->portsc_on_reconnecting == 1)
+		return -EBUSY;
+
+	udc_ep0_ptr = &crg_udc->udc_ep[0];
+	udc_ep0_ptr->generation++;
+	/* If we reinit ep0 on bus reset, we just make ep0 dequeue pointer
+	 * align with enqueue pointer, all remaining xfer trbs became dumb once
+	 * which will not produce xfer event anymore.
+	 *
+	 * If we considering the opposite solution, we should wait all ep0 xfer
+	 * trbs be completed(with some err complete code)
+	 */
+	if (crg_udc->reinit_ep0_on_bus_reset) {
+		struct crg_uccr __iomem *uccr = crg_udc->uccr;
+		enum crg_ep_state ep0_ep_state;
+		u32 generation;
+		u32 running;
+
+		/* Block requeue from giveback callbacks, then quiesce DMA. */
+		crg_udc->ep0_state = EP0_STATE_ABORTING;
+		/* A HALTED EP0 may still own a queued STALL TRB. */
+		ep0_ep_state = crg_udc_get_ep_state(crg_udc, 0);
+		if (ep0_ep_state != EP_STATE_DISABLED &&
+		    ep0_ep_state != EP_STATE_STOPPED) {
+			ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP,
+						    BIT(0), 0);
+			if (ret)
+				goto reset_error;
+
+			ret = readl_poll_timeout_atomic(&uccr->ep_running, running,
+							!(running & BIT(0)), 10, 10000);
+			if (ret)
+				goto reset_error;
+		}
+		udc_ep0_ptr->ep_state = EP_STATE_STOPPED;
+		generation = udc_ep0_ptr->generation;
+		crg_udc_ep_nuke(udc_ep0_ptr, -ESHUTDOWN);
+		/* The giveback callback may have disconnected the gadget. */
+		if (!crg_udc_gadget_ready(crg_udc) ||
+		    generation != udc_ep0_ptr->generation)
+			return -ECONNRESET;
+
+		/* context related ops */
+		dqptaddr = crg_udc_tran_trb_virt_to_dma(udc_ep0_ptr,
+							udc_ep0_ptr->enq_pt);
+		if (dma_mapping_error(crg_udc->dev, dqptaddr)) {
+			ret = -EIO;
+			goto reset_error;
+		}
+		cmd_param0 =
+			(lower_32_bits(dqptaddr) & CRG_CMD0_0_DQPTRLO_MASK) |
+			FIELD_PREP(CRG_CMD0_0_DCS_MASK, udc_ep0_ptr->pcs);
+		cmd_param1 = upper_32_bits(dqptaddr);
+		ret = crg_udc_issue_command(crg_udc, CRG_CMD_INIT_EP0,
+						cmd_param0, cmd_param1);
+		if (ret)
+			goto reset_error;
+
+		udc_ep0_ptr->deq_pt = udc_ep0_ptr->enq_pt;
+		udc_ep0_ptr->tran_ring_full = false;
+		udc_ep0_ptr->ep_state = EP_STATE_RUNNING;
+		crg_udc->ep0_state = EP0_STATE_IDLE;
+	} else {
+		if (!list_empty(&udc_ep0_ptr->queue)) {
+			dev_err(crg_udc->dev,
+				"remaining ep0 xfer trbs on reset\n");
+			return -EBUSY;
+		}
+	}
+
+	crg_udc_enable_setup(crg_udc);
+
+	return 0;
+
+reset_error:
+	crg_udc->ep0_state = EP0_STATE_ERROR;
+	crg_udc_schedule_endpoint_recovery(crg_udc);
+	return ret;
+}
+
+static void set_address_cmpl(struct crg_gadget_dev *crg_udc)
+{
+	if (crg_udc->device_state == USB_STATE_DEFAULT &&
+	    crg_udc->dev_addr != 0) {
+		crg_udc->device_state = USB_STATE_ADDRESS;
+		usb_gadget_set_state(&crg_udc->gadget, USB_STATE_ADDRESS);
+	} else if (crg_udc->device_state == USB_STATE_ADDRESS) {
+		if (crg_udc->dev_addr == 0) {
+			crg_udc->device_state = USB_STATE_DEFAULT;
+			usb_gadget_set_state(&crg_udc->gadget,
+					     USB_STATE_DEFAULT);
+		}
+
+	}
+}
+
+int crg_udc_set_ep0_halt(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+	enum crg_ep_state ep_state;
+	int ret;
+
+	ep_state = crg_udc_get_ep_state(crg_udc, udc_ep_ptr->dci);
+	if (ep_state == EP_STATE_HALTED)
+		return 0;
+	if (ep_state == EP_STATE_DISABLED)
+		return -ESHUTDOWN;
+
+	/* STALL terminates the current DATA/STATUS sequence. */
+	crg_udc->setup_status = STATUS_STAGE_XFER;
+	crg_udc->setup_fn_call_back = NULL;
+
+	ret = crg_udc_build_ep0_status(udc_ep_ptr, true, -EINVAL,
+				       NULL, 0, 1);
+	if (ret) {
+		/* A failed STALL publication leaves EP0 needing recovery. */
+		crg_udc->ep0_state = EP0_STATE_ERROR;
+		return ret;
+	}
+
+	udc_ep_ptr->ep_state = EP_STATE_HALTED;
+
+	return 0;
+}
+
+static void set_sel_cmpl(struct usb_ep *ep, struct usb_request *req)
+{
+	struct crg_udc_ep *udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+	struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+	const struct usb_set_sel_req *sel;
+	unsigned long flags;
+	void *buf;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	buf = req->buf;
+	if (buf && !req->status && req->actual >= sizeof(*sel)) {
+		sel = buf;
+		crg_udc->sel_value.u1_sel_value = sel->u1_sel;
+		crg_udc->sel_value.u1_pel_value = sel->u1_pel;
+		crg_udc->sel_value.u2_sel_value = le16_to_cpu(sel->u2_sel);
+		crg_udc->sel_value.u2_pel_value = le16_to_cpu(sel->u2_pel);
+	}
+	req->buf = NULL;
+	crg_udc->status_req_callback_pending = false;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	kfree(buf);
+}
+
+static void set_test_mode_cmpl(struct crg_gadget_dev *crg_udc)
+{
+	if (crg_udc->set_tm != 0) {
+		u32 tmp;
+		struct crg_uccr __iomem *uccr = crg_udc->uccr;
+
+		tmp = readl(&uccr->u2portpmsc);
+		FIELD_MODIFY(CRG_U3DC_U2PORTPM_TM_MASK, &tmp, crg_udc->set_tm);
+		writel(tmp, &uccr->u2portpmsc);
+
+		crg_udc->set_tm = 0;
+	}
+}
+
+static void setaddressrequest(struct crg_gadget_dev *crg_udc, u16 value,
+			      u16 index, u16 length)
+{
+	u32 param0;
+
+	if (value > 127 || index || length ||
+	    (crg_udc->device_state != USB_STATE_DEFAULT &&
+	     crg_udc->device_state != USB_STATE_ADDRESS))
+		goto invalid;
+
+	/* SET_ADDRESS(0) is legal in the Default state. */
+	param0 = FIELD_PREP(CRG_CMD2_0_DEV_ADDR_MASK, value);
+	if (crg_udc_issue_command(crg_udc, CRG_CMD_SET_ADDR, param0, 0))
+		goto invalid;
+
+	crg_udc->dev_addr = value;
+	crg_udc->setup_status = STATUS_STAGE_XFER;
+	crg_udc->setup_fn_call_back = set_address_cmpl;
+	crg_udc_submit_ep0_status(crg_udc, -EINPROGRESS, 1);
+	return;
+
+invalid:
+	crg_udc->setup_fn_call_back = NULL;
+	crg_udc_set_ep0_halt(crg_udc);
+}
+
+static bool setfeaturesrequest(struct crg_gadget_dev *crg_udc, u8 request_type,
+			       u8 b_request, u16 value, u16 index, u16 length,
+			       unsigned long *flags)
+{
+	int status = -EINPROGRESS;
+	u8 dci;
+	struct crg_udc_ep *udc_ep_ptr;
+	u32 tmp;
+	u32 set_feat = 0;
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+
+	if (length || (request_type & USB_DIR_IN))
+		goto set_feature_error;
+
+	/* TEST_MODE is permitted in the Default state; other features aren't. */
+	if (crg_udc->device_state == USB_STATE_DEFAULT &&
+	    !((request_type & USB_RECIP_MASK) == USB_RECIP_DEVICE &&
+	      value == USB_DEVICE_TEST_MODE &&
+	      b_request == USB_REQ_SET_FEATURE))
+		goto set_feature_error;
+
+	set_feat = (b_request == USB_REQ_SET_FEATURE) ? 1 : 0;
+	if ((request_type & (USB_RECIP_MASK | USB_TYPE_MASK)) ==
+	    (USB_RECIP_ENDPOINT | USB_TYPE_STANDARD)) {
+		dev_dbg(crg_udc->dev, "Halt/Unhalt EP\n");
+		if (crg_udc->device_state == USB_STATE_ADDRESS) {
+			if (index != 0) {
+				status = -EINVAL;
+				goto set_feature_error;
+			}
+		}
+
+		if (value != USB_ENDPOINT_HALT)
+			goto set_feature_error;
+
+		if (index & ~(USB_ENDPOINT_NUMBER_MASK | USB_DIR_IN))
+			goto set_feature_error;
+
+		dci = crg_udc_index2dci(index);
+		if (dci == 1 || dci >= CRG_NUM_EP_CX)
+			goto set_feature_error;
+		if (dci != 0 && crg_udc->device_state == USB_STATE_CONFIGURED &&
+		    crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_DISABLED)
+			goto set_feature_error;
+
+		udc_ep_ptr = &crg_udc->udc_ep[dci];
+		status = crg_udc_ep_halt(
+			udc_ep_ptr, (b_request == USB_REQ_SET_FEATURE) ? 1 : 0,
+			0, flags);
+
+		if (status < 0)
+			goto set_feature_error;
+	} else if ((request_type & (USB_RECIP_MASK | USB_TYPE_MASK)) ==
+		   (USB_RECIP_DEVICE | USB_TYPE_STANDARD)) {
+		switch (value) {
+		case USB_DEVICE_REMOTE_WAKEUP:
+			/* REMOTE_WAKEUP selector is not used by USB3.0 */
+			if (index ||
+			    crg_udc->device_state < USB_STATE_DEFAULT ||
+			    crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+				status = -EINVAL;
+				goto set_feature_error;
+			}
+
+			crg_udc->u2_rwe = set_feat;
+			crg_udc->gadget.wakeup_armed = set_feat;
+
+			break;
+		case USB_DEVICE_U1_ENABLE:
+		case USB_DEVICE_U2_ENABLE: {
+			u32 timeout_val;
+
+			if (crg_udc->device_state != USB_STATE_CONFIGURED) {
+				dev_err(crg_udc->dev,
+					"u12 enable fail, usb state=%d\n",
+					crg_udc->device_state);
+				status = -EINVAL;
+				goto set_feature_error;
+			}
+
+			if (index != 0 ||
+			    crg_udc->gadget.speed < USB_SPEED_SUPER) {
+				status = -EINVAL;
+				goto set_feature_error;
+			}
+
+			if (set_feat == 1 && crg_udc->u12_forbidden) {
+				status = -EINVAL;
+				goto set_feature_error;
+			}
+
+			tmp = readl(&uccr->u3portpmsc);
+			if (value == USB_DEVICE_U1_ENABLE) {
+				if (!crg_udc->u12_initiate_forbidden) {
+					tmp &= (~CRG_U3DC_U3PORTPM_U1IEN);
+					tmp |= FIELD_PREP(
+						CRG_U3DC_U3PORTPM_U1IEN,
+						set_feat);
+				}
+				if (!crg_udc->u12_forbidden)
+					timeout_val =
+						set_feat ? CRG_UDC_U1_TIMEOUT_VAL : 0;
+				else
+					timeout_val = 0;
+
+				FIELD_MODIFY(CRG_U3DC_U3PORTPM_U1TMOUT_MASK,
+					     &tmp, timeout_val);
+
+				crg_udc->feature_u1_enable = set_feat;
+			}
+
+			if (value == USB_DEVICE_U2_ENABLE) {
+				if (!crg_udc->u12_initiate_forbidden) {
+					tmp &= (~CRG_U3DC_U3PORTPM_U2IEN);
+					tmp |= FIELD_PREP(
+						CRG_U3DC_U3PORTPM_U2IEN,
+						set_feat);
+				}
+				if (!crg_udc->u12_forbidden)
+					timeout_val =
+						set_feat ? CRG_UDC_U2_TIMEOUT_VAL : 0;
+				else
+					timeout_val = 0;
+
+				FIELD_MODIFY(CRG_U3DC_U3PORTPM_U2TMOUT_MASK,
+					     &tmp, timeout_val);
+
+				crg_udc->feature_u2_enable = set_feat;
+			}
+			writel(tmp, &uccr->u3portpmsc);
+
+			break;
+		}
+		case USB_DEVICE_TEST_MODE: {
+			u32 u_pattern;
+
+			if (b_request != USB_REQ_SET_FEATURE ||
+			    crg_udc->gadget.speed != USB_SPEED_HIGH ||
+			    crg_udc->device_state < USB_STATE_DEFAULT ||
+			    (index & 0xff) || (index >> 8) == 0 ||
+			    (index >> 8) > 6)
+				goto set_feature_error;
+			u_pattern = index >> 8;
+			crg_udc->set_tm = u_pattern;
+			crg_udc->setup_fn_call_back = &set_test_mode_cmpl;
+
+			break;
+		}
+
+		default:
+			goto set_feature_error;
+		}
+
+	} else if ((request_type & (USB_RECIP_MASK | USB_TYPE_MASK)) ==
+		   (USB_RECIP_INTERFACE | USB_TYPE_STANDARD)) {
+		if (crg_udc->device_state != USB_STATE_CONFIGURED) {
+			dev_err(crg_udc->dev,
+				"interface u12 enable fail, usb state=%d\n",
+				crg_udc->device_state);
+			status = -EINVAL;
+			goto set_feature_error;
+		}
+
+		/* Suspend Option */
+		if (value != USB_INTRF_FUNC_SUSPEND) {
+			status = -EINVAL;
+			goto set_feature_error;
+		}
+
+		/* Both cleared LP/RW bits and either suspend option are valid. */
+		if (index & ~(0xff | USB_INTRF_FUNC_SUSPEND_LP |
+			      USB_INTRF_FUNC_SUSPEND_RW))
+			goto set_feature_error;
+
+		if (index & USB_INTRF_FUNC_SUSPEND_RW)
+			dev_dbg(crg_udc->dev, "Interface En Remote Wakeup\n");
+		else
+			dev_dbg(crg_udc->dev, "Interface Dis RemoteWakeup\n");
+
+		/* Do not need to return status stage here
+		 * Pass to composite gadget driver to process
+		 * the request
+		 */
+		return false;
+	} else {
+		/* Unknown recipient must not receive a successful status stage. */
+		goto set_feature_error;
+	}
+
+	crg_udc->setup_status = STATUS_STAGE_XFER;
+	crg_udc_submit_ep0_status(crg_udc, status, 0);
+	return true;
+
+set_feature_error:
+	crg_udc_set_ep0_halt(crg_udc);
+	return true;
+}
+
+static void setselrequest(struct crg_gadget_dev *crg_udc, u16 value, u16 index,
+			  u16 length)
+{
+	struct crg_udc_request *udc_req_ptr = crg_udc->status_req;
+	struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+	struct usb_set_sel_req *sel;
+	int ret;
+
+	if (!udc_req_ptr || crg_udc->status_req_callback_pending ||
+	    !list_empty(&udc_req_ptr->queue) ||
+	    crg_udc->device_state == USB_STATE_DEFAULT || value || index ||
+	    length != sizeof(struct usb_set_sel_req)) {
+		crg_udc_set_ep0_halt(crg_udc);
+		return;
+	}
+
+	kfree(udc_req_ptr->usb_req.buf);
+
+	udc_req_ptr->usb_req.buf = kzalloc_obj(*sel, GFP_ATOMIC);
+	if (!udc_req_ptr->usb_req.buf) {
+		crg_udc_set_ep0_halt(crg_udc);
+		return;
+	}
+
+	sel = udc_req_ptr->usb_req.buf;
+	udc_req_ptr->usb_req.length = sizeof(*sel);
+	udc_req_ptr->usb_req.status = -EINPROGRESS;
+	udc_req_ptr->usb_req.actual = 0;
+	udc_req_ptr->usb_req.complete = set_sel_cmpl;
+	crg_udc->status_req_callback_pending = true;
+	ret = crg_udc_bounce_map_request(udc_ep_ptr, udc_req_ptr,
+					 DMA_FROM_DEVICE, GFP_ATOMIC);
+	if (ret) {
+		kfree(udc_req_ptr->usb_req.buf);
+		udc_req_ptr->usb_req.buf = NULL;
+		crg_udc->status_req_callback_pending = false;
+		crg_udc_set_ep0_halt(crg_udc);
+		return;
+	}
+
+	crg_udc->setup_status = DATA_STAGE_RECV;
+
+	ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+	if (ret) {
+		crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr, 0);
+		kfree(udc_req_ptr->usb_req.buf);
+		udc_req_ptr->usb_req.buf = NULL;
+		crg_udc->status_req_callback_pending = false;
+		crg_udc_set_ep0_halt(crg_udc);
+		return;
+	}
+	list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+}
+
+int crg_udc_update_ep0_maxpacketsize(struct crg_gadget_dev *crg_udc)
+{
+	u16 maxpacketsize = 0;
+	struct crg_udc_ep *udc_ep0 = &crg_udc->udc_ep[0];
+	u32 param0;
+
+	if (crg_udc->gadget.speed >= USB_SPEED_SUPER)
+		maxpacketsize = 512;
+	else
+		maxpacketsize = 64;
+
+	param0 = FIELD_PREP(CRG_CMD1_0_MPS_MASK, maxpacketsize);
+	if (crg_udc_issue_command(crg_udc, CRG_CMD_UPDATE_EP0_CFG, param0, 0))
+		return -EIO;
+
+	crg_udc->ep0_desc.wMaxPacketSize = cpu_to_le16(maxpacketsize);
+	udc_ep0->usb_ep.maxpacket = maxpacketsize;
+
+	return 0;
+}
+
+void crg_udc_handle_setup_pkt(struct crg_gadget_dev *crg_udc,
+			      struct usb_ctrlrequest *setup_pkt, u8 setup_tag,
+			      unsigned long *flags)
+{
+	u16 w_value = le16_to_cpu(setup_pkt->wValue);
+	u16 w_index = le16_to_cpu(setup_pkt->wIndex);
+	u16 w_length = le16_to_cpu(setup_pkt->wLength);
+	struct usb_gadget_driver *driver, *setup_wait_driver;
+	u32 ep0_generation, event_generation;
+	u32 wait_ep0_generation, wait_event_generation;
+	u8 wait_setup_tag, active_setup_tag;
+	int setup_ret;
+	u32 tmp;
+
+	/* Guard EP0 across flush_work(), which temporarily drops udc_lock. */
+	if (!crg_udc_gadget_ready(crg_udc) ||
+	    crg_udc->udc_ep[0].ep_state != EP_STATE_RUNNING ||
+	    crg_udc->ep0_state != EP0_STATE_IDLE)
+		return;
+	wait_ep0_generation = crg_udc->udc_ep[0].generation;
+	wait_event_generation = crg_udc->udc_event[0].generation;
+	setup_wait_driver = crg_udc->gadget_driver;
+	wait_setup_tag = crg_udc->setup_tag;
+	crg_udc->ep0_state = EP0_STATE_ABORTING;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+	flush_work(&crg_udc->udc_reset_work);
+	spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+	/* Another SETUP or reset may have taken ownership of EP0. */
+	if (wait_ep0_generation != crg_udc->udc_ep[0].generation ||
+	    wait_event_generation != crg_udc->udc_event[0].generation ||
+	    wait_setup_tag != crg_udc->setup_tag ||
+	    crg_udc->ep0_state != EP0_STATE_ABORTING)
+		return;
+
+	if (!crg_udc_gadget_ready(crg_udc) ||
+	    crg_udc->device_state == USB_STATE_RECONNECTING ||
+	    crg_udc->bus_reset_pending || crg_udc->port_reset_pending ||
+	    crg_udc->udc_ep[0].ep_state != EP_STATE_RUNNING ||
+	    crg_udc->gadget_driver != setup_wait_driver) {
+		/*
+		 * Leave EP0 blocked: reset/reinit or a new SETUP must reclaim it.
+		 * Returning to IDLE here could allow queues during teardown.
+		 */
+		return;
+	}
+	crg_udc->ep0_state = EP0_STATE_IDLE;
+
+	trace_crg_udc_ctrl_req(setup_pkt);
+
+	/* New SETUP supersedes any mapping of the previous transaction. */
+	crg_udc->udc_ep[0].generation++;
+	/* EP0 come backs to running when new setup packet comes */
+	crg_udc->udc_ep[0].ep_state = EP_STATE_RUNNING;
+
+	crg_udc->setup_tag = setup_tag;
+	crg_udc->setup_status = SETUP_PKT_PROCESS_IN_PROGRESS;
+	crg_udc->setup_fn_call_back = NULL;
+
+	if ((setup_pkt->bRequestType & USB_TYPE_MASK) == USB_TYPE_STANDARD) {
+		switch (setup_pkt->bRequest) {
+		case USB_REQ_GET_STATUS:
+			if (setup_pkt->bRequestType !=
+				    (USB_DIR_IN | USB_RECIP_DEVICE |
+				     USB_TYPE_STANDARD) &&
+			    setup_pkt->bRequestType !=
+				    (USB_DIR_IN | USB_RECIP_INTERFACE |
+				     USB_TYPE_STANDARD) &&
+			    setup_pkt->bRequestType !=
+				    (USB_DIR_IN | USB_RECIP_ENDPOINT |
+				     USB_TYPE_STANDARD)) {
+				crg_udc_set_ep0_halt(crg_udc);
+				return;
+			}
+
+			getstatusrequest(crg_udc, setup_pkt->bRequestType,
+					 w_value, w_index, w_length);
+			return;
+		case USB_REQ_SET_ADDRESS:
+			if (setup_pkt->bRequestType !=
+			    (USB_DIR_OUT | USB_RECIP_DEVICE |
+			     USB_TYPE_STANDARD)) {
+				crg_udc_set_ep0_halt(crg_udc);
+				return;
+			}
+
+			setaddressrequest(crg_udc, w_value, w_index, w_length);
+			return;
+		case USB_REQ_SET_SEL:
+			if (setup_pkt->bRequestType !=
+			    (USB_DIR_OUT | USB_RECIP_DEVICE |
+			     USB_TYPE_STANDARD)) {
+				crg_udc_set_ep0_halt(crg_udc);
+				return;
+			}
+			crg_udc->setup_status = DATA_STAGE_RECV;
+			setselrequest(crg_udc, w_value, w_index, w_length);
+			return;
+		case USB_REQ_SET_ISOCH_DELAY:
+			if (setup_pkt->bRequestType !=
+				    (USB_DIR_OUT | USB_RECIP_DEVICE |
+				     USB_TYPE_STANDARD) ||
+			    w_index || w_length ||
+			    w_value >
+				    FIELD_MAX(
+					    CRG_U3DC_PTMCTRL_ISOCH_DELAY_MASK)) {
+				crg_udc_set_ep0_halt(crg_udc);
+				return;
+			}
+
+			tmp = readl(&crg_udc->uccr->ptmctrl);
+			FIELD_MODIFY(CRG_U3DC_PTMCTRL_ISOCH_DELAY_MASK, &tmp,
+				     w_value);
+			writel(tmp, &crg_udc->uccr->ptmctrl);
+			crg_udc->setup_status = STATUS_STAGE_XFER;
+			crg_udc_submit_ep0_status(crg_udc, -EINPROGRESS, 0);
+			return;
+
+		case USB_REQ_CLEAR_FEATURE:
+		case USB_REQ_SET_FEATURE:
+			/* Need composite gadget driver
+			 * to process the function remote wakeup request
+			 */
+			if (setfeaturesrequest(crg_udc, setup_pkt->bRequestType,
+					       setup_pkt->bRequest, w_value,
+					       w_index, w_length, flags)) {
+				/* Get here if request has been processed.*/
+				return;
+			}
+			break;
+		case USB_REQ_SET_CONFIGURATION:
+			if (setup_pkt->bRequestType !=
+				    (USB_DIR_OUT | USB_RECIP_DEVICE |
+				     USB_TYPE_STANDARD) ||
+			    w_length || w_value > 0xff) {
+				crg_udc_set_ep0_halt(crg_udc);
+				return;
+			}
+			/* In theory we need to clear RUN bit before
+			 * status stage of deconfig request sent.
+			 * But seeing problem if we do it before all the
+			 * endpoints belonging to the configuration
+			 * disabled.
+			 */
+			if (crg_udc->device_state <= USB_STATE_DEFAULT) {
+				crg_udc_set_ep0_halt(crg_udc);
+				return;
+			}
+
+			break;
+		default:
+			dev_dbg(crg_udc->dev,
+				"USB_REQ default bRequest=%d, bRequestType=%d\n",
+				setup_pkt->bRequest, setup_pkt->bRequestType);
+		}
+	}
+
+	if (w_length) {
+		/* data phase from gadget like GET_CONFIGURATION
+		 * call the setup routine of gadget driver.
+		 * remember the request direction.
+		 */
+		crg_udc->setup_status = (setup_pkt->bRequestType & USB_DIR_IN) ?
+						DATA_STAGE_XFER :
+						DATA_STAGE_RECV;
+	}
+	/* The callback drops udc_lock; a newer SETUP or reset can supersede us. */
+	ep0_generation = crg_udc->udc_ep[0].generation;
+	event_generation = crg_udc->udc_event[0].generation;
+	active_setup_tag = crg_udc->setup_tag;
+	driver = crg_udc->async_callbacks ? crg_udc->gadget_driver : NULL;
+	crg_udc_callback_enter(crg_udc);
+	spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+
+	if (!driver || !READ_ONCE(crg_udc->async_callbacks) ||
+	    READ_ONCE(crg_udc->gadget_driver) != driver)
+		setup_ret = -ESHUTDOWN;
+	else
+		setup_ret = driver->setup(&crg_udc->gadget, setup_pkt);
+
+	spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+	crg_udc_callback_exit(crg_udc);
+	/* Never STALL or update device state for an obsolete SETUP. */
+	if (!crg_udc_gadget_ready(crg_udc) ||
+	    crg_udc->bus_reset_pending || crg_udc->port_reset_pending ||
+	    crg_udc->device_state == USB_STATE_RECONNECTING ||
+	    !driver || crg_udc->gadget_driver != driver ||
+	    crg_udc->udc_ep[0].generation != ep0_generation ||
+	    crg_udc->udc_event[0].generation != event_generation ||
+	    crg_udc->setup_tag != active_setup_tag ||
+	    crg_udc->ep0_state != EP0_STATE_IDLE ||
+	    crg_udc->udc_ep[0].ep_state != EP_STATE_RUNNING)
+		return;
+
+	if (setup_ret < 0) {
+		crg_udc_set_ep0_halt(crg_udc);
+		return;
+	}
+
+	if (setup_pkt->bRequest == USB_REQ_SET_CONFIGURATION &&
+	    (setup_pkt->bRequestType & (USB_TYPE_MASK | USB_RECIP_MASK)) ==
+		    (USB_TYPE_STANDARD | USB_RECIP_DEVICE)) {
+		if (w_value) {
+			crg_udc->device_state = USB_STATE_CONFIGURED;
+			usb_gadget_set_state(&crg_udc->gadget,
+					     USB_STATE_CONFIGURED);
+		} else {
+			/* Endpoint disable may already have updated device_state. */
+			crg_udc->device_state = USB_STATE_ADDRESS;
+			usb_gadget_set_state(&crg_udc->gadget,
+					     USB_STATE_ADDRESS);
+		}
+	}
+}
+
+int crg_udc_init_ep0(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+	u32 cmd_param0;
+	u32 cmd_param1;
+
+	if (!udc_ep_ptr->tran_ring_info.cpu_addr)
+		return -ENOMEM;
+
+	udc_ep_ptr->generation++;
+	udc_ep_ptr->first_trb = udc_ep_ptr->tran_ring_info.cpu_addr;
+	udc_ep_ptr->last_trb = udc_ep_ptr->first_trb +
+			       CRG_UDC_CONTROL_EP_TD_RING_SIZE - 1;
+	memset(udc_ep_ptr->first_trb, 0, udc_ep_ptr->tran_ring_info.len);
+	/* Ensure that transfer ring is updated */
+	dma_wmb();
+	udc_ep_ptr->enq_pt = udc_ep_ptr->first_trb;
+	udc_ep_ptr->deq_pt = udc_ep_ptr->first_trb;
+	udc_ep_ptr->pcs = 1;
+	udc_ep_ptr->tran_ring_full = false;
+
+	crg_udc_setup_link_trb(udc_ep_ptr->last_trb, true,
+			       udc_ep_ptr->tran_ring_info.dma_addr);
+
+	/* context related ops */
+	cmd_param0 = (lower_32_bits(udc_ep_ptr->tran_ring_info.dma_addr) &
+		      CRG_CMD0_0_DQPTRLO_MASK) |
+		     FIELD_PREP(CRG_CMD0_0_DCS_MASK, udc_ep_ptr->pcs);
+	cmd_param1 = upper_32_bits(udc_ep_ptr->tran_ring_info.dma_addr);
+
+	if (crg_udc_issue_command(crg_udc, CRG_CMD_INIT_EP0, cmd_param0,
+				  cmd_param1))
+		return -EIO;
+
+	udc_ep_ptr->desc = &crg_udc->ep0_desc;
+	udc_ep_ptr->ep_state = EP_STATE_RUNNING;
+	udc_ep_ptr->pcs = 1;
+
+	return 0;
+}
+
+int crg_udc_queue_ctrl(struct crg_udc_ep *udc_ep_ptr,
+		       struct crg_udc_request *udc_req_ptr,
+		       u32 num_of_trbs_needed)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+	enum crg_ep_state ep_state;
+	struct crg_transfer_trb *enq_pt = udc_ep_ptr->enq_pt;
+	struct crg_transfer_trb *dq_pt = udc_ep_ptr->deq_pt;
+	struct usb_request *usb_req = &udc_req_ptr->usb_req;
+	struct crg_transfer_trb *p_trb;
+	u32 transfer_length;
+	bool ioc;
+	bool azp;
+	u8 dir = 0;
+	u8 setup_tag = crg_udc->setup_tag;
+
+	/* A new SETUP aborts the previous EP0 transaction. The completion
+	 * callback of the aborted request is allowed to run, but it must not
+	 * requeue the old request before the new SETUP has been handed to the
+	 * gadget driver.
+	 */
+	if (crg_udc->ep0_state == EP0_STATE_ABORTING)
+		return -ECONNRESET;
+
+	ep_state = crg_udc_get_ep_state(crg_udc, 0);
+	/* Need to queue the request even ep is paused or halted */
+	if (ep_state != EP_STATE_RUNNING) {
+		dev_dbg(crg_udc->dev, "EP State = 0x%x\n", ep_state);
+		return -EINVAL;
+	}
+	if (list_empty(&udc_ep_ptr->queue)) {
+		/* For control endpoint, we can handle one setup request at a
+		 * time. so if there are TD pending in the transfer ring.
+		 * wait for the sequence number error event. Then put the new
+		 * request to transfer ring
+		 */
+		if (enq_pt == dq_pt && !udc_ep_ptr->tran_ring_full) {
+			u32 i, td_done = 0;
+			bool need_zlp = false;
+			struct crg_udc_bounce *bounce = udc_req_ptr->bounce;
+			dma_addr_t trb_dma = bounce ? bounce->dma : 0;
+			u64 remaining = usb_req->length;
+			/* Transfer ring is empty
+			 * setup data stage TRBs
+			 */
+			udc_req_ptr->first_trb = udc_ep_ptr->enq_pt;
+			if (crg_udc->setup_status == DATA_STAGE_XFER)
+				dir = 0;
+			else if (crg_udc->setup_status == DATA_STAGE_RECV)
+				dir = 1;
+			else
+				dev_dbg(crg_udc->dev,
+					"unexpected setup_status!%d\n",
+					crg_udc->setup_status);
+			if (!udc_ep_ptr->usb_ep.maxpacket)
+				return -EINVAL;
+			if (udc_req_ptr->usb_req.zero == 1 &&
+			    udc_req_ptr->usb_req.length != 0 &&
+			    ((udc_req_ptr->usb_req.length %
+			      udc_ep_ptr->usb_ep.maxpacket) == 0))
+				need_zlp = true; /* zero length packet */
+			if (num_of_trbs_needed > 32)
+				return -EMSGSIZE;
+
+			if (num_of_trbs_needed >
+			    crg_udc_room_on_ring(
+				    crg_udc, CRG_UDC_CONTROL_EP_TD_RING_SIZE,
+				    udc_ep_ptr->first_trb, enq_pt, dq_pt))
+				return -ENOSPC;
+
+			u8 original_pcs = udc_ep_ptr->pcs;
+
+			for (i = 0; i < num_of_trbs_needed; i++) {
+				p_trb = enq_pt;
+				transfer_length = crg_udc_trb_chunk_len(
+					trb_dma, remaining);
+				ioc = (i == num_of_trbs_needed - 1);
+				azp = ioc && need_zlp;
+				td_done += transfer_length;
+				crg_udc_setup_datastage_trb(
+					p_trb, trb_dma,
+					i == 0 ? (original_pcs ^ 1) :
+						 udc_ep_ptr->pcs,
+					transfer_length,
+					crg_udc_td_size(usb_req->length, td_done,
+						udc_ep_ptr->usb_ep.maxpacket, ioc),
+					ioc,
+					!ioc, azp, dir, setup_tag);
+				udc_req_ptr->all_trbs_queued = 1;
+				trb_dma += transfer_length;
+				remaining -= transfer_length;
+				enq_pt++;
+				if (FIELD_GET(TRB_TYPE_MASK,
+					      le32_to_cpu(enq_pt->dw3)) ==
+				    TRB_TYPE_LINK) {
+					if (FIELD_GET(
+						    TRB_LINK_TOGGLE_CYCLE_MASK,
+						    le32_to_cpu(enq_pt->dw3))) {
+						dma_wmb();
+						le32p_replace_bits(
+							&enq_pt->dw3,
+							udc_ep_ptr->pcs,
+							TRB_CYCLE_BIT_MASK);
+						udc_ep_ptr->pcs ^= 0x1;
+					}
+					enq_pt = udc_ep_ptr->first_trb;
+				}
+			}
+			dma_wmb();
+			le32p_replace_bits(&udc_req_ptr->first_trb->dw3,
+					   original_pcs, TRB_CYCLE_BIT_MASK);
+			udc_ep_ptr->enq_pt = enq_pt;
+			crg_udc_knock_doorbell(crg_udc, 0);
+			if (udc_ep_ptr->enq_pt == udc_ep_ptr->first_trb)
+				udc_req_ptr->last_trb =
+					udc_ep_ptr->last_trb - 1;
+			else
+				udc_req_ptr->last_trb = udc_ep_ptr->enq_pt - 1;
+		} else {
+			/* The SETUP handler normally flushes stale control TRBs.
+			 * Do not accept an unscheduled request if EP0 is still busy:
+			 * SETUP_TAG_MISMATCH events are intentionally ignored.
+			 */
+			dev_err(crg_udc->dev,
+				"ep0 ring not idle on new setup\n");
+			return -EBUSY;
+		}
+	} else {
+		dev_err(crg_udc->dev, "udc_ep_ptr->queue not empty\n");
+		/* New setup packet came
+		 * Drop this request.
+		 */
+		return -EINVAL;
+	}
+	return 0;
+}

-- 
2.34.1


^ permalink raw reply related	[flat|nested] 13+ messages in thread

* [PATCH v5 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations
  2026-11-16 19:06 [PATCH v5 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
                   ` (3 preceding siblings ...)
  2026-11-16 19:06 ` [PATCH v5 4/6] usb: gadget: udc: crg: Add endpoint 0 handling Petar Stepanovic
@ 2026-11-16 19:06 ` Petar Stepanovic
  2026-11-16 20:47   ` sashiko-bot
  2026-11-16 19:06 ` [PATCH v5 6/6] usb: gadget: udc: crg: Add core driver and build integration Petar Stepanovic
  5 siblings, 1 reply; 13+ messages in thread
From: Petar Stepanovic @ 2026-11-16 19:06 UTC (permalink / raw)
  To: Prasad Bolisetty, Greg Kroah-Hartman, Rob Herring,
	Krzysztof Kozlowski, Conor Dooley, Tzu-Hao Wei,
	Karthikeyan Mitran, Philipp Zabel, Neill Kapron
  Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
	Petar Stepanovic, Krutik Shah, Naveen Kumar Rajgiri Bassappa

This patch implements the gadget and endpoint operations for the
Axiado AX3000/AX3005 UDC driver.

Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Tzu-Hao Wei <twei@axiado.com>
Signed-off-by: Tzu-Hao Wei <twei@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
 drivers/usb/gadget/udc/crg/crg-udc-gadget.c | 3255 +++++++++++++++++++++++++++
 drivers/usb/gadget/udc/crg/crg-udc-trace.c  |    9 +
 2 files changed, 3264 insertions(+)

diff --git a/drivers/usb/gadget/udc/crg/crg-udc-gadget.c b/drivers/usb/gadget/udc/crg/crg-udc-gadget.c
new file mode 100644
index 000000000000..dddfc5cf3d3e
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-gadget.c
@@ -0,0 +1,3255 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/gfp.h>
+#include <linux/highmem.h>
+#include <linux/interrupt.h>
+#include <linux/iopoll.h>
+#include <linux/jiffies.h>
+#include <linux/slab.h>
+
+#include "crg-udc.h"
+#include "crg-udc-trace.h"
+
+/* Forward declarations for functions used before their definition. */
+static int crg_udc_epcx_setup(struct crg_udc_ep *udc_ep);
+static int crg_udc_epcx_update_dqptr(struct crg_udc_ep *udc_ep);
+static int crg_udc_queue_trbs(struct crg_udc_ep *udc_ep_ptr,
+			      struct crg_udc_request *udc_req_ptr, bool b_isoc,
+			      u32 xfer_ring_size, u32 num_trbs_needed,
+			      u64 buffer_length);
+static int crg_udc_reinit(struct crg_gadget_dev *crg_udc, bool notify_reset);
+static bool crg_udc_sg_advance(struct crg_udc_request *udc_req, u32 bytes);
+static bool crg_udc_sg_current(struct crg_udc_request *udc_req, dma_addr_t *dma,
+			       u32 *len);
+static int crg_udc_sg_init(struct crg_udc_request *udc_req);
+static int crg_udc_validate_ep_desc(struct crg_udc_ep *udc_ep,
+				    const struct usb_endpoint_descriptor *desc);
+static bool is_request_dequeued(struct crg_gadget_dev *crg_udc,
+				struct crg_udc_ep *udc_ep,
+				struct crg_event_trb *event);
+static int crg_udc_stop_error_ep(struct crg_gadget_dev *crg_udc, u8 dci);
+
+static int crg_gadget_get_frame(struct usb_gadget *g)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+	unsigned long flags;
+	int frame = -ESHUTDOWN;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	if (!crg_udc->gadget_stopping && !crg_udc->shutting_down &&
+	    !crg_udc->resetting && !crg_udc->controller_recovering &&
+	    !crg_udc->suspended)
+		frame = FIELD_GET(CRG_U3DC_MFINDEX_FRAME_MASK,
+				  readl(&crg_udc->uccr->mfindex));
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	return frame;
+}
+
+static int crg_gadget_pullup(struct usb_gadget *g, int is_on)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+	unsigned long flags;
+	int ret = 0;
+
+	/*
+	 * The core can call ->disconnect() as soon as pullup(0) returns.
+	 * Return the requests even if hardware stop required quarantine.
+	 */
+	if (!is_on) {
+		ret = crg_udc_stop_gadget(crg_udc);
+		if (!ret)
+			trace_crg_udc_pullup(false);
+		return ret;
+	}
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	if (crg_udc->shutting_down || crg_udc->controller_recovering ||
+	    (!crg_udc->gadget_stopping &&
+	     (crg_udc->resetting || crg_udc->controller_faulted))) {
+		ret = -ESHUTDOWN;
+		goto out;
+	}
+	if (crg_udc->gadget_stopping) {
+		ret = crg_udc_restart_gadget(crg_udc, &flags);
+		if (ret)
+			goto out;
+	} else if (crg_udc->stopped_needs_reset) {
+		ret = crg_udc_recover(crg_udc, &flags);
+		if (ret)
+			goto out;
+	}
+
+	crg_udc->pullup_on = true;
+	if (!crg_udc->suspended)
+		crg_udc_set_pullup(crg_udc, true);
+out:
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	if (!ret)
+		trace_crg_udc_pullup(true);
+	return ret;
+}
+
+static int crg_gadget_set_remote_wakeup(struct usb_gadget *g, int set)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+	unsigned long flags;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	crg_udc->u2_rwe = !!set;
+	g->wakeup_armed = !!set;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	return 0;
+}
+
+static int crg_gadget_set_selfpowered(struct usb_gadget *g, int set)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+	unsigned long flags;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	g->is_selfpowered = !!set;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	return 0;
+}
+
+static void crg_udc_set_speed(struct usb_gadget *gadget,
+			      enum usb_device_speed speed)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(gadget);
+
+	switch (speed) {
+	case USB_SPEED_FULL:
+	case USB_SPEED_HIGH:
+	case USB_SPEED_SUPER:
+		break;
+	case USB_SPEED_LOW:
+		speed = USB_SPEED_FULL;
+		break;
+	default:
+		speed = gadget->max_speed;
+		break;
+	}
+	crg_udc->max_speed = min(speed, gadget->max_speed);
+}
+
+static int crg_gadget_start(struct usb_gadget *g,
+			    struct usb_gadget_driver *driver)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+	unsigned long flags;
+	int ret;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	if (crg_udc->shutting_down || crg_udc->controller_recovering ||
+	    (!crg_udc->gadget_stopping &&
+	     (crg_udc->resetting || crg_udc->controller_faulted))) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EBUSY;
+	}
+	if (crg_udc->gadget_stopping) {
+		ret = crg_udc_restart_gadget(crg_udc, &flags);
+	} else {
+		ret = crg_udc_program_speed(crg_udc);
+		if (!ret)
+			crg_udc_set_vbus_detect(crg_udc, true);
+	}
+	if (!ret)
+		crg_udc->gadget_driver = driver;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	return ret;
+}
+
+static int crg_gadget_stop(struct usb_gadget *g)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+	unsigned long flags;
+	int i, ret;
+
+	ret = crg_udc_stop_gadget(crg_udc);
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	crg_udc->gadget_driver = NULL;
+	for (i = 2; i < CRG_NUM_EP_CX; i++)
+		crg_udc_ep_clear_enabled(&crg_udc->udc_ep[i]);
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	return ret;
+}
+
+static int crg_gadget_wakeup(struct usb_gadget *g)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	unsigned long flags;
+	u32 portsc;
+	int ret = 0;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	if (crg_udc->resetting) {
+		ret = -ESHUTDOWN;
+		goto out;
+	}
+	if (!crg_udc->connected || g->speed > USB_SPEED_HIGH) {
+		ret = -EOPNOTSUPP;
+		goto out;
+	}
+	if (!crg_udc->u2_rwe) {
+		ret = -EACCES;
+		goto out;
+	}
+
+	portsc = readl(&uccr->portsc);
+	if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, portsc) !=
+	    CRG_U3DC_PORTSC_PLS_U3) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	portsc &= ~(CRG_U3DC_PORTSC_W1C_MASK | CRG_U3DC_PORTSC_PLS_MASK);
+	portsc |= CRG_U3DC_PORTSC_LWS;
+	writel(portsc, &uccr->portsc);
+out:
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	return ret;
+}
+
+static void crg_gadget_async_callbacks(struct usb_gadget *g, bool enable)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+	unsigned long flags;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	crg_udc->async_callbacks = enable;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	/* Also cover unbind paths that skip pullup(0). */
+	if (!enable)
+		crg_udc_stop_gadget(crg_udc);
+}
+
+struct usb_request *crg_udc_alloc_request(struct usb_ep *_ep, gfp_t gfp_flags)
+{
+	struct crg_udc_request *udc_req_ptr;
+
+	udc_req_ptr = kzalloc_obj(*udc_req_ptr, gfp_flags);
+
+	if (!udc_req_ptr)
+		return NULL;
+
+	udc_req_ptr->usb_req.dma = DMA_MAPPING_ERROR;
+	INIT_LIST_HEAD(&udc_req_ptr->queue);
+
+	trace_crg_udc_alloc_request(
+		container_of(_ep, struct crg_udc_ep, usb_ep), udc_req_ptr);
+
+	return &udc_req_ptr->usb_req;
+}
+
+int crg_udc_build_td(struct crg_udc_ep *udc_ep_ptr,
+		     struct crg_udc_request *udc_req_ptr)
+{
+	int status = 0;
+	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+	u32 num_trbs_needed;
+	u64 buffer_length;
+
+	if (udc_req_ptr->trbs_needed) {
+		/* If this is called to complete pending TRB transfers
+		 * of previous Request
+		 */
+		buffer_length = udc_req_ptr->buff_len_left;
+		num_trbs_needed = udc_req_ptr->trbs_needed;
+	} else {
+		buffer_length = udc_req_ptr->usb_req.length;
+		if (udc_req_ptr->bounce && udc_req_ptr->bounce->sgt.nents)
+			num_trbs_needed = crg_udc_count_trbs_sg(udc_req_ptr);
+		else
+			num_trbs_needed = crg_udc_count_trbs_linear(
+				udc_req_ptr->bounce ? udc_req_ptr->bounce->dma : 0,
+				buffer_length);
+		if (!num_trbs_needed && buffer_length)
+			return -EINVAL;
+		if (!num_trbs_needed)
+			num_trbs_needed = 1;
+	}
+
+	if (usb_endpoint_xfer_control(udc_ep_ptr->desc))
+		status = crg_udc_queue_ctrl(udc_ep_ptr, udc_req_ptr,
+					    num_trbs_needed);
+	else if (usb_endpoint_xfer_isoc(udc_ep_ptr->desc))
+		status = crg_udc_queue_trbs(udc_ep_ptr, udc_req_ptr, true,
+					    CRG_UDC_ISOC_EP_TD_RING_SIZE,
+					    num_trbs_needed, buffer_length);
+	else if (usb_endpoint_xfer_bulk(udc_ep_ptr->desc))
+		status = crg_udc_queue_trbs(udc_ep_ptr, udc_req_ptr, false,
+					    CRG_UDC_BULK_EP_TD_RING_SIZE,
+					    num_trbs_needed, buffer_length);
+	else
+		/* interrupt endpoint */
+		status = crg_udc_queue_trbs(udc_ep_ptr, udc_req_ptr, false,
+					    CRG_UDC_INT_EP_TD_RING_SIZE,
+					    num_trbs_needed, buffer_length);
+
+	if (!status) {
+		udc_req_ptr->td_pending = true;
+		if (!usb_endpoint_xfer_control(udc_ep_ptr->desc))
+			crg_udc_knock_doorbell(crg_udc, udc_ep_ptr->dci);
+	}
+
+	return status;
+}
+
+void crg_udc_clear_portpm(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 tmp;
+
+	tmp = readl(&uccr->u3portpmsc);
+
+	tmp &= (~CRG_U3DC_U3PORTPM_U1IEN);
+	FIELD_MODIFY(CRG_U3DC_U3PORTPM_U1TMOUT_MASK, &tmp, 0);
+	tmp &= (~CRG_U3DC_U3PORTPM_U2IEN);
+	FIELD_MODIFY(CRG_U3DC_U3PORTPM_U2TMOUT_MASK, &tmp, 0);
+
+	writel(tmp, &uccr->u3portpmsc);
+
+	crg_udc->feature_u1_enable = 0;
+	crg_udc->feature_u2_enable = 0;
+}
+
+/* HALTED endpoints may hold requests for CLEAR_FEATURE(ENDPOINT_HALT).
+ * STOPPED endpoints may accept a requeue only while ep_dequeue() has
+ * confirmed STOP_EP and is giving a cancelled request back.
+ * Neither state may publish new TRBs until RUNNING again.
+ * Called with udc_lock held.
+ */
+static bool crg_udc_ep_can_queue(struct crg_udc_ep *ep)
+{
+	if (ep->nuking)
+		return false;
+
+	if (ep->ep_state == EP_STATE_RUNNING)
+		return true;
+
+	if (ep->dci == 0)
+		return false;
+
+	if (ep->ep_state == EP_STATE_HALTED)
+		return true;
+
+	return ep->ep_state == EP_STATE_STOPPED && ep->dequeue_paused;
+}
+
+static int crg_udc_ep_dequeue(struct usb_ep *_ep, struct usb_request *_req)
+{
+	struct crg_udc_ep *udc_ep_ptr;
+	struct crg_gadget_dev *crg_udc;
+	struct crg_uccr __iomem *uccr;
+	struct crg_udc_request *udc_req = NULL;
+	struct crg_udc_request *iter;
+	struct crg_ep_cx *p_ep_cx;
+	struct crg_transfer_trb *pause_pt;
+	enum crg_ep_state old_ep_state;
+	u32 tmp, deq_pt_lo, deq_pt_hi;
+	dma_addr_t dq_pt_addr;
+	u8 dci;
+	u32 generation;
+	unsigned long flags;
+	const struct usb_endpoint_descriptor *old_desc;
+	u32 event_generation;
+	int ret;
+
+	if (!_ep || !_req)
+		return -EINVAL;
+
+	udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+	crg_udc = udc_ep_ptr->crg_udc;
+	uccr = crg_udc->uccr;
+	dci = udc_ep_ptr->dci;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	list_for_each_entry(iter, &udc_ep_ptr->queue, queue) {
+		if (&iter->usb_req == _req) {
+			udc_req = iter;
+			break;
+		}
+	}
+
+	if (!udc_req) {
+		ret = -EINVAL;
+		goto out;
+	}
+	if (crg_udc->shutting_down) {
+		ret = -ESHUTDOWN;
+		goto out;
+	}
+	if (crg_udc->resetting || crg_udc->controller_recovering ||
+	    crg_udc->gadget_stopping || crg_udc->endpoint_recovery_pending ||
+	    udc_ep_ptr->nuking || udc_ep_ptr->ep_state == EP_STATE_DISABLED) {
+		/* The pending teardown will give back this queued request. */
+		ret = 0;
+		goto out;
+	}
+	if (dci == 0) {
+		/* The abort schedules recovery on hardware failure. */
+		if (crg_udc->port_reset_pending || crg_udc->bus_reset_pending ||
+		    crg_udc_abort_ep0_for_setup(crg_udc) == -ESHUTDOWN)
+			crg_udc_schedule_endpoint_recovery(crg_udc);
+		ret = 0;
+		goto out;
+	}
+	if (crg_udc->port_reset_pending || crg_udc->bus_reset_pending) {
+		ret = 0;
+		goto out;
+	}
+	if (udc_ep_ptr->halt_failed) {
+		ret = -EIO;
+		goto recover;
+	}
+
+	old_ep_state = udc_ep_ptr->ep_state;
+	old_desc = udc_ep_ptr->desc;
+	event_generation = crg_udc->udc_event[0].generation;
+	generation = udc_ep_ptr->generation;
+
+	if (!udc_req->first_trb) {
+		/* This request was queued during HALT / STOP; no live TRB. */
+		crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return 0;
+	}
+
+	/*
+	 * RUN=0 quiesces endpoint DMA but prevents STOP_EP / SET_TR_DQPTR.
+	 * Give the request back without touching the stopped hardware ring.
+	 * Force a full controller rebuild before RUN can be asserted again.
+	 */
+	if (!(readl(&uccr->control) & CRG_U3DC_CTRL_RUN)) {
+		ret = readl_poll_timeout_atomic(&uccr->ep_running, tmp,
+					      !(tmp & BIT(dci)), 10, 10000);
+		if (ret)
+			goto recover;
+		crg_udc->stopped_needs_reset = true;
+		crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return 0;
+	}
+
+	/* A stopped ring may still contain published requests. */
+	if (old_ep_state == EP_STATE_STOPPED) {
+		ret = -EBUSY;
+		goto recover;
+	}
+
+	if (old_ep_state == EP_STATE_RUNNING) {
+		ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP,
+						BIT(dci), 0);
+
+		if (ret)
+			goto recover;
+		ret = readl_poll_timeout_atomic(&uccr->ep_running, tmp,
+					       !(tmp & BIT(dci)), 10, 10000);
+		if (ret)
+			goto recover;
+		udc_ep_ptr->ep_state = EP_STATE_STOPPED;
+		udc_ep_ptr->dequeue_paused = true;
+	}
+
+	p_ep_cx = &crg_udc->p_epcx[dci - 2];
+	deq_pt_lo = le32_to_cpu(p_ep_cx->dw2) & EP_CX_TR_DQPT_LO_MASK;
+	deq_pt_hi = le32_to_cpu(p_ep_cx->dw3);
+	dq_pt_addr = (u64)deq_pt_lo | ((u64)deq_pt_hi << 32);
+	pause_pt = crg_udc_tran_trb_dma_to_virt(udc_ep_ptr, dq_pt_addr);
+	if (!pause_pt) {
+		ret = -EIO;
+		goto recover;
+	}
+
+	if (crg_udc_is_pointer_less_than(pause_pt, udc_req->first_trb,
+					 udc_ep_ptr)) {
+		crg_udc_squeeze_xfer_ring(udc_ep_ptr, udc_req);
+		/* Give back the cancelled request BEFORE scheduling the tail:
+		 * its callback may add/remove requests, and dropping udc_lock
+		 * while traversing the old list is unsafe.
+		 */
+		crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+	} else if (udc_req->last_trb &&
+		   crg_udc_is_pointer_less_than(udc_req->last_trb, pause_pt,
+						udc_ep_ptr)) {
+		crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+	} else {
+		struct crg_udc_request *next_req = NULL;
+
+		udc_req->usb_req.actual =
+			crg_udc_actual_before(udc_ep_ptr, udc_req, pause_pt);
+
+		if (!list_is_last(&udc_req->queue, &udc_ep_ptr->queue))
+			next_req = list_entry(udc_req->queue.next,
+					      struct crg_udc_request, queue);
+
+		if (next_req && next_req->first_trb)
+			udc_ep_ptr->deq_pt = next_req->first_trb;
+		else
+			udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
+
+		ret = crg_udc_epcx_update_dqptr(udc_ep_ptr);
+		if (ret)
+			goto recover;
+		udc_ep_ptr->tran_ring_full = false;
+		crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+	}
+
+	/* Stop accepting deferred queues before resuming the endpoint. */
+	if (udc_ep_ptr->generation == generation &&
+	    udc_ep_ptr->desc == old_desc)
+		udc_ep_ptr->dequeue_paused = false;
+
+	if (old_ep_state == EP_STATE_RUNNING && !crg_udc->resetting &&
+	    !crg_udc->port_reset_pending &&
+	    !crg_udc->endpoint_recovery_pending &&
+	    !udc_ep_ptr->halt_failed &&
+	    udc_ep_ptr->desc == old_desc &&
+	    event_generation == crg_udc->udc_event[0].generation &&
+	    udc_ep_ptr->generation == generation &&
+	    udc_ep_ptr->ep_state == EP_STATE_STOPPED) {
+		udc_ep_ptr->ep_state = EP_STATE_RUNNING;
+		udc_ep_ptr->tran_ring_full = false;
+		if (crg_udc_queue_pending_trbs(udc_ep_ptr))
+			udc_ep_ptr->tran_ring_full = true;
+		/* Queue processing can give back requests and drop udc_lock. */
+		if (!crg_udc->resetting && !crg_udc->port_reset_pending &&
+		    !crg_udc->endpoint_recovery_pending &&
+		    !udc_ep_ptr->halt_failed && udc_ep_ptr->desc == old_desc &&
+		    udc_ep_ptr->generation == generation &&
+		    udc_ep_ptr->ep_state == EP_STATE_RUNNING)
+			crg_udc_knock_doorbell(crg_udc, dci);
+	}
+
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	return 0;
+
+recover:
+	/*
+	 * A gadget may hold its own lock across dequeue. Leave requests
+	 * mapped and let the worker reset the controller and give them back.
+	 */
+	dev_err(crg_udc->dev, "EP%d dequeue failed (%d); scheduling recovery\n",
+		dci, ret);
+	udc_ep_ptr->dequeue_paused = false;
+	udc_ep_ptr->halt_failed = true;
+	udc_ep_ptr->ep_state = EP_STATE_STOPPED;
+	crg_udc_schedule_endpoint_recovery(crg_udc);
+	ret = 0;
+out:
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	return ret;
+}
+
+/* endpoint operations */
+
+static int crg_udc_ep_disable(struct usb_ep *ep)
+{
+	struct crg_udc_ep *udc_ep;
+	struct crg_gadget_dev *crg_udc;
+	struct crg_ep_cx *p_ep_cx;
+	struct crg_uccr __iomem *uccr;
+	unsigned long flags;
+	u32 running;
+	int ret;
+
+	if (!ep)
+		return -EINVAL;
+	udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+	crg_udc = udc_ep->crg_udc;
+	if (udc_ep->dci == 0)
+		return 0;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	/* A completion callback must not restart disable during giveback. */
+	if (udc_ep->nuking) {
+		ret = -EBUSY;
+		goto out;
+	}
+	if (udc_ep->configuring) {
+		ret = -EBUSY;
+		goto out;
+	}
+	if (udc_ep->ep_state == EP_STATE_DISABLED) {
+		/*
+		 * A reset can mark this endpoint disabled before reaching its
+		 * queue. A callback from another endpoint may disable it now.
+		 * Give back its requests before returning success.
+		 */
+		crg_udc_ep_nuke(udc_ep, -ESHUTDOWN);
+		udc_ep->desc = NULL;
+		udc_ep->comp_desc = NULL;
+		udc_ep->usb_ep.enabled = false;
+		ret = 0;
+		goto out;
+	}
+	if (crg_udc->resetting || crg_udc->controller_recovering) {
+		ret = -ESHUTDOWN;
+		goto out;
+	}
+
+	uccr = crg_udc->uccr;
+	p_ep_cx = &crg_udc->p_epcx[udc_ep->dci - 2];
+	/* The disable attempt owns this endpoint until DMA is quiescent. */
+	udc_ep->ep_state = EP_STATE_STOPPED;
+	udc_ep->dequeue_paused = false;
+	writel(BIT(udc_ep->dci), &uccr->ep_enable);
+	/*
+	 * Wait once for each hardware state, with a bounded timeout. DWC3's
+	 * delayed-stop retry uses a controller-specific SETUP event which CRG
+	 * does not currently have. An asynchronous retry must not outlive
+	 * usb_ep_disable() while Gadget owns the original request buffers.
+	 */
+	ret = readl_poll_timeout_atomic(&uccr->ep_enable, running,
+					!(running & BIT(udc_ep->dci)), 10, 10000);
+	if (ret)
+		goto disable_failed;
+	/* A cleared enable bit alone does not prove DMA is idle. */
+	ret = readl_poll_timeout_atomic(&uccr->ep_running, running,
+					!(running & BIT(udc_ep->dci)), 10, 10000);
+	if (ret)
+		goto disable_failed;
+
+	/* Endpoint DMA is stopped: mappings can now be released. */
+	udc_ep->disable_failed = false;
+	udc_ep->halt_failed = false;
+	udc_ep->dequeue_paused = false;
+	udc_ep->generation++;
+	udc_ep->desc = NULL;
+	udc_ep->comp_desc = NULL;
+	udc_ep->ep_state = EP_STATE_DISABLED;
+	udc_ep->tran_ring_full = false;
+	if (crg_udc->num_enabled_eps)
+		crg_udc->num_enabled_eps--;
+	memset(p_ep_cx, 0, sizeof(*p_ep_cx));
+
+	if (!crg_udc->num_enabled_eps &&
+	    crg_udc->device_state == USB_STATE_CONFIGURED)
+		crg_udc->device_state = USB_STATE_ADDRESS;
+
+	/* crg_udc_req_done() drops the lock during giveback. All endpoint
+	 * state has to be finalized before that can happen.
+	 */
+	crg_udc_ep_nuke(udc_ep, -ESHUTDOWN);
+	udc_ep->usb_ep.enabled = false;
+	ret = 0;
+	goto out;
+
+disable_failed:
+	/*
+	 * usb_ep_disable() may be called with IRQs disabled, and a gadget may
+	 * release its requests as soon as this call returns. A background
+	 * reset alone cannot make the outstanding DMA buffers safe to free.
+	 * Block the endpoint first, then quiesce the entire controller now.
+	 */
+	udc_ep->disable_failed = true;
+	udc_ep->halt_failed = true;
+	dev_err(crg_udc->dev,
+		"EP%d disable failed (%d); attempting controller reset\n",
+		udc_ep->dci, ret);
+
+	ret = crg_udc_recover_endpoints(crg_udc, &flags);
+	if (ret && udc_ep->ep_state == EP_STATE_DISABLED && !udc_ep->nuking &&
+	    list_empty(&udc_ep->queue))
+		udc_ep->usb_ep.enabled = false;
+
+out:
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	return ret;
+}
+
+static int crg_udc_ep_enable(struct usb_ep *ep,
+			     const struct usb_endpoint_descriptor *desc)
+{
+	struct crg_udc_ep *udc_ep;
+	struct crg_gadget_dev *crg_udc;
+	size_t len;
+	unsigned long flags;
+	u32 ring_size, param0;
+	int ret;
+
+	if (!ep || !desc || desc->bDescriptorType != USB_DT_ENDPOINT)
+		return -EINVAL;
+	udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+	if (udc_ep->dci == 0)
+		return 0;
+	crg_udc = udc_ep->crg_udc;
+
+	if (usb_endpoint_xfer_bulk(desc))
+		ring_size = CRG_UDC_BULK_EP_TD_RING_SIZE;
+	else if (usb_endpoint_xfer_isoc(desc))
+		ring_size = CRG_UDC_ISOC_EP_TD_RING_SIZE;
+	else if (usb_endpoint_xfer_int(desc))
+		ring_size = CRG_UDC_INT_EP_TD_RING_SIZE;
+	else
+		return -EINVAL;
+	len = ring_size * sizeof(struct crg_transfer_trb);
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	if (!crg_udc->gadget_driver || crg_udc->controller_faulted ||
+	    crg_udc->resetting || crg_udc->gadget_stopping ||
+	    crg_udc->controller_recovering ||
+	    crg_udc->endpoint_recovery_pending ||
+	    crg_udc->shutting_down || crg_udc->bus_reset_pending ||
+	    crg_udc->port_reset_pending) {
+		ret = -ESHUTDOWN;
+		goto out;
+	}
+	if (udc_ep->nuking || udc_ep->configuring ||
+	    udc_ep->disable_failed ||
+	    udc_ep->ep_state != EP_STATE_DISABLED) {
+		ret = -EBUSY;
+		goto out;
+	}
+	udc_ep->configuring = true;
+	udc_ep->desc = desc;
+	udc_ep->comp_desc = ep->comp_desc;
+	ret = crg_udc_validate_ep_desc(udc_ep, desc);
+	if (ret)
+		goto err;
+
+	/* Rings are allocated at probe, outside the atomic enable path. */
+	if (!udc_ep->tran_ring_info.cpu_addr ||
+	    udc_ep->tran_ring_info.len < len) {
+		ret = -ENOMEM;
+		goto err;
+	}
+
+	udc_ep->first_trb = udc_ep->tran_ring_info.cpu_addr;
+	udc_ep->last_trb = udc_ep->first_trb + ring_size - 1;
+	memset(udc_ep->first_trb, 0, len);
+	dma_wmb();
+	crg_udc_setup_link_trb(udc_ep->last_trb, true,
+				 udc_ep->tran_ring_info.dma_addr);
+	udc_ep->enq_pt = udc_ep->first_trb;
+	udc_ep->deq_pt = udc_ep->first_trb;
+	udc_ep->pcs = 1;
+	udc_ep->tran_ring_full = false;
+	ret = crg_udc_epcx_setup(udc_ep);
+	if (ret)
+		goto err;
+	param0 = BIT(udc_ep->dci);
+	ret = crg_udc_issue_command(crg_udc, CRG_CMD_CONFIG_EP, param0, 0);
+	if (ret)
+		goto err;
+
+	udc_ep->generation++;
+	/* usb_ep.maxpacket must reflect the descriptor actually configured. */
+	udc_ep->usb_ep.maxpacket = usb_endpoint_maxp(desc);
+	udc_ep->wedge = false;
+	udc_ep->halt_failed = false;
+	udc_ep->disable_failed = false;
+	udc_ep->dequeue_paused = false;
+	udc_ep->usb_ep.enabled = 1;
+	udc_ep->ep_state = EP_STATE_RUNNING;
+	udc_ep->configuring = false;
+	crg_udc->num_enabled_eps++;
+	if (crg_udc->device_state == USB_STATE_ADDRESS)
+		crg_udc->device_state = USB_STATE_CONFIGURED;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	return 0;
+
+err:
+	udc_ep->desc = NULL;
+	udc_ep->comp_desc = NULL;
+	udc_ep->configuring = false;
+	/* Keep the preallocated ring for a later retry. */
+	udc_ep->ep_state = EP_STATE_DISABLED;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	return ret;
+out:
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	return ret;
+}
+
+void crg_udc_ep_nuke(struct crg_udc_ep *udc_ep, int status)
+{
+	struct crg_udc_request *req;
+
+	/*
+	 * A giveback may re-enter ep_nuke() via disable or reset. The outer
+	 * drain still owns all originally queued requests, so do not nest it.
+	 * Also reject new queue/enable operations until the drain is finished.
+	 */
+	if (udc_ep->nuking)
+		return;
+
+	udc_ep->nuking = true;
+	while (!list_empty(&udc_ep->queue)) {
+		req = list_first_entry(&udc_ep->queue,
+				       struct crg_udc_request, queue);
+		crg_udc_req_done(udc_ep, req, status);
+	}
+	udc_ep->nuking = false;
+	if (udc_ep->clear_enabled_on_nuke) {
+		udc_ep->clear_enabled_on_nuke = false;
+		udc_ep->usb_ep.enabled = false;
+	}
+}
+
+/* The CPU page list stays intact even if dma_map_sgtable() merges entries. */
+static size_t crg_udc_copy_bounce_pages(struct crg_udc_bounce *bounce,
+				       struct usb_request *req, size_t len)
+{
+	bool to_bounce = bounce->dir == DMA_TO_DEVICE;
+	struct sg_mapping_iter miter;
+	struct page *page;
+	unsigned int sg_flags = to_bounce ? SG_MITER_FROM_SG : SG_MITER_TO_SG;
+	size_t done = 0;
+
+	if (req->num_sgs)
+		sg_miter_start(&miter, req->sg, req->num_sgs,
+			       SG_MITER_ATOMIC | sg_flags);
+
+	list_for_each_entry(page, &bounce->pages, lru) {
+		size_t count = min_t(size_t, PAGE_SIZE, len - done);
+		size_t offset = 0;
+		void *vaddr;
+
+		if (!count)
+			break;
+
+		vaddr = kmap_local_page(page);
+		if (req->num_sgs) {
+			while (offset < count && sg_miter_next(&miter)) {
+				size_t chunk = min(miter.length, count - offset);
+
+				if (to_bounce)
+					memcpy(vaddr + offset, miter.addr, chunk);
+				else
+					memcpy(miter.addr, vaddr + offset, chunk);
+				miter.consumed = chunk;
+				offset += chunk;
+			}
+			sg_miter_stop(&miter);
+		} else {
+			if (to_bounce)
+				memcpy(vaddr, req->buf + done, count);
+			else
+				memcpy(req->buf + done, vaddr, count);
+			offset = count;
+		}
+		kunmap_local(vaddr);
+
+		done += offset;
+		if (offset != count)
+			break;
+	}
+	return done;
+}
+
+static int crg_udc_bounce_map_pages(struct device *dev,
+				   struct crg_udc_bounce *bounce,
+				   struct usb_request *req, gfp_t gfp)
+{
+	unsigned int nr_pages = DIV_ROUND_UP_ULL(req->length, PAGE_SIZE);
+	size_t remaining = req->length;
+	struct scatterlist *sg;
+	unsigned int i;
+	int ret;
+
+	if (bounce->dir == DMA_FROM_DEVICE)
+		gfp |= __GFP_ZERO;
+	ret = sg_alloc_table(&bounce->sgt, nr_pages, gfp);
+	if (ret)
+		return ret;
+
+	for_each_sgtable_sg(&bounce->sgt, sg, i) {
+		size_t len = min_t(size_t, remaining, PAGE_SIZE);
+		struct page *page = alloc_page(gfp);
+
+		if (!page)
+			return -ENOMEM;
+
+		list_add_tail(&page->lru, &bounce->pages);
+		sg_set_page(sg, page, len, 0);
+		remaining -= len;
+	}
+
+	if (bounce->dir == DMA_TO_DEVICE &&
+	    crg_udc_copy_bounce_pages(bounce, req, req->length) != req->length)
+		return -EINVAL;
+
+	return dma_map_sgtable(dev, &bounce->sgt, bounce->dir, 0);
+}
+
+static int crg_udc_bounce_map_ep0(struct device *dev,
+				 struct crg_udc_bounce *bounce,
+				 struct usb_request *req, gfp_t gfp)
+{
+	bounce->buf = kzalloc(req->length, gfp);
+	if (!bounce->buf)
+		return -ENOMEM;
+
+	if (bounce->dir == DMA_TO_DEVICE)
+		memcpy(bounce->buf, req->buf, req->length);
+
+	bounce->dma = dma_map_single(dev, bounce->buf, bounce->len, bounce->dir);
+	return dma_mapping_error(dev, bounce->dma) ? -EIO : 0;
+}
+
+int crg_udc_bounce_map_request(struct crg_udc_ep *udc_ep,
+			       struct crg_udc_request *udc_req,
+			       enum dma_data_direction dir, gfp_t gfp)
+{
+	struct device *dev = udc_ep->crg_udc->dev;
+	struct usb_request *req = &udc_req->usb_req;
+	struct crg_udc_bounce *bounce;
+	int nents, ret;
+
+	if (!req->length)
+		return 0;
+	if (udc_req->bounce)
+		return -EBUSY;
+
+	/* CPU copies require buffers the UDC is allowed to access. */
+	if (req->sg_was_mapped || req->dma_mapped)
+		return -EOPNOTSUPP;
+	if (req->num_sgs) {
+		if (!req->sg)
+			return -EINVAL;
+		nents = sg_nents_for_len(req->sg, req->length);
+		if (nents < 0 || nents > req->num_sgs)
+			return -EINVAL;
+	} else if (!req->buf) {
+		return -EINVAL;
+	}
+
+	bounce = kzalloc(sizeof(*bounce), gfp);
+	if (!bounce)
+		return -ENOMEM;
+	INIT_LIST_HEAD(&bounce->pages);
+	bounce->dir = dir;
+	bounce->len = req->length;
+	udc_req->bounce = bounce;
+
+	if (udc_ep->dci)
+		ret = crg_udc_bounce_map_pages(dev, bounce, req, gfp);
+	else
+		ret = crg_udc_bounce_map_ep0(dev, bounce, req, gfp);
+
+	if (ret) {
+		crg_udc_unmap_request(udc_ep, udc_req, 0);
+		return ret;
+	}
+
+	bounce->mapped = true;
+	return 0;
+}
+
+static int crg_udc_ep_queue(struct usb_ep *_ep, struct usb_request *_req,
+			    gfp_t gfp_flags)
+{
+	struct crg_udc_request *udc_req_ptr;
+	struct crg_udc_ep *udc_ep_ptr;
+	struct crg_gadget_dev *crg_udc;
+	enum dma_data_direction dma_data_dir;
+	const struct usb_endpoint_descriptor *desc;
+	unsigned long flags;
+	int status;
+	bool is_sg;
+	u8 setup_tag = 0;
+	enum crg_setup_status setup_status = WAIT_FOR_SETUP;
+	u32 generation = 0;
+
+	if (!_ep || !_req)
+		return -EINVAL;
+
+	udc_req_ptr = container_of(_req, struct crg_udc_request, usb_req);
+	udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+	crg_udc = udc_ep_ptr->crg_udc;
+	is_sg = _req->num_sgs != 0 && _req->length != 0;
+
+	if (is_sg && !_req->sg)
+		return -EINVAL;
+	if (_req->stream_id)
+		return -EOPNOTSUPP;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	/*
+	 * A completion callback for an aborted EP0 transfer may requeue the
+	 * same request. Reject EP0 requests while the previous control
+	 * transfer is being aborted or EP0 is in an error state. EP0 becomes
+	 * available again only after the new SETUP has been accepted and
+	 * the EP0 transfer ring has been successfully reinitialized.
+	 */
+	if (udc_ep_ptr->dci == 0 && crg_udc->ep0_state != EP0_STATE_IDLE) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -ECONNRESET;
+	}
+	if (!udc_ep_ptr->first_trb || !_req->complete ||
+	    (_req->length && !is_sg && !_req->buf) ||
+	    !list_empty(&udc_req_ptr->queue)) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EINVAL;
+	}
+	/*
+	 * pullup(false) stops RUN without necessarily changing the EP
+	 * generation. Do not accept requests into a stopped USB session.
+	 */
+	if (!crg_udc_gadget_ready(crg_udc) ||
+	    crg_udc->bus_reset_pending || crg_udc->port_reset_pending) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -ESHUTDOWN;
+	}
+	if (!udc_ep_ptr->desc) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EINVAL;
+	}
+	desc = udc_ep_ptr->desc;
+	if (udc_ep_ptr->halt_failed ||
+	    !crg_udc_ep_can_queue(udc_ep_ptr) ||
+	    udc_ep_ptr->configuring) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -ESHUTDOWN;
+	}
+	setup_tag = crg_udc->setup_tag;
+	if (udc_ep_ptr->dci == 0)
+		setup_status = crg_udc->setup_status;
+	generation = udc_ep_ptr->generation;
+
+	if (usb_endpoint_xfer_control(udc_ep_ptr->desc) && is_sg) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EOPNOTSUPP;
+	}
+
+	if (usb_endpoint_xfer_control(udc_ep_ptr->desc) && !_req->length) {
+		if (crg_udc->setup_status == SETUP_PKT_PROCESS_IN_PROGRESS) {
+			/* Only a control request without DATA needs STATUS now. */
+			crg_udc_clear_req_container(udc_req_ptr);
+			crg_udc->setup_status = STATUS_STAGE_XFER;
+			status = crg_udc_build_ep0_status(udc_ep_ptr, false,
+						 -EINPROGRESS, udc_req_ptr, 0, 0);
+			if (status)
+				crg_udc->setup_status = SETUP_PKT_PROCESS_IN_PROGRESS;
+			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+			return status;
+		}
+		/* A zero-length DATA stage still needs a DATA-stage TRB. */
+		if (crg_udc->setup_status != DATA_STAGE_XFER &&
+		    crg_udc->setup_status != DATA_STAGE_RECV) {
+			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+			return -EINVAL;
+		}
+	}
+
+	if (usb_endpoint_xfer_control(udc_ep_ptr->desc)) {
+		if (crg_udc->setup_status == DATA_STAGE_XFER ||
+		    crg_udc->setup_status == STATUS_STAGE_XFER) {
+			dma_data_dir = DMA_TO_DEVICE;
+		} else if (crg_udc->setup_status == DATA_STAGE_RECV ||
+			   crg_udc->setup_status == STATUS_STAGE_RECV) {
+			dma_data_dir = DMA_FROM_DEVICE;
+		} else {
+			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+			return -EINVAL;
+		}
+	} else {
+		dma_data_dir = usb_endpoint_dir_in(udc_ep_ptr->desc) ?
+				       DMA_TO_DEVICE :
+				       DMA_FROM_DEVICE;
+	}
+	crg_udc->queueing++;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	crg_udc_clear_req_container(udc_req_ptr);
+
+	status = crg_udc_bounce_map_request(udc_ep_ptr, udc_req_ptr,
+					   dma_data_dir, gfp_flags);
+	if (status)
+		goto out_queueing;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	if (udc_ep_ptr->desc != desc || !udc_req_ptr->usb_req.complete ||
+	    !list_empty(&udc_req_ptr->queue) ||
+	    !crg_udc_gadget_ready(crg_udc) ||
+	    crg_udc->bus_reset_pending || crg_udc->port_reset_pending ||
+	    udc_ep_ptr->configuring || udc_ep_ptr->halt_failed ||
+	    !crg_udc_ep_can_queue(udc_ep_ptr) ||
+	    udc_ep_ptr->generation != generation ||
+	    (udc_ep_ptr->dci == 0 &&
+	     (crg_udc->ep0_state != EP0_STATE_IDLE ||
+	      crg_udc->setup_status != setup_status ||
+	      crg_udc->setup_tag != setup_tag))) {
+		status = -ESHUTDOWN;
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		goto unmap;
+	}
+
+	_req->status = -EINPROGRESS;
+	_req->actual = 0;
+
+	/* A halted or temporarily stopped endpoint may accept requests,
+	 * but no TRBs may be published until it resumes.
+	 */
+	if (udc_ep_ptr->ep_state != EP_STATE_RUNNING) {
+		status = 0;
+	} else if (!list_empty(&udc_ep_ptr->queue)) {
+		struct crg_udc_request *tail;
+
+		tail = list_last_entry(&udc_ep_ptr->queue,
+				       struct crg_udc_request, queue);
+		status = (!tail->all_trbs_queued ||
+			  udc_ep_ptr->tran_ring_full) ? 0 :
+			 crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+	} else if (udc_ep_ptr->tran_ring_full) {
+		status = 0;
+	} else {
+		status = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+	}
+
+	if (!status) {
+		list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+		goto out_locked;
+	}
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+unmap:
+	crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr, 0);
+out_queueing:
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+out_locked:
+	crg_udc->queueing--;
+	if (crg_udc->gadget_stopping && !crg_udc->queueing)
+		wake_up(&crg_udc->teardown_wait);
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	return status;
+}
+
+static int crg_udc_ep_set_halt(struct usb_ep *_ep, int value)
+{
+	struct crg_udc_ep *udc_ep_ptr;
+	int status;
+	unsigned long flags = 0;
+	struct crg_gadget_dev *crg_udc;
+
+	if (!_ep)
+		return -EINVAL;
+
+	udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+	crg_udc = udc_ep_ptr->crg_udc;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	if (!udc_ep_ptr->desc) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EINVAL;
+	}
+	/*
+	 * A previous HALT/STOP failure retained the queue for DMA safety.
+	 * Do not reject the HALT retry merely because requests remain queued.
+	 */
+	if (value && !udc_ep_ptr->halt_failed &&
+	    usb_endpoint_dir_in(udc_ep_ptr->desc) &&
+	    !list_empty(&udc_ep_ptr->queue)) {
+		dev_err(crg_udc->dev, "set_halt: list not empty\n");
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EAGAIN;
+	}
+
+	status = crg_udc_ep_halt(udc_ep_ptr, value, 1, &flags);
+
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	return status;
+}
+
+static int crg_udc_ep_set_wedge(struct usb_ep *_ep)
+{
+	struct crg_udc_ep *udc_ep_ptr;
+	bool was_wedged;
+	int status;
+	unsigned long flags = 0;
+	struct crg_gadget_dev *crg_udc;
+
+	if (!_ep)
+		return -EINVAL;
+
+	udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+	crg_udc = udc_ep_ptr->crg_udc;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+
+	if (!udc_ep_ptr->desc) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EINVAL;
+	}
+
+	/* Like set_halt(), do not cancel queued IN requests.  A failed
+	 * HALT/STOP may be retried even though its queue remains populated.
+	 */
+	if (!udc_ep_ptr->halt_failed &&
+	    usb_endpoint_dir_in(udc_ep_ptr->desc) &&
+	    !list_empty(&udc_ep_ptr->queue)) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EAGAIN;
+	}
+
+	was_wedged = udc_ep_ptr->wedge;
+	udc_ep_ptr->wedge = true;
+	status = crg_udc_ep_halt(udc_ep_ptr, 1, 1, &flags);
+	if (status)
+		udc_ep_ptr->wedge = was_wedged;
+
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	return status;
+}
+
+static int crg_udc_epcx_setup(struct crg_udc_ep *udc_ep)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+	const struct usb_endpoint_descriptor *desc = udc_ep->desc;
+	const struct usb_ss_ep_comp_descriptor *comp_desc = udc_ep->comp_desc;
+	u8 dci = udc_ep->dci;
+	struct crg_ep_cx *epcx = &crg_udc->p_epcx[dci - 2];
+	enum crg_ep_type ep_type;
+	u16 maxburst = 0;
+	u8 maxstreams = 0;
+	u16 maxsize;
+	u32 dw;
+
+	/* corigine gadget dir should be opposite to host dir */
+	if (usb_endpoint_dir_out(desc))
+		ep_type = usb_endpoint_type(desc) + EP_TYPE_INVALID2;
+	else
+		ep_type = usb_endpoint_type(desc);
+
+	maxsize = usb_endpoint_maxp(desc) & 0x07ff;
+
+	if (crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+		if (!comp_desc)
+			return -EINVAL;
+		maxburst = comp_desc->bMaxBurst;
+
+		if (usb_endpoint_xfer_bulk(udc_ep->desc))
+			maxstreams = comp_desc->bmAttributes & 0x1f;
+
+	} else if ((crg_udc->gadget.speed == USB_SPEED_HIGH ||
+		    crg_udc->gadget.speed == USB_SPEED_FULL) &&
+		   (usb_endpoint_xfer_int(udc_ep->desc) ||
+		    usb_endpoint_xfer_isoc(udc_ep->desc))) {
+		if (crg_udc->gadget.speed == USB_SPEED_HIGH)
+			maxburst = usb_endpoint_maxp_mult(desc) - 1;
+		if (maxburst == 0x3) {
+			dev_err(crg_udc->dev, "invalid maxburst\n");
+			maxburst = 0x2;
+		}
+	}
+
+	/* fill ep_dw0 */
+	dw = 0;
+	FIELD_MODIFY(EP_CX_LOGICAL_EP_NUM_MASK, &dw, udc_ep->dci / 2);
+	FIELD_MODIFY(EP_CX_INTERVAL_MASK, &dw, desc->bInterval);
+	if (maxstreams) {
+		dev_err(crg_udc->dev, "maxstream=%d is not expected\n",
+			maxstreams);
+	}
+	epcx->dw0 = cpu_to_le32(dw);
+
+	/* fill ep_dw1 */
+	dw = 0;
+	FIELD_MODIFY(EP_CX_EP_TYPE_MASK, &dw, ep_type);
+	FIELD_MODIFY(EP_CX_MAX_PACKET_SIZE_MASK, &dw, maxsize);
+	FIELD_MODIFY(EP_CX_MAX_BURST_SIZE_MASK, &dw, maxburst);
+	epcx->dw1 = cpu_to_le32(dw);
+
+	/* fill ep_dw2 */
+	dw = lower_32_bits(udc_ep->tran_ring_info.dma_addr) &
+	     EP_CX_TR_DQPT_LO_MASK;
+	FIELD_MODIFY(EP_CX_DEQ_CYC_STATE_MASK, &dw, udc_ep->pcs);
+	epcx->dw2 = cpu_to_le32(dw);
+
+	/* fill ep_dw3 */
+	dw = upper_32_bits(udc_ep->tran_ring_info.dma_addr);
+	epcx->dw3 = cpu_to_le32(dw);
+	/* Ensure that epcx is updated */
+	dma_wmb();
+	return 0;
+}
+
+static int crg_udc_epcx_update_dqptr(struct crg_udc_ep *udc_ep)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+	u8 dci = udc_ep->dci;
+	struct crg_ep_cx *epcx;
+	u32 dw;
+	dma_addr_t dqptaddr;
+	u32 cmd_param0;
+
+	if (dci < 2 || dci >= CRG_NUM_EP_CX) {
+		dev_err(crg_udc->dev, "invalid DCI %u for dequeue pointer\n", dci);
+		return -EINVAL;
+	}
+
+	epcx = &crg_udc->p_epcx[dci - 2];
+	dqptaddr = crg_udc_tran_trb_virt_to_dma(udc_ep, udc_ep->deq_pt);
+
+	if (dqptaddr == DMA_MAPPING_ERROR)
+		return -EINVAL;
+
+	/* fill ep_dw2 */
+	dw = lower_32_bits(dqptaddr) & EP_CX_TR_DQPT_LO_MASK;
+	/* The consumer cycle can lag the producer by one ring wrap. */
+	FIELD_MODIFY(EP_CX_DEQ_CYC_STATE_MASK, &dw,
+		     udc_ep->deq_pt > udc_ep->enq_pt ?
+		     udc_ep->pcs ^ 1 : udc_ep->pcs);
+	epcx->dw2 = cpu_to_le32(dw);
+
+	/* fill ep_dw3 */
+	dw = upper_32_bits(dqptaddr);
+	epcx->dw3 = cpu_to_le32(dw);
+
+	cmd_param0 = BIT(udc_ep->dci);
+	/* Ensure that dqptr is updated */
+	dma_wmb();
+
+	return crg_udc_issue_command(crg_udc, CRG_CMD_SET_TR_DQPTR, cmd_param0, 0);
+}
+
+void crg_udc_free_request(struct usb_ep *_ep, struct usb_request *_req)
+{
+	struct crg_udc_request *udc_req_ptr;
+
+	if (!_ep || !_req)
+		return;
+
+	udc_req_ptr = container_of(_req, struct crg_udc_request, usb_req);
+
+	trace_crg_udc_free_request(container_of(_ep, struct crg_udc_ep, usb_ep),
+				   udc_req_ptr);
+
+	/* A queued request is returned through its completion callback. */
+	if (!list_empty(&udc_req_ptr->queue) || udc_req_ptr->bounce) {
+		struct crg_udc_ep *ep = container_of(_ep, struct crg_udc_ep, usb_ep);
+
+		dev_err_ratelimited(ep->crg_udc->dev, "freeing active request on %s\n",
+				    _ep->name);
+		return;
+	}
+
+	kfree(udc_req_ptr);
+}
+
+static int crg_udc_queue_trbs(struct crg_udc_ep *udc_ep_ptr,
+			      struct crg_udc_request *udc_req_ptr, bool b_isoc,
+			      u32 xfer_ring_size, u32 num_trbs_needed,
+			      u64 buffer_length)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+	struct crg_transfer_trb *p_xfer_ring = udc_ep_ptr->first_trb;
+	struct crg_transfer_trb *enq_pt = udc_ep_ptr->enq_pt;
+	struct usb_request *usb_req = &udc_req_ptr->usb_req;
+	/* Plan the whole TD before touching hardware-visible TRBs. */
+	struct {
+		dma_addr_t dma;
+		u32 len;
+	} plan[32];
+	struct crg_udc_request sg_cursor = {
+		.bounce = udc_req_ptr->bounce,
+		.sg = udc_req_ptr->sg,
+		.sg_index = udc_req_ptr->sg_index,
+		.sg_offset = udc_req_ptr->sg_offset,
+	};
+	u64 remaining = buffer_length;
+	u32 num_trbs_ava, count, i;
+	u32 td_trbs, td_size, td_bytes, td_done = 0;
+	u32 td_start_offset = usb_req->length - buffer_length;
+	u32 j = 1;
+	u32 intr_rate = b_isoc ? CRG_UDC_ISOC_EP_INTERRUPT_RATE :
+				 CRG_UDC_BULK_EP_INTERRUPT_RATE;
+	bool full_td = true;
+	bool need_zlp = false;
+	bool is_sg = udc_req_ptr->bounce && udc_req_ptr->bounce->sgt.nents;
+	u8 original_pcs = udc_ep_ptr->pcs;
+
+	if (!udc_ep_ptr->usb_ep.maxpacket)
+		return -EINVAL;
+
+	if (!b_isoc && usb_req->zero && usb_req->length &&
+	    !(usb_req->length % udc_ep_ptr->usb_ep.maxpacket))
+		need_zlp = true;
+
+	if (!num_trbs_needed)
+		return -EINVAL;
+
+	/* TD_SIZE is five bits. A request may span multiple 32-TRB TDs. */
+	td_trbs = min_t(u32, num_trbs_needed, ARRAY_SIZE(plan));
+	num_trbs_ava = crg_udc_room_on_ring(crg_udc, xfer_ring_size,
+					    p_xfer_ring, udc_ep_ptr->enq_pt,
+					    udc_ep_ptr->deq_pt);
+
+	if (num_trbs_ava < td_trbs) {
+		if (b_isoc)
+			return -EBUSY;
+		trace_crg_udc_no_room_on_ring(
+			"deferring TRBs to next doorbell");
+		count = num_trbs_ava;
+		full_td = false;
+		need_zlp = false;
+	} else {
+		count = td_trbs;
+	}
+
+	if (!count) {
+		/* There really is no room, not a malformed SG list. */
+		udc_ep_ptr->tran_ring_full = true;
+		return -ENOSPC;
+	}
+
+	/* Preflight all buffers, including every SG boundary.  No request SG
+	 * state or hardware ring state changes if any entry is invalid.
+	 */
+	if (is_sg && !sg_cursor.sg && crg_udc_sg_init(&sg_cursor))
+		return -EINVAL;
+
+	for (i = 0; i < count; i++) {
+		dma_addr_t dma;
+		u32 len, sg_len = 0;
+
+		if (is_sg) {
+			if (!crg_udc_sg_current(&sg_cursor, &dma, &sg_len) ||
+			    !sg_len || dma_mapping_error(crg_udc->dev, dma))
+				return -EINVAL;
+			len = min_t(u64, remaining,
+					min_t(u32, sg_len,
+						crg_udc_trb_chunk_len(dma, remaining)));
+		} else {
+			if (remaining)
+				return -EINVAL;
+			dma = 0;
+			len = crg_udc_trb_chunk_len(dma, remaining);
+		}
+
+		if (!len && remaining)
+			return -EINVAL;
+		if (is_sg && !crg_udc_sg_advance(&sg_cursor, len))
+			return -EINVAL;
+
+		plan[i].dma = dma;
+		plan[i].len = len;
+		remaining -= len;
+	}
+
+	/* All validation is complete.  The following loop cannot fail; the
+	 * first TRB's cycle bit is published last, after the complete TD.
+	 */
+	td_bytes = buffer_length - remaining;
+	remaining = buffer_length;
+	for (i = 0; i < count; i++) {
+		u32 trb_len = plan[i].len;
+		bool chain_bit = i != count - 1;
+		bool ioc = !chain_bit;
+		bool short_pkt = usb_endpoint_dir_out(udc_ep_ptr->desc);
+
+		remaining -= trb_len;
+		td_done += trb_len;
+		td_size = crg_udc_td_size(td_bytes, td_done,
+					  udc_ep_ptr->usb_ep.maxpacket, !chain_bit);
+		if (!full_td && j == intr_rate) {
+			ioc = true;
+			j = 0;
+		}
+
+		if (b_isoc) {
+			crg_udc_setup_trb(
+				enq_pt, trb_len, plan[i].dma, td_size,
+				i == 0 ? (original_pcs ^ 1) : udc_ep_ptr->pcs,
+				i == 0 ? TRB_TYPE_XFER_DATA_ISOCH :
+					 TRB_TYPE_XFER_NORMAL,
+				short_pkt, chain_bit, ioc, false, 0,
+				i == 0, 0, i == 0, 0);
+		} else {
+			bool azp = ioc && !chain_bit && !remaining && need_zlp;
+
+			crg_udc_setup_trb(
+				enq_pt, trb_len, plan[i].dma, td_size,
+				i == 0 ? (original_pcs ^ 1) : udc_ep_ptr->pcs,
+				TRB_TYPE_XFER_NORMAL, short_pkt, chain_bit, ioc,
+				false, 0, false, 0, 0, azp);
+		}
+
+		trace_crg_udc_queue_trb(udc_ep_ptr, enq_pt);
+		enq_pt++;
+		j++;
+
+		if (enq_pt == udc_ep_ptr->last_trb) {
+			dma_wmb();
+			le32p_replace_bits(&enq_pt->dw3, udc_ep_ptr->pcs,
+					   TRB_CYCLE_BIT_MASK);
+			udc_ep_ptr->pcs ^= 1;
+			enq_pt = udc_ep_ptr->first_trb;
+		}
+	}
+
+	dma_wmb();
+	le32p_replace_bits(&udc_ep_ptr->enq_pt->dw3, original_pcs,
+			   TRB_CYCLE_BIT_MASK);
+
+	/* Commit request bookkeeping only after publishing the whole TD. */
+	udc_req_ptr->first_trb = udc_ep_ptr->enq_pt;
+	udc_req_ptr->td_start_offset = td_start_offset;
+	udc_ep_ptr->enq_pt = enq_pt;
+	udc_ep_ptr->tran_ring_full = count == num_trbs_ava;
+	udc_req_ptr->buff_len_left = remaining;
+	udc_req_ptr->td_pending = true;
+	udc_req_ptr->trbs_needed = remaining ? num_trbs_needed - count : 0;
+	if (!remaining)
+		udc_req_ptr->all_trbs_queued = true;
+	if (is_sg) {
+		udc_req_ptr->sg = sg_cursor.sg;
+		udc_req_ptr->sg_index = sg_cursor.sg_index;
+		udc_req_ptr->sg_offset = sg_cursor.sg_offset;
+	}
+
+	if (udc_ep_ptr->enq_pt == udc_ep_ptr->first_trb)
+		udc_req_ptr->last_trb = udc_ep_ptr->last_trb - 1;
+	else
+		udc_req_ptr->last_trb = udc_ep_ptr->enq_pt - 1;
+
+	return 0;
+}
+
+static int crg_udc_reinit(struct crg_gadget_dev *crg_udc, bool notify_reset)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	struct crg_udc_ep *ep;
+	u32 i, tmp;
+	int ret;
+
+	if (crg_udc->shutting_down || crg_udc->gadget_stopping)
+		return -ESHUTDOWN;
+	if (crg_udc->controller_recovering)
+		return -EBUSY;
+	crg_udc_recovery_begin(crg_udc);
+
+	/* A newer bus reset invalidates any already-running reset worker. */
+	crg_udc->bus_reset_generation++;
+	/* Invalidate EP0 mappings and SETUPs belonging to the old bus state. */
+	crg_udc->udc_ep[0].generation++;
+	crg_udc->pending_setup_valid = false;
+	crg_udc->setup_status = WAIT_FOR_SETUP;
+	crg_udc->device_state = USB_STATE_RECONNECTING;
+	crg_udc->port_reset_pending = false;
+
+	/*
+	 * The controller datasheet defines EP Enabled only in bits 31:2
+	 * (RW1C). Bits 1:0 are reserved and may read as nonzero (bit 0
+	 * stayed set on AX3000), so neither write them nor wait for them
+	 * to clear when disabling endpoints before DMA cleanup.
+	 */
+	tmp = readl(&uccr->ep_enable) & GENMASK(31, 2);
+	writel(tmp, &uccr->ep_enable);
+	if (readl_poll_timeout_atomic(&uccr->ep_enable, tmp,
+				      !(tmp & GENMASK(31, 2)), 10, 10000)) {
+		/* Still DMA-visible: only SWRST can safely release these TRBs. */
+		dev_err(crg_udc->dev,
+			"bus reset: EP_ENABLE did not clear (0x%x)\n", tmp);
+		crg_udc->bus_reset_pending = false;
+		crg_udc->resetting = true;
+		ret = -ETIMEDOUT;
+		goto out;
+	}
+
+	/*
+	 * Disabling an endpoint does not itself prove that DMA stopped.
+	 * Before unmapping any requests, verify every data endpoint is idle.
+	 */
+	if (readl_poll_timeout_atomic(&uccr->ep_running, tmp,
+					      !(tmp & GENMASK(CRG_NUM_EP_CX - 1, 2)),
+					      10, 10000)) {
+		dev_err(crg_udc->dev,
+			"bus reset: EP_RUNNING did not clear (0x%x)\n", tmp);
+		crg_udc->bus_reset_pending = false;
+		crg_udc->resetting = true;
+		ret = -ETIMEDOUT;
+		goto out;
+	}
+
+	/*
+	 * No callbacks until every endpoint has been marked inactive.  When
+	 * notifying composite, keep usb_ep.enabled and descriptors intact:
+	 * its ->disable() callbacks must still reach usb_ep_disable().
+	 */
+	crg_udc->bus_reset_pending = notify_reset;
+	for (i = 2; i < CRG_NUM_EP_CX; i++) {
+		ep = &crg_udc->udc_ep[i];
+		ep->generation++;
+		ep->ep_state = EP_STATE_DISABLED;
+		ep->halt_failed = false;
+		ep->disable_failed = false;
+		ep->dequeue_paused = false;
+		ep->tran_ring_full = false;
+		if (!notify_reset) {
+			ep->desc = NULL;
+			ep->comp_desc = NULL;
+		}
+	}
+	crg_udc->num_enabled_eps = 0;
+
+	for (i = 2; i < CRG_NUM_EP_CX; i++)
+		crg_udc_ep_nuke(&crg_udc->udc_ep[i], -ESHUTDOWN);
+	if (!notify_reset)
+		for (i = 2; i < CRG_NUM_EP_CX; i++)
+			crg_udc_ep_clear_enabled(&crg_udc->udc_ep[i]);
+
+	/*
+	 * Givebacks release udc_lock. A system suspend/shutdown may have
+	 * taken ownership of the controller during that window.
+	 */
+	if (crg_udc->suspended || crg_udc->shutting_down || crg_udc->resetting) {
+		ret = -ESHUTDOWN;
+		goto out;
+	}
+	/* A gadget disconnect may have stopped RUN during giveback. */
+	if (crg_udc->stopped_needs_reset) {
+		ret = -ECONNRESET;
+		goto out;
+	}
+
+	/* EP0 is reinitialized after the event ring has been drained. */
+	if (crg_udc->dev_addr != 0) {
+		u32 param0 = FIELD_PREP(CRG_CMD2_0_DEV_ADDR_MASK, 0);
+
+		if (crg_udc_issue_command(crg_udc, CRG_CMD_SET_ADDR,
+					  param0, 0)) {
+			/* Cleanup must still run, but no new transfers may start. */
+			crg_udc->resetting = true;
+			ret = -EIO;
+			goto out;
+		}
+		crg_udc->dev_addr = 0;
+	}
+
+	crg_udc_clear_portpm(crg_udc);
+	ret = 0;
+out:
+	crg_udc_recovery_end(crg_udc);
+	return ret;
+}
+
+void crg_udc_set_pullup(struct crg_gadget_dev *crg_udc, bool is_on)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 val;
+
+	if (is_on) {
+		if (crg_udc->controller_faulted || crg_udc->resetting ||
+		    crg_udc->controller_recovering || crg_udc->bus_reset_pending ||
+		    crg_udc->endpoint_recovery_pending ||
+		    crg_udc->shutting_down || crg_udc->gadget_stopping)
+			return;
+		/* Enable event reporting and interrupts, then assert RUN */
+		val = readl(&uccr->config1);
+		val |= (CRG_U3DC_CFG1_CSC_EVENT_EN |
+			CRG_U3DC_CFG1_PEC_EVENT_EN |
+			CRG_U3DC_CFG1_PPC_EVENT_EN |
+			CRG_U3DC_CFG1_PRC_EVENT_EN |
+			CRG_U3DC_CFG1_PLC_EVENT_EN |
+			CRG_U3DC_CFG1_CEC_EVENT_EN);
+		writel(val, &uccr->config1);
+
+		val = readl(&uccr->control);
+		val |= (CRG_U3DC_CTRL_SYSERR_EN | CRG_U3DC_CTRL_INT_EN |
+			CRG_U3DC_CTRL_RUN);
+		crg_udc->hw_dma_armed = true;
+		crg_udc->transfer_rings_live = true;
+		writel(val, &uccr->control);
+	} else {
+		val = readl(&uccr->control);
+		val &= ~CRG_U3DC_CTRL_RUN;
+		writel(val, &uccr->control);
+	}
+}
+
+void crg_udc_set_vbus_detect(struct crg_gadget_dev *crg_udc, bool enable)
+{
+	u32 val = readl(crg_udc->mmio_virt_base + CRG_UDC_VENDOR_REG);
+
+	if (enable)
+		val |= CRG_UDC_VBUS_DETECT;
+	else
+		val &= ~CRG_UDC_VBUS_DETECT;
+
+	writel(val, crg_udc->mmio_virt_base + CRG_UDC_VENDOR_REG);
+}
+
+static bool crg_udc_sg_advance(struct crg_udc_request *udc_req, u32 bytes)
+{
+	struct scatterlist *sg = udc_req->sg;
+	unsigned int i = udc_req->sg_index;
+
+	while (bytes) {
+		u32 len;
+
+		if (!sg || i >= udc_req->bounce->sgt.nents)
+			return false;
+
+		len = sg_dma_len(sg);
+		if (!len || udc_req->sg_offset == len) {
+			sg = sg_next(sg);
+			i++;
+			udc_req->sg_offset = 0;
+			continue;
+		}
+
+		len -= udc_req->sg_offset;
+		if (bytes < len) {
+			udc_req->sg_offset += bytes;
+			bytes = 0;
+			break;
+		}
+
+		bytes -= len;
+		sg = sg_next(sg);
+		i++;
+		udc_req->sg_offset = 0;
+	}
+
+	udc_req->sg = sg;
+	udc_req->sg_index = i;
+	return true;
+}
+
+static bool crg_udc_sg_current(struct crg_udc_request *udc_req, dma_addr_t *dma,
+			       u32 *len)
+{
+	struct scatterlist *sg = udc_req->sg;
+
+	while (sg && udc_req->sg_index < udc_req->bounce->sgt.nents &&
+	       !sg_dma_len(sg)) {
+		sg = sg_next(sg);
+		udc_req->sg_index++;
+		udc_req->sg_offset = 0;
+	}
+	udc_req->sg = sg;
+
+	if (!sg || udc_req->sg_index >= udc_req->bounce->sgt.nents)
+		return false;
+
+	if (udc_req->sg_offset >= sg_dma_len(sg))
+		return false;
+
+	*dma = sg_dma_address(sg) + udc_req->sg_offset;
+	*len = sg_dma_len(sg) - udc_req->sg_offset;
+	return true;
+}
+
+static int crg_udc_sg_init(struct crg_udc_request *udc_req)
+{
+	struct scatterlist *sg;
+	unsigned int i;
+
+	for_each_sgtable_dma_sg(&udc_req->bounce->sgt, sg, i) {
+		if (sg_dma_len(sg)) {
+			udc_req->sg = sg;
+			udc_req->sg_index = i;
+			udc_req->sg_offset = 0;
+			return 0;
+		}
+	}
+
+	return -EINVAL;
+}
+
+static void crg_udc_bounce_unmap(struct device *dev,
+				struct crg_udc_bounce *bounce)
+{
+	if (!bounce->mapped)
+		return;
+
+	if (bounce->buf)
+		dma_unmap_single(dev, bounce->dma, bounce->len, bounce->dir);
+	else
+		dma_unmap_sgtable(dev, &bounce->sgt, bounce->dir, 0);
+}
+
+static void crg_udc_bounce_free(struct crg_udc_bounce *bounce)
+{
+	struct page *page, *next;
+
+	list_for_each_entry_safe(page, next, &bounce->pages, lru) {
+		list_del(&page->lru);
+		__free_page(page);
+	}
+	sg_free_table(&bounce->sgt);
+	kfree(bounce->buf);
+	kfree(bounce);
+}
+
+/* Call only after hardware can no longer access the retained mappings. */
+void crg_udc_release_quarantine(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_udc_bounce *bounce, *next;
+
+	list_for_each_entry_safe(bounce, next, &crg_udc->dma_quarantine,
+				 quarantine) {
+		list_del(&bounce->quarantine);
+		crg_udc_bounce_unmap(crg_udc->dev, bounce);
+		crg_udc_bounce_free(bounce);
+	}
+}
+
+void crg_udc_unmap_request(struct crg_udc_ep *udc_ep,
+			   struct crg_udc_request *udc_req, size_t received)
+{
+	struct crg_udc_bounce *bounce = udc_req->bounce;
+	struct usb_request *req = &udc_req->usb_req;
+
+	if (!bounce)
+		return;
+
+	udc_req->bounce = NULL;
+	crg_udc_bounce_unmap(udc_ep->crg_udc->dev, bounce);
+	if (bounce->dir == DMA_FROM_DEVICE && received) {
+		size_t len = min(received, bounce->len);
+
+		if (bounce->buf) {
+			memcpy(req->buf, bounce->buf, len);
+		} else {
+			size_t copied = crg_udc_copy_bounce_pages(bounce, req, len);
+
+			if (copied != len) {
+				req->status = -EFAULT;
+				req->actual = copied;
+			}
+		}
+	}
+	crg_udc_bounce_free(bounce);
+}
+
+static int crg_udc_validate_ep_desc(struct crg_udc_ep *udc_ep,
+				    const struct usb_endpoint_descriptor *desc)
+{
+	const struct usb_ss_ep_comp_descriptor *comp = udc_ep->comp_desc;
+	u16 maxpacket = usb_endpoint_maxp(desc);
+
+	if (usb_endpoint_xfer_control(desc) || !maxpacket ||
+	    maxpacket > CRG_MAX_PACKET_SIZE)
+		return -EINVAL;
+
+	if (crg_udc_index2dci(desc->bEndpointAddress) != udc_ep->dci)
+		return -EINVAL;
+
+	if (usb_endpoint_xfer_bulk(desc) && comp && usb_ss_max_streams(comp))
+		return -EOPNOTSUPP;
+
+	if (usb_endpoint_xfer_isoc(desc) && desc->bInterval == 0)
+		return -EINVAL;
+
+	if (udc_ep->crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+		if (!comp || comp->bMaxBurst > 15)
+			return -EINVAL;
+		if (usb_endpoint_xfer_isoc(desc) &&
+		    ((comp->bmAttributes & 0x3) > 2))
+			return -EINVAL;
+	} else if (udc_ep->crg_udc->gadget.speed == USB_SPEED_HIGH) {
+		if (desc->bInterval > 16 && (usb_endpoint_xfer_int(desc) ||
+					     usb_endpoint_xfer_isoc(desc)))
+			return -EINVAL;
+		if (usb_endpoint_xfer_isoc(desc) &&
+		    usb_endpoint_maxp_mult(desc) > 3)
+			return -EINVAL;
+	} else if (udc_ep->crg_udc->gadget.speed == USB_SPEED_FULL) {
+		if (maxpacket > 64 && !usb_endpoint_xfer_isoc(desc))
+			return -EINVAL;
+		if (usb_endpoint_xfer_isoc(desc) && maxpacket > 1023)
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
+static bool crg_udc_vbus_connected(struct crg_gadget_dev *crg_udc)
+{
+	return !!(readl(&crg_udc->uccr->portsc) & CRG_U3DC_PORTSC_PP);
+}
+
+static void crg_ep_struct_setup(struct crg_gadget_dev *crg_udc, u32 dci,
+				const char *name)
+{
+	struct crg_udc_ep *ep = &crg_udc->udc_ep[dci];
+
+	ep->dci = dci;
+
+	if (ep->dci > 1) {
+		strscpy_pad(ep->name, name, sizeof(ep->name));
+		ep->usb_ep.name = ep->name;
+
+		ep->usb_ep.caps.type_iso = 1;
+		ep->usb_ep.caps.type_bulk = 1;
+		ep->usb_ep.caps.type_int = 1;
+		if (ep->dci % 2)
+			ep->usb_ep.caps.dir_out = 1;
+		else
+			ep->usb_ep.caps.dir_in = 1;
+		usb_ep_set_maxpacket_limit(&ep->usb_ep, CRG_MAX_PACKET_SIZE);
+	} else {
+		strscpy(ep->name, "ep0", sizeof(ep->name));
+		ep->usb_ep.name = ep->name;
+		ep->usb_ep.maxpacket = 512;
+		ep->usb_ep.caps.type_control = 1;
+		ep->usb_ep.caps.dir_in = 1;
+		ep->usb_ep.caps.dir_out = 1;
+	}
+
+	ep->usb_ep.ops = &crg_udc_ep_ops;
+	ep->crg_udc = crg_udc;
+
+	INIT_LIST_HEAD(&ep->queue);
+	if (ep->dci > 1)
+		list_add_tail(&ep->usb_ep.ep_list, &crg_udc->gadget.ep_list);
+}
+
+void crg_udc_reset_work(struct work_struct *work)
+{
+	struct crg_gadget_dev *crg_udc =
+		container_of(work, struct crg_gadget_dev, udc_reset_work);
+	struct usb_gadget_driver *driver;
+	unsigned long flags;
+	u32 generation;
+	bool requeue;
+	int i, ret;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	if (crg_udc->resetting || crg_udc->controller_recovering ||
+	    crg_udc->shutting_down || crg_udc->gadget_stopping ||
+	    crg_udc->suspended)
+		goto unlock;
+	generation = crg_udc->bus_reset_generation;
+	driver = crg_udc->bus_reset_pending && crg_udc->async_callbacks ?
+		crg_udc->gadget_driver : NULL;
+	/* Defer until callbacks initiated before the reset have returned. */
+	if (driver && crg_udc->gadget_callbacks_active)
+		goto unlock;
+	if (driver)
+		crg_udc->reset_callback_active = true;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	if (driver)
+		usb_gadget_udc_reset(&crg_udc->gadget, driver);
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	if (generation == crg_udc->bus_reset_generation &&
+	    crg_udc->bus_reset_pending) {
+		/*
+		 * DMA was stopped by crg_udc_reinit() before scheduling this work.
+		 * Some gadget functions do not call usb_ep_disable() when reset.
+		 * Finalize these endpoints under the lock and generation check;
+		 * calling usb_ep_disable() after unlocking would let a newer
+		 * reset reuse the endpoint before the disable operation.
+		 */
+		for (i = 2; i < CRG_NUM_EP_CX; i++) {
+			struct crg_udc_ep *ep = &crg_udc->udc_ep[i];
+			struct crg_ep_cx *epcx;
+
+			if (ep->ep_state != EP_STATE_DISABLED ||
+			    !ep->usb_ep.enabled)
+				continue;
+
+			dev_warn(crg_udc->dev,
+				 "EP%d remained enabled after gadget reset\n", i);
+			epcx = &crg_udc->p_epcx[i - 2];
+			ep->generation++;
+			ep->desc = NULL;
+			ep->comp_desc = NULL;
+			ep->tran_ring_full = false;
+			ep->halt_failed = false;
+			ep->disable_failed = false;
+			ep->dequeue_paused = false;
+			memset(epcx, 0, sizeof(*epcx));
+			/*
+			 * The controller disabled this endpoint and drained the
+			 * requests; reflect that in the USB core's enabled flag.
+			 */
+			crg_udc_ep_clear_enabled(ep);
+		}
+		if (!crg_udc->resetting)
+			crg_udc->bus_reset_pending = false;
+	}
+
+	/*
+	 * Keep port reset events deferred through endpoint cleanup, not just
+	 * through the gadget callback. Replay after clearing the guard.
+	 */
+	if (driver)
+		crg_udc->reset_callback_active = false;
+	if (!crg_udc->resetting && !crg_udc->suspended &&
+	    !crg_udc->shutting_down && crg_udc->deferred_system_error) {
+		crg_udc->deferred_system_error = false;
+		crg_udc_recover_link(crg_udc, &flags, false);
+	}
+	if (!crg_udc->resetting && !crg_udc->suspended &&
+	    !crg_udc->shutting_down && crg_udc->deferred_port_events) {
+		ret = crg_udc_handle_port_status(crg_udc, &flags);
+		if (ret && ret != -ECONNRESET)
+			dev_err(crg_udc->dev,
+				"deferred port status failed: %d\n", ret);
+	}
+
+	/* Port-event replay may have initiated another bus reset. */
+	requeue = generation != crg_udc->bus_reset_generation &&
+		  crg_udc->bus_reset_pending && crg_udc->async_callbacks &&
+		  crg_udc->gadget_driver;
+	/* A failed configured-session reset defers RUN until the gadget's
+	 * reset callback has dropped the old composite configuration.
+	 */
+	if (!requeue && !crg_udc->bus_reset_pending && !crg_udc->connected &&
+	    !crg_udc->resetting && !crg_udc->suspended &&
+	    !crg_udc->shutting_down && crg_udc->pullup_on &&
+	    crg_udc->gadget_driver && crg_udc->async_callbacks &&
+	    (crg_udc->restart_after_reset || crg_udc_vbus_connected(crg_udc))) {
+		crg_udc->restart_after_reset = false;
+		crg_udc_set_pullup(crg_udc, true);
+	}
+
+	if (requeue || crg_udc->external_reset_pending ||
+	    crg_udc->endpoint_recovery_pending)
+		crg_udc_kick_reset_work(crg_udc);
+unlock:
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+}
+
+/*
+ * Called with udc_lock held from the port IRQ or its retry worker.
+ * The old bus-reset path may have failed to quiesce endpoint DMA.  Never
+ * unmap requests or restart RUN based on a failed EP_ENABLE/EP_RUNNING
+ * poll; SWRST must succeed and rebuild all rings first.
+ *
+ * A successful full reset invalidates the current PORTSC snapshot/event
+ * ring, so callers must stop processing that event (-ECONNRESET).
+ */
+static int crg_udc_recover_port_failure(struct crg_gadget_dev *crg_udc,
+					unsigned long *flags, bool notify_reset)
+{
+	int ret;
+
+	if (crg_udc->controller_recovering || crg_udc->suspended ||
+	    crg_udc->shutting_down || !crg_udc->gadget_driver)
+		return -ESHUTDOWN;
+
+	ret = crg_udc_recover_link(crg_udc, flags, notify_reset);
+	return ret ?: -ECONNRESET;
+}
+
+/* PRC is W1C at detection, not a completion notification. */
+#define CRG_PORT_RESET_RETRY_MS 50
+#define CRG_PORT_RESET_MAX_RETRIES 40
+
+void crg_udc_port_retry_work(struct work_struct *work)
+{
+	struct crg_gadget_dev *crg_udc =
+		container_of(to_delayed_work(work), struct crg_gadget_dev,
+			     port_retry_work);
+	unsigned long flags;
+	int ret;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	if (!crg_udc->port_reset_pending && !crg_udc->deferred_port_events)
+		goto out;
+	if (crg_udc->resetting || crg_udc->suspended ||
+	    crg_udc->shutting_down || !crg_udc->pullup_on ||
+	    !crg_udc->gadget_driver || !crg_udc->async_callbacks)
+		goto out;
+	/* Reset work replays deferred events after the callback. A controller
+	 * reconstruction is different: retry once its register state is stable.
+	 */
+	if (crg_udc->controller_recovering) {
+		mod_delayed_work(system_wq, &crg_udc->port_retry_work,
+				 msecs_to_jiffies(CRG_PORT_RESET_RETRY_MS));
+		goto out;
+	}
+	if (crg_udc->port_reset_pending &&
+	    crg_udc->port_reset_retry_count++ >= CRG_PORT_RESET_MAX_RETRIES) {
+		bool notify_reset = crg_udc->device_state == USB_STATE_CONFIGURED;
+
+		dev_err(crg_udc->dev,
+			"port reset did not become ready; resetting controller\n");
+		ret = crg_udc_recover_port_failure(crg_udc, &flags,
+						   notify_reset);
+		if (ret != -ECONNRESET)
+			dev_err(crg_udc->dev,
+				"port-reset recovery returned %d\n", ret);
+		goto out;
+	}
+	ret = crg_udc_handle_port_status(crg_udc, &flags);
+	if (ret && ret != -ECONNRESET)
+		dev_dbg(crg_udc->dev, "port reset poll: %d\n", ret);
+out:
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+}
+
+int crg_udc_handle_port_status(struct crg_gadget_dev *crg_udc,
+			       unsigned long *flags)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	bool was_connected = crg_udc->connected;
+	u32 portsc_val;
+	u32 tmp;
+	u32 speed;
+	int ret;
+
+	/* Snapshot and acknowledge every W1C status bit exactly once. */
+	portsc_val = readl(&uccr->portsc);
+	trace_crg_udc_portsc(portsc_val);
+	writel(portsc_val & CRG_U3DC_PORTSC_W1C_MASK, &uccr->portsc);
+
+	/* A new port reset must not tear down endpoints from inside a gadget
+	 * reset callback. Preserve the acknowledged W1C events for replay.
+	 */
+	if (crg_udc->reset_callback_active || crg_udc->suspended ||
+	    crg_udc->shutting_down || crg_udc->controller_recovering) {
+		if (crg_udc->suspended || crg_udc->shutting_down)
+			return -ESHUTDOWN;
+		crg_udc->deferred_port_events |=
+			portsc_val & CRG_U3DC_PORTSC_W1C_MASK;
+		if (portsc_val & CRG_U3DC_PORTSC_PRC)
+			crg_udc->port_reset_pending = true;
+		return 0;
+	}
+
+	portsc_val |= crg_udc->deferred_port_events;
+	crg_udc->deferred_port_events = 0;
+
+	/* Process link changes before any disconnect path can reset the UDC. */
+	if (portsc_val & CRG_U3DC_PORTSC_PLC) {
+		tmp = readl(&uccr->portsc);
+		if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) == 0xf) {
+			tmp &= ~CRG_U3DC_PORTSC_W1C_MASK;
+			FIELD_MODIFY(CRG_U3DC_PORTSC_PLS_MASK, &tmp, 0x0);
+			tmp |= CRG_U3DC_PORTSC_LWS;
+			writel(tmp, &uccr->portsc);
+		}
+	}
+
+	/* A reset may finish before the port is ready; retry on later changes. */
+	if (portsc_val & CRG_U3DC_PORTSC_PRC) {
+		if (!crg_udc->port_reset_pending)
+			crg_udc->port_reset_retry_count = 0;
+		crg_udc->port_reset_pending = true;
+	}
+
+	/* Retry even when no new W1C event was raised: PRC describes when
+	 * the reset was detected, not when PED becomes ready.
+	 */
+	if (crg_udc->port_reset_pending) {
+		tmp = readl(&uccr->portsc);
+		if (tmp & CRG_U3DC_PORTSC_PRC) {
+			dev_dbg(crg_udc->dev, "PRC is still set\n");
+		} else if (tmp & CRG_U3DC_PORTSC_PR) {
+			/* first port status change event for port reset */
+			dev_dbg(crg_udc->dev,
+				"PRC is not set, but PR is set\n");
+		} else {
+			if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) != 0 ||
+			    !(tmp & CRG_U3DC_PORTSC_PED)) {
+				dev_dbg(crg_udc->dev, "portsc=0x%x no PED\n",
+					tmp);
+				goto prc_done;
+			}
+
+			switch (FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, tmp)) {
+			case CRG_U3DC_PORTSC_SPEED_SSP:
+				speed = USB_SPEED_SUPER_PLUS;
+				break;
+			case CRG_U3DC_PORTSC_SPEED_SS:
+				speed = USB_SPEED_SUPER;
+				break;
+			case CRG_U3DC_PORTSC_SPEED_HS:
+				speed = USB_SPEED_HIGH;
+				break;
+
+			case CRG_U3DC_PORTSC_SPEED_FS:
+				speed = USB_SPEED_FULL;
+				break;
+
+			case CRG_U3DC_PORTSC_SPEED_LS:
+				speed = USB_SPEED_LOW;
+				break;
+			default:
+				speed = USB_SPEED_UNKNOWN;
+				goto prc_done;
+			}
+
+			/* All prerequisites are met; the deferred reset is consumed. */
+			crg_udc->port_reset_pending = false;
+
+			if (crg_udc->device_state >= USB_STATE_DEFAULT) {
+				/*
+				 * A single enumeration commonly involves two
+				 * bus resets (an initial one, then another
+				 * after SET_ADDRESS before SET_CONFIGURATION).
+				 * By the second reset device_state is already
+				 * USB_STATE_ADDRESS, which is >= DEFAULT but
+				 * has no active configuration to tear down.
+				 * Only notify composite when a configuration
+				 * was actually active: otherwise this races
+				 * with the SET_CONFIGURATION that's about to
+				 * arrive, since the deferred work below can
+				 * run after fsg_set_alt() has already raised
+				 * its own FSG_STATE_CONFIG_CHANGE exception
+				 * and reset_config()'s ->disable() would
+				 * overwrite that exception's argument back to
+				 * NULL before the function's thread consumes
+				 * it, silently skipping the real enable.
+				 */
+				bool was_configured = crg_udc->device_state ==
+						       USB_STATE_CONFIGURED;
+				bool notify_reset = was_configured &&
+					crg_udc->async_callbacks &&
+					crg_udc->gadget_driver;
+				int reinit_ret;
+
+				reinit_ret = crg_udc_reinit(crg_udc, notify_reset);
+				if (reinit_ret) {
+					dev_err(crg_udc->dev,
+						"bus reset reinit failed: %d\n",
+						reinit_ret);
+					/* SWRST is required before releasing DMA buffers.
+					 * Do not resume processing the stale port event.
+					 */
+					if (reinit_ret == -ETIMEDOUT ||
+					    reinit_ret == -EIO)
+						return crg_udc_recover_port_failure(
+							crg_udc, flags, notify_reset);
+					return reinit_ret;
+				}
+				if (notify_reset && crg_udc->bus_reset_pending)
+					schedule_work(&crg_udc->udc_reset_work);
+			}
+
+			crg_udc->gadget.speed = speed;
+
+			ret = crg_udc_update_ep0_maxpacketsize(crg_udc);
+			if (ret) {
+				dev_err(crg_udc->dev,
+					"failed to configure EP0 maxpacket: %d\n", ret);
+				return crg_udc_recover_port_failure(
+					crg_udc, flags,
+					crg_udc->bus_reset_pending ||
+					crg_udc->device_state == USB_STATE_CONFIGURED);
+			}
+
+			crg_udc->connected = 1;
+
+			if (crg_udc->device_state < USB_STATE_RECONNECTING)
+				crg_udc_enable_setup(crg_udc);
+		}
+	}
+prc_done:
+	/* handle Port Reset end */
+
+	/* Port-enable and power changes can also complete initialization. */
+	if (portsc_val & (CRG_U3DC_PORTSC_CSC |
+			 CRG_U3DC_PORTSC_PEC |
+			 CRG_U3DC_PORTSC_PPC)) {
+		tmp = readl(&uccr->portsc);
+		if ((tmp & (CRG_U3DC_PORTSC_CCS | CRG_U3DC_PORTSC_PP)) ==
+		    (CRG_U3DC_PORTSC_CCS | CRG_U3DC_PORTSC_PP)) {
+			/* Do not configure EP0 until the pending port reset is handled. */
+			if (crg_udc->port_reset_pending)
+				goto csc_done;
+			if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) != 0 ||
+			    (!(tmp & CRG_U3DC_PORTSC_PED)))
+				goto csc_done;
+
+			switch (FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, tmp)) {
+			case CRG_U3DC_PORTSC_SPEED_SSP:
+				speed = USB_SPEED_SUPER_PLUS;
+				break;
+			case CRG_U3DC_PORTSC_SPEED_SS:
+				speed = USB_SPEED_SUPER;
+				break;
+			case CRG_U3DC_PORTSC_SPEED_HS:
+				speed = USB_SPEED_HIGH;
+				break;
+
+			case CRG_U3DC_PORTSC_SPEED_FS:
+				speed = USB_SPEED_FULL;
+				break;
+
+			case CRG_U3DC_PORTSC_SPEED_LS:
+				speed = USB_SPEED_LOW;
+				break;
+			default:
+				speed = USB_SPEED_UNKNOWN;
+				goto csc_done;
+			}
+			crg_udc->gadget.speed = speed;
+
+			ret = crg_udc_update_ep0_maxpacketsize(crg_udc);
+			if (ret) {
+				dev_err(crg_udc->dev,
+					"failed to configure EP0 maxpacket: %d\n", ret);
+				return crg_udc_recover_port_failure(
+					crg_udc, flags,
+					crg_udc->bus_reset_pending ||
+					crg_udc->device_state == USB_STATE_CONFIGURED);
+			}
+
+			crg_udc->connected = 1;
+
+			if (crg_udc->device_state < USB_STATE_RECONNECTING)
+				crg_udc_enable_setup(crg_udc);
+
+		} else if ((portsc_val & (CRG_U3DC_PORTSC_CSC |
+					  CRG_U3DC_PORTSC_PPC)) &&
+			   (!(tmp & CRG_U3DC_PORTSC_CCS) ||
+			    !(tmp & CRG_U3DC_PORTSC_PP))) {
+			bool cable_connected;
+			int ccs_drop_ignore = 0;
+
+			if ((FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) == 0x0) &&
+			    (FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, tmp) <
+			     CRG_U3DC_PORTSC_SPEED_SS)) {
+				ccs_drop_ignore = 1;
+				dev_err(crg_udc->dev, "ccs glitch detect on HS/FS!\n");
+			}
+
+			cable_connected = crg_udc_vbus_connected(crg_udc);
+			/* A missing port-power bit overrides the HS/FS CCS glitch rule. */
+			if (!cable_connected)
+				ccs_drop_ignore = 0;
+			if (!ccs_drop_ignore)
+				crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+			if ((!ccs_drop_ignore || !cable_connected) &&
+			    was_connected && crg_udc->connected) {
+				u32 generation = crg_udc->udc_event[0].generation;
+
+				/* The callback drops udc_lock: mark link down first. */
+				crg_udc->connected = 0;
+				crg_udc_notify_disconnect(crg_udc, flags);
+				if (crg_udc->resetting ||
+				    generation != crg_udc->udc_event[0].generation)
+					return -ECONNRESET;
+
+				/* Another thread may have handled a new connection. */
+				if (crg_udc->connected)
+					goto csc_done;
+
+				/* A new CCS=1 alone is not proof that software was reset. */
+				cable_connected = crg_udc_vbus_connected(crg_udc);
+			}
+			if (cable_connected && !ccs_drop_ignore) {
+				bool notify_reset = crg_udc->device_state ==
+							 USB_STATE_CONFIGURED;
+				int ret;
+
+				ret = crg_udc_reinit(crg_udc, false);
+				if (ret == -ETIMEDOUT || ret == -EIO)
+					return crg_udc_recover_port_failure(
+						crg_udc, flags, notify_reset);
+				if (ret)
+					return ret;
+				crg_udc->connected = 0;
+			} else if (!cable_connected) {
+				dev_dbg(crg_udc->dev,
+					"cable disconnected, rst controller\n");
+				/* Keep RUN and port interrupts enabled for the next cable. */
+				crg_udc_recover_link(crg_udc, flags, false);
+				return -ECONNRESET;
+			}
+		}
+	}
+csc_done:
+	/* PRC was already acknowledged. It may never interrupt us again if
+	 * PED becomes ready without a further CSC/PLC transition.
+	 */
+	if (crg_udc->port_reset_pending && !crg_udc->resetting &&
+	    !crg_udc->shutting_down && !crg_udc->suspended &&
+	    crg_udc->pullup_on && crg_udc->gadget_driver &&
+	    crg_udc->async_callbacks &&
+	    crg_udc->port_reset_retry_count < CRG_PORT_RESET_MAX_RETRIES)
+		mod_delayed_work(system_wq, &crg_udc->port_retry_work,
+				 msecs_to_jiffies(CRG_PORT_RESET_RETRY_MS));
+	return 0;
+}
+
+/*
+ * A giveback drops udc_lock; its callback can reset, stall, or replace
+ * the control transfer. Never advance an old EP0 transaction afterwards.
+ */
+static void crg_udc_ep0_complete_if_current(struct crg_gadget_dev *crg_udc,
+					    struct crg_udc_ep *ep, u32 generation,
+					    u32 event_generation, u8 setup_tag,
+					    enum crg_setup_status setup_status,
+					    unsigned long *flags)
+{
+	if (crg_udc->resetting || crg_udc->bus_reset_pending ||
+	    crg_udc->port_reset_pending ||
+	    crg_udc->device_state == USB_STATE_RECONNECTING ||
+	    crg_udc->ep0_state != EP0_STATE_IDLE ||
+	    ep->ep_state != EP_STATE_RUNNING || ep->nuking || !ep->desc ||
+	    !usb_endpoint_xfer_control(ep->desc) ||
+	    ep->generation != generation ||
+	    crg_udc->udc_event[0].generation != event_generation ||
+	    crg_udc->setup_tag != setup_tag ||
+	    crg_udc->setup_status != setup_status)
+		return;
+
+	crg_udc_ep0_req_complete(ep, flags);
+}
+
+int crg_udc_handle_xfer_event(struct crg_gadget_dev *crg_udc,
+			      struct crg_event_trb *event, unsigned long *flags)
+{
+	u32 dw3 = le32_to_cpu(event->dw3);
+	u8 dci = FIELD_GET(EVE_TRB_ENDPOINT_ID_MASK, dw3);
+	u16 comp_code =
+		FIELD_GET(EVE_TRB_COMPL_CODE_MASK, le32_to_cpu(event->dw2));
+	struct crg_udc_ep *udc_ep_ptr;
+	struct crg_udc_request *udc_req_ptr;
+	struct crg_transfer_trb *trb;
+	bool trbs_dequeued = false;
+	u32 ep_generation = 0, event_generation = 0;
+	enum crg_setup_status setup_status = WAIT_FOR_SETUP;
+	u8 setup_tag = 0;
+	u32 residual;
+
+	if (dci >= CRG_NUM_EP_CX || dci == 1)
+		return -EINVAL;
+
+	udc_ep_ptr = &crg_udc->udc_ep[dci];
+	if (dci == 0) {
+		ep_generation = udc_ep_ptr->generation;
+		event_generation = crg_udc->udc_event[0].generation;
+		setup_tag = crg_udc->setup_tag;
+		setup_status = crg_udc->setup_status;
+	}
+	/* Recovery owns the mappings while DMA state is uncertain. */
+	if (crg_udc->endpoint_recovery_pending || crg_udc->port_reset_pending ||
+	    udc_ep_ptr->halt_failed)
+		return 0;
+	if (!udc_ep_ptr->first_trb ||
+	    crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_DISABLED)
+		return -ENODEV;
+
+	/* EP0 transfer events from an abort must not affect the new ring. */
+	if (dci == 0 && crg_udc->ep0_state != EP0_STATE_IDLE)
+		return 0;
+
+	if (comp_code != CMPL_CODE_STOPPED &&
+	    comp_code != CMPL_CODE_STOPPED_LENGTH_INVALID &&
+	    comp_code != CMPL_CODE_DISABLED &&
+	    comp_code != CMPL_CODE_DISABLED_LENGTH_INVALID &&
+	    comp_code != CMPL_CODE_HALTED &&
+	    comp_code != CMPL_CODE_HALTED_LENGTH_INVALID &&
+	    comp_code != CMPL_CODE_SETUP_TAG_MISMATCH) {
+		/*
+		 * Only live requests may advance the transfer-ring dequeue
+		 * pointer. EP0 data and status requests both track their TRBs.
+		 */
+		if (list_empty(&udc_ep_ptr->queue) ||
+		    is_request_dequeued(crg_udc, udc_ep_ptr, event))
+			return 0;
+
+		crg_udc_update_dequeue_pt(event, udc_ep_ptr);
+	}
+
+	switch (comp_code) {
+	case CMPL_CODE_SUCCESS:
+		trb = crg_udc_tran_trb_dma_to_virt(
+			udc_ep_ptr,
+			(u64)le32_to_cpu(event->dw0) |
+				((u64)le32_to_cpu(event->dw1) << 32));
+		if (!trb || list_empty(&udc_ep_ptr->queue))
+			break;
+
+		/*
+		 * A control transfer has separate data-stage and status-stage
+		 * TRBs.  The status request is an internal zero-length request
+		 * and its successful completion must not go through
+		 * crg_udc_build_td().  Doing that attempts to queue another
+		 * control TD while the status request is already on the EP0
+		 * software queue, leaving it stuck in -EINPROGRESS.  The next
+		 * SETUP then sees a non-empty queue and enumeration fails.
+		 */
+		if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(trb->dw3)) ==
+		    TRB_TYPE_XFER_STATUS_STAGE) {
+			udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+						       struct crg_udc_request,
+						       queue);
+			if (dci == 0) {
+				udc_req_ptr->usb_req.actual = 0;
+				trbs_dequeued = true;
+				crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
+				crg_udc_ep0_complete_if_current(crg_udc, udc_ep_ptr,
+						 ep_generation, event_generation,
+						 setup_tag, setup_status, flags);
+			} else {
+				dev_err(crg_udc->dev,
+					"status-stage TRB on endpoint %u\n",
+					dci);
+			}
+			break;
+		}
+
+		if (!FIELD_GET(TRB_CHAIN_BIT_MASK, le32_to_cpu(trb->dw3))) {
+			udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+						       struct crg_udc_request,
+						       queue);
+			trbs_dequeued = true;
+			udc_req_ptr->td_pending = false;
+
+			if (udc_req_ptr->all_trbs_queued) {
+				udc_req_ptr->usb_req.actual =
+					udc_req_ptr->usb_req.length -
+					udc_req_ptr->buff_len_left;
+				residual = FIELD_GET(EVE_TRB_TRAN_LEN_MASK,
+						     le32_to_cpu(event->dw2));
+				if (residual > udc_req_ptr->usb_req.actual)
+					residual = 0;
+				udc_req_ptr->usb_req.actual -= residual;
+				crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
+
+				if (dci == 0)
+					crg_udc_ep0_complete_if_current(crg_udc, udc_ep_ptr,
+						 ep_generation, event_generation,
+						 setup_tag, setup_status, flags);
+			} else {
+				int ret;
+
+				ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+				if (ret == -ENOSPC || ret == -EBUSY) {
+					/* A previous TD completed, but this next TD
+					 * could not yet be published. Retry later.
+					 */
+					udc_ep_ptr->tran_ring_full = true;
+				} else if (ret) {
+					udc_req_ptr->usb_req.actual =
+						udc_req_ptr->usb_req.length -
+						udc_req_ptr->buff_len_left;
+					crg_udc_req_done(udc_ep_ptr, udc_req_ptr,
+							 ret);
+				}
+			}
+		}
+		break;
+
+	case CMPL_CODE_SHORT_PKT:
+		if (list_empty(&udc_ep_ptr->queue))
+			break;
+
+		udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+					       struct crg_udc_request, queue);
+		trb = crg_udc_tran_trb_dma_to_virt(
+			udc_ep_ptr,
+			(u64)le32_to_cpu(event->dw0) |
+			((u64)le32_to_cpu(event->dw1) << 32));
+		if (!trb)
+			break;
+
+		residual = FIELD_GET(EVE_TRB_TRAN_LEN_MASK,
+				     le32_to_cpu(event->dw2));
+		udc_req_ptr->usb_req.actual =
+			crg_udc_actual_before(udc_ep_ptr, udc_req_ptr, trb) +
+			FIELD_GET(TRB_TRANSFER_LEN_MASK,
+					  le32_to_cpu(trb->dw2)) -
+			min_t(u32, residual,
+			      FIELD_GET(TRB_TRANSFER_LEN_MASK,
+						le32_to_cpu(trb->dw2)));
+		udc_req_ptr->usb_req.actual = min(udc_req_ptr->usb_req.actual,
+						    udc_req_ptr->usb_req.length);
+
+		/*
+		 * Advance past the completed request before giveback drops
+		 * udc_lock and a completion callback can modify the queue.
+		 */
+		trbs_dequeued = true;
+		crg_udc_advance_dequeue_pt(udc_ep_ptr);
+		if (udc_req_ptr->usb_req.short_not_ok &&
+		    usb_endpoint_dir_out(udc_ep_ptr->desc))
+			crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EREMOTEIO);
+		else
+			crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
+		if (dci == 0)
+			crg_udc_ep0_complete_if_current(crg_udc, udc_ep_ptr,
+						 ep_generation, event_generation,
+						 setup_tag, setup_status, flags);
+		break;
+
+	case CMPL_CODE_PROTOCOL_STALL:
+	case CMPL_CODE_TRB_STALL:
+		if (!list_empty(&udc_ep_ptr->queue)) {
+			udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+						       struct crg_udc_request,
+						       queue);
+			crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EPIPE);
+			trbs_dequeued = true;
+		}
+		if (dci == 0)
+			crg_udc->setup_status = WAIT_FOR_SETUP;
+		break;
+
+	case CMPL_CODE_SETUP_TAG_MISMATCH:
+		/*
+		 * EP0 SETUP changes are serialized by the SETUP event handler.
+		 * When a new SETUP arrives, crg_udc_abort_ep0_for_setup() stops
+		 * EP0, gives back the old request, resets the EP0 ring, and then
+		 * accepts the new SETUP. A SETUP_TAG_MISMATCH event generated
+		 * afterwards therefore belongs to the aborted control transfer.
+		 *
+		 * Do not complete the current EP0 request or change setup_status
+		 * here: doing so could operate on the new control transfer and
+		 * recreate the EP0 queue/state race this path is intended to avoid.
+		 */
+		if (dci == 0) {
+			dev_dbg(crg_udc->dev,
+				"ignoring stale EP0 SETUP_TAG_MISMATCH\n");
+			break;
+		}
+
+		/* This completion code is meaningful only for EP0. */
+		dev_warn(crg_udc->dev,
+			 "SETUP_TAG_MISMATCH on non-EP0 endpoint %u\n", dci);
+		break;
+
+	case CMPL_CODE_BABBLE_DETECTED_ERR:
+	case CMPL_CODE_DATA_BUFFER_ERR:
+	case CMPL_CODE_INVALID_STREAM_TYPE_ERR:
+	case CMPL_CODE_RING_UNDERRUN:
+	case CMPL_CODE_RING_OVERRUN:
+	case CMPL_CODE_EVENT_RING_FULL_ERR:
+	case CMPL_CODE_ISOCH_BUFFER_OVERRUN:
+	case CMPL_CODE_USB_TRANS_ERR:
+	case CMPL_CODE_TRB_ERR:
+	default:
+		/* Unknown codes are not proof that endpoint DMA has stopped. */
+		dev_err(crg_udc->dev, "transfer error, completion code 0x%x\n",
+			comp_code);
+		if (dci == 0) {
+			int ret;
+
+			/* EP0 must stop DMA before completing a failed transfer. */
+			if (!list_empty(&udc_ep_ptr->queue)) {
+				udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+							       struct crg_udc_request,
+							       queue);
+				if (udc_req_ptr->usb_req.status == -EINPROGRESS)
+					udc_req_ptr->usb_req.status = -EIO;
+			}
+			ret = crg_udc_abort_ep0_for_setup(crg_udc);
+			if (ret)
+				dev_err(crg_udc->dev,
+					"failed to stop EP0 after transfer error: %d\n",
+					ret);
+		} else {
+			int ret = crg_udc_stop_error_ep(crg_udc, dci);
+
+			if (ret)
+				dev_err(crg_udc->dev,
+					"failed to stop EP%u after transfer error: %d\n",
+					dci, ret);
+		}
+		break;
+
+	case CMPL_CODE_STOPPED:
+	case CMPL_CODE_STOPPED_LENGTH_INVALID:
+	case CMPL_CODE_HALTED:
+	case CMPL_CODE_HALTED_LENGTH_INVALID:
+	case CMPL_CODE_DISABLED:
+	case CMPL_CODE_DISABLED_LENGTH_INVALID:
+		break;
+	}
+
+	/* A completed TD frees room, but this event may also have published
+	 * the next TD and filled the ring again.  Recompute from the current
+	 * pointers: unconditionally clearing tran_ring_full loses that state.
+	 */
+	if (trbs_dequeued && dci != 0 &&
+	    udc_ep_ptr->ep_state == EP_STATE_RUNNING &&
+	    !crg_udc->resetting && !crg_udc->port_reset_pending &&
+	    !udc_ep_ptr->halt_failed) {
+		u32 ring_size = udc_ep_ptr->last_trb -
+				udc_ep_ptr->first_trb + 1;
+		u32 room = crg_udc_room_on_ring(crg_udc, ring_size,
+					udc_ep_ptr->first_trb,
+					udc_ep_ptr->enq_pt,
+					udc_ep_ptr->deq_pt);
+
+		udc_ep_ptr->tran_ring_full = !room;
+		if (room && crg_udc_queue_pending_trbs(udc_ep_ptr))
+			udc_ep_ptr->tran_ring_full = true;
+	}
+
+	return 0;
+}
+
+void crg_udc_dispatch_setup(struct crg_gadget_dev *crg_udc,
+			    struct usb_ctrlrequest *setup, u8 setup_tag,
+			    unsigned long *flags)
+{
+	struct crg_udc_ep *ep0 = &crg_udc->udc_ep[0];
+
+	/* A completed control transfer can leave old TRBs on the ring even
+	 * after its software queue becomes empty. A new SETUP aborts those
+	 * TRBs: stop and reinitialize EP0 before accepting another request.
+	 */
+	if (crg_udc->setup_status != WAIT_FOR_SETUP ||
+	    crg_udc->ep0_state != EP0_STATE_IDLE ||
+	    ep0->ep_state != EP_STATE_RUNNING ||
+	    !list_empty(&ep0->queue) || ep0->enq_pt != ep0->deq_pt ||
+	    ep0->tran_ring_full) {
+		/* Do not submit TRBs if STOP_EP failed. */
+		if (crg_udc_abort_ep0_for_setup(crg_udc))
+			return;
+	}
+
+	crg_udc_handle_setup_pkt(crg_udc, setup, setup_tag, flags);
+}
+
+int crg_udc_handle_event(struct crg_gadget_dev *crg_udc,
+			 struct crg_event_trb *event, unsigned long *flags)
+{
+	int ret;
+
+	trace_crg_udc_event_trb(event);
+
+	switch (FIELD_GET(EVE_TRB_TYPE_MASK, le32_to_cpu(event->dw3))) {
+	case TRB_TYPE_EVT_PORT_STATUS_CHANGE:
+		if (crg_udc->device_state == USB_STATE_RECONNECTING) {
+			crg_udc->portsc_on_reconnecting = 1;
+			break;
+		}
+		/* 1.Port Reset  2.Port Connection Change 3.Port Link Change */
+		ret = crg_udc_handle_port_status(crg_udc, flags);
+		if (ret)
+			return ret;
+
+		break;
+	case TRB_TYPE_EVT_TRANSFER:
+		if (crg_udc->device_state < USB_STATE_RECONNECTING) {
+			dev_err(crg_udc->dev,
+				"Xfer compl event rcved when dev state=%d !\n",
+				crg_udc->device_state);
+			break;
+		}
+
+		ret = crg_udc_handle_xfer_event(crg_udc, event, flags);
+		if (ret)
+			return ret;
+		break;
+	case TRB_TYPE_EVT_SETUP_PKT: {
+		struct usb_ctrlrequest setup;
+		u32 dw0 = le32_to_cpu(event->dw0);
+		u32 dw1 = le32_to_cpu(event->dw1);
+		u8 setup_tag;
+
+		BUILD_BUG_ON(sizeof(struct usb_ctrlrequest) != 8);
+		setup.bRequestType = dw0 & 0xff;
+		setup.bRequest = (dw0 >> 8) & 0xff;
+		setup.wValue = cpu_to_le16((dw0 >> 16) & 0xffff);
+		setup.wIndex = cpu_to_le16(dw1 & 0xffff);
+		setup.wLength = cpu_to_le16((dw1 >> 16) & 0xffff);
+
+		setup_tag = FIELD_GET(EVE_TRB_SETUP_TAG_MASK,
+				      le32_to_cpu(event->dw3));
+		if (crg_udc->device_state == USB_STATE_RECONNECTING) {
+			/* EP0 is not ready until the old event ring is drained. */
+			crg_udc->pending_setup = setup;
+			crg_udc->pending_setup_tag = setup_tag;
+			crg_udc->pending_setup_valid = true;
+			break;
+		}
+		crg_udc_dispatch_setup(crg_udc, &setup, setup_tag, flags);
+
+		break;
+	}
+	default:
+		dev_dbg(crg_udc->dev, "unexpected TRB_TYPE = 0x%lx",
+			FIELD_GET(EVE_TRB_TYPE_MASK, le32_to_cpu(event->dw3)));
+		break;
+	}
+
+	return 0;
+}
+
+/*
+ * Invoke the bound gadget driver's ->disconnect() callback, if any.
+ *
+ * The gadget disconnect callback may be called in interrupt context.
+ * Release only udc_lock while keeping local IRQs disabled.
+ */
+void crg_udc_notify_disconnect(struct crg_gadget_dev *crg_udc,
+			       unsigned long *flags)
+{
+	struct usb_gadget_driver *driver = crg_udc->gadget_driver;
+
+	/* Report link loss even when there is no gadget callback. */
+	usb_gadget_set_state(&crg_udc->gadget, USB_STATE_NOTATTACHED);
+
+	if (!crg_udc->async_callbacks || !driver || !driver->disconnect)
+		return;
+
+	/*
+	 * The unbind path disables async callbacks and synchronizes the IRQ
+	 * before freeing the gadget driver. Keep this pointer across unlock.
+	 */
+	crg_udc_callback_enter(crg_udc);
+	spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+	driver->disconnect(&crg_udc->gadget);
+	spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+	crg_udc_callback_exit(crg_udc);
+}
+
+int crg_udc_ep_halt(struct crg_udc_ep *udc_ep_ptr, int halt, int ignore_wedge,
+		    unsigned long *flags)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	enum crg_ep_state ep_state;
+	bool reset_seq_only = false;
+	bool recovering;
+	int do_halt, ret;
+	u32 param0;
+	u32 tmp;
+
+	if (crg_udc->resetting || crg_udc->endpoint_recovery_pending)
+		return -ESHUTDOWN;
+	if (!udc_ep_ptr->desc) {
+		dev_err(crg_udc->dev, "NULL desc\n");
+		return -EINVAL;
+	}
+	/* Failed disable is only recoverable by disable retry or reset. */
+	if (udc_ep_ptr->disable_failed)
+		return -ESHUTDOWN;
+
+	if (usb_endpoint_xfer_isoc(udc_ep_ptr->desc)) {
+		dev_err(crg_udc->dev, "Isoc ep, halt not supported\n");
+		return -EOPNOTSUPP;
+	}
+
+	if (udc_ep_ptr->dci == 0) {
+		/* EP0 stalls with a STATUS-stage TRB, not SET_HALT. */
+		if (!halt)
+			return -EOPNOTSUPP;
+
+		return crg_udc_set_ep0_halt(crg_udc);
+	}
+
+	/* A giveback can drop udc_lock while ep_nuke() still owns requests.
+	 * Never resume DMA until the entire cancellation drain is finished.
+	 */
+	if (udc_ep_ptr->nuking && !halt)
+		return -EBUSY;
+
+	/* Do not resume hardware while dequeue() still owns its ring. */
+	if (udc_ep_ptr->dequeue_paused && !halt)
+		return -EBUSY;
+
+	/* A failed halt cannot be cleared until DMA is safely stopped. */
+	if (udc_ep_ptr->halt_failed && !halt)
+		return -EIO;
+	recovering = udc_ep_ptr->halt_failed;
+
+	ep_state = crg_udc_get_ep_state(crg_udc, udc_ep_ptr->dci);
+
+	if (ep_state == EP_STATE_DISABLED)
+		return 0;
+
+	if (ep_state == EP_STATE_HALTED) {
+		if (halt != 0)
+			return 0;
+
+		/* want unhalt an halted ep */
+		if (udc_ep_ptr->wedge && !ignore_wedge) {
+			do_halt = -1;
+			reset_seq_only = true;
+		} else {
+			do_halt = 0;
+		}
+
+	} else {
+		/* ep state == running or stopped */
+		if (halt != 0) {
+			/* want halt a running ep */
+			do_halt = 1;
+		} else {
+			/* reset a running ep */
+			do_halt = 0;
+			reset_seq_only = true;
+		}
+	}
+
+	trace_crg_udc_ep_halt(udc_ep_ptr, do_halt == 1, udc_ep_ptr->wedge);
+
+	param0 = BIT(udc_ep_ptr->dci);
+	if (do_halt == 1) {
+		/* Stop hardware DMA before updating the ring or giving back. */
+		ret = crg_udc_issue_command(crg_udc, CRG_CMD_SET_HALT,
+					    param0, 0);
+		if (!ret)
+			ret = readl_poll_timeout_atomic(&uccr->ep_running, tmp,
+							       !(tmp & param0), 10, 10000);
+		if (ret) {
+			/* The controller might still own the mapped buffers. */
+			udc_ep_ptr->halt_failed = true;
+			udc_ep_ptr->ep_state = EP_STATE_STOPPED;
+			dev_err(crg_udc->dev,
+				"EP%d halt failed (%d); DMA mappings retained\n",
+				udc_ep_ptr->dci, ret);
+			return ret;
+		}
+		udc_ep_ptr->halt_failed = false;
+		udc_ep_ptr->dequeue_paused = false;
+
+		if (recovering || crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+			udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
+			udc_ep_ptr->tran_ring_full = false;
+		}
+		udc_ep_ptr->ep_state = EP_STATE_HALTED;
+		/*
+		 * A failed stop retained old DMA mappings even at HS/FS.
+		 * After a confirmed retry, all those requests can be given back.
+		 */
+		if (recovering || crg_udc->gadget.speed >= USB_SPEED_SUPER)
+			crg_udc_ep_nuke(udc_ep_ptr, -ECONNRESET);
+	} else if (do_halt == 0) {
+		/* clearing ep halt state */
+		/* reset sequence number */
+		if (crg_udc_issue_command(crg_udc, CRG_CMD_RESET_SEQNUM, param0,
+					  0))
+			return -EIO;
+
+		if (!reset_seq_only) {
+			/* Clear halt for a halted EP. */
+			/* NOTE: we must CLEAR_HALT first,
+			 * then SET_TR_DQPTR
+			 */
+			ret = crg_udc_issue_command(crg_udc,
+						    CRG_CMD_CLEAR_HALT, param0, 0);
+			if (ret)
+				goto clear_halt_failed;
+			ret = crg_udc_epcx_update_dqptr(udc_ep_ptr);
+			if (ret)
+				goto clear_halt_failed;
+		}
+
+		udc_ep_ptr->wedge = 0;
+		udc_ep_ptr->ep_state = EP_STATE_RUNNING;
+		/* Requests queued during HALT have DMA mappings but no TRBs.
+		 * Build them before notifying the resumed controller.
+		 */
+		if (!list_empty(&udc_ep_ptr->queue)) {
+			u32 generation = udc_ep_ptr->generation;
+
+			ret = crg_udc_queue_pending_trbs(udc_ep_ptr);
+			if (ret == -ENOSPC || ret == -EBUSY)
+				udc_ep_ptr->tran_ring_full = true;
+			else if (ret)
+				return ret;
+
+			/* Pending-request giveback can drop udc_lock. */
+			if (crg_udc->resetting || crg_udc->port_reset_pending ||
+			    udc_ep_ptr->halt_failed ||
+			    udc_ep_ptr->generation != generation ||
+			    udc_ep_ptr->ep_state != EP_STATE_RUNNING ||
+			    !udc_ep_ptr->desc)
+				return -ESHUTDOWN;
+			crg_udc_knock_doorbell(crg_udc, udc_ep_ptr->dci);
+		}
+	} else {
+		/* wedged EP deny CLEAR HALT */
+		dev_dbg(crg_udc->dev, "wedged EP deny CLEAR HALT dci = %d\n",
+			udc_ep_ptr->dci);
+		/* reset sequence number */
+		if (reset_seq_only &&
+		    crg_udc_issue_command(crg_udc, CRG_CMD_RESET_SEQNUM, param0,
+					  0))
+			return -EIO;
+	}
+
+	return 0;
+
+clear_halt_failed:
+	/*
+	 * CLEAR_HALT may have taken effect even if the following command
+	 * failed. Block software queues and retain DMA mappings until a
+	 * confirmed SET_HALT retry, endpoint disable, or controller reset.
+	 */
+	udc_ep_ptr->halt_failed = true;
+	udc_ep_ptr->dequeue_paused = false;
+	udc_ep_ptr->ep_state = EP_STATE_STOPPED;
+	dev_err(crg_udc->dev,
+		"EP%d CLEAR_HALT recovery failed (%d); DMA mappings retained\n",
+		udc_ep_ptr->dci, ret);
+	return ret;
+}
+
+enum crg_ep_state crg_udc_get_ep_state(struct crg_gadget_dev *crg_udc, int dci)
+{
+	struct crg_udc_ep *udc_ep_ptr;
+
+	/*
+	 * DCI 0 is ep0; DCI 1 is unused in the Corigine ep-context layout
+	 * (ep contexts start from DCI 2 for ep1in/ep1out). This is why the
+	 * ep-context array is indexed as p_epcx[dci - 2] elsewhere.
+	 */
+	if (dci < 0 || dci >= CRG_NUM_EP_CX || dci == 1)
+		return EP_STATE_DISABLED;
+
+	udc_ep_ptr = &crg_udc->udc_ep[dci];
+
+	return udc_ep_ptr->ep_state;
+}
+
+u32 crg_udc_index2dci(u16 index)
+{
+	if (index == 0)
+		return 0;
+
+	return (index & USB_ENDPOINT_NUMBER_MASK) * 2 +
+	       ((index & USB_DIR_IN) ? 0 : 1);
+}
+
+int crg_udc_init_device_context(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	int ret;
+	u32 buff_length;
+	dma_addr_t mapping;
+
+	/* ep0 is not included in ep contexts in crg udc */
+	buff_length = (CRG_NUM_EP_CX - 2) * sizeof(struct crg_ep_cx);
+
+	if (!crg_udc->ep_cx.cpu_addr) {
+		crg_udc->ep_cx.cpu_addr = dma_alloc_coherent(
+			crg_udc->dev, buff_length, &mapping, GFP_ATOMIC);
+		if (!crg_udc->ep_cx.cpu_addr) {
+			ret = -ENOMEM;
+			return ret;
+		}
+	} else {
+		mapping = crg_udc->ep_cx.dma_addr;
+	}
+
+	crg_udc->p_epcx = crg_udc->ep_cx.cpu_addr;
+	crg_udc->ep_cx.len = buff_length;
+	crg_udc->ep_cx.dma_addr = mapping;
+	memset(crg_udc->p_epcx, 0, buff_length);
+	/*
+	 * Ensure the endpoint contexts are initialized before making their
+	 * DMA address visible to the controller.
+	 */
+	dma_wmb();
+	/* hw ops DCBAPLO DCBAPHI */
+	writel(lower_32_bits(crg_udc->ep_cx.dma_addr), &uccr->dcbaplo);
+	writel(upper_32_bits(crg_udc->ep_cx.dma_addr), &uccr->dcbaphi);
+
+	return 0;
+}
+
+void crg_udc_init_ep_info(struct crg_gadget_dev *crg_udc)
+{
+	int i;
+
+	/* udc_ep[0] is reserved */
+	crg_ep_struct_setup(crg_udc, 0, NULL);
+
+	for (i = 1; i < CRG_NUM_EP_CX / 2; i++) {
+		char name[14];
+
+		snprintf(name, sizeof(name), "ep%din", i);
+		crg_ep_struct_setup(crg_udc, i * 2, name);
+		snprintf(name, sizeof(name), "ep%dout", i);
+		crg_ep_struct_setup(crg_udc, i * 2 + 1, name);
+	}
+}
+
+static bool is_request_dequeued(struct crg_gadget_dev *crg_udc,
+				struct crg_udc_ep *udc_ep,
+				struct crg_event_trb *event)
+{
+	struct crg_udc_request *udc_req;
+	u32 trb_pt_lo = le32_to_cpu(event->dw0);
+	u32 trb_pt_hi = le32_to_cpu(event->dw1);
+	u64 trb_addr = (u64)trb_pt_lo | ((u64)trb_pt_hi << 32);
+	struct crg_transfer_trb *trb_pt;
+
+	if (list_empty(&udc_ep->queue))
+		return true;
+
+	trb_pt = crg_udc_tran_trb_dma_to_virt(udc_ep, trb_addr);
+	if (!trb_pt)
+		return true;
+
+	/* The transfer event handler consumes the queue head only. Do not
+	 * accept an event for a later request and attribute it to the head.
+	 */
+	udc_req = list_first_entry(&udc_ep->queue, struct crg_udc_request, queue);
+	if (trb_pt == udc_req->first_trb || trb_pt == udc_req->last_trb)
+		return false;
+
+	return !(crg_udc_is_pointer_less_than(trb_pt, udc_req->last_trb,
+						      udc_ep) &&
+		 crg_udc_is_pointer_less_than(udc_req->first_trb, trb_pt,
+						     udc_ep));
+}
+
+void crg_udc_knock_doorbell(struct crg_gadget_dev *crg_udc, int dci)
+{
+	u32 tmp;
+	struct crg_uccr __iomem *uccr;
+
+	uccr = crg_udc->uccr;
+	/* Ensure everything is written before notifying the HW */
+	dma_wmb();
+
+	tmp = FIELD_PREP(CRG_U3DC_DB_TARGET_MASK, dci);
+	writel(tmp, &uccr->doorbell);
+
+	trace_crg_udc_doorbell(crg_udc, dci);
+}
+
+/*
+ * Complete a request and give it back to the gadget layer via the
+ * standard usb_gadget_giveback_request() helper.
+ *
+ * Must be called with udc_lock held and interrupts disabled.
+ *
+ * The request's ->complete() callback must not sleep. crg_udc_req_done() drops
+ * udc_lock before the giveback and re-acquires it before returning.
+ * Callers therefore cannot assume state protected by udc_lock is
+ * unchanged across this call.
+ */
+void crg_udc_req_done(struct crg_udc_ep *udc_ep,
+		      struct crg_udc_request *udc_req, int status)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+
+	if (likely(udc_req->usb_req.status == -EINPROGRESS))
+		udc_req->usb_req.status = status;
+
+	trace_crg_udc_request_giveback(udc_ep, udc_req);
+
+	list_del_init(&udc_req->queue);
+
+	/*
+	 * Only published payloads can still be owned by the DMA master.
+	 * Move the mapping off the request before giving it back. No CPU
+	 * access to the bounce pages is allowed while they are quarantined.
+	 */
+	if (crg_udc->dma_quiesce_failed && udc_req->bounce &&
+	    udc_req->first_trb) {
+		list_add_tail(&udc_req->bounce->quarantine,
+			      &crg_udc->dma_quarantine);
+		udc_req->bounce = NULL;
+		udc_req->usb_req.status = -ESHUTDOWN;
+		udc_req->usb_req.actual = 0;
+	} else {
+		crg_udc_unmap_request(udc_ep, udc_req, udc_req->usb_req.actual);
+	}
+
+	if (udc_req->usb_req.complete) {
+		crg_udc_callback_enter(crg_udc);
+		spin_unlock(&crg_udc->udc_lock);
+		usb_gadget_giveback_request(&udc_ep->usb_ep, &udc_req->usb_req);
+		spin_lock(&crg_udc->udc_lock);
+		crg_udc_callback_exit(crg_udc);
+	}
+}
+
+/*
+ * A transfer error must stop hardware DMA before giving requests back.
+ * Keep the endpoint blocked if the stop command fails, so callers cannot
+ * queue more TRBs while old requests still own DMA mappings.
+ */
+static int crg_udc_stop_error_ep(struct crg_gadget_dev *crg_udc, u8 dci)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[dci];
+	struct crg_udc_request *req;
+	bool isoc;
+	u32 tmp;
+	int ret;
+
+	if (crg_udc->resetting || !udc_ep_ptr->desc ||
+	    udc_ep_ptr->ep_state == EP_STATE_DISABLED)
+		return -ESHUTDOWN;
+
+	/* Preserve -EIO for the failed request, including after recovery. */
+	if (!list_empty(&udc_ep_ptr->queue)) {
+		req = list_first_entry(&udc_ep_ptr->queue,
+				       struct crg_udc_request, queue);
+		if (req->usb_req.status == -EINPROGRESS)
+			req->usb_req.status = -EIO;
+	}
+
+	isoc = usb_endpoint_xfer_isoc(udc_ep_ptr->desc);
+	if (udc_ep_ptr->ep_state != EP_STATE_HALTED) {
+		/* Block queues while stopping, including on command failure. */
+		udc_ep_ptr->ep_state = EP_STATE_STOPPED;
+		udc_ep_ptr->dequeue_paused = false;
+		ret = crg_udc_issue_command(crg_udc,
+				isoc ? CRG_CMD_STOP_EP : CRG_CMD_SET_HALT,
+				BIT(dci), 0);
+		if (ret)
+			goto stop_failed;
+	}
+
+	ret = readl_poll_timeout_atomic(&uccr->ep_running, tmp,
+					 !(tmp & BIT(dci)), 10, 10000);
+	if (ret)
+		goto stop_failed;
+
+	/* The controller is no longer reading request buffers. */
+	udc_ep_ptr->halt_failed = false;
+	udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
+	udc_ep_ptr->tran_ring_full = false;
+	udc_ep_ptr->ep_state = isoc ? EP_STATE_STOPPED : EP_STATE_HALTED;
+
+	crg_udc_ep_nuke(udc_ep_ptr, -ECONNRESET);
+
+	return 0;
+
+stop_failed:
+	udc_ep_ptr->halt_failed = true;
+	udc_ep_ptr->ep_state = EP_STATE_STOPPED;
+	crg_udc_schedule_endpoint_recovery(crg_udc);
+	return ret;
+}
+
+void crg_udc_clear_req_container(struct crg_udc_request *udc_req_ptr)
+{
+	udc_req_ptr->buff_len_left = 0;
+	udc_req_ptr->trbs_needed = 0;
+	udc_req_ptr->all_trbs_queued = 0;
+	udc_req_ptr->td_pending = false;
+	udc_req_ptr->first_trb = NULL;
+	udc_req_ptr->last_trb = NULL;
+	udc_req_ptr->td_start_offset = 0;
+	udc_req_ptr->sg = NULL;
+	udc_req_ptr->sg_index = 0;
+	udc_req_ptr->sg_offset = 0;
+}
+
+const struct usb_ep_ops crg_udc_ep_ops = {
+	.enable = crg_udc_ep_enable,
+	.disable = crg_udc_ep_disable,
+	.alloc_request = crg_udc_alloc_request,
+	.free_request = crg_udc_free_request,
+	.queue = crg_udc_ep_queue,
+	.dequeue = crg_udc_ep_dequeue,
+	.set_halt = crg_udc_ep_set_halt,
+	.set_wedge = crg_udc_ep_set_wedge,
+};
+
+const struct usb_gadget_ops crg_gadget_ops = {
+	.get_frame = crg_gadget_get_frame,
+	.wakeup = crg_gadget_wakeup,
+	.set_remote_wakeup = crg_gadget_set_remote_wakeup,
+	.set_selfpowered = crg_gadget_set_selfpowered,
+	.pullup = crg_gadget_pullup,
+	.udc_start = crg_gadget_start,
+	.udc_stop = crg_gadget_stop,
+	.udc_set_speed = crg_udc_set_speed,
+	.udc_async_callbacks = crg_gadget_async_callbacks,
+};
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-trace.c b/drivers/usb/gadget/udc/crg/crg-udc-trace.c
new file mode 100644
index 000000000000..ee091708dbe0
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-trace.c
@@ -0,0 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ * Trace Support.
+ *
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#define CREATE_TRACE_POINTS
+#include "crg-udc-trace.h"

-- 
2.34.1


^ permalink raw reply related	[flat|nested] 13+ messages in thread

* [PATCH v5 6/6] usb: gadget: udc: crg: Add core driver and build integration
  2026-11-16 19:06 [PATCH v5 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
                   ` (4 preceding siblings ...)
  2026-11-16 19:06 ` [PATCH v5 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations Petar Stepanovic
@ 2026-11-16 19:06 ` Petar Stepanovic
  2026-11-16 21:15   ` sashiko-bot
  5 siblings, 1 reply; 13+ messages in thread
From: Petar Stepanovic @ 2026-11-16 19:06 UTC (permalink / raw)
  To: Prasad Bolisetty, Greg Kroah-Hartman, Rob Herring,
	Krzysztof Kozlowski, Conor Dooley, Tzu-Hao Wei,
	Karthikeyan Mitran, Philipp Zabel, Neill Kapron
  Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
	Petar Stepanovic, Krutik Shah, Naveen Kumar Rajgiri Bassappa

This patch introduces the core probe/remove functions for the Axiado
AX3000/AX3005 UDC driver, and wires up the directory into the build
system via Kconfig and Makefile.

Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Tzu-Hao Wei <twei@axiado.com>
Signed-off-by: Tzu-Hao Wei <twei@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
 drivers/usb/gadget/udc/Kconfig            |    2 +
 drivers/usb/gadget/udc/Makefile           |    1 +
 drivers/usb/gadget/udc/crg/Kconfig        |   16 +
 drivers/usb/gadget/udc/crg/Makefile       |    9 +
 drivers/usb/gadget/udc/crg/crg-udc-core.c | 1376 +++++++++++++++++++++++++++++
 5 files changed, 1404 insertions(+)

diff --git a/drivers/usb/gadget/udc/Kconfig b/drivers/usb/gadget/udc/Kconfig
index 7ddbc2606686..72a5a8a04982 100644
--- a/drivers/usb/gadget/udc/Kconfig
+++ b/drivers/usb/gadget/udc/Kconfig
@@ -421,6 +421,8 @@ source "drivers/usb/gadget/udc/aspeed-vhub/Kconfig"
 
 source "drivers/usb/gadget/udc/cdns2/Kconfig"
 
+source "drivers/usb/gadget/udc/crg/Kconfig"
+
 #
 # LAST -- dummy/emulated controller
 #
diff --git a/drivers/usb/gadget/udc/Makefile b/drivers/usb/gadget/udc/Makefile
index 1b9b1a4f9c57..227d2a0b525b 100644
--- a/drivers/usb/gadget/udc/Makefile
+++ b/drivers/usb/gadget/udc/Makefile
@@ -38,3 +38,4 @@ obj-$(CONFIG_USB_ASPEED_UDC)	+= aspeed_udc.o
 obj-$(CONFIG_USB_BDC_UDC)	+= bdc/
 obj-$(CONFIG_USB_MAX3420_UDC)	+= max3420_udc.o
 obj-$(CONFIG_USB_CDNS2_UDC)	+= cdns2/
+obj-$(CONFIG_USB_CRG_UDC)	+= crg/
diff --git a/drivers/usb/gadget/udc/crg/Kconfig b/drivers/usb/gadget/udc/crg/Kconfig
new file mode 100644
index 000000000000..7167c4943389
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/Kconfig
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: GPL-2.0
+config USB_CRG_UDC
+	tristate "Corigine / Axiado USB Device Controller"
+	depends on USB_GADGET
+	depends on HAS_DMA
+	depends on ARCH_AXIADO || COMPILE_TEST
+	help
+	  Say Y here to support the USB device controller found on Axiado AX3000
+	  and AX3005 SoCs, which is based on the Corigine USB 3.1 IP.
+
+	  This controller supports High-Speed and SuperSpeed.
+
+	  Say "y" to link the driver statically, or "m" to build a
+	  dynamically linked module called "crg-udc.ko".
+
+	  If unsure, say N.
diff --git a/drivers/usb/gadget/udc/crg/Makefile b/drivers/usb/gadget/udc/crg/Makefile
new file mode 100644
index 000000000000..96b209fb59fc
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/Makefile
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: GPL-2.0
+
+# define_trace.h needs to know how to find our trace header
+CFLAGS_crg-udc-trace.o		:= -I$(src)
+
+obj-$(CONFIG_USB_CRG_UDC)	+= crg-udc.o
+
+crg-udc-y			:= crg-udc-core.o crg-udc-ring.o crg-udc-ep0.o crg-udc-gadget.o
+crg-udc-$(CONFIG_TRACING)	+= crg-udc-trace.o
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-core.c b/drivers/usb/gadget/udc/crg/crg-udc-core.c
new file mode 100644
index 000000000000..e508dbd45339
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-core.c
@@ -0,0 +1,1376 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/clk.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/iopoll.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/phy/phy.h>
+#include <linux/platform_device.h>
+#include <linux/pm.h>
+#include <linux/property.h>
+#include <linux/reset.h>
+#include <linux/slab.h>
+
+#include "crg-udc.h"
+
+static int crg_gadget_irq_init(struct platform_device *pdev,
+			       struct crg_gadget_dev *crg_udc);
+static void crg_udc_phy_exit(void *data);
+static void crg_udc_recovery_work(struct work_struct *work);
+static int crg_udc_reset(struct crg_gadget_dev *crg_udc, unsigned long *flags);
+
+static int crg_udc_alloc_ep_rings(struct crg_gadget_dev *crg_udc)
+{
+	int i;
+
+	/*
+	 * Preallocate two disjoint rings per endpoint. Endpoint enable and
+	 * controller recovery can run in atomic context.
+	 */
+	for (i = 0; i < CRG_NUM_EP_CX; i++) {
+		struct crg_buffer_info *ring = &crg_udc->udc_ep[i].tran_ring_alloc;
+		size_t len = (i ? CRG_UDC_BULK_EP_TD_RING_SIZE :
+			      CRG_UDC_CONTROL_EP_TD_RING_SIZE) *
+			     sizeof(struct crg_transfer_trb);
+
+		if (i == 1)
+			continue;
+
+		ring->cpu_addr = dma_alloc_coherent(crg_udc->dev, 2 * len,
+						    &ring->dma_addr, GFP_KERNEL);
+		if (!ring->cpu_addr)
+			return -ENOMEM;
+		ring->len = 2 * len;
+	}
+
+	return 0;
+}
+
+static void crg_udc_select_transfer_rings(struct crg_gadget_dev *crg_udc)
+{
+	int i;
+
+	/*
+	 * Transfer events identify TRBs by DMA address. After confirmed
+	 * quiescence, use the other bank for the next session so a completion
+	 * for the previous bank cannot match a new request. Keep the selected
+	 * bank if a rebuild failed before RUN was set. Reusing a bank from
+	 * older sessions relies on the halt/reset and event-ring clear.
+	 */
+	if (crg_udc->transfer_rings_live) {
+		crg_udc->transfer_ring_bank = !crg_udc->transfer_ring_bank;
+		crg_udc->transfer_rings_live = false;
+	}
+
+	for (i = 0; i < CRG_NUM_EP_CX; i++) {
+		struct crg_udc_ep *ep = &crg_udc->udc_ep[i];
+		struct crg_buffer_info *alloc = &ep->tran_ring_alloc;
+		u32 len = alloc->len / 2;
+		u32 offset = crg_udc->transfer_ring_bank ? len : 0;
+		void *addr;
+
+		if (!alloc->cpu_addr)
+			continue;
+
+		addr = (u8 *)alloc->cpu_addr + offset;
+		if (ep->tran_ring_info.cpu_addr == addr)
+			continue;
+
+		ep->tran_ring_info.cpu_addr = addr;
+		ep->tran_ring_info.dma_addr = alloc->dma_addr + offset;
+		ep->tran_ring_info.len = len;
+		ep->first_trb = NULL;
+		ep->last_trb = NULL;
+		ep->enq_pt = NULL;
+		ep->deq_pt = NULL;
+	}
+}
+
+static void crg_udc_gadget_release(struct device *dev)
+{
+	struct crg_gadget_dev *crg_udc;
+
+	crg_udc = container_of(dev, struct crg_gadget_dev, gadget.dev);
+	kfree(crg_udc);
+}
+
+/*
+ * command related ops
+ */
+int crg_udc_issue_command(struct crg_gadget_dev *crg_udc,
+			  enum crg_cmd_type type, u32 param0, u32 param1)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 status;
+	u32 tmp;
+	bool running;
+
+	running = !!(readl(&uccr->control) & CRG_U3DC_CTRL_RUN);
+	/* Only INIT_EP0 may be staged while the controller is stopped. */
+	if (!running && type != CRG_CMD_INIT_EP0)
+		return -ESHUTDOWN;
+
+	/*
+	 * A staged command must complete before the command registers can
+	 * be reused. Hardware does not consume commands while RUN is clear.
+	 */
+	tmp = readl(&uccr->cmd_control);
+	if (tmp & CRG_U3DC_CMD_CTRL_ACTIVE) {
+		if (!running)
+			return -EBUSY;
+		if (readl_poll_timeout_atomic(&uccr->cmd_control, tmp,
+					      !(tmp & CRG_U3DC_CMD_CTRL_ACTIVE),
+					      1, 10000))
+			return -ETIMEDOUT;
+	}
+	/* Ensure that everything is written before issuing new command */
+	dma_wmb();
+
+	writel(param0, &uccr->cmd_param0);
+	writel(param1, &uccr->cmd_param1);
+
+	tmp = CRG_U3DC_CMD_CTRL_ACTIVE |
+	      FIELD_PREP(CRG_U3DC_CMD_CTRL_TYPE_MASK, type);
+	writel(tmp, &uccr->cmd_control);
+
+	if (!running)
+		return 0;
+
+	if (readl_poll_timeout_atomic(&uccr->cmd_control, tmp,
+				      !(tmp & CRG_U3DC_CMD_CTRL_ACTIVE), 1,
+				      1000)) {
+		dev_dbg(crg_udc->dev, "command timed out\n");
+		return -ETIMEDOUT;
+	}
+
+	status = FIELD_GET(CRG_U3DC_CMD_CTRL_STATUS_MASK, tmp);
+	if (status) {
+		dev_dbg(crg_udc->dev, "command failed with status %u\n",
+			status);
+		return -EIO;
+	}
+
+	return 0;
+}
+
+static void crg_udc_clk_disable(void *data)
+{
+	clk_disable_unprepare(data);
+}
+
+/*
+ * Stop the controller and wait for pending event writes before resetting it.
+ * Corigine USB 3.1 DRD v1.5, sections 8.3.3 and 8.3.4: Controller Halted
+ * requires Controller Idle, at which point queued events have reached the
+ * event ring. Discard those events when rebuilding the ring after reset.
+ */
+static int crg_udc_soft_reset(struct crg_gadget_dev *crg_udc, bool atomic)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 ctrl, val;
+	int ret;
+
+	ctrl = readl(&uccr->control);
+	ctrl &= ~(CRG_U3DC_CTRL_RUN | CRG_U3DC_CTRL_INT_EN |
+		  CRG_U3DC_CTRL_SYSERR_EN | CRG_U3DC_CTRL_SWRST);
+	writel(ctrl, &uccr->control);
+
+	if (atomic)
+		ret = readl_poll_timeout_atomic(&uccr->status, val,
+					       val & CRG_U3DC_STATUS_DEV_CTRL_HALT,
+					       10, 10000);
+	else
+		ret = readl_poll_timeout(&uccr->status, val,
+					val & CRG_U3DC_STATUS_DEV_CTRL_HALT,
+					10, 10000);
+	if (ret) {
+		dev_err(crg_udc->dev, "controller halt timed out before reset\n");
+		return ret;
+	}
+
+	writel(ctrl | CRG_U3DC_CTRL_SWRST, &uccr->control);
+	if (atomic)
+		ret = readl_poll_timeout_atomic(&uccr->control, val,
+					       !(val & CRG_U3DC_CTRL_SWRST), 10, 10000);
+	else
+		ret = readl_poll_timeout(&uccr->control, val,
+					!(val & CRG_U3DC_CTRL_SWRST), 10, 10000);
+	if (ret)
+		dev_err(crg_udc->dev, "software reset timed out\n");
+
+	return ret;
+}
+
+/*
+ * Stop bus-master DMA with new submissions blocked. This can sleep.
+ * An asserted SoC reset is a fallback when halting or SWRST times out.
+ */
+static int crg_udc_quiesce_hw(struct crg_gadget_dev *crg_udc)
+{
+	int ret;
+
+	if (crg_udc->hw_reset_asserted || !crg_udc->hw_dma_armed)
+		return 0;
+	if (!crg_udc->uccr)
+		return -ENODEV;
+
+	ret = crg_udc_soft_reset(crg_udc, false);
+	if (!ret) {
+		crg_udc->hw_dma_armed = false;
+		return 0;
+	}
+
+	if (!crg_udc->rst)
+		return ret;
+
+	ret = reset_control_assert(crg_udc->rst);
+	if (!ret) {
+		crg_udc->hw_reset_asserted = true;
+		crg_udc->hw_dma_armed = false;
+	}
+
+	return ret;
+}
+
+static void crg_udc_free_resources(struct crg_gadget_dev *crg_udc)
+{
+	unsigned long flags;
+	int i, ret;
+
+	/* Block any in-flight threaded handler before releasing DMA memory. */
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	crg_udc->shutting_down = true;
+	if (crg_udc->uccr && !crg_udc->hw_reset_asserted &&
+	    crg_udc->hw_dma_armed) {
+		u32 val = readl(&crg_udc->uccr->control);
+
+		val &= ~(CRG_U3DC_CTRL_INT_EN | CRG_U3DC_CTRL_RUN |
+			 CRG_U3DC_CTRL_SYSERR_EN);
+		writel(val, &crg_udc->uccr->control);
+	}
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	if (crg_udc->irq >= 0) {
+		synchronize_irq(crg_udc->irq);
+		devm_free_irq(crg_udc->dev, crg_udc->irq, crg_udc);
+		crg_udc->irq = -1;
+		crg_udc->gadget.irq = 0;
+	}
+
+	if (crg_udc->status_req) {
+		kfree(crg_udc->status_req->usb_req.buf);
+		crg_udc_free_request(&crg_udc->udc_ep[0].usb_ep,
+				     &crg_udc->status_req->usb_req);
+		crg_udc->status_req = NULL;
+	}
+
+	ret = crg_udc_quiesce_hw(crg_udc);
+	if (ret) {
+		crg_udc->dma_quiesce_failed = true;
+		/*
+		 * Retain potentially live DMA mappings and coherent allocations.
+		 * A platform with detachable DMA translations must isolate the
+		 * master independently before tearing down its DMA domain.
+		 */
+		dev_crit(crg_udc->dev,
+			 "DMA stop failed (%d); retaining allocations, platform isolation required\n",
+			 ret);
+		return;
+	}
+	crg_udc->dma_quiesce_failed = false;
+	crg_udc_release_quarantine(crg_udc);
+
+	for (i = 0; i < CRG_RING_NUM; i++) {
+		struct crg_udc_event *udc_event = &crg_udc->udc_event[i];
+
+		if (udc_event->event_ring.cpu_addr) {
+			dma_free_coherent(crg_udc->dev,
+					  udc_event->event_ring.len,
+					  udc_event->event_ring.cpu_addr,
+					  udc_event->event_ring.dma_addr);
+			udc_event->event_ring.cpu_addr = NULL;
+			udc_event->event_ring.dma_addr = DMA_MAPPING_ERROR;
+			udc_event->event_ring.len = 0;
+			udc_event->evt_dq_pt = NULL;
+		}
+		if (udc_event->erst.cpu_addr) {
+			dma_free_coherent(crg_udc->dev, udc_event->erst.len,
+					  udc_event->erst.cpu_addr,
+					  udc_event->erst.dma_addr);
+			udc_event->erst.cpu_addr = NULL;
+			udc_event->erst.dma_addr = DMA_MAPPING_ERROR;
+			udc_event->erst.len = 0;
+			udc_event->p_erst = NULL;
+		}
+	}
+
+	if (crg_udc->ep_cx.cpu_addr) {
+		dma_free_coherent(crg_udc->dev, crg_udc->ep_cx.len,
+				  crg_udc->ep_cx.cpu_addr,
+				  crg_udc->ep_cx.dma_addr);
+		crg_udc->ep_cx.cpu_addr = NULL;
+		crg_udc->ep_cx.dma_addr = DMA_MAPPING_ERROR;
+		crg_udc->ep_cx.len = 0;
+		crg_udc->p_epcx = NULL;
+	}
+
+	for (i = 0; i < CRG_NUM_EP_CX; i++) {
+		struct crg_udc_ep *udc_ep = &crg_udc->udc_ep[i];
+		struct crg_buffer_info *alloc = &udc_ep->tran_ring_alloc;
+
+		if (alloc->cpu_addr) {
+			dma_free_coherent(crg_udc->dev, alloc->len,
+					  alloc->cpu_addr, alloc->dma_addr);
+			*alloc = (struct crg_buffer_info){};
+			udc_ep->tran_ring_info.cpu_addr = NULL;
+			udc_ep->tran_ring_info.dma_addr = DMA_MAPPING_ERROR;
+			udc_ep->tran_ring_info.len = 0;
+			udc_ep->first_trb = NULL;
+			udc_ep->last_trb = NULL;
+			udc_ep->enq_pt = NULL;
+			udc_ep->deq_pt = NULL;
+		}
+	}
+}
+
+static int crg_udc_init_hw(struct crg_gadget_dev *crg_udc)
+{
+	struct device *dev = crg_udc->dev;
+	struct reset_control *rst;
+	int ret;
+
+	crg_udc->clk = devm_clk_get_optional(dev, NULL);
+	if (IS_ERR(crg_udc->clk))
+		return dev_err_probe(dev, PTR_ERR(crg_udc->clk),
+				     "failed to get clock\n");
+
+	ret = clk_prepare_enable(crg_udc->clk);
+	if (ret)
+		return dev_err_probe(dev, ret, "failed to enable clock\n");
+
+	ret = devm_add_action_or_reset(dev, crg_udc_clk_disable, crg_udc->clk);
+	if (ret)
+		return ret;
+
+	rst = devm_reset_control_get_optional_exclusive_deasserted(dev, NULL);
+	if (IS_ERR(rst))
+		return dev_err_probe(dev, PTR_ERR(rst),
+				     "failed to get reset\n");
+	crg_udc->rst = rst;
+
+	crg_udc->phy = devm_phy_optional_get(dev, NULL);
+	if (IS_ERR(crg_udc->phy))
+		return dev_err_probe(dev, PTR_ERR(crg_udc->phy),
+				     "failed to get phy\n");
+
+	ret = phy_init(crg_udc->phy);
+	if (ret)
+		return dev_err_probe(dev, ret, "failed to init phy\n");
+
+	ret = phy_power_on(crg_udc->phy);
+	if (ret) {
+		phy_exit(crg_udc->phy);
+		return dev_err_probe(dev, ret, "failed to power on phy\n");
+	}
+
+	return devm_add_action_or_reset(dev, crg_udc_phy_exit, crg_udc->phy);
+}
+
+static void crg_udc_phy_exit(void *data)
+{
+	struct phy *phy = data;
+
+	phy_power_off(phy);
+	phy_exit(phy);
+}
+
+static int crg_udc_probe(struct platform_device *pdev)
+{
+	unsigned long flags = 0;
+	int ret;
+	int i;
+	struct crg_gadget_dev *crg_udc;
+	const struct crg_udc_priv *priv;
+
+	priv = device_get_match_data(&pdev->dev);
+	if (!priv)
+		return -ENODEV;
+
+	crg_udc = kzalloc_obj(*crg_udc);
+	if (!crg_udc)
+		return -ENOMEM;
+	usb_initialize_gadget(&pdev->dev, &crg_udc->gadget,
+			      crg_udc_gadget_release);
+
+	crg_udc->dev = &pdev->dev;
+	crg_udc->irq = -1;
+	/* Probe errors can call free_resources() before hardware init. */
+	spin_lock_init(&crg_udc->udc_lock);
+	init_waitqueue_head(&crg_udc->teardown_wait);
+	INIT_LIST_HEAD(&crg_udc->dma_quarantine);
+
+	ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64));
+	if (ret) {
+		ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32));
+		if (ret) {
+			dev_err(&pdev->dev, "no usable DMA mask\n");
+			goto err_free_resources;
+		}
+	}
+
+	ret = crg_udc_init_hw(crg_udc);
+	if (ret)
+		goto err_free_resources;
+
+	crg_udc->ep0_desc.bLength = USB_DT_ENDPOINT_SIZE;
+	crg_udc->ep0_desc.bDescriptorType = USB_DT_ENDPOINT;
+	crg_udc->ep0_desc.bEndpointAddress = 0;
+	crg_udc->ep0_desc.bmAttributes = USB_ENDPOINT_XFER_CONTROL;
+	crg_udc->ep0_desc.wMaxPacketSize = cpu_to_le16(64);
+
+	crg_udc->u12_forbidden = priv->quirks & CRG_QUIRK_DIS_U1U2_ENTRY;
+	crg_udc->u12_initiate_forbidden = priv->quirks &
+					  CRG_QUIRK_DIS_U1U2_INITIATION;
+	crg_udc->reinit_ep0_on_bus_reset = priv->quirks &
+					   CRG_QUIRK_REINIT_EP0_ON_RESET;
+
+	INIT_WORK(&crg_udc->udc_reset_work, crg_udc_reset_work);
+	INIT_WORK(&crg_udc->recovery_work, crg_udc_recovery_work);
+	INIT_DELAYED_WORK(&crg_udc->port_retry_work, crg_udc_port_retry_work);
+	platform_set_drvdata(pdev, crg_udc);
+
+	crg_udc->gadget.ops = &crg_gadget_ops;
+	crg_udc->gadget.ep0 = &crg_udc->udc_ep[0].usb_ep;
+	INIT_LIST_HEAD(&crg_udc->gadget.ep_list);
+	/*
+	 * The controller supports up to SuperSpeed. Honour the standard
+	 * "maximum-speed" property to cap it; default to the hardware
+	 * maximum when the property is absent.
+	 */
+	crg_udc->gadget.max_speed = usb_get_maximum_speed(&pdev->dev);
+	/* Hardware only supports Full-, High-, and SuperSpeed. */
+	switch (crg_udc->gadget.max_speed) {
+	case USB_SPEED_FULL:
+	case USB_SPEED_HIGH:
+	case USB_SPEED_SUPER:
+		break;
+	case USB_SPEED_LOW:
+		crg_udc->gadget.max_speed = USB_SPEED_FULL;
+		break;
+	default:
+		crg_udc->gadget.max_speed = USB_SPEED_SUPER;
+		break;
+	}
+	crg_udc->max_speed = crg_udc->gadget.max_speed;
+	crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+	crg_udc->gadget.name = KBUILD_MODNAME;
+	crg_udc->gadget.sg_supported = true;
+	crg_udc->gadget.wakeup_capable = true;
+	crg_udc->connected = 0;
+	crg_udc->dev_addr = 0;
+
+	crg_udc->mmio_virt_base = devm_platform_ioremap_resource(pdev, 0);
+	if (IS_ERR(crg_udc->mmio_virt_base)) {
+		ret = dev_err_probe(&pdev->dev,
+				    PTR_ERR(crg_udc->mmio_virt_base),
+				    "failed to map registers\n");
+		goto err_put_gadget;
+	}
+
+	/* set controller device role */
+	writel((readl(crg_udc->mmio_virt_base + CRG_UDC_MODE_REG) |
+		CRG_UDC_ROLE_DEVICE),
+	       crg_udc->mmio_virt_base + CRG_UDC_MODE_REG);
+	for (i = 0; i < CRG_RING_NUM; i++) {
+		crg_udc->uicr[i] = crg_udc->mmio_virt_base + CRG_UICR_OFFSET +
+				   i * CRG_UICR_STRIDE;
+	}
+	crg_udc->uccr = crg_udc->mmio_virt_base + CRG_UCCR_OFFSET;
+
+	/* Initialize endpoint objects before reset, DMA setup or any IRQ. */
+	crg_udc_init_ep_info(crg_udc);
+
+	ret = crg_udc_alloc_ep_rings(crg_udc);
+	if (ret)
+		goto err_free_resources;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	ret = crg_udc_reset(crg_udc, &flags);
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	if (ret)
+		goto err_free_resources;
+
+	crg_udc_clear_portpm(crg_udc);
+
+	ret = crg_udc_reset_data_struct(crg_udc);
+	if (ret) {
+		dev_err(crg_udc->dev, "crg_udc_reset_data_struct error\n");
+		goto err_free_resources;
+	}
+
+	ret = crg_udc_init_ep0(crg_udc);
+	if (ret)
+		goto err_free_resources;
+
+	crg_udc->udc_ep[0].desc = &crg_udc->ep0_desc;
+	/* Release the reset gate only after EP0 and all event rings exist. */
+	crg_udc->resetting = false;
+
+	ret = crg_gadget_irq_init(pdev, crg_udc);
+	if (ret)
+		goto err_free_resources;
+
+	ret = usb_add_gadget(&crg_udc->gadget);
+	if (ret)
+		goto err_free_resources;
+
+	return 0;
+
+err_free_resources:
+	crg_udc_free_resources(crg_udc);
+err_put_gadget:
+	usb_put_gadget(&crg_udc->gadget);
+	return ret;
+}
+
+static void crg_udc_remove(struct platform_device *pdev)
+{
+	struct crg_gadget_dev *crg_udc = platform_get_drvdata(pdev);
+	unsigned long flags;
+	u32 tmp;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	crg_udc->shutting_down = true;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	/* The gadget core calls pullup(0) before the disconnect callback. */
+	usb_del_gadget(&crg_udc->gadget);
+	cancel_delayed_work_sync(&crg_udc->port_retry_work);
+	cancel_work_sync(&crg_udc->recovery_work);
+	cancel_work_sync(&crg_udc->udc_reset_work);
+	crg_udc_free_resources(crg_udc);
+
+	/* Do not access registers while the external reset is asserted. */
+	if (!crg_udc->hw_reset_asserted && !crg_udc->dma_quiesce_failed) {
+		tmp = readl(crg_udc->mmio_virt_base + CRG_UDC_MODE_REG) & ~0x1;
+		writel(tmp, crg_udc->mmio_virt_base + CRG_UDC_MODE_REG);
+	}
+
+	platform_set_drvdata(pdev, NULL);
+	usb_put_gadget(&crg_udc->gadget);
+}
+
+static void crg_udc_shutdown(struct platform_device *pdev)
+{
+	struct crg_gadget_dev *crg_udc = platform_get_drvdata(pdev);
+	unsigned long flags;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	crg_udc->pullup_on = false;
+	crg_udc->shutting_down = true;
+	crg_udc->resetting = true;
+	crg_udc->device_state = USB_STATE_ATTACHED;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	if (crg_udc->irq >= 0)
+		synchronize_irq(crg_udc->irq);
+	cancel_delayed_work_sync(&crg_udc->port_retry_work);
+	cancel_work_sync(&crg_udc->recovery_work);
+	cancel_work_sync(&crg_udc->udc_reset_work);
+
+	/* Reset and gadget stop may own the registers across a lock drop. */
+	spin_lock_irq(&crg_udc->udc_lock);
+	wait_event_lock_irq(crg_udc->teardown_wait,
+			    !crg_udc->controller_recovering, crg_udc->udc_lock);
+	if (crg_udc->uccr && !crg_udc->hw_reset_asserted) {
+		u32 val = readl(&crg_udc->uccr->control);
+
+		crg_udc_set_vbus_detect(crg_udc, false);
+		val &= ~(CRG_U3DC_CTRL_INT_EN | CRG_U3DC_CTRL_RUN |
+			 CRG_U3DC_CTRL_SYSERR_EN);
+		writel(val, &crg_udc->uccr->control);
+	}
+	spin_unlock_irq(&crg_udc->udc_lock);
+}
+
+static int crg_udc_suspend(struct device *dev)
+{
+	struct crg_gadget_dev *crg_udc = dev_get_drvdata(dev);
+	unsigned long flags;
+	u32 val;
+	int ret;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	if (crg_udc->controller_recovering || crg_udc->shutting_down ||
+	    crg_udc->external_reset_pending ||
+	    crg_udc->endpoint_recovery_pending ||
+	    (crg_udc->hw_reset_asserted && !crg_udc->gadget_stopping)) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EBUSY;
+	}
+	crg_udc->suspended = true;
+	if (crg_udc->gadget_stopping) {
+		if (crg_udc->hw_dma_armed) {
+			crg_udc->suspended = false;
+			ret = -EBUSY;
+		} else {
+			ret = 0;
+		}
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return ret;
+	}
+	crg_udc_set_pullup(crg_udc, false);
+	val = readl(&crg_udc->uccr->control);
+	val &= ~CRG_U3DC_CTRL_INT_EN;
+	writel(val, &crg_udc->uccr->control);
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	if (crg_udc->irq >= 0)
+		synchronize_irq(crg_udc->irq);
+	cancel_delayed_work_sync(&crg_udc->port_retry_work);
+	cancel_work_sync(&crg_udc->recovery_work);
+	cancel_work_sync(&crg_udc->udc_reset_work);
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	if (crg_udc->gadget_stopping || crg_udc->shutting_down ||
+	    crg_udc->controller_recovering || crg_udc->external_reset_pending ||
+	    crg_udc->endpoint_recovery_pending) {
+		ret = -EBUSY;
+		goto out;
+	}
+	/*
+	 * A reset may already have marked the link disconnected while its
+	 * gadget notification was still pending. Deliver disconnect before
+	 * resetting the software state, including that cancelled work.
+	 */
+	if (crg_udc->connected || crg_udc->bus_reset_pending)
+		crg_udc_notify_disconnect(crg_udc, &flags);
+	crg_udc->connected = false;
+	crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+	if (crg_udc->external_reset_pending || crg_udc->endpoint_recovery_pending) {
+		ret = -EBUSY;
+		goto out;
+	}
+	ret = crg_udc_recover(crg_udc, &flags);
+	if (ret)
+		dev_err(crg_udc->dev, "suspend controller reset failed: %d\n", ret);
+out:
+	if (ret) {
+		crg_udc->suspended = false;
+		crg_udc_kick_reset_work(crg_udc);
+	}
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	return ret;
+}
+
+static int crg_udc_resume(struct device *dev)
+{
+	struct crg_gadget_dev *crg_udc = dev_get_drvdata(dev);
+	unsigned long flags;
+	int ret = 0;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	crg_udc->suspended = false;
+	if (!crg_udc->gadget_stopping) {
+		if (crg_udc->shutting_down || crg_udc->resetting)
+			ret = -ESHUTDOWN;
+		else if (crg_udc->gadget_driver && crg_udc->pullup_on)
+			crg_udc_set_pullup(crg_udc, true);
+	}
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	return ret;
+}
+
+/* Program the speed limit while the controller is stopped. */
+int crg_udc_program_speed(struct crg_gadget_dev *crg_udc)
+{
+	u32 cfg, speed;
+
+	switch (crg_udc->max_speed) {
+	case USB_SPEED_FULL:
+		speed = CRG_U3DC_CFG0_MAXSPEED_FS;
+		break;
+	case USB_SPEED_HIGH:
+		speed = CRG_U3DC_CFG0_MAXSPEED_HS;
+		break;
+	case USB_SPEED_SUPER:
+		speed = CRG_U3DC_CFG0_MAXSPEED_SS;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	cfg = readl(&crg_udc->uccr->config0);
+	cfg &= ~CRG_U3DC_CFG0_MAXSPEED_MASK;
+	cfg |= FIELD_PREP(CRG_U3DC_CFG0_MAXSPEED_MASK, speed);
+	writel(cfg, &crg_udc->uccr->config0);
+
+	return 0;
+}
+
+int crg_udc_reset_data_struct(struct crg_gadget_dev *crg_udc)
+{
+	int i;
+	int ret;
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 val;
+
+	crg_udc->ep0_state = EP0_STATE_IDLE;
+
+	val = readl(&uccr->control);
+	val &= (~(CRG_U3DC_CTRL_INT_EN | CRG_U3DC_CTRL_RUN));
+	writel(val, &uccr->control);
+
+	ret = crg_udc_program_speed(crg_udc);
+	if (ret)
+		return ret;
+
+	crg_udc_select_transfer_rings(crg_udc);
+
+	for (i = 0; i < CRG_RING_NUM; i++) {
+		ret = crg_udc_init_event_ring(crg_udc, i);
+		if (ret)
+			return ret;
+	}
+
+	ret = crg_udc_init_device_context(crg_udc);
+	if (ret)
+		return ret;
+
+	if (!crg_udc->status_req) {
+		struct usb_request *req;
+
+		req = crg_udc_alloc_request(&crg_udc->udc_ep[0].usb_ep,
+					    GFP_ATOMIC);
+		if (!req)
+			return -ENOMEM;
+
+		crg_udc->status_req =
+			container_of(req, struct crg_udc_request, usb_req);
+	}
+
+	/* other hw ops */
+	return 0;
+}
+
+static void crg_gadget_handle_interrupt(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	unsigned long flags = 0;
+	u32 tmp_status;
+	int ret;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+
+	/*
+	 * A reset started on another CPU owns the controller registers and
+	 * the event rings until it has rebuilt them. It drops udc_lock while
+	 * polling SWRST and during givebacks, so this handler can get in.
+	 */
+	if (crg_udc->resetting || crg_udc->suspended ||
+	    crg_udc->shutting_down || crg_udc->controller_recovering) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return;
+	}
+
+	tmp_status = readl(&uccr->status);
+
+	if (tmp_status & CRG_U3DC_STATUS_SYS_ERR) {
+		bool was_connected = crg_udc->connected;
+		u32 generation = crg_udc->udc_event[0].generation;
+
+		/* A controller reset must not overlap composite's ->reset(). */
+		if (crg_udc->reset_callback_active) {
+			writel(CRG_U3DC_STATUS_SYS_ERR, &uccr->status);
+			crg_udc->deferred_system_error = true;
+			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+			return;
+		}
+
+		dev_err(crg_udc->dev, "controller system error\n");
+		writel(CRG_U3DC_STATUS_SYS_ERR, &uccr->status);
+		/* Function drivers must see enabled endpoints on disconnect. */
+		if (was_connected)
+			crg_udc_notify_disconnect(crg_udc, &flags);
+		/*
+		 * The disconnect callback drops udc_lock: another CPU may be
+		 * resetting now, or may have reset and rebuilt the rings
+		 * already, in which case this error has been recovered.
+		 */
+		if (crg_udc->resetting || crg_udc->suspended ||
+		    crg_udc->shutting_down ||
+		    generation != crg_udc->udc_event[0].generation) {
+			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+			return;
+		}
+		crg_udc_recover_link(crg_udc, &flags, false);
+
+		/*
+		 * tmp_status was captured before the reset above, which tore
+		 * down and rebuilt the event rings. It no longer reflects
+		 * the controller's current state, and if recovery failed the
+		 * rings may not be consistent at all. Skip the rest of this
+		 * interrupt's processing; any event still pending will raise
+		 * a fresh interrupt once the controller is running again.
+		 */
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return;
+	}
+
+	if (tmp_status & CRG_U3DC_STATUS_EINT) {
+		int i;
+
+		writel(CRG_U3DC_STATUS_EINT, &uccr->status);
+
+		for (i = 0; i < CRG_RING_NUM; i++) {
+			u32 generation = crg_udc->udc_event[i].generation;
+
+			ret = crg_udc_process_event_ring(crg_udc, i, &flags);
+			/*
+			 * The event handlers drop udc_lock, so a reset may have
+			 * started (resetting) or already completed and rebuilt
+			 * the rings (generation) while they ran. Either way
+			 * tmp_status and the checks below are stale.
+			 */
+			if (unlikely(crg_udc->resetting || crg_udc->suspended ||
+				     crg_udc->shutting_down ||
+				     generation != crg_udc->udc_event[i].generation)) {
+				spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+				return;
+			}
+			if (ret)
+				dev_err(crg_udc->dev,
+					"process event ring error: %d\n", ret);
+		}
+	}
+
+	if (crg_udc->device_state == USB_STATE_RECONNECTING &&
+	    crg_udc->portsc_on_reconnecting == 1 &&
+	    crg_udc_is_event_rings_empty(crg_udc)) {
+		crg_udc->portsc_on_reconnecting = 0;
+		crg_udc_handle_port_status(crg_udc, &flags);
+		/* The port handler drops udc_lock. */
+		if (crg_udc->resetting) {
+			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+			return;
+		}
+	}
+
+	if (crg_udc->device_state == USB_STATE_RECONNECTING &&
+	    crg_udc->connected == 1) {
+		ret = crg_udc_prepare_for_setup(crg_udc);
+		if (!ret && crg_udc->pending_setup_valid &&
+		    !crg_udc->resetting && crg_udc->connected &&
+		    crg_udc->device_state == USB_STATE_DEFAULT) {
+			struct usb_ctrlrequest setup = crg_udc->pending_setup;
+			u8 tag = crg_udc->pending_setup_tag;
+
+			crg_udc->pending_setup_valid = false;
+			crg_udc_dispatch_setup(crg_udc, &setup, tag, &flags);
+		} else if (ret && ret != -EBUSY) {
+			dev_err(crg_udc->dev,
+				"failed to prepare EP0 after reset: %d\n", ret);
+		}
+	}
+
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+}
+
+/*
+ * Primary handler only checks whether the shared interrupt belongs to the
+ * controller. All controller work, including reset polling, runs in the
+ * threaded handler and therefore never busy-waits in hard IRQ context.
+ */
+static irqreturn_t crg_udc_common_irq(int irq, void *dev)
+{
+	struct crg_gadget_dev *crg_udc = dev;
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 status;
+
+	if (READ_ONCE(crg_udc->gadget_stopping) ||
+	    READ_ONCE(crg_udc->shutting_down) ||
+	    READ_ONCE(crg_udc->resetting))
+		return IRQ_NONE;
+
+	status = readl(&uccr->status);
+	if (!(status & (CRG_U3DC_STATUS_EINT | CRG_U3DC_STATUS_SYS_ERR)))
+		return IRQ_NONE;
+
+	return IRQ_WAKE_THREAD;
+}
+
+static irqreturn_t crg_udc_irq_thread(int irq, void *dev)
+{
+	struct crg_gadget_dev *crg_udc = dev;
+
+	crg_gadget_handle_interrupt(crg_udc);
+
+	return IRQ_HANDLED;
+}
+
+static int crg_gadget_irq_init(struct platform_device *pdev,
+			       struct crg_gadget_dev *crg_udc)
+{
+	int irq;
+	int ret;
+
+	irq = platform_get_irq(pdev, 0);
+	if (irq < 0)
+		return irq;
+
+	/*
+	 * IRQF_ONESHOT keeps this (level-triggered) line masked until
+	 * crg_udc_irq_thread() has cleared the status bits, which is the
+	 * correct pairing for IRQF_SHARED with a threaded handler.
+	 */
+	ret = devm_request_threaded_irq(&pdev->dev, irq, crg_udc_common_irq,
+					crg_udc_irq_thread,
+					IRQF_SHARED | IRQF_ONESHOT, "crg_udc",
+					crg_udc);
+	if (ret) {
+		dev_err(crg_udc->dev, "request of irq%d failed\n", irq);
+	} else {
+		crg_udc->irq = irq;
+		crg_udc->gadget.irq = irq;
+	}
+
+	return ret;
+}
+
+/* Invalidate the old software session before dropping udc_lock. */
+static void crg_udc_reset_session(struct crg_gadget_dev *crg_udc)
+{
+	crg_udc->resetting = true;
+	crg_udc->bus_reset_generation++;
+	crg_udc->bus_reset_pending = false;
+	crg_udc->external_reset_pending = false;
+	crg_udc->endpoint_recovery_pending = false;
+	crg_udc->restart_after_reset = false;
+	crg_udc->deferred_port_events = 0;
+	crg_udc->deferred_system_error = false;
+	crg_udc->portsc_on_reconnecting = false;
+	crg_udc->setup_fn_call_back = NULL;
+	crg_udc->set_tm = 0;
+	crg_udc->udc_ep[0].generation++;
+	crg_udc->pending_setup_valid = false;
+}
+
+static void crg_udc_reset_queues(struct crg_gadget_dev *crg_udc,
+				 bool preserve_enabled)
+{
+	int i;
+
+	crg_udc->setup_status = WAIT_FOR_SETUP;
+	crg_udc->device_state = USB_STATE_ATTACHED;
+	crg_udc->dev_addr = 0;
+	crg_udc->port_reset_pending = false;
+	crg_udc->num_enabled_eps = 0;
+
+	crg_udc->udc_ep[0].desc = NULL;
+	crg_udc->udc_ep[0].ep_state = EP_STATE_DISABLED;
+
+	/* Data endpoint contexts start at DCI 2. DCI 1 has no queue. */
+	for (i = 2; i < CRG_NUM_EP_CX; i++) {
+		struct crg_udc_ep *ep = &crg_udc->udc_ep[i];
+
+		ep->generation++;
+		ep->desc = NULL;
+		ep->comp_desc = NULL;
+		ep->tran_ring_full = false;
+		ep->halt_failed = false;
+		ep->disable_failed = false;
+		ep->dequeue_paused = false;
+		ep->ep_state = EP_STATE_DISABLED;
+	}
+
+	for (i = 2; i < CRG_NUM_EP_CX; i++)
+		crg_udc_ep_nuke(&crg_udc->udc_ep[i], -ESHUTDOWN);
+	crg_udc_ep_nuke(&crg_udc->udc_ep[0], -ESHUTDOWN);
+
+	/* Preserve Gadget's endpoint state through all givebacks. */
+	if (!preserve_enabled)
+		for (i = 2; i < CRG_NUM_EP_CX; i++)
+			crg_udc_ep_clear_enabled(&crg_udc->udc_ep[i]);
+}
+
+static int crg_udc_reset(struct crg_gadget_dev *crg_udc, unsigned long *flags)
+{
+	int ret;
+
+	crg_udc_reset_session(crg_udc);
+	spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+	ret = crg_udc_soft_reset(crg_udc, true);
+	spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+	if (ret)
+		return ret;
+
+	crg_udc->hw_dma_armed = false;
+	crg_udc->dma_quiesce_failed = false;
+	crg_udc_release_quarantine(crg_udc);
+	crg_udc_clear_portpm(crg_udc);
+	crg_udc_reset_queues(crg_udc, crg_udc->preserve_ep_enabled_on_reset);
+	return 0;
+}
+
+/*
+ * The UDC core serializes pullup, async-callback and stop operations.
+ * Return requests before disconnect or unbind can free their storage.
+ */
+int crg_udc_stop_gadget(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 val;
+	int ret;
+
+	spin_lock_irq(&crg_udc->udc_lock);
+	if (crg_udc->gadget_stopping) {
+		/* A previous stop or a failed restart can leave DMA armed. */
+		bool retry = crg_udc->hw_dma_armed;
+
+		if (!retry) {
+			spin_unlock_irq(&crg_udc->udc_lock);
+			return 0;
+		}
+		goto quiesce;
+	}
+	crg_udc->gadget_stopping = true;
+	crg_udc->pullup_on = false;
+	crg_udc_reset_session(crg_udc);
+	if (!crg_udc->controller_recovering && crg_udc->hw_dma_armed) {
+		val = readl(&uccr->control);
+		val &= ~(CRG_U3DC_CTRL_RUN | CRG_U3DC_CTRL_INT_EN |
+			 CRG_U3DC_CTRL_SYSERR_EN);
+		writel(val, &uccr->control);
+	}
+	spin_unlock_irq(&crg_udc->udc_lock);
+
+	if (crg_udc->irq >= 0)
+		synchronize_irq(crg_udc->irq);
+	cancel_delayed_work_sync(&crg_udc->port_retry_work);
+	cancel_work_sync(&crg_udc->recovery_work);
+	cancel_work_sync(&crg_udc->udc_reset_work);
+
+	spin_lock_irq(&crg_udc->udc_lock);
+	wait_event_lock_irq(crg_udc->teardown_wait,
+			    !crg_udc->controller_recovering &&
+			    !crg_udc->gadget_callbacks_active && !crg_udc->queueing,
+			    crg_udc->udc_lock);
+quiesce:
+	crg_udc_recovery_begin(crg_udc);
+	spin_unlock_irq(&crg_udc->udc_lock);
+	ret = crg_udc_quiesce_hw(crg_udc);
+	if (ret)
+		dev_err(crg_udc->dev,
+			"cannot stop DMA during gadget teardown: %d\n", ret);
+
+	spin_lock_irq(&crg_udc->udc_lock);
+	crg_udc->dma_quiesce_failed = !!ret;
+	crg_udc->controller_faulted = !!ret;
+	if (!ret) {
+		if (!crg_udc->hw_reset_asserted)
+			crg_udc_set_vbus_detect(crg_udc, false);
+		crg_udc_release_quarantine(crg_udc);
+	}
+	/* Function drivers can inspect usb_ep.enabled during disconnect. */
+	crg_udc_reset_queues(crg_udc, true);
+	crg_udc->connected = false;
+	crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+	crg_udc_recovery_end(crg_udc);
+	spin_unlock_irq(&crg_udc->udc_lock);
+
+	usb_gadget_set_state(&crg_udc->gadget, USB_STATE_NOTATTACHED);
+	return ret;
+}
+
+/* Called with udc_lock held and IRQ and queue entry blocked. */
+static int crg_udc_deassert_reset(struct crg_gadget_dev *crg_udc,
+				  unsigned long *flags)
+{
+	int ret;
+
+	spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+	ret = reset_control_deassert(crg_udc->rst);
+	spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+	if (!ret)
+		crg_udc->hw_reset_asserted = false;
+	return ret;
+}
+
+static int crg_udc_rebuild_after_stop(struct crg_gadget_dev *crg_udc)
+{
+	void __iomem *mode = crg_udc->mmio_virt_base + CRG_UDC_MODE_REG;
+	int ret;
+
+	writel(readl(mode) | CRG_UDC_ROLE_DEVICE, mode);
+	crg_udc_clear_portpm(crg_udc);
+	ret = crg_udc_reset_data_struct(crg_udc);
+	if (!ret)
+		ret = crg_udc_init_ep0(crg_udc);
+	if (!ret) {
+		crg_udc_set_vbus_detect(crg_udc, true);
+		crg_udc->stopped_needs_reset = false;
+		crg_udc->resetting = false;
+	}
+	return ret;
+}
+
+/*
+ * The gadget core serializes start and pullup with stop. Keep the IRQ and
+ * queue paths gated by gadget_stopping while the external reset may sleep.
+ */
+int crg_udc_restart_gadget(struct crg_gadget_dev *crg_udc,
+			   unsigned long *flags)
+{
+	int i, ret;
+
+	if (crg_udc->suspended)
+		return -EBUSY;
+
+	crg_udc_recovery_begin(crg_udc);
+	/*
+	 * A previous teardown or failed reconstruction may have left DMA
+	 * armed. Retry quiescing it before releasing any retained mappings.
+	 */
+	spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+	ret = crg_udc_quiesce_hw(crg_udc);
+	spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+	crg_udc->dma_quiesce_failed = !!ret;
+	if (ret)
+		goto out;
+
+	crg_udc_release_quarantine(crg_udc);
+	if (crg_udc->shutting_down) {
+		ret = -ESHUTDOWN;
+		goto out;
+	}
+	if (crg_udc->hw_reset_asserted) {
+		ret = crg_udc_deassert_reset(crg_udc, flags);
+		if (ret)
+			goto out;
+	}
+	if (crg_udc->shutting_down) {
+		ret = -ESHUTDOWN;
+		goto out;
+	}
+	ret = crg_udc_rebuild_after_stop(crg_udc);
+	if (ret)
+		goto out;
+
+	for (i = 2; i < CRG_NUM_EP_CX; i++)
+		crg_udc_ep_clear_enabled(&crg_udc->udc_ep[i]);
+	crg_udc->gadget_stopping = false;
+out:
+	crg_udc->controller_faulted = !!ret;
+	crg_udc_recovery_end(crg_udc);
+	return ret;
+}
+
+/*
+ * Run the optional reset in process context after the atomic SWRST path
+ * has returned the requests and quarantined any DMA-visible mappings.
+ */
+static int crg_udc_recover_external(struct crg_gadget_dev *crg_udc,
+				     unsigned long *flags)
+{
+	int ret;
+
+	crg_udc->external_reset_pending = false;
+	crg_udc_recovery_begin(crg_udc);
+	spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+	if (crg_udc->irq >= 0)
+		synchronize_irq(crg_udc->irq);
+	ret = reset_control_assert(crg_udc->rst);
+	spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+	if (ret)
+		goto out;
+
+	crg_udc->hw_reset_asserted = true;
+	crg_udc->hw_dma_armed = false;
+	crg_udc->dma_quiesce_failed = false;
+	crg_udc_release_quarantine(crg_udc);
+
+	if (crg_udc->shutting_down || crg_udc->gadget_stopping) {
+		ret = -ESHUTDOWN;
+		goto out;
+	}
+	ret = crg_udc_deassert_reset(crg_udc, flags);
+	if (ret)
+		goto out;
+	if (crg_udc->shutting_down || crg_udc->gadget_stopping) {
+		ret = -ESHUTDOWN;
+		goto out;
+	}
+	ret = crg_udc_rebuild_after_stop(crg_udc);
+out:
+	if (ret)
+		crg_udc->bus_reset_pending = false;
+	crg_udc->controller_faulted = !!ret;
+	crg_udc_recovery_end(crg_udc);
+	return ret;
+}
+
+static void crg_udc_restart_link(struct crg_gadget_dev *crg_udc,
+				 bool notify_reset)
+{
+	if (!crg_udc->async_callbacks || !crg_udc->gadget_driver ||
+	    crg_udc->reset_callback_active)
+		return;
+
+	if (notify_reset) {
+		crg_udc->bus_reset_pending = true;
+		crg_udc->restart_after_reset = true;
+		crg_udc_kick_reset_work(crg_udc);
+	} else if (!crg_udc->suspended && crg_udc->pullup_on) {
+		crg_udc_set_pullup(crg_udc, true);
+	}
+}
+
+static void crg_udc_recovery_work(struct work_struct *work)
+{
+	struct crg_gadget_dev *crg_udc =
+		container_of(work, struct crg_gadget_dev, recovery_work);
+	unsigned long flags;
+	int ret;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	if (crg_udc->controller_recovering || crg_udc->gadget_callbacks_active ||
+	    crg_udc->reset_callback_active ||
+	    crg_udc->suspended || crg_udc->shutting_down || crg_udc->gadget_stopping)
+		goto unlock;
+
+	if (crg_udc->endpoint_recovery_pending) {
+		crg_udc_recover_endpoints(crg_udc, &flags);
+		goto unlock;
+	}
+
+	if (!crg_udc->external_reset_pending)
+		goto unlock;
+
+	ret = crg_udc_recover_external(crg_udc, &flags);
+	if (ret) {
+		if (ret != -ESHUTDOWN)
+			dev_err(crg_udc->dev, "external reset recovery failed: %d\n", ret);
+		goto unlock;
+	}
+
+	crg_udc_restart_link(crg_udc, crg_udc->bus_reset_pending);
+unlock:
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+}
+
+/*
+ * Called with udc_lock held. Keep the entire controller reconstruction
+ * serialized, not just the SWRST register poll. In particular, request
+ * callbacks drop udc_lock, allowing stop/suspend and IRQs to re-enter.
+ */
+int crg_udc_recover(struct crg_gadget_dev *crg_udc, unsigned long *flags)
+{
+	int ret;
+
+	if (crg_udc->controller_recovering || crg_udc->shutting_down ||
+	    crg_udc->gadget_stopping || crg_udc->hw_reset_asserted)
+		return -EBUSY;
+
+	crg_udc_recovery_begin(crg_udc);
+	ret = crg_udc_reset(crg_udc, flags);
+	if (ret) {
+		crg_udc->controller_faulted = true;
+		crg_udc->dma_quiesce_failed = crg_udc->hw_dma_armed;
+		crg_udc_reset_queues(crg_udc, crg_udc->preserve_ep_enabled_on_reset);
+	}
+	if (!ret && !crg_udc->shutting_down && !crg_udc->gadget_stopping)
+		ret = crg_udc_reset_data_struct(crg_udc);
+	if (!ret && !crg_udc->shutting_down && !crg_udc->gadget_stopping)
+		ret = crg_udc_init_ep0(crg_udc);
+	if (!ret && (crg_udc->shutting_down || crg_udc->gadget_stopping))
+		ret = -ESHUTDOWN;
+
+	crg_udc->controller_faulted = !!ret;
+	crg_udc->resetting = !!ret;
+	if (!ret)
+		crg_udc->stopped_needs_reset = false;
+	crg_udc_recovery_end(crg_udc);
+	return ret;
+}
+
+/* Called with udc_lock held for recovery of a live USB session. */
+int crg_udc_recover_link(struct crg_gadget_dev *crg_udc, unsigned long *flags,
+			 bool notify_reset)
+{
+	int ret;
+
+	if (crg_udc->controller_recovering || crg_udc->shutting_down ||
+	    crg_udc->gadget_stopping || crg_udc->suspended ||
+	    crg_udc->hw_reset_asserted)
+		return -EBUSY;
+
+	notify_reset = notify_reset && crg_udc->async_callbacks &&
+		       crg_udc->gadget_driver && !crg_udc->reset_callback_active;
+	crg_udc->connected = false;
+	crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+	crg_udc->preserve_ep_enabled_on_reset = notify_reset;
+	ret = crg_udc_recover(crg_udc, flags);
+	crg_udc->preserve_ep_enabled_on_reset = false;
+
+	if (crg_udc->shutting_down || crg_udc->gadget_stopping)
+		return ret;
+
+	if (ret) {
+		dev_err(crg_udc->dev, "link recovery failed: %d\n", ret);
+		if (ret == -ETIMEDOUT && crg_udc->dma_quiesce_failed &&
+		    crg_udc->rst) {
+			crg_udc->external_reset_pending = true;
+			crg_udc->bus_reset_pending = notify_reset;
+			crg_udc_kick_reset_work(crg_udc);
+		}
+		return ret;
+	}
+
+	crg_udc_restart_link(crg_udc, notify_reset);
+	return 0;
+}
+
+/* A full endpoint recovery also invalidates the gadget configuration. */
+int crg_udc_recover_endpoints(struct crg_gadget_dev *crg_udc,
+			      unsigned long *flags)
+{
+	bool notify_reset = crg_udc->device_state == USB_STATE_CONFIGURED ||
+			    crg_udc->bus_reset_pending ||
+			    crg_udc->gadget_callbacks_active;
+
+	return crg_udc_recover_link(crg_udc, flags, notify_reset);
+}
+
+static const struct crg_udc_priv crg_udc_data = {
+	.quirks = CRG_UDC_QUIRKS_DEFAULT,
+};
+
+static const struct of_device_id of_crg_udc_match[] = {
+	{ .compatible = "axiado,ax3000-udc", .data = &crg_udc_data },
+	{ },
+};
+MODULE_DEVICE_TABLE(of, of_crg_udc_match);
+
+static DEFINE_SIMPLE_DEV_PM_OPS(crg_udc_pm_ops, crg_udc_suspend,
+				crg_udc_resume);
+
+static struct platform_driver crg_udc_driver = {
+	.probe = crg_udc_probe,
+	.remove = crg_udc_remove,
+	.shutdown = crg_udc_shutdown,
+	.driver = {
+		.name = KBUILD_MODNAME,
+		.of_match_table = of_crg_udc_match,
+		.pm = pm_sleep_ptr(&crg_udc_pm_ops),
+	},
+};
+module_platform_driver(crg_udc_driver);
+
+MODULE_AUTHOR("Axiado Corporation");
+MODULE_DESCRIPTION("Axiado AX3000/AX3005 USB Device Controller");
+MODULE_LICENSE("GPL");

-- 
2.34.1


^ permalink raw reply related	[flat|nested] 13+ messages in thread

* Re: [PATCH v5 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC
  2026-11-16 19:06 ` [PATCH v5 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC Petar Stepanovic
@ 2026-11-16 19:12   ` sashiko-bot
  0 siblings, 0 replies; 13+ messages in thread
From: sashiko-bot @ 2026-11-16 19:12 UTC (permalink / raw)
  To: Petar Stepanovic; +Cc: linux-usb

> Add a Devicetree binding for the USB device controller integrated into
> the Axiado AX3000 and AX3005 SoCs.
> 
> The controller is based on a Corigine USB 3.1 IP core that is
> dual-role capable (device and host), used here in device role. It
> provides SuperSpeed (5 Gb/s) or High-Speed (480 Mb/s) operation, and
> supports control, bulk, interrupt, and isochronous transfer types
> across multiple configurable endpoints.
> 
> Co-developed-by: Krutik Shah <krutikshah@axiado.com>
> Signed-off-by: Krutik Shah <krutikshah@axiado.com>
> Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
> Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
> Co-developed-by: Tzu-Hao Wei <twei@axiado.com>
> Signed-off-by: Tzu-Hao Wei <twei@axiado.com>
> Reviewed-by: Krzysztof Kozlowski <krzysztof.kozlowski@oss.qualcomm.com>
> Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261116-axiado-ax3000-usb-device-controller-v5-0-49271ed5c793@axiado.com?part=1


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [PATCH v5 2/6] usb: gadget: udc: crg: Add hardware definitions and structs
  2026-11-16 19:06 ` [PATCH v5 2/6] usb: gadget: udc: crg: Add hardware definitions and structs Petar Stepanovic
@ 2026-11-16 19:24   ` sashiko-bot
  0 siblings, 0 replies; 13+ messages in thread
From: sashiko-bot @ 2026-11-16 19:24 UTC (permalink / raw)
  To: Petar Stepanovic; +Cc: linux-usb

> This patch introduces the hardware definitions, register offsets, and
> fundamental data structures for the Axiado AX3000/AX3005 USB Device
> Controller driver.
> 
> Co-developed-by: Krutik Shah <krutikshah@axiado.com>
> Signed-off-by: Krutik Shah <krutikshah@axiado.com>
> Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
> Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
> Co-developed-by: Tzu-Hao Wei <twei@axiado.com>
> Signed-off-by: Tzu-Hao Wei <twei@axiado.com>
> Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261116-axiado-ax3000-usb-device-controller-v5-0-49271ed5c793@axiado.com?part=2


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [PATCH v5 3/6] usb: gadget: udc: crg: Add ring and transfer management
  2026-11-16 19:06 ` [PATCH v5 3/6] usb: gadget: udc: crg: Add ring and transfer management Petar Stepanovic
@ 2026-11-16 19:46   ` sashiko-bot
  0 siblings, 0 replies; 13+ messages in thread
From: sashiko-bot @ 2026-11-16 19:46 UTC (permalink / raw)
  To: Petar Stepanovic; +Cc: linux-usb, devicetree, robh, conor+dt

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] usb: gadget: udc: crg: missing session check in
  crg_udc_queue_pending_trbs()
--

commit 4959c938e3393a398d380564ec6cf72791b96459
Author: Petar Stepanovic <pstepanovic@axiado.com>

usb: gadget: udc: crg: Add ring and transfer management

This patch adds event ring initialization, event processing, and transfer
ring TRB management for the Axiado AX3000/AX3005 UDC driver.

> diff --git a/drivers/usb/gadget/udc/crg/crg-udc-ring.c b/drivers/usb/gadget/udc/crg/crg-udc-ring.c
> new file mode 100644
> index 0000000000000..348d2b611a332
> --- /dev/null
> +++ b/drivers/usb/gadget/udc/crg/crg-udc-ring.c
[ ... ]
> +int crg_udc_queue_pending_trbs(struct crg_udc_ep *udc_ep_ptr)
> +{
> +	struct crg_udc_request *req;
> +	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
> +	u32 generation = crg_udc->udc_event[0].generation;
> +	u32 ep_generation = udc_ep_ptr->generation;
> +	int ret;
> +
> +restart:
> +	/* A giveback can reset/disable the endpoint while udc_lock is dropped. */
> +	if (crg_udc->resetting || crg_udc->port_reset_pending ||
> +	    crg_udc->endpoint_recovery_pending ||
> +	    !udc_ep_ptr->desc || udc_ep_ptr->halt_failed || udc_ep_ptr->nuking ||
> +	    udc_ep_ptr->ep_state != EP_STATE_RUNNING)
> +		return 0;
> +
> +	list_for_each_entry(req, &udc_ep_ptr->queue, queue) {
> +		if (req->all_trbs_queued)
> +			continue;
> +
> +		/* A large request must finish its current TD before the next
> +		 * TD (or the next USB request) can be submitted: the event
> +		 * handler uses all_trbs_queued to complete the USB request.
> +		 */
> +		if (req->td_pending)
> +			return 0;
> +
> +		ret = crg_udc_build_td(udc_ep_ptr, req);
> +		if (ret == -ENOSPC || ret == -EBUSY)
> +			return ret;
> +		if (ret) {
> +			dev_err(crg_udc->dev,
> +				"failed to build pending transfer: %d\n", ret);
> +			/* Giveback removes req and drops udc_lock: callbacks can
> +			 * remove or free every saved request-list iterator.
> +			 */
> +			crg_udc_req_done(udc_ep_ptr, req, ret);
> +			if (generation != crg_udc->udc_event[0].generation ||
> +			    ep_generation != udc_ep_ptr->generation)
> +				return 0;
> +			goto restart;
> +		}

[Severity: Medium]
Can crg_udc_queue_pending_trbs() publish TRBs and ring the doorbell after a
concurrent suspend or teardown has stopped the controller?

When crg_udc_req_done() drops udc_lock (either inside this loop or in a
caller right before invoking crg_udc_queue_pending_trbs()), another context
can stop the session before controller reset runs:

CPU0                                  CPU1
crg_udc_handle_xfer_event()
  crg_udc_req_done()
    spin_unlock(&crg_udc->udc_lock)
                                      crg_udc_suspend()
                                        crg_udc->suspended = true;
                                        crg_udc_set_pullup(..., false)
                                          /* clears CRG_U3DC_CTRL_RUN */
                                        spin_unlock(&crg_udc->udc_lock)
                                        synchronize_irq()
    spin_lock(&crg_udc->udc_lock)
  crg_udc_queue_pending_trbs()
    crg_udc_build_td()                <-- publishes TRBs and rings doorbell

During this window, crg_udc->resetting is still false, generations have not
changed, and data endpoints remain in EP_STATE_RUNNING, so the check at
restart passes even though crg_udc_session_ready(crg_udc) is false.

Checking !crg_udc_session_ready(crg_udc) at restart would prevent queueing
TRBs once the session is no longer ready.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261116-axiado-ax3000-usb-device-controller-v5-0-49271ed5c793@axiado.com?part=3

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [PATCH v5 4/6] usb: gadget: udc: crg: Add endpoint 0 handling
  2026-11-16 19:06 ` [PATCH v5 4/6] usb: gadget: udc: crg: Add endpoint 0 handling Petar Stepanovic
@ 2026-11-16 20:16   ` sashiko-bot
  0 siblings, 0 replies; 13+ messages in thread
From: sashiko-bot @ 2026-11-16 20:16 UTC (permalink / raw)
  To: Petar Stepanovic; +Cc: linux-usb, devicetree, robh, conor+dt

Thank you for your contribution! Sashiko AI review found 5 potential
issues to consider:
- [High] usb: gadget: udc: crg: broken EP0 halt and lock-drop handling
  in setfeaturesrequest()
- [Medium] usb: gadget: udc: crg: clear U1 and U2 LPM state on
  SET_CONFIGURATION(0)
- [Medium] usb: gadget: udc: crg: mishandled interface GET_STATUS and
  FUNC_SUSPEND on EP0
- [Medium] usb: gadget: crg: missing setup_status error return and
  ep0_state checks on EP0
- [Medium] usb: gadget: crg: missing ep0_state checks in EP0 status and
  halt paths
--

commit 38b91eca05d2bb158aa1aa4be9dbbe69096a9ef9
Author: Petar Stepanovic <pstepanovic@axiado.com>

usb: gadget: udc: crg: Add endpoint 0 handling

This commit adds endpoint 0 setup packet, status stage, and control transfer
handling for the Axiado AX3000/AX3005 UDC driver.

> diff --git a/drivers/usb/gadget/udc/crg/crg-udc-ep0.c b/drivers/usb/gadget/udc/crg/crg-udc-ep0.c
> new file mode 100644
> index 0000000000000..4bd0dd4da8c67
> --- /dev/null
> +++ b/drivers/usb/gadget/udc/crg/crg-udc-ep0.c
> @@ -0,0 +1,1264 @@
[ ... ]
> +int crg_udc_build_ep0_status(struct crg_udc_ep *udc_ep_ptr, bool default_value,
> +			      u32 status, struct crg_udc_request *udc_req_ptr,
> +			      u8 set_addr, u8 stall)
> +{
> +	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
> +	struct crg_transfer_trb *enq_pt = udc_ep_ptr->enq_pt;
> +
> +	/* Internal status requests bypass usb_ep_queue(), so they must also
> +	 * respect a teardown that is giving back the old EP0 queue.
> +	 */
> +	if (udc_ep_ptr->nuking) {
> +		crg_udc->ep0_state = EP0_STATE_ERROR;
> +		return -EBUSY;
> +	}
> +
> +	/* Reserve a free transfer-ring slot before touching request state or
> +	 * publishing the STATUS TRB. crg_udc_room_on_ring() treats equal
> +	 * pointers as empty, so check tran_ring_full separately.
> +	 */
> +	if (!udc_ep_ptr->first_trb || !udc_ep_ptr->last_trb ||
> +	    !udc_ep_ptr->enq_pt || !udc_ep_ptr->deq_pt)
> +		return -ENODEV;
> +	if (udc_ep_ptr->ep_state != EP_STATE_RUNNING)
> +		return -ESHUTDOWN;

[Severity: Medium]
Should crg_udc_build_ep0_status() and crg_udc_set_ep0_halt() check that
crg_udc->ep0_state is EP0_STATE_IDLE before queuing a status or stall TRB?

crg_udc_handle_setup_pkt() sets crg_udc->ep0_state = EP0_STATE_ABORTING
while udc_ep[0].ep_state is still EP_STATE_RUNNING, then drops udc_lock
across flush_work(&crg_udc->udc_reset_work):

CPU0                              CPU1
crg_udc_handle_setup_pkt()
  ep0_state = EP0_STATE_ABORTING
  spin_unlock(&udc_lock)
  flush_work(&udc_reset_work)
                                  crg_udc_ep_set_halt(ep0)
                                    ...
                                      crg_udc_set_ep0_halt()
                                        crg_udc_build_ep0_status()
                                        ep_state = EP_STATE_HALTED
  spin_lock(&udc_lock)
  if (ep_state != EP_STATE_RUNNING)
    return;  <-- drops new SETUP with ep0_state stuck in EP0_STATE_ABORTING

Because neither crg_udc_set_ep0_halt() nor crg_udc_build_ep0_status() checks
crg_udc->ep0_state, a concurrent halt or status submission while ep0_state
is EP0_STATE_ABORTING or EP0_STATE_ERROR queues status_req with the previous
setup_tag, rings doorbell 0, and sets ep_state to EP_STATE_HALTED without
bumping generation, causing crg_udc_handle_setup_pkt() to abort the incoming
SETUP and leave EP0 blocked in EP0_STATE_ABORTING.

[ ... ]

> +static bool setfeaturesrequest(struct crg_gadget_dev *crg_udc, u8 request_type,
> +			       u8 b_request, u16 value, u16 index, u16 length,
> +			       unsigned long *flags)
> +{
> +	int status = -EINPROGRESS;
> +	u8 dci;
> +	struct crg_udc_ep *udc_ep_ptr;
> +	u32 tmp;
> +	u32 set_feat = 0;
> +	struct crg_uccr __iomem *uccr = crg_udc->uccr;
[ ... ]
> +		udc_ep_ptr = &crg_udc->udc_ep[dci];
> +		status = crg_udc_ep_halt(
> +			udc_ep_ptr, (b_request == USB_REQ_SET_FEATURE) ? 1 : 0,
> +			0, flags);
> +
> +		if (status < 0)
> +			goto set_feature_error;
[ ... ]
> +	} else if ((request_type & (USB_RECIP_MASK | USB_TYPE_MASK)) ==
> +		   (USB_RECIP_INTERFACE | USB_TYPE_STANDARD)) {
[ ... ]
> +		/* Do not need to return status stage here
> +		 * Pass to composite gadget driver to process
> +		 * the request
> +		 */
> +		return false;
> +	} else {
> +		/* Unknown recipient must not receive a successful status stage. */
> +		goto set_feature_error;
> +	}
> +
> +	crg_udc->setup_status = STATUS_STAGE_XFER;
> +	crg_udc_submit_ep0_status(crg_udc, status, 0);
> +	return true;
> +
> +set_feature_error:
> +	crg_udc_set_ep0_halt(crg_udc);
> +	return true;
> +}

[Severity: High]
Does the USB_RECIP_ENDPOINT path in setfeaturesrequest() mishandle EP0
targets, status-stage request state, and lock drops across
crg_udc_ep_halt()?

First, when SET_FEATURE(USB_ENDPOINT_HALT) targets EP0 (dci == 0),
crg_udc_ep_halt() calls crg_udc_set_ep0_halt(), which already queues
status_req with stall = 1 and sets udc_ep[0].ep_state = EP_STATE_HALTED;
falling through to crg_udc_submit_ep0_status() then fails with -ESHUTDOWN
and forces crg_udc->ep0_state into EP0_STATE_ERROR.

Second, crg_udc_ep_halt() can drop udc_lock via flags during request
givebacks in crg_udc_ep_nuke() or crg_udc_queue_pending_trbs(), but
setfeaturesrequest() does not revalidate EP0/event generations or reset
state before calling crg_udc_submit_ep0_status() or crg_udc_set_ep0_halt(),
which can publish a status or stall TRB and ring doorbell 0 during a reset
or on a reinitialized EP0 ring.

Third, assigning status = crg_udc_ep_halt(...) overwrites status from
-EINPROGRESS to 0 on success, so crg_udc_submit_ep0_status() queues the
in-flight status_req with usb_req.status = 0 before hardware completion.

[ ... ]

> +	if ((setup_pkt->bRequestType & USB_TYPE_MASK) == USB_TYPE_STANDARD) {
> +		switch (setup_pkt->bRequest) {
> +		case USB_REQ_GET_STATUS:
> +			if (setup_pkt->bRequestType !=
> +				    (USB_DIR_IN | USB_RECIP_DEVICE |
> +				     USB_TYPE_STANDARD) &&
> +			    setup_pkt->bRequestType !=
> +				    (USB_DIR_IN | USB_RECIP_INTERFACE |
> +				     USB_TYPE_STANDARD) &&
> +			    setup_pkt->bRequestType !=
> +				    (USB_DIR_IN | USB_RECIP_ENDPOINT |
> +				     USB_TYPE_STANDARD)) {
> +				crg_udc_set_ep0_halt(crg_udc);
> +				return;
> +			}
> +
> +			getstatusrequest(crg_udc, setup_pkt->bRequestType,
> +					 w_value, w_index, w_length);
> +			return;

[Severity: Medium]
Should SuperSpeed USB_REQ_GET_STATUS for USB_RECIP_INTERFACE be delegated to
driver->setup(), and should setfeaturesrequest() reject
USB_INTRF_FUNC_SUSPEND below SuperSpeed?

setfeaturesrequest() delegates USB_INTRF_FUNC_SUSPEND to driver->setup() so
the composite driver can update function wakeup state, but
crg_udc_handle_setup_pkt() intercepts USB_REQ_GET_STATUS for
USB_RECIP_INTERFACE and getstatusrequest() always replies with 0 instead of
letting composite_setup() report USB_INTRF_STAT_FUNC_RW_CAP and
USB_INTRF_STAT_FUNC_RW.  In addition, the USB_RECIP_INTERFACE branch in
setfeaturesrequest() does not check crg_udc->gadget.speed >= USB_SPEED_SUPER,
and composite_setup() only checks gadget_is_superspeed() (which tests
max_speed), so USB_INTRF_FUNC_SUSPEND is accepted at High-Speed and
Full-Speed instead of stalling EP0.

[ ... ]

> +	if (setup_pkt->bRequest == USB_REQ_SET_CONFIGURATION &&
> +	    (setup_pkt->bRequestType & (USB_TYPE_MASK | USB_RECIP_MASK)) ==
> +		    (USB_TYPE_STANDARD | USB_RECIP_DEVICE)) {
> +		if (w_value) {
> +			crg_udc->device_state = USB_STATE_CONFIGURED;
> +			usb_gadget_set_state(&crg_udc->gadget,
> +					     USB_STATE_CONFIGURED);
> +		} else {
> +			/* Endpoint disable may already have updated device_state. */
> +			crg_udc->device_state = USB_STATE_ADDRESS;
> +			usb_gadget_set_state(&crg_udc->gadget,
> +					     USB_STATE_ADDRESS);
> +		}
> +	}
> +}

[Severity: Medium]
When USB_REQ_SET_CONFIGURATION with w_value == 0 transitions the device back
to USB_STATE_ADDRESS, should it also clear U1/U2 link power management state
via crg_udc_clear_portpm()?

Without clearing uccr->u3portpmsc, feature_u1_enable, and feature_u2_enable
on deconfiguration, hardware U1/U2 initiation and timeouts stay active in
USB_STATE_ADDRESS, getstatusrequest() continues reporting
USB_DEV_STAT_U1_ENABLED and USB_DEV_STAT_U2_ENABLED, and the host cannot
clear them via CLEAR_FEATURE because setfeaturesrequest() rejects U1/U2
requests outside USB_STATE_CONFIGURED.

[ ... ]

> +int crg_udc_queue_ctrl(struct crg_udc_ep *udc_ep_ptr,
> +		       struct crg_udc_request *udc_req_ptr,
> +		       u32 num_of_trbs_needed)
> +{
[ ... ]
> +	if (crg_udc->ep0_state == EP0_STATE_ABORTING)
> +		return -ECONNRESET;
> +
> +	ep_state = crg_udc_get_ep_state(crg_udc, 0);
> +	/* Need to queue the request even ep is paused or halted */
> +	if (ep_state != EP_STATE_RUNNING) {
> +		dev_dbg(crg_udc->dev, "EP State = 0x%x\n", ep_state);
> +		return -EINVAL;
> +	}
> +	if (list_empty(&udc_ep_ptr->queue)) {
> +		/* For control endpoint, we can handle one setup request at a
> +		 * time. so if there are TD pending in the transfer ring.
> +		 * wait for the sequence number error event. Then put the new
> +		 * request to transfer ring
> +		 */
> +		if (enq_pt == dq_pt && !udc_ep_ptr->tran_ring_full) {
> +			u32 i, td_done = 0;
> +			bool need_zlp = false;
> +			struct crg_udc_bounce *bounce = udc_req_ptr->bounce;
> +			dma_addr_t trb_dma = bounce ? bounce->dma : 0;
> +			u64 remaining = usb_req->length;
> +			/* Transfer ring is empty
> +			 * setup data stage TRBs
> +			 */
> +			udc_req_ptr->first_trb = udc_ep_ptr->enq_pt;
> +			if (crg_udc->setup_status == DATA_STAGE_XFER)
> +				dir = 0;
> +			else if (crg_udc->setup_status == DATA_STAGE_RECV)
> +				dir = 1;
> +			else
> +				dev_dbg(crg_udc->dev,
> +					"unexpected setup_status!%d\n",
> +					crg_udc->setup_status);

[Severity: Medium]
Should crg_udc_queue_ctrl() return -EINVAL when setup_status is neither
DATA_STAGE_XFER nor DATA_STAGE_RECV, and also reject requests when
crg_udc->ep0_state != EP0_STATE_IDLE?

In crg_udc_ep_queue(), a non-zero-length EP0 request queued while
setup_status is STATUS_STAGE_XFER or STATUS_STAGE_RECV (for example, from a
completion callback during status-stage giveback before setup_status is
reset to WAIT_FOR_SETUP) passes the !_req->length check and reaches
crg_udc_queue_ctrl().  Because the else branch only logs dev_dbg(),
crg_udc_queue_ctrl() proceeds to queue TRB_TYPE_XFER_DATA_STAGE TRBs with
dir = 0 and ring the EP0 doorbell.  Similarly, checking only
ep0_state == EP0_STATE_ABORTING at entry allows control requests to be
queued while ep0_state is EP0_STATE_ERROR.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261116-axiado-ax3000-usb-device-controller-v5-0-49271ed5c793@axiado.com?part=4

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [PATCH v5 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations
  2026-11-16 19:06 ` [PATCH v5 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations Petar Stepanovic
@ 2026-11-16 20:47   ` sashiko-bot
  0 siblings, 0 replies; 13+ messages in thread
From: sashiko-bot @ 2026-11-16 20:47 UTC (permalink / raw)
  To: Petar Stepanovic; +Cc: linux-usb, devicetree, robh, conor+dt

Thank you for your contribution! Sashiko AI review found 10 potential
issues to consider:
- [High] usb: gadget: crg: stale deq_pt in crg_udc_ep_dequeue() when
  pause_pt passed TD
- [High] usb: gadget: crg: stuck USB_STATE_RECONNECTING in worker
  port-status paths
- [High] usb: gadget: crg: isoc endpoints wedged in EP_STATE_STOPPED on
  xfer error
- [Medium] usb: gadget: crg: broken EP0 USB_ENDPOINT_HALT handling in
  crg_udc_ep_halt()
- [Medium] usb: gadget: crg: request rejected on ring-full in
  crg_udc_ep_queue()
- [Medium] usb: gadget: crg: premature USB_STATE_ADDRESS in
  crg_udc_ep_disable()
- [Medium] usb: gadget: udc: crg: lost port reset poll in
  crg_udc_port_retry_work()
- [Medium] usb: gadget: udc: crg: controller reset on nested
  crg_udc_ep_dequeue()
- [Medium] usb: gadget: udc: crg: dequeue pointer rewind in
  crg_udc_ep_halt()
- [Low] usb: gadget: crg: incomplete cleanup in crg_udc_ep_disable()
  when disabled
--

commit 4ce545406bad41a031bcbc10a89cb66370b3d324
Author: Petar Stepanovic <pstepanovic@axiado.com>

usb: gadget: udc: crg: Add gadget and endpoint operations

This commit adds the USB gadget and endpoint operations, transfer ring
management, port status handling, and error recovery for the Axiado
AX3000/AX3005 UDC driver.

> diff --git a/drivers/usb/gadget/udc/crg/crg-udc-gadget.c b/drivers/usb/gadget/udc/crg-udc-gadget.c
> new file mode 100644
> index 0000000000000..dddfc5cf3d3ec
> --- /dev/null
> +++ b/drivers/usb/gadget/udc/crg/crg-udc-gadget.c

[ ... ]

> +static int crg_udc_ep_dequeue(struct usb_ep *_ep, struct usb_request *_req)
> +{

[ ... ]

> +	/* A stopped ring may still contain published requests. */
> +	if (old_ep_state == EP_STATE_STOPPED) {
> +		ret = -EBUSY;
> +		goto recover;
> +	}
> +
> +	if (old_ep_state == EP_STATE_RUNNING) {
> +		ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP,
> +						BIT(dci), 0);
> +
> +		if (ret)
> +			goto recover;
> +		ret = readl_poll_timeout_atomic(&uccr->ep_running, tmp,
> +					       !(tmp & BIT(dci)), 10, 10000);
> +		if (ret)
> +			goto recover;
> +		udc_ep_ptr->ep_state = EP_STATE_STOPPED;
> +		udc_ep_ptr->dequeue_paused = true;
> +	}

[Severity: Medium]
Does a nested or concurrent crg_udc_ep_dequeue() trigger a full controller
recovery reset when a second request with live TRBs is dequeued during
giveback?

crg_udc_ep_dequeue(ep, req1)
  ep->ep_state = EP_STATE_STOPPED
  ep->dequeue_paused = true
  crg_udc_req_done(ep, req1, -ECONNRESET)
    spin_unlock(&crg_udc->udc_lock)
    usb_gadget_giveback_request()
      req1->complete()
        usb_ep_dequeue(ep, req2)
          crg_udc_ep_dequeue(ep, req2)
            old_ep_state == EP_STATE_STOPPED
            goto recover  <-- schedules controller reset

When req2 already has first_trb set on the stopped ring, the inner
crg_udc_ep_dequeue() sees old_ep_state == EP_STATE_STOPPED without checking
udc_ep_ptr->dequeue_paused, marks halt_failed = true, and schedules a full
controller reset that drops the USB connection.

> +	p_ep_cx = &crg_udc->p_epcx[dci - 2];
> +	deq_pt_lo = le32_to_cpu(p_ep_cx->dw2) & EP_CX_TR_DQPT_LO_MASK;
> +	deq_pt_hi = le32_to_cpu(p_ep_cx->dw3);
> +	dq_pt_addr = (u64)deq_pt_lo | ((u64)deq_pt_hi << 32);
> +	pause_pt = crg_udc_tran_trb_dma_to_virt(udc_ep_ptr, dq_pt_addr);
> +	if (!pause_pt) {
> +		ret = -EIO;
> +		goto recover;
> +	}
> +
> +	if (crg_udc_is_pointer_less_than(pause_pt, udc_req->first_trb,
> +					 udc_ep_ptr)) {
> +		crg_udc_squeeze_xfer_ring(udc_ep_ptr, udc_req);
> +		/* Give back the cancelled request BEFORE scheduling the tail:
> +		 * its callback may add/remove requests, and dropping udc_lock
> +		 * while traversing the old list is unsafe.
> +		 */
> +		crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
> +	} else if (udc_req->last_trb &&
> +		   crg_udc_is_pointer_less_than(udc_req->last_trb, pause_pt,
> +						udc_ep_ptr)) {
> +		crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
> +	} else {

[Severity: High]
Does the else-if branch leave udc_ep_ptr->deq_pt stuck at udc_req->first_trb
when pause_pt has already advanced past udc_req->last_trb?

When hardware has consumed the TD's TRBs and its Transfer Event is still
pending on the event ring, crg_udc_req_done() unlinks udc_req from
udc_ep_ptr->queue and unmaps udc_req->bounce with usb_req.actual still 0
(discarding received OUT data).  When crg_udc_handle_xfer_event() later
processes the pending event, is_request_dequeued() sees that udc_req is no
longer on the queue and returns before calling crg_udc_update_dequeue_pt().
That leaks the request's TRB slots in crg_udc_room_on_ring(), and if the
endpoint is later halted and unhalted at High or Full Speed,
crg_udc_ep_halt() calls crg_udc_epcx_update_dqptr() and reprograms the
hardware dequeue pointer back to the unmapped TRBs.

[ ... ]

> +static int crg_udc_ep_disable(struct usb_ep *ep)
> +{

[ ... ]

> +	if (udc_ep->ep_state == EP_STATE_DISABLED) {
> +		/*
> +		 * A reset can mark this endpoint disabled before reaching its
> +		 * queue. A callback from another endpoint may disable it now.
> +		 * Give back its requests before returning success.
> +		 */
> +		crg_udc_ep_nuke(udc_ep, -ESHUTDOWN);
> +		udc_ep->desc = NULL;
> +		udc_ep->comp_desc = NULL;
> +		udc_ep->usb_ep.enabled = false;
> +		ret = 0;
> +		goto out;
> +	}

[Severity: Low]
Should this path clear udc_ep->desc, udc_ep->comp_desc, and
crg_udc->p_epcx[udc_ep->dci - 2] before calling crg_udc_ep_nuke()?

When crg_udc_reinit() disables endpoints for a bus reset with notify_reset
set, it preserves desc, comp_desc, usb_ep.enabled, and p_epcx for the
gadget's usb_ep_disable() call.  Taking this early branch leaves p_epcx
unzeroed (since crg_udc_reset_work() skips endpoints once usb_ep.enabled is
false) and exposes non-NULL descriptors while crg_udc_ep_nuke() drops
udc_lock for request givebacks.

[ ... ]

> +	if (!crg_udc->num_enabled_eps &&
> +	    crg_udc->device_state == USB_STATE_CONFIGURED)
> +		crg_udc->device_state = USB_STATE_ADDRESS;

[Severity: Medium]
Should disabling the last non-EP0 endpoint downgrade crg_udc->device_state
to USB_STATE_ADDRESS when the USB configuration is still active?

If a configured gadget disables all data endpoints (for example, when
switching an interface to alternate setting 0), the device is still in the
Configured state until SET_CONFIGURATION(0) or a bus reset occurs.
Setting USB_STATE_ADDRESS here causes getstatusrequest() and
setfeaturesrequest() to stall valid Configured-state control requests, and
causes crg_udc_handle_port_status() to see was_configured == false on a
subsequent bus reset and skip usb_gadget_udc_reset().

[ ... ]

> +static int crg_udc_ep_queue(struct usb_ep *_ep, struct usb_request *_req,
> +			    gfp_t gfp_flags)
> +{

[ ... ]

> +	if (udc_ep_ptr->ep_state != EP_STATE_RUNNING) {
> +		status = 0;
> +	} else if (!list_empty(&udc_ep_ptr->queue)) {
> +		struct crg_udc_request *tail;
> +
> +		tail = list_last_entry(&udc_ep_ptr->queue,
> +				       struct crg_udc_request, queue);
> +		status = (!tail->all_trbs_queued ||
> +			  udc_ep_ptr->tran_ring_full) ? 0 :
> +			 crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
> +	} else if (udc_ep_ptr->tran_ring_full) {
> +		status = 0;
> +	} else {
> +		status = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
> +	}
> +
> +	if (!status) {
> +		list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
> +		goto out_locked;
> +	}
> +	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +
> +unmap:
> +	crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr, 0);

[Severity: Medium]
When crg_udc_build_td() returns -EBUSY or -ENOSPC, should crg_udc_ep_queue()
set tran_ring_full = true and keep the request on udc_ep_ptr->queue instead
of unmapping and failing it?

On an isochronous endpoint with fewer free ring slots than td_trbs,
crg_udc_queue_trbs() returns -EBUSY without setting tran_ring_full.
Similarly, crg_udc_ep_dequeue() clears tran_ring_full = false before calling
crg_udc_queue_pending_trbs(), which returns 0 without calling
crg_udc_build_td() if all queued requests already have all_trbs_queued or
td_pending set; the next crg_udc_ep_queue() call on that full ring then
invokes crg_udc_build_td() and fails with -ENOSPC.  All other callers of
crg_udc_build_td() treat -ENOSPC and -EBUSY as ring-full conditions that
defer the request.

[ ... ]

> +void crg_udc_port_retry_work(struct work_struct *work)
> +{
> +	struct crg_gadget_dev *crg_udc =
> +		container_of(to_delayed_work(work), struct crg_gadget_dev,
> +			     port_retry_work);
> +	unsigned long flags;
> +	int ret;
> +
> +	spin_lock_irqsave(&crg_udc->udc_lock, flags);
> +	if (!crg_udc->port_reset_pending && !crg_udc->deferred_port_events)
> +		goto out;
> +	if (crg_udc->resetting || crg_udc->suspended ||
> +	    crg_udc->shutting_down || !crg_udc->pullup_on ||
> +	    !crg_udc->gadget_driver || !crg_udc->async_callbacks)
> +		goto out;
> +	/* Reset work replays deferred events after the callback. A controller
> +	 * reconstruction is different: retry once its register state is stable.
> +	 */
> +	if (crg_udc->controller_recovering) {
> +		mod_delayed_work(system_wq, &crg_udc->port_retry_work,
> +				 msecs_to_jiffies(CRG_PORT_RESET_RETRY_MS));
> +		goto out;
> +	}
> +	if (crg_udc->port_reset_pending &&
> +	    crg_udc->port_reset_retry_count++ >= CRG_PORT_RESET_MAX_RETRIES) {
> +		bool notify_reset = crg_udc->device_state == USB_STATE_CONFIGURED;
> +
> +		dev_err(crg_udc->dev,
> +			"port reset did not become ready; resetting controller\n");
> +		ret = crg_udc_recover_port_failure(crg_udc, &flags,
> +						   notify_reset);
> +		if (ret != -ECONNRESET)
> +			dev_err(crg_udc->dev,
> +				"port-reset recovery returned %d\n", ret);
> +		goto out;
> +	}
> +	ret = crg_udc_handle_port_status(crg_udc, &flags);
> +	if (ret && ret != -ECONNRESET)
> +		dev_dbg(crg_udc->dev, "port reset poll: %d\n", ret);
> +out:
> +	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +}

[Severity: Medium]
Can a port reset poll be permanently lost if crg_udc_port_retry_work() runs
while crg_udc->reset_callback_active is true?

kworker (udc_reset_work)          kworker (port_retry_work)
crg_udc_reset_work()
  reset_callback_active = true
  spin_unlock(&udc_lock)
  usb_gadget_udc_reset()
                                  crg_udc_port_retry_work()
                                    crg_udc_handle_port_status()
                                      portsc_val & W1C_MASK == 0
                                      if (reset_callback_active)
                                        deferred_port_events |= 0
                                        return 0
  spin_lock(&udc_lock)
  reset_callback_active = false
  if (deferred_port_events)  <-- false; port_reset_pending stays true

Because CRG_U3DC_PORTSC_PRC was already W1C-cleared when port_reset_pending
was first set, crg_udc_handle_port_status() sees no new W1C bits during
reset_callback_active and returns early without rescheduling
port_retry_work.  crg_udc_reset_work() then only checks
deferred_port_events, leaving port_reset_pending stuck true and blocking EP0
setup handling, endpoint enablement, and request queueing.  In addition, if
port_reset_retry_count reaches CRG_PORT_RESET_MAX_RETRIES while
reset_callback_active is true, crg_udc_port_retry_work() runs
crg_udc_recover_port_failure() concurrently with usb_gadget_udc_reset().

[ ... ]

> +int crg_udc_handle_port_status(struct crg_gadget_dev *crg_udc,
> +			       unsigned long *flags)
> +{

[ ... ]

> +			if (crg_udc->device_state >= USB_STATE_DEFAULT) {

[ ... ]

> +				reinit_ret = crg_udc_reinit(crg_udc, notify_reset);

[ ... ]

> +			}
> +
> +			crg_udc->gadget.speed = speed;
> +
> +			ret = crg_udc_update_ep0_maxpacketsize(crg_udc);
> +			if (ret) {
> +				dev_err(crg_udc->dev,
> +					"failed to configure EP0 maxpacket: %d\n", ret);
> +				return crg_udc_recover_port_failure(
> +					crg_udc, flags,
> +					crg_udc->bus_reset_pending ||
> +					crg_udc->device_state == USB_STATE_CONFIGURED);
> +			}
> +
> +			crg_udc->connected = 1;
> +
> +			if (crg_udc->device_state < USB_STATE_RECONNECTING)
> +				crg_udc_enable_setup(crg_udc);
> +		}
> +	}

[Severity: High]
Does crg_udc->device_state remain stuck in USB_STATE_RECONNECTING when a
port reset completes via crg_udc_port_retry_work() or crg_udc_reset_work(),
or after a link drop with VBUS present?

When device_state >= USB_STATE_DEFAULT, crg_udc_reinit() sets device_state
to USB_STATE_RECONNECTING, so crg_udc_handle_port_status() skips
crg_udc_enable_setup() expecting the caller to drain the event ring and call
crg_udc_prepare_for_setup().  Neither crg_udc_port_retry_work() nor
crg_udc_reset_work() calls crg_udc_prepare_for_setup() after
crg_udc_handle_port_status(), and the link-drop path (cable_connected &&
!ccs_drop_ignore) also calls crg_udc_reinit() with connected = 0 so a later
CSC/PEC/PPC reconnect skips crg_udc_enable_setup() as well.

Also, while port_reset_pending is true before crg_udc_reinit() runs,
device_state is not yet USB_STATE_RECONNECTING; if a new SETUP event arrives
before port_retry_work polls PED, crg_udc_handle_event() dispatches it to
crg_udc_handle_setup_pkt(), which sets ep0_state = EP0_STATE_ABORTING and
returns early on port_reset_pending, dropping the SETUP packet and wedging
EP0.

[ ... ]

> +int crg_udc_ep_halt(struct crg_udc_ep *udc_ep_ptr, int halt, int ignore_wedge,
> +		    unsigned long *flags)
> +{

[ ... ]

> +	if (udc_ep_ptr->dci == 0) {
> +		/* EP0 stalls with a STATUS-stage TRB, not SET_HALT. */
> +		if (!halt)
> +			return -EOPNOTSUPP;
> +
> +		return crg_udc_set_ep0_halt(crg_udc);
> +	}

[Severity: Medium]
Does this break USB_REQ_SET_FEATURE and USB_REQ_CLEAR_FEATURE(ENDPOINT_HALT)
targeting EP0 in setfeaturesrequest()?

drivers/usb/gadget/udc/crg/crg-udc-ep0.c:setfeaturesrequest() {
    ...
		udc_ep_ptr = &crg_udc->udc_ep[dci];
		status = crg_udc_ep_halt(
			udc_ep_ptr, (b_request == USB_REQ_SET_FEATURE) ? 1 : 0,
			0, flags);

		if (status < 0)
			goto set_feature_error;
    ...
	crg_udc->setup_status = STATUS_STAGE_XFER;
	crg_udc_submit_ep0_status(crg_udc, status, 0);
}

On CLEAR_FEATURE(USB_ENDPOINT_HALT) for EP0, returning -EOPNOTSUPP causes
setfeaturesrequest() to stall EP0 instead of completing the status stage.
On SET_FEATURE(USB_ENDPOINT_HALT) for EP0, crg_udc_set_ep0_halt() queues a
STALL status TRB using crg_udc->status_req, sets ep_state = EP_STATE_HALTED,
and returns 0; setfeaturesrequest() then calls crg_udc_submit_ep0_status() a
second time, which fails on the halted endpoint and sets ep0_state =
EP0_STATE_ERROR so crg_udc_handle_xfer_event() ignores the STALL completion.

[ ... ]

> +		if (recovering || crg_udc->gadget.speed >= USB_SPEED_SUPER) {
> +			udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
> +			udc_ep_ptr->tran_ring_full = false;
> +		}
> +		udc_ep_ptr->ep_state = EP_STATE_HALTED;
> +		/*
> +		 * A failed stop retained old DMA mappings even at HS/FS.
> +		 * After a confirmed retry, all those requests can be given back.
> +		 */
> +		if (recovering || crg_udc->gadget.speed >= USB_SPEED_SUPER)
> +			crg_udc_ep_nuke(udc_ep_ptr, -ECONNRESET);
> +	} else if (do_halt == 0) {
> +		/* clearing ep halt state */
> +		/* reset sequence number */
> +		if (crg_udc_issue_command(crg_udc, CRG_CMD_RESET_SEQNUM, param0,
> +					  0))
> +			return -EIO;
> +
> +		if (!reset_seq_only) {
> +			/* Clear halt for a halted EP. */
> +			/* NOTE: we must CLEAR_HALT first,
> +			 * then SET_TR_DQPTR
> +			 */
> +			ret = crg_udc_issue_command(crg_udc,
> +						    CRG_CMD_CLEAR_HALT, param0, 0);
> +			if (ret)
> +				goto clear_halt_failed;
> +			ret = crg_udc_epcx_update_dqptr(udc_ep_ptr);
> +			if (ret)
> +				goto clear_halt_failed;
> +		}

[Severity: Medium]
Can clearing a halt at High or Full Speed rewind the hardware dequeue
pointer to the start of a partially completed multi-TRB request?

Chained TRBs before the final TRB of a TD have ioc = false, so hardware
advances TR_DQPT in p_epcx as each TRB completes while udc_ep_ptr->deq_pt
remains at first_trb.  At High or Full Speed, halting the endpoint retains
the in-flight request without updating udc_ep_ptr->deq_pt, and clearing the
halt then calls crg_udc_epcx_update_dqptr(), overwriting the hardware
dequeue pointer with udc_ep_ptr->deq_pt and re-transferring the already
completed TRBs.

[ ... ]

> +static int crg_udc_stop_error_ep(struct crg_gadget_dev *crg_udc, u8 dci)
> +{

[ ... ]

> +	/* The controller is no longer reading request buffers. */
> +	udc_ep_ptr->halt_failed = false;
> +	udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
> +	udc_ep_ptr->tran_ring_full = false;
> +	udc_ep_ptr->ep_state = isoc ? EP_STATE_STOPPED : EP_STATE_HALTED;
> +
> +	crg_udc_ep_nuke(udc_ep_ptr, -ECONNRESET);
> +
> +	return 0;

[Severity: High]
Does setting ep_state = EP_STATE_STOPPED here permanently wedge isochronous
endpoints after any transfer error event?

With ep_state == EP_STATE_STOPPED and dequeue_paused == false,
crg_udc_ep_can_queue() returns false and crg_udc_ep_queue() rejects all
subsequent requests with -ESHUTDOWN.  Unlike bulk and interrupt endpoints,
crg_udc_ep_halt() rejects isochronous endpoints with -EOPNOTSUPP, and
udc_ep_ptr->deq_pt is advanced to enq_pt in software without calling
crg_udc_epcx_update_dqptr(), leaving no way for the gadget to resume the
isochronous stream without disabling and re-enabling the endpoint.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261116-axiado-ax3000-usb-device-controller-v5-0-49271ed5c793@axiado.com?part=5

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: [PATCH v5 6/6] usb: gadget: udc: crg: Add core driver and build integration
  2026-11-16 19:06 ` [PATCH v5 6/6] usb: gadget: udc: crg: Add core driver and build integration Petar Stepanovic
@ 2026-11-16 21:15   ` sashiko-bot
  0 siblings, 0 replies; 13+ messages in thread
From: sashiko-bot @ 2026-11-16 21:15 UTC (permalink / raw)
  To: Petar Stepanovic; +Cc: linux-usb, devicetree, robh, conor+dt

Thank you for your contribution! Sashiko AI review found 10 potential
issues to consider:
- [High] usb: gadget: crg: unhandled level IRQ storm in
  crg_udc_common_irq()
- [High] usb: gadget: crg: broken endpoint teardown and UAF in
  crg_udc_suspend()
- [Medium] usb: gadget: crg: missing crg_udc_prepare_for_setup() in port
  reset workers
- [Medium] usb: gadget: crg: unscheduled external reset on
  crg_udc_recover() timeout
- [Medium] usb: gadget: crg: lost link restart in crg_udc_restart_link()
  during reset
- [Medium] usb: gadget: crg: lost disconnect on SYS_ERR when
  bus_reset_pending is set
- [Medium] usb: gadget: crg: uncancelled work items on probe error and
  remove
- [Medium] usb: gadget: crg: uncleared connected flag before disconnect
  on SYS_ERR
- [Medium] usb: gadget: crg: missing DMA quiescence wait in
  crg_udc_shutdown()
- [Medium] usb: gadget: udc: crg: lost reset notification in
  crg_udc_handle_port_status()
--

commit 808e1b308d74053a13eba452fa900f8bf2bf60a5
Author: Petar Stepanovic <pstepanovic@axiado.com>

usb: gadget: udc: crg: Add core driver and build integration

This patch adds the core platform driver, interrupt handling, reset and
recovery logic, and Kconfig/Makefile integration for the Corigine/Axiado
AX3000/AX3005 USB Device Controller.

> diff --git a/drivers/usb/gadget/udc/crg/crg-udc-core.c b/drivers/usb/gadget/udc/crg/crg-udc-core.c
> new file mode 100644
> index 0000000000000..e508dbd45339f
> --- /dev/null
> +++ b/drivers/usb/gadget/udc/crg/crg-udc-core.c
[ ... ]
> +	crg_udc->resetting = false;
> +
> +	ret = crg_gadget_irq_init(pdev, crg_udc);
> +	if (ret)
> +		goto err_free_resources;
> +
> +	ret = usb_add_gadget(&crg_udc->gadget);
> +	if (ret)
> +		goto err_free_resources;
> +
> +	return 0;
> +
> +err_free_resources:
> +	crg_udc_free_resources(crg_udc);
> +err_put_gadget:
> +	usb_put_gadget(&crg_udc->gadget);
> +	return ret;
> +}
> +
> +static void crg_udc_remove(struct platform_device *pdev)
> +{
> +	struct crg_gadget_dev *crg_udc = platform_get_drvdata(pdev);
> +	unsigned long flags;
> +	u32 tmp;
> +
> +	spin_lock_irqsave(&crg_udc->udc_lock, flags);
> +	crg_udc->shutting_down = true;
> +	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +
> +	/* The gadget core calls pullup(0) before the disconnect callback. */
> +	usb_del_gadget(&crg_udc->gadget);
> +	cancel_delayed_work_sync(&crg_udc->port_retry_work);
> +	cancel_work_sync(&crg_udc->recovery_work);
> +	cancel_work_sync(&crg_udc->udc_reset_work);
> +	crg_udc_free_resources(crg_udc);

[Severity: Medium]
Can an interrupt scheduled between crg_gadget_irq_init() and a failing
usb_add_gadget() leave port_retry_work, recovery_work, or udc_reset_work
queued after crg_udc is freed?

crg_udc_free_resources() synchronizes and frees the IRQ, then frees the DMA
rings, without cancelling those three work items before usb_put_gadget()
frees crg_udc.  In crg_udc_remove(), the work items are also cancelled
before crg_udc_free_resources() synchronizes and frees the IRQ rather than
after.

> +
> +	/* Do not access registers while the external reset is asserted. */
> +	if (!crg_udc->hw_reset_asserted && !crg_udc->dma_quiesce_failed) {
> +		tmp = readl(crg_udc->mmio_virt_base + CRG_UDC_MODE_REG) & ~0x1;
> +		writel(tmp, crg_udc->mmio_virt_base + CRG_UDC_MODE_REG);
> +	}
> +
> +	platform_set_drvdata(pdev, NULL);
> +	usb_put_gadget(&crg_udc->gadget);
> +}
> +
> +static void crg_udc_shutdown(struct platform_device *pdev)
> +{
> +	struct crg_gadget_dev *crg_udc = platform_get_drvdata(pdev);
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&crg_udc->udc_lock, flags);
> +	crg_udc->pullup_on = false;
> +	crg_udc->shutting_down = true;
> +	crg_udc->resetting = true;
> +	crg_udc->device_state = USB_STATE_ATTACHED;
> +	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +	if (crg_udc->irq >= 0)
> +		synchronize_irq(crg_udc->irq);
> +	cancel_delayed_work_sync(&crg_udc->port_retry_work);
> +	cancel_work_sync(&crg_udc->recovery_work);
> +	cancel_work_sync(&crg_udc->udc_reset_work);
> +
> +	/* Reset and gadget stop may own the registers across a lock drop. */
> +	spin_lock_irq(&crg_udc->udc_lock);
> +	wait_event_lock_irq(crg_udc->teardown_wait,
> +			    !crg_udc->controller_recovering, crg_udc->udc_lock);
> +	if (crg_udc->uccr && !crg_udc->hw_reset_asserted) {
> +		u32 val = readl(&crg_udc->uccr->control);
> +
> +		crg_udc_set_vbus_detect(crg_udc, false);
> +		val &= ~(CRG_U3DC_CTRL_INT_EN | CRG_U3DC_CTRL_RUN |
> +			 CRG_U3DC_CTRL_SYSERR_EN);
> +		writel(val, &crg_udc->uccr->control);
> +	}
> +	spin_unlock_irq(&crg_udc->udc_lock);
> +}

[Severity: Medium]
Does crg_udc_shutdown() return before bus-master DMA has actually stopped?

As noted in crg_udc_soft_reset(), clearing CRG_U3DC_CTRL_RUN does not halt
DMA or pending event-ring writes until CRG_U3DC_STATUS_DEV_CTRL_HALT
asserts.  Returning without waiting for halt or calling crg_udc_quiesce_hw()
leaves DMA active across shutdown or kexec.

> +
> +static int crg_udc_suspend(struct device *dev)
> +{
> +	struct crg_gadget_dev *crg_udc = dev_get_drvdata(dev);
> +	unsigned long flags;
> +	u32 val;
> +	int ret;
> +
> +	spin_lock_irqsave(&crg_udc->udc_lock, flags);
> +	if (crg_udc->controller_recovering || crg_udc->shutting_down ||
> +	    crg_udc->external_reset_pending ||
> +	    crg_udc->endpoint_recovery_pending ||
> +	    (crg_udc->hw_reset_asserted && !crg_udc->gadget_stopping)) {
> +		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +		return -EBUSY;
> +	}
> +	crg_udc->suspended = true;
> +	if (crg_udc->gadget_stopping) {
> +		if (crg_udc->hw_dma_armed) {
> +			crg_udc->suspended = false;
> +			ret = -EBUSY;
> +		} else {
> +			ret = 0;
> +		}
> +		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +		return ret;
> +	}
> +	crg_udc_set_pullup(crg_udc, false);
> +	val = readl(&crg_udc->uccr->control);
> +	val &= ~CRG_U3DC_CTRL_INT_EN;
> +	writel(val, &crg_udc->uccr->control);
> +	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +
> +	if (crg_udc->irq >= 0)
> +		synchronize_irq(crg_udc->irq);
> +	cancel_delayed_work_sync(&crg_udc->port_retry_work);
> +	cancel_work_sync(&crg_udc->recovery_work);
> +	cancel_work_sync(&crg_udc->udc_reset_work);
> +
> +	spin_lock_irqsave(&crg_udc->udc_lock, flags);
> +	if (crg_udc->gadget_stopping || crg_udc->shutting_down ||
> +	    crg_udc->controller_recovering || crg_udc->external_reset_pending ||
> +	    crg_udc->endpoint_recovery_pending) {
> +		ret = -EBUSY;
> +		goto out;
> +	}
> +	/*
> +	 * A reset may already have marked the link disconnected while its
> +	 * gadget notification was still pending. Deliver disconnect before
> +	 * resetting the software state, including that cancelled work.
> +	 */
> +	if (crg_udc->connected || crg_udc->bus_reset_pending)
> +		crg_udc_notify_disconnect(crg_udc, &flags);
> +	crg_udc->connected = false;
> +	crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
> +	if (crg_udc->external_reset_pending || crg_udc->endpoint_recovery_pending) {
> +		ret = -EBUSY;
> +		goto out;
> +	}
> +	ret = crg_udc_recover(crg_udc, &flags);
> +	if (ret)
> +		dev_err(crg_udc->dev, "suspend controller reset failed: %d\n", ret);
> +out:
> +	if (ret) {
> +		crg_udc->suspended = false;
> +		crg_udc_kick_reset_work(crg_udc);
> +	}
> +	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +	return ret;
> +}

[Severity: High]
Can crg_udc_suspend() cause a use-after-free on queued usb_requests when
crg_udc_notify_disconnect() disables endpoints?

crg_udc_suspend() sets crg_udc->suspended = true and clears CRG_U3DC_CTRL_RUN
via crg_udc_set_pullup(crg_udc, false) before calling
crg_udc_notify_disconnect() and crg_udc_recover().  When the gadget driver's
->disconnect() callback invokes usb_ep_disable() -> crg_udc_ep_disable() on
an active endpoint, polling uccr->ep_enable or uccr->ep_running can time out
with RUN cleared and call crg_udc_recover_endpoints() ->
crg_udc_recover_link().  Because crg_udc_recover_link() rejects
crg_udc->suspended with -EBUSY, usb_ep_disable() returns without stopping DMA
or draining udc_ep->queue, and the function driver frees its usb_requests
before crg_udc_recover() -> crg_udc_reset_queues() -> crg_udc_ep_nuke() gives
them back.

In addition, if usb_ep_dequeue() runs during that window,
crg_udc_abort_ep0_for_setup() or CRG_CMD_STOP_EP fails because suspended is
set and RUN is clear, setting endpoint_recovery_pending = true and causing
crg_udc_suspend() to abort with -EBUSY instead of running crg_udc_recover().
And if crg_udc_suspend() takes the early goto out before crg_udc_recover(),
CRG_U3DC_CTRL_RUN and CRG_U3DC_CTRL_INT_EN remain cleared while
crg_udc->suspended is reset to false.

[ ... ]
> +	if (tmp_status & CRG_U3DC_STATUS_SYS_ERR) {
> +		bool was_connected = crg_udc->connected;
> +		u32 generation = crg_udc->udc_event[0].generation;
> +
> +		/* A controller reset must not overlap composite's ->reset(). */
> +		if (crg_udc->reset_callback_active) {
> +			writel(CRG_U3DC_STATUS_SYS_ERR, &uccr->status);
> +			crg_udc->deferred_system_error = true;
> +			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +			return;
> +		}
> +
> +		dev_err(crg_udc->dev, "controller system error\n");
> +		writel(CRG_U3DC_STATUS_SYS_ERR, &uccr->status);
> +		/* Function drivers must see enabled endpoints on disconnect. */
> +		if (was_connected)
> +			crg_udc_notify_disconnect(crg_udc, &flags);

[Severity: Medium]
If CRG_U3DC_STATUS_SYS_ERR fires after a bus reset has already cleared
crg_udc->connected while crg_udc->bus_reset_pending is still true, does this
drop the gadget reset/disconnect notification?

Unlike crg_udc_suspend(), which checks crg_udc->connected ||
crg_udc->bus_reset_pending, this path only checks crg_udc->connected and
then calls crg_udc_recover_link(crg_udc, &flags, false), which clears
bus_reset_pending and marks all endpoints disabled without notifying the
gadget driver.

> +		/*
> +		 * The disconnect callback drops udc_lock: another CPU may be
> +		 * resetting now, or may have reset and rebuilt the rings
> +		 * already, in which case this error has been recovered.
> +		 */
> +		if (crg_udc->resetting || crg_udc->suspended ||
> +		    crg_udc->shutting_down ||
> +		    generation != crg_udc->udc_event[0].generation) {
> +			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +			return;
> +		}
> +		crg_udc_recover_link(crg_udc, &flags, false);

[Severity: Medium]
Should crg_udc->connected (and device_state) be cleared before calling
crg_udc_notify_disconnect() here?

Because crg_udc_notify_disconnect() drops udc_lock while crg_udc->connected
is still true and crg_udc->device_state is still USB_STATE_CONFIGURED, a
concurrent crg_udc_suspend() or port status handler can deliver a second
disconnect notification.  Also, if usb_ep_disable() inside ->disconnect()
times out on the faulted controller and calls crg_udc_recover_endpoints(),
notify_reset evaluates to true (via device_state == USB_STATE_CONFIGURED and
gadget_callbacks_active) and queues a bus reset on top of the disconnect.

[ ... ]
> +	if (crg_udc->device_state == USB_STATE_RECONNECTING &&
> +	    crg_udc->portsc_on_reconnecting == 1 &&
> +	    crg_udc_is_event_rings_empty(crg_udc)) {
> +		crg_udc->portsc_on_reconnecting = 0;
> +		crg_udc_handle_port_status(crg_udc, &flags);
> +		/* The port handler drops udc_lock. */
> +		if (crg_udc->resetting) {
> +			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +			return;
> +		}
> +	}

[Severity: Medium]
Can a second port event while crg_udc->bus_reset_pending is true clear
bus_reset_pending without notifying the gadget driver?

When an initial bus reset on a configured gadget runs crg_udc_reinit(crg_udc,
true), crg_udc->device_state becomes USB_STATE_RECONNECTING and
crg_udc->bus_reset_pending is set to true until udc_reset_work runs.  If a
second port reset, a port-reset timeout in crg_udc_port_retry_work(), or a
CCS drop with VBUS present arrives before udc_reset_work runs, those paths
only check crg_udc->device_state == USB_STATE_CONFIGURED (or pass false to
crg_udc_reinit()):

drivers/usb/gadget/udc/crg/crg-udc-gadget.c:crg_udc_handle_port_status() {
    ...
				bool was_configured = crg_udc->device_state ==
						       USB_STATE_CONFIGURED;
				bool notify_reset = was_configured &&
					crg_udc->async_callbacks &&
					crg_udc->gadget_driver;
				int reinit_ret;

				reinit_ret = crg_udc_reinit(crg_udc, notify_reset);
    ...
}

crg_udc_reinit(crg_udc, false) then overwrites crg_udc->bus_reset_pending
with false and clears usb_ep.enabled on all endpoints without running
usb_gadget_udc_reset().

> +
> +	if (crg_udc->device_state == USB_STATE_RECONNECTING &&
> +	    crg_udc->connected == 1) {
> +		ret = crg_udc_prepare_for_setup(crg_udc);
> +		if (!ret && crg_udc->pending_setup_valid &&
> +		    !crg_udc->resetting && crg_udc->connected &&
> +		    crg_udc->device_state == USB_STATE_DEFAULT) {
> +			struct usb_ctrlrequest setup = crg_udc->pending_setup;
> +			u8 tag = crg_udc->pending_setup_tag;
> +
> +			crg_udc->pending_setup_valid = false;
> +			crg_udc_dispatch_setup(crg_udc, &setup, tag, &flags);
> +		} else if (ret && ret != -EBUSY) {
> +			dev_err(crg_udc->dev,
> +				"failed to prepare EP0 after reset: %d\n", ret);
> +		}
> +	}

[Severity: Medium]
What prepares EP0 and transitions device_state out of USB_STATE_RECONNECTING
when crg_udc_handle_port_status() is called from crg_udc_port_retry_work() or
crg_udc_reset_work()?

When a port reset on a session with device_state >= USB_STATE_DEFAULT
completes inside crg_udc_handle_port_status(), crg_udc_reinit() sets
device_state = USB_STATE_RECONNECTING and crg_udc_handle_port_status() sets
connected = 1 without calling crg_udc_enable_setup().  Only
crg_gadget_handle_interrupt() runs the USB_STATE_RECONNECTING block above to
call crg_udc_prepare_for_setup() and dispatch pending_setup; the two worker
callers of crg_udc_handle_port_status() return with the controller stuck in
USB_STATE_RECONNECTING.

> +
> +	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +}
> +
> +/*
> + * Primary handler only checks whether the shared interrupt belongs to the
> + * controller. All controller work, including reset polling, runs in the
> + * threaded handler and therefore never busy-waits in hard IRQ context.
> + */
> +static irqreturn_t crg_udc_common_irq(int irq, void *dev)
> +{
> +	struct crg_gadget_dev *crg_udc = dev;
> +	struct crg_uccr __iomem *uccr = crg_udc->uccr;
> +	u32 status;
> +
> +	if (READ_ONCE(crg_udc->gadget_stopping) ||
> +	    READ_ONCE(crg_udc->shutting_down) ||
> +	    READ_ONCE(crg_udc->resetting))
> +		return IRQ_NONE;
> +
> +	status = readl(&uccr->status);
> +	if (!(status & (CRG_U3DC_STATUS_EINT | CRG_U3DC_STATUS_SYS_ERR)))
> +		return IRQ_NONE;
> +
> +	return IRQ_WAKE_THREAD;
> +}

[Severity: High]
Can returning IRQ_NONE here while the level-triggered interrupt is still
asserted cause a hard-IRQ storm that disables the shared IRQ line?

The IRQ is registered with IRQF_SHARED | IRQF_ONESHOT, so the line is only
masked when crg_udc_common_irq() returns IRQ_WAKE_THREAD.  Several paths set
resetting, shutting_down, or gadget_stopping and re-enable local interrupts
before CRG_U3DC_CTRL_INT_EN and CRG_U3DC_CTRL_SYSERR_EN are cleared in
uccr->control:

crg_udc_reset()
  crg_udc_reset_session()
    crg_udc->resetting = true;
  spin_unlock_irqrestore(&crg_udc->udc_lock, *flags)
  <hardirq>
    crg_udc_common_irq()
      if (READ_ONCE(crg_udc->resetting))
        return IRQ_NONE;  <-- level IRQ still asserted; storms

The same window exists in crg_udc_remove() during usb_del_gadget(), in
crg_udc_shutdown() before synchronize_irq(), and in crg_udc_stop_gadget()
when controller_recovering is true.  Similarly, when suspended or
controller_recovering is true while resetting is false (such as during
crg_udc_reinit()), crg_udc_common_irq() wakes crg_udc_irq_thread(), which
returns early in crg_gadget_handle_interrupt() without clearing
uccr->status and unmasks the still-asserted level interrupt in a tight loop.

[ ... ]
> +static void crg_udc_restart_link(struct crg_gadget_dev *crg_udc,
> +				 bool notify_reset)
> +{
> +	if (!crg_udc->async_callbacks || !crg_udc->gadget_driver ||
> +	    crg_udc->reset_callback_active)
> +		return;
> +
> +	if (notify_reset) {
> +		crg_udc->bus_reset_pending = true;
> +		crg_udc->restart_after_reset = true;
> +		crg_udc_kick_reset_work(crg_udc);
> +	} else if (!crg_udc->suspended && crg_udc->pullup_on) {
> +		crg_udc_set_pullup(crg_udc, true);
> +	}
> +}

[Severity: Medium]
If crg_udc_recover_endpoints() runs while crg_udc->reset_callback_active is
true, does this lose the post-reset pullup restart?

When usb_ep_disable() times out inside usb_gadget_udc_reset() and calls
crg_udc_recover_endpoints() -> crg_udc_recover_link(), crg_udc_reset() clears
crg_udc->restart_after_reset = false and resets the controller (clearing
CRG_U3DC_CTRL_RUN).  crg_udc_restart_link() then returns early because
reset_callback_active is true, leaving restart_after_reset false when
crg_udc_reset_work() finishes.

[ ... ]
> +int crg_udc_recover(struct crg_gadget_dev *crg_udc, unsigned long *flags)
> +{
> +	int ret;
> +
> +	if (crg_udc->controller_recovering || crg_udc->shutting_down ||
> +	    crg_udc->gadget_stopping || crg_udc->hw_reset_asserted)
> +		return -EBUSY;
> +
> +	crg_udc_recovery_begin(crg_udc);
> +	ret = crg_udc_reset(crg_udc, flags);
> +	if (ret) {
> +		crg_udc->controller_faulted = true;
> +		crg_udc->dma_quiesce_failed = crg_udc->hw_dma_armed;
> +		crg_udc_reset_queues(crg_udc, crg_udc->preserve_ep_enabled_on_reset);
> +	}
> +	if (!ret && !crg_udc->shutting_down && !crg_udc->gadget_stopping)
> +		ret = crg_udc_reset_data_struct(crg_udc);
> +	if (!ret && !crg_udc->shutting_down && !crg_udc->gadget_stopping)
> +		ret = crg_udc_init_ep0(crg_udc);
> +	if (!ret && (crg_udc->shutting_down || crg_udc->gadget_stopping))
> +		ret = -ESHUTDOWN;
> +
> +	crg_udc->controller_faulted = !!ret;
> +	crg_udc->resetting = !!ret;
> +	if (!ret)
> +		crg_udc->stopped_needs_reset = false;
> +	crg_udc_recovery_end(crg_udc);
> +	return ret;
> +}

[Severity: Medium]
When crg_udc_reset() times out in crg_udc_recover(), should this set
crg_udc->external_reset_pending when crg_udc->dma_quiesce_failed &&
crg_udc->rst, like crg_udc_recover_link() does?

crg_udc_reset_session() clears external_reset_pending, and on timeout
crg_udc_reset_queues() quarantines any active DMA bounce buffers into
crg_udc->dma_quarantine.  When crg_udc_recover() is called directly from
crg_udc_suspend() or crg_gadget_pullup(), external_reset_pending is never
set, so crg_udc_kick_reset_work() does not schedule recovery_work to assert
the external reset and release the quarantined buffers.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261116-axiado-ax3000-usb-device-controller-v5-0-49271ed5c793@axiado.com?part=6

^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2026-11-16 21:15 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-16 19:06 [PATCH v5 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
2026-11-16 19:06 ` [PATCH v5 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC Petar Stepanovic
2026-11-16 19:12   ` sashiko-bot
2026-11-16 19:06 ` [PATCH v5 2/6] usb: gadget: udc: crg: Add hardware definitions and structs Petar Stepanovic
2026-11-16 19:24   ` sashiko-bot
2026-11-16 19:06 ` [PATCH v5 3/6] usb: gadget: udc: crg: Add ring and transfer management Petar Stepanovic
2026-11-16 19:46   ` sashiko-bot
2026-11-16 19:06 ` [PATCH v5 4/6] usb: gadget: udc: crg: Add endpoint 0 handling Petar Stepanovic
2026-11-16 20:16   ` sashiko-bot
2026-11-16 19:06 ` [PATCH v5 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations Petar Stepanovic
2026-11-16 20:47   ` sashiko-bot
2026-11-16 19:06 ` [PATCH v5 6/6] usb: gadget: udc: crg: Add core driver and build integration Petar Stepanovic
2026-11-16 21:15   ` sashiko-bot

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox