[PATCH v3 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC)

[PATCH v3 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC)

[Petar Stepanovic]

Dear maintainers,

This series continues the work of adding support for the USB device
controller (UDC) integrated into the Axiado AX3000 and AX3005 SoCs.

The controller is based on the Corigine USB 3.1 IP core, which is
dual-role capable (device and host) at the hardware level. Which role
is active depends on which driver is bound to it: this series binds
it in device role, while host role would instead use a standard xHCI
host driver - the hardware itself does not fix the role. Endpoint 0
shares the same command-ring / event-ring / per-endpoint transfer-ring
TRB hardware as every other endpoint, so the driver tracks control
transfer stages (SETUP/DATA/STATUS) explicitly instead of relying on a
dedicated hardware FIFO. Other upstream TRB/ring based UDC drivers,
such as cdns2 (whose file layout this series' structure is now modeled
on), share this same shape for the same reason.

Since v2, the driver has been restructured to follow the file layout
and conventions of drivers/usb/gadget/udc/cdns2, split into
core/gadget/ep0/ring source files with ftrace tracepoint support, and
brought in line with cdns2's static/global function ratio. See the
detailed changelog below.

The devicetree binding also picks up one item left over from the v2
review: the description now notes the underlying core is dual-role
capable (device and host), used here in device role.

Tested on an Axiado AX3005 EVK with a mass-storage + composite HID
(keyboard/mouse) gadget, including repeated bind/unbind/rebind cycles
and host-driven bus resets.

Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
Changes in v3:

Devicetree binding:
- Note that the underlying Corigine core is dual-role capable (device
  and host), used here in device role.
- Drop the explicit maximum-speed property declaration. Krzysztof noted
  in the v2 review that compatible already implies SuperSpeed
capability,
  and testing confirms the property is not required.
- Reference the generic usb.yaml schema and replace additionalProperties
  with unevaluatedProperties to permit standard USB properties while
  rejecting unsupported properties.
- Validated the updated binding with dt_binding_check.

Driver structure (matching drivers/usb/gadget/udc/cdns2):
- Split crg_udc.c/crg_udc.h into crg-udc-core.c, crg-udc-gadget.c,
  crg-udc-ep0.c, crg-udc-ring.c and crg-udc.h, one concern per file.
- Add ftrace tracepoint support (crg-udc-trace.h/.c) covering pullup,
  doorbell, TRB queueing, event processing, control requests, and
  request alloc/free/giveback, plus a decode-helper header
  (crg-udc-debug.h) for the PORTSC/completion-code/TRB-type strings
  used by the trace TP_printk()s.
- Make internal-only functions static and normalize the remaining
  exported ones to a consistent crg_udc_ prefix.
- Reorganize #include lists so each file pulls in only the headers it
  actually uses, instead of funneling every kernel header the driver
  needs through the shared crg-udc.h.
- Match the SPDX/license comment style used elsewhere in the tree:
  a line comment for the SPDX tag, block comment for the description
  and copyright.

Bug fixes:
- Restore the VBUS-detect vendor-register write on udc_start(),
  udc_stop(), remove() and shutdown(). Without it the controller never
  considers itself powered and enumeration is unreliable.
- Restore the DATA_STAGE_XFER/DATA_STAGE_RECV cases dropped from the
  EP0 status-stage completion handler. Without them, any control read
  (e.g. GET_DESCRIPTOR) never arms its STATUS stage and the host times
  out waiting for it.
- Fix the transfer-ring TRB-deferral check to defer building a TRB
  only when the ring is actually full, not merely because another
  request is already queued on the same endpoint. The stricter check
  silently broke back-to-back completions on one endpoint, such as
  mass storage's CSW immediately following its data stage.
- Restore the RUN-gated completion check in the hardware command
  helper, so a command issued after the controller has already been
  stopped (e.g. during udc_stop()) does not spin waiting on a
  completion the hardware will never post.
- Call usb_gadget_udc_reset() on a bus reset so the composite/function
  drivers correctly re-run set_alt() and re-enable their endpoints
  after a reset, instead of composite silently treating a repeated
  SET_CONFIGURATION as a no-op. This is deferred to a workqueue rather
  than called directly from the port-status interrupt path, since the
  synchronous call re-enters the function drivers and would otherwise
  require dropping the driver lock in the middle of event processing.
- Clear the generic usb_ep.enabled flag and endpoint descriptors during
  internal endpoint teardown and controller reset, preventing stale
  endpoint state across sessions and allowing endpoints to be correctly
  re-enabled after a new bind or reset.

Additional fixes from Sashiko review:
- Use dma_addr_t for DMA offset calculations and validate DMA address
  translation failures before programming hardware registers.
- Reserve transfer-ring capacity and publish multi-TRB descriptors
  only after all TRBs have been initialized.
- Track remaining TRBs and buffer length for requests spanning
  multiple transfer descriptors.
- Advance the hardware dequeue pointer before completing canceled
  requests, and restore the producer cycle state when rewinding rings.
- Detach pending requests before completing them during endpoint
  teardown, and complete requests that fail during deferred queueing.
- Reduce the hardware reset timeout from 500 ms to 10 ms and release
  the spinlock while polling for completion.
- Protect pullup register updates with the controller spinlock.
- Update the event-ring dequeue pointer before yielding the CPU.
- Unregister the interrupt handler before freeing DMA resources.

- Link to v2: https://patch.msgid.link/20260817-axiado-ax3000-usb-device-controller-v2-0-0ef033dd0a68@axiado.com

Changes in v2:
- Drop the dual (GPL-2.0+ OR MIT) license; the driver and header are now
  GPL-2.0+ only.
- Register via module_platform_driver() and add MODULE_AUTHOR/MODULE_ALIAS
  instead of the hand-written module_init/module_exit pair.
- Acquire clock, reset and PHY through the driver model
  (devm_clk_get_optional, devm_reset_control_get_optional_exclusive,
  devm_phy_optional_get), with teardown wired up via
  devm_add_action_or_reset.
- Switch to devm_request_irq() and drop the manual free_irq() paths.
- Support multiple instances using an IDA (DEFINE_IDA/ida_alloc/ida_free)
  instead of a file-static device counter.
- Read match data with device_get_match_data() and move the former
  compile-time feature switches to DT properties (dis-u1u2-entry-quirk,
  dis-u1u2-initiation-quirk, reinit-ep0-on-bus-reset).
- Set the DMA mask via dma_set_mask_and_coherent() with a 32-bit fallback,
  and convert PM to SIMPLE_DEV_PM_OPS.
- Fix endianness handling for TRBs, events and endpoint contexts using
  cpu_to_le32()/le32_to_cpu(), and decode setup packets without casting.
- Locking: pass the caller's IRQ flags through the completion paths so the
  interrupt state is correctly restored across the unlock/lock window around
  gadget callbacks.
- Properly free all coherent DMA allocations and add
  dma_mapping_error() checks after dma_map_single()
- Fix DMA directions on the GET_STATUS/SET_SEL control transfers.
- Replace open-coded register busy-wait loops with
  readl_poll_timeout_atomic() so they time out instead of spinning forever,
  and return proper errnos (-EBUSY/-ETIMEDOUT/-EIO) instead of -1.
- Replace the custom SETF_VAR/GETF/MAKEF_VAR shift-and-mask macros with
  FIELD_PREP/FIELD_GET (include linux/bitfield.h), and use BIT()/GENMASK()
  and dma_wmb()/dma_rmb().
- Use DMA_MAPPING_ERROR instead of a private DMA_ADDR_INVALID.
- Remove the U-Boot-derived VBUS cable-detect routine, the VBUS-detect
  kthread and its wait queue, dead defines and struct fields, the unused
  interrupt-register-array macros, the mdelay()-based reset delays and the
  bulk of the dev_dbg() tracing.
- Implement the pullup(is_on == 0) path and route suspend/shutdown through
  pullup(0); mark crg_udc_ep_ops const and return IRQ_NONE from the shared
  IRQ handler when the interrupt is not ours.
- Use dev_err_probe() with structured goto-based unwinding in probe.
- Reword the Kconfig help text for USB_CRG_UDC to name the crg_udc module.
- Link to v1: https://lore.kernel.org/r/20260202-axiado-ax3000-usb-device-controller-v1-0-45ce0a8b014f@axiado.com

To: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
To: Prasad Bolisetty <pbolisetty@axiado.com>
To: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: Rob Herring <robh@kernel.org>
To: Krzysztof Kozlowski <krzk+dt@kernel.org>
To: Conor Dooley <conor+dt@kernel.org>
To: Tzu-Hao Wei <twei@axiado.com>
To: Karthikeyan Mitran <kmitran@axiado.com>
To: Philipp Zabel <p.zabel@pengutronix.de>
To: Petar Stepanovic <pstepanovic@axiado.com>
Cc: linux-usb@vger.kernel.org
Cc: devicetree@vger.kernel.org
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org

---
Petar Stepanovic (6):
      dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC
      usb: gadget: udc: crg: Add hardware definitions and structs
      usb: gadget: udc: crg: Add ring and transfer management
      usb: gadget: udc: crg: Add endpoint 0 handling
      usb: gadget: udc: crg: Add gadget and endpoint operations
      usb: gadget: udc: crg: Add core driver and build integration

 .../devicetree/bindings/usb/axiado,ax3000-udc.yaml |   69 +
 MAINTAINERS                                        |    9 +
 drivers/usb/gadget/udc/Kconfig                     |    2 +
 drivers/usb/gadget/udc/Makefile                    |    1 +
 drivers/usb/gadget/udc/crg/Kconfig                 |   16 +
 drivers/usb/gadget/udc/crg/Makefile                |    9 +
 drivers/usb/gadget/udc/crg/crg-udc-core.c          |  706 ++++++
 drivers/usb/gadget/udc/crg/crg-udc-debug.h         |  143 ++
 drivers/usb/gadget/udc/crg/crg-udc-ep0.c           | 1106 ++++++++++
 drivers/usb/gadget/udc/crg/crg-udc-gadget.c        | 2299 ++++++++++++++++++++
 drivers/usb/gadget/udc/crg/crg-udc-ring.c          |  664 ++++++
 drivers/usb/gadget/udc/crg/crg-udc-trace.c         |    9 +
 drivers/usb/gadget/udc/crg/crg-udc-trace.h         |  164 ++
 drivers/usb/gadget/udc/crg/crg-udc.h               |  722 ++++++
 14 files changed, 5919 insertions(+)
---
base-commit: c68641afacd9ea5707a489b30f204d5e89b324e0
change-id: 20260128-axiado-ax3000-usb-device-controller-156d3f1840c9

Best regards,
--  
Petar Stepanovic <pstepanovic@axiado.com>

[PATCH v3 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC

[Petar Stepanovic]

Add a Devicetree binding for the USB device controller integrated into
the Axiado AX3000 and AX3005 SoCs.

The controller is based on a Corigine USB 3.1 IP core that is
dual-role capable (device and host), used here in device role. It
provides SuperSpeed (5 Gb/s) or High-Speed (480 Mb/s) operation, and
supports control, bulk, interrupt, and isochronous transfer types
across multiple configurable endpoints.

Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
 .../devicetree/bindings/usb/axiado,ax3000-udc.yaml | 69 ++++++++++++++++++++++
 MAINTAINERS                                        |  8 +++
 2 files changed, 77 insertions(+)

diff --git a/Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml b/Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
new file mode 100644
index 000000000000..b9f7f4879ee0
--- /dev/null
+++ b/Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
@@ -0,0 +1,69 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/usb/axiado,ax3000-udc.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Axiado AX3000/AX3005 USB Device Controller
+
+maintainers:
+  - Petar Stepanovic <pstepanovic@axiado.com>
+  - Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
+  - Prasad Bolisetty <pbolisetty@axiado.com>
+
+description:
+  Axiado USB dual-role controller based on the Corigine USB 3.1 IP core.
+  Supports USB 3.1 SuperSpeed and USB 2.0 High-Speed device operation
+  with control, bulk, interrupt, and isochronous transfers.
+
+allOf:
+  - $ref: usb.yaml#
+
+properties:
+  compatible:
+    oneOf:
+      - const: axiado,ax3000-udc
+      - items:
+          - const: axiado,ax3005-udc
+          - const: axiado,ax3000-udc
+
+  reg:
+    maxItems: 1
+
+  interrupts:
+    maxItems: 1
+
+  clocks:
+    maxItems: 1
+
+  resets:
+    maxItems: 1
+
+  phys:
+    maxItems: 1
+
+required:
+  - compatible
+  - reg
+  - interrupts
+
+unevaluatedProperties: false
+
+examples:
+  - |
+    #include <dt-bindings/interrupt-controller/arm-gic.h>
+    #include <dt-bindings/interrupt-controller/irq.h>
+
+    soc {
+        #address-cells = <2>;
+        #size-cells = <2>;
+
+        usb@81200000 {
+            compatible = "axiado,ax3000-udc";
+            reg = <0x0 0x81200000 0x0 0x100000>;
+            interrupt-parent = <&gic500>;
+            interrupts = <GIC_SPI 21 IRQ_TYPE_LEVEL_HIGH>;
+            resets = <&reset 0>;
+            phys = <&usb_phy>;
+        };
+    };
diff --git a/MAINTAINERS b/MAINTAINERS
index 65e8a4b5c90b..58c5c8e00870 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4508,6 +4508,14 @@ F:	Documentation/devicetree/bindings/spi/axiado,ax3000-spi.yaml
 F:	drivers/spi/spi-axiado.c
 F:	drivers/spi/spi-axiado.h
 
+AXIADO USB UDC DRIVER
+M:	Petar Stepanovic <pstepanovic@axiado.com>
+M:	Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
+M:	Prasad Bolisetty <pbolisetty@axiado.com>
+L:	linux-usb@vger.kernel.org
+S:	Supported
+F:	Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
+
 AYANEO PLATFORM EC DRIVER
 M:	Antheas Kapenekakis <lkml@antheas.dev>
 L:	platform-driver-x86@vger.kernel.org

-- 
2.34.1

[PATCH v3 2/6] usb: gadget: udc: crg: Add hardware definitions and structs

[Petar Stepanovic]

This patch introduces the hardware definitions, register offsets, and
fundamental data structures for the Axiado AX3000/AX3005 USB Device
Controller driver.

Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
 drivers/usb/gadget/udc/crg/crg-udc-debug.h | 143 ++++++
 drivers/usb/gadget/udc/crg/crg-udc-trace.h | 164 +++++++
 drivers/usb/gadget/udc/crg/crg-udc.h       | 722 +++++++++++++++++++++++++++++
 3 files changed, 1029 insertions(+)

diff --git a/drivers/usb/gadget/udc/crg/crg-udc-debug.h b/drivers/usb/gadget/udc/crg/crg-udc-debug.h
new file mode 100644
index 000000000000..f73c60a79ade
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-debug.h
@@ -0,0 +1,143 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ * Debug/trace decode helpers.
+ *
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+
+#ifndef __CRG_UDC_DEBUG_H
+#define __CRG_UDC_DEBUG_H
+
+#include "crg-udc.h"
+
+static inline const char *crg_udc_trb_type_string(u32 type)
+{
+	switch (type) {
+	case TRB_TYPE_XFER_NORMAL:
+		return "Normal";
+	case TRB_TYPE_XFER_DATA_STAGE:
+		return "Data Stage";
+	case TRB_TYPE_XFER_STATUS_STAGE:
+		return "Status Stage";
+	case TRB_TYPE_XFER_DATA_ISOCH:
+		return "Isoch";
+	case TRB_TYPE_LINK:
+		return "Link";
+	case TRB_TYPE_NO_OP:
+		return "No-Op";
+	case TRB_TYPE_EVT_TRANSFER:
+		return "Transfer Event";
+	case TRB_TYPE_EVT_CMD_COMPLETION:
+		return "Command Completion Event";
+	case TRB_TYPE_EVT_PORT_STATUS_CHANGE:
+		return "Port Status Change Event";
+	case TRB_TYPE_EVT_MFINDEX_WRAP:
+		return "MFINDEX Wrap Event";
+	case TRB_TYPE_EVT_SETUP_PKT:
+		return "Setup Packet Event";
+	default:
+		return "Reserved";
+	}
+}
+
+static inline const char *crg_udc_cmpl_code_string(u32 code)
+{
+	switch (code) {
+	case CMPL_CODE_INVALID:
+		return "Invalid";
+	case CMPL_CODE_SUCCESS:
+		return "Success";
+	case CMPL_CODE_DATA_BUFFER_ERR:
+		return "Data Buffer Error";
+	case CMPL_CODE_BABBLE_DETECTED_ERR:
+		return "Babble Detected";
+	case CMPL_CODE_USB_TRANS_ERR:
+		return "USB Transaction Error";
+	case CMPL_CODE_TRB_ERR:
+		return "TRB Error";
+	case CMPL_CODE_TRB_STALL:
+		return "Stall";
+	case CMPL_CODE_INVALID_STREAM_TYPE_ERR:
+		return "Invalid Stream Type";
+	case CMPL_CODE_SHORT_PKT:
+		return "Short Packet";
+	case CMPL_CODE_RING_UNDERRUN:
+		return "Ring Underrun";
+	case CMPL_CODE_RING_OVERRUN:
+		return "Ring Overrun";
+	case CMPL_CODE_EVENT_RING_FULL_ERR:
+		return "Event Ring Full";
+	case CMPL_CODE_STOPPED:
+		return "Stopped";
+	case CMPL_CODE_STOPPED_LENGTH_INVALID:
+		return "Stopped - Length Invalid";
+	case CMPL_CODE_ISOCH_BUFFER_OVERRUN:
+		return "Isoch Buffer Overrun";
+	case CMPL_CODE_PROTOCOL_STALL:
+		return "Protocol Stall";
+	case CMPL_CODE_SETUP_TAG_MISMATCH:
+		return "Setup Tag Mismatch";
+	case CMPL_CODE_HALTED:
+		return "Halted";
+	case CMPL_CODE_HALTED_LENGTH_INVALID:
+		return "Halted - Length Invalid";
+	case CMPL_CODE_DISABLED:
+		return "Disabled";
+	case CMPL_CODE_DISABLED_LENGTH_INVALID:
+		return "Disabled - Length Invalid";
+	default:
+		return "Unknown";
+	}
+}
+
+static inline const char *crg_udc_decode_portsc(char *str, size_t size,
+						u32 portsc)
+{
+	scnprintf(str, size, "0x%08x, PLS: %lu, Speed: %lu, %s%s%s%s",
+		 portsc, FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, portsc),
+		 FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, portsc),
+		 (portsc & CRG_U3DC_PORTSC_CCS) ? "CCS " : "",
+		 (portsc & CRG_U3DC_PORTSC_PED) ? "PED " : "",
+		 (portsc & CRG_U3DC_PORTSC_PR) ? "PR " : "",
+		 (portsc & CRG_U3DC_PORTSC_PP) ? "PP " : "");
+
+	return str;
+}
+
+static inline const char *crg_udc_decode_trb(char *str, size_t size,
+					     struct crg_transfer_trb *trb)
+{
+	u32 dw2 = le32_to_cpu(trb->dw2);
+	u32 dw3 = le32_to_cpu(trb->dw3);
+	u32 type = FIELD_GET(TRB_TYPE_MASK, dw3);
+
+	scnprintf(str, size,
+		  "type: %s, len: %lu, cycle: %lu, chain: %lu, ioc: %lu",
+		  crg_udc_trb_type_string(type),
+		  FIELD_GET(TRB_TRANSFER_LEN_MASK, dw2),
+		  dw3 & TRB_CYCLE_BIT_MASK ? 1UL : 0UL,
+		  dw3 & TRB_CHAIN_BIT_MASK ? 1UL : 0UL,
+		  dw3 & TRB_INTR_ON_COMPLETION_MASK ? 1UL : 0UL);
+
+	return str;
+}
+
+static inline const char *crg_udc_decode_event_trb(char *str, size_t size,
+						   struct crg_event_trb *trb)
+{
+	u32 dw2 = le32_to_cpu(trb->dw2);
+	u32 dw3 = le32_to_cpu(trb->dw3);
+	u32 type = FIELD_GET(EVE_TRB_TYPE_MASK, dw3);
+
+	scnprintf(str, size, "type: %s, cmpl: %s, len: %lu, dci: %lu",
+		  crg_udc_trb_type_string(type),
+		  crg_udc_cmpl_code_string(
+			  FIELD_GET(EVE_TRB_COMPL_CODE_MASK, dw2)),
+		  FIELD_GET(EVE_TRB_TRAN_LEN_MASK, dw2),
+		  FIELD_GET(EVE_TRB_ENDPOINT_ID_MASK, dw3));
+
+	return str;
+}
+
+#endif /* __CRG_UDC_DEBUG_H */
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-trace.h b/drivers/usb/gadget/udc/crg/crg-udc-trace.h
new file mode 100644
index 000000000000..672a719a041b
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-trace.h
@@ -0,0 +1,164 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ * Trace support header file.
+ *
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM crg-udc
+
+#undef TRACE_SYSTEM_VAR
+#define TRACE_SYSTEM_VAR crg_udc
+
+#if !defined(__CRG_UDC_TRACE_H) || defined(TRACE_HEADER_MULTI_READ)
+#define __CRG_UDC_TRACE_H
+
+#include <linux/types.h>
+#include <linux/tracepoint.h>
+#include <linux/usb/ch9.h>
+#include "crg-udc.h"
+#include "crg-udc-debug.h"
+
+#define CRG_MSG_MAX 200
+
+DECLARE_EVENT_CLASS(crg_udc_log_enable_disable, TP_PROTO(int set), TP_ARGS(set),
+		    TP_STRUCT__entry(__field(int, set)),
+		    TP_fast_assign(__entry->set = set;),
+		    TP_printk("%s", __entry->set ? "enabled" : "disabled"));
+
+DEFINE_EVENT(crg_udc_log_enable_disable, crg_udc_pullup, TP_PROTO(int set),
+	     TP_ARGS(set));
+
+TRACE_EVENT(crg_udc_no_room_on_ring, TP_PROTO(char *msg), TP_ARGS(msg),
+	    TP_STRUCT__entry(__string(text, msg)),
+	    TP_fast_assign(__assign_str(text);),
+	    TP_printk("%s", __get_str(text)));
+
+TRACE_EVENT(crg_udc_ep_halt,
+	    TP_PROTO(struct crg_udc_ep *udc_ep, u8 halt, u8 wedge),
+	    TP_ARGS(udc_ep, halt, wedge),
+	    TP_STRUCT__entry(__string(name, udc_ep->usb_ep.name)
+				     __field(u8, halt) __field(u8, wedge)),
+	    TP_fast_assign(__assign_str(name); __entry->halt = halt;
+			   __entry->wedge = wedge;),
+	    TP_printk("%s: %s%s", __get_str(name),
+		      __entry->halt ? "halted" : "cleared",
+		      __entry->wedge ? " (wedged)" : ""));
+
+TRACE_EVENT(crg_udc_portsc, TP_PROTO(u32 portsc), TP_ARGS(portsc),
+	    TP_STRUCT__entry(__field(u32, portsc)),
+	    TP_fast_assign(__entry->portsc = portsc;),
+	    TP_printk("%s",
+		      crg_udc_decode_portsc(__get_buf(CRG_MSG_MAX), CRG_MSG_MAX,
+					    __entry->portsc)));
+
+TRACE_EVENT(crg_udc_doorbell, TP_PROTO(struct crg_gadget_dev *crg_udc, int dci),
+	    TP_ARGS(crg_udc, dci),
+	    TP_STRUCT__entry(__string(name, crg_udc->udc_ep[dci].usb_ep.name)
+				     __field(int, dci)),
+	    TP_fast_assign(__assign_str(name); __entry->dci = dci;),
+	    TP_printk("%s, dci %d", __get_str(name), __entry->dci));
+
+DECLARE_EVENT_CLASS(
+	crg_udc_log_ctrl, TP_PROTO(struct usb_ctrlrequest *ctrl), TP_ARGS(ctrl),
+	TP_STRUCT__entry(__field(u8, bRequestType) __field(u8, bRequest)
+				 __field(u16, wValue) __field(u16, wIndex)
+					 __field(u16, wLength)),
+	TP_fast_assign(__entry->bRequestType = ctrl->bRequestType;
+		       __entry->bRequest = ctrl->bRequest;
+		       __entry->wValue = le16_to_cpu(ctrl->wValue);
+		       __entry->wIndex = le16_to_cpu(ctrl->wIndex);
+		       __entry->wLength = le16_to_cpu(ctrl->wLength);),
+	TP_printk("%s", usb_decode_ctrl(__get_buf(CRG_MSG_MAX), CRG_MSG_MAX,
+					__entry->bRequestType,
+					__entry->bRequest, __entry->wValue,
+					__entry->wIndex, __entry->wLength)));
+
+DEFINE_EVENT(crg_udc_log_ctrl, crg_udc_ctrl_req,
+	     TP_PROTO(struct usb_ctrlrequest *ctrl), TP_ARGS(ctrl));
+
+DECLARE_EVENT_CLASS(
+	crg_udc_log_trb,
+	TP_PROTO(struct crg_udc_ep *udc_ep, struct crg_transfer_trb *trb),
+	TP_ARGS(udc_ep, trb),
+	TP_STRUCT__entry(__string(name, udc_ep->usb_ep.name)
+				 __field(struct crg_transfer_trb *, trb)
+					 __dynamic_array(char, decoded,
+							 CRG_MSG_MAX)),
+	TP_fast_assign(__assign_str(name); __entry->trb = trb;
+		       crg_udc_decode_trb(__get_dynamic_array(decoded),
+					  CRG_MSG_MAX, trb);),
+	TP_printk("%s: trb %p, %s", __get_str(name), __entry->trb,
+		  __get_str(decoded)));
+
+DEFINE_EVENT(crg_udc_log_trb, crg_udc_queue_trb,
+	     TP_PROTO(struct crg_udc_ep *udc_ep, struct crg_transfer_trb *trb),
+	     TP_ARGS(udc_ep, trb));
+
+TRACE_EVENT(
+	crg_udc_event_trb, TP_PROTO(struct crg_event_trb *trb), TP_ARGS(trb),
+	TP_STRUCT__entry(__field(struct crg_event_trb *, trb)
+				 __dynamic_array(char, decoded, CRG_MSG_MAX)),
+	TP_fast_assign(__entry->trb = trb;
+		       crg_udc_decode_event_trb(__get_dynamic_array(decoded),
+						CRG_MSG_MAX, trb);),
+	TP_printk("trb %p, %s", __entry->trb, __get_str(decoded)));
+
+DECLARE_EVENT_CLASS(
+	crg_udc_log_request,
+	TP_PROTO(struct crg_udc_ep *udc_ep, struct crg_udc_request *udc_req),
+	TP_ARGS(udc_ep, udc_req),
+	TP_STRUCT__entry(
+		__string(name, udc_ep->usb_ep.name) __field(
+			struct usb_request *, request) __field(void *, buf)
+			__field(unsigned int, actual) __field(unsigned int,
+							      length)
+				__field(int, status) __field(dma_addr_t, dma)
+					__field(int, zero) __field(int,
+								   short_not_ok)
+						__field(int, no_interrupt)),
+	TP_fast_assign(__assign_str(name); __entry->request = &udc_req->usb_req;
+		       __entry->buf = udc_req->usb_req.buf;
+		       __entry->actual = udc_req->usb_req.actual;
+		       __entry->length = udc_req->usb_req.length;
+		       __entry->status = udc_req->usb_req.status;
+		       __entry->dma = udc_req->usb_req.dma;
+		       __entry->zero = udc_req->usb_req.zero;
+		       __entry->short_not_ok = udc_req->usb_req.short_not_ok;
+		       __entry->no_interrupt = udc_req->usb_req.no_interrupt;),
+	TP_printk(
+		"%s: req: %p, buf: %p, length: %u/%u, status: %d, dma: %pad, %s%s%s",
+		__get_str(name), __entry->request, __entry->buf,
+		__entry->actual, __entry->length, __entry->status,
+		&__entry->dma, __entry->zero ? "Z" : "z",
+		__entry->short_not_ok ? "S" : "s",
+		__entry->no_interrupt ? "I" : "i"));
+
+DEFINE_EVENT(crg_udc_log_request, crg_udc_alloc_request,
+	     TP_PROTO(struct crg_udc_ep *udc_ep,
+		      struct crg_udc_request *udc_req),
+	     TP_ARGS(udc_ep, udc_req));
+
+DEFINE_EVENT(crg_udc_log_request, crg_udc_free_request,
+	     TP_PROTO(struct crg_udc_ep *udc_ep,
+		      struct crg_udc_request *udc_req),
+	     TP_ARGS(udc_ep, udc_req));
+
+DEFINE_EVENT(crg_udc_log_request, crg_udc_request_giveback,
+	     TP_PROTO(struct crg_udc_ep *udc_ep,
+		      struct crg_udc_request *udc_req),
+	     TP_ARGS(udc_ep, udc_req));
+
+#endif /* __CRG_UDC_TRACE_H */
+
+/* This part must be outside header guard. */
+
+#undef TRACE_INCLUDE_PATH
+#define TRACE_INCLUDE_PATH .
+
+#undef TRACE_INCLUDE_FILE
+#define TRACE_INCLUDE_FILE crg-udc-trace
+
+#include <trace/define_trace.h>
diff --git a/drivers/usb/gadget/udc/crg/crg-udc.h b/drivers/usb/gadget/udc/crg/crg-udc.h
new file mode 100644
index 000000000000..f3b545db5599
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc.h
@@ -0,0 +1,722 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+
+#ifndef CRG_UDC_H
+#define CRG_UDC_H
+
+#include <linux/bitfield.h>
+#include <linux/bits.h>
+#include <linux/dma-mapping.h>
+#include <linux/io.h>
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/workqueue.h>
+
+#include <linux/usb/ch9.h>
+#include <linux/usb/gadget.h>
+
+#define CRG_UCCR_OFFSET 0x2400
+#define CRG_UICR_OFFSET 0x2500
+#define CRG_UICR_STRIDE 0x20
+
+/* Corigine USB 3.1 device core register layout. */
+struct crg_uccr {
+	u32 capability;
+	u32 resv0[3];
+
+	u32 config0;
+	u32 config1;
+	u32 resv1[2];
+
+	u32 control;
+	u32 status;
+	u32 dcbaplo;
+	u32 dcbaphi;
+	u32 portsc;
+	u32 u3portpmsc;
+	u32 u2portpmsc;
+	u32 u3portli;
+
+	u32 doorbell;
+	u32 mfindex;
+	u32 ptmctrl;
+	u32 resv3[5];
+
+	u32 ep_enable;
+	u32 ep_running;
+	u32 resv4[2];
+
+	u32 cmd_param0;
+	u32 cmd_param1;
+	u32 cmd_control;
+	u32 resv5[1];
+
+	u32 odb_capability;
+	u32 resv6[3];
+	u32 odb_config[8];
+
+	u32 debug0;
+};
+
+struct crg_uicr {
+	u32 iman;
+	u32 imod;
+	u32 erstsz;
+	u32 resv0;
+
+	u32 erstbalo;
+	u32 erstbahi;
+	u32 erdplo;
+	u32 erdphi;
+};
+
+/* config0/config1 registers */
+#define CRG_U3DC_CFG0_MAXSPEED_MASK GENMASK(3, 0)
+#define CRG_U3DC_CFG0_MAXSPEED_FS 1
+#define CRG_U3DC_CFG0_MAXSPEED_HS 3
+#define CRG_U3DC_CFG0_MAXSPEED_SS 4
+#define CRG_U3DC_CFG0_MAXSPEED_SSP 5
+
+#define CRG_U3DC_CFG1_CSC_EVENT_EN BIT(0)
+#define CRG_U3DC_CFG1_PEC_EVENT_EN BIT(1)
+#define CRG_U3DC_CFG1_PPC_EVENT_EN BIT(3)
+#define CRG_U3DC_CFG1_PRC_EVENT_EN BIT(4)
+#define CRG_U3DC_CFG1_PLC_EVENT_EN BIT(5)
+#define CRG_U3DC_CFG1_CEC_EVENT_EN BIT(6)
+
+#define CRG_U3DC_CFG1_U3_ENTRY_EN BIT(8)
+#define CRG_U3DC_CFG1_L1_ENTRY_EN BIT(9)
+#define CRG_U3DC_CFG1_U3_RESUME_EN BIT(10)
+#define CRG_U3DC_CFG1_L1_RESUME_EN BIT(11)
+#define CRG_U3DC_CFG1_INACTIVE_PLC_EN BIT(12)
+#define CRG_U3DC_CFG1_U3_RESUME_NORESP_PLC_EN BIT(13)
+#define CRG_U3DC_CFG1_U2_RESUME_NORESP_PLC_EN BIT(14)
+
+#define CRG_U3DC_CFG1_SETUP_EVENT_EN BIT(16)
+
+/* control register */
+#define CRG_U3DC_CTRL_RUN BIT(0)
+#define CRG_U3DC_CTRL_STOP 0
+#define CRG_U3DC_CTRL_SWRST BIT(1)
+#define CRG_U3DC_CTRL_INT_EN BIT(2)
+#define CRG_U3DC_CTRL_SYSERR_EN BIT(3)
+#define CRG_U3DC_CTRL_EWE BIT(10)
+#define CRG_U3DC_CTRL_KP_CNCT BIT(11)
+
+/* status register */
+#define CRG_U3DC_STATUS_DEV_CTRL_HALT BIT(0)
+#define CRG_U3DC_STATUS_SYS_ERR BIT(2)
+#define CRG_U3DC_STATUS_EINT BIT(3)
+
+/* portsc register */
+#define CRG_U3DC_PORTSC_CCS BIT(0)
+#define CRG_U3DC_PORTSC_PED BIT(1)
+#define CRG_U3DC_PORTSC_PP BIT(3)
+#define CRG_U3DC_PORTSC_PR BIT(4)
+#define CRG_U3DC_PORTSC_PLS_MASK GENMASK(8, 5)
+#define CRG_U3DC_PORTSC_PLS_U3 3
+
+#define CRG_U3DC_PORTSC_SPEED_MASK GENMASK(13, 10)
+#define CRG_U3DC_PORTSC_SPEED_FS 0x1
+#define CRG_U3DC_PORTSC_SPEED_LS 0x2
+#define CRG_U3DC_PORTSC_SPEED_HS 0x3
+#define CRG_U3DC_PORTSC_SPEED_SS 0x4
+#define CRG_U3DC_PORTSC_SPEED_SSP 0x5
+
+#define CRG_U3DC_PORTSC_LWS BIT(16)
+#define CRG_U3DC_PORTSC_CSC BIT(17)
+#define CRG_U3DC_PORTSC_PEC BIT(18)
+#define CRG_U3DC_PORTSC_PPC BIT(20)
+#define CRG_U3DC_PORTSC_PRC BIT(21)
+#define CRG_U3DC_PORTSC_PLC BIT(22)
+#define CRG_U3DC_PORTSC_CEC BIT(23)
+#define CRG_U3DC_PORTSC_WCE BIT(25)
+#define CRG_U3DC_PORTSC_WDE BIT(26)
+#define CRG_U3DC_PORTSC_WPR BIT(31)
+
+#define CRG_U3DC_PORTSC_W1C_MASK                                           \
+	(CRG_U3DC_PORTSC_CSC | CRG_U3DC_PORTSC_PEC | CRG_U3DC_PORTSC_PPC | \
+	 CRG_U3DC_PORTSC_PRC | CRG_U3DC_PORTSC_PLC | CRG_U3DC_PORTSC_CEC)
+
+/* mfindex register */
+#define CRG_U3DC_MFINDEX_CURRENT_MASK GENMASK(17, 4)
+#define CRG_U3DC_MFINDEX_FRAME_MASK GENMASK(17, 7)
+
+/* ptmctrl register */
+#define CRG_U3DC_PTMCTRL_ISOCH_DELAY_MASK GENMASK(13, 0)
+
+/* u3portpmsc */
+#define CRG_U3DC_U3PORTPM_U1TMOUT_MASK GENMASK(7, 0)
+#define CRG_U3DC_U3PORTPM_U2TMOUT_MASK GENMASK(15, 8)
+#define CRG_U3DC_U3PORTPM_FLA BIT(16)
+#define CRG_U3DC_U3PORTPM_U1IEN BIT(20)
+#define CRG_U3DC_U3PORTPM_U2IEN BIT(21)
+#define CRG_U3DC_U3PORTPM_U1AEN BIT(22)
+#define CRG_U3DC_U3PORTPM_U2AEN BIT(23)
+
+#define CRG_U3DC_U3PORTPM_U1U2TMOUT_MASK GENMASK(31, 24)
+
+/* u2portpmsc */
+#define CRG_U3DC_U2PORTPM_RJ_TH_MASK GENMASK(3, 0)
+#define CRG_U3DC_U2PORTPM_DS_TH_MASK GENMASK(7, 4)
+#define CRG_U3DC_U2PORTPM_LPM_EN BIT(8)
+#define CRG_U3DC_U2PORTPM_RJ_TH_EN BIT(9)
+#define CRG_U3DC_U2PORTPM_DS_EN BIT(10)
+#define CRG_U3DC_U2PORTPM_SLP_EN BIT(11)
+#define CRG_U3DC_U2PORTPM_LPM_FACK BIT(12)
+#define CRG_U3DC_U2PORTPM_L1_AEX BIT(13)
+#define CRG_U3DC_U2PORTPM_H_B_MASK GENMASK(19, 16)
+#define CRG_U3DC_U2PORTPM_RWE BIT(20)
+#define CRG_U3DC_U2PORTPM_TM_MASK GENMASK(31, 28)
+
+/* doorbell register */
+#define CRG_U3DC_DB_TARGET_MASK GENMASK(4, 0)
+
+/* odb registers */
+#define CRG_U3DC_ODBCFG_2N_OFFSET_MASK GENMASK(9, 0)
+#define CRG_U3DC_ODBCFG_2N_SIZE_MASK GENMASK(12, 10)
+#define CRG_U3DC_ODBCFG_2N1_OFFSET_MASK GENMASK(25, 16)
+#define CRG_U3DC_ODBCFG_2N1_SIZE_MASK GENMASK(28, 26)
+
+/* command control register */
+#define CRG_U3DC_CMD_CTRL_ACTIVE BIT(0)
+#define CRG_U3DC_CMD_CTRL_IOC_EN BIT(1)
+#define CRG_U3DC_CMD_CTRL_TYPE_MASK GENMASK(7, 4)
+#define CRG_U3DC_CMD_CTRL_STATUS_MASK GENMASK(19, 16)
+
+/* interrupt register / IMAN bits */
+#define CRG_U3DC_IMAN_INT_PEND BIT(0)
+#define CRG_U3DC_IMAN_INT_EN BIT(1)
+
+/* erdp bits */
+#define CRG_U3DC_ERDPLO_EHB BIT(3)
+
+/* command parameters: init EP0 */
+#define CRG_CMD0_0_DQPTRLO_MASK GENMASK(31, 4)
+#define CRG_CMD0_0_DCS_MASK BIT(0)
+
+/* command parameters: update EP0 */
+#define CRG_CMD1_0_MPS_MASK GENMASK(31, 16)
+
+/* command parameters: set address */
+#define CRG_CMD2_0_DEV_ADDR_MASK GENMASK(7, 0)
+
+#define CRG_UDC_MODE_REG 0x20fc
+#define CRG_UDC_ROLE_DEVICE 0x1
+
+#define CRG_UDC_VENDOR_REG 0x210c
+#define CRG_UDC_VBUS_DETECT BIT(26)
+
+/*
+ * Device-visible data structures. The controller reads and writes these
+ * over DMA, so the layouts and field encodings below are fixed by the
+ * hardware in the same way as the register definitions above.
+ */
+
+/* Transfer and event TRB types */
+enum crg_trb_type {
+	TRB_TYPE_RSVD = 0,
+	TRB_TYPE_XFER_NORMAL,
+	TRB_TYPE_RSVD2,
+	TRB_TYPE_XFER_DATA_STAGE,
+	TRB_TYPE_XFER_STATUS_STAGE,
+	TRB_TYPE_XFER_DATA_ISOCH,
+	TRB_TYPE_LINK,
+	TRB_TYPE_RSVD7,
+	TRB_TYPE_NO_OP,
+
+	TRB_TYPE_EVT_TRANSFER = 32,
+	TRB_TYPE_EVT_CMD_COMPLETION = 33,
+	TRB_TYPE_EVT_PORT_STATUS_CHANGE = 34,
+	TRB_TYPE_EVT_MFINDEX_WRAP = 39,
+	TRB_TYPE_EVT_SETUP_PKT = 40,
+};
+
+/* Event TRB completion codes */
+enum crg_trb_cmpl_code {
+	CMPL_CODE_INVALID = 0,
+	CMPL_CODE_SUCCESS,
+	CMPL_CODE_DATA_BUFFER_ERR,
+	CMPL_CODE_BABBLE_DETECTED_ERR,
+	CMPL_CODE_USB_TRANS_ERR,
+	CMPL_CODE_TRB_ERR,
+	CMPL_CODE_TRB_STALL,
+	CMPL_CODE_INVALID_STREAM_TYPE_ERR = 10,
+	CMPL_CODE_SHORT_PKT = 13,
+	CMPL_CODE_RING_UNDERRUN,
+	CMPL_CODE_RING_OVERRUN,
+	CMPL_CODE_EVENT_RING_FULL_ERR = 21,
+	CMPL_CODE_STOPPED = 26,
+	CMPL_CODE_STOPPED_LENGTH_INVALID = 27,
+	CMPL_CODE_ISOCH_BUFFER_OVERRUN = 31,
+
+	CMPL_CODE_PROTOCOL_STALL = 192,
+	CMPL_CODE_SETUP_TAG_MISMATCH = 193,
+	CMPL_CODE_HALTED = 194,
+	CMPL_CODE_HALTED_LENGTH_INVALID = 195,
+	CMPL_CODE_DISABLED = 196,
+	CMPL_CODE_DISABLED_LENGTH_INVALID = 197,
+};
+
+/* Endpoint type encoding in the endpoint context */
+enum crg_ep_type {
+	EP_TYPE_INVALID = 0,
+	EP_TYPE_ISOCH_OUTBOUND,
+	EP_TYPE_BULK_OUTBOUND,
+	EP_TYPE_INTR_OUTBOUND,
+	EP_TYPE_INVALID2,
+	EP_TYPE_ISOCH_INBOUND,
+	EP_TYPE_BULK_INBOUND,
+	EP_TYPE_INTR_INBOUND,
+};
+
+/* crg_transfer_trb dw2 fields */
+#define TRB_TRANSFER_LEN_MASK GENMASK(16, 0)
+#define TRB_TD_SIZE_MASK GENMASK(21, 17)
+#define TRB_INTR_TARGET_MASK GENMASK(31, 22)
+
+/* crg_transfer_trb dw3 fields */
+#define TRB_CYCLE_BIT_MASK BIT(0)
+#define TRB_LINK_TOGGLE_CYCLE_MASK BIT(1)
+#define TRB_INTR_ON_SHORT_PKT_MASK BIT(2)
+#define TRB_NO_SNOOP_MASK BIT(3)
+#define TRB_CHAIN_BIT_MASK BIT(4)
+#define TRB_INTR_ON_COMPLETION_MASK BIT(5)
+#define TRB_APPEND_ZLP_MASK BIT(7)
+#define TRB_BLOCK_EVENT_INT_MASK BIT(9)
+#define TRB_TYPE_MASK GENMASK(15, 10)
+#define DATA_STAGE_TRB_DIR_MASK BIT(16)
+#define TRB_SETUP_TAG_MASK GENMASK(18, 17)
+#define STATUS_STAGE_TRB_STALL_MASK BIT(19)
+#define STATUS_STAGE_TRB_SET_ADDR_MASK BIT(20)
+#define ISOC_TRB_FRAME_ID_MASK GENMASK(30, 20)
+#define ISOC_TRB_SIA_MASK BIT(31)
+
+struct crg_transfer_trb {
+	__le32 dw0;
+	__le32 dw1;
+	__le32 dw2;
+	__le32 dw3;
+};
+
+/* crg_event_trb dw2 fields */
+#define EVE_TRB_TRAN_LEN_MASK GENMASK(16, 0)
+#define EVE_TRB_COMPL_CODE_MASK GENMASK(31, 24)
+
+/* crg_event_trb dw3 fields */
+#define EVE_TRB_CYCLE_BIT_MASK BIT(0)
+#define EVE_TRB_TYPE_MASK GENMASK(15, 10)
+#define EVE_TRB_ENDPOINT_ID_MASK GENMASK(20, 16)
+#define EVE_TRB_SETUP_TAG_MASK GENMASK(22, 21)
+
+struct crg_event_trb {
+	__le32 dw0;
+	__le32 dw1;
+	__le32 dw2;
+	__le32 dw3;
+};
+
+/* crg_ep_cx dw0 fields */
+#define EP_CX_LOGICAL_EP_NUM_MASK GENMASK(6, 3)
+#define EP_CX_INTERVAL_MASK GENMASK(23, 16)
+
+/* crg_ep_cx dw1 fields */
+#define EP_CX_EP_TYPE_MASK GENMASK(5, 3)
+#define EP_CX_MAX_BURST_SIZE_MASK GENMASK(15, 8)
+#define EP_CX_MAX_PACKET_SIZE_MASK GENMASK(31, 16)
+
+/* crg_ep_cx dw2 fields */
+#define EP_CX_DEQ_CYC_STATE_MASK BIT(0)
+#define EP_CX_TR_DQPT_LO_MASK GENMASK(31, 4)
+
+struct crg_ep_cx {
+	__le32 dw0;
+	__le32 dw1;
+	__le32 dw2;
+	__le32 dw3;
+};
+
+struct crg_erst {
+	/* 64-bit event ring segment address */
+	__le32 seg_addr_lo;
+	__le32 seg_addr_hi;
+	__le32 seg_size;
+	/* Set to zero */
+	__le32 rsvd;
+};
+
+/* command type */
+enum crg_cmd_type {
+	CRG_CMD_INIT_EP0 = 0,
+	CRG_CMD_UPDATE_EP0_CFG = 1,
+	CRG_CMD_SET_ADDR = 2,
+	CRG_CMD_SEND_DEV_NOTIFICATION = 3,
+	CRG_CMD_CONFIG_EP = 4,
+	CRG_CMD_SET_HALT = 5,
+	CRG_CMD_CLEAR_HALT = 6,
+	CRG_CMD_RESET_SEQNUM = 7,
+	CRG_CMD_STOP_EP = 8,
+	CRG_CMD_SET_TR_DQPTR = 9,
+	CRG_CMD_FORCE_FLOW_CONTROL = 10,
+	CRG_CMD_REQ_LDM_EXCHANGE = 11,
+};
+
+#define CRG_MAX_PACKET_SIZE 1024
+
+#define CRG_ERST_SIZE 1
+#define CRG_EVENT_RING_SIZE 256
+#define CRG_NUM_EP_CX 32
+
+/*
+ * Cap on events drained per udc_lock acquisition in crg_udc_process_event_ring().
+ * Command polling and crg_udc_reset()'s hardware-reset poll both run from
+ * crg_udc_handle_event() with udc_lock held, so without a cap a busy event
+ * ring could keep the lock, and local IRQs, held for as long as events
+ * keep arriving.
+ */
+#define CRG_EVENT_RING_BATCH 32
+
+#define CRG_UDC_TRB_MAX_BUFFER_SIZE 65536
+#define CRG_UDC_CONTROL_EP_TD_RING_SIZE 16
+#define CRG_UDC_BULK_EP_TD_RING_SIZE 1024
+#define CRG_UDC_ISOC_EP_TD_RING_SIZE 32
+#define CRG_UDC_INT_EP_TD_RING_SIZE 8
+
+#define CRG_UDC_U1_TIMEOUT_VAL 0x70
+#define CRG_UDC_U2_TIMEOUT_VAL 0x70
+
+enum crg_ep_state {
+	EP_STATE_DISABLED = 0,
+	EP_STATE_RUNNING = 1,
+	EP_STATE_HALTED = 2,
+	EP_STATE_STOPPED = 3
+};
+
+/* control-transfer setup state machine */
+enum crg_setup_status {
+	WAIT_FOR_SETUP = 0,
+	SETUP_PKT_PROCESS_IN_PROGRESS,
+	DATA_STAGE_XFER,
+	DATA_STAGE_RECV,
+	STATUS_STAGE_XFER,
+	STATUS_STAGE_RECV,
+};
+
+/* interrupt-on-completion moderation rate (in TRBs) per endpoint type */
+enum {
+	CRG_UDC_BULK_EP_INTERRUPT_RATE = 5,
+	CRG_UDC_ISOC_EP_INTERRUPT_RATE = 1,
+};
+
+struct crg_buffer_info {
+	void *cpu_addr;
+	dma_addr_t dma_addr;
+	u32 len;
+};
+
+struct crg_sel_value {
+	u16 u2_pel_value;
+	u16 u2_sel_value;
+	u8 u1_pel_value;
+	u8 u1_sel_value;
+};
+
+struct crg_udc_request {
+	struct usb_request usb_req;
+	struct list_head queue;
+	bool mapped;
+	/*
+	 * Direction used for the DMA mapping call that produced the mapping
+	 * tracked above. ep0 is bidirectional and its
+	 * descriptor address is always 0, so usb_endpoint_dir_in() on the
+	 * endpoint cannot tell an unmap apart from a control IN or OUT
+	 * transfer; the direction actually used to map is recorded here
+	 * instead and reused as-is at unmap time.
+	 */
+	enum dma_data_direction map_dir;
+	struct scatterlist *sg;
+	unsigned int sg_index;
+	u32 sg_offset;
+	u64 buff_len_left;
+	u32 trbs_needed;
+	struct crg_transfer_trb *first_trb;
+	struct crg_transfer_trb *last_trb;
+	bool all_trbs_queued;
+};
+
+struct crg_udc_ep {
+	struct usb_ep usb_ep;
+
+	struct crg_buffer_info tran_ring_info;
+	struct crg_transfer_trb *first_trb;
+	struct crg_transfer_trb *last_trb;
+
+	struct crg_transfer_trb *enq_pt;
+	struct crg_transfer_trb *deq_pt;
+	u8 pcs;
+
+	char name[10];
+	u8 dci;
+	struct list_head queue;
+	const struct usb_endpoint_descriptor *desc;
+	const struct usb_ss_ep_comp_descriptor *comp_desc;
+	bool tran_ring_full;
+	struct crg_gadget_dev *crg_udc;
+
+	enum crg_ep_state ep_state;
+
+	bool wedge;
+};
+
+#define CRG_RING_NUM 1
+
+struct crg_udc_event {
+	struct crg_buffer_info erst;
+	struct crg_erst *p_erst;
+	struct crg_buffer_info event_ring;
+	struct crg_event_trb *evt_dq_pt;
+	u8 ccs;
+	struct crg_event_trb *evt_seg0_last_trb;
+};
+
+enum crg_ep0_state {
+	EP0_STATE_IDLE = 0,
+	EP0_STATE_ABORTING,
+	EP0_STATE_ERROR,
+};
+
+struct crg_gadget_dev {
+	void __iomem *mmio_virt_base;
+	struct crg_uccr __iomem *uccr;
+	struct crg_uicr __iomem *uicr[CRG_RING_NUM];
+
+	/* udc_lock device lock */
+	spinlock_t udc_lock;
+
+	struct device *dev;
+	struct clk *clk;
+	struct phy *phy;
+	struct reset_control *rst;
+
+	struct usb_gadget gadget;
+	struct usb_gadget_driver *gadget_driver;
+	/* Speed selected for the currently bound gadget driver. */
+	enum usb_device_speed max_speed;
+
+	int irq;
+
+	struct usb_endpoint_descriptor ep0_desc;
+
+	struct crg_udc_ep udc_ep[CRG_NUM_EP_CX];
+	struct crg_buffer_info ep_cx;
+	struct crg_ep_cx *p_epcx;
+
+	struct crg_udc_event udc_event[CRG_RING_NUM];
+
+	struct crg_udc_request *status_req;
+	struct crg_sel_value sel_value;
+	void (*setup_fn_call_back)(struct crg_gadget_dev *a);
+
+	enum crg_setup_status setup_status;
+	/* EP0 rejects new requests while the previous control transfer is
+	 * being aborted and its completion callbacks are being given back.
+	 */
+	enum crg_ep0_state ep0_state;
+
+	u8 device_state;
+	u16 dev_addr;
+	u8 setup_tag;
+	u8 set_tm;
+
+	u32 num_enabled_eps;
+
+	bool connected;
+	bool pullup_on;
+	bool async_callbacks;
+	/*
+	 * Set while crg_udc_reset() has udc_lock dropped and is polling for
+	 * SWRST to clear. Checked by crg_udc_ep_queue() so a request queued
+	 * on another endpoint during that window is rejected instead of
+	 * being built into a ring and doorbell-rung while the controller is
+	 * still mid-reset.
+	 */
+	bool resetting;
+
+	bool u2_rwe;
+	bool feature_u1_enable;
+	bool feature_u2_enable;
+
+	bool portsc_on_reconnecting;
+	bool u12_forbidden;
+	bool u12_initiate_forbidden;
+	bool reinit_ep0_on_bus_reset;
+
+	/*
+	 * usb_gadget_udc_reset() re-enters the gadget/function drivers
+	 * synchronously. Calling it directly from the port-status IRQ path
+	 * would require dropping udc_lock mid-event-processing, opening a
+	 * window for a second port-status event to run crg_udc_reinit()
+	 * again while the first event's SET_CONFIGURATION/set_alt() work is
+	 * still landing on another thread, tearing the just-enabled
+	 * endpoints back down. Defer the notification to this work item so
+	 * the IRQ-thread's event processing never has to drop the lock.
+	 */
+	struct work_struct udc_reset_work;
+};
+
+/* Shared prototypes */
+void crg_udc_build_ep0_status(struct crg_udc_ep *udc_ep_ptr, bool default_value,
+			      u32 status, struct crg_udc_request *udc_req_ptr,
+			      u8 set_addr, u8 stall);
+/*
+ * Abort the current EP0 control transfer when a new SETUP packet arrives.
+ *
+ * A SETUP packet is allowed to abort any in-progress control transfer. Do
+ * not defer the new SETUP until the old transfer completes: doing so leaves
+ * the old request on the EP0 software queue and subsequent gadget requests
+ * are rejected as "queue not empty".
+ *
+ * The controller's EP0 ring is reinitialized at the ring start, making all
+ * TRBs belonging to the aborted control transfer stale. The new SETUP is
+ * then handled immediately with a fresh setup tag.
+ */
+int crg_udc_abort_ep0_for_setup(struct crg_gadget_dev *crg_udc);
+int crg_udc_enable_setup(struct crg_gadget_dev *crg_udc);
+void crg_udc_ep0_req_complete(struct crg_udc_ep *udc_ep_ptr,
+			      unsigned long *flags);
+int crg_udc_prepare_for_setup(struct crg_gadget_dev *crg_udc);
+int crg_udc_set_ep0_halt(struct crg_gadget_dev *crg_udc);
+int crg_udc_update_ep0_maxpacketsize(struct crg_gadget_dev *crg_udc);
+void crg_udc_handle_setup_pkt(struct crg_gadget_dev *crg_udc,
+			      struct usb_ctrlrequest *setup_pkt, u8 setup_tag,
+			      unsigned long *flags);
+int crg_udc_init_ep0(struct crg_gadget_dev *crg_udc);
+int crg_udc_queue_ctrl(struct crg_udc_ep *udc_ep_ptr,
+		       struct crg_udc_request *udc_req_ptr,
+		       u32 num_of_trbs_needed);
+void crg_udc_advance_dequeue_pt(struct crg_udc_ep *udc_ep);
+unsigned int crg_udc_count_trbs_linear(dma_addr_t addr, u64 len);
+unsigned int crg_udc_count_trbs_sg(struct crg_udc_request *udc_req);
+int crg_udc_init_event_ring(struct crg_gadget_dev *crg_udc, int index);
+bool crg_udc_is_event_rings_empty(struct crg_gadget_dev *crg_udc);
+bool crg_udc_is_pointer_less_than(struct crg_transfer_trb *a,
+				  struct crg_transfer_trb *b,
+				  struct crg_udc_ep *udc_ep);
+int crg_udc_process_event_ring(struct crg_gadget_dev *crg_udc, int index,
+			       unsigned long *flags);
+/* This function will go through the list of the USB requests for the
+ * given endpoint and schedule any unscheduled trb's to the xfer ring
+ */
+int crg_udc_queue_pending_trbs(struct crg_udc_ep *udc_ep_ptr);
+/* num_trbs here is the size of the ring. */
+u32 crg_udc_room_on_ring(struct crg_gadget_dev *crg_udc, u32 num_trbs,
+			 struct crg_transfer_trb *p_ring,
+			 struct crg_transfer_trb *enq_pt,
+			 struct crg_transfer_trb *dq_pt);
+void crg_udc_setup_datastage_trb(struct crg_transfer_trb *p_trb, dma_addr_t dma,
+				 u8 pcs, u32 transfer_length, u32 td_size,
+				 bool ioc, bool azp, u8 dir, u8 setup_tag);
+void crg_udc_setup_link_trb(struct crg_transfer_trb *link_trb, bool toggle,
+			    dma_addr_t next_trb);
+void crg_udc_setup_status_trb(struct crg_gadget_dev *crg_udc,
+			      struct crg_transfer_trb *p_trb,
+			      struct usb_request *usb_req, u8 pcs, u8 set_addr,
+			      u8 stall);
+void crg_udc_setup_trb(struct crg_transfer_trb *p_trb, u32 xfer_len,
+		       dma_addr_t xfer_buf_addr, u8 td_size, u8 pcs,
+		       u8 trb_type, bool short_pkt, bool chain_bit,
+		       bool intr_on_compl, bool b_setup_stage, u8 usb_dir,
+		       bool b_isoc, u16 frame_id, bool sia, bool azp);
+void crg_udc_squeeze_xfer_ring(struct crg_udc_ep *udc_ep_ptr,
+			       struct crg_udc_request *udc_req_ptr);
+struct crg_transfer_trb *crg_udc_tran_trb_dma_to_virt(struct crg_udc_ep *udc_ep,
+						    dma_addr_t address);
+dma_addr_t crg_udc_tran_trb_virt_to_dma(struct crg_udc_ep *udc_ep,
+					struct crg_transfer_trb *trb);
+void crg_udc_update_dequeue_pt(struct crg_event_trb *event,
+			       struct crg_udc_ep *udc_ep);
+u32 crg_udc_trb_chunk_len(dma_addr_t dma, u64 remaining);
+u32 crg_udc_actual_before(struct crg_udc_ep *udc_ep,
+			  struct crg_udc_request *udc_req,
+			  struct crg_transfer_trb *pause);
+/*
+ * command related ops
+ */
+int crg_udc_issue_command(struct crg_gadget_dev *crg_udc,
+			  enum crg_cmd_type type, u32 param0, u32 param1);
+void crg_udc_free_resources(struct crg_gadget_dev *crg_udc);
+int crg_udc_program_speed(struct crg_gadget_dev *crg_udc);
+int crg_udc_reset_data_struct(struct crg_gadget_dev *crg_udc);
+int crg_udc_reset(struct crg_gadget_dev *crg_udc, unsigned long *flags);
+struct usb_request *crg_udc_alloc_request(struct usb_ep *_ep, gfp_t gfp_flags);
+int crg_udc_build_td(struct crg_udc_ep *udc_ep_ptr,
+		     struct crg_udc_request *udc_req_ptr);
+void crg_udc_clear_portpm(struct crg_gadget_dev *crg_udc);
+void crg_udc_ep_nuke(struct crg_udc_ep *udc_ep, int status);
+void crg_udc_free_request(struct usb_ep *_ep, struct usb_request *_req);
+void crg_udc_set_pullup(struct crg_gadget_dev *crg_udc, bool is_on);
+void crg_udc_reset_work(struct work_struct *work);
+void crg_udc_set_vbus_detect(struct crg_gadget_dev *crg_udc, bool enable);
+/*
+ * Undo whatever crg_udc_ep_queue() mapped for this request: the merged
+ * SG list, the single buffer, or neither if the request carried a
+ * pre-mapped DMA address.
+ */
+void crg_udc_unmap_request(struct crg_udc_ep *udc_ep,
+			   struct crg_udc_request *udc_req);
+int crg_udc_handle_port_status(struct crg_gadget_dev *crg_udc,
+			       unsigned long *flags);
+int crg_udc_handle_xfer_event(struct crg_gadget_dev *crg_udc,
+			      struct crg_event_trb *event, unsigned long *flags);
+int crg_udc_handle_event(struct crg_gadget_dev *crg_udc,
+			 struct crg_event_trb *event, unsigned long *flags);
+/*
+ * Invoke the bound gadget driver's ->disconnect() callback, if any.
+ *
+ * The gadget disconnect callback may be called in interrupt context.
+ * Release only udc_lock while keeping local IRQs disabled.
+ */
+void crg_udc_notify_disconnect(struct crg_gadget_dev *crg_udc,
+			       unsigned long *flags);
+int crg_udc_ep_halt(struct crg_udc_ep *udc_ep_ptr, int halt, int ignore_wedge,
+		    unsigned long *flags);
+enum crg_ep_state crg_udc_get_ep_state(struct crg_gadget_dev *crg_udc, int dci);
+u32 crg_udc_index2dci(u16 index);
+int crg_udc_init_device_context(struct crg_gadget_dev *crg_udc);
+void crg_udc_init_ep_info(struct crg_gadget_dev *crg_udc);
+void crg_udc_knock_doorbell(struct crg_gadget_dev *crg_udc, int dci);
+/*
+ * Complete a request and give it back to the gadget layer via the
+ * standard usb_gadget_giveback_request() helper.
+ *
+ * Must be called with udc_lock held and interrupts disabled.
+ *
+ * The request's ->complete() callback must not sleep. crg_udc_req_done() drops
+ * udc_lock before the giveback and re-acquires it before returning.
+ * Callers therefore cannot assume state protected by udc_lock is
+ * unchanged across this call.
+ */
+void crg_udc_req_done(struct crg_udc_ep *udc_ep,
+		      struct crg_udc_request *udc_req, int status);
+void crg_udc_clear_req_container(struct crg_udc_request *udc_req_ptr);
+
+#define CRG_QUIRK_DIS_U1U2_ENTRY BIT(0)
+#define CRG_QUIRK_DIS_U1U2_INITIATION BIT(1)
+#define CRG_QUIRK_REINIT_EP0_ON_RESET BIT(2)
+
+#define CRG_UDC_QUIRKS_DEFAULT                                      \
+	(CRG_QUIRK_DIS_U1U2_ENTRY | CRG_QUIRK_DIS_U1U2_INITIATION | \
+	 CRG_QUIRK_REINIT_EP0_ON_RESET)
+
+struct crg_udc_priv {
+	u32 quirks;
+};
+
+extern const struct usb_gadget_ops crg_gadget_ops;
+extern const struct usb_ep_ops crg_udc_ep_ops;
+#define gadget_to_udc(g) (container_of(g, struct crg_gadget_dev, gadget))
+
+#endif /* CRG_UDC_H */

-- 
2.34.1

[PATCH v3 3/6] usb: gadget: udc: crg: Add ring and transfer management

[Petar Stepanovic]

This patch introduces the memory management and ring buffer logic
(TRBs and event rings) required by the Axiado AX3000/AX3005 UDC.

Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
 drivers/usb/gadget/udc/crg/crg-udc-ring.c | 664 ++++++++++++++++++++++++++++++
 1 file changed, 664 insertions(+)

diff --git a/drivers/usb/gadget/udc/crg/crg-udc-ring.c b/drivers/usb/gadget/udc/crg/crg-udc-ring.c
new file mode 100644
index 000000000000..1c57dc1787d4
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-ring.c
@@ -0,0 +1,664 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/sched.h>
+#include <linux/scatterlist.h>
+
+#include "crg-udc.h"
+
+void crg_udc_advance_dequeue_pt(struct crg_udc_ep *udc_ep)
+{
+	struct crg_udc_request *udc_req;
+
+	if (!list_empty(&udc_ep->queue)) {
+		udc_req = list_entry(udc_ep->queue.next, struct crg_udc_request,
+				     queue);
+
+		if (udc_req->first_trb)
+			udc_ep->deq_pt = udc_req->first_trb;
+		else
+			udc_ep->deq_pt = udc_ep->enq_pt;
+	} else {
+		udc_ep->deq_pt = udc_ep->enq_pt;
+	}
+}
+
+unsigned int crg_udc_count_trbs_linear(dma_addr_t addr, u64 len)
+{
+	unsigned int num_trbs = 0;
+
+	while (len) {
+		u32 chunk = crg_udc_trb_chunk_len(addr, len);
+
+		if (!chunk)
+			return 0;
+		num_trbs++;
+		addr += chunk;
+		len -= chunk;
+	}
+
+	return num_trbs;
+}
+
+unsigned int crg_udc_count_trbs_sg(struct crg_udc_request *udc_req)
+{
+	struct scatterlist *sg;
+	u64 remaining = udc_req->usb_req.length;
+	unsigned int i;
+	unsigned int num_trbs = 0;
+
+	for_each_sg(udc_req->usb_req.sg, sg, udc_req->usb_req.num_sgs, i) {
+		dma_addr_t addr;
+		u64 len;
+
+		if (!sg_dma_len(sg))
+			continue;
+		if (!remaining)
+			break;
+
+		len = min_t(u64, sg_dma_len(sg), remaining);
+		addr = sg_dma_address(sg);
+		while (len) {
+			u32 chunk = crg_udc_trb_chunk_len(addr, len);
+
+			if (!chunk)
+				return 0;
+			num_trbs++;
+			addr += chunk;
+			len -= chunk;
+			remaining -= chunk;
+		}
+	}
+
+	return remaining ? 0 : num_trbs;
+}
+
+static dma_addr_t event_trb_virt_to_dma(struct crg_udc_event *udc_event,
+					struct crg_event_trb *event)
+{
+	dma_addr_t dma_addr = 0;
+	unsigned long seg_offset;
+
+	if (!udc_event || !event)
+		return DMA_MAPPING_ERROR;
+
+	if ((void *)event < udc_event->event_ring.cpu_addr ||
+	    (void *)event >=
+		    udc_event->event_ring.cpu_addr + udc_event->event_ring.len)
+		return DMA_MAPPING_ERROR;
+
+	seg_offset = (void *)event - udc_event->event_ring.cpu_addr;
+	dma_addr = udc_event->event_ring.dma_addr + seg_offset;
+
+	return dma_addr;
+}
+
+int crg_udc_init_event_ring(struct crg_gadget_dev *crg_udc, int index)
+{
+	struct crg_uicr __iomem *uicr = crg_udc->uicr[index];
+	struct crg_udc_event *udc_event = &crg_udc->udc_event[index];
+
+	int ret;
+	u32 buff_length;
+	dma_addr_t mapping;
+	bool erst_allocated = false;
+
+	buff_length = CRG_ERST_SIZE * sizeof(struct crg_erst);
+	if (!udc_event->erst.cpu_addr) {
+		udc_event->erst.cpu_addr = dma_alloc_coherent(
+			crg_udc->dev, buff_length, &mapping, GFP_ATOMIC);
+
+		if (!udc_event->erst.cpu_addr)
+			return -ENOMEM;
+		erst_allocated = true;
+	} else {
+		mapping = udc_event->erst.dma_addr;
+	}
+
+	udc_event->erst.len = buff_length;
+	udc_event->erst.dma_addr = mapping;
+	udc_event->p_erst = udc_event->erst.cpu_addr;
+
+	buff_length = CRG_EVENT_RING_SIZE * sizeof(struct crg_event_trb);
+	if (!udc_event->event_ring.cpu_addr) {
+		udc_event->event_ring.cpu_addr = dma_alloc_coherent(
+			crg_udc->dev, buff_length, &mapping, GFP_ATOMIC);
+
+		if (!udc_event->event_ring.cpu_addr) {
+			ret = -ENOMEM;
+			goto err_free_erst;
+		}
+	} else {
+		mapping = udc_event->event_ring.dma_addr;
+	}
+
+	udc_event->event_ring.len = buff_length;
+	udc_event->event_ring.dma_addr = mapping;
+	udc_event->evt_dq_pt = udc_event->event_ring.cpu_addr;
+
+	udc_event->evt_seg0_last_trb =
+		(struct crg_event_trb *)(udc_event->event_ring.cpu_addr) +
+		(CRG_EVENT_RING_SIZE - 1);
+
+	udc_event->ccs = 1;
+
+	udc_event->p_erst->seg_addr_lo =
+		cpu_to_le32(lower_32_bits(udc_event->event_ring.dma_addr));
+	udc_event->p_erst->seg_addr_hi =
+		cpu_to_le32(upper_32_bits(udc_event->event_ring.dma_addr));
+	udc_event->p_erst->seg_size = cpu_to_le32(CRG_EVENT_RING_SIZE);
+	udc_event->p_erst->rsvd = cpu_to_le32(0);
+	/* Ensure that event ring is updated */
+	dma_wmb();
+	/* clear the event ring, to avoid hw unexpected ops
+	 * because of dirty data
+	 */
+	memset(udc_event->event_ring.cpu_addr, 0, buff_length);
+
+	/* Ensure that event ring is clear before programming
+	 * ERSTBA/ERSTSZ/ERDP
+	 */
+	dma_wmb();
+
+	writel(CRG_ERST_SIZE, &uicr->erstsz);
+	writel(lower_32_bits(udc_event->erst.dma_addr), &uicr->erstbalo);
+	writel(upper_32_bits(udc_event->erst.dma_addr), &uicr->erstbahi);
+	writel(lower_32_bits(udc_event->event_ring.dma_addr) |
+		       CRG_U3DC_ERDPLO_EHB,
+	       &uicr->erdplo);
+	writel(upper_32_bits(udc_event->event_ring.dma_addr), &uicr->erdphi);
+
+	writel((CRG_U3DC_IMAN_INT_EN | CRG_U3DC_IMAN_INT_PEND), &uicr->iman);
+	/* Interrupt moderation: 4000 * 250ns = 1ms */
+	writel(4000, &uicr->imod);
+	return 0;
+
+err_free_erst:
+	/* Only free what this call allocated; a pre-existing erst is owned
+	 * by an earlier successful init and released by crg_udc_free_resources().
+	 */
+	if (erst_allocated) {
+		dma_free_coherent(crg_udc->dev, udc_event->erst.len,
+				  udc_event->erst.cpu_addr,
+				  udc_event->erst.dma_addr);
+		udc_event->erst.cpu_addr = NULL;
+	}
+	return ret;
+}
+
+static bool is_event_ring_x_empty(struct crg_gadget_dev *crg_udc, int index)
+{
+	struct crg_event_trb *event;
+	struct crg_udc_event *udc_event;
+
+	udc_event = &crg_udc->udc_event[index];
+	if (udc_event->evt_dq_pt) {
+		event = (struct crg_event_trb *)udc_event->evt_dq_pt;
+
+		if (FIELD_GET(EVE_TRB_CYCLE_BIT_MASK,
+			      le32_to_cpu(event->dw3)) != udc_event->ccs)
+			return true;
+	}
+
+	return false;
+}
+
+bool crg_udc_is_event_rings_empty(struct crg_gadget_dev *crg_udc)
+{
+	int i;
+
+	for (i = 0; i < CRG_RING_NUM; i++) {
+		if (!is_event_ring_x_empty(crg_udc, i)) {
+			dev_err(crg_udc->dev, "event ring not empty\n");
+			return false;
+		}
+	}
+	return true;
+}
+
+bool crg_udc_is_pointer_less_than(struct crg_transfer_trb *a,
+				  struct crg_transfer_trb *b,
+				  struct crg_udc_ep *udc_ep)
+{
+	if (!a || !b || !udc_ep->first_trb || !udc_ep->last_trb)
+		return false;
+	if (b > a && (udc_ep->enq_pt >= b || udc_ep->enq_pt < a))
+		return true;
+	if (b < a && (udc_ep->enq_pt >= b && udc_ep->enq_pt < a))
+		return true;
+	return false;
+}
+
+int crg_udc_process_event_ring(struct crg_gadget_dev *crg_udc, int index,
+			       unsigned long *flags)
+{
+	struct crg_event_trb *event;
+	struct crg_udc_event *udc_event;
+	struct crg_uicr __iomem *uicr = crg_udc->uicr[index];
+	u32 val;
+	dma_addr_t erdp;
+	int ret;
+	unsigned int processed = 0;
+
+	if (!uicr)
+		return -ENODEV;
+
+	val = readl(&uicr->iman);
+	val |= CRG_U3DC_IMAN_INT_PEND;
+	writel(val, &uicr->iman);
+
+	udc_event = &crg_udc->udc_event[index];
+	while (udc_event->evt_dq_pt) {
+		event = (struct crg_event_trb *)udc_event->evt_dq_pt;
+
+		if (FIELD_GET(EVE_TRB_CYCLE_BIT_MASK,
+			      le32_to_cpu(event->dw3)) != udc_event->ccs)
+			break;
+		/* Read the payload only after observing device ownership. */
+		dma_rmb();
+
+		ret = crg_udc_handle_event(crg_udc, event, flags);
+		if (ret == -ECONNRESET) {
+			/* The disconnect path rebuilt and reprogrammed the ring. */
+			return 0;
+		}
+		if (ret < 0)
+			dev_dbg(crg_udc->dev, "failed to process event: %d\n",
+				ret);
+
+		/* Consume malformed events so they cannot pin the ring head. */
+		if (event == udc_event->evt_seg0_last_trb) {
+			udc_event->ccs = udc_event->ccs ? 0 : 1;
+			udc_event->evt_dq_pt = udc_event->event_ring.cpu_addr;
+		} else {
+			udc_event->evt_dq_pt++;
+		}
+
+		/*
+		 * udc_event state is only re-read from crg_udc, never cached
+		 * across this drop, so releasing and reacquiring the lock
+		 * here is safe and lets other udc_lock users, and this CPU's
+		 * other interrupts, make progress during a busy event ring.
+		 */
+		if (++processed % CRG_EVENT_RING_BATCH == 0) {
+			erdp = event_trb_virt_to_dma(udc_event,
+						     udc_event->evt_dq_pt);
+			if (!dma_mapping_error(crg_udc->dev, erdp)) {
+				writel(upper_32_bits(erdp), &uicr->erdphi);
+				writel(lower_32_bits(erdp) |
+					       CRG_U3DC_ERDPLO_EHB,
+				       &uicr->erdplo);
+			}
+			spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+			cond_resched();
+			spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+		}
+	}
+
+	/* update dequeue pointer */
+	erdp = event_trb_virt_to_dma(udc_event, udc_event->evt_dq_pt);
+	if (dma_mapping_error(crg_udc->dev, erdp))
+		return -EINVAL;
+	writel(upper_32_bits(erdp), &uicr->erdphi);
+	writel(lower_32_bits(erdp) | CRG_U3DC_ERDPLO_EHB, &uicr->erdplo);
+
+	return 0;
+}
+
+/* This function will go through the list of the USB requests for the
+ * given endpoint and schedule any unscheduled trb's to the xfer ring
+ */
+int crg_udc_queue_pending_trbs(struct crg_udc_ep *udc_ep_ptr)
+{
+	struct crg_udc_request *udc_req_ptr, *tmp;
+	int ret;
+
+	list_for_each_entry_safe(udc_req_ptr, tmp, &udc_ep_ptr->queue, queue) {
+		if (udc_req_ptr->all_trbs_queued)
+			continue;
+
+		ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+		if (ret == -ENOSPC || ret == -EBUSY) {
+			/*
+			 * Ring is full; leave this request queued and retry
+			 * it (and anything behind it) on the next TRB
+			 * completion via crg_udc_queue_pending_trbs().
+			 */
+			return ret;
+		} else if (ret) {
+			/*
+			 * A non-capacity failure is not transient: fail this
+			 * request instead of leaving it stuck forever, and
+			 * keep rebuilding the rest of the queue.
+			 */
+			dev_err(udc_ep_ptr->crg_udc->dev,
+				"failed to build pending transfer: %d\n", ret);
+			list_del_init(&udc_req_ptr->queue);
+			crg_udc_req_done(udc_ep_ptr, udc_req_ptr, ret);
+			continue;
+		}
+		break;
+	}
+
+	return 0;
+
+}
+
+/* num_trbs here is the size of the ring. */
+u32 crg_udc_room_on_ring(struct crg_gadget_dev *crg_udc, u32 num_trbs,
+			 struct crg_transfer_trb *p_ring,
+			 struct crg_transfer_trb *enq_pt,
+			 struct crg_transfer_trb *dq_pt)
+{
+	u32 i = 0;
+
+	if (enq_pt == dq_pt) {
+		/* ring is empty */
+		return num_trbs - 2;
+	}
+
+	while (enq_pt != dq_pt) {
+		i++;
+
+		enq_pt++;
+
+		if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(enq_pt->dw3)) ==
+		    TRB_TYPE_LINK)
+			enq_pt = p_ring;
+
+		if (i > num_trbs)
+			break;
+	}
+
+	return i - 1;
+}
+
+void crg_udc_setup_datastage_trb(struct crg_transfer_trb *p_trb, dma_addr_t dma,
+				 u8 pcs, u32 transfer_length, u32 td_size,
+				 bool ioc, bool azp, u8 dir, u8 setup_tag)
+{
+	u32 tmp;
+
+	p_trb->dw0 = cpu_to_le32(lower_32_bits(dma));
+	p_trb->dw1 = cpu_to_le32(upper_32_bits(dma));
+
+	/* TRB_Transfer_Length
+	 * For USB_DIR_OUT, this field is the number of data bytes expected
+	 * from xhc. For USB_DIR_IN, this field is the number of data bytes
+	 * the device will send.
+	 */
+	tmp = 0;
+	FIELD_MODIFY(TRB_TRANSFER_LEN_MASK, &tmp, transfer_length);
+	FIELD_MODIFY(TRB_TD_SIZE_MASK, &tmp, td_size);
+	FIELD_MODIFY(TRB_INTR_TARGET_MASK, &tmp, 0);
+	p_trb->dw2 = cpu_to_le32(tmp);
+
+	tmp = 0;
+	FIELD_MODIFY(TRB_CYCLE_BIT_MASK, &tmp, pcs);
+	FIELD_MODIFY(TRB_INTR_ON_SHORT_PKT_MASK, &tmp, 1);
+	FIELD_MODIFY(TRB_INTR_ON_COMPLETION_MASK, &tmp, ioc);
+	FIELD_MODIFY(TRB_TYPE_MASK, &tmp, TRB_TYPE_XFER_DATA_STAGE);
+	FIELD_MODIFY(TRB_APPEND_ZLP_MASK, &tmp, azp);
+	FIELD_MODIFY(DATA_STAGE_TRB_DIR_MASK, &tmp, dir);
+	FIELD_MODIFY(TRB_SETUP_TAG_MASK, &tmp, setup_tag);
+
+	/* Publish ownership only after the TRB payload is visible. */
+	dma_wmb();
+	p_trb->dw3 = cpu_to_le32(tmp);
+}
+
+void crg_udc_setup_link_trb(struct crg_transfer_trb *link_trb, bool toggle,
+			    dma_addr_t next_trb)
+{
+	u32 dw = 0;
+
+	link_trb->dw0 = cpu_to_le32(lower_32_bits(next_trb));
+	link_trb->dw1 = cpu_to_le32(upper_32_bits(next_trb));
+	link_trb->dw2 = cpu_to_le32(0);
+
+	FIELD_MODIFY(TRB_TYPE_MASK, &dw, TRB_TYPE_LINK);
+	if (toggle)
+		FIELD_MODIFY(TRB_LINK_TOGGLE_CYCLE_MASK, &dw, 1);
+	else
+		FIELD_MODIFY(TRB_LINK_TOGGLE_CYCLE_MASK, &dw, 0);
+
+	/* Publish ownership only after the link address is visible. */
+	dma_wmb();
+	link_trb->dw3 = cpu_to_le32(dw);
+}
+
+void crg_udc_setup_status_trb(struct crg_gadget_dev *crg_udc,
+			      struct crg_transfer_trb *p_trb,
+			      struct usb_request *usb_req, u8 pcs, u8 set_addr,
+			      u8 stall)
+{
+	u32 tmp, dir = 0;
+
+	/* There are some cases where crg_udc_setup_status_trb() is called with
+	 * usb_req set to NULL.
+	 */
+	p_trb->dw0 = cpu_to_le32(0);
+	p_trb->dw1 = cpu_to_le32(0);
+
+	tmp = 0;
+	FIELD_MODIFY(TRB_INTR_TARGET_MASK, &tmp, 0);
+	p_trb->dw2 = cpu_to_le32(tmp);
+
+	tmp = 0;
+	FIELD_MODIFY(TRB_CYCLE_BIT_MASK, &tmp, pcs);
+	FIELD_MODIFY(TRB_INTR_ON_COMPLETION_MASK, &tmp, 1);
+	FIELD_MODIFY(TRB_TYPE_MASK, &tmp, TRB_TYPE_XFER_STATUS_STAGE);
+
+	dir = (crg_udc->setup_status == STATUS_STAGE_XFER) ? 0 : 1;
+	FIELD_MODIFY(DATA_STAGE_TRB_DIR_MASK, &tmp, dir);
+
+	FIELD_MODIFY(TRB_SETUP_TAG_MASK, &tmp, crg_udc->setup_tag);
+	FIELD_MODIFY(STATUS_STAGE_TRB_STALL_MASK, &tmp, stall);
+	FIELD_MODIFY(STATUS_STAGE_TRB_SET_ADDR_MASK, &tmp, set_addr);
+
+	/* Publish ownership only after the TRB payload is visible. */
+	dma_wmb();
+	p_trb->dw3 = cpu_to_le32(tmp);
+}
+
+void crg_udc_setup_trb(struct crg_transfer_trb *p_trb, u32 xfer_len,
+		       dma_addr_t xfer_buf_addr, u8 td_size, u8 pcs,
+		       u8 trb_type, bool short_pkt, bool chain_bit,
+		       bool intr_on_compl, bool b_setup_stage, u8 usb_dir,
+		       bool b_isoc, u16 frame_id, bool sia, bool azp)
+{
+	u32 tmp;
+
+	p_trb->dw0 = cpu_to_le32(lower_32_bits(xfer_buf_addr));
+	p_trb->dw1 = cpu_to_le32(upper_32_bits(xfer_buf_addr));
+
+	tmp = 0;
+	FIELD_MODIFY(TRB_TRANSFER_LEN_MASK, &tmp, xfer_len);
+	FIELD_MODIFY(TRB_TD_SIZE_MASK, &tmp, td_size);
+	FIELD_MODIFY(TRB_INTR_TARGET_MASK, &tmp, 0);
+
+	p_trb->dw2 = cpu_to_le32(tmp);
+
+	tmp = 0;
+	FIELD_MODIFY(TRB_CYCLE_BIT_MASK, &tmp, pcs);
+	FIELD_MODIFY(TRB_INTR_ON_SHORT_PKT_MASK, &tmp, short_pkt);
+	FIELD_MODIFY(TRB_CHAIN_BIT_MASK, &tmp, chain_bit);
+	FIELD_MODIFY(TRB_INTR_ON_COMPLETION_MASK, &tmp, intr_on_compl);
+	FIELD_MODIFY(TRB_APPEND_ZLP_MASK, &tmp, azp);
+	FIELD_MODIFY(TRB_TYPE_MASK, &tmp, trb_type);
+
+	if (b_setup_stage)
+		FIELD_MODIFY(DATA_STAGE_TRB_DIR_MASK, &tmp, usb_dir);
+
+	if (b_isoc) {
+		FIELD_MODIFY(ISOC_TRB_FRAME_ID_MASK, &tmp, frame_id);
+		FIELD_MODIFY(ISOC_TRB_SIA_MASK, &tmp, sia);
+	}
+
+	/* Publish ownership only after the TRB payload is visible. */
+	dma_wmb();
+	p_trb->dw3 = cpu_to_le32(tmp);
+}
+
+void crg_udc_squeeze_xfer_ring(struct crg_udc_ep *udc_ep_ptr,
+			       struct crg_udc_request *udc_req_ptr)
+{
+	struct crg_transfer_trb *trb = udc_req_ptr->first_trb;
+	struct crg_udc_request *next_req, *tmp;
+	u8 new_pcs;
+
+	if (!trb)
+		return;
+
+	/* The canceled request becomes the new enqueue point. Its original
+	 * cycle bit is the cycle expected at that point after the tail is
+	 * removed.
+	 */
+	new_pcs = FIELD_GET(TRB_CYCLE_BIT_MASK, le32_to_cpu(trb->dw3));
+
+	while (trb != udc_ep_ptr->enq_pt) {
+		memset(trb, 0, sizeof(*trb));
+		trb++;
+		if (trb == udc_ep_ptr->last_trb)
+			trb = udc_ep_ptr->first_trb;
+	}
+
+	udc_ep_ptr->enq_pt = udc_req_ptr->first_trb;
+	udc_ep_ptr->pcs = new_pcs;
+	udc_ep_ptr->tran_ring_full = false;
+	crg_udc_setup_link_trb(udc_ep_ptr->last_trb, true,
+			udc_ep_ptr->tran_ring_info.dma_addr);
+	dma_wmb();
+
+	if (list_is_last(&udc_req_ptr->queue, &udc_ep_ptr->queue))
+		return;
+
+	next_req = udc_req_ptr;
+	list_for_each_entry_safe_continue(next_req, tmp, &udc_ep_ptr->queue,
+					  queue) {
+		int ret;
+
+		next_req->usb_req.status = -EINPROGRESS;
+		next_req->usb_req.actual = 0;
+		crg_udc_clear_req_container(next_req);
+		ret = crg_udc_build_td(udc_ep_ptr, next_req);
+		if (ret == -ENOSPC) {
+			/* Ring is full; a later TRB completion will retry
+			 * this and any remaining requests via
+			 * crg_udc_queue_pending_trbs().
+			 */
+			udc_ep_ptr->tran_ring_full = true;
+			break;
+		} else if (ret) {
+			/*
+			 * A non-capacity failure is not transient: fail this
+			 * request instead of leaving it stuck forever, and
+			 * keep rebuilding the rest of the queue.
+			 */
+			dev_err(udc_ep_ptr->crg_udc->dev,
+				"failed to rebuild queued request: %d\n", ret);
+			crg_udc_req_done(udc_ep_ptr, next_req, ret);
+			continue;
+		}
+		if (udc_ep_ptr->tran_ring_full)
+			break;
+	}
+}
+
+struct crg_transfer_trb *crg_udc_tran_trb_dma_to_virt(struct crg_udc_ep *udc_ep,
+						    dma_addr_t address)
+{
+	dma_addr_t offset;
+	struct crg_transfer_trb *trb_virt;
+
+	if (lower_32_bits(address) & 0xf)
+		return NULL;
+
+	offset = address - udc_ep->tran_ring_info.dma_addr;
+	if (unlikely(offset >= udc_ep->tran_ring_info.len))
+		return NULL;
+	offset = offset / sizeof(struct crg_transfer_trb);
+	trb_virt = udc_ep->first_trb + offset;
+	return trb_virt;
+}
+
+dma_addr_t crg_udc_tran_trb_virt_to_dma(struct crg_udc_ep *udc_ep,
+					struct crg_transfer_trb *trb)
+{
+	unsigned long offset;
+	int trb_idx;
+	dma_addr_t dma_addr = 0;
+
+	trb_idx = trb - udc_ep->first_trb;
+	if (unlikely(trb_idx < 0))
+		return DMA_MAPPING_ERROR;
+
+	offset = trb_idx * sizeof(*trb);
+	if (unlikely(offset >= udc_ep->tran_ring_info.len))
+		return DMA_MAPPING_ERROR;
+	dma_addr = udc_ep->tran_ring_info.dma_addr + offset;
+	return dma_addr;
+}
+
+void crg_udc_update_dequeue_pt(struct crg_event_trb *event,
+			       struct crg_udc_ep *udc_ep)
+{
+	u32 deq_pt_lo = le32_to_cpu(event->dw0);
+	u32 deq_pt_hi = le32_to_cpu(event->dw1);
+	u64 dq_pt_addr = (u64)deq_pt_lo | ((u64)deq_pt_hi << 32);
+	struct crg_transfer_trb *deq_pt;
+
+	deq_pt = crg_udc_tran_trb_dma_to_virt(udc_ep, dq_pt_addr);
+	if (!deq_pt)
+		return;
+	deq_pt++;
+	if (deq_pt == udc_ep->last_trb)
+		deq_pt = udc_ep->first_trb;
+
+	if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(deq_pt->dw3)) == TRB_TYPE_LINK)
+		deq_pt = udc_ep->first_trb;
+
+	udc_ep->deq_pt = deq_pt;
+}
+
+u32 crg_udc_trb_chunk_len(dma_addr_t dma, u64 remaining)
+{
+	u32 boundary = CRG_UDC_TRB_MAX_BUFFER_SIZE -
+		       (lower_32_bits(dma) & (CRG_UDC_TRB_MAX_BUFFER_SIZE - 1));
+
+	return min_t(u64, remaining, boundary);
+}
+
+u32 crg_udc_actual_before(struct crg_udc_ep *udc_ep,
+			  struct crg_udc_request *udc_req,
+			  struct crg_transfer_trb *pause)
+{
+	struct crg_transfer_trb *trb = udc_req->first_trb;
+	u32 actual = 0;
+	u32 guard = 0;
+	u32 max_iter = udc_ep->tran_ring_info.len / sizeof(*trb);
+
+	if (!trb || !pause)
+		return 0;
+
+	while (trb != pause && trb != udc_ep->enq_pt && guard++ < max_iter) {
+		u32 len;
+
+		if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(trb->dw3)) ==
+		    TRB_TYPE_LINK) {
+			trb = udc_ep->first_trb;
+			continue;
+		}
+		len = FIELD_GET(TRB_TRANSFER_LEN_MASK, le32_to_cpu(trb->dw2));
+		actual += len;
+		trb++;
+		if (trb == udc_ep->last_trb)
+			trb = udc_ep->first_trb;
+	}
+
+	return min(actual, udc_req->usb_req.length);
+}

-- 
2.34.1

[PATCH v3 4/6] usb: gadget: udc: crg: Add endpoint 0 handling

[Petar Stepanovic]

This patch adds the endpoint 0 setup packet, status, and control
transfer handling for the Axiado AX3000/AX3005 UDC.

Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
 drivers/usb/gadget/udc/crg/crg-udc-ep0.c | 1106 ++++++++++++++++++++++++++++++
 1 file changed, 1106 insertions(+)

diff --git a/drivers/usb/gadget/udc/crg/crg-udc-ep0.c b/drivers/usb/gadget/udc/crg/crg-udc-ep0.c
new file mode 100644
index 000000000000..dce36d23d3ec
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-ep0.c
@@ -0,0 +1,1106 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/iopoll.h>
+
+#include "crg-udc.h"
+#include "crg-udc-trace.h"
+
+static void enable_setup_event(struct crg_gadget_dev *crg_udc);
+
+void crg_udc_build_ep0_status(struct crg_udc_ep *udc_ep_ptr, bool default_value,
+			      u32 status, struct crg_udc_request *udc_req_ptr,
+			      u8 set_addr, u8 stall)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+	struct crg_transfer_trb *enq_pt = udc_ep_ptr->enq_pt;
+
+	if (default_value) {
+		udc_req_ptr = crg_udc->status_req;
+		udc_req_ptr->usb_req.length = 0;
+		udc_req_ptr->usb_req.status = status;
+		udc_req_ptr->usb_req.actual = 0;
+		udc_req_ptr->usb_req.complete = NULL;
+	} else {
+		udc_req_ptr->usb_req.status = status;
+		udc_req_ptr->usb_req.actual = 0;
+	}
+
+	crg_udc_setup_status_trb(crg_udc, enq_pt, &udc_req_ptr->usb_req,
+				 udc_ep_ptr->pcs, set_addr, stall);
+
+	enq_pt++;
+
+	/* check if we are at end of trb segment.  If so, update
+	 * pcs and enq for next segment
+	 */
+	if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(enq_pt->dw3)) ==
+	    TRB_TYPE_LINK) {
+		if (FIELD_GET(TRB_LINK_TOGGLE_CYCLE_MASK,
+			      le32_to_cpu(enq_pt->dw3))) {
+			dma_wmb();
+			le32p_replace_bits(&enq_pt->dw3, udc_ep_ptr->pcs,
+					   TRB_CYCLE_BIT_MASK);
+			udc_ep_ptr->pcs ^= 0x1;
+		}
+		enq_pt = udc_ep_ptr->first_trb;
+	}
+	udc_ep_ptr->enq_pt = enq_pt;
+
+	crg_udc_knock_doorbell(crg_udc, 0);
+
+	list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+}
+
+/*
+ * Abort the current EP0 control transfer when a new SETUP packet arrives.
+ *
+ * A SETUP packet is allowed to abort any in-progress control transfer. Do
+ * not defer the new SETUP until the old transfer completes: doing so leaves
+ * the old request on the EP0 software queue and subsequent gadget requests
+ * are rejected as "queue not empty".
+ *
+ * The controller's EP0 ring is reinitialized at the ring start, making all
+ * TRBs belonging to the aborted control transfer stale. The new SETUP is
+ * then handled immediately with a fresh setup tag.
+ */
+int crg_udc_abort_ep0_for_setup(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_udc_ep *udc_ep0 = &crg_udc->udc_ep[0];
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	dma_addr_t dqptaddr;
+	u32 cmd_param0;
+	u32 cmd_param1;
+	u32 tmp;
+	int ret;
+
+	/*
+	 * A new SETUP is allowed to abort the current control transfer.
+	 * Keep EP0 in ABORTING for the complete stop/nuke/reinitialize
+	 * sequence. crg_udc_req_done() drops udc_lock while invoking the gadget
+	 * callback, so the callback may re-enter the UDC during the nuke.
+	 */
+	crg_udc->setup_status = WAIT_FOR_SETUP;
+	crg_udc->ep0_state = EP0_STATE_ABORTING;
+
+	/*
+	 * Stop EP0 before touching the transfer ring. The controller may still
+	 * own the old TRBs even when no EP0 request is running from the gadget's
+	 * point of view. Do not clear the DMA-visible ring until the hardware
+	 * reports that EP0 is no longer running.
+	 */
+	ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP, BIT(0), 0);
+	if (ret) {
+		dev_err(crg_udc->dev,
+			"failed to stop EP0 for SETUP abort: %d\n", ret);
+		crg_udc->ep0_state = EP0_STATE_ERROR;
+		return ret;
+	}
+
+	ret = readl_poll_timeout_atomic(&uccr->ep_running, tmp, !(tmp & BIT(0)),
+					10, 10000);
+	if (ret) {
+		dev_err(crg_udc->dev, "timeout stopping EP0 for SETUP abort\n");
+		crg_udc->ep0_state = EP0_STATE_ERROR;
+		return ret;
+	}
+
+	udc_ep0->ep_state = EP_STATE_STOPPED;
+	/* Complete requests belonging to the aborted control transfer. */
+	crg_udc_ep_nuke(udc_ep0, -ECONNRESET);
+
+	if (!udc_ep0->first_trb || !udc_ep0->tran_ring_info.cpu_addr) {
+		crg_udc->ep0_state = EP0_STATE_ERROR;
+		return -ENODEV;
+	}
+
+	/*
+	 * The controller has stopped consuming EP0 TRBs. It is now safe to
+	 * discard the old transfer descriptors and restart the ring from its
+	 * beginning. No ring memory is freed here.
+	 */
+	memset(udc_ep0->first_trb, 0, udc_ep0->tran_ring_info.len);
+	udc_ep0->enq_pt = udc_ep0->first_trb;
+	udc_ep0->deq_pt = udc_ep0->first_trb;
+	udc_ep0->pcs = 1;
+	udc_ep0->tran_ring_full = false;
+	crg_udc_setup_link_trb(udc_ep0->last_trb, true,
+			       udc_ep0->tran_ring_info.dma_addr);
+
+	/* Make the new ring contents visible before handing the ring to HW. */
+	dma_wmb();
+
+	dqptaddr = crg_udc_tran_trb_virt_to_dma(udc_ep0, udc_ep0->first_trb);
+	if (dma_mapping_error(crg_udc->dev, dqptaddr)) {
+		crg_udc->ep0_state = EP0_STATE_ERROR;
+		return -EIO;
+	}
+
+	cmd_param0 = (lower_32_bits(dqptaddr) & CRG_CMD0_0_DQPTRLO_MASK) |
+		     FIELD_PREP(CRG_CMD0_0_DCS_MASK, udc_ep0->pcs);
+	cmd_param1 = upper_32_bits(dqptaddr);
+
+	ret = crg_udc_issue_command(crg_udc, CRG_CMD_INIT_EP0, cmd_param0,
+				    cmd_param1);
+	if (ret) {
+		dev_err(crg_udc->dev,
+			"failed to reinitialize EP0 for SETUP: %d\n", ret);
+		crg_udc->ep0_state = EP0_STATE_ERROR;
+		return ret;
+	}
+
+	udc_ep0->ep_state = EP_STATE_RUNNING;
+	crg_udc->ep0_state = EP0_STATE_IDLE;
+	return 0;
+}
+
+int crg_udc_enable_setup(struct crg_gadget_dev *crg_udc)
+{
+	enable_setup_event(crg_udc);
+	crg_udc->device_state = USB_STATE_DEFAULT;
+	usb_gadget_set_state(&crg_udc->gadget, USB_STATE_DEFAULT);
+	crg_udc->setup_status = WAIT_FOR_SETUP;
+
+	return 0;
+}
+
+static void enable_setup_event(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 val;
+
+	val = readl(&uccr->config1);
+	val |= CRG_U3DC_CFG1_SETUP_EVENT_EN;
+	writel(val, &uccr->config1);
+}
+
+void crg_udc_ep0_req_complete(struct crg_udc_ep *udc_ep_ptr,
+			      unsigned long *flags)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+
+	switch (crg_udc->setup_status) {
+	case DATA_STAGE_XFER:
+		crg_udc->setup_status = STATUS_STAGE_RECV;
+		crg_udc_build_ep0_status(udc_ep_ptr, true, -EINPROGRESS, NULL,
+					 0, 0);
+		break;
+	case DATA_STAGE_RECV:
+		crg_udc->setup_status = STATUS_STAGE_XFER;
+		crg_udc_build_ep0_status(udc_ep_ptr, true, -EINPROGRESS, NULL,
+					 0, 0);
+		break;
+	default:
+		if (crg_udc->setup_fn_call_back) {
+			crg_udc->setup_fn_call_back(crg_udc);
+			crg_udc->setup_fn_call_back = NULL;
+		}
+		crg_udc->setup_status = WAIT_FOR_SETUP;
+		break;
+	}
+}
+
+static void get_status_cmpl(struct usb_ep *ep, struct usb_request *req)
+{
+	struct crg_udc_ep *udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+	struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+	unsigned long flags;
+
+	kfree(req->buf);
+	req->buf = NULL;
+
+	/*
+	 * A non-zero status means this request was completed early (e.g.
+	 * crg_udc_ep_nuke(..., -ECONNRESET) from crg_udc_abort_ep0_for_setup()
+	 * for a new SETUP). EP0 is being reinitialized for that new control
+	 * transfer; queuing a status stage here would race with it.
+	 */
+	if (req->status)
+		return;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	crg_udc->setup_status = STATUS_STAGE_RECV;
+	crg_udc_build_ep0_status(udc_ep, true, -EINPROGRESS, NULL, 0, 0);
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+}
+
+static void getstatusrequest(struct crg_gadget_dev *crg_udc, u8 request_type,
+			     u16 value, u16 index, u16 length)
+{
+	struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+	struct crg_udc_request *udc_req_ptr = crg_udc->status_req;
+	u16 status_val = 0;
+	u32 dci;
+	int ret;
+
+	if (!udc_req_ptr || !list_empty(&udc_req_ptr->queue) || value ||
+	    length != sizeof(u16)) {
+		crg_udc_set_ep0_halt(crg_udc);
+		return;
+	}
+
+	switch (request_type & USB_RECIP_MASK) {
+	case USB_RECIP_DEVICE:
+		if (index)
+			goto invalid;
+		if (crg_udc->gadget.speed == USB_SPEED_HIGH ||
+		    crg_udc->gadget.speed == USB_SPEED_FULL) {
+			if (crg_udc->u2_rwe)
+				status_val |= BIT(USB_DEVICE_REMOTE_WAKEUP);
+		} else if (crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+			if (crg_udc->feature_u1_enable)
+				status_val |= BIT(USB_DEV_STAT_U1_ENABLED);
+			if (crg_udc->feature_u2_enable)
+				status_val |= BIT(USB_DEV_STAT_U2_ENABLED);
+		}
+		if (crg_udc->gadget.is_selfpowered)
+			status_val |= BIT(USB_DEVICE_SELF_POWERED);
+		break;
+
+	case USB_RECIP_INTERFACE:
+		if (crg_udc->device_state != USB_STATE_CONFIGURED ||
+		    (index & 0xff00))
+			goto invalid;
+		break;
+
+	case USB_RECIP_ENDPOINT:
+		if (index & ~(USB_ENDPOINT_NUMBER_MASK | USB_DIR_IN))
+			goto invalid;
+		dci = crg_udc_index2dci(index);
+		if (dci == 1 || dci >= CRG_NUM_EP_CX)
+			goto invalid;
+		if (crg_udc->device_state < USB_STATE_ADDRESS && dci != 0)
+			goto invalid;
+		if (crg_udc->device_state == USB_STATE_ADDRESS && dci != 0)
+			goto invalid;
+		if (crg_udc->device_state == USB_STATE_CONFIGURED &&
+		    crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_DISABLED)
+			goto invalid;
+		if (crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_HALTED)
+			status_val = BIT(USB_ENDPOINT_HALT);
+		break;
+
+	default:
+		goto invalid;
+	}
+
+	kfree(udc_req_ptr->usb_req.buf);
+
+	udc_req_ptr->usb_req.buf = kzalloc_obj(status_val, GFP_ATOMIC);
+	if (!udc_req_ptr->usb_req.buf)
+		goto invalid;
+
+	*(__le16 *)udc_req_ptr->usb_req.buf = cpu_to_le16(status_val);
+	udc_req_ptr->usb_req.length = sizeof(status_val);
+	udc_req_ptr->usb_req.status = -EINPROGRESS;
+	udc_req_ptr->usb_req.actual = 0;
+	udc_req_ptr->usb_req.complete = get_status_cmpl;
+
+	udc_req_ptr->map_dir = DMA_TO_DEVICE;
+	ret = usb_gadget_map_request_by_dev(crg_udc->dev, &udc_req_ptr->usb_req,
+					    true);
+	if (ret) {
+		kfree(udc_req_ptr->usb_req.buf);
+		udc_req_ptr->usb_req.buf = NULL;
+		goto invalid;
+	}
+
+	crg_udc->setup_status = DATA_STAGE_XFER;
+
+	ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+	if (ret) {
+		crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr);
+		kfree(udc_req_ptr->usb_req.buf);
+		udc_req_ptr->usb_req.buf = NULL;
+		crg_udc_set_ep0_halt(crg_udc);
+		return;
+	}
+	list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+	return;
+
+invalid:
+	crg_udc_set_ep0_halt(crg_udc);
+}
+
+int crg_udc_prepare_for_setup(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_udc_ep *udc_ep0_ptr;
+
+	dma_addr_t dqptaddr;
+	u32 cmd_param0;
+	u32 cmd_param1;
+
+	if (!crg_udc_is_event_rings_empty(crg_udc) ||
+	    crg_udc->portsc_on_reconnecting == 1)
+		return -EBUSY;
+
+	udc_ep0_ptr = &crg_udc->udc_ep[0];
+	/* If we reinit ep0 on bus reset, we just make ep0 dequeue pointer
+	 * align with enqueue pointer, all remaining xfer trbs became dumb once
+	 * which will not produce xfer event anymore.
+	 *
+	 * If we considering the opposite solution, we should wait all ep0 xfer
+	 * trbs be completed(with some err complete code)
+	 */
+	if (crg_udc->reinit_ep0_on_bus_reset) {
+		/* Complete any reqs on EP0 queue */
+		crg_udc_ep_nuke(udc_ep0_ptr, -ESHUTDOWN);
+
+		/* context related ops */
+		dqptaddr = crg_udc_tran_trb_virt_to_dma(udc_ep0_ptr,
+							udc_ep0_ptr->enq_pt);
+		if (dma_mapping_error(crg_udc->dev, dqptaddr))
+			return -EIO;
+		cmd_param0 =
+			(lower_32_bits(dqptaddr) & CRG_CMD0_0_DQPTRLO_MASK) |
+			FIELD_PREP(CRG_CMD0_0_DCS_MASK, udc_ep0_ptr->pcs);
+		cmd_param1 = upper_32_bits(dqptaddr);
+		if (crg_udc_issue_command(crg_udc, CRG_CMD_INIT_EP0, cmd_param0,
+					  cmd_param1))
+			return -EIO;
+
+		udc_ep0_ptr->deq_pt = udc_ep0_ptr->enq_pt;
+		udc_ep0_ptr->tran_ring_full = false;
+	} else {
+		if (!list_empty(&udc_ep0_ptr->queue)) {
+			dev_err(crg_udc->dev,
+				"remaining ep0 xfer trbs on reset\n");
+			return -EBUSY;
+		}
+	}
+
+	crg_udc_enable_setup(crg_udc);
+
+	return 0;
+}
+
+static void set_address_cmpl(struct crg_gadget_dev *crg_udc)
+{
+	if (crg_udc->device_state == USB_STATE_DEFAULT &&
+	    crg_udc->dev_addr != 0) {
+		crg_udc->device_state = USB_STATE_ADDRESS;
+		usb_gadget_set_state(&crg_udc->gadget, USB_STATE_ADDRESS);
+	} else if (crg_udc->device_state == USB_STATE_ADDRESS) {
+		if (crg_udc->dev_addr == 0) {
+			crg_udc->device_state = USB_STATE_DEFAULT;
+			usb_gadget_set_state(&crg_udc->gadget,
+					     USB_STATE_DEFAULT);
+		}
+
+	}
+}
+
+int crg_udc_set_ep0_halt(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+	enum crg_ep_state ep_state;
+
+	ep_state = crg_udc_get_ep_state(crg_udc, udc_ep_ptr->dci);
+	if (ep_state == EP_STATE_HALTED || ep_state == EP_STATE_DISABLED)
+		return 0;
+
+	crg_udc_build_ep0_status(udc_ep_ptr, true, -EINVAL, NULL, 0, 1);
+
+	udc_ep_ptr->ep_state = EP_STATE_HALTED;
+
+	return 0;
+}
+
+static void set_sel_cmpl(struct usb_ep *ep, struct usb_request *req)
+{
+	struct crg_udc_ep *udc_ep;
+	struct crg_gadget_dev *crg_udc;
+	struct crg_sel_value *sel_value;
+
+	udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+	crg_udc = udc_ep->crg_udc;
+
+	if (req->buf) {
+		if (!req->status && req->actual >= sizeof(*sel_value)) {
+			sel_value = req->buf;
+			crg_udc->sel_value.u2_pel_value =
+				le16_to_cpu(sel_value->u2_pel_value);
+			crg_udc->sel_value.u2_sel_value =
+				le16_to_cpu(sel_value->u2_sel_value);
+			crg_udc->sel_value.u1_pel_value = sel_value->u1_pel_value;
+			crg_udc->sel_value.u1_sel_value = sel_value->u1_sel_value;
+		}
+
+		kfree(req->buf);
+		req->buf = NULL;
+	}
+}
+
+static void set_test_mode_cmpl(struct crg_gadget_dev *crg_udc)
+{
+	if (crg_udc->set_tm != 0) {
+		u32 tmp;
+		struct crg_uccr __iomem *uccr = crg_udc->uccr;
+
+		tmp = readl(&uccr->u2portpmsc);
+		FIELD_MODIFY(CRG_U3DC_U2PORTPM_TM_MASK, &tmp, crg_udc->set_tm);
+		writel(tmp, &uccr->u2portpmsc);
+
+		crg_udc->set_tm = 0;
+	}
+}
+
+static void setaddressrequest(struct crg_gadget_dev *crg_udc, u16 value,
+			      u16 index, u16 length)
+{
+	int status = -EINPROGRESS;
+	u8 status_set_addr = 0;
+
+	if (value > 127 || index != 0 || length != 0) {
+		status = -EINVAL;
+		goto set_address_error;
+	}
+
+	if ((crg_udc->device_state == USB_STATE_DEFAULT && value != 0) ||
+	    crg_udc->device_state == USB_STATE_ADDRESS) {
+		u32 param0;
+
+		param0 = FIELD_PREP(CRG_CMD2_0_DEV_ADDR_MASK, value);
+		if (crg_udc_issue_command(crg_udc, CRG_CMD_SET_ADDR, param0,
+					  0)) {
+			status = -EIO;
+		} else {
+			crg_udc->dev_addr = value;
+			status_set_addr = 1;
+		}
+	} else {
+		status = -EINVAL;
+	}
+
+set_address_error:
+	dev_dbg(crg_udc->dev, "crg_udc_build_ep0_status for Address Device\n");
+
+	crg_udc->setup_status = STATUS_STAGE_XFER;
+	crg_udc->setup_fn_call_back = &set_address_cmpl;
+	crg_udc_build_ep0_status(&crg_udc->udc_ep[0], true, status, NULL,
+				 status_set_addr, status != -EINPROGRESS);
+}
+
+static bool setfeaturesrequest(struct crg_gadget_dev *crg_udc, u8 request_type,
+			       u8 b_request, u16 value, u16 index, u16 length,
+			       unsigned long *flags)
+{
+	int status = -EINPROGRESS;
+	u8 dci;
+	struct crg_udc_ep *udc_ep_ptr;
+	u32 tmp;
+	u32 set_feat = 0;
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+
+	if (length != 0) {
+		status = -EINVAL;
+		goto set_feature_error;
+	}
+
+	if (crg_udc->device_state == USB_STATE_DEFAULT) {
+		status = -EINVAL;
+		goto set_feature_error;
+	}
+
+	set_feat = (b_request == USB_REQ_SET_FEATURE) ? 1 : 0;
+	if ((request_type & (USB_RECIP_MASK | USB_TYPE_MASK)) ==
+	    (USB_RECIP_ENDPOINT | USB_TYPE_STANDARD)) {
+		dev_dbg(crg_udc->dev, "Halt/Unhalt EP\n");
+		if (crg_udc->device_state == USB_STATE_ADDRESS) {
+			if (index != 0) {
+				status = -EINVAL;
+				goto set_feature_error;
+			}
+		}
+
+		if (value != USB_ENDPOINT_HALT)
+			goto set_feature_error;
+
+		if (index & ~(USB_ENDPOINT_NUMBER_MASK | USB_DIR_IN))
+			goto set_feature_error;
+
+		dci = crg_udc_index2dci(index);
+		if (dci == 1 || dci >= CRG_NUM_EP_CX)
+			goto set_feature_error;
+		if (dci != 0 && crg_udc->device_state == USB_STATE_CONFIGURED &&
+		    crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_DISABLED)
+			goto set_feature_error;
+
+		udc_ep_ptr = &crg_udc->udc_ep[dci];
+		status = crg_udc_ep_halt(
+			udc_ep_ptr, (b_request == USB_REQ_SET_FEATURE) ? 1 : 0,
+			0, flags);
+
+		if (status < 0)
+			goto set_feature_error;
+	} else if ((request_type & (USB_RECIP_MASK | USB_TYPE_MASK)) ==
+		   (USB_RECIP_DEVICE | USB_TYPE_STANDARD)) {
+		switch (value) {
+		case USB_DEVICE_REMOTE_WAKEUP:
+			/* REMOTE_WAKEUP selector is not used by USB3.0 */
+			if (index ||
+			    crg_udc->device_state < USB_STATE_DEFAULT ||
+			    crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+				status = -EINVAL;
+				goto set_feature_error;
+			}
+
+			crg_udc->u2_rwe = set_feat;
+			crg_udc->gadget.wakeup_armed = set_feat;
+
+			break;
+		case USB_DEVICE_U1_ENABLE:
+		case USB_DEVICE_U2_ENABLE: {
+			u32 timeout_val;
+
+			if (crg_udc->device_state != USB_STATE_CONFIGURED) {
+				dev_err(crg_udc->dev,
+					"u12 enable fail, usb state=%d\n",
+					crg_udc->device_state);
+				status = -EINVAL;
+				goto set_feature_error;
+			}
+
+			if (index != 0 ||
+			    crg_udc->gadget.speed < USB_SPEED_SUPER) {
+				status = -EINVAL;
+				goto set_feature_error;
+			}
+
+			if (set_feat == 1 && crg_udc->u12_forbidden) {
+				status = -EINVAL;
+				goto set_feature_error;
+			}
+
+			tmp = readl(&uccr->u3portpmsc);
+			if (value == USB_DEVICE_U1_ENABLE) {
+				if (!crg_udc->u12_initiate_forbidden) {
+					tmp &= (~CRG_U3DC_U3PORTPM_U1IEN);
+					tmp |= FIELD_PREP(
+						CRG_U3DC_U3PORTPM_U1IEN,
+						set_feat);
+				}
+				if (!crg_udc->u12_forbidden)
+					timeout_val =
+						set_feat ? CRG_UDC_U1_TIMEOUT_VAL : 0;
+				else
+					timeout_val = 0;
+
+				FIELD_MODIFY(CRG_U3DC_U3PORTPM_U1TMOUT_MASK,
+					     &tmp, timeout_val);
+
+				crg_udc->feature_u1_enable = set_feat;
+			}
+
+			if (value == USB_DEVICE_U2_ENABLE) {
+				if (!crg_udc->u12_initiate_forbidden) {
+					tmp &= (~CRG_U3DC_U3PORTPM_U2IEN);
+					tmp |= FIELD_PREP(
+						CRG_U3DC_U3PORTPM_U2IEN,
+						set_feat);
+				}
+				if (!crg_udc->u12_forbidden)
+					timeout_val =
+						set_feat ? CRG_UDC_U2_TIMEOUT_VAL : 0;
+				else
+					timeout_val = 0;
+
+				FIELD_MODIFY(CRG_U3DC_U3PORTPM_U2TMOUT_MASK,
+					     &tmp, timeout_val);
+
+				crg_udc->feature_u2_enable = set_feat;
+			}
+			writel(tmp, &uccr->u3portpmsc);
+
+			break;
+		}
+		case USB_DEVICE_TEST_MODE: {
+			u32 u_pattern;
+
+			if (b_request != USB_REQ_SET_FEATURE ||
+			    crg_udc->gadget.speed != USB_SPEED_HIGH ||
+			    crg_udc->device_state < USB_STATE_DEFAULT ||
+			    (index & 0xff) || (index >> 8) == 0 ||
+			    (index >> 8) > 6)
+				goto set_feature_error;
+			u_pattern = index >> 8;
+			crg_udc->set_tm = u_pattern;
+			crg_udc->setup_fn_call_back = &set_test_mode_cmpl;
+
+			break;
+		}
+
+		default:
+			goto set_feature_error;
+		}
+
+	} else if ((request_type & (USB_RECIP_MASK | USB_TYPE_MASK)) ==
+		   (USB_RECIP_INTERFACE | USB_TYPE_STANDARD)) {
+		if (crg_udc->device_state != USB_STATE_CONFIGURED) {
+			dev_err(crg_udc->dev,
+				"interface u12 enable fail, usb state=%d\n",
+				crg_udc->device_state);
+			status = -EINVAL;
+			goto set_feature_error;
+		}
+
+		/* Suspend Option */
+		if (value != USB_INTRF_FUNC_SUSPEND) {
+			status = -EINVAL;
+			goto set_feature_error;
+		}
+
+		if (!(index & USB_INTR_FUNC_SUSPEND_OPT_MASK &
+		      USB_INTRF_FUNC_SUSPEND_LP)) {
+			status = -EINVAL;
+			goto set_feature_error;
+		}
+
+		if (index & USB_INTRF_FUNC_SUSPEND_RW)
+			dev_dbg(crg_udc->dev, "Interface En Remote Wakeup\n");
+		else
+			dev_dbg(crg_udc->dev, "Interface Dis RemoteWakeup\n");
+
+		/* Do not need to return status stage here
+		 * Pass to composite gadget driver to process
+		 * the request
+		 */
+		return false;
+	}
+
+	crg_udc->setup_status = STATUS_STAGE_XFER;
+	crg_udc_build_ep0_status(&crg_udc->udc_ep[0], true, status, NULL, 0, 0);
+	return true;
+
+set_feature_error:
+	crg_udc_set_ep0_halt(crg_udc);
+	return true;
+}
+
+static void setselrequest(struct crg_gadget_dev *crg_udc, u16 value, u16 index,
+			  u16 length)
+{
+	struct crg_udc_request *udc_req_ptr = crg_udc->status_req;
+	struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+	struct crg_sel_value *sel;
+	int ret;
+
+	if (!udc_req_ptr || !list_empty(&udc_req_ptr->queue) ||
+	    crg_udc->device_state == USB_STATE_DEFAULT || value || index ||
+	    length != sizeof(struct crg_sel_value)) {
+		crg_udc_set_ep0_halt(crg_udc);
+		return;
+	}
+
+	kfree(udc_req_ptr->usb_req.buf);
+
+	udc_req_ptr->usb_req.buf = kzalloc_obj(*sel, GFP_ATOMIC);
+	if (!udc_req_ptr->usb_req.buf) {
+		crg_udc_set_ep0_halt(crg_udc);
+		return;
+	}
+
+	sel = udc_req_ptr->usb_req.buf;
+	udc_req_ptr->usb_req.length = sizeof(*sel);
+	udc_req_ptr->usb_req.status = -EINPROGRESS;
+	udc_req_ptr->usb_req.actual = 0;
+	udc_req_ptr->usb_req.complete = set_sel_cmpl;
+	udc_req_ptr->map_dir = DMA_FROM_DEVICE;
+	ret = usb_gadget_map_request_by_dev(crg_udc->dev, &udc_req_ptr->usb_req,
+					    false);
+	if (ret) {
+		kfree(udc_req_ptr->usb_req.buf);
+		udc_req_ptr->usb_req.buf = NULL;
+		crg_udc_set_ep0_halt(crg_udc);
+		return;
+	}
+
+	crg_udc->setup_status = DATA_STAGE_RECV;
+
+	ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+	if (ret) {
+		crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr);
+		kfree(udc_req_ptr->usb_req.buf);
+		udc_req_ptr->usb_req.buf = NULL;
+		crg_udc_set_ep0_halt(crg_udc);
+		return;
+	}
+	list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+}
+
+int crg_udc_update_ep0_maxpacketsize(struct crg_gadget_dev *crg_udc)
+{
+	u16 maxpacketsize = 0;
+	struct crg_udc_ep *udc_ep0 = &crg_udc->udc_ep[0];
+	u32 param0;
+
+	if (crg_udc->gadget.speed >= USB_SPEED_SUPER)
+		maxpacketsize = 512;
+	else
+		maxpacketsize = 64;
+
+	param0 = FIELD_PREP(CRG_CMD1_0_MPS_MASK, maxpacketsize);
+	if (crg_udc_issue_command(crg_udc, CRG_CMD_UPDATE_EP0_CFG, param0, 0))
+		return -EIO;
+
+	crg_udc->ep0_desc.wMaxPacketSize = cpu_to_le16(maxpacketsize);
+	udc_ep0->usb_ep.maxpacket = maxpacketsize;
+
+	return 0;
+}
+
+void crg_udc_handle_setup_pkt(struct crg_gadget_dev *crg_udc,
+			      struct usb_ctrlrequest *setup_pkt, u8 setup_tag,
+			      unsigned long *flags)
+{
+	u16 w_value = le16_to_cpu(setup_pkt->wValue);
+	u16 w_index = le16_to_cpu(setup_pkt->wIndex);
+	u16 w_length = le16_to_cpu(setup_pkt->wLength);
+	struct usb_gadget_driver *driver;
+	u32 tmp;
+
+	trace_crg_udc_ctrl_req(setup_pkt);
+
+	/* EP0 come backs to running when new setup packet comes */
+	crg_udc->udc_ep[0].ep_state = EP_STATE_RUNNING;
+
+	crg_udc->setup_tag = setup_tag;
+	crg_udc->setup_status = SETUP_PKT_PROCESS_IN_PROGRESS;
+	crg_udc->setup_fn_call_back = NULL;
+
+	if ((setup_pkt->bRequestType & USB_TYPE_MASK) == USB_TYPE_STANDARD) {
+		switch (setup_pkt->bRequest) {
+		case USB_REQ_GET_STATUS:
+			if (setup_pkt->bRequestType !=
+				    (USB_DIR_IN | USB_RECIP_DEVICE |
+				     USB_TYPE_STANDARD) &&
+			    setup_pkt->bRequestType !=
+				    (USB_DIR_IN | USB_RECIP_INTERFACE |
+				     USB_TYPE_STANDARD) &&
+			    setup_pkt->bRequestType !=
+				    (USB_DIR_IN | USB_RECIP_ENDPOINT |
+				     USB_TYPE_STANDARD)) {
+				crg_udc_set_ep0_halt(crg_udc);
+				return;
+			}
+
+			getstatusrequest(crg_udc, setup_pkt->bRequestType,
+					 w_value, w_index, w_length);
+			return;
+		case USB_REQ_SET_ADDRESS:
+			if (setup_pkt->bRequestType !=
+			    (USB_DIR_OUT | USB_RECIP_DEVICE |
+			     USB_TYPE_STANDARD)) {
+				crg_udc_set_ep0_halt(crg_udc);
+				return;
+			}
+
+			setaddressrequest(crg_udc, w_value, w_index, w_length);
+			return;
+		case USB_REQ_SET_SEL:
+			if (setup_pkt->bRequestType !=
+			    (USB_DIR_OUT | USB_RECIP_DEVICE |
+			     USB_TYPE_STANDARD)) {
+				crg_udc_set_ep0_halt(crg_udc);
+				return;
+			}
+			crg_udc->setup_status = DATA_STAGE_RECV;
+			setselrequest(crg_udc, w_value, w_index, w_length);
+			return;
+		case USB_REQ_SET_ISOCH_DELAY:
+			if (setup_pkt->bRequestType !=
+				    (USB_DIR_OUT | USB_RECIP_DEVICE |
+				     USB_TYPE_STANDARD) ||
+			    w_index || w_length ||
+			    w_value >
+				    FIELD_MAX(
+					    CRG_U3DC_PTMCTRL_ISOCH_DELAY_MASK)) {
+				crg_udc_set_ep0_halt(crg_udc);
+				return;
+			}
+
+			tmp = readl(&crg_udc->uccr->ptmctrl);
+			FIELD_MODIFY(CRG_U3DC_PTMCTRL_ISOCH_DELAY_MASK, &tmp,
+				     w_value);
+			writel(tmp, &crg_udc->uccr->ptmctrl);
+			crg_udc->setup_status = STATUS_STAGE_XFER;
+			crg_udc_build_ep0_status(&crg_udc->udc_ep[0], true,
+						 -EINPROGRESS, NULL, 0, 0);
+			return;
+
+		case USB_REQ_CLEAR_FEATURE:
+		case USB_REQ_SET_FEATURE:
+			/* Need composite gadget driver
+			 * to process the function remote wakeup request
+			 */
+			if (setfeaturesrequest(crg_udc, setup_pkt->bRequestType,
+					       setup_pkt->bRequest, w_value,
+					       w_index, w_length, flags)) {
+				/* Get here if request has been processed.*/
+				return;
+			}
+			break;
+		case USB_REQ_SET_CONFIGURATION:
+			if (setup_pkt->bRequestType !=
+				    (USB_DIR_OUT | USB_RECIP_DEVICE |
+				     USB_TYPE_STANDARD) ||
+			    w_length || w_value > 0xff) {
+				crg_udc_set_ep0_halt(crg_udc);
+				return;
+			}
+			/* In theory we need to clear RUN bit before
+			 * status stage of deconfig request sent.
+			 * But seeing problem if we do it before all the
+			 * endpoints belonging to the configuration
+			 * disabled.
+			 */
+			if (crg_udc->device_state <= USB_STATE_DEFAULT) {
+				crg_udc_set_ep0_halt(crg_udc);
+				return;
+			}
+
+			break;
+		default:
+			dev_dbg(crg_udc->dev,
+				"USB_REQ default bRequest=%d, bRequestType=%d\n",
+				setup_pkt->bRequest, setup_pkt->bRequestType);
+		}
+	}
+
+	if (w_length) {
+		/* data phase from gadget like GET_CONFIGURATION
+		 * call the setup routine of gadget driver.
+		 * remember the request direction.
+		 */
+		crg_udc->setup_status = (setup_pkt->bRequestType & USB_DIR_IN) ?
+						DATA_STAGE_XFER :
+						DATA_STAGE_RECV;
+	}
+	driver = crg_udc->async_callbacks ? crg_udc->gadget_driver : NULL;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+
+	/*
+	 * Make sure any reset notification queued by
+	 * crg_udc_handle_port_status() has actually run before this SETUP
+	 * (which may be SET_CONFIGURATION) reaches the function driver.
+	 * Otherwise udc_reset_work's usb_gadget_udc_reset() could run after
+	 * the new configuration is already active and tear it back down.
+	 */
+	flush_work(&crg_udc->udc_reset_work);
+
+	if (!driver || driver->setup(&crg_udc->gadget, setup_pkt) < 0) {
+		spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+		crg_udc_set_ep0_halt(crg_udc);
+		return;
+	}
+	spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+	if (setup_pkt->bRequest == USB_REQ_SET_CONFIGURATION &&
+	    (setup_pkt->bRequestType & (USB_TYPE_MASK | USB_RECIP_MASK)) ==
+		    (USB_TYPE_STANDARD | USB_RECIP_DEVICE)) {
+		if (w_value) {
+			crg_udc->device_state = USB_STATE_CONFIGURED;
+			usb_gadget_set_state(&crg_udc->gadget,
+					     USB_STATE_CONFIGURED);
+		} else {
+			/* Endpoint disable may already have updated device_state. */
+			crg_udc->device_state = USB_STATE_ADDRESS;
+			usb_gadget_set_state(&crg_udc->gadget,
+					     USB_STATE_ADDRESS);
+		}
+	}
+}
+
+int crg_udc_init_ep0(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+	u32 cmd_param0;
+	u32 cmd_param1;
+
+	/* setup transfer ring */
+	if (!udc_ep_ptr->tran_ring_info.cpu_addr) {
+		dma_addr_t dma;
+		u32 ring_size = CRG_UDC_CONTROL_EP_TD_RING_SIZE;
+		void *vaddr;
+		size_t len;
+
+		len = ring_size * sizeof(struct crg_transfer_trb);
+		vaddr = dma_alloc_coherent(crg_udc->dev, len, &dma, GFP_KERNEL);
+		if (!vaddr)
+			return -ENOMEM;
+
+		udc_ep_ptr->tran_ring_info.cpu_addr = vaddr;
+		udc_ep_ptr->tran_ring_info.dma_addr = dma;
+		udc_ep_ptr->tran_ring_info.len = len;
+		udc_ep_ptr->first_trb = vaddr;
+		udc_ep_ptr->last_trb = udc_ep_ptr->first_trb + ring_size - 1;
+	}
+
+	memset(udc_ep_ptr->first_trb, 0, udc_ep_ptr->tran_ring_info.len);
+	/* Ensure that transfer ring is updated */
+	dma_wmb();
+	udc_ep_ptr->enq_pt = udc_ep_ptr->first_trb;
+	udc_ep_ptr->deq_pt = udc_ep_ptr->first_trb;
+	udc_ep_ptr->pcs = 1;
+	udc_ep_ptr->tran_ring_full = false;
+
+	crg_udc_setup_link_trb(udc_ep_ptr->last_trb, true,
+			       udc_ep_ptr->tran_ring_info.dma_addr);
+
+	/* context related ops */
+	cmd_param0 = (lower_32_bits(udc_ep_ptr->tran_ring_info.dma_addr) &
+		      CRG_CMD0_0_DQPTRLO_MASK) |
+		     FIELD_PREP(CRG_CMD0_0_DCS_MASK, udc_ep_ptr->pcs);
+	cmd_param1 = upper_32_bits(udc_ep_ptr->tran_ring_info.dma_addr);
+
+	if (crg_udc_issue_command(crg_udc, CRG_CMD_INIT_EP0, cmd_param0,
+				  cmd_param1))
+		return -EIO;
+
+	udc_ep_ptr->desc = &crg_udc->ep0_desc;
+	udc_ep_ptr->ep_state = EP_STATE_RUNNING;
+	udc_ep_ptr->pcs = 1;
+
+	return 0;
+}
+
+int crg_udc_queue_ctrl(struct crg_udc_ep *udc_ep_ptr,
+		       struct crg_udc_request *udc_req_ptr,
+		       u32 num_of_trbs_needed)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+	enum crg_ep_state ep_state;
+	struct crg_transfer_trb *enq_pt = udc_ep_ptr->enq_pt;
+	struct crg_transfer_trb *dq_pt = udc_ep_ptr->deq_pt;
+	struct usb_request *usb_req = &udc_req_ptr->usb_req;
+	struct crg_transfer_trb *p_trb;
+	u32 transfer_length;
+	bool ioc;
+	bool azp;
+	u8 dir = 0;
+	u8 setup_tag = crg_udc->setup_tag;
+
+	/* A new SETUP aborts the previous EP0 transaction. The completion
+	 * callback of the aborted request is allowed to run, but it must not
+	 * requeue the old request before the new SETUP has been handed to the
+	 * gadget driver.
+	 */
+	if (crg_udc->ep0_state == EP0_STATE_ABORTING)
+		return -ECONNRESET;
+
+	ep_state = crg_udc_get_ep_state(crg_udc, 0);
+	/* Need to queue the request even ep is paused or halted */
+	if (ep_state != EP_STATE_RUNNING) {
+		dev_dbg(crg_udc->dev, "EP State = 0x%x\n", ep_state);
+		return -EINVAL;
+	}
+	if (list_empty(&udc_ep_ptr->queue)) {
+		/* For control endpoint, we can handle one setup request at a
+		 * time. so if there are TD pending in the transfer ring.
+		 * wait for the sequence number error event. Then put the new
+		 * request to transfer ring
+		 */
+		if (enq_pt == dq_pt) {
+			u32 i;
+			bool need_zlp = false;
+			dma_addr_t trb_dma = usb_req->dma;
+			u64 remaining = usb_req->length;
+			/* Transfer ring is empty
+			 * setup data stage TRBs
+			 */
+			udc_req_ptr->first_trb = udc_ep_ptr->enq_pt;
+			if (crg_udc->setup_status == DATA_STAGE_XFER)
+				dir = 0;
+			else if (crg_udc->setup_status == DATA_STAGE_RECV)
+				dir = 1;
+			else
+				dev_dbg(crg_udc->dev,
+					"unexpected setup_status!%d\n",
+					crg_udc->setup_status);
+			if (udc_req_ptr->usb_req.zero == 1 &&
+			    udc_req_ptr->usb_req.length != 0 &&
+			    ((udc_req_ptr->usb_req.length %
+			      udc_ep_ptr->usb_ep.maxpacket) == 0))
+				need_zlp = true; /* zero length packet */
+			if (num_of_trbs_needed > 32)
+				return -EMSGSIZE;
+
+			if (num_of_trbs_needed >
+			    crg_udc_room_on_ring(
+				    crg_udc, CRG_UDC_CONTROL_EP_TD_RING_SIZE,
+				    udc_ep_ptr->first_trb, enq_pt, dq_pt))
+				return -ENOSPC;
+
+			u8 original_pcs = udc_ep_ptr->pcs;
+
+			for (i = 0; i < num_of_trbs_needed; i++) {
+				p_trb = enq_pt;
+				transfer_length = crg_udc_trb_chunk_len(
+					trb_dma, remaining);
+				ioc = (i == num_of_trbs_needed - 1);
+				azp = ioc && need_zlp;
+				crg_udc_setup_datastage_trb(
+					p_trb, trb_dma,
+					i == 0 ? (original_pcs ^ 1) :
+						 udc_ep_ptr->pcs,
+					transfer_length,
+					num_of_trbs_needed - i - 1, ioc, azp,
+					dir, setup_tag);
+				udc_req_ptr->all_trbs_queued = 1;
+				trb_dma += transfer_length;
+				remaining -= transfer_length;
+				enq_pt++;
+				if (FIELD_GET(TRB_TYPE_MASK,
+					      le32_to_cpu(enq_pt->dw3)) ==
+				    TRB_TYPE_LINK) {
+					if (FIELD_GET(
+						    TRB_LINK_TOGGLE_CYCLE_MASK,
+						    le32_to_cpu(enq_pt->dw3))) {
+						dma_wmb();
+						le32p_replace_bits(
+							&enq_pt->dw3,
+							udc_ep_ptr->pcs,
+							TRB_CYCLE_BIT_MASK);
+						udc_ep_ptr->pcs ^= 0x1;
+					}
+					enq_pt = udc_ep_ptr->first_trb;
+				}
+			}
+			dma_wmb();
+			le32p_replace_bits(&udc_req_ptr->first_trb->dw3,
+					   original_pcs, TRB_CYCLE_BIT_MASK);
+			udc_ep_ptr->enq_pt = enq_pt;
+			crg_udc_knock_doorbell(crg_udc, 0);
+			if (udc_ep_ptr->enq_pt == udc_ep_ptr->first_trb)
+				udc_req_ptr->last_trb =
+					udc_ep_ptr->last_trb - 1;
+			else
+				udc_req_ptr->last_trb = udc_ep_ptr->enq_pt - 1;
+		} else {
+			/*
+			 * The software queue is empty but the transfer ring
+			 * still holds pending TDs from a previous control
+			 * transfer (the host issued a new setup before the
+			 * old one drained). We process one setup at a time,
+			 * so we intentionally do not queue TRBs here.
+			 * Returning success leaves this request on the
+			 * endpoint queue; the stale TDs and this request are
+			 * flushed and completed (with -EINVAL) by the
+			 * sequence-number-error / setup-tag-mismatch event in
+			 * crg_udc_handle_xfer_event().
+			 */
+			dev_err(crg_udc->dev,
+				"ep0 ring not idle on new setup\n");
+		}
+	} else {
+		dev_err(crg_udc->dev, "udc_ep_ptr->queue not empty\n");
+		/* New setup packet came
+		 * Drop this request.
+		 */
+		return -EINVAL;
+	}
+	return 0;
+}

-- 
2.34.1

[PATCH v3 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations

[Petar Stepanovic]

This patch implements the gadget and endpoint operations for the
Axiado AX3000/AX3005 UDC driver.

Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
 drivers/usb/gadget/udc/crg/crg-udc-gadget.c | 2299 +++++++++++++++++++++++++++
 drivers/usb/gadget/udc/crg/crg-udc-trace.c  |    9 +
 2 files changed, 2308 insertions(+)

diff --git a/drivers/usb/gadget/udc/crg/crg-udc-gadget.c b/drivers/usb/gadget/udc/crg/crg-udc-gadget.c
new file mode 100644
index 000000000000..a330e857a950
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-gadget.c
@@ -0,0 +1,2299 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/iopoll.h>
+#include <linux/scatterlist.h>
+
+#include "crg-udc.h"
+#include "crg-udc-trace.h"
+
+/* Forward declarations for functions used before their definition. */
+static int crg_udc_epcx_setup(struct crg_udc_ep *udc_ep);
+static int crg_udc_epcx_update_dqptr(struct crg_udc_ep *udc_ep);
+static int crg_udc_queue_trbs(struct crg_udc_ep *udc_ep_ptr,
+			      struct crg_udc_request *udc_req_ptr, bool b_isoc,
+			      u32 xfer_ring_size, u32 num_trbs_needed,
+			      u64 buffer_length);
+static int crg_udc_reinit(struct crg_gadget_dev *crg_udc);
+static bool crg_udc_sg_advance(struct crg_udc_request *udc_req, u32 bytes);
+static bool crg_udc_sg_current(struct crg_udc_request *udc_req, dma_addr_t *dma,
+			       u32 *len);
+static int crg_udc_sg_init(struct crg_udc_request *udc_req);
+static int crg_udc_validate_ep_desc(struct crg_udc_ep *udc_ep,
+				    const struct usb_endpoint_descriptor *desc);
+static bool is_request_dequeued(struct crg_gadget_dev *crg_udc,
+				struct crg_udc_ep *udc_ep,
+				struct crg_event_trb *event);
+static int set_ep_halt(struct crg_gadget_dev *crg_udc, int dci,
+		       unsigned long *flags);
+
+static int crg_gadget_get_frame(struct usb_gadget *g)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+
+	return FIELD_GET(CRG_U3DC_MFINDEX_FRAME_MASK,
+			 readl(&crg_udc->uccr->mfindex));
+}
+
+static int crg_gadget_pullup(struct usb_gadget *g, int is_on)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+	unsigned long flags;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	crg_udc->pullup_on = !!is_on;
+	crg_udc_set_pullup(crg_udc, crg_udc->pullup_on);
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	trace_crg_udc_pullup(crg_udc->pullup_on);
+	return 0;
+}
+
+static int crg_gadget_set_remote_wakeup(struct usb_gadget *g, int set)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+	unsigned long flags;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	crg_udc->u2_rwe = !!set;
+	g->wakeup_armed = !!set;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	return 0;
+}
+
+static int crg_gadget_set_selfpowered(struct usb_gadget *g, int set)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+	unsigned long flags;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	g->is_selfpowered = !!set;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	return 0;
+}
+
+static void crg_udc_set_speed(struct usb_gadget *gadget,
+			      enum usb_device_speed speed)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(gadget);
+
+	crg_udc->max_speed = speed;
+}
+
+static int crg_gadget_start(struct usb_gadget *g,
+			    struct usb_gadget_driver *driver)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+	int ret;
+
+	/* The UDC core calls udc_set_speed() before udc_start(). */
+	ret = crg_udc_program_speed(crg_udc);
+	if (ret)
+		return ret;
+
+	crg_udc->gadget_driver = driver;
+	crg_udc_set_vbus_detect(crg_udc, true);
+	return 0;
+}
+
+static int crg_gadget_stop(struct usb_gadget *g)
+{
+	struct crg_gadget_dev *crg_udc;
+	unsigned long flags = 0;
+
+	crg_udc = gadget_to_udc(g);
+
+	/*
+	 * Clear gadget_driver first so crg_udc_handle_port_status() can no
+	 * longer schedule udc_reset_work (it only does so while
+	 * gadget_driver is set), then cancel_work_sync() is guaranteed to
+	 * be the last word: no new reset notification can be queued after
+	 * it returns.
+	 */
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	crg_udc->gadget_driver = NULL;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	cancel_work_sync(&crg_udc->udc_reset_work);
+
+	crg_udc->device_state = USB_STATE_ATTACHED;
+	crg_udc_set_vbus_detect(crg_udc, false);
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+
+	/*
+	 * crg_udc_reset() completes any queued requests via
+	 * crg_udc_ep_nuke() -> crg_udc_req_done(). crg_udc_req_done() drops udc_lock
+	 * (IRQ-safe) around each ->complete() callback and re-acquires it
+	 * afterwards, so gadget completion callbacks are never invoked with
+	 * the spinlock held. flags is threaded through for that purpose.
+	 */
+	if (crg_udc_reset(crg_udc, &flags)) {
+		dev_err(crg_udc->dev, "controller reset failed during stop\n");
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EIO;
+	}
+	if (crg_udc_reset_data_struct(crg_udc)) {
+		dev_err(crg_udc->dev,
+			"controller data reset failed during stop\n");
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EIO;
+	}
+
+	crg_udc->connected = 0;
+	crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+
+	if (crg_udc_init_ep0(crg_udc)) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EIO;
+	}
+
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	return 0;
+}
+
+static int crg_gadget_wakeup(struct usb_gadget *g)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	unsigned long flags;
+	u32 portsc;
+	int ret = 0;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	if (!crg_udc->connected || g->speed > USB_SPEED_HIGH) {
+		ret = -EOPNOTSUPP;
+		goto out;
+	}
+	if (!crg_udc->u2_rwe) {
+		ret = -EACCES;
+		goto out;
+	}
+
+	portsc = readl(&uccr->portsc);
+	if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, portsc) !=
+	    CRG_U3DC_PORTSC_PLS_U3) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	portsc &= ~(CRG_U3DC_PORTSC_W1C_MASK | CRG_U3DC_PORTSC_PLS_MASK);
+	portsc |= CRG_U3DC_PORTSC_LWS;
+	writel(portsc, &uccr->portsc);
+out:
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	return ret;
+}
+
+static void crg_gadget_async_callbacks(struct usb_gadget *g, bool enable)
+{
+	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+	unsigned long flags;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	crg_udc->async_callbacks = enable;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+}
+
+struct usb_request *crg_udc_alloc_request(struct usb_ep *_ep, gfp_t gfp_flags)
+{
+	struct crg_udc_request *udc_req_ptr;
+
+	udc_req_ptr = kzalloc_obj(*udc_req_ptr, gfp_flags);
+
+	if (!udc_req_ptr)
+		return NULL;
+
+	udc_req_ptr->usb_req.dma = DMA_MAPPING_ERROR;
+	INIT_LIST_HEAD(&udc_req_ptr->queue);
+
+	trace_crg_udc_alloc_request(
+		container_of(_ep, struct crg_udc_ep, usb_ep), udc_req_ptr);
+
+	return &udc_req_ptr->usb_req;
+}
+
+int crg_udc_build_td(struct crg_udc_ep *udc_ep_ptr,
+		     struct crg_udc_request *udc_req_ptr)
+{
+	int status = 0;
+	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+	u32 num_trbs_needed;
+	u64 buffer_length;
+
+	if (udc_req_ptr->trbs_needed) {
+		/* If this is called to complete pending TRB transfers
+		 * of previous Request
+		 */
+		buffer_length = udc_req_ptr->buff_len_left;
+		num_trbs_needed = udc_req_ptr->trbs_needed;
+	} else {
+		buffer_length = udc_req_ptr->usb_req.length;
+		if (udc_req_ptr->usb_req.num_mapped_sgs)
+			num_trbs_needed = crg_udc_count_trbs_sg(udc_req_ptr);
+		else
+			num_trbs_needed = crg_udc_count_trbs_linear(
+				udc_req_ptr->usb_req.dma, buffer_length);
+		if (!num_trbs_needed && buffer_length)
+			return -EINVAL;
+		if (!num_trbs_needed)
+			num_trbs_needed = 1;
+	}
+
+	if (usb_endpoint_xfer_control(udc_ep_ptr->desc))
+		status = crg_udc_queue_ctrl(udc_ep_ptr, udc_req_ptr,
+					    num_trbs_needed);
+	else if (usb_endpoint_xfer_isoc(udc_ep_ptr->desc))
+		status = crg_udc_queue_trbs(udc_ep_ptr, udc_req_ptr, true,
+					    CRG_UDC_ISOC_EP_TD_RING_SIZE,
+					    num_trbs_needed, buffer_length);
+	else if (usb_endpoint_xfer_bulk(udc_ep_ptr->desc))
+		status = crg_udc_queue_trbs(udc_ep_ptr, udc_req_ptr, false,
+					    CRG_UDC_BULK_EP_TD_RING_SIZE,
+					    num_trbs_needed, buffer_length);
+	else
+		/* interrupt endpoint */
+		status = crg_udc_queue_trbs(udc_ep_ptr, udc_req_ptr, false,
+					    CRG_UDC_INT_EP_TD_RING_SIZE,
+					    num_trbs_needed, buffer_length);
+
+	if (!status && !usb_endpoint_xfer_control(udc_ep_ptr->desc))
+		crg_udc_knock_doorbell(crg_udc, udc_ep_ptr->dci);
+
+	return status;
+}
+
+void crg_udc_clear_portpm(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 tmp;
+
+	tmp = readl(&uccr->u3portpmsc);
+
+	tmp &= (~CRG_U3DC_U3PORTPM_U1IEN);
+	FIELD_MODIFY(CRG_U3DC_U3PORTPM_U1TMOUT_MASK, &tmp, 0);
+	tmp &= (~CRG_U3DC_U3PORTPM_U2IEN);
+	FIELD_MODIFY(CRG_U3DC_U3PORTPM_U2TMOUT_MASK, &tmp, 0);
+
+	writel(tmp, &uccr->u3portpmsc);
+
+	crg_udc->feature_u1_enable = 0;
+	crg_udc->feature_u2_enable = 0;
+}
+
+static int crg_udc_ep_dequeue(struct usb_ep *_ep, struct usb_request *_req)
+{
+	struct crg_udc_ep *udc_ep_ptr;
+	struct crg_gadget_dev *crg_udc;
+	struct crg_uccr __iomem *uccr;
+	struct crg_udc_request *udc_req = NULL;
+	struct crg_udc_request *iter;
+	struct crg_ep_cx *p_ep_cx;
+	struct crg_transfer_trb *pause_pt;
+	enum crg_ep_state old_ep_state;
+	u32 tmp, deq_pt_lo, deq_pt_hi;
+	dma_addr_t dq_pt_addr;
+	u8 dci;
+	bool found = false;
+	unsigned long flags;
+
+	if (!_ep || !_req)
+		return -EINVAL;
+
+	udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+	crg_udc = udc_ep_ptr->crg_udc;
+	uccr = crg_udc->uccr;
+	dci = udc_ep_ptr->dci;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	old_ep_state = udc_ep_ptr->ep_state;
+
+	list_for_each_entry(iter, &udc_ep_ptr->queue, queue) {
+		if (&iter->usb_req == _req) {
+			udc_req = iter;
+			found = true;
+			break;
+		}
+	}
+
+	if (!found) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EINVAL;
+	}
+
+	if (!udc_req->first_trb) {
+		crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return 0;
+	}
+
+	if (dci == 0) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EINVAL;
+	}
+
+	if (old_ep_state == EP_STATE_RUNNING) {
+		int ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP,
+						BIT(dci), 0);
+
+		if (ret) {
+			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+			return ret;
+		}
+		if (readl_poll_timeout_atomic(&uccr->ep_running, tmp,
+					      !(tmp & BIT(dci)), 10, 10000)) {
+			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+			return -ETIMEDOUT;
+		}
+		udc_ep_ptr->ep_state = EP_STATE_STOPPED;
+	}
+
+	p_ep_cx = &crg_udc->p_epcx[dci - 2];
+	deq_pt_lo = le32_to_cpu(p_ep_cx->dw2) & EP_CX_TR_DQPT_LO_MASK;
+	deq_pt_hi = le32_to_cpu(p_ep_cx->dw3);
+	dq_pt_addr = (dma_addr_t)deq_pt_lo | ((dma_addr_t)deq_pt_hi << 32);
+	pause_pt = crg_udc_tran_trb_dma_to_virt(udc_ep_ptr, dq_pt_addr);
+	if (!pause_pt) {
+		if (old_ep_state == EP_STATE_RUNNING) {
+			udc_ep_ptr->ep_state = EP_STATE_RUNNING;
+			crg_udc_knock_doorbell(crg_udc, dci);
+		}
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EIO;
+	}
+
+	if (crg_udc_is_pointer_less_than(pause_pt, udc_req->first_trb,
+					 udc_ep_ptr)) {
+		crg_udc_squeeze_xfer_ring(udc_ep_ptr, udc_req);
+		crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+	} else if (udc_req->last_trb &&
+		   crg_udc_is_pointer_less_than(udc_req->last_trb, pause_pt,
+						udc_ep_ptr)) {
+		crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+	} else {
+		struct crg_udc_request *next_req = NULL;
+
+		udc_req->usb_req.actual =
+			crg_udc_actual_before(udc_ep_ptr, udc_req, pause_pt);
+
+		if (!list_is_last(&udc_req->queue, &udc_ep_ptr->queue))
+			next_req = list_entry(udc_req->queue.next,
+					      struct crg_udc_request, queue);
+
+		if (next_req && next_req->first_trb)
+			udc_ep_ptr->deq_pt = next_req->first_trb;
+		else
+			udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
+
+		if (crg_udc_epcx_update_dqptr(udc_ep_ptr)) {
+			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+			return -EIO;
+		}
+		crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+	}
+
+	if (old_ep_state == EP_STATE_RUNNING) {
+		udc_ep_ptr->ep_state = EP_STATE_RUNNING;
+		crg_udc_knock_doorbell(crg_udc, dci);
+	}
+
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	return 0;
+}
+
+/* endpoint operations */
+
+static int crg_udc_ep_disable(struct usb_ep *ep)
+{
+	struct crg_udc_ep *udc_ep;
+	struct crg_gadget_dev *crg_udc;
+	struct crg_ep_cx *p_ep_cx;
+	enum crg_ep_state ep_state;
+	struct crg_uccr __iomem *uccr;
+	unsigned long flags = 0;
+
+	if (!ep)
+		return -EINVAL;
+
+	udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+	crg_udc = udc_ep->crg_udc;
+
+	if (udc_ep->dci == 0)
+		return 0;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+
+	uccr = crg_udc->uccr;
+	p_ep_cx = &crg_udc->p_epcx[udc_ep->dci - 2];
+
+	ep_state = crg_udc_get_ep_state(crg_udc, udc_ep->dci);
+	if (ep_state == EP_STATE_DISABLED) {
+		/* get here if ep is already disabled */
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EINVAL;
+	}
+
+	/* Maybe we need to halt ep before ep disable */
+
+	/*
+	 * ep_enable is a write-1-to-disable register: endpoints are enabled
+	 * via the CONFIG_EP command, but writing a set bit here disables the
+	 * corresponding endpoint.
+	 */
+	writel(BIT(udc_ep->dci), &uccr->ep_enable);
+
+	udc_ep->desc = NULL;
+	udc_ep->comp_desc = NULL;
+
+	/* clean up the request queue */
+	crg_udc_ep_nuke(udc_ep, -ESHUTDOWN);
+
+	/* decrement ep counters */
+	crg_udc->num_enabled_eps--;
+	/* clean up the endpoint context */
+	memset(p_ep_cx, 0, sizeof(struct crg_ep_cx));
+
+	/* If device state was changed to default by port
+	 * reset, should not overwrite it again
+	 */
+	if (crg_udc->num_enabled_eps == 0 &&
+	    crg_udc->device_state == USB_STATE_CONFIGURED) {
+		crg_udc->device_state = USB_STATE_ADDRESS;
+	}
+
+	udc_ep->ep_state = EP_STATE_DISABLED;
+
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	return 0;
+}
+
+static int crg_udc_ep_enable(struct usb_ep *ep,
+			     const struct usb_endpoint_descriptor *desc)
+{
+	struct crg_udc_ep *udc_ep;
+	struct crg_gadget_dev *crg_udc;
+	u32 param0;
+	int status;
+	unsigned long flags = 0;
+
+	if (!ep || !desc || desc->bDescriptorType != USB_DT_ENDPOINT)
+		return -EINVAL;
+	udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+	/* ep0 is always running */
+	if (udc_ep->dci == 0)
+		return 0;
+	crg_udc = udc_ep->crg_udc;
+	if (!crg_udc->gadget_driver)
+		return -ESHUTDOWN;
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	/* crg ep context start from ep1 */
+	if (crg_udc_get_ep_state(crg_udc, udc_ep->dci) != EP_STATE_DISABLED) {
+		int ret;
+
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		ret = crg_udc_ep_disable(ep);
+		if (ret)
+			return ret;
+		spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	}
+	udc_ep->desc = desc;
+	udc_ep->comp_desc = ep->comp_desc;
+	status = crg_udc_validate_ep_desc(udc_ep, desc);
+	if (status) {
+		udc_ep->desc = NULL;
+		udc_ep->comp_desc = NULL;
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return status;
+	}
+	/* setup endpoint context for regular endpoint
+	 * the endpoint context for control endpoint has been
+	 * set up in probe function
+	 */
+	if (udc_ep->dci) {
+		u32 ring_size = 0;
+		size_t len;
+
+		if (usb_endpoint_xfer_bulk(desc))
+			ring_size = CRG_UDC_BULK_EP_TD_RING_SIZE;
+		else if (usb_endpoint_xfer_isoc(desc))
+			ring_size = CRG_UDC_ISOC_EP_TD_RING_SIZE;
+		else if (usb_endpoint_xfer_int(desc))
+			ring_size = CRG_UDC_INT_EP_TD_RING_SIZE;
+
+		if (!ring_size) {
+			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+			return -EINVAL;
+		}
+		len = ring_size * sizeof(struct crg_transfer_trb);
+
+		if (udc_ep->tran_ring_info.cpu_addr &&
+		    udc_ep->tran_ring_info.len != len) {
+			void *old_vaddr = udc_ep->tran_ring_info.cpu_addr;
+			dma_addr_t old_dma = udc_ep->tran_ring_info.dma_addr;
+			size_t old_len = udc_ep->tran_ring_info.len;
+
+			udc_ep->tran_ring_info.cpu_addr = NULL;
+			udc_ep->enq_pt = NULL;
+			udc_ep->deq_pt = NULL;
+
+			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+			dma_free_coherent(crg_udc->dev, old_len, old_vaddr,
+					  old_dma);
+			spin_lock_irqsave(&crg_udc->udc_lock, flags);
+
+			if (udc_ep->desc != desc ||
+			    udc_ep->ep_state != EP_STATE_DISABLED) {
+				spin_unlock_irqrestore(&crg_udc->udc_lock,
+						       flags);
+				return -EAGAIN;
+			}
+		}
+
+		/* setup transfer ring */
+		if (!udc_ep->tran_ring_info.cpu_addr) {
+			dma_addr_t dma;
+			void *vaddr;
+
+			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+			vaddr = dma_alloc_coherent(crg_udc->dev, len, &dma,
+						   GFP_KERNEL);
+			if (!vaddr)
+				return -ENOMEM;
+			spin_lock_irqsave(&crg_udc->udc_lock, flags);
+			if (udc_ep->desc != desc ||
+			    udc_ep->ep_state != EP_STATE_DISABLED) {
+				spin_unlock_irqrestore(&crg_udc->udc_lock,
+						       flags);
+				dma_free_coherent(crg_udc->dev, len, vaddr,
+						  dma);
+				return -EAGAIN;
+			}
+			udc_ep->tran_ring_info.cpu_addr = vaddr;
+			udc_ep->tran_ring_info.dma_addr = dma;
+			udc_ep->tran_ring_info.len = len;
+			udc_ep->first_trb = vaddr;
+			udc_ep->last_trb = udc_ep->first_trb + ring_size - 1;
+		}
+		memset(udc_ep->first_trb, 0, udc_ep->tran_ring_info.len);
+		/* Ensure that transfer ring is updated */
+		dma_wmb();
+		crg_udc_setup_link_trb(udc_ep->last_trb, true,
+				       udc_ep->tran_ring_info.dma_addr);
+		udc_ep->enq_pt = udc_ep->first_trb;
+		udc_ep->deq_pt = udc_ep->first_trb;
+		udc_ep->pcs = 1;
+		udc_ep->tran_ring_full = false;
+		crg_udc->num_enabled_eps++;
+		status = crg_udc_epcx_setup(udc_ep);
+		if (status)
+			goto err_ep_enable;
+	}
+	param0 = BIT(udc_ep->dci);
+	status = crg_udc_issue_command(crg_udc, CRG_CMD_CONFIG_EP, param0, 0);
+	if (status)
+		goto err_ep_enable;
+	if (crg_udc->device_state == USB_STATE_ADDRESS)
+		crg_udc->device_state = USB_STATE_CONFIGURED;
+	udc_ep->wedge = false;
+	udc_ep->usb_ep.enabled = 1;
+	udc_ep->ep_state = EP_STATE_RUNNING;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	return 0;
+
+err_ep_enable:
+	if (udc_ep->dci && crg_udc->num_enabled_eps)
+		crg_udc->num_enabled_eps--;
+	udc_ep->desc = NULL;
+	udc_ep->comp_desc = NULL;
+	udc_ep->ep_state = EP_STATE_DISABLED;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	if (udc_ep->tran_ring_info.cpu_addr) {
+		dma_free_coherent(crg_udc->dev, udc_ep->tran_ring_info.len,
+				  udc_ep->tran_ring_info.cpu_addr,
+				  udc_ep->tran_ring_info.dma_addr);
+		udc_ep->tran_ring_info.cpu_addr = NULL;
+		udc_ep->tran_ring_info.dma_addr = DMA_MAPPING_ERROR;
+		udc_ep->tran_ring_info.len = 0;
+		udc_ep->first_trb = NULL;
+		udc_ep->last_trb = NULL;
+	}
+	return status;
+}
+
+void crg_udc_ep_nuke(struct crg_udc_ep *udc_ep, int status)
+{
+	struct crg_udc_request *req;
+	struct list_head nuke_list;
+
+	list_replace_init(&udc_ep->queue, &nuke_list);
+
+	while (!list_empty(&nuke_list)) {
+		req = list_first_entry(&nuke_list, struct crg_udc_request,
+				       queue);
+
+		crg_udc_req_done(udc_ep, req, status);
+	}
+}
+
+static int crg_udc_ep_queue(struct usb_ep *_ep, struct usb_request *_req,
+			    gfp_t gfp_flags)
+{
+	struct crg_udc_request *udc_req_ptr;
+	struct crg_udc_ep *udc_ep_ptr;
+	struct crg_gadget_dev *crg_udc;
+	enum dma_data_direction dma_data_dir;
+	const struct usb_endpoint_descriptor *desc;
+	unsigned long flags;
+	int status;
+	bool is_sg;
+
+	if (!_ep || !_req)
+		return -EINVAL;
+
+	udc_req_ptr = container_of(_req, struct crg_udc_request, usb_req);
+	udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+	crg_udc = udc_ep_ptr->crg_udc;
+	is_sg = _req->num_sgs != 0 && _req->length != 0;
+
+	if (is_sg && !_req->sg)
+		return -EINVAL;
+	if (_req->stream_id)
+		return -EOPNOTSUPP;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	/*
+	 * A completion callback for an aborted EP0 transfer may requeue the
+	 * same request. Reject EP0 requests while the previous control
+	 * transfer is being aborted or EP0 is in an error state. EP0 becomes
+	 * available again only after the new SETUP has been accepted and
+	 * the EP0 transfer ring has been successfully reinitialized.
+	 */
+	if (udc_ep_ptr->dci == 0 && crg_udc->ep0_state != EP0_STATE_IDLE) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -ECONNRESET;
+	}
+	if (!udc_ep_ptr->first_trb || !_req->complete ||
+	    (!is_sg && !_req->buf) || !list_empty(&udc_req_ptr->queue)) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EINVAL;
+	}
+	if (crg_udc->resetting) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -ESHUTDOWN;
+	}
+	if (!udc_ep_ptr->desc) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EINVAL;
+	}
+	desc = udc_ep_ptr->desc;
+
+	if (usb_endpoint_xfer_control(udc_ep_ptr->desc) && is_sg) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EOPNOTSUPP;
+	}
+
+	if (usb_endpoint_xfer_control(udc_ep_ptr->desc) && !_req->length) {
+		if (crg_udc->setup_status == DATA_STAGE_XFER)
+			crg_udc->setup_status = STATUS_STAGE_RECV;
+		else
+			crg_udc->setup_status = STATUS_STAGE_XFER;
+		crg_udc_build_ep0_status(&crg_udc->udc_ep[0], false,
+					 -EINPROGRESS, udc_req_ptr, 0, 0);
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return 0;
+	}
+
+	if (usb_endpoint_xfer_control(udc_ep_ptr->desc)) {
+		if (crg_udc->setup_status == DATA_STAGE_XFER ||
+		    crg_udc->setup_status == STATUS_STAGE_XFER) {
+			dma_data_dir = DMA_TO_DEVICE;
+		} else if (crg_udc->setup_status == DATA_STAGE_RECV ||
+			   crg_udc->setup_status == STATUS_STAGE_RECV) {
+			dma_data_dir = DMA_FROM_DEVICE;
+		} else {
+			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+			return -EINVAL;
+		}
+	} else {
+		dma_data_dir = usb_endpoint_dir_in(udc_ep_ptr->desc) ?
+				       DMA_TO_DEVICE :
+				       DMA_FROM_DEVICE;
+	}
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	crg_udc_clear_req_container(udc_req_ptr);
+	udc_req_ptr->mapped = false;
+
+	if (_req->length) {
+		udc_req_ptr->map_dir = dma_data_dir;
+		status = usb_gadget_map_request_by_dev(
+			crg_udc->dev, _req, dma_data_dir == DMA_TO_DEVICE);
+		if (status)
+			goto err_unmap;
+
+		if (is_sg) {
+			if (!_req->num_mapped_sgs ||
+			    crg_udc_sg_init(udc_req_ptr)) {
+				status = -EIO;
+				goto err_unmap;
+			}
+		} else if (_req->dma_mapped) {
+			udc_req_ptr->mapped = true;
+		}
+	}
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	if (udc_ep_ptr->desc != desc || !udc_req_ptr->usb_req.complete ||
+	    !list_empty(&udc_req_ptr->queue) || crg_udc->resetting) {
+		status = -ESHUTDOWN;
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr);
+		return status;
+	}
+
+	_req->status = -EINPROGRESS;
+	_req->actual = 0;
+
+	if (udc_ep_ptr->tran_ring_full)
+		status = 0;
+	else
+		status = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+
+	if (!status)
+		list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	if (status)
+		crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr);
+	return status;
+
+err_unmap:
+	crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr);
+	return status;
+}
+
+static int crg_udc_ep_set_halt(struct usb_ep *_ep, int value)
+{
+	struct crg_udc_ep *udc_ep_ptr;
+	int status;
+	unsigned long flags = 0;
+	struct crg_gadget_dev *crg_udc;
+
+	if (!_ep)
+		return -EINVAL;
+
+	udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+	crg_udc = udc_ep_ptr->crg_udc;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	if (!udc_ep_ptr->desc) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EINVAL;
+	}
+	if (value && usb_endpoint_dir_in(udc_ep_ptr->desc) &&
+	    !list_empty(&udc_ep_ptr->queue)) {
+		dev_err(crg_udc->dev, "set_halt: list not empty\n");
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EAGAIN;
+	}
+
+	status = crg_udc_ep_halt(udc_ep_ptr, value, 1, &flags);
+
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	return status;
+}
+
+static int crg_udc_ep_set_wedge(struct usb_ep *_ep)
+{
+	struct crg_udc_ep *udc_ep_ptr;
+	int status;
+	unsigned long flags = 0;
+	struct crg_gadget_dev *crg_udc;
+
+	if (!_ep)
+		return -EINVAL;
+
+	udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+	crg_udc = udc_ep_ptr->crg_udc;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+
+	if (!udc_ep_ptr->desc) {
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return -EINVAL;
+	}
+
+	udc_ep_ptr->wedge = 1;
+
+	status = crg_udc_ep_halt(udc_ep_ptr, 1, 1, &flags);
+
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	return status;
+}
+
+static int crg_udc_epcx_setup(struct crg_udc_ep *udc_ep)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+	const struct usb_endpoint_descriptor *desc = udc_ep->desc;
+	const struct usb_ss_ep_comp_descriptor *comp_desc = udc_ep->comp_desc;
+	u8 dci = udc_ep->dci;
+	struct crg_ep_cx *epcx = &crg_udc->p_epcx[dci - 2];
+	enum crg_ep_type ep_type;
+	u16 maxburst = 0;
+	u8 maxstreams = 0;
+	u16 maxsize;
+	u32 dw;
+
+	/* corigine gadget dir should be opposite to host dir */
+	if (usb_endpoint_dir_out(desc))
+		ep_type = usb_endpoint_type(desc) + EP_TYPE_INVALID2;
+	else
+		ep_type = usb_endpoint_type(desc);
+
+	maxsize = usb_endpoint_maxp(desc) & 0x07ff;
+
+	if (crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+		if (!comp_desc)
+			return -EINVAL;
+		maxburst = comp_desc->bMaxBurst;
+
+		if (usb_endpoint_xfer_bulk(udc_ep->desc))
+			maxstreams = comp_desc->bmAttributes & 0x1f;
+
+	} else if ((crg_udc->gadget.speed == USB_SPEED_HIGH ||
+		    crg_udc->gadget.speed == USB_SPEED_FULL) &&
+		   (usb_endpoint_xfer_int(udc_ep->desc) ||
+		    usb_endpoint_xfer_isoc(udc_ep->desc))) {
+		if (crg_udc->gadget.speed == USB_SPEED_HIGH)
+			maxburst = (usb_endpoint_maxp(desc) >> 11) & 0x3;
+		if (maxburst == 0x3) {
+			dev_err(crg_udc->dev, "invalid maxburst\n");
+			maxburst = 0x2;
+		}
+	}
+
+	/* fill ep_dw0 */
+	dw = 0;
+	FIELD_MODIFY(EP_CX_LOGICAL_EP_NUM_MASK, &dw, udc_ep->dci / 2);
+	FIELD_MODIFY(EP_CX_INTERVAL_MASK, &dw, desc->bInterval);
+	if (maxstreams) {
+		dev_err(crg_udc->dev, "maxstream=%d is not expected\n",
+			maxstreams);
+	}
+	epcx->dw0 = cpu_to_le32(dw);
+
+	/* fill ep_dw1 */
+	dw = 0;
+	FIELD_MODIFY(EP_CX_EP_TYPE_MASK, &dw, ep_type);
+	FIELD_MODIFY(EP_CX_MAX_PACKET_SIZE_MASK, &dw, maxsize);
+	FIELD_MODIFY(EP_CX_MAX_BURST_SIZE_MASK, &dw, maxburst);
+	epcx->dw1 = cpu_to_le32(dw);
+
+	/* fill ep_dw2 */
+	dw = lower_32_bits(udc_ep->tran_ring_info.dma_addr) &
+	     EP_CX_TR_DQPT_LO_MASK;
+	FIELD_MODIFY(EP_CX_DEQ_CYC_STATE_MASK, &dw, udc_ep->pcs);
+	epcx->dw2 = cpu_to_le32(dw);
+
+	/* fill ep_dw3 */
+	dw = upper_32_bits(udc_ep->tran_ring_info.dma_addr);
+	epcx->dw3 = cpu_to_le32(dw);
+	/* Ensure that epcx is updated */
+	dma_wmb();
+	return 0;
+}
+
+static int crg_udc_epcx_update_dqptr(struct crg_udc_ep *udc_ep)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+	u8 dci = udc_ep->dci;
+	struct crg_ep_cx *epcx;
+	u32 dw;
+	dma_addr_t dqptaddr;
+	u32 cmd_param0;
+
+	if (dci < 2 || dci >= CRG_NUM_EP_CX) {
+		dev_err(crg_udc->dev, "invalid DCI %u for dequeue pointer\n", dci);
+		return -EINVAL;
+	}
+
+	epcx = &crg_udc->p_epcx[dci - 2];
+	dqptaddr = crg_udc_tran_trb_virt_to_dma(udc_ep, udc_ep->deq_pt);
+
+	if (dqptaddr == DMA_MAPPING_ERROR)
+		return -EINVAL;
+
+	/* fill ep_dw2 */
+	dw = lower_32_bits(dqptaddr) & EP_CX_TR_DQPT_LO_MASK;
+	FIELD_MODIFY(EP_CX_DEQ_CYC_STATE_MASK, &dw, udc_ep->pcs);
+	epcx->dw2 = cpu_to_le32(dw);
+
+	/* fill ep_dw3 */
+	dw = upper_32_bits(dqptaddr);
+	epcx->dw3 = cpu_to_le32(dw);
+
+	cmd_param0 = BIT(udc_ep->dci);
+	/* Ensure that dqptr is updated */
+	dma_wmb();
+
+	return crg_udc_issue_command(crg_udc, CRG_CMD_SET_TR_DQPTR, cmd_param0, 0);
+}
+
+void crg_udc_free_request(struct usb_ep *_ep, struct usb_request *_req)
+{
+	struct crg_udc_request *udc_req_ptr;
+
+	if (!_ep || !_req)
+		return;
+
+	udc_req_ptr = container_of(_req, struct crg_udc_request, usb_req);
+
+	trace_crg_udc_free_request(container_of(_ep, struct crg_udc_ep, usb_ep),
+				   udc_req_ptr);
+
+	if (udc_req_ptr->mapped || udc_req_ptr->usb_req.num_mapped_sgs)
+		crg_udc_unmap_request(container_of(_ep, struct crg_udc_ep,
+						   usb_ep),
+				      udc_req_ptr);
+	kfree(udc_req_ptr);
+}
+
+static int crg_udc_queue_trbs(struct crg_udc_ep *udc_ep_ptr,
+			      struct crg_udc_request *udc_req_ptr, bool b_isoc,
+			      u32 xfer_ring_size, u32 num_trbs_needed,
+			      u64 buffer_length)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+	struct crg_transfer_trb *p_xfer_ring = udc_ep_ptr->first_trb;
+	struct crg_transfer_trb *enq_pt = udc_ep_ptr->enq_pt;
+	struct usb_request *usb_req = &udc_req_ptr->usb_req;
+	u32 num_trbs_ava, count, i;
+	u32 td_trbs;
+	u32 td_size;
+	u32 j = 1;
+	u32 intr_rate = b_isoc ? CRG_UDC_ISOC_EP_INTERRUPT_RATE :
+				 CRG_UDC_BULK_EP_INTERRUPT_RATE;
+	bool full_td = true;
+	bool need_zlp = false;
+	bool is_sg = usb_req->num_mapped_sgs;
+
+	if (!b_isoc && usb_req->zero && usb_req->length &&
+	    !(usb_req->length % udc_ep_ptr->usb_ep.maxpacket))
+		need_zlp = true;
+
+	if (!num_trbs_needed)
+		return -EINVAL;
+
+	/* TD_SIZE is five bits and encodes the number of TRBs remaining
+	 * in this TD minus one. A request may contain multiple TDs.
+	 */
+	td_trbs = min_t(u32, num_trbs_needed, 32);
+
+	num_trbs_ava = crg_udc_room_on_ring(crg_udc, xfer_ring_size,
+					    p_xfer_ring, udc_ep_ptr->enq_pt,
+					    udc_ep_ptr->deq_pt);
+
+	if (num_trbs_ava < td_trbs) {
+		if (b_isoc)
+			return -EBUSY;
+		trace_crg_udc_no_room_on_ring(
+			"deferring TRBs to next doorbell");
+		count = num_trbs_ava;
+		full_td = false;
+		udc_ep_ptr->tran_ring_full = true;
+		need_zlp = false;
+	} else {
+		count = td_trbs;
+	}
+
+	if (!count)
+		return -ENOSPC;
+
+	td_size = count - 1;
+
+	if (is_sg && !udc_req_ptr->sg && crg_udc_sg_init(udc_req_ptr))
+		return -EINVAL;
+
+	u8 original_pcs = udc_ep_ptr->pcs;
+
+	for (i = 0; i < count; i++) {
+		dma_addr_t trb_buf_addr;
+		u32 trb_len, sg_len = 0;
+		bool chain_bit = true;
+		bool ioc = false;
+		bool short_pkt = usb_endpoint_dir_out(udc_ep_ptr->desc);
+
+		if (is_sg) {
+			if (!crg_udc_sg_current(udc_req_ptr, &trb_buf_addr,
+						&sg_len))
+				return -EINVAL;
+			if (!sg_len ||
+			    dma_mapping_error(crg_udc->dev, trb_buf_addr))
+				return -EINVAL;
+			trb_len = min_t(
+				u64, buffer_length,
+				min_t(u32, sg_len,
+				      crg_udc_trb_chunk_len(trb_buf_addr,
+							    buffer_length)));
+		} else {
+			if (buffer_length)
+				trb_buf_addr = usb_req->dma + (usb_req->length -
+							       buffer_length);
+			else
+				trb_buf_addr = 0;
+			trb_len = crg_udc_trb_chunk_len(trb_buf_addr,
+							buffer_length);
+		}
+
+		if (!trb_len && buffer_length)
+			return -EINVAL;
+
+		buffer_length -= trb_len;
+
+		if (i == count - 1) {
+			chain_bit = false;
+			ioc = true;
+			if (!buffer_length)
+				udc_req_ptr->all_trbs_queued = true;
+		}
+
+		if (!full_td && j == intr_rate) {
+			ioc = true;
+			j = 0;
+		}
+
+		if (b_isoc) {
+			crg_udc_setup_trb(
+				enq_pt, trb_len, trb_buf_addr, td_size,
+				i == 0 ? (original_pcs ^ 1) : udc_ep_ptr->pcs,
+				TRB_TYPE_XFER_DATA_ISOCH, short_pkt, chain_bit,
+				ioc, false, 0, true, 0, 1, 0);
+		} else {
+			bool azp = ioc && !chain_bit && need_zlp;
+
+			crg_udc_setup_trb(
+				enq_pt, trb_len, trb_buf_addr, td_size,
+				i == 0 ? (original_pcs ^ 1) : udc_ep_ptr->pcs,
+				TRB_TYPE_XFER_NORMAL, short_pkt, chain_bit, ioc,
+				false, 0, false, 0, 0, azp);
+		}
+
+		trace_crg_udc_queue_trb(udc_ep_ptr, enq_pt);
+
+		if (is_sg && !crg_udc_sg_advance(udc_req_ptr, trb_len))
+			return -EINVAL;
+
+		td_size--;
+		enq_pt++;
+		j++;
+
+		if (enq_pt == udc_ep_ptr->last_trb) {
+			dma_wmb();
+			le32p_replace_bits(&enq_pt->dw3, udc_ep_ptr->pcs,
+					   TRB_CYCLE_BIT_MASK);
+			udc_ep_ptr->pcs ^= 1;
+			enq_pt = udc_ep_ptr->first_trb;
+		}
+	}
+
+	dma_wmb();
+	le32p_replace_bits(&udc_ep_ptr->enq_pt->dw3, original_pcs,
+			   TRB_CYCLE_BIT_MASK);
+
+	if (!udc_req_ptr->trbs_needed)
+		udc_req_ptr->first_trb = udc_ep_ptr->enq_pt;
+
+	udc_ep_ptr->enq_pt = enq_pt;
+	if (count == num_trbs_ava)
+		udc_ep_ptr->tran_ring_full = true;
+	udc_req_ptr->buff_len_left = buffer_length;
+	if (buffer_length)
+		udc_req_ptr->trbs_needed = num_trbs_needed - count;
+	else
+		udc_req_ptr->trbs_needed = 0;
+
+	if (!udc_req_ptr->buff_len_left) {
+		if (udc_ep_ptr->enq_pt == udc_req_ptr->first_trb)
+			udc_req_ptr->last_trb = udc_ep_ptr->last_trb - 1;
+		else
+			udc_req_ptr->last_trb = udc_ep_ptr->enq_pt - 1;
+	}
+
+	return 0;
+}
+
+static int crg_udc_reinit(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 i, tmp;
+	struct crg_udc_ep *udc_ep_ptr;
+
+	crg_udc->setup_status = WAIT_FOR_SETUP;
+	/* Base on Figure 9-1, default USB_STATE is attached */
+	crg_udc->device_state = USB_STATE_RECONNECTING;
+
+	/* halt all the endpoints */
+
+	/* disable all the endpoints */
+	tmp = readl(&uccr->ep_enable);
+	writel(tmp, &uccr->ep_enable);
+	if (readl_poll_timeout_atomic(&uccr->ep_enable, tmp, !tmp, 1, 50)) {
+		dev_dbg(crg_udc->dev,
+			"timeout waiting for all endpoints to disable\n");
+		return -ETIMEDOUT;
+	}
+
+	for (i = 2; i < CRG_NUM_EP_CX; i++) {
+		udc_ep_ptr = &crg_udc->udc_ep[i];
+		udc_ep_ptr->usb_ep.enabled = 0;
+		if (udc_ep_ptr->desc) {
+			udc_ep_ptr->desc = NULL;
+			udc_ep_ptr->comp_desc = NULL;
+			crg_udc_ep_nuke(udc_ep_ptr, -ESHUTDOWN);
+		}
+		udc_ep_ptr->tran_ring_full = false;
+		udc_ep_ptr->ep_state = EP_STATE_DISABLED;
+	}
+	crg_udc->num_enabled_eps = 0;
+
+	/* we don't handle ep0 here, we crg_udc_init_ep0 when event ring is empty */
+
+	if (crg_udc->dev_addr != 0) {
+		u32 param0;
+
+		param0 = FIELD_PREP(CRG_CMD2_0_DEV_ADDR_MASK, 0);
+		if (crg_udc_issue_command(crg_udc, CRG_CMD_SET_ADDR, param0, 0))
+			return -EIO;
+		crg_udc->dev_addr = 0;
+	}
+
+	crg_udc_clear_portpm(crg_udc);
+
+	return 0;
+}
+
+void crg_udc_set_pullup(struct crg_gadget_dev *crg_udc, bool is_on)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 val;
+
+	if (is_on) {
+		/* Enable event reporting and interrupts, then assert RUN */
+		val = readl(&uccr->config1);
+		val |= (CRG_U3DC_CFG1_CSC_EVENT_EN |
+			CRG_U3DC_CFG1_PEC_EVENT_EN |
+			CRG_U3DC_CFG1_PPC_EVENT_EN |
+			CRG_U3DC_CFG1_PRC_EVENT_EN |
+			CRG_U3DC_CFG1_PLC_EVENT_EN |
+			CRG_U3DC_CFG1_CEC_EVENT_EN);
+		writel(val, &uccr->config1);
+
+		val = readl(&uccr->control);
+		val |= (CRG_U3DC_CTRL_SYSERR_EN | CRG_U3DC_CTRL_INT_EN |
+			CRG_U3DC_CTRL_RUN);
+		writel(val, &uccr->control);
+	} else {
+		val = readl(&uccr->control);
+		val &= ~CRG_U3DC_CTRL_RUN;
+		writel(val, &uccr->control);
+	}
+}
+
+void crg_udc_set_vbus_detect(struct crg_gadget_dev *crg_udc, bool enable)
+{
+	u32 val = readl(crg_udc->mmio_virt_base + CRG_UDC_VENDOR_REG);
+
+	if (enable)
+		val |= CRG_UDC_VBUS_DETECT;
+	else
+		val &= ~CRG_UDC_VBUS_DETECT;
+
+	writel(val, crg_udc->mmio_virt_base + CRG_UDC_VENDOR_REG);
+}
+
+static bool crg_udc_sg_advance(struct crg_udc_request *udc_req, u32 bytes)
+{
+	struct scatterlist *sg = udc_req->sg;
+	unsigned int i = udc_req->sg_index;
+
+	while (bytes) {
+		u32 len;
+
+		if (!sg || i >= udc_req->usb_req.num_sgs)
+			return false;
+
+		len = sg_dma_len(sg);
+		if (!len || udc_req->sg_offset == len) {
+			sg = sg_next(sg);
+			i++;
+			udc_req->sg_offset = 0;
+			continue;
+		}
+
+		len -= udc_req->sg_offset;
+		if (bytes < len) {
+			udc_req->sg_offset += bytes;
+			bytes = 0;
+			break;
+		}
+
+		bytes -= len;
+		sg = sg_next(sg);
+		i++;
+		udc_req->sg_offset = 0;
+	}
+
+	udc_req->sg = sg;
+	udc_req->sg_index = i;
+	return true;
+}
+
+static bool crg_udc_sg_current(struct crg_udc_request *udc_req, dma_addr_t *dma,
+			       u32 *len)
+{
+	struct scatterlist *sg = udc_req->sg;
+
+	while (sg && udc_req->sg_index < udc_req->usb_req.num_sgs &&
+	       !sg_dma_len(sg)) {
+		sg = sg_next(sg);
+		udc_req->sg_index++;
+		udc_req->sg_offset = 0;
+	}
+	udc_req->sg = sg;
+
+	if (!sg || udc_req->sg_index >= udc_req->usb_req.num_sgs)
+		return false;
+
+	if (udc_req->sg_offset >= sg_dma_len(sg))
+		return false;
+
+	*dma = sg_dma_address(sg) + udc_req->sg_offset;
+	*len = sg_dma_len(sg) - udc_req->sg_offset;
+	return true;
+}
+
+static int crg_udc_sg_init(struct crg_udc_request *udc_req)
+{
+	struct scatterlist *sg;
+	unsigned int i;
+
+	for_each_sg(udc_req->usb_req.sg, sg, udc_req->usb_req.num_sgs, i) {
+		if (sg_dma_len(sg)) {
+			udc_req->sg = sg;
+			udc_req->sg_index = i;
+			udc_req->sg_offset = 0;
+			return 0;
+		}
+	}
+
+	return -EINVAL;
+}
+
+/*
+ * Undo whatever crg_udc_ep_queue() mapped for this request: the merged
+ * SG list, the single buffer, or neither if the request carried a
+ * pre-mapped DMA address.
+ */
+void crg_udc_unmap_request(struct crg_udc_ep *udc_ep,
+			   struct crg_udc_request *udc_req)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+	struct usb_request *req = &udc_req->usb_req;
+	bool is_in = udc_req->map_dir == DMA_TO_DEVICE;
+
+	if (req->num_mapped_sgs || req->dma_mapped)
+		usb_gadget_unmap_request_by_dev(crg_udc->dev, req, is_in);
+
+	if (udc_req->mapped) {
+		req->dma = DMA_MAPPING_ERROR;
+		udc_req->mapped = false;
+	}
+
+	udc_req->sg = NULL;
+	udc_req->sg_index = 0;
+	udc_req->sg_offset = 0;
+}
+
+static int crg_udc_validate_ep_desc(struct crg_udc_ep *udc_ep,
+				    const struct usb_endpoint_descriptor *desc)
+{
+	const struct usb_ss_ep_comp_descriptor *comp = udc_ep->comp_desc;
+	u16 maxpacket = usb_endpoint_maxp(desc);
+
+	if (usb_endpoint_xfer_control(desc) || !maxpacket ||
+	    maxpacket > CRG_MAX_PACKET_SIZE)
+		return -EINVAL;
+
+	if (crg_udc_index2dci(desc->bEndpointAddress) != udc_ep->dci)
+		return -EINVAL;
+
+	if (usb_endpoint_xfer_bulk(desc) && comp && usb_ss_max_streams(comp))
+		return -EOPNOTSUPP;
+
+	if (usb_endpoint_xfer_isoc(desc) && desc->bInterval == 0)
+		return -EINVAL;
+
+	if (udc_ep->crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+		if (!comp || comp->bMaxBurst > 15)
+			return -EINVAL;
+		if (usb_endpoint_xfer_isoc(desc) &&
+		    ((comp->bmAttributes & 0x3) > 2))
+			return -EINVAL;
+	} else if (udc_ep->crg_udc->gadget.speed == USB_SPEED_HIGH) {
+		if (desc->bInterval > 16 && (usb_endpoint_xfer_int(desc) ||
+					     usb_endpoint_xfer_isoc(desc)))
+			return -EINVAL;
+		if (usb_endpoint_xfer_isoc(desc) &&
+		    usb_endpoint_maxp_mult(desc) > 3)
+			return -EINVAL;
+	} else if (udc_ep->crg_udc->gadget.speed == USB_SPEED_FULL) {
+		if (maxpacket > 64 && !usb_endpoint_xfer_isoc(desc))
+			return -EINVAL;
+		if (usb_endpoint_xfer_isoc(desc) && maxpacket > 1023)
+			return -EINVAL;
+	}
+
+	return 0;
+}
+
+static bool crg_udc_vbus_connected(struct crg_gadget_dev *crg_udc)
+{
+	return !!(readl(&crg_udc->uccr->portsc) & CRG_U3DC_PORTSC_PP);
+}
+
+static void crg_ep_struct_setup(struct crg_gadget_dev *crg_udc, u32 dci,
+				const char *name)
+{
+	struct crg_udc_ep *ep = &crg_udc->udc_ep[dci];
+
+	ep->dci = dci;
+
+	if (ep->dci > 1) {
+		strscpy_pad(ep->name, name, sizeof(ep->name));
+		ep->usb_ep.name = ep->name;
+
+		ep->usb_ep.caps.type_iso = 1;
+		ep->usb_ep.caps.type_bulk = 1;
+		ep->usb_ep.caps.type_int = 1;
+		if (ep->dci % 2)
+			ep->usb_ep.caps.dir_out = 1;
+		else
+			ep->usb_ep.caps.dir_in = 1;
+		usb_ep_set_maxpacket_limit(&ep->usb_ep, CRG_MAX_PACKET_SIZE);
+	} else {
+		strscpy(ep->name, "ep0", sizeof(ep->name));
+		ep->usb_ep.name = ep->name;
+		ep->usb_ep.maxpacket = 512;
+		ep->usb_ep.caps.type_control = 1;
+		ep->usb_ep.caps.dir_in = 1;
+		ep->usb_ep.caps.dir_out = 1;
+	}
+
+	ep->usb_ep.ops = &crg_udc_ep_ops;
+	ep->crg_udc = crg_udc;
+
+	INIT_LIST_HEAD(&ep->queue);
+	if (ep->dci > 1)
+		list_add_tail(&ep->usb_ep.ep_list, &crg_udc->gadget.ep_list);
+}
+
+void crg_udc_reset_work(struct work_struct *work)
+{
+	struct crg_gadget_dev *crg_udc =
+		container_of(work, struct crg_gadget_dev, udc_reset_work);
+
+	if (crg_udc->gadget_driver)
+		usb_gadget_udc_reset(&crg_udc->gadget, crg_udc->gadget_driver);
+}
+
+int crg_udc_handle_port_status(struct crg_gadget_dev *crg_udc,
+			       unsigned long *flags)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	bool was_connected = crg_udc->connected;
+	u32 portsc_val;
+	u32 tmp;
+	u32 speed;
+
+	/* handle Port Reset */
+	portsc_val = readl(&uccr->portsc);
+	trace_crg_udc_portsc(portsc_val);
+	writel(portsc_val & CRG_U3DC_PORTSC_W1C_MASK, &uccr->portsc);
+
+	tmp = readl(&uccr->portsc);
+
+	if (portsc_val & CRG_U3DC_PORTSC_PRC) {
+		tmp = readl(&uccr->portsc);
+		if (tmp & CRG_U3DC_PORTSC_PRC) {
+			dev_dbg(crg_udc->dev, "PRC is still set\n");
+		} else if (tmp & CRG_U3DC_PORTSC_PR) {
+			/* first port status change event for port reset */
+			dev_dbg(crg_udc->dev,
+				"PRC is not set, but PR is set\n");
+		} else {
+			if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) != 0 ||
+			    (!(tmp & CRG_U3DC_PORTSC_PED))) {
+				dev_dbg(crg_udc->dev, "portsc=0x%x no PED\n",
+					tmp);
+				return 0;
+			}
+
+			switch (FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, tmp)) {
+			case CRG_U3DC_PORTSC_SPEED_SSP:
+				speed = USB_SPEED_SUPER_PLUS;
+				break;
+			case CRG_U3DC_PORTSC_SPEED_SS:
+				speed = USB_SPEED_SUPER;
+				break;
+			case CRG_U3DC_PORTSC_SPEED_HS:
+				speed = USB_SPEED_HIGH;
+				break;
+
+			case CRG_U3DC_PORTSC_SPEED_FS:
+				speed = USB_SPEED_FULL;
+				break;
+
+			case CRG_U3DC_PORTSC_SPEED_LS:
+				speed = USB_SPEED_LOW;
+				break;
+			default:
+				speed = USB_SPEED_UNKNOWN;
+				return 0;
+			}
+
+			if (crg_udc->device_state >= USB_STATE_DEFAULT) {
+				/*
+				 * A single enumeration commonly involves two
+				 * bus resets (an initial one, then another
+				 * after SET_ADDRESS before SET_CONFIGURATION).
+				 * By the second reset device_state is already
+				 * USB_STATE_ADDRESS, which is >= DEFAULT but
+				 * has no active configuration to tear down.
+				 * Only notify composite when a configuration
+				 * was actually active: otherwise this races
+				 * with the SET_CONFIGURATION that's about to
+				 * arrive, since the deferred work below can
+				 * run after fsg_set_alt() has already raised
+				 * its own FSG_STATE_CONFIG_CHANGE exception
+				 * and reset_config()'s ->disable() would
+				 * overwrite that exception's argument back to
+				 * NULL before the function's thread consumes
+				 * it, silently skipping the real enable.
+				 */
+				bool was_configured = crg_udc->device_state ==
+						       USB_STATE_CONFIGURED;
+				int reinit_ret;
+
+				reinit_ret = crg_udc_reinit(crg_udc);
+				if (reinit_ret) {
+					dev_dbg(crg_udc->dev,
+						"bus reset reinit failed: %d\n",
+						reinit_ret);
+					return reinit_ret;
+				}
+
+				/*
+				 * Notify the function/composite driver of the
+				 * bus reset so it tears down the active
+				 * configuration (usb_gadget_udc_reset() ->
+				 * ->reset()). Without this, a host that
+				 * re-sends the same SET_CONFIGURATION value
+				 * after a reset can see composite treat it as
+				 * a no-op and skip re-running set_alt(), even
+				 * though every endpoint was just disabled in
+				 * software above.
+				 *
+				 * This must not be called synchronously here:
+				 * usb_gadget_udc_reset() re-enters the
+				 * function drivers, which requires dropping
+				 * udc_lock. Doing that mid-event-processing
+				 * would let a second port-status event run
+				 * crg_udc_reinit() again while this event's
+				 * SET_CONFIGURATION/set_alt() work is still
+				 * landing on another thread, tearing the
+				 * just-enabled endpoints back down before
+				 * they can be used. Defer it to a work item
+				 * so this function never has to drop the
+				 * lock.
+				 */
+				if (was_configured && crg_udc->gadget_driver)
+					schedule_work(&crg_udc->udc_reset_work);
+			}
+
+			crg_udc->gadget.speed = speed;
+
+			crg_udc_update_ep0_maxpacketsize(crg_udc);
+
+			crg_udc->connected = 1;
+
+			if (crg_udc->device_state < USB_STATE_RECONNECTING)
+				crg_udc_enable_setup(crg_udc);
+		}
+	}
+	/* handle Port Reset end */
+
+	/* handle Port Connection Change */
+	tmp = readl(&uccr->portsc);
+	if ((tmp & (CRG_U3DC_PORTSC_CCS | CRG_U3DC_PORTSC_PP)) ==
+	    (CRG_U3DC_PORTSC_CCS | CRG_U3DC_PORTSC_PP)) {
+		if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) != 0 ||
+		    (!(tmp & CRG_U3DC_PORTSC_PED)))
+			return 0;
+
+		switch (FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, tmp)) {
+		case CRG_U3DC_PORTSC_SPEED_SSP:
+			speed = USB_SPEED_SUPER_PLUS;
+			break;
+		case CRG_U3DC_PORTSC_SPEED_SS:
+			speed = USB_SPEED_SUPER;
+			break;
+		case CRG_U3DC_PORTSC_SPEED_HS:
+			speed = USB_SPEED_HIGH;
+			break;
+
+		case CRG_U3DC_PORTSC_SPEED_FS:
+			speed = USB_SPEED_FULL;
+			break;
+
+		case CRG_U3DC_PORTSC_SPEED_LS:
+			speed = USB_SPEED_LOW;
+			break;
+		default:
+			speed = USB_SPEED_UNKNOWN;
+			return 0;
+		}
+		crg_udc->gadget.speed = speed;
+
+		crg_udc_update_ep0_maxpacketsize(crg_udc);
+
+		crg_udc->connected = 1;
+
+		if (crg_udc->device_state < USB_STATE_RECONNECTING)
+			crg_udc_enable_setup(crg_udc);
+
+	} else if (!(tmp & CRG_U3DC_PORTSC_CCS)) {
+		bool cable_connected;
+		int ccs_drop_ignore = 0;
+
+		if ((FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) == 0x0) &&
+		    (FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, tmp) <
+		     CRG_U3DC_PORTSC_SPEED_SS)) {
+			ccs_drop_ignore = 1;
+			dev_err(crg_udc->dev, "ccs glitch detect on HS/FS!\n");
+		}
+
+		if (!ccs_drop_ignore)
+			crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+
+		cable_connected = crg_udc_vbus_connected(crg_udc);
+		if (cable_connected && !ccs_drop_ignore) {
+			crg_udc->device_state = USB_STATE_POWERED;
+			crg_udc_reinit(crg_udc);
+			crg_udc->connected = 0;
+		} else if (!cable_connected) {
+			dev_dbg(crg_udc->dev,
+				"cable disconnected, rst controller\n");
+			if (was_connected)
+				crg_udc_notify_disconnect(crg_udc, flags);
+			if (crg_udc_reset(crg_udc, flags))
+				dev_err(crg_udc->dev,
+					"controller reset failed on disconnect\n");
+			crg_udc_reset_data_struct(crg_udc);
+			crg_udc->connected = 0;
+			crg_udc_init_ep0(crg_udc);
+
+			crg_udc->device_state = USB_STATE_ATTACHED;
+			return -ECONNRESET;
+		}
+	}
+	/* handle Port Connection Change end*/
+
+	if (portsc_val & CRG_U3DC_PORTSC_PLC) {
+		tmp = readl(&uccr->portsc);
+		if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) == 0xf) {
+			FIELD_MODIFY(CRG_U3DC_PORTSC_PLS_MASK, &tmp, 0x0);
+			writel(tmp, &uccr->portsc);
+		}
+	}
+
+	return 0;
+}
+
+int crg_udc_handle_xfer_event(struct crg_gadget_dev *crg_udc,
+			      struct crg_event_trb *event, unsigned long *flags)
+{
+	u32 dw3 = le32_to_cpu(event->dw3);
+	u8 dci = FIELD_GET(EVE_TRB_ENDPOINT_ID_MASK, dw3);
+	u16 comp_code =
+		FIELD_GET(EVE_TRB_COMPL_CODE_MASK, le32_to_cpu(event->dw2));
+	struct crg_udc_ep *udc_ep_ptr;
+	struct crg_udc_request *udc_req_ptr;
+	struct crg_transfer_trb *trb;
+	bool trbs_dequeued = false;
+	u32 residual;
+
+	if (dci >= CRG_NUM_EP_CX || dci == 1)
+		return -EINVAL;
+
+	udc_ep_ptr = &crg_udc->udc_ep[dci];
+	if (!udc_ep_ptr->first_trb ||
+	    crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_DISABLED)
+		return -ENODEV;
+
+	if (comp_code != CMPL_CODE_STOPPED &&
+	    comp_code != CMPL_CODE_STOPPED_LENGTH_INVALID &&
+	    comp_code != CMPL_CODE_DISABLED &&
+	    comp_code != CMPL_CODE_DISABLED_LENGTH_INVALID &&
+	    comp_code != CMPL_CODE_HALTED &&
+	    comp_code != CMPL_CODE_HALTED_LENGTH_INVALID)
+		crg_udc_update_dequeue_pt(event, udc_ep_ptr);
+
+	if (is_request_dequeued(crg_udc, udc_ep_ptr, event)) {
+		trbs_dequeued = true;
+		goto queue_more_trbs;
+	}
+
+	switch (comp_code) {
+	case CMPL_CODE_SUCCESS:
+		trb = crg_udc_tran_trb_dma_to_virt(
+			udc_ep_ptr,
+			(dma_addr_t)le32_to_cpu(event->dw0) |
+				((dma_addr_t)le32_to_cpu(event->dw1) << 32));
+		if (!trb || list_empty(&udc_ep_ptr->queue))
+			break;
+
+		/*
+		 * A control transfer has separate data-stage and status-stage
+		 * TRBs.  The status request is an internal zero-length request
+		 * and its successful completion must not go through
+		 * crg_udc_build_td().  Doing that attempts to queue another
+		 * control TD while the status request is already on the EP0
+		 * software queue, leaving it stuck in -EINPROGRESS.  The next
+		 * SETUP then sees a non-empty queue and enumeration fails.
+		 */
+		if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(trb->dw3)) ==
+		    TRB_TYPE_XFER_STATUS_STAGE) {
+			udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+						       struct crg_udc_request,
+						       queue);
+			if (dci == 0) {
+				udc_req_ptr->usb_req.actual = 0;
+				trbs_dequeued = true;
+				crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
+				crg_udc_ep0_req_complete(udc_ep_ptr, flags);
+			} else {
+				dev_err(crg_udc->dev,
+					"status-stage TRB on endpoint %u\n",
+					dci);
+			}
+			break;
+		}
+
+		if (!FIELD_GET(TRB_CHAIN_BIT_MASK, le32_to_cpu(trb->dw3))) {
+			udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+						       struct crg_udc_request,
+						       queue);
+			trbs_dequeued = true;
+
+			if (udc_req_ptr->all_trbs_queued) {
+				udc_req_ptr->usb_req.actual =
+					udc_req_ptr->usb_req.length -
+					udc_req_ptr->buff_len_left;
+				residual = FIELD_GET(EVE_TRB_TRAN_LEN_MASK,
+						     le32_to_cpu(event->dw2));
+				if (residual > udc_req_ptr->usb_req.actual)
+					residual = 0;
+				udc_req_ptr->usb_req.actual -= residual;
+				crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
+
+				if (dci == 0 && udc_ep_ptr->desc &&
+				    usb_endpoint_xfer_control(udc_ep_ptr->desc))
+					crg_udc_ep0_req_complete(udc_ep_ptr,
+								 flags);
+			} else {
+				int ret;
+
+				ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+				if (ret) {
+					udc_req_ptr->usb_req.actual =
+						udc_req_ptr->usb_req.length -
+						udc_req_ptr->buff_len_left;
+					crg_udc_req_done(udc_ep_ptr, udc_req_ptr,
+							 -EIO);
+				} else {
+					crg_udc_knock_doorbell(crg_udc, dci);
+				}
+			}
+		}
+		break;
+
+	case CMPL_CODE_SHORT_PKT:
+		if (list_empty(&udc_ep_ptr->queue))
+			break;
+
+		udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+					       struct crg_udc_request, queue);
+		udc_req_ptr->usb_req.actual = udc_req_ptr->usb_req.length -
+					      udc_req_ptr->buff_len_left;
+		residual = FIELD_GET(EVE_TRB_TRAN_LEN_MASK,
+				     le32_to_cpu(event->dw2));
+		if (residual > udc_req_ptr->usb_req.actual)
+			residual = 0;
+		udc_req_ptr->usb_req.actual -= residual;
+
+		if (udc_req_ptr->usb_req.short_not_ok &&
+		    usb_endpoint_dir_out(udc_ep_ptr->desc))
+			crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EREMOTEIO);
+		else
+			crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
+
+		trbs_dequeued = true;
+		crg_udc_advance_dequeue_pt(udc_ep_ptr);
+		break;
+
+	case CMPL_CODE_PROTOCOL_STALL:
+	case CMPL_CODE_TRB_STALL:
+		if (!list_empty(&udc_ep_ptr->queue)) {
+			udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+						       struct crg_udc_request,
+						       queue);
+			crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EPIPE);
+			trbs_dequeued = true;
+		}
+		if (dci == 0)
+			crg_udc->setup_status = WAIT_FOR_SETUP;
+		break;
+
+	case CMPL_CODE_SETUP_TAG_MISMATCH:
+		/*
+		 * EP0 SETUP changes are serialized by the SETUP event handler.
+		 * When a new SETUP arrives, crg_udc_abort_ep0_for_setup() stops
+		 * EP0, gives back the old request, resets the EP0 ring, and then
+		 * accepts the new SETUP. A SETUP_TAG_MISMATCH event generated
+		 * afterwards therefore belongs to the aborted control transfer.
+		 *
+		 * Do not complete the current EP0 request or change setup_status
+		 * here: doing so could operate on the new control transfer and
+		 * recreate the EP0 queue/state race this path is intended to avoid.
+		 */
+		if (dci == 0) {
+			dev_dbg(crg_udc->dev,
+				"ignoring stale EP0 SETUP_TAG_MISMATCH\n");
+			break;
+		}
+
+		/* This completion code is meaningful only for EP0. */
+		dev_warn(crg_udc->dev,
+			 "SETUP_TAG_MISMATCH on non-EP0 endpoint %u\n", dci);
+		break;
+
+	case CMPL_CODE_BABBLE_DETECTED_ERR:
+	case CMPL_CODE_DATA_BUFFER_ERR:
+	case CMPL_CODE_INVALID_STREAM_TYPE_ERR:
+	case CMPL_CODE_RING_UNDERRUN:
+	case CMPL_CODE_RING_OVERRUN:
+	case CMPL_CODE_EVENT_RING_FULL_ERR:
+	case CMPL_CODE_ISOCH_BUFFER_OVERRUN:
+	case CMPL_CODE_USB_TRANS_ERR:
+	case CMPL_CODE_TRB_ERR:
+		dev_err(crg_udc->dev, "transfer error, completion code 0x%x\n",
+			comp_code);
+		if (!list_empty(&udc_ep_ptr->queue)) {
+			udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+						       struct crg_udc_request,
+						       queue);
+			crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EIO);
+			trbs_dequeued = true;
+		}
+		if (dci)
+			set_ep_halt(crg_udc, dci, flags);
+		break;
+
+	case CMPL_CODE_STOPPED:
+	case CMPL_CODE_STOPPED_LENGTH_INVALID:
+	case CMPL_CODE_HALTED:
+	case CMPL_CODE_HALTED_LENGTH_INVALID:
+	case CMPL_CODE_DISABLED:
+	case CMPL_CODE_DISABLED_LENGTH_INVALID:
+		break;
+
+	default:
+		dev_err(crg_udc->dev,
+			"unsupported transfer completion code 0x%x\n",
+			comp_code);
+		if (!list_empty(&udc_ep_ptr->queue)) {
+			udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+						       struct crg_udc_request,
+						       queue);
+			crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EIO);
+			trbs_dequeued = true;
+		}
+		break;
+	}
+
+queue_more_trbs:
+	if (trbs_dequeued && udc_ep_ptr->tran_ring_full) {
+		udc_ep_ptr->tran_ring_full = false;
+		if (crg_udc_queue_pending_trbs(udc_ep_ptr))
+			udc_ep_ptr->tran_ring_full = true;
+	}
+
+	return 0;
+}
+
+int crg_udc_handle_event(struct crg_gadget_dev *crg_udc,
+			 struct crg_event_trb *event, unsigned long *flags)
+{
+	int ret;
+
+	trace_crg_udc_event_trb(event);
+
+	switch (FIELD_GET(EVE_TRB_TYPE_MASK, le32_to_cpu(event->dw3))) {
+	case TRB_TYPE_EVT_PORT_STATUS_CHANGE:
+		if (crg_udc->device_state == USB_STATE_RECONNECTING) {
+			crg_udc->portsc_on_reconnecting = 1;
+			break;
+		}
+		/* 1.Port Reset  2.Port Connection Change 3.Port Link Change */
+		ret = crg_udc_handle_port_status(crg_udc, flags);
+		if (ret)
+			return ret;
+
+		break;
+	case TRB_TYPE_EVT_TRANSFER:
+		if (crg_udc->device_state < USB_STATE_RECONNECTING) {
+			dev_err(crg_udc->dev,
+				"Xfer compl event rcved when dev state=%d !\n",
+				crg_udc->device_state);
+			break;
+		}
+
+		ret = crg_udc_handle_xfer_event(crg_udc, event, flags);
+		if (ret)
+			return ret;
+		break;
+	case TRB_TYPE_EVT_SETUP_PKT: {
+		struct usb_ctrlrequest setup;
+		u32 dw0 = le32_to_cpu(event->dw0);
+		u32 dw1 = le32_to_cpu(event->dw1);
+		u8 setup_tag;
+
+		BUILD_BUG_ON(sizeof(struct usb_ctrlrequest) != 8);
+		setup.bRequestType = dw0 & 0xff;
+		setup.bRequest = (dw0 >> 8) & 0xff;
+		setup.wValue = cpu_to_le16((dw0 >> 16) & 0xffff);
+		setup.wIndex = cpu_to_le16(dw1 & 0xffff);
+		setup.wLength = cpu_to_le16((dw1 >> 16) & 0xffff);
+
+		setup_tag = FIELD_GET(EVE_TRB_SETUP_TAG_MASK,
+				      le32_to_cpu(event->dw3));
+		if (crg_udc->setup_status != WAIT_FOR_SETUP) {
+			/* A new SETUP aborts the previous control transfer. The old
+			 * EP0 request must be completed and its TRBs discarded before
+			 * the new setup is handed to the gadget driver.
+			 */
+			if (crg_udc_abort_ep0_for_setup(crg_udc)) {
+				crg_udc_set_ep0_halt(crg_udc);
+				break;
+			}
+		}
+
+		crg_udc_handle_setup_pkt(crg_udc, &setup, setup_tag, flags);
+
+		break;
+	}
+	default:
+		dev_dbg(crg_udc->dev, "unexpected TRB_TYPE = 0x%lx",
+			FIELD_GET(EVE_TRB_TYPE_MASK, le32_to_cpu(event->dw3)));
+		break;
+	}
+
+	return 0;
+}
+
+/*
+ * Invoke the bound gadget driver's ->disconnect() callback, if any.
+ *
+ * The gadget disconnect callback may be called in interrupt context.
+ * Release only udc_lock while keeping local IRQs disabled.
+ */
+void crg_udc_notify_disconnect(struct crg_gadget_dev *crg_udc,
+			       unsigned long *flags)
+{
+	/* Report link loss even when there is no gadget callback. */
+	usb_gadget_set_state(&crg_udc->gadget, USB_STATE_NOTATTACHED);
+
+	if (!crg_udc->async_callbacks || !crg_udc->gadget_driver ||
+	    !crg_udc->gadget_driver->disconnect)
+		return;
+
+	spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+	crg_udc->gadget_driver->disconnect(&crg_udc->gadget);
+	spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+}
+
+int crg_udc_ep_halt(struct crg_udc_ep *udc_ep_ptr, int halt, int ignore_wedge,
+		    unsigned long *flags)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	enum crg_ep_state ep_state;
+	bool reset_seq_only = false;
+	int do_halt;
+	u32 param0;
+	u32 tmp;
+
+	if (!udc_ep_ptr->desc) {
+		dev_err(crg_udc->dev, "NULL desc\n");
+		return -EINVAL;
+	}
+
+	if (usb_endpoint_xfer_isoc(udc_ep_ptr->desc)) {
+		dev_err(crg_udc->dev, "Isoc ep, halt not supported\n");
+		return -EOPNOTSUPP;
+	}
+
+	if (udc_ep_ptr->dci == 0)
+		return 0;
+
+	ep_state = crg_udc_get_ep_state(crg_udc, udc_ep_ptr->dci);
+
+	if (ep_state == EP_STATE_DISABLED)
+		return 0;
+
+	if (ep_state == EP_STATE_HALTED) {
+		if (halt != 0)
+			return 0;
+
+		/* want unhalt an halted ep */
+		if (udc_ep_ptr->wedge && !ignore_wedge) {
+			do_halt = -1;
+			reset_seq_only = true;
+		} else {
+			do_halt = 0;
+		}
+
+	} else {
+		/* ep state == running or stopped */
+		if (halt != 0) {
+			/* want halt a running ep */
+			do_halt = 1;
+		} else {
+			/* reset a running ep */
+			do_halt = 0;
+			reset_seq_only = true;
+		}
+	}
+
+	trace_crg_udc_ep_halt(udc_ep_ptr, do_halt == 1, udc_ep_ptr->wedge);
+
+	param0 = BIT(udc_ep_ptr->dci);
+	if (do_halt == 1) {
+		/* setting ep to halt */
+		if (crg_udc_issue_command(crg_udc, CRG_CMD_SET_HALT, param0, 0))
+			return -EIO;
+		if (readl_poll_timeout_atomic(&uccr->ep_running, tmp,
+					      !(tmp & param0), 10, 10000))
+			dev_warn(crg_udc->dev, "ep %d halt wait timed out\n",
+				 udc_ep_ptr->dci);
+
+		if (crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+			/* clean up the request queue */
+			crg_udc_ep_nuke(udc_ep_ptr, -ECONNRESET);
+
+			udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
+			udc_ep_ptr->tran_ring_full = false;
+		}
+		/* clean up the request queue */
+		udc_ep_ptr->ep_state = EP_STATE_HALTED;
+	} else if (do_halt == 0) {
+		/* clearing ep halt state */
+		/* reset sequence number */
+		if (crg_udc_issue_command(crg_udc, CRG_CMD_RESET_SEQNUM, param0,
+					  0))
+			return -EIO;
+
+		if (!reset_seq_only) {
+			/* Clear halt for a halted EP. */
+			/* NOTE: we must CLEAR_HALT first,
+			 * then SET_TR_DQPTR
+			 */
+			if (crg_udc_issue_command(crg_udc, CRG_CMD_CLEAR_HALT,
+						  param0, 0))
+				return -EIO;
+			if (crg_udc_epcx_update_dqptr(udc_ep_ptr))
+				return -EIO;
+		}
+
+		udc_ep_ptr->wedge = 0;
+		udc_ep_ptr->ep_state = EP_STATE_RUNNING;
+		/* set endpoint to running state */
+		/* clear pause for the endpoint */
+		if (!list_empty(&udc_ep_ptr->queue)) {
+			struct crg_transfer_trb *next = udc_ep_ptr->deq_pt + 1;
+
+			if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(next->dw3)) ==
+			    TRB_TYPE_LINK) {
+				udc_ep_ptr->pcs ^= 1;
+				if (crg_udc_epcx_update_dqptr(udc_ep_ptr)) {
+					udc_ep_ptr->pcs ^= 1;
+					return -EIO;
+				}
+				udc_ep_ptr->pcs ^= 1;
+			}
+			crg_udc_knock_doorbell(crg_udc, udc_ep_ptr->dci);
+		}
+	} else {
+		/* wedged EP deny CLEAR HALT */
+		dev_dbg(crg_udc->dev, "wedged EP deny CLEAR HALT dci = %d\n",
+			udc_ep_ptr->dci);
+		/* reset sequence number */
+		if (reset_seq_only &&
+		    crg_udc_issue_command(crg_udc, CRG_CMD_RESET_SEQNUM, param0,
+					  0))
+			return -EIO;
+	}
+
+	return 0;
+}
+
+enum crg_ep_state crg_udc_get_ep_state(struct crg_gadget_dev *crg_udc, int dci)
+{
+	struct crg_udc_ep *udc_ep_ptr;
+
+	/*
+	 * DCI 0 is ep0; DCI 1 is unused in the Corigine ep-context layout
+	 * (ep contexts start from DCI 2 for ep1in/ep1out). This is why the
+	 * ep-context array is indexed as p_epcx[dci - 2] elsewhere.
+	 */
+	if (dci < 0 || dci >= CRG_NUM_EP_CX || dci == 1)
+		return EP_STATE_DISABLED;
+
+	udc_ep_ptr = &crg_udc->udc_ep[dci];
+
+	return udc_ep_ptr->ep_state;
+}
+
+u32 crg_udc_index2dci(u16 index)
+{
+	if (index == 0)
+		return 0;
+
+	return (index & USB_ENDPOINT_NUMBER_MASK) * 2 +
+	       ((index & USB_DIR_IN) ? 0 : 1);
+}
+
+int crg_udc_init_device_context(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	int ret;
+	u32 buff_length;
+	dma_addr_t mapping;
+
+	/* ep0 is not included in ep contexts in crg udc */
+	buff_length = (CRG_NUM_EP_CX - 2) * sizeof(struct crg_ep_cx);
+
+	if (!crg_udc->ep_cx.cpu_addr) {
+		crg_udc->ep_cx.cpu_addr = dma_alloc_coherent(
+			crg_udc->dev, buff_length, &mapping, GFP_ATOMIC);
+		if (!crg_udc->ep_cx.cpu_addr) {
+			ret = -ENOMEM;
+			return ret;
+		}
+	} else {
+		mapping = crg_udc->ep_cx.dma_addr;
+	}
+
+	crg_udc->p_epcx = crg_udc->ep_cx.cpu_addr;
+	crg_udc->ep_cx.len = buff_length;
+	crg_udc->ep_cx.dma_addr = mapping;
+	memset(crg_udc->p_epcx, 0, buff_length);
+	/*
+	 * Ensure the endpoint contexts are initialized before making their
+	 * DMA address visible to the controller.
+	 */
+	dma_wmb();
+	/* hw ops DCBAPLO DCBAPHI */
+	writel(lower_32_bits(crg_udc->ep_cx.dma_addr), &uccr->dcbaplo);
+	writel(upper_32_bits(crg_udc->ep_cx.dma_addr), &uccr->dcbaphi);
+
+	return 0;
+}
+
+void crg_udc_init_ep_info(struct crg_gadget_dev *crg_udc)
+{
+	int i;
+
+	/* udc_ep[0] is reserved */
+	crg_ep_struct_setup(crg_udc, 0, NULL);
+
+	for (i = 1; i < CRG_NUM_EP_CX / 2; i++) {
+		char name[14];
+
+		snprintf(name, sizeof(name), "ep%din", i);
+		crg_ep_struct_setup(crg_udc, i * 2, name);
+		snprintf(name, sizeof(name), "ep%dout", i);
+		crg_ep_struct_setup(crg_udc, i * 2 + 1, name);
+	}
+}
+
+static bool is_request_dequeued(struct crg_gadget_dev *crg_udc,
+				struct crg_udc_ep *udc_ep,
+				struct crg_event_trb *event)
+{
+	struct crg_udc_request *udc_req;
+	u32 trb_pt_lo = le32_to_cpu(event->dw0);
+	u32 trb_pt_hi = le32_to_cpu(event->dw1);
+	u64 trb_addr = (u64)trb_pt_lo | ((u64)trb_pt_hi << 32);
+	struct crg_transfer_trb *trb_pt;
+	bool status = true;
+
+	if (udc_ep->dci == 0 || list_empty(&udc_ep->queue))
+		return false;
+
+	trb_pt = crg_udc_tran_trb_dma_to_virt(udc_ep, trb_addr);
+	if (!trb_pt)
+		return false;
+	list_for_each_entry(udc_req, &udc_ep->queue, queue) {
+		if (trb_pt == udc_req->last_trb ||
+		    trb_pt == udc_req->first_trb) {
+			status = false;
+			break;
+		}
+
+		if (crg_udc_is_pointer_less_than(trb_pt, udc_req->last_trb,
+						 udc_ep) &&
+		    crg_udc_is_pointer_less_than(udc_req->first_trb, trb_pt,
+						 udc_ep)) {
+			status = false;
+			break;
+		}
+	}
+
+	return status;
+}
+
+void crg_udc_knock_doorbell(struct crg_gadget_dev *crg_udc, int dci)
+{
+	u32 tmp;
+	struct crg_uccr __iomem *uccr;
+
+	uccr = crg_udc->uccr;
+	/* Ensure everything is written before notifying the HW */
+	dma_wmb();
+
+	tmp = FIELD_PREP(CRG_U3DC_DB_TARGET_MASK, dci);
+	writel(tmp, &uccr->doorbell);
+
+	trace_crg_udc_doorbell(crg_udc, dci);
+}
+
+/*
+ * Complete a request and give it back to the gadget layer via the
+ * standard usb_gadget_giveback_request() helper.
+ *
+ * Must be called with udc_lock held and interrupts disabled.
+ *
+ * The request's ->complete() callback must not sleep. crg_udc_req_done() drops
+ * udc_lock before the giveback and re-acquires it before returning.
+ * Callers therefore cannot assume state protected by udc_lock is
+ * unchanged across this call.
+ */
+void crg_udc_req_done(struct crg_udc_ep *udc_ep,
+		      struct crg_udc_request *udc_req, int status)
+{
+	struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+
+	if (likely(udc_req->usb_req.status == -EINPROGRESS))
+		udc_req->usb_req.status = status;
+
+	trace_crg_udc_request_giveback(udc_ep, udc_req);
+
+	list_del_init(&udc_req->queue);
+
+	crg_udc_unmap_request(udc_ep, udc_req);
+
+	if (udc_req->usb_req.complete) {
+		spin_unlock(&crg_udc->udc_lock);
+		usb_gadget_giveback_request(&udc_ep->usb_ep, &udc_req->usb_req);
+		spin_lock(&crg_udc->udc_lock);
+	}
+}
+
+static int set_ep_halt(struct crg_gadget_dev *crg_udc, int dci,
+		       unsigned long *flags)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[dci];
+	u32 param0;
+	u32 tmp;
+
+	if (dci == 0)
+		return 0;
+
+	if (udc_ep_ptr->ep_state == EP_STATE_DISABLED ||
+	    udc_ep_ptr->ep_state == EP_STATE_HALTED)
+		return 0;
+	if (udc_ep_ptr->desc && usb_endpoint_xfer_isoc(udc_ep_ptr->desc))
+		return -EOPNOTSUPP;
+
+	param0 = BIT(dci);
+	if (crg_udc_issue_command(crg_udc, CRG_CMD_SET_HALT, param0, 0))
+		return -EIO;
+	if (readl_poll_timeout_atomic(&uccr->ep_running, tmp, !(tmp & param0),
+				      10, 10000))
+		return -ETIMEDOUT;
+	/* clean up the request queue */
+	crg_udc_ep_nuke(udc_ep_ptr, -ECONNRESET);
+
+	udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
+	udc_ep_ptr->tran_ring_full = false;
+	udc_ep_ptr->ep_state = EP_STATE_HALTED;
+
+	return 0;
+}
+
+void crg_udc_clear_req_container(struct crg_udc_request *udc_req_ptr)
+{
+	udc_req_ptr->buff_len_left = 0;
+	udc_req_ptr->trbs_needed = 0;
+	udc_req_ptr->all_trbs_queued = 0;
+	udc_req_ptr->first_trb = NULL;
+	udc_req_ptr->last_trb = NULL;
+	udc_req_ptr->sg = NULL;
+	udc_req_ptr->sg_index = 0;
+	udc_req_ptr->sg_offset = 0;
+}
+
+const struct usb_ep_ops crg_udc_ep_ops = {
+	.enable = crg_udc_ep_enable,
+	.disable = crg_udc_ep_disable,
+	.alloc_request = crg_udc_alloc_request,
+	.free_request = crg_udc_free_request,
+	.queue = crg_udc_ep_queue,
+	.dequeue = crg_udc_ep_dequeue,
+	.set_halt = crg_udc_ep_set_halt,
+	.set_wedge = crg_udc_ep_set_wedge,
+};
+
+const struct usb_gadget_ops crg_gadget_ops = {
+	.get_frame = crg_gadget_get_frame,
+	.wakeup = crg_gadget_wakeup,
+	.set_remote_wakeup = crg_gadget_set_remote_wakeup,
+	.set_selfpowered = crg_gadget_set_selfpowered,
+	.pullup = crg_gadget_pullup,
+	.udc_start = crg_gadget_start,
+	.udc_stop = crg_gadget_stop,
+	.udc_set_speed = crg_udc_set_speed,
+	.udc_async_callbacks = crg_gadget_async_callbacks,
+};
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-trace.c b/drivers/usb/gadget/udc/crg/crg-udc-trace.c
new file mode 100644
index 000000000000..ee091708dbe0
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-trace.c
@@ -0,0 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ * Trace Support.
+ *
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#define CREATE_TRACE_POINTS
+#include "crg-udc-trace.h"

-- 
2.34.1

[PATCH v3 6/6] usb: gadget: udc: crg: Add core driver and build integration

[Petar Stepanovic]

This patch introduces the core probe/remove functions for the Axiado
AX3000/AX3005 UDC driver, and wires up the directory into the build
system via Kconfig and Makefile.

Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
 MAINTAINERS                               |   1 +
 drivers/usb/gadget/udc/Kconfig            |   2 +
 drivers/usb/gadget/udc/Makefile           |   1 +
 drivers/usb/gadget/udc/crg/Kconfig        |  16 +
 drivers/usb/gadget/udc/crg/Makefile       |   9 +
 drivers/usb/gadget/udc/crg/crg-udc-core.c | 706 ++++++++++++++++++++++++++++++
 6 files changed, 735 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 58c5c8e00870..d5b6a04a95ad 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4515,6 +4515,7 @@ M:	Prasad Bolisetty <pbolisetty@axiado.com>
 L:	linux-usb@vger.kernel.org
 S:	Supported
 F:	Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
+F:	drivers/usb/gadget/udc/crg/
 
 AYANEO PLATFORM EC DRIVER
 M:	Antheas Kapenekakis <lkml@antheas.dev>
diff --git a/drivers/usb/gadget/udc/Kconfig b/drivers/usb/gadget/udc/Kconfig
index 7ddbc2606686..72a5a8a04982 100644
--- a/drivers/usb/gadget/udc/Kconfig
+++ b/drivers/usb/gadget/udc/Kconfig
@@ -421,6 +421,8 @@ source "drivers/usb/gadget/udc/aspeed-vhub/Kconfig"
 
 source "drivers/usb/gadget/udc/cdns2/Kconfig"
 
+source "drivers/usb/gadget/udc/crg/Kconfig"
+
 #
 # LAST -- dummy/emulated controller
 #
diff --git a/drivers/usb/gadget/udc/Makefile b/drivers/usb/gadget/udc/Makefile
index 1b9b1a4f9c57..227d2a0b525b 100644
--- a/drivers/usb/gadget/udc/Makefile
+++ b/drivers/usb/gadget/udc/Makefile
@@ -38,3 +38,4 @@ obj-$(CONFIG_USB_ASPEED_UDC)	+= aspeed_udc.o
 obj-$(CONFIG_USB_BDC_UDC)	+= bdc/
 obj-$(CONFIG_USB_MAX3420_UDC)	+= max3420_udc.o
 obj-$(CONFIG_USB_CDNS2_UDC)	+= cdns2/
+obj-$(CONFIG_USB_CRG_UDC)	+= crg/
diff --git a/drivers/usb/gadget/udc/crg/Kconfig b/drivers/usb/gadget/udc/crg/Kconfig
new file mode 100644
index 000000000000..7167c4943389
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/Kconfig
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: GPL-2.0
+config USB_CRG_UDC
+	tristate "Corigine / Axiado USB Device Controller"
+	depends on USB_GADGET
+	depends on HAS_DMA
+	depends on ARCH_AXIADO || COMPILE_TEST
+	help
+	  Say Y here to support the USB device controller found on Axiado AX3000
+	  and AX3005 SoCs, which is based on the Corigine USB 3.1 IP.
+
+	  This controller supports High-Speed and SuperSpeed.
+
+	  Say "y" to link the driver statically, or "m" to build a
+	  dynamically linked module called "crg-udc.ko".
+
+	  If unsure, say N.
diff --git a/drivers/usb/gadget/udc/crg/Makefile b/drivers/usb/gadget/udc/crg/Makefile
new file mode 100644
index 000000000000..96b209fb59fc
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/Makefile
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: GPL-2.0
+
+# define_trace.h needs to know how to find our trace header
+CFLAGS_crg-udc-trace.o		:= -I$(src)
+
+obj-$(CONFIG_USB_CRG_UDC)	+= crg-udc.o
+
+crg-udc-y			:= crg-udc-core.o crg-udc-ring.o crg-udc-ep0.o crg-udc-gadget.o
+crg-udc-$(CONFIG_TRACING)	+= crg-udc-trace.o
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-core.c b/drivers/usb/gadget/udc/crg/crg-udc-core.c
new file mode 100644
index 000000000000..80a4fc6a61e8
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-core.c
@@ -0,0 +1,706 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/clk.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/iopoll.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/phy/phy.h>
+#include <linux/platform_device.h>
+#include <linux/pm.h>
+#include <linux/property.h>
+#include <linux/reset.h>
+#include <linux/slab.h>
+
+#include "crg-udc.h"
+
+static int crg_gadget_irq_init(struct platform_device *pdev,
+			       struct crg_gadget_dev *crg_udc);
+static void crg_udc_phy_exit(void *data);
+
+static void crg_udc_gadget_release(struct device *dev)
+{
+	struct crg_gadget_dev *crg_udc;
+
+	crg_udc = container_of(dev, struct crg_gadget_dev, gadget.dev);
+	kfree(crg_udc);
+}
+
+/*
+ * command related ops
+ */
+int crg_udc_issue_command(struct crg_gadget_dev *crg_udc,
+			  enum crg_cmd_type type, u32 param0, u32 param1)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 status;
+	bool check_complete = false;
+	u32 tmp;
+
+	/*
+	 * The command state machine only runs while the controller itself is
+	 * running. Once RUN is cleared (e.g. during udc_stop()/reset()), the
+	 * hardware never processes queued commands, so ACTIVE never clears.
+	 * Skip the busy/completion checks in that case instead of getting
+	 * stuck on a command the hardware will never service.
+	 */
+	tmp = readl(&uccr->control);
+	if (tmp & CRG_U3DC_CTRL_RUN)
+		check_complete = true;
+
+	if (check_complete) {
+		tmp = readl(&uccr->cmd_control);
+		if (tmp & CRG_U3DC_CMD_CTRL_ACTIVE) {
+			dev_err(crg_udc->dev,
+				"previous command is not complete\n");
+			return -EBUSY;
+		}
+	}
+	/* Ensure that everything is written before issuing new command */
+	dma_wmb();
+
+	writel(param0, &uccr->cmd_param0);
+	writel(param1, &uccr->cmd_param1);
+
+	tmp = CRG_U3DC_CMD_CTRL_ACTIVE |
+	      FIELD_PREP(CRG_U3DC_CMD_CTRL_TYPE_MASK, type);
+	writel(tmp, &uccr->cmd_control);
+
+	if (!check_complete)
+		return 0;
+
+	if (readl_poll_timeout_atomic(&uccr->cmd_control, tmp,
+				      !(tmp & CRG_U3DC_CMD_CTRL_ACTIVE), 1,
+				      1000)) {
+		dev_dbg(crg_udc->dev, "command timed out\n");
+		return -ETIMEDOUT;
+	}
+
+	status = FIELD_GET(CRG_U3DC_CMD_CTRL_STATUS_MASK, tmp);
+	if (status) {
+		dev_dbg(crg_udc->dev, "command failed with status %u\n",
+			status);
+		return -EIO;
+	}
+
+	return 0;
+}
+
+static void crg_udc_clk_disable(void *data)
+{
+	clk_disable_unprepare(data);
+}
+
+void crg_udc_free_resources(struct crg_gadget_dev *crg_udc)
+{
+	int i;
+
+	if (crg_udc->uccr) {
+		u32 val = readl(&crg_udc->uccr->control);
+
+		val &= ~(CRG_U3DC_CTRL_INT_EN | CRG_U3DC_CTRL_RUN);
+		writel(val, &crg_udc->uccr->control);
+	}
+	if (crg_udc->irq >= 0) {
+		devm_free_irq(crg_udc->dev, crg_udc->irq, crg_udc);
+		crg_udc->irq = -1;
+		crg_udc->gadget.irq = 0;
+	}
+
+	for (i = 0; i < CRG_RING_NUM; i++) {
+		struct crg_udc_event *udc_event = &crg_udc->udc_event[i];
+
+		if (udc_event->event_ring.cpu_addr) {
+			dma_free_coherent(crg_udc->dev,
+					  udc_event->event_ring.len,
+					  udc_event->event_ring.cpu_addr,
+					  udc_event->event_ring.dma_addr);
+			udc_event->event_ring.cpu_addr = NULL;
+			udc_event->event_ring.dma_addr = DMA_MAPPING_ERROR;
+			udc_event->event_ring.len = 0;
+			udc_event->evt_dq_pt = NULL;
+		}
+		if (udc_event->erst.cpu_addr) {
+			dma_free_coherent(crg_udc->dev, udc_event->erst.len,
+					  udc_event->erst.cpu_addr,
+					  udc_event->erst.dma_addr);
+			udc_event->erst.cpu_addr = NULL;
+			udc_event->erst.dma_addr = DMA_MAPPING_ERROR;
+			udc_event->erst.len = 0;
+			udc_event->p_erst = NULL;
+		}
+	}
+
+	if (crg_udc->ep_cx.cpu_addr) {
+		dma_free_coherent(crg_udc->dev, crg_udc->ep_cx.len,
+				  crg_udc->ep_cx.cpu_addr,
+				  crg_udc->ep_cx.dma_addr);
+		crg_udc->ep_cx.cpu_addr = NULL;
+		crg_udc->ep_cx.dma_addr = DMA_MAPPING_ERROR;
+		crg_udc->ep_cx.len = 0;
+		crg_udc->p_epcx = NULL;
+	}
+
+	for (i = 0; i < CRG_NUM_EP_CX; i++) {
+		struct crg_udc_ep *udc_ep = &crg_udc->udc_ep[i];
+
+		if (udc_ep->tran_ring_info.cpu_addr) {
+			dma_free_coherent(crg_udc->dev,
+					  udc_ep->tran_ring_info.len,
+					  udc_ep->tran_ring_info.cpu_addr,
+					  udc_ep->tran_ring_info.dma_addr);
+			udc_ep->tran_ring_info.cpu_addr = NULL;
+			udc_ep->tran_ring_info.dma_addr = DMA_MAPPING_ERROR;
+			udc_ep->tran_ring_info.len = 0;
+			udc_ep->first_trb = NULL;
+			udc_ep->last_trb = NULL;
+			udc_ep->enq_pt = NULL;
+			udc_ep->deq_pt = NULL;
+		}
+	}
+
+	if (crg_udc->status_req) {
+		crg_udc_free_request(&crg_udc->udc_ep[0].usb_ep,
+				     &crg_udc->status_req->usb_req);
+		crg_udc->status_req = NULL;
+	}
+}
+
+static int crg_udc_init_hw(struct crg_gadget_dev *crg_udc)
+{
+	struct device *dev = crg_udc->dev;
+	int ret;
+
+	crg_udc->clk = devm_clk_get_optional(dev, NULL);
+	if (IS_ERR(crg_udc->clk))
+		return dev_err_probe(dev, PTR_ERR(crg_udc->clk),
+				     "failed to get clock\n");
+
+	ret = clk_prepare_enable(crg_udc->clk);
+	if (ret)
+		return dev_err_probe(dev, ret, "failed to enable clock\n");
+
+	ret = devm_add_action_or_reset(dev, crg_udc_clk_disable, crg_udc->clk);
+	if (ret)
+		return ret;
+
+	crg_udc->rst =
+		devm_reset_control_get_optional_exclusive_deasserted(dev, NULL);
+	if (IS_ERR(crg_udc->rst))
+		return dev_err_probe(dev, PTR_ERR(crg_udc->rst),
+				     "failed to get reset\n");
+
+	crg_udc->phy = devm_phy_optional_get(dev, NULL);
+	if (IS_ERR(crg_udc->phy))
+		return dev_err_probe(dev, PTR_ERR(crg_udc->phy),
+				     "failed to get phy\n");
+
+	ret = phy_init(crg_udc->phy);
+	if (ret)
+		return dev_err_probe(dev, ret, "failed to init phy\n");
+
+	ret = phy_power_on(crg_udc->phy);
+	if (ret) {
+		phy_exit(crg_udc->phy);
+		return dev_err_probe(dev, ret, "failed to power on phy\n");
+	}
+
+	return devm_add_action_or_reset(dev, crg_udc_phy_exit, crg_udc->phy);
+}
+
+static void crg_udc_phy_exit(void *data)
+{
+	struct phy *phy = data;
+
+	phy_power_off(phy);
+	phy_exit(phy);
+}
+
+static int crg_udc_probe(struct platform_device *pdev)
+{
+	unsigned long flags = 0;
+	int ret;
+	int i;
+	struct crg_gadget_dev *crg_udc;
+	const struct crg_udc_priv *priv;
+
+	priv = device_get_match_data(&pdev->dev);
+	if (!priv)
+		return -ENODEV;
+
+	crg_udc = kzalloc_obj(*crg_udc, GFP_KERNEL);
+	if (!crg_udc)
+		return -ENOMEM;
+	usb_initialize_gadget(&pdev->dev, &crg_udc->gadget,
+			      crg_udc_gadget_release);
+
+	crg_udc->dev = &pdev->dev;
+	crg_udc->irq = -1;
+
+	ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64));
+	if (ret) {
+		ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32));
+		if (ret) {
+			dev_err(&pdev->dev, "no usable DMA mask\n");
+			goto err_free_resources;
+		}
+	}
+
+	ret = crg_udc_init_hw(crg_udc);
+	if (ret)
+		goto err_free_resources;
+
+	crg_udc->ep0_desc.bLength = USB_DT_ENDPOINT_SIZE;
+	crg_udc->ep0_desc.bDescriptorType = USB_DT_ENDPOINT;
+	crg_udc->ep0_desc.bEndpointAddress = 0;
+	crg_udc->ep0_desc.bmAttributes = USB_ENDPOINT_XFER_CONTROL;
+	crg_udc->ep0_desc.wMaxPacketSize = cpu_to_le16(64);
+
+	crg_udc->u12_forbidden = priv->quirks & CRG_QUIRK_DIS_U1U2_ENTRY;
+	crg_udc->u12_initiate_forbidden = priv->quirks &
+					  CRG_QUIRK_DIS_U1U2_INITIATION;
+	crg_udc->reinit_ep0_on_bus_reset = priv->quirks &
+					   CRG_QUIRK_REINIT_EP0_ON_RESET;
+
+	spin_lock_init(&crg_udc->udc_lock);
+	INIT_WORK(&crg_udc->udc_reset_work, crg_udc_reset_work);
+	platform_set_drvdata(pdev, crg_udc);
+
+	crg_udc->gadget.ops = &crg_gadget_ops;
+	crg_udc->gadget.ep0 = &crg_udc->udc_ep[0].usb_ep;
+	INIT_LIST_HEAD(&crg_udc->gadget.ep_list);
+	/*
+	 * The controller supports up to SuperSpeed. Honour the standard
+	 * "maximum-speed" property to cap it; default to the hardware
+	 * maximum when the property is absent.
+	 */
+	crg_udc->gadget.max_speed = usb_get_maximum_speed(&pdev->dev);
+	if (crg_udc->gadget.max_speed == USB_SPEED_UNKNOWN)
+		crg_udc->gadget.max_speed = USB_SPEED_SUPER;
+	crg_udc->max_speed = crg_udc->gadget.max_speed;
+	crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+	crg_udc->gadget.name = KBUILD_MODNAME;
+	crg_udc->gadget.sg_supported = true;
+	crg_udc->gadget.wakeup_capable = true;
+	crg_udc->connected = 0;
+	crg_udc->dev_addr = 0;
+
+	crg_udc->mmio_virt_base = devm_platform_ioremap_resource(pdev, 0);
+	if (IS_ERR(crg_udc->mmio_virt_base)) {
+		ret = dev_err_probe(&pdev->dev,
+				    PTR_ERR(crg_udc->mmio_virt_base),
+				    "failed to map registers\n");
+		goto err_put_gadget;
+	}
+
+	/* set controller device role */
+	writel((readl(crg_udc->mmio_virt_base + CRG_UDC_MODE_REG) |
+		CRG_UDC_ROLE_DEVICE),
+	       crg_udc->mmio_virt_base + CRG_UDC_MODE_REG);
+	for (i = 0; i < CRG_RING_NUM; i++) {
+		crg_udc->uicr[i] = crg_udc->mmio_virt_base + CRG_UICR_OFFSET +
+				   i * CRG_UICR_STRIDE;
+	}
+	crg_udc->uccr = crg_udc->mmio_virt_base + CRG_UCCR_OFFSET;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	ret = crg_udc_reset(crg_udc, &flags);
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	if (ret)
+		goto err_free_resources;
+
+	crg_udc_clear_portpm(crg_udc);
+
+	ret = crg_gadget_irq_init(pdev, crg_udc);
+	if (ret)
+		goto err_free_resources;
+
+	ret = crg_udc_reset_data_struct(crg_udc);
+	if (ret) {
+		dev_err(crg_udc->dev, "crg_udc_reset_data_struct error\n");
+		goto err_free_resources;
+	}
+
+	crg_udc_init_ep_info(crg_udc);
+	ret = crg_udc_init_ep0(crg_udc);
+	if (ret)
+		goto err_free_resources;
+
+	crg_udc->udc_ep[0].desc = &crg_udc->ep0_desc;
+
+	ret = usb_add_gadget(&crg_udc->gadget);
+	if (ret)
+		goto err_free_resources;
+
+	return 0;
+
+err_free_resources:
+	crg_udc_free_resources(crg_udc);
+err_put_gadget:
+	usb_put_gadget(&crg_udc->gadget);
+	return ret;
+}
+
+static void crg_udc_remove(struct platform_device *pdev)
+{
+	struct crg_gadget_dev *crg_udc = platform_get_drvdata(pdev);
+	u32 tmp;
+
+	crg_udc->device_state = USB_STATE_ATTACHED;
+	crg_udc_set_vbus_detect(crg_udc, false);
+
+	usb_del_gadget(&crg_udc->gadget);
+	cancel_work_sync(&crg_udc->udc_reset_work);
+	crg_udc_free_resources(crg_udc);
+
+	/* set controller host role */
+	tmp = readl(crg_udc->mmio_virt_base + CRG_UDC_MODE_REG) & ~0x1;
+	writel(tmp, crg_udc->mmio_virt_base + CRG_UDC_MODE_REG);
+
+	platform_set_drvdata(pdev, NULL);
+	usb_put_gadget(&crg_udc->gadget);
+}
+
+static void crg_udc_shutdown(struct platform_device *pdev)
+{
+	struct crg_gadget_dev *crg_udc = platform_get_drvdata(pdev);
+	unsigned long flags;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	crg_udc->device_state = USB_STATE_ATTACHED;
+	crg_udc_set_pullup(crg_udc, false);
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	crg_udc_set_vbus_detect(crg_udc, false);
+	if (crg_udc->uccr) {
+		u32 val = readl(&crg_udc->uccr->control);
+
+		val &= ~CRG_U3DC_CTRL_INT_EN;
+		writel(val, &crg_udc->uccr->control);
+	}
+	if (crg_udc->irq >= 0)
+		synchronize_irq(crg_udc->irq);
+}
+
+static int __maybe_unused crg_udc_suspend(struct device *dev)
+{
+	struct crg_gadget_dev *crg_udc = dev_get_drvdata(dev);
+	unsigned long flags;
+
+	/* Stop the pullup while the controller is suspended. */
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	crg_udc_set_pullup(crg_udc, false);
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+	return 0;
+}
+
+static int __maybe_unused crg_udc_resume(struct device *dev)
+{
+	struct crg_gadget_dev *crg_udc = dev_get_drvdata(dev);
+	unsigned long flags;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+	if (crg_udc->gadget_driver && crg_udc->pullup_on)
+		crg_udc_set_pullup(crg_udc, true);
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+	return 0;
+}
+
+/* Program the speed limit while the controller is stopped. */
+int crg_udc_program_speed(struct crg_gadget_dev *crg_udc)
+{
+	u32 cfg, speed;
+
+	switch (crg_udc->max_speed) {
+	case USB_SPEED_FULL:
+		speed = CRG_U3DC_CFG0_MAXSPEED_FS;
+		break;
+	case USB_SPEED_HIGH:
+		speed = CRG_U3DC_CFG0_MAXSPEED_HS;
+		break;
+	case USB_SPEED_SUPER:
+		speed = CRG_U3DC_CFG0_MAXSPEED_SS;
+		break;
+	case USB_SPEED_SUPER_PLUS:
+		speed = CRG_U3DC_CFG0_MAXSPEED_SSP;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	cfg = readl(&crg_udc->uccr->config0);
+	cfg &= ~CRG_U3DC_CFG0_MAXSPEED_MASK;
+	cfg |= FIELD_PREP(CRG_U3DC_CFG0_MAXSPEED_MASK, speed);
+	writel(cfg, &crg_udc->uccr->config0);
+
+	return 0;
+}
+
+int crg_udc_reset_data_struct(struct crg_gadget_dev *crg_udc)
+{
+	int i;
+	int ret;
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 val;
+
+	crg_udc->ep0_state = EP0_STATE_IDLE;
+
+	val = readl(&uccr->control);
+	val &= (~(CRG_U3DC_CTRL_INT_EN | CRG_U3DC_CTRL_RUN));
+	writel(val, &uccr->control);
+
+	ret = crg_udc_program_speed(crg_udc);
+	if (ret)
+		return ret;
+
+	for (i = 0; i < CRG_RING_NUM; i++) {
+		ret = crg_udc_init_event_ring(crg_udc, i);
+		if (ret)
+			return ret;
+	}
+
+	ret = crg_udc_init_device_context(crg_udc);
+	if (ret)
+		return ret;
+
+	if (!crg_udc->status_req) {
+		struct usb_request *req;
+
+		req = crg_udc_alloc_request(&crg_udc->udc_ep[0].usb_ep,
+					    GFP_ATOMIC);
+		if (!req)
+			return -ENOMEM;
+
+		crg_udc->status_req =
+			container_of(req, struct crg_udc_request, usb_req);
+	}
+
+	/* other hw ops */
+	return 0;
+}
+
+static void crg_gadget_handle_interrupt(struct crg_gadget_dev *crg_udc)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	unsigned long flags = 0;
+	u32 tmp_status;
+	int ret;
+
+	spin_lock_irqsave(&crg_udc->udc_lock, flags);
+
+	tmp_status = readl(&uccr->status);
+
+	if (tmp_status & CRG_U3DC_STATUS_SYS_ERR) {
+		bool was_connected = crg_udc->connected;
+
+		dev_err(crg_udc->dev, "controller system error\n");
+		writel(CRG_U3DC_STATUS_SYS_ERR, &uccr->status);
+		ret = crg_udc_reset(crg_udc, &flags);
+		if (!ret)
+			ret = crg_udc_reset_data_struct(crg_udc);
+		if (!ret)
+			ret = crg_udc_init_ep0(crg_udc);
+		if (ret)
+			dev_err(crg_udc->dev,
+				"failed to recover from controller error: %d\n",
+				ret);
+		crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+		crg_udc->connected = 0;
+		if (was_connected)
+			crg_udc_notify_disconnect(crg_udc, &flags);
+		if (!ret && crg_udc->pullup_on)
+			crg_udc_set_pullup(crg_udc, true);
+
+		/*
+		 * tmp_status was captured before the reset above, which tore
+		 * down and rebuilt the event rings. It no longer reflects
+		 * the controller's current state, and if recovery failed the
+		 * rings may not be consistent at all. Skip the rest of this
+		 * interrupt's processing; any event still pending will raise
+		 * a fresh interrupt once the controller is running again.
+		 */
+		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+		return;
+	}
+
+	if (tmp_status & CRG_U3DC_STATUS_EINT) {
+		int i;
+
+		writel(CRG_U3DC_STATUS_EINT, &uccr->status);
+
+		for (i = 0; i < CRG_RING_NUM; i++) {
+			ret = crg_udc_process_event_ring(crg_udc, i, &flags);
+			if (ret)
+				dev_err(crg_udc->dev,
+					"process event ring error: %d\n", ret);
+		}
+	}
+
+	if (crg_udc->device_state == USB_STATE_RECONNECTING &&
+	    crg_udc->portsc_on_reconnecting == 1 &&
+	    crg_udc_is_event_rings_empty(crg_udc)) {
+		crg_udc->portsc_on_reconnecting = 0;
+		crg_udc_handle_port_status(crg_udc, &flags);
+	}
+
+	if (crg_udc->device_state == USB_STATE_RECONNECTING &&
+	    crg_udc->connected == 1)
+		crg_udc_prepare_for_setup(crg_udc);
+
+	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+}
+
+/*
+ * Primary handler only checks whether the shared interrupt belongs to the
+ * controller. All controller work, including reset polling, runs in the
+ * threaded handler and therefore never busy-waits in hard IRQ context.
+ */
+static irqreturn_t crg_udc_common_irq(int irq, void *dev)
+{
+	struct crg_gadget_dev *crg_udc = dev;
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 status = readl(&uccr->status);
+
+	if (!(status & (CRG_U3DC_STATUS_EINT | CRG_U3DC_STATUS_SYS_ERR)))
+		return IRQ_NONE;
+
+	return IRQ_WAKE_THREAD;
+}
+
+static irqreturn_t crg_udc_irq_thread(int irq, void *dev)
+{
+	struct crg_gadget_dev *crg_udc = dev;
+
+	crg_gadget_handle_interrupt(crg_udc);
+
+	return IRQ_HANDLED;
+}
+
+static int crg_gadget_irq_init(struct platform_device *pdev,
+			       struct crg_gadget_dev *crg_udc)
+{
+	int irq;
+	int ret;
+
+	irq = platform_get_irq(pdev, 0);
+	if (irq < 0)
+		return irq;
+
+	/*
+	 * IRQF_ONESHOT keeps this (level-triggered) line masked until
+	 * crg_udc_irq_thread() has cleared the status bits, which is the
+	 * correct pairing for IRQF_SHARED with a threaded handler.
+	 */
+	ret = devm_request_threaded_irq(&pdev->dev, irq, crg_udc_common_irq,
+					crg_udc_irq_thread,
+					IRQF_SHARED | IRQF_ONESHOT, "crg_udc",
+					crg_udc);
+	if (ret) {
+		dev_err(crg_udc->dev, "request of irq%d failed\n", irq);
+	} else {
+		crg_udc->irq = irq;
+		crg_udc->gadget.irq = irq;
+	}
+
+	return ret;
+}
+
+int crg_udc_reset(struct crg_gadget_dev *crg_udc, unsigned long *flags)
+{
+	struct crg_uccr __iomem *uccr = crg_udc->uccr;
+	u32 i, tmp;
+	struct crg_udc_ep *udc_ep_ptr;
+	u32 val;
+
+	val = readl(&uccr->control);
+	val |= CRG_U3DC_CTRL_SWRST;
+	writel(val, &uccr->control);
+	crg_udc->resetting = true;
+	spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+	/* Poll until SWRST clears; hardware reset takes up to 10 ms */
+	if (readl_poll_timeout_atomic(&uccr->control, tmp,
+			       !(tmp & CRG_U3DC_CTRL_SWRST), 10, 10000)) {
+		spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+		crg_udc->resetting = false;
+		dev_err(crg_udc->dev, "reset error\n");
+		return -ETIMEDOUT;
+	}
+	spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+	crg_udc->resetting = false;
+
+	crg_udc_clear_portpm(crg_udc);
+
+	crg_udc->setup_status = WAIT_FOR_SETUP;
+	/* Base on Figure 9-1, default USB_STATE is attached */
+	crg_udc->device_state = USB_STATE_ATTACHED;
+	crg_udc->dev_addr = 0;
+
+	/* Complete any reqs on EP1-EP15 queue */
+	for (i = 2; i < CRG_NUM_EP_CX; i++) {
+		udc_ep_ptr = &crg_udc->udc_ep[i];
+
+		if (udc_ep_ptr->desc) {
+			udc_ep_ptr->desc = NULL;
+			udc_ep_ptr->comp_desc = NULL;
+			crg_udc_ep_nuke(udc_ep_ptr, -ESHUTDOWN);
+		}
+		udc_ep_ptr->tran_ring_full = false;
+		udc_ep_ptr->ep_state = EP_STATE_DISABLED;
+		/*
+		 * This bypasses the generic usb_ep_disable() API, so the
+		 * framework's own usb_ep.enabled bookkeeping needs to be
+		 * cleared directly here too (crg_udc_reinit() already does
+		 * this for the live bus-reset case). Otherwise a later
+		 * usb_ep_enable() call for this endpoint short-circuits on
+		 * the stale "already enabled" flag without ever calling our
+		 * .enable callback, leaving desc/the ring never set up for
+		 * the new session.
+		 */
+		udc_ep_ptr->usb_ep.enabled = 0;
+	}
+	crg_udc->num_enabled_eps = 0;
+
+	/* Complete any reqs on EP0 queue */
+	udc_ep_ptr = &crg_udc->udc_ep[0];
+	if (udc_ep_ptr->desc) {
+		udc_ep_ptr->desc = NULL;
+		udc_ep_ptr->comp_desc = NULL;
+		crg_udc_ep_nuke(udc_ep_ptr, -ESHUTDOWN);
+	}
+
+	return 0;
+}
+
+static const struct crg_udc_priv crg_udc_data = {
+	.quirks = CRG_UDC_QUIRKS_DEFAULT,
+};
+
+static const struct of_device_id of_crg_udc_match[] = {
+	{ .compatible = "axiado,ax3000-udc", .data = &crg_udc_data },
+	{ },
+};
+MODULE_DEVICE_TABLE(of, of_crg_udc_match);
+
+static SIMPLE_DEV_PM_OPS(crg_udc_pm_ops, crg_udc_suspend, crg_udc_resume);
+
+static struct platform_driver crg_udc_driver = {
+	.probe = crg_udc_probe,
+	.remove = crg_udc_remove,
+	.shutdown = crg_udc_shutdown,
+	.driver = {
+		.name = KBUILD_MODNAME,
+		.of_match_table = of_crg_udc_match,
+		.pm = &crg_udc_pm_ops,
+	},
+};
+module_platform_driver(crg_udc_driver);
+
+MODULE_AUTHOR("Axiado Corporation");
+MODULE_DESCRIPTION("Axiado AX3000/AX3005 USB Device Controller");
+MODULE_LICENSE("GPL");

-- 
2.34.1

Re: [PATCH v3 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC

[sashiko-bot]

> Add a Devicetree binding for the USB device controller integrated into
> the Axiado AX3000 and AX3005 SoCs.
> 
> The controller is based on a Corigine USB 3.1 IP core that is
> dual-role capable (device and host), used here in device role. It
> provides SuperSpeed (5 Gb/s) or High-Speed (480 Mb/s) operation, and
> supports control, bulk, interrupt, and isochronous transfer types
> across multiple configurable endpoints.
> 
> Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
> Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
> Co-developed-by: Krutik Shah <krutikshah@axiado.com>
> Signed-off-by: Krutik Shah <krutikshah@axiado.com>
> Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261020-axiado-ax3000-usb-device-controller-v3-0-3012ecfc0a93@axiado.com?part=1

Re: [PATCH v3 2/6] usb: gadget: udc: crg: Add hardware definitions and structs

[sashiko-bot]

> This patch introduces the hardware definitions, register offsets, and
> fundamental data structures for the Axiado AX3000/AX3005 USB Device
> Controller driver.
> 
> Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
> Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
> Co-developed-by: Krutik Shah <krutikshah@axiado.com>
> Signed-off-by: Krutik Shah <krutikshah@axiado.com>
> Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>

Sashiko has reviewed this patch and found no issues.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261020-axiado-ax3000-usb-device-controller-v3-0-3012ecfc0a93@axiado.com?part=2

Re: [PATCH v3 3/6] usb: gadget: udc: crg: Add ring and transfer management

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider:
- [High] usb: gadget: crg: cycle bit corruption in crg_udc_squeeze_xfer_ring()
- [High] usb: gadget: crg: premature loop break in crg_udc_queue_pending_trbs()
- [High] usb: gadget: crg: unsafe queue iteration across crg_udc_req_done() lock drop
- [High] usb: gadget: udc: crg: stale request state in crg_udc_squeeze_xfer_ring()
- [High] usb: gadget: crg: race updating evt_dq_pt after lock drop in crg_udc_process_event_ring()
--

commit ac2a648308943ddb5f71e791a2ef85c73d0020b9
Author: Petar Stepanovic <pstepanovic@axiado.com>

usb: gadget: udc: crg: Add ring and transfer management

This patch adds the memory management and ring buffer logic for TRBs and
event rings used by the Axiado AX3000/AX3005 USB device controller.

> diff --git a/drivers/usb/gadget/udc/crg/crg-udc-ring.c b/drivers/usb/gadget/udc/crg/crg-udc-ring.c
> new file mode 100644
> index 0000000000000..1c57dc1787d42
> --- /dev/null
> +++ b/drivers/usb/gadget/udc/crg/crg-udc-ring.c

[ ... ]

> +int crg_udc_process_event_ring(struct crg_gadget_dev *crg_udc, int index,
> +			       unsigned long *flags)
> +{
> +	struct crg_event_trb *event;
> +	struct crg_udc_event *udc_event;
> +	struct crg_uicr __iomem *uicr = crg_udc->uicr[index];
> +	u32 val;
> +	dma_addr_t erdp;
> +	int ret;
> +	unsigned int processed = 0;
> +
> +	if (!uicr)
> +		return -ENODEV;
> +
> +	val = readl(&uicr->iman);
> +	val |= CRG_U3DC_IMAN_INT_PEND;
> +	writel(val, &uicr->iman);
> +
> +	udc_event = &crg_udc->udc_event[index];
> +	while (udc_event->evt_dq_pt) {
> +		event = (struct crg_event_trb *)udc_event->evt_dq_pt;
> +
> +		if (FIELD_GET(EVE_TRB_CYCLE_BIT_MASK,
> +			      le32_to_cpu(event->dw3)) != udc_event->ccs)
> +			break;
> +		/* Read the payload only after observing device ownership. */
> +		dma_rmb();
> +
> +		ret = crg_udc_handle_event(crg_udc, event, flags);
> +		if (ret == -ECONNRESET) {
> +			/* The disconnect path rebuilt and reprogrammed the ring. */
> +			return 0;
> +		}
> +		if (ret < 0)
> +			dev_dbg(crg_udc->dev, "failed to process event: %d\n",
> +				ret);
> +
> +		/* Consume malformed events so they cannot pin the ring head. */
> +		if (event == udc_event->evt_seg0_last_trb) {
> +			udc_event->ccs = udc_event->ccs ? 0 : 1;
> +			udc_event->evt_dq_pt = udc_event->event_ring.cpu_addr;
> +		} else {
> +			udc_event->evt_dq_pt++;
> +		}

[Severity: High]
Can crg_udc_process_event_ring() corrupt udc_event->evt_dq_pt and
udc_event->ccs if the event ring is reinitialized while crg_udc->udc_lock
is dropped inside crg_udc_handle_event()?

Several paths inside crg_udc_handle_event() drop and re-acquire udc_lock
and then return 0 (for example, crg_udc_req_done() during transfer
completion, or crg_udc_handle_setup_pkt() during flush_work() and
driver->setup()).

If crg_gadget_stop() runs concurrently on another CPU during that lock
drop, it acquires udc_lock and calls:

crg_gadget_stop()
  crg_udc_reset_data_struct()
    crg_udc_init_event_ring()

which zeroes the event ring, resets udc_event->evt_dq_pt to
event_ring.cpu_addr, sets udc_event->ccs to 1, and programs uicr->erdplo
and uicr->erdphi to the start of the ring.

When crg_udc_handle_event() re-acquires udc_lock and returns 0,
crg_udc_process_event_ring() uses the stale pre-call event pointer to
advance udc_event->evt_dq_pt to index 1 (or, if event was
evt_seg0_last_trb, toggles udc_event->ccs from 1 to 0, causing the
subsequent loop iterations to treat all zeroed TRBs in the freshly cleared
ring as valid events).

[ ... ]

> +int crg_udc_queue_pending_trbs(struct crg_udc_ep *udc_ep_ptr)
> +{
> +	struct crg_udc_request *udc_req_ptr, *tmp;
> +	int ret;
> +
> +	list_for_each_entry_safe(udc_req_ptr, tmp, &udc_ep_ptr->queue, queue) {
> +		if (udc_req_ptr->all_trbs_queued)
> +			continue;
> +
> +		ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
> +		if (ret == -ENOSPC || ret == -EBUSY) {
> +			/*
> +			 * Ring is full; leave this request queued and retry
> +			 * it (and anything behind it) on the next TRB
> +			 * completion via crg_udc_queue_pending_trbs().
> +			 */
> +			return ret;
> +		} else if (ret) {
> +			/*
> +			 * A non-capacity failure is not transient: fail this
> +			 * request instead of leaving it stuck forever, and
> +			 * keep rebuilding the rest of the queue.
> +			 */
> +			dev_err(udc_ep_ptr->crg_udc->dev,
> +				"failed to build pending transfer: %d\n", ret);
> +			list_del_init(&udc_req_ptr->queue);
> +			crg_udc_req_done(udc_ep_ptr, udc_req_ptr, ret);
> +			continue;
> +		}

[Severity: High]
Is it safe to continue list_for_each_entry_safe() after calling
crg_udc_req_done() here (and in crg_udc_squeeze_xfer_ring() below)?

As documented on crg_udc_req_done() in crg-udc.h, crg_udc_req_done() drops
crg_udc->udc_lock around usb_gadget_giveback_request() and re-acquires it
before returning.

While udc_lock is dropped, the completion callback or another CPU (via
crg_udc_ep_dequeue(), crg_udc_ep_disable(), or crg_udc_ep_nuke()) can
unlink via list_del_init() and free the pre-fetched tmp request. Resuming
iteration with the cached tmp pointer can then trigger a use-after-free or
spin forever if tmp->queue was reinitialized with list_del_init().

> +		break;
> +	}

[Severity: High]
Should this unconditional break instead check udc_ep_ptr->tran_ring_full
like crg_udc_squeeze_xfer_ring() does?

With an unconditional break after the first crg_udc_build_td() succeeds,
crg_udc_queue_pending_trbs() schedules at most one pending request per
call even when space remains on the transfer ring. If multiple requests
were queued while the ring was full and the first rebuilt request has
usb_req.no_interrupt set, no completion interrupt will fire and the
remaining pending requests in udc_ep_ptr->queue will stall.

[ ... ]

> +void crg_udc_setup_link_trb(struct crg_transfer_trb *link_trb, bool toggle,
> +			    dma_addr_t next_trb)
> +{
> +	u32 dw = 0;
> +
> +	link_trb->dw0 = cpu_to_le32(lower_32_bits(next_trb));
> +	link_trb->dw1 = cpu_to_le32(upper_32_bits(next_trb));
> +	link_trb->dw2 = cpu_to_le32(0);
> +
> +	FIELD_MODIFY(TRB_TYPE_MASK, &dw, TRB_TYPE_LINK);
> +	if (toggle)
> +		FIELD_MODIFY(TRB_LINK_TOGGLE_CYCLE_MASK, &dw, 1);
> +	else
> +		FIELD_MODIFY(TRB_LINK_TOGGLE_CYCLE_MASK, &dw, 0);
> +
> +	/* Publish ownership only after the link address is visible. */
> +	dma_wmb();
> +	link_trb->dw3 = cpu_to_le32(dw);
> +}

[ ... ]

> +void crg_udc_squeeze_xfer_ring(struct crg_udc_ep *udc_ep_ptr,
> +			       struct crg_udc_request *udc_req_ptr)
> +{
> +	struct crg_transfer_trb *trb = udc_req_ptr->first_trb;
> +	struct crg_udc_request *next_req, *tmp;
> +	u8 new_pcs;
> +
> +	if (!trb)
> +		return;
> +
> +	/* The canceled request becomes the new enqueue point. Its original
> +	 * cycle bit is the cycle expected at that point after the tail is
> +	 * removed.
> +	 */
> +	new_pcs = FIELD_GET(TRB_CYCLE_BIT_MASK, le32_to_cpu(trb->dw3));
> +
> +	while (trb != udc_ep_ptr->enq_pt) {
> +		memset(trb, 0, sizeof(*trb));
> +		trb++;
> +		if (trb == udc_ep_ptr->last_trb)
> +			trb = udc_ep_ptr->first_trb;
> +	}
> +
> +	udc_ep_ptr->enq_pt = udc_req_ptr->first_trb;
> +	udc_ep_ptr->pcs = new_pcs;
> +	udc_ep_ptr->tran_ring_full = false;
> +	crg_udc_setup_link_trb(udc_ep_ptr->last_trb, true,
> +			udc_ep_ptr->tran_ring_info.dma_addr);
> +	dma_wmb();

[Severity: High]
Does zeroing the TRBs and unconditionally resetting udc_ep_ptr->last_trb
corrupt the transfer ring cycle state when new_pcs is 0 or when an active
request wraps the ring?

Because the link TRB toggles the cycle state on each ring wrap, the
expected producer/consumer cycle state alternates between 1 and 0.

First, memset(trb, 0, sizeof(*trb)) clears TRB_CYCLE_BIT_MASK to 0 and
TRB_TYPE_MASK to 0 (TRB_TYPE_RSVD). On a ring pass where new_pcs == 0 (or
if the cleared range wraps past last_trb into a pcs == 0 pass), a cycle
bit of 0 marks the cleared slots as hardware-owned rather than unowned.
If udc_req_ptr was the last queued request (or rebuilt requests consume
fewer TRBs than were wiped), the TRB at the rewound udc_ep_ptr->enq_pt will
have cycle bit 0 == udc_ep_ptr->pcs and type TRB_TYPE_RSVD, allowing the
controller to fetch a zeroed TRB instead of stopping at enq_pt.

Second, crg_udc_setup_link_trb() always writes dw3 with cycle bit 0 and
chain bit 0. Calling it unconditionally on udc_ep_ptr->last_trb will
overwrite last_trb's cycle bit and chain bit even if an earlier,
still-active request ahead of udc_req_ptr in udc_ep_ptr->queue wrapped
across last_trb during a pcs == 1 pass, invalidating the link TRB before
hardware consumes it.

> +	if (list_is_last(&udc_req_ptr->queue, &udc_ep_ptr->queue))
> +		return;
> +
> +	next_req = udc_req_ptr;
> +	list_for_each_entry_safe_continue(next_req, tmp, &udc_ep_ptr->queue,
> +					  queue) {
> +		int ret;
> +
> +		next_req->usb_req.status = -EINPROGRESS;
> +		next_req->usb_req.actual = 0;
> +		crg_udc_clear_req_container(next_req);
> +		ret = crg_udc_build_td(udc_ep_ptr, next_req);
> +		if (ret == -ENOSPC) {
> +			/* Ring is full; a later TRB completion will retry
> +			 * this and any remaining requests via
> +			 * crg_udc_queue_pending_trbs().
> +			 */
> +			udc_ep_ptr->tran_ring_full = true;
> +			break;
> +		} else if (ret) {
> +			/*
> +			 * A non-capacity failure is not transient: fail this
> +			 * request instead of leaving it stuck forever, and
> +			 * keep rebuilding the rest of the queue.
> +			 */
> +			dev_err(udc_ep_ptr->crg_udc->dev,
> +				"failed to rebuild queued request: %d\n", ret);
> +			crg_udc_req_done(udc_ep_ptr, next_req, ret);
> +			continue;
> +		}
> +		if (udc_ep_ptr->tran_ring_full)
> +			break;
> +	}
> +}

[Severity: High]
What happens to the remaining requests in udc_ep_ptr->queue if this loop
breaks out early when the ring fills up?

The while (trb != udc_ep_ptr->enq_pt) loop above already wiped the hardware
TRBs for all requests after udc_req_ptr, but crg_udc_clear_req_container()
is only called inside list_for_each_entry_safe_continue() right before each
crg_udc_build_td() call.

If an earlier request fills the ring and hits either break, any subsequent
requests in udc_ep_ptr->queue that previously had TRBs queued are left with
all_trbs_queued == true, buff_len_left == 0, and stale first_trb/last_trb
pointers. When a later TRB completion runs crg_udc_queue_pending_trbs(), it
checks:

	if (udc_req_ptr->all_trbs_queued)
		continue;

and skips those requests permanently.

Also, for isochronous endpoints, crg_udc_build_td() -> crg_udc_queue_trbs()
returns -EBUSY when the ring lacks space. Unlike crg_udc_queue_pending_trbs()
which checks for both -ENOSPC and -EBUSY, this check only tests for
ret == -ENOSPC, so -EBUSY falls into the error branch and fails the request
via crg_udc_req_done().

[ ... ]

完美

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261020-axiado-ax3000-usb-device-controller-v3-0-3012ecfc0a93@axiado.com?part=3

Re: [PATCH v3 6/6] usb: gadget: udc: crg: Add core driver and build integration

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 8 potential issue(s) to consider:
- [High] usb: gadget: crg: crg_udc_reset() clears ep->enabled before disconnect()
- [High] usb: gadget: crg: premature resetting clear and missing checks in crg_udc_reset()
- [Medium] usb: gadget: crg: premature IRQ and data init before ep setup in crg_udc_probe()
- [Medium] usb: gadget: crg: premature unlocked state reset in crg_udc_remove()
- [Medium] usb: gadget: crg: race in crg_udc_shutdown() on uccr->control and pullup_on
- [Medium] usb: gadget: udc: crg: 32-bit dma_addr_t shift overflow under COMPILE_TEST
- [Medium] usb: gadget: crg: crg_udc_issue_command() overwrites pending command when RUN is 0
- [Low] usb: gadget: udc: crg: deprecated SIMPLE_DEV_PM_OPS and missing pm_sleep_ptr()
--

commit 0ef71bda637190f9d2012396d35661e2929b2b80
Author: Petar Stepanovic <pstepanovic@axiado.com>

usb: gadget: udc: crg: Add core driver and build integration

This patch adds the core probe, remove, reset, and interrupt handling
functions for the Axiado AX3000/AX3005 UDC driver and wires the driver
into Kconfig and Makefile.

> diff --git a/drivers/usb/gadget/udc/crg/Kconfig b/drivers/usb/gadget/udc/crg/Kconfig
> new file mode 100644
> index 0000000000000..7167c4943389b
> --- /dev/null
> +++ b/drivers/usb/gadget/udc/crg/Kconfig
> @@ -0,0 +1,16 @@
> +# SPDX-License-Identifier: GPL-2.0
> +config USB_CRG_UDC
> +	tristate "Corigine / Axiado USB Device Controller"
> +	depends on USB_GADGET
> +	depends on HAS_DMA
> +	depends on ARCH_AXIADO || COMPILE_TEST

[Severity: Medium]
Can enabling COMPILE_TEST here break 32-bit builds where dma_addr_t is
32 bits wide (CONFIG_ARCH_DMA_ADDR_T_64BIT=n)?

In drivers/usb/gadget/udc/crg/crg-udc-gadget.c, crg_udc_ep_dequeue() and
crg_udc_handle_xfer_event() shift values cast to dma_addr_t left by 32 bits:

drivers/usb/gadget/udc/crg/crg-udc-gadget.c:crg_udc_ep_dequeue() {
	...
	dq_pt_addr = (dma_addr_t)deq_pt_lo | ((dma_addr_t)deq_pt_hi << 32);
	...
}

drivers/usb/gadget/udc/crg/crg-udc-gadget.c:crg_udc_handle_xfer_event() {
	...
		trb = crg_udc_tran_trb_dma_to_virt(
			udc_ep_ptr,
			(dma_addr_t)le32_to_cpu(event->dw0) |
				((dma_addr_t)le32_to_cpu(event->dw1) << 32));
	...
}

On 32-bit architectures without 64-bit dma_addr_t, shifting a 32-bit type
by 32 bits triggers -Wshift-count-overflow. Should those shifts cast to u64
first (like is_request_dequeued() and crg_udc_update_dequeue_pt() already
do)?

[ ... ]

> diff --git a/drivers/usb/gadget/udc/crg/crg-udc-core.c b/drivers/usb/gadget/udc/crg/crg-udc-core.c
> new file mode 100644
> index 0000000000000..80a4fc6a61e88
> --- /dev/null
> +++ b/drivers/usb/gadget/udc/crg/crg-udc-core.c
> @@ -0,0 +1,706 @@

[ ... ]

> +/*
> + * command related ops
> + */
> +int crg_udc_issue_command(struct crg_gadget_dev *crg_udc,
> +			  enum crg_cmd_type type, u32 param0, u32 param1)
> +{
> +	struct crg_uccr __iomem *uccr = crg_udc->uccr;
> +	u32 status;
> +	bool check_complete = false;
> +	u32 tmp;
> +
> +	/*
> +	 * The command state machine only runs while the controller itself is
> +	 * running. Once RUN is cleared (e.g. during udc_stop()/reset()), the
> +	 * hardware never processes queued commands, so ACTIVE never clears.
> +	 * Skip the busy/completion checks in that case instead of getting
> +	 * stuck on a command the hardware will never service.
> +	 */
> +	tmp = readl(&uccr->control);
> +	if (tmp & CRG_U3DC_CTRL_RUN)
> +		check_complete = true;
> +
> +	if (check_complete) {
> +		tmp = readl(&uccr->cmd_control);
> +		if (tmp & CRG_U3DC_CMD_CTRL_ACTIVE) {
> +			dev_err(crg_udc->dev,
> +				"previous command is not complete\n");
> +			return -EBUSY;
> +		}
> +	}
> +	/* Ensure that everything is written before issuing new command */
> +	dma_wmb();
> +
> +	writel(param0, &uccr->cmd_param0);
> +	writel(param1, &uccr->cmd_param1);
> +
> +	tmp = CRG_U3DC_CMD_CTRL_ACTIVE |
> +	      FIELD_PREP(CRG_U3DC_CMD_CTRL_TYPE_MASK, type);
> +	writel(tmp, &uccr->cmd_control);
> +
> +	if (!check_complete)
> +		return 0;

[Severity: Medium]
What happens when multiple commands are issued while CRG_U3DC_CTRL_RUN is 0,
or when a command is issued immediately after CRG_U3DC_CTRL_RUN is asserted?

Because the controller has a single set of command registers
(uccr->cmd_param0, uccr->cmd_param1, uccr->cmd_control) and the hardware
does not clear CRG_U3DC_CMD_CTRL_ACTIVE while CRG_U3DC_CTRL_RUN is 0:

1. crg_udc_init_ep0() issues CRG_CMD_INIT_EP0 while CRG_U3DC_CTRL_RUN is 0
   in crg_udc_probe(), crg_gadget_stop(), crg_udc_handle_port_status() on
   disconnect, and crg_gadget_handle_interrupt() on CRG_U3DC_STATUS_SYS_ERR,
   leaving CRG_CMD_INIT_EP0 pending in uccr->cmd_control. Similarly,
   crg_udc_set_pullup(crg_udc, false) in crg_gadget_pullup() and
   crg_udc_suspend() clears CRG_U3DC_CTRL_RUN while endpoints remain
   enabled.

2. If another command is issued before CRG_U3DC_CTRL_RUN is set (for
   example via crg_udc_ep_enable(), crg_udc_ep_dequeue() issuing
   CRG_CMD_STOP_EP followed by CRG_CMD_SET_TR_DQPTR, or crg_udc_ep_halt()
   issuing CRG_CMD_RESET_SEQNUM, CRG_CMD_CLEAR_HALT, and
   CRG_CMD_SET_TR_DQPTR back-to-back), check_complete is false and
   crg_udc_issue_command() overwrites the unserviced command in
   uccr->cmd_control.

3. When crg_udc_set_pullup(crg_udc, true) later sets CRG_U3DC_CTRL_RUN, it
   does not wait for a pending CRG_U3DC_CMD_CTRL_ACTIVE command to finish,
   so a command issued right after RUN is set can see
   CRG_U3DC_CMD_CTRL_ACTIVE still set and fail with -EBUSY.

Can this cause CRG_CMD_INIT_EP0 or endpoint commands to be lost or fail
with -EBUSY?

[ ... ]

> +static int crg_udc_probe(struct platform_device *pdev)
> +{

[ ... ]

> +	spin_lock_irqsave(&crg_udc->udc_lock, flags);
> +	ret = crg_udc_reset(crg_udc, &flags);
> +	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +	if (ret)
> +		goto err_free_resources;
> +
> +	crg_udc_clear_portpm(crg_udc);
> +
> +	ret = crg_gadget_irq_init(pdev, crg_udc);
> +	if (ret)
> +		goto err_free_resources;
> +
> +	ret = crg_udc_reset_data_struct(crg_udc);
> +	if (ret) {
> +		dev_err(crg_udc->dev, "crg_udc_reset_data_struct error\n");
> +		goto err_free_resources;
> +	}
> +
> +	crg_udc_init_ep_info(crg_udc);
> +	ret = crg_udc_init_ep0(crg_udc);
> +	if (ret)
> +		goto err_free_resources;

[Severity: Medium]
Should crg_udc_init_ep_info(), crg_udc_reset_data_struct(), and
crg_udc_init_ep0() run before crg_gadget_irq_init(), with
crg_udc_init_ep_info() called before crg_udc_reset_data_struct()?

With the current ordering in crg_udc_probe():

1. crg_udc_reset_data_struct() runs before crg_udc_init_ep_info() has
   initialized crg_udc->udc_ep[0], and calls:

   crg_udc_reset_data_struct()
     -> crg_udc_alloc_request(&crg_udc->udc_ep[0].usb_ep, GFP_ATOMIC)
       -> trace_crg_udc_alloc_request(udc_ep, udc_req_ptr)

   where udc_ep[0].usb_ep.name, ops, and crg_udc are still NULL.

2. crg_gadget_irq_init() registers the shared IRQ (IRQF_SHARED |
   IRQF_ONESHOT) before crg_udc_reset_data_struct(), crg_udc_init_ep_info(),
   and crg_udc_init_ep0() have initialized the event rings, endpoint list
   heads, and EP0 transfer ring. Because crg_udc_common_irq() only checks
   uccr->status for CRG_U3DC_STATUS_EINT | CRG_U3DC_STATUS_SYS_ERR without
   checking CRG_U3DC_CTRL_INT_EN, an interrupt arriving in this window runs
   crg_gadget_handle_interrupt() under spin_lock_irqsave(&crg_udc->udc_lock,
   flags) while udc_ep[0].tran_ring_info.cpu_addr is still NULL, causing
   crg_udc_init_ep0() to call dma_alloc_coherent(..., GFP_KERNEL) while
   holding the spinlock:

drivers/usb/gadget/udc/crg/crg-udc-ep0.c:crg_udc_init_ep0() {
	...
	if (!udc_ep_ptr->tran_ring_info.cpu_addr) {
		...
		vaddr = dma_alloc_coherent(crg_udc->dev, len, &dma, GFP_KERNEL);
	...
}

> +
> +	crg_udc->udc_ep[0].desc = &crg_udc->ep0_desc;
> +
> +	ret = usb_add_gadget(&crg_udc->gadget);
> +	if (ret)
> +		goto err_free_resources;
> +
> +	return 0;
> +
> +err_free_resources:
> +	crg_udc_free_resources(crg_udc);
> +err_put_gadget:
> +	usb_put_gadget(&crg_udc->gadget);
> +	return ret;
> +}
> +
> +static void crg_udc_remove(struct platform_device *pdev)
> +{
> +	struct crg_gadget_dev *crg_udc = platform_get_drvdata(pdev);
> +	u32 tmp;
> +
> +	crg_udc->device_state = USB_STATE_ATTACHED;
> +	crg_udc_set_vbus_detect(crg_udc, false);
> +
> +	usb_del_gadget(&crg_udc->gadget);
> +	cancel_work_sync(&crg_udc->udc_reset_work);
> +	crg_udc_free_resources(crg_udc);

[Severity: Medium]
In crg_udc_remove(), why are crg_udc->device_state and VBUS detection
updated before usb_del_gadget(&crg_udc->gadget) and without holding
crg_udc->udc_lock?

While a gadget driver is bound and active, the controller and its interrupt
handler remain active until usb_del_gadget() unbinds the gadget driver and
invokes crg_gadget_stop() (which already sets crg_udc->device_state =
USB_STATE_ATTACHED and calls crg_udc_set_vbus_detect(crg_udc, false)).

Setting crg_udc->device_state = USB_STATE_ATTACHED before usb_del_gadget()
causes any concurrent crg_udc_handle_event() execution for
TRB_TYPE_EVT_TRANSFER to see crg_udc->device_state < USB_STATE_RECONNECTING
and drop in-flight transfer completion events:

drivers/usb/gadget/udc/crg/crg-udc-gadget.c:crg_udc_handle_event() {
	...
	case TRB_TYPE_EVT_TRANSFER:
		if (crg_udc->device_state < USB_STATE_RECONNECTING) {
			dev_err(crg_udc->dev,
				"Xfer compl event rcved when dev state=%d !\n",
				crg_udc->device_state);
			break;
		}

		ret = crg_udc_handle_xfer_event(crg_udc, event, flags);
	...
}

> +
> +	/* set controller host role */
> +	tmp = readl(crg_udc->mmio_virt_base + CRG_UDC_MODE_REG) & ~0x1;
> +	writel(tmp, crg_udc->mmio_virt_base + CRG_UDC_MODE_REG);
> +
> +	platform_set_drvdata(pdev, NULL);
> +	usb_put_gadget(&crg_udc->gadget);
> +}
> +
> +static void crg_udc_shutdown(struct platform_device *pdev)
> +{
> +	struct crg_gadget_dev *crg_udc = platform_get_drvdata(pdev);
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&crg_udc->udc_lock, flags);
> +	crg_udc->device_state = USB_STATE_ATTACHED;
> +	crg_udc_set_pullup(crg_udc, false);
> +	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +	crg_udc_set_vbus_detect(crg_udc, false);
> +	if (crg_udc->uccr) {
> +		u32 val = readl(&crg_udc->uccr->control);
> +
> +		val &= ~CRG_U3DC_CTRL_INT_EN;
> +		writel(val, &crg_udc->uccr->control);
> +	}
> +	if (crg_udc->irq >= 0)
> +		synchronize_irq(crg_udc->irq);
> +}

[Severity: Medium]
Can crg_udc_shutdown() race with crg_gadget_handle_interrupt() and leave
the controller running with interrupts enabled?

In crg_udc_shutdown():
1. crg_udc_set_pullup(crg_udc, false) clears CRG_U3DC_CTRL_RUN, but
   crg_udc->pullup_on is not cleared.
2. &crg_udc->uccr->control is read-modify-written to clear
   CRG_U3DC_CTRL_INT_EN after dropping crg_udc->udc_lock and before
   synchronize_irq(crg_udc->irq).
3. If crg_udc_irq_thread() -> crg_gadget_handle_interrupt() is running
   concurrently (for instance, recovering from CRG_U3DC_STATUS_SYS_ERR where
   crg_udc_reset() drops and reacquires udc_lock), it observes
   crg_udc->pullup_on == true and calls crg_udc_set_pullup(crg_udc, true),
   which sets CRG_U3DC_CTRL_SYSERR_EN | CRG_U3DC_CTRL_INT_EN |
   CRG_U3DC_CTRL_RUN in uccr->control again.
4. crg_udc->udc_reset_work is not cancelled with cancel_work_sync().

Should crg_udc->pullup_on = false and the CRG_U3DC_CTRL_INT_EN clear be
done under crg_udc->udc_lock, followed by synchronize_irq() and
cancel_work_sync(&crg_udc->udc_reset_work)?

[ ... ]

> +static void crg_gadget_handle_interrupt(struct crg_gadget_dev *crg_udc)
> +{
> +	struct crg_uccr __iomem *uccr = crg_udc->uccr;
> +	unsigned long flags = 0;
> +	u32 tmp_status;
> +	int ret;
> +
> +	spin_lock_irqsave(&crg_udc->udc_lock, flags);
> +
> +	tmp_status = readl(&uccr->status);
> +
> +	if (tmp_status & CRG_U3DC_STATUS_SYS_ERR) {
> +		bool was_connected = crg_udc->connected;
> +
> +		dev_err(crg_udc->dev, "controller system error\n");
> +		writel(CRG_U3DC_STATUS_SYS_ERR, &uccr->status);
> +		ret = crg_udc_reset(crg_udc, &flags);
> +		if (!ret)
> +			ret = crg_udc_reset_data_struct(crg_udc);
> +		if (!ret)
> +			ret = crg_udc_init_ep0(crg_udc);
> +		if (ret)
> +			dev_err(crg_udc->dev,
> +				"failed to recover from controller error: %d\n",
> +				ret);
> +		crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
> +		crg_udc->connected = 0;
> +		if (was_connected)
> +			crg_udc_notify_disconnect(crg_udc, &flags);

[Severity: High]
Does calling crg_udc_reset() before crg_udc_notify_disconnect() here
prevent gadget function drivers from cleaning up on disconnect?

In crg_udc_reset(), udc_ep_ptr->usb_ep.enabled is cleared to 0 for all
non-EP0 endpoints:

	for (i = 2; i < CRG_NUM_EP_CX; i++) {
		...
		udc_ep_ptr->ep_state = EP_STATE_DISABLED;
		udc_ep_ptr->usb_ep.enabled = 0;
	}

When crg_udc_notify_disconnect() then invokes
crg_udc->gadget_driver->disconnect() -> composite_disconnect() ->
reset_config(), function drivers whose ->disable() callbacks guard teardown
on ep->enabled (such as rndis_disable() checking rndis->notify->enabled, or
ecm_disable() and eem_disable() checking port.in_ep->enabled) see
enabled == 0 and return without calling rndis_uninit() or
gether_disconnect(). On subsequent re-enumeration, their ->set_alt()
callbacks also check in_ep->enabled before calling gether_disconnect(),
leaking the previous session's usb_request objects.

Should crg_udc_notify_disconnect() be called before crg_udc_reset(), matching
the disconnect path in crg_udc_handle_port_status()?

[ ... ]

> +int crg_udc_reset(struct crg_gadget_dev *crg_udc, unsigned long *flags)
> +{
> +	struct crg_uccr __iomem *uccr = crg_udc->uccr;
> +	u32 i, tmp;
> +	struct crg_udc_ep *udc_ep_ptr;
> +	u32 val;
> +
> +	val = readl(&uccr->control);
> +	val |= CRG_U3DC_CTRL_SWRST;
> +	writel(val, &uccr->control);
> +	crg_udc->resetting = true;
> +	spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
> +	/* Poll until SWRST clears; hardware reset takes up to 10 ms */
> +	if (readl_poll_timeout_atomic(&uccr->control, tmp,
> +			       !(tmp & CRG_U3DC_CTRL_SWRST), 10, 10000)) {
> +		spin_lock_irqsave(&crg_udc->udc_lock, *flags);
> +		crg_udc->resetting = false;
> +		dev_err(crg_udc->dev, "reset error\n");
> +		return -ETIMEDOUT;
> +	}
> +	spin_lock_irqsave(&crg_udc->udc_lock, *flags);
> +	crg_udc->resetting = false;
> +
> +	crg_udc_clear_portpm(crg_udc);
> +
> +	crg_udc->setup_status = WAIT_FOR_SETUP;
> +	/* Base on Figure 9-1, default USB_STATE is attached */
> +	crg_udc->device_state = USB_STATE_ATTACHED;
> +	crg_udc->dev_addr = 0;
> +
> +	/* Complete any reqs on EP1-EP15 queue */
> +	for (i = 2; i < CRG_NUM_EP_CX; i++) {
> +		udc_ep_ptr = &crg_udc->udc_ep[i];
> +
> +		if (udc_ep_ptr->desc) {
> +			udc_ep_ptr->desc = NULL;
> +			udc_ep_ptr->comp_desc = NULL;
> +			crg_udc_ep_nuke(udc_ep_ptr, -ESHUTDOWN);
> +		}
> +		udc_ep_ptr->tran_ring_full = false;
> +		udc_ep_ptr->ep_state = EP_STATE_DISABLED;
> +		/*
> +		 * This bypasses the generic usb_ep_disable() API, so the
> +		 * framework's own usb_ep.enabled bookkeeping needs to be
> +		 * cleared directly here too (crg_udc_reinit() already does
> +		 * this for the live bus-reset case). Otherwise a later
> +		 * usb_ep_enable() call for this endpoint short-circuits on
> +		 * the stale "already enabled" flag without ever calling our
> +		 * .enable callback, leaving desc/the ring never set up for
> +		 * the new session.
> +		 */
> +		udc_ep_ptr->usb_ep.enabled = 0;
> +	}
> +	crg_udc->num_enabled_eps = 0;
> +
> +	/* Complete any reqs on EP0 queue */
> +	udc_ep_ptr = &crg_udc->udc_ep[0];
> +	if (udc_ep_ptr->desc) {
> +		udc_ep_ptr->desc = NULL;
> +		udc_ep_ptr->comp_desc = NULL;
> +		crg_udc_ep_nuke(udc_ep_ptr, -ESHUTDOWN);
> +	}
> +
> +	return 0;
> +}

[Severity: High]
Can dropping crg_udc->udc_lock during crg_udc_reset() allow other threads or
completion callbacks to touch registers and ring doorbells while the
controller is mid-reset or before its rings and contexts are reinitialized?

There appear to be two windows here when crg_udc_reset() is called at
runtime (from crg_gadget_handle_interrupt(), crg_udc_handle_port_status(),
or crg_gadget_stop()):

1. While udc_lock is dropped polling for CRG_U3DC_CTRL_SWRST to clear, all
   endpoints still have desc != NULL and ep_state == EP_STATE_RUNNING, and
   only crg_udc_ep_queue() checks crg_udc->resetting. Concurrent calls to
   crg_udc_ep_dequeue(), crg_udc_ep_disable(), crg_udc_ep_enable(),
   crg_udc_ep_set_halt(), crg_udc_ep_set_wedge(), crg_gadget_pullup(),
   crg_gadget_wakeup(), crg_udc_suspend(), or crg_udc_resume() can acquire
   udc_lock and issue commands or write registers while SWRST is active.

2. crg_udc->resetting is set back to false immediately after SWRST clears,
   before the loop over udc_ep[2..CRG_NUM_EP_CX - 1] and udc_ep[0], and
   before the caller runs crg_udc_reset_data_struct() and
   crg_udc_init_ep0(). Inside the loop, crg_udc_ep_nuke() ->
   crg_udc_req_done() drops udc_lock around usb_gadget_giveback_request()
   while crg_udc->resetting is already false and remaining endpoints j > i
   (as well as udc_ep[0]) still have desc != NULL:

drivers/usb/gadget/udc/crg/crg-udc-gadget.c:crg_udc_ep_queue() {
	...
	if (udc_ep_ptr->desc != desc || !udc_req_ptr->usb_req.complete ||
	    !list_empty(&udc_req_ptr->queue) || crg_udc->resetting) {
	...
}

   A completion callback or another CPU waiting in crg_udc_ep_queue() for a
   later endpoint j > i (or EP0) will see both udc_ep_ptr->desc == desc and
   crg_udc->resetting == false, and will queue TRBs and ring the doorbell on
   a wiped controller before crg_udc_reset_data_struct() and
   crg_udc_init_ep0() have reprogrammed DCBAP, the event rings, and EP0.

> +
> +static const struct crg_udc_priv crg_udc_data = {
> +	.quirks = CRG_UDC_QUIRKS_DEFAULT,
> +};
> +
> +static const struct of_device_id of_crg_udc_match[] = {
> +	{ .compatible = "axiado,ax3000-udc", .data = &crg_udc_data },
> +	{ },
> +};
> +MODULE_DEVICE_TABLE(of, of_crg_udc_match);
> +
> +static SIMPLE_DEV_PM_OPS(crg_udc_pm_ops, crg_udc_suspend, crg_udc_resume);
> +
> +static struct platform_driver crg_udc_driver = {
> +	.probe = crg_udc_probe,
> +	.remove = crg_udc_remove,
> +	.shutdown = crg_udc_shutdown,
> +	.driver = {
> +		.name = KBUILD_MODNAME,
> +		.of_match_table = of_crg_udc_match,
> +		.pm = &crg_udc_pm_ops,
> +	},
> +};

[Severity: Low]
SIMPLE_DEV_PM_OPS() is deprecated in include/linux/pm.h. Could this use
DEFINE_SIMPLE_DEV_PM_OPS(crg_udc_pm_ops, crg_udc_suspend, crg_udc_resume)
with pm_sleep_ptr(&crg_udc_pm_ops) (and drop __maybe_unused from
crg_udc_suspend() and crg_udc_resume()) so the PM callbacks and ops struct
are omitted when CONFIG_PM_SLEEP is disabled?

[ ... ]

>
endasi

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261020-axiado-ax3000-usb-device-controller-v3-0-3012ecfc0a93@axiado.com?part=6

Re: [PATCH v3 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 12 potential issue(s) to consider:
- [Critical] usb: gadget: crg: TRB interleaving, premature ZLP, and stall on multi-TD rings
- [Critical] usb: gadget: crg: dangling ring pointers and unlocked free in crg_udc_ep_enable()
- [Critical] usb: gadget: crg: race and missing async_callbacks check in crg_udc_reset_work()
- [High] usb: gadget: crg: broken last_trb calculation in crg_udc_queue_trbs()
- [High] usb: gadget: crg: broken CMPL_CODE_SHORT_PKT handling in crg_udc_handle_xfer_event()
- [High] usb: gadget: crg: wrong dequeue cycle state and stalled ring on dequeue/unhalt
- [High] usb: gadget: crg: missing hardware disable wait and lock-drop race in crg_udc_ep_disable()
- [High] usb: gadget: crg: stale state updates after lock drop in crg_udc_req_done()
- [High] usb: gadget: crg: High-Speed maxburst always 0 in crg_udc_epcx_setup()
- [High] usb: gadget: crg: broken PLC resume, state, and disconnect in crg_udc_handle_port_status()
- [High] usb: gadget: crg: missing state revalidation after lock drop in crg_udc_ep_queue()
- [Medium] usb: gadget: crg: lockless state updates in crg_gadget_stop() and start()
--

commit 61353c2b5c4b6d93cbb68f036c35a32b502efad8
Author: Petar Stepanovic <pstepanovic@axiado.com>

usb: gadget: udc: crg: Add gadget and endpoint operations

This commit implements the USB gadget and endpoint operations, TRB ring
queueing, and event handling for the Axiado AX3000/AX3005 UDC driver.

> diff --git a/drivers/usb/gadget/udc/crg/crg-udc-gadget.c b/drivers/usb/gadget/udc/crg/crg-udc-gadget.c
> new file mode 100644
> index 0000000000000..a330e857a9509
> --- /dev/null
> +++ b/drivers/usb/gadget/udc/crg/crg-udc-gadget.c
> @@ -0,0 +1,2299 @@

[ ... ]

> +static int crg_gadget_start(struct usb_gadget *g,
> +			    struct usb_gadget_driver *driver)
> +{
> +	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
> +	int ret;
> +
> +	/* The UDC core calls udc_set_speed() before udc_start(). */
> +	ret = crg_udc_program_speed(crg_udc);
> +	if (ret)
> +		return ret;
> +
> +	crg_udc->gadget_driver = driver;
> +	crg_udc_set_vbus_detect(crg_udc, true);
> +	return 0;
> +}
> +
> +static int crg_gadget_stop(struct usb_gadget *g)
> +{
> +	struct crg_gadget_dev *crg_udc;
> +	unsigned long flags = 0;
> +
> +	crg_udc = gadget_to_udc(g);
> +
> +	/*
> +	 * Clear gadget_driver first so crg_udc_handle_port_status() can no
> +	 * longer schedule udc_reset_work (it only does so while
> +	 * gadget_driver is set), then cancel_work_sync() is guaranteed to
> +	 * be the last word: no new reset notification can be queued after
> +	 * it returns.
> +	 */
> +	spin_lock_irqsave(&crg_udc->udc_lock, flags);
> +	crg_udc->gadget_driver = NULL;
> +	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +
> +	cancel_work_sync(&crg_udc->udc_reset_work);
> +
> +	crg_udc->device_state = USB_STATE_ATTACHED;
> +	crg_udc_set_vbus_detect(crg_udc, false);
> +
> +	spin_lock_irqsave(&crg_udc->udc_lock, flags);

[Severity: Medium]
Can updating crg_udc->device_state and calling crg_udc_set_vbus_detect()
without holding crg_udc->udc_lock race with the threaded interrupt handler?

Before crg_gadget_stop() runs crg_udc_reset() and
crg_udc_reset_data_struct(), hardware interrupts remain enabled in
uccr->control (crg_gadget_pullup() only clears CRG_U3DC_CTRL_RUN, and sysfs
soft_connect_store() can invoke udc_stop() without synchronize_irq()).

If crg_gadget_handle_interrupt() is concurrently running under udc_lock in
crg_udc_handle_port_status() or crg_udc_handle_event(), setting
crg_udc->device_state = USB_STATE_ATTACHED outside udc_lock can race with the
state checks (such as crg_udc->device_state < USB_STATE_RECONNECTING),
causing crg_udc_handle_port_status() to prematurely call
crg_udc_enable_setup() or crg_udc_handle_event() to drop events. Similarly,
crg_gadget_start() updates config0, crg_udc->gadget_driver, and
CRG_UDC_VENDOR_REG without acquiring crg_udc->udc_lock.

Should these state and register updates in crg_gadget_stop() and
crg_gadget_start() be performed under crg_udc->udc_lock?

[ ... ]

> +static void crg_gadget_async_callbacks(struct usb_gadget *g, bool enable)
> +{
> +	struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&crg_udc->udc_lock, flags);
> +	crg_udc->async_callbacks = enable;
> +	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +}

[ ... ]

> +static int crg_udc_ep_dequeue(struct usb_ep *_ep, struct usb_request *_req)
> +{

[ ... ]

> +	if (crg_udc_is_pointer_less_than(pause_pt, udc_req->first_trb,
> +					 udc_ep_ptr)) {
> +		crg_udc_squeeze_xfer_ring(udc_ep_ptr, udc_req);
> +		crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
> +	} else if (udc_req->last_trb &&
> +		   crg_udc_is_pointer_less_than(udc_req->last_trb, pause_pt,
> +						udc_ep_ptr)) {
> +		crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
> +	} else {
> +		struct crg_udc_request *next_req = NULL;
> +
> +		udc_req->usb_req.actual =
> +			crg_udc_actual_before(udc_ep_ptr, udc_req, pause_pt);
> +
> +		if (!list_is_last(&udc_req->queue, &udc_ep_ptr->queue))
> +			next_req = list_entry(udc_req->queue.next,
> +					      struct crg_udc_request, queue);
> +
> +		if (next_req && next_req->first_trb)
> +			udc_ep_ptr->deq_pt = next_req->first_trb;
> +		else
> +			udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
> +
> +		if (crg_udc_epcx_update_dqptr(udc_ep_ptr)) {
> +			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +			return -EIO;
> +		}
> +		crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
> +	}
> +
> +	if (old_ep_state == EP_STATE_RUNNING) {
> +		udc_ep_ptr->ep_state = EP_STATE_RUNNING;
> +		crg_udc_knock_doorbell(crg_udc, dci);
> +	}
> +
> +	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +	return 0;
> +}

[Severity: High]
When the active request is dequeued in the else branch above, advancing
udc_ep_ptr->deq_pt frees space on the transfer ring, but tran_ring_full is
never cleared and crg_udc_queue_pending_trbs() is not called. If
tran_ring_full was true and next_req has first_trb == NULL, won't setting
deq_pt = enq_pt while leaving tran_ring_full true permanently stall the
endpoint queue?

In addition, crg_udc_req_done() drops crg_udc->udc_lock while
udc_ep_ptr->ep_state is EP_STATE_STOPPED. When it returns, checking the saved
old_ep_state == EP_STATE_RUNNING unconditionally sets udc_ep_ptr->ep_state =
EP_STATE_RUNNING and rings the doorbell. If the endpoint was disabled or
halted while udc_lock was dropped during the completion callback, won't this
clobber the new endpoint state?

> +
> +/* endpoint operations */
> +
> +static int crg_udc_ep_disable(struct usb_ep *ep)
> +{

[ ... ]

> +	/*
> +	 * ep_enable is a write-1-to-disable register: endpoints are enabled
> +	 * via the CONFIG_EP command, but writing a set bit here disables the
> +	 * corresponding endpoint.
> +	 */
> +	writel(BIT(udc_ep->dci), &uccr->ep_enable);
> +
> +	udc_ep->desc = NULL;
> +	udc_ep->comp_desc = NULL;
> +
> +	/* clean up the request queue */
> +	crg_udc_ep_nuke(udc_ep, -ESHUTDOWN);
> +
> +	/* decrement ep counters */
> +	crg_udc->num_enabled_eps--;
> +	/* clean up the endpoint context */
> +	memset(p_ep_cx, 0, sizeof(struct crg_ep_cx));
> +
> +	/* If device state was changed to default by port
> +	 * reset, should not overwrite it again
> +	 */
> +	if (crg_udc->num_enabled_eps == 0 &&
> +	    crg_udc->device_state == USB_STATE_CONFIGURED) {
> +		crg_udc->device_state = USB_STATE_ADDRESS;
> +	}
> +
> +	udc_ep->ep_state = EP_STATE_DISABLED;
> +
> +	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +
> +	return 0;
> +}

[Severity: High]
Does crg_udc_ep_disable() need to wait for the hardware to clear
BIT(udc_ep->dci) in uccr->ep_enable (similar to the readl_poll_timeout_atomic
in crg_udc_reinit()) before unmapping DMA buffers in crg_udc_ep_nuke() and
zeroing p_ep_cx?

Also, crg_udc_ep_nuke() calls crg_udc_req_done(), which drops and re-acquires
crg_udc->udc_lock around each request's completion callback. Because
udc_ep->ep_state, crg_udc->num_enabled_eps--, and memset(p_ep_cx, 0, ...) are
only updated after crg_udc_ep_nuke() returns:
- If a bus reset runs crg_udc_reinit() while udc_lock is dropped,
  crg_udc->num_enabled_eps is reset to 0, and crg_udc_ep_disable() will then
  underflow num_enabled_eps to 0xffffffff.
- Because udc_ep->ep_state is still EP_STATE_RUNNING during the callback, a
  concurrent re-enable or disable can race with the post-nuke memset(p_ep_cx)
  and udc_ep->ep_state = EP_STATE_DISABLED assignment.

Should the hardware disable poll, endpoint context cleanup, num_enabled_eps
decrement, and EP_STATE_DISABLED transition happen before calling
crg_udc_ep_nuke()?

> +
> +static int crg_udc_ep_enable(struct usb_ep *ep,
> +			     const struct usb_endpoint_descriptor *desc)
> +{

[ ... ]

> +	if (udc_ep->dci) {
> +		u32 ring_size = 0;
> +		size_t len;
> +
> +		if (usb_endpoint_xfer_bulk(desc))
> +			ring_size = CRG_UDC_BULK_EP_TD_RING_SIZE;
> +		else if (usb_endpoint_xfer_isoc(desc))
> +			ring_size = CRG_UDC_ISOC_EP_TD_RING_SIZE;
> +		else if (usb_endpoint_xfer_int(desc))
> +			ring_size = CRG_UDC_INT_EP_TD_RING_SIZE;
> +
> +		if (!ring_size) {
> +			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +			return -EINVAL;
> +		}
> +		len = ring_size * sizeof(struct crg_transfer_trb);
> +
> +		if (udc_ep->tran_ring_info.cpu_addr &&
> +		    udc_ep->tran_ring_info.len != len) {
> +			void *old_vaddr = udc_ep->tran_ring_info.cpu_addr;
> +			dma_addr_t old_dma = udc_ep->tran_ring_info.dma_addr;
> +			size_t old_len = udc_ep->tran_ring_info.len;
> +
> +			udc_ep->tran_ring_info.cpu_addr = NULL;
> +			udc_ep->enq_pt = NULL;
> +			udc_ep->deq_pt = NULL;
> +
> +			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +			dma_free_coherent(crg_udc->dev, old_len, old_vaddr,
> +					  old_dma);
> +			spin_lock_irqsave(&crg_udc->udc_lock, flags);
> +
> +			if (udc_ep->desc != desc ||
> +			    udc_ep->ep_state != EP_STATE_DISABLED) {
> +				spin_unlock_irqrestore(&crg_udc->udc_lock,
> +						       flags);
> +				return -EAGAIN;
> +			}
> +		}
> +
> +		/* setup transfer ring */
> +		if (!udc_ep->tran_ring_info.cpu_addr) {
> +			dma_addr_t dma;
> +			void *vaddr;
> +
> +			spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +			vaddr = dma_alloc_coherent(crg_udc->dev, len, &dma,
> +						   GFP_KERNEL);
> +			if (!vaddr)
> +				return -ENOMEM;
> +			spin_lock_irqsave(&crg_udc->udc_lock, flags);
> +			if (udc_ep->desc != desc ||
> +			    udc_ep->ep_state != EP_STATE_DISABLED) {
> +				spin_unlock_irqrestore(&crg_udc->udc_lock,
> +						       flags);
> +				dma_free_coherent(crg_udc->dev, len, vaddr,
> +						  dma);
> +				return -EAGAIN;
> +			}
> +			udc_ep->tran_ring_info.cpu_addr = vaddr;
> +			udc_ep->tran_ring_info.dma_addr = dma;
> +			udc_ep->tran_ring_info.len = len;
> +			udc_ep->first_trb = vaddr;
> +			udc_ep->last_trb = udc_ep->first_trb + ring_size - 1;
> +		}
> +		memset(udc_ep->first_trb, 0, udc_ep->tran_ring_info.len);
> +		/* Ensure that transfer ring is updated */
> +		dma_wmb();
> +		crg_udc_setup_link_trb(udc_ep->last_trb, true,
> +				       udc_ep->tran_ring_info.dma_addr);
> +		udc_ep->enq_pt = udc_ep->first_trb;
> +		udc_ep->deq_pt = udc_ep->first_trb;
> +		udc_ep->pcs = 1;
> +		udc_ep->tran_ring_full = false;
> +		crg_udc->num_enabled_eps++;
> +		status = crg_udc_epcx_setup(udc_ep);
> +		if (status)
> +			goto err_ep_enable;
> +	}
> +	param0 = BIT(udc_ep->dci);
> +	status = crg_udc_issue_command(crg_udc, CRG_CMD_CONFIG_EP, param0, 0);
> +	if (status)
> +		goto err_ep_enable;
> +	if (crg_udc->device_state == USB_STATE_ADDRESS)
> +		crg_udc->device_state = USB_STATE_CONFIGURED;
> +	udc_ep->wedge = false;
> +	udc_ep->usb_ep.enabled = 1;
> +	udc_ep->ep_state = EP_STATE_RUNNING;
> +	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +	return 0;
> +
> +err_ep_enable:
> +	if (udc_ep->dci && crg_udc->num_enabled_eps)
> +		crg_udc->num_enabled_eps--;
> +	udc_ep->desc = NULL;
> +	udc_ep->comp_desc = NULL;
> +	udc_ep->ep_state = EP_STATE_DISABLED;
> +	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +
> +	if (udc_ep->tran_ring_info.cpu_addr) {
> +		dma_free_coherent(crg_udc->dev, udc_ep->tran_ring_info.len,
> +				  udc_ep->tran_ring_info.cpu_addr,
> +				  udc_ep->tran_ring_info.dma_addr);
> +		udc_ep->tran_ring_info.cpu_addr = NULL;
> +		udc_ep->tran_ring_info.dma_addr = DMA_MAPPING_ERROR;
> +		udc_ep->tran_ring_info.len = 0;
> +		udc_ep->first_trb = NULL;
> +		udc_ep->last_trb = NULL;
> +	}
> +	return status;
> +}

[Severity: Critical]
Can the transfer ring resize and error paths in crg_udc_ep_enable() leave
dangling ring pointers or race on udc_ep->tran_ring_info?

1. When resizing an existing ring (udc_ep->tran_ring_info.len != len),
   udc_ep->tran_ring_info.cpu_addr, udc_ep->enq_pt, and udc_ep->deq_pt are
   set to NULL and old_vaddr is freed, but udc_ep->first_trb and
   udc_ep->last_trb still point to old_vaddr. If the subsequent
   dma_alloc_coherent() returns -ENOMEM (or the state re-check returns
   -EAGAIN, or !ring_size returns -EINVAL), crg_udc_ep_enable() returns
   without clearing udc_ep->desc, udc_ep->comp_desc, udc_ep->first_trb, or
   udc_ep->last_trb. A subsequent crg_udc_ep_queue() will see non-NULL
   udc_ep_ptr->first_trb and udc_ep_ptr->desc and dereference NULL
   udc_ep_ptr->enq_pt (or freed first_trb).

2. Under err_ep_enable, spin_unlock_irqrestore(&crg_udc->udc_lock, flags) is
   called before checking, freeing, and clearing udc_ep->tran_ring_info and
   first_trb/last_trb (and enq_pt/deq_pt are left non-NULL). Once udc_lock is
   dropped with udc_ep->ep_state == EP_STATE_DISABLED, can a concurrent
   crg_udc_ep_enable() or teardown path race on udc_ep->tran_ring_info.cpu_addr
   and double-free or use-after-free the coherent ring buffer?

[ ... ]

> +static int crg_udc_ep_queue(struct usb_ep *_ep, struct usb_request *_req,
> +			    gfp_t gfp_flags)
> +{

[ ... ]

> +	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +
> +	crg_udc_clear_req_container(udc_req_ptr);
> +	udc_req_ptr->mapped = false;
> +
> +	if (_req->length) {
> +		udc_req_ptr->map_dir = dma_data_dir;
> +		status = usb_gadget_map_request_by_dev(
> +			crg_udc->dev, _req, dma_data_dir == DMA_TO_DEVICE);
> +		if (status)
> +			goto err_unmap;
> +
> +		if (is_sg) {
> +			if (!_req->num_mapped_sgs ||
> +			    crg_udc_sg_init(udc_req_ptr)) {
> +				status = -EIO;
> +				goto err_unmap;
> +			}
> +		} else if (_req->dma_mapped) {
> +			udc_req_ptr->mapped = true;
> +		}
> +	}
> +
> +	spin_lock_irqsave(&crg_udc->udc_lock, flags);
> +	if (udc_ep_ptr->desc != desc || !udc_req_ptr->usb_req.complete ||
> +	    !list_empty(&udc_req_ptr->queue) || crg_udc->resetting) {
> +		status = -ESHUTDOWN;
> +		spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +		crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr);
> +		return status;
> +	}
> +
> +	_req->status = -EINPROGRESS;
> +	_req->actual = 0;
> +
> +	if (udc_ep_ptr->tran_ring_full)
> +		status = 0;
> +	else
> +		status = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
> +
> +	if (!status)
> +		list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
> +	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);

[Severity: High]
When crg_udc_ep_queue() drops crg_udc->udc_lock to map the DMA buffer and then
re-acquires it, is checking udc_ep_ptr->desc != desc sufficient?

For EP0 (dci == 0), udc_ep_ptr->desc always points to &crg_udc->ep0_desc. If
a new SETUP packet arrives while udc_lock is dropped during DMA mapping,
crg_udc_abort_ep0_for_setup() and crg_udc_handle_setup_pkt() can complete and
start a new control transfer, yet udc_ep_ptr->desc != desc will still be
false. Without revalidating ep0_state, setup_status, and setup_tag after
re-acquiring udc_lock, can a stale EP0 request (potentially mapped with the
wrong DMA direction) be queued onto the new control transfer?

Similarly, for non-EP0 endpoints, crg_udc_ep_queue() never checks that
udc_ep_ptr->ep_state == EP_STATE_RUNNING (either before or after dropping
udc_lock), which allows requests to be queued while an endpoint is still
mid-enable in crg_udc_ep_enable() or after it was disabled and re-enabled with
the same descriptor pointer.

[ ... ]

> +static int crg_udc_epcx_setup(struct crg_udc_ep *udc_ep)
> +{

[ ... ]

> +	} else if ((crg_udc->gadget.speed == USB_SPEED_HIGH ||
> +		    crg_udc->gadget.speed == USB_SPEED_FULL) &&
> +		   (usb_endpoint_xfer_int(udc_ep->desc) ||
> +		    usb_endpoint_xfer_isoc(udc_ep->desc))) {
> +		if (crg_udc->gadget.speed == USB_SPEED_HIGH)
> +			maxburst = (usb_endpoint_maxp(desc) >> 11) & 0x3;
> +		if (maxburst == 0x3) {
> +			dev_err(crg_udc->dev, "invalid maxburst\n");
> +			maxburst = 0x2;
> +		}
> +	}

[Severity: High]
In crg_udc_epcx_setup(), usb_endpoint_maxp(desc) masks wMaxPacketSize with
USB_ENDPOINT_MAXP_MASK (0x07ff), stripping bits [15:11]. Doesn't shifting the
result right by 11 bits always evaluate to 0, leaving High-Speed
high-bandwidth isochronous and interrupt endpoints configured with
maxburst = 0 instead of usb_endpoint_maxp_mult(desc) - 1?

[ ... ]

> +static int crg_udc_epcx_update_dqptr(struct crg_udc_ep *udc_ep)
> +{
> +	struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
> +	u8 dci = udc_ep->dci;
> +	struct crg_ep_cx *epcx;
> +	u32 dw;
> +	dma_addr_t dqptaddr;
> +	u32 cmd_param0;
> +
> +	if (dci < 2 || dci >= CRG_NUM_EP_CX) {
> +		dev_err(crg_udc->dev, "invalid DCI %u for dequeue pointer\n", dci);
> +		return -EINVAL;
> +	}
> +
> +	epcx = &crg_udc->p_epcx[dci - 2];
> +	dqptaddr = crg_udc_tran_trb_virt_to_dma(udc_ep, udc_ep->deq_pt);
> +
> +	if (dqptaddr == DMA_MAPPING_ERROR)
> +		return -EINVAL;
> +
> +	/* fill ep_dw2 */
> +	dw = lower_32_bits(dqptaddr) & EP_CX_TR_DQPT_LO_MASK;
> +	FIELD_MODIFY(EP_CX_DEQ_CYC_STATE_MASK, &dw, udc_ep->pcs);
> +	epcx->dw2 = cpu_to_le32(dw);

[Severity: High]
Does crg_udc_epcx_update_dqptr() program the wrong Dequeue Cycle State (DCS)
when udc_ep->enq_pt has wrapped past the Link TRB?

udc_ep->pcs tracks the producer cycle state at udc_ep->enq_pt and is toggled
whenever enq_pt wraps to udc_ep->first_trb in crg_udc_queue_trbs(). When
udc_ep->deq_pt != udc_ep->enq_pt and enq_pt has wrapped while deq_pt has not
yet wrapped (deq_pt > enq_pt), the TRBs at udc_ep->deq_pt have cycle bit
udc_ep->pcs ^ 1, not udc_ep->pcs. Programming EP_CX_DEQ_CYC_STATE_MASK with
udc_ep->pcs in crg_udc_ep_dequeue() or crg_udc_ep_halt() will cause the
controller to see a cycle bit mismatch at deq_pt and stall the endpoint ring.

[ ... ]

> +static int crg_udc_queue_trbs(struct crg_udc_ep *udc_ep_ptr,
> +			      struct crg_udc_request *udc_req_ptr, bool b_isoc,
> +			      u32 xfer_ring_size, u32 num_trbs_needed,
> +			      u64 buffer_length)
> +{
> +	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
> +	struct crg_transfer_trb *p_xfer_ring = udc_ep_ptr->first_trb;
> +	struct crg_transfer_trb *enq_pt = udc_ep_ptr->enq_pt;
> +	struct usb_request *usb_req = &udc_req_ptr->usb_req;
> +	u32 num_trbs_ava, count, i;
> +	u32 td_trbs;
> +	u32 td_size;
> +	u32 j = 1;
> +	u32 intr_rate = b_isoc ? CRG_UDC_ISOC_EP_INTERRUPT_RATE :
> +				 CRG_UDC_BULK_EP_INTERRUPT_RATE;
> +	bool full_td = true;
> +	bool need_zlp = false;
> +	bool is_sg = usb_req->num_mapped_sgs;
> +
> +	if (!b_isoc && usb_req->zero && usb_req->length &&
> +	    !(usb_req->length % udc_ep_ptr->usb_ep.maxpacket))
> +		need_zlp = true;
> +
> +	if (!num_trbs_needed)
> +		return -EINVAL;
> +
> +	/* TD_SIZE is five bits and encodes the number of TRBs remaining
> +	 * in this TD minus one. A request may contain multiple TDs.
> +	 */
> +	td_trbs = min_t(u32, num_trbs_needed, 32);
> +
> +	num_trbs_ava = crg_udc_room_on_ring(crg_udc, xfer_ring_size,
> +					    p_xfer_ring, udc_ep_ptr->enq_pt,
> +					    udc_ep_ptr->deq_pt);
> +
> +	if (num_trbs_ava < td_trbs) {
> +		if (b_isoc)
> +			return -EBUSY;
> +		trace_crg_udc_no_room_on_ring(
> +			"deferring TRBs to next doorbell");
> +		count = num_trbs_ava;
> +		full_td = false;
> +		udc_ep_ptr->tran_ring_full = true;
> +		need_zlp = false;
> +	} else {
> +		count = td_trbs;
> +	}

[ ... ]

> +		if (b_isoc) {
> +			crg_udc_setup_trb(
> +				enq_pt, trb_len, trb_buf_addr, td_size,
> +				i == 0 ? (original_pcs ^ 1) : udc_ep_ptr->pcs,
> +				TRB_TYPE_XFER_DATA_ISOCH, short_pkt, chain_bit,
> +				ioc, false, 0, true, 0, 1, 0);
> +		} else {
> +			bool azp = ioc && !chain_bit && need_zlp;
> +
> +			crg_udc_setup_trb(
> +				enq_pt, trb_len, trb_buf_addr, td_size,
> +				i == 0 ? (original_pcs ^ 1) : udc_ep_ptr->pcs,
> +				TRB_TYPE_XFER_NORMAL, short_pkt, chain_bit, ioc,
> +				false, 0, false, 0, 0, azp);
> +		}

[Severity: Critical]
How are requests requiring more than 32 TRBs (or queued behind a full ring)
handled across crg_udc_queue_trbs(), crg_udc_ep_queue(), and
crg_udc_handle_xfer_event()?

1. When num_trbs_needed > 32 and num_trbs_ava >= 32, td_trbs is capped at 32,
   so buffer_length is still non-zero at the 32nd TRB (i == count - 1), but
   need_zlp is only cleared when num_trbs_ava < td_trbs. Won't
   azp = ioc && !chain_bit && need_zlp set TRB_APPEND_ZLP_MASK on the 32nd TRB
   in the middle of the transfer, sending a premature zero-length packet
   before the remaining TRBs of the request are queued?

2. In that same case (num_trbs_needed > 32 and num_trbs_ava > 32),
   crg_udc_queue_trbs() queues the first 32 TRBs and leaves
   udc_req_ptr->all_trbs_queued = false and udc_ep_ptr->tran_ring_full =
   false. If a second request is then submitted via crg_udc_ep_queue(), it
   checks only if (udc_ep_ptr->tran_ring_full) and immediately calls
   crg_udc_build_td() for the second request, interleaving the second
   request's TRBs onto the hardware ring ahead of the remaining TRBs of the
   first request.

3. In crg_udc_handle_xfer_event() at queue_more_trbs, udc_ep_ptr->tran_ring_full
   is cleared to false before calling crg_udc_queue_pending_trbs(). Because
   crg_udc_queue_pending_trbs() breaks out of its loop and returns 0 after
   building a single TD for the first pending request, if that TD does not
   fill the ring, tran_ring_full stays false even when later requests in
   udc_ep_ptr->queue still have !all_trbs_queued (first_trb == NULL), leaving
   them unscheduled on subsequent completions and allowing newly queued
   requests in crg_udc_ep_queue() to bypass them.

> +
> +		trace_crg_udc_queue_trb(udc_ep_ptr, enq_pt);
> +
> +		if (is_sg && !crg_udc_sg_advance(udc_req_ptr, trb_len))
> +			return -EINVAL;
> +
> +		td_size--;
> +		enq_pt++;
> +		j++;
> +
> +		if (enq_pt == udc_ep_ptr->last_trb) {
> +			dma_wmb();
> +			le32p_replace_bits(&enq_pt->dw3, udc_ep_ptr->pcs,
> +					   TRB_CYCLE_BIT_MASK);
> +			udc_ep_ptr->pcs ^= 1;
> +			enq_pt = udc_ep_ptr->first_trb;
> +		}
> +	}
> +
> +	dma_wmb();
> +	le32p_replace_bits(&udc_ep_ptr->enq_pt->dw3, original_pcs,
> +			   TRB_CYCLE_BIT_MASK);
> +
> +	if (!udc_req_ptr->trbs_needed)
> +		udc_req_ptr->first_trb = udc_ep_ptr->enq_pt;
> +
> +	udc_ep_ptr->enq_pt = enq_pt;
> +	if (count == num_trbs_ava)
> +		udc_ep_ptr->tran_ring_full = true;
> +	udc_req_ptr->buff_len_left = buffer_length;
> +	if (buffer_length)
> +		udc_req_ptr->trbs_needed = num_trbs_needed - count;
> +	else
> +		udc_req_ptr->trbs_needed = 0;
> +
> +	if (!udc_req_ptr->buff_len_left) {
> +		if (udc_ep_ptr->enq_pt == udc_req_ptr->first_trb)
> +			udc_req_ptr->last_trb = udc_ep_ptr->last_trb - 1;
> +		else
> +			udc_req_ptr->last_trb = udc_ep_ptr->enq_pt - 1;
> +	}
> +
> +	return 0;
> +}

[Severity: High]
There appear to be two issues with how udc_req_ptr->last_trb is computed here:

1. When the final TRB of a request is placed at udc_ep_ptr->last_trb - 1, the
   loop increments enq_pt to udc_ep_ptr->last_trb and wraps it to
   udc_ep_ptr->first_trb. However, the check at the end compares:

	if (udc_ep_ptr->enq_pt == udc_req_ptr->first_trb)

   instead of udc_ep_ptr->first_trb (unlike crg_udc_queue_ctrl() in
   crg-udc-ep0.c). For any request that did not start at udc_ep_ptr->first_trb,
   this condition is false on wrap and udc_req_ptr->last_trb is set to
   udc_ep_ptr->first_trb - 1 (pointing out-of-bounds before the start of the
   ring buffer).

2. udc_req_ptr->last_trb is only assigned when !udc_req_ptr->buff_len_left.
   When a request is split across multiple TDs (because num_trbs_needed > 32
   or num_trbs_ava < td_trbs), buff_len_left is non-zero and
   udc_req_ptr->last_trb remains NULL while the first TD is in flight. When
   the completion event for the last TRB of that TD (at count - 1 > 0, so
   trb_pt != udc_req->first_trb) arrives in crg_udc_handle_xfer_event(),
   is_request_dequeued() sees udc_req->last_trb == NULL,
   crg_udc_is_pointer_less_than(trb_pt, NULL, udc_ep) returns false, and
   is_request_dequeued() returns true. crg_udc_handle_xfer_event() then jumps
   to queue_more_trbs and skips the !udc_req_ptr->all_trbs_queued continuation
   path, leaving the transfer hung.

[ ... ]

> +void crg_udc_reset_work(struct work_struct *work)
> +{
> +	struct crg_gadget_dev *crg_udc =
> +		container_of(work, struct crg_gadget_dev, udc_reset_work);
> +
> +	if (crg_udc->gadget_driver)
> +		usb_gadget_udc_reset(&crg_udc->gadget, crg_udc->gadget_driver);
> +}

[Severity: Critical]
Can crg_udc_reset_work() dereference a NULL crg_udc->gadget_driver or call
driver->reset() after usb_gadget_disable_async_callbacks()?

1. crg_udc_reset_work() reads crg_udc->gadget_driver twice without holding
   crg_udc->udc_lock. If crg_gadget_stop() sets crg_udc->gadget_driver = NULL
   on another CPU right after the if (crg_udc->gadget_driver) check,
   usb_gadget_udc_reset() can be passed NULL and dereference driver->reset().

2. During unbind, the UDC core calls usb_gadget_disable_async_callbacks()
   (which calls crg_gadget_async_callbacks(g, false)), synchronize_irq(), and
   driver->unbind() before calling usb_gadget_udc_stop_locked()
   (crg_gadget_stop()). Because crg_udc_handle_port_status() and
   crg_udc_reset_work() do not check crg_udc->async_callbacks and
   crg_gadget_async_callbacks() does not cancel udc_reset_work, can
   udc_reset_work run concurrently with or after driver->unbind()?

> +
> +int crg_udc_handle_port_status(struct crg_gadget_dev *crg_udc,
> +			       unsigned long *flags)
> +{

[ ... ]

> +	/* handle Port Connection Change */
> +	tmp = readl(&uccr->portsc);
> +	if ((tmp & (CRG_U3DC_PORTSC_CCS | CRG_U3DC_PORTSC_PP)) ==
> +	    (CRG_U3DC_PORTSC_CCS | CRG_U3DC_PORTSC_PP)) {
> +		if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) != 0 ||
> +		    (!(tmp & CRG_U3DC_PORTSC_PED)))
> +			return 0;

[ ... ]

> +	} else if (!(tmp & CRG_U3DC_PORTSC_CCS)) {
> +		bool cable_connected;
> +		int ccs_drop_ignore = 0;
> +
> +		if ((FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) == 0x0) &&
> +		    (FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, tmp) <
> +		     CRG_U3DC_PORTSC_SPEED_SS)) {
> +			ccs_drop_ignore = 1;
> +			dev_err(crg_udc->dev, "ccs glitch detect on HS/FS!\n");
> +		}
> +
> +		if (!ccs_drop_ignore)
> +			crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
> +
> +		cable_connected = crg_udc_vbus_connected(crg_udc);
> +		if (cable_connected && !ccs_drop_ignore) {
> +			crg_udc->device_state = USB_STATE_POWERED;
> +			crg_udc_reinit(crg_udc);
> +			crg_udc->connected = 0;
> +		} else if (!cable_connected) {
> +			dev_dbg(crg_udc->dev,
> +				"cable disconnected, rst controller\n");
> +			if (was_connected)
> +				crg_udc_notify_disconnect(crg_udc, flags);
> +			if (crg_udc_reset(crg_udc, flags))
> +				dev_err(crg_udc->dev,
> +					"controller reset failed on disconnect\n");
> +			crg_udc_reset_data_struct(crg_udc);
> +			crg_udc->connected = 0;
> +			crg_udc_init_ep0(crg_udc);
> +
> +			crg_udc->device_state = USB_STATE_ATTACHED;
> +			return -ECONNRESET;
> +		}
> +	}
> +	/* handle Port Connection Change end*/
> +
> +	if (portsc_val & CRG_U3DC_PORTSC_PLC) {
> +		tmp = readl(&uccr->portsc);
> +		if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) == 0xf) {
> +			FIELD_MODIFY(CRG_U3DC_PORTSC_PLS_MASK, &tmp, 0x0);
> +			writel(tmp, &uccr->portsc);
> +		}
> +	}
> +
> +	return 0;
> +}

[Severity: High]
There seem to be several issues in crg_udc_handle_port_status():

1. The Port Connection Change block runs unconditionally even when
   CRG_U3DC_PORTSC_CSC is not set in portsc_val, and returns 0 whenever
   (tmp & (CRG_U3DC_PORTSC_CCS | CRG_U3DC_PORTSC_PP)) ==
   (CRG_U3DC_PORTSC_CCS | CRG_U3DC_PORTSC_PP) and PLS != 0. When a Port Link
   State Change (CRG_U3DC_PORTSC_PLC) event arrives with PLS == 0xf (Resume)
   on a connected port, PLS != 0 is true, so the function returns 0 early and
   never reaches the if (portsc_val & CRG_U3DC_PORTSC_PLC) block below.

2. In the CRG_U3DC_PORTSC_PLC block, writing PLS = 0x0 to &uccr->portsc does
   not set CRG_U3DC_PORTSC_LWS (Link Write Strobe) and does not mask out
   CRG_U3DC_PORTSC_W1C_MASK (unlike crg_gadget_wakeup()). Without LWS, will
   the hardware ignore the PLS write while clearing any newly asserted W1C
   bits in portsc?

3. In the !(tmp & CRG_U3DC_PORTSC_CCS) branch when cable_connected &&
   !ccs_drop_ignore is true:
   - crg_udc->device_state is set to USB_STATE_POWERED, but the immediate call
     to crg_udc_reinit(crg_udc) overwrites crg_udc->device_state with
     USB_STATE_RECONNECTING.
   - crg_udc->connected is set to 0 without calling
     crg_udc_notify_disconnect(). If a cable unplug drops CCS before VBUS
     (CRG_U3DC_PORTSC_PP) decays, crg_udc->connected becomes 0 on the first
     event, so when VBUS subsequently drops and enters else if
     (!cable_connected), was_connected is false and
     crg_udc_notify_disconnect() is never called.

> +
> +int crg_udc_handle_xfer_event(struct crg_gadget_dev *crg_udc,
> +			      struct crg_event_trb *event, unsigned long *flags)
> +{

[ ... ]

> +			if (udc_req_ptr->all_trbs_queued) {
> +				udc_req_ptr->usb_req.actual =
> +					udc_req_ptr->usb_req.length -
> +					udc_req_ptr->buff_len_left;
> +				residual = FIELD_GET(EVE_TRB_TRAN_LEN_MASK,
> +						     le32_to_cpu(event->dw2));
> +				if (residual > udc_req_ptr->usb_req.actual)
> +					residual = 0;
> +				udc_req_ptr->usb_req.actual -= residual;
> +				crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
> +
> +				if (dci == 0 && udc_ep_ptr->desc &&
> +				    usb_endpoint_xfer_control(udc_ep_ptr->desc))
> +					crg_udc_ep0_req_complete(udc_ep_ptr,
> +								 flags);

[Severity: High]
For a standard USB_REQ_GET_STATUS request on EP0, getstatusrequest() in
crg-udc-ep0.c sets udc_req_ptr->usb_req.complete = get_status_cmpl and
crg_udc->setup_status = DATA_STAGE_XFER.

When the data stage completes here, crg_udc_req_done() runs get_status_cmpl(),
which sets crg_udc->setup_status = STATUS_STAGE_RECV and calls
crg_udc_build_ep0_status() to queue the status stage TRB. Immediately after
crg_udc_req_done() returns, crg_udc_ep0_req_complete() is called; because
setup_status is already STATUS_STAGE_RECV instead of DATA_STAGE_XFER,
crg_udc_ep0_req_complete() hits its default case and resets
crg_udc->setup_status = WAIT_FOR_SETUP while the status stage TRB is still in
flight.

> +			} else {
> +				int ret;
> +
> +				ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
> +				if (ret) {
> +					udc_req_ptr->usb_req.actual =
> +						udc_req_ptr->usb_req.length -
> +						udc_req_ptr->buff_len_left;
> +					crg_udc_req_done(udc_ep_ptr, udc_req_ptr,
> +							 -EIO);
> +				} else {
> +					crg_udc_knock_doorbell(crg_udc, dci);
> +				}
> +			}
> +		}
> +		break;
> +
> +	case CMPL_CODE_SHORT_PKT:
> +		if (list_empty(&udc_ep_ptr->queue))
> +			break;
> +
> +		udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
> +					       struct crg_udc_request, queue);
> +		udc_req_ptr->usb_req.actual = udc_req_ptr->usb_req.length -
> +					      udc_req_ptr->buff_len_left;
> +		residual = FIELD_GET(EVE_TRB_TRAN_LEN_MASK,
> +				     le32_to_cpu(event->dw2));
> +		if (residual > udc_req_ptr->usb_req.actual)
> +			residual = 0;
> +		udc_req_ptr->usb_req.actual -= residual;
> +
> +		if (udc_req_ptr->usb_req.short_not_ok &&
> +		    usb_endpoint_dir_out(udc_ep_ptr->desc))
> +			crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EREMOTEIO);
> +		else
> +			crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
> +
> +		trbs_dequeued = true;
> +		crg_udc_advance_dequeue_pt(udc_ep_ptr);
> +		break;

[Severity: High]
Can CMPL_CODE_SHORT_PKT cause EP0 control OUT transfers to hang and miscalculate
usb_req.actual on multi-TRB transfers?

1. crg_udc_setup_datastage_trb() sets TRB_INTR_ON_SHORT_PKT_MASK on EP0 data
   stage TRBs. If a host sends a short packet during a control OUT data stage
   (DATA_STAGE_RECV), completion arrives with CMPL_CODE_SHORT_PKT, which
   completes the request via crg_udc_req_done() but never calls
   crg_udc_ep0_req_complete(udc_ep_ptr, flags) when dci == 0, so the status
   stage TRB is never queued.

2. EVE_TRB_TRAN_LEN_MASK in event->dw2 is the residual byte count of the
   single TRB that short-terminated, whereas
   udc_req_ptr->usb_req.length - udc_req_ptr->buff_len_left is the total byte
   count of all TRBs queued in the TD. If an OUT request spans multiple TRBs
   (for example, a scatter-gather list or a buffer crossing a 64 KB boundary)
   and short-terminates on an earlier TRB, subtracting only that TRB's
   residual counts all subsequent unexecuted TRBs in the TD as received.

3. Calling crg_udc_req_done() before crg_udc_advance_dequeue_pt(udc_ep_ptr)
   means udc_lock is dropped for the completion callback while udc_ep_ptr->deq_pt
   is still stale; if the callback queues a new request, advancing deq_pt
   afterwards can corrupt the dequeue pointer. (A similar ordering issue
   exists in crg_udc_ep_halt() and set_ep_halt(), where crg_udc_ep_nuke()
   drops udc_lock before udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt and
   udc_ep_ptr->ep_state = EP_STATE_HALTED are set.)

[ ... ]

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261020-axiado-ax3000-usb-device-controller-v3-0-3012ecfc0a93@axiado.com?part=5

Re: [PATCH v3 4/6] usb: gadget: udc: crg: Add endpoint 0 handling

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 9 potential issue(s) to consider:
- [High] usb: gadget: crg: NULL dereference and list corruption in crg_udc_build_ep0_status()
- [High] usb: gadget: crg: duplicate status stage transition in get_status_cmpl()
- [High] usb: gadget: crg: race on unlocked status_req buffer and EP0 state in completion
- [Medium] usb: gadget: crg: broken feature validation and recipient handling in setfeaturesrequest()
- [Medium] usb: gadget: crg: broken SET_ADDRESS(0) and error handling in setaddressrequest()
- [Medium] usb: gadget: crg: reversed SET_SEL payload layout in set_sel_cmpl()
- [Medium] usb: gadget: udc: crg: missing chain bit in crg_udc_queue_ctrl()
- [Medium] usb: gadget: crg: unsafe EP0 nuke ordering in crg_udc_prepare_for_setup()
- [Medium] usb: gadget: crg: race with udc_reset_work in crg_udc_handle_setup_pkt()
--

commit 594f5b35fd6f32d435f7e79db65654d684456116
Author: Petar Stepanovic <pstepanovic@axiado.com>

usb: gadget: udc: crg: Add endpoint 0 handling

This patch adds endpoint 0 setup packet, status stage, and control transfer
handling for the Axiado AX3000/AX3005 UDC driver.

> diff --git a/drivers/usb/gadget/udc/crg/crg-udc-ep0.c b/drivers/usb/gadget/udc/crg/crg-udc-ep0.c
> new file mode 100644
> index 0000000000000..dce36d23d3ece
> --- /dev/null
> +++ b/drivers/usb/gadget/udc/crg/crg-udc-ep0.c
> @@ -0,0 +1,1106 @@
> +// SPDX-License-Identifier: GPL-2.0+
> +/*
> + * Axiado AX3000/AX3005 USB Device Controller driver.
> + *
> + * Copyright (c) 2019 Corigine Inc.
> + * Copyright (c) 2021-2026 Axiado Corporation.
> + */
> +#include <linux/iopoll.h>
> +
> +#include "crg-udc.h"
> +#include "crg-udc-trace.h"
> +
> +static void enable_setup_event(struct crg_gadget_dev *crg_udc);
> +
> +void crg_udc_build_ep0_status(struct crg_udc_ep *udc_ep_ptr, bool default_value,
> +			      u32 status, struct crg_udc_request *udc_req_ptr,
> +			      u8 set_addr, u8 stall)
> +{
> +	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
> +	struct crg_transfer_trb *enq_pt = udc_ep_ptr->enq_pt;
> +
> +	if (default_value) {
> +		udc_req_ptr = crg_udc->status_req;
> +		udc_req_ptr->usb_req.length = 0;
> +		udc_req_ptr->usb_req.status = status;
> +		udc_req_ptr->usb_req.actual = 0;
> +		udc_req_ptr->usb_req.complete = NULL;
> +	} else {
> +		udc_req_ptr->usb_req.status = status;
> +		udc_req_ptr->usb_req.actual = 0;
> +	}
> +
> +	crg_udc_setup_status_trb(crg_udc, enq_pt, &udc_req_ptr->usb_req,
> +				 udc_ep_ptr->pcs, set_addr, stall);
> +
> +	enq_pt++;
> +
> +	/* check if we are at end of trb segment.  If so, update
> +	 * pcs and enq for next segment
> +	 */
> +	if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(enq_pt->dw3)) ==
> +	    TRB_TYPE_LINK) {
> +		if (FIELD_GET(TRB_LINK_TOGGLE_CYCLE_MASK,
> +			      le32_to_cpu(enq_pt->dw3))) {
> +			dma_wmb();
> +			le32p_replace_bits(&enq_pt->dw3, udc_ep_ptr->pcs,
> +					   TRB_CYCLE_BIT_MASK);
> +			udc_ep_ptr->pcs ^= 0x1;
> +		}
> +		enq_pt = udc_ep_ptr->first_trb;
> +	}
> +	udc_ep_ptr->enq_pt = enq_pt;
> +
> +	crg_udc_knock_doorbell(crg_udc, 0);
> +
> +	list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
> +}

[Severity: High]
Can crg_udc_build_ep0_status() dereference a NULL crg_udc->status_req or
corrupt udc_ep_ptr->queue when called via crg_udc_set_ep0_halt()?

In getstatusrequest() and setselrequest(), udc_req_ptr is initialized to
crg_udc->status_req and checked with:

	if (!udc_req_ptr || !list_empty(&udc_req_ptr->queue) || ...) {
		crg_udc_set_ep0_halt(crg_udc);
		return;
	}

However, crg_udc_set_ep0_halt() immediately calls:

	crg_udc_build_ep0_status(udc_ep_ptr, true, -EINVAL, NULL, 0, 1);

When default_value is true, crg_udc_build_ep0_status() sets udc_req_ptr =
crg_udc->status_req and dereferences it without checking for NULL. If
status_req is already queued on udc_ep_ptr->queue from a previous in-flight
request, this also overwrites udc_req_ptr->usb_req.complete with NULL
(leaking any allocated status_req->usb_req.buf and its DMA mapping) and
calls list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue) on a list node
that is already in the list.

[ ... ]

> +void crg_udc_ep0_req_complete(struct crg_udc_ep *udc_ep_ptr,
> +			      unsigned long *flags)
> +{
> +	struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
> +
> +	switch (crg_udc->setup_status) {
> +	case DATA_STAGE_XFER:
> +		crg_udc->setup_status = STATUS_STAGE_RECV;
> +		crg_udc_build_ep0_status(udc_ep_ptr, true, -EINPROGRESS, NULL,
> +					 0, 0);
> +		break;
> +	case DATA_STAGE_RECV:
> +		crg_udc->setup_status = STATUS_STAGE_XFER;
> +		crg_udc_build_ep0_status(udc_ep_ptr, true, -EINPROGRESS, NULL,
> +					 0, 0);
> +		break;
> +	default:
> +		if (crg_udc->setup_fn_call_back) {
> +			crg_udc->setup_fn_call_back(crg_udc);
> +			crg_udc->setup_fn_call_back = NULL;
> +		}
> +		crg_udc->setup_status = WAIT_FOR_SETUP;
> +		break;
> +	}
> +}
> +
> +static void get_status_cmpl(struct usb_ep *ep, struct usb_request *req)
> +{
> +	struct crg_udc_ep *udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
> +	struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
> +	unsigned long flags;
> +
> +	kfree(req->buf);
> +	req->buf = NULL;
> +
> +	/*
> +	 * A non-zero status means this request was completed early (e.g.
> +	 * crg_udc_ep_nuke(..., -ECONNRESET) from crg_udc_abort_ep0_for_setup()
> +	 * for a new SETUP). EP0 is being reinitialized for that new control
> +	 * transfer; queuing a status stage here would race with it.
> +	 */
> +	if (req->status)
> +		return;
> +
> +	spin_lock_irqsave(&crg_udc->udc_lock, flags);
> +	crg_udc->setup_status = STATUS_STAGE_RECV;
> +	crg_udc_build_ep0_status(udc_ep, true, -EINPROGRESS, NULL, 0, 0);
> +	spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +}

[Severity: High]
Does get_status_cmpl() conflict with crg_udc_ep0_req_complete() when a
USB_REQ_GET_STATUS data stage finishes?

When the data-stage TRB for getstatusrequest() completes with
CMPL_CODE_SUCCESS, crg_udc_handle_xfer_event() calls crg_udc_req_done()
followed immediately by crg_udc_ep0_req_complete().

Inside crg_udc_req_done(), get_status_cmpl() sets crg_udc->setup_status =
STATUS_STAGE_RECV and calls crg_udc_build_ep0_status(), which queues the
status TRB and appends crg_udc->status_req to udc_ep[0].queue.

When crg_udc_ep0_req_complete() runs next, crg_udc->setup_status is already
STATUS_STAGE_RECV instead of DATA_STAGE_XFER, so it hits the default branch
and resets crg_udc->setup_status to WAIT_FOR_SETUP while the status stage is
still pending on the ring and in udc_ep[0].queue. If another SETUP packet
arrives before the status TRB completes, crg_udc_handle_event() will see
WAIT_FOR_SETUP, skip crg_udc_abort_ep0_for_setup(), and corrupt
udc_ep[0].queue when queuing the next request.

Additionally, crg_udc_req_done() removes status_req from udc_ep[0].queue via
list_del_init(&udc_req->queue) and drops crg_udc->udc_lock before invoking
get_status_cmpl() (or set_sel_cmpl()). Because list_empty(&status_req->queue)
is now true while udc_lock is dropped, can a concurrent getstatusrequest()
or setselrequest() acquire udc_lock and run kfree(udc_req_ptr->usb_req.buf)
and kzalloc_obj() concurrently with the unlocked kfree(req->buf) and
req->buf = NULL in get_status_cmpl() or set_sel_cmpl()?

[ ... ]

> +int crg_udc_prepare_for_setup(struct crg_gadget_dev *crg_udc)
> +{
> +	struct crg_udc_ep *udc_ep0_ptr;
> +
> +	dma_addr_t dqptaddr;
> +	u32 cmd_param0;
> +	u32 cmd_param1;
> +
> +	if (!crg_udc_is_event_rings_empty(crg_udc) ||
> +	    crg_udc->portsc_on_reconnecting == 1)
> +		return -EBUSY;
> +
> +	udc_ep0_ptr = &crg_udc->udc_ep[0];
> +	/* If we reinit ep0 on bus reset, we just make ep0 dequeue pointer
> +	 * align with enqueue pointer, all remaining xfer trbs became dumb once
> +	 * which will not produce xfer event anymore.
> +	 *
> +	 * If we considering the opposite solution, we should wait all ep0 xfer
> +	 * trbs be completed(with some err complete code)
> +	 */
> +	if (crg_udc->reinit_ep0_on_bus_reset) {
> +		/* Complete any reqs on EP0 queue */
> +		crg_udc_ep_nuke(udc_ep0_ptr, -ESHUTDOWN);
> +
> +		/* context related ops */
> +		dqptaddr = crg_udc_tran_trb_virt_to_dma(udc_ep0_ptr,
> +							udc_ep0_ptr->enq_pt);
> +		if (dma_mapping_error(crg_udc->dev, dqptaddr))
> +			return -EIO;
> +		cmd_param0 =
> +			(lower_32_bits(dqptaddr) & CRG_CMD0_0_DQPTRLO_MASK) |
> +			FIELD_PREP(CRG_CMD0_0_DCS_MASK, udc_ep0_ptr->pcs);
> +		cmd_param1 = upper_32_bits(dqptaddr);
> +		if (crg_udc_issue_command(crg_udc, CRG_CMD_INIT_EP0, cmd_param0,
> +					  cmd_param1))
> +			return -EIO;
> +
> +		udc_ep0_ptr->deq_pt = udc_ep0_ptr->enq_pt;
> +		udc_ep0_ptr->tran_ring_full = false;
> +	} else {

[Severity: Medium]
In crg_udc_prepare_for_setup(), is it safe to call crg_udc_ep_nuke() before
stopping EP0 or issuing CRG_CMD_INIT_EP0?

crg_udc_ep_nuke() unmaps the DMA buffers and drops crg_udc->udc_lock while
giving back requests to completion callbacks before CRG_CMD_INIT_EP0 updates
the hardware dequeue pointer.

Furthermore, because crg_udc->ep0_state is not set to EP0_STATE_ABORTING
while udc_lock is dropped during crg_udc_ep_nuke(), a concurrent
crg_udc_ep_queue() on EP0 could enqueue new TRBs and add a request to
udc_ep0_ptr->queue, which would then be skipped in hardware when
CRG_CMD_INIT_EP0 sets the dequeue pointer to udc_ep0_ptr->enq_pt and
udc_ep0_ptr->deq_pt = udc_ep0_ptr->enq_pt. Unlike crg_udc_init_ep0() and
crg_udc_abort_ep0_for_setup(), this path also does not restore
udc_ep0_ptr->ep_state to EP_STATE_RUNNING.

[ ... ]

> +static void set_sel_cmpl(struct usb_ep *ep, struct usb_request *req)
> +{
> +	struct crg_udc_ep *udc_ep;
> +	struct crg_gadget_dev *crg_udc;
> +	struct crg_sel_value *sel_value;
> +
> +	udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
> +	crg_udc = udc_ep->crg_udc;
> +
> +	if (req->buf) {
> +		if (!req->status && req->actual >= sizeof(*sel_value)) {
> +			sel_value = req->buf;
> +			crg_udc->sel_value.u2_pel_value =
> +				le16_to_cpu(sel_value->u2_pel_value);
> +			crg_udc->sel_value.u2_sel_value =
> +				le16_to_cpu(sel_value->u2_sel_value);
> +			crg_udc->sel_value.u1_pel_value = sel_value->u1_pel_value;
> +			crg_udc->sel_value.u1_sel_value = sel_value->u1_sel_value;
> +		}
> +
> +		kfree(req->buf);
> +		req->buf = NULL;
> +	}
> +}

[Severity: Medium]
Does set_sel_cmpl() parse the 6-byte USB_REQ_SET_SEL payload in the wrong
field order?

Per USB 3.2 Section 9.4.12 and struct usb_set_sel_req in
include/uapi/linux/usb/ch9.h, the wire layout is u1_sel (u8), u1_pel (u8),
u2_sel (__le16), and u2_pel (__le16).

However, set_sel_cmpl() casts req->buf to struct crg_sel_value, which is
defined in drivers/usb/gadget/udc/crg/crg-udc.h in the reverse order:

struct crg_sel_value {
	u16 u2_pel_value;
	u16 u2_sel_value;
	u8 u1_pel_value;
	u8 u1_sel_value;
};

This reads u1_sel and u1_pel into u2_pel_value, reads u2_pel into
u1_pel_value and u1_sel_value, and passes host-endian u16 members to
le16_to_cpu(). Also, crg_udc->sel_value is modified here without holding
crg_udc->udc_lock.

[ ... ]

> +static void setaddressrequest(struct crg_gadget_dev *crg_udc, u16 value,
> +			      u16 index, u16 length)
> +{
> +	int status = -EINPROGRESS;
> +	u8 status_set_addr = 0;
> +
> +	if (value > 127 || index != 0 || length != 0) {
> +		status = -EINVAL;
> +		goto set_address_error;
> +	}
> +
> +	if ((crg_udc->device_state == USB_STATE_DEFAULT && value != 0) ||
> +	    crg_udc->device_state == USB_STATE_ADDRESS) {
> +		u32 param0;
> +
> +		param0 = FIELD_PREP(CRG_CMD2_0_DEV_ADDR_MASK, value);
> +		if (crg_udc_issue_command(crg_udc, CRG_CMD_SET_ADDR, param0,
> +					  0)) {
> +			status = -EIO;
> +		} else {
> +			crg_udc->dev_addr = value;
> +			status_set_addr = 1;
> +		}
> +	} else {
> +		status = -EINVAL;
> +	}
> +
> +set_address_error:
> +	dev_dbg(crg_udc->dev, "crg_udc_build_ep0_status for Address Device\n");
> +
> +	crg_udc->setup_status = STATUS_STAGE_XFER;
> +	crg_udc->setup_fn_call_back = &set_address_cmpl;
> +	crg_udc_build_ep0_status(&crg_udc->udc_ep[0], true, status, NULL,
> +				 status_set_addr, status != -EINPROGRESS);
> +}

[Severity: Medium]
Should setaddressrequest() allow value == 0 when crg_udc->device_state is
USB_STATE_DEFAULT, and avoid registering set_address_cmpl on error?

Per USB 2.0/3.2 Section 9.4.6, a device in the Default state must accept
SET_ADDRESS(0), whereas the check above rejects value == 0 in
USB_STATE_DEFAULT with -EINVAL.

In addition, when setaddressrequest() fails validation or
crg_udc_issue_command() fails, jumping to set_address_error still sets
crg_udc->setup_fn_call_back = &set_address_cmpl and leaves
udc_ep[0].ep_state as EP_STATE_RUNNING instead of EP_STATE_HALTED.

> +
> +static bool setfeaturesrequest(struct crg_gadget_dev *crg_udc, u8 request_type,
> +			       u8 b_request, u16 value, u16 index, u16 length,
> +			       unsigned long *flags)
> +{
> +	int status = -EINPROGRESS;
> +	u8 dci;
> +	struct crg_udc_ep *udc_ep_ptr;
> +	u32 tmp;
> +	u32 set_feat = 0;
> +	struct crg_uccr __iomem *uccr = crg_udc->uccr;
> +
> +	if (length != 0) {
> +		status = -EINVAL;
> +		goto set_feature_error;
> +	}
> +
> +	if (crg_udc->device_state == USB_STATE_DEFAULT) {
> +		status = -EINVAL;
> +		goto set_feature_error;
> +	}

[Severity: Medium]
Does rejecting USB_STATE_DEFAULT here prevent USB_DEVICE_TEST_MODE from ever
working in the Default state?

Further down in the USB_DEVICE_TEST_MODE case, the code checks:

	if (b_request != USB_REQ_SET_FEATURE ||
	    crg_udc->gadget.speed != USB_SPEED_HIGH ||
	    crg_udc->device_state < USB_STATE_DEFAULT ||
	    ...)

Per USB 2.0 Section 9.4.9, SetFeature(Test_Mode) must be supported in the
Default, Address, and Configured states, but the early check for
crg_udc->device_state == USB_STATE_DEFAULT stalls it before reaching the
USB_DEVICE_TEST_MODE handler.

[ ... ]

> +	} else if ((request_type & (USB_RECIP_MASK | USB_TYPE_MASK)) ==
> +		   (USB_RECIP_INTERFACE | USB_TYPE_STANDARD)) {
> +		if (crg_udc->device_state != USB_STATE_CONFIGURED) {
> +			dev_err(crg_udc->dev,
> +				"interface u12 enable fail, usb state=%d\n",
> +				crg_udc->device_state);
> +			status = -EINVAL;
> +			goto set_feature_error;
> +		}
> +
> +		/* Suspend Option */
> +		if (value != USB_INTRF_FUNC_SUSPEND) {
> +			status = -EINVAL;
> +			goto set_feature_error;
> +		}
> +
> +		if (!(index & USB_INTR_FUNC_SUSPEND_OPT_MASK &
> +		      USB_INTRF_FUNC_SUSPEND_LP)) {
> +			status = -EINVAL;
> +			goto set_feature_error;
> +		}
> +
> +		if (index & USB_INTRF_FUNC_SUSPEND_RW)
> +			dev_dbg(crg_udc->dev, "Interface En Remote Wakeup\n");
> +		else
> +			dev_dbg(crg_udc->dev, "Interface Dis RemoteWakeup\n");
> +
> +		/* Do not need to return status stage here
> +		 * Pass to composite gadget driver to process
> +		 * the request
> +		 */
> +		return false;
> +	}
> +
> +	crg_udc->setup_status = STATUS_STAGE_XFER;
> +	crg_udc_build_ep0_status(&crg_udc->udc_ep[0], true, status, NULL, 0, 0);
> +	return true;
> +
> +set_feature_error:
> +	crg_udc_set_ep0_halt(crg_udc);
> +	return true;
> +}

[Severity: Medium]
In setfeaturesrequest(), doesn't the check:

		if (!(index & USB_INTR_FUNC_SUSPEND_OPT_MASK &
		      USB_INTRF_FUNC_SUSPEND_LP)) {
			status = -EINVAL;
			goto set_feature_error;
		}

stall EP0 whenever the Low Power Suspend bit (USB_INTRF_FUNC_SUSPEND_LP) is
cleared? Per USB 3.2 Section 9.4.9 and composite_setup(), the host clears
function suspend or disables function remote wakeup by sending
SetFeature(FUNCTION_SUSPEND) with LP=0 or ClearFeature(FUNCTION_SUSPEND)
with wIndex=0, both of which have USB_INTRF_FUNC_SUSPEND_LP == 0.

Also, because neither crg_udc_handle_setup_pkt() nor setfeaturesrequest()
verifies that (request_type & USB_DIR_IN) == USB_DIR_OUT, and the recipient
if/else-if chain has no final else branch, any SET_FEATURE or CLEAR_FEATURE
request targeting USB_RECIP_OTHER (or with USB_DIR_IN set) falls through to
crg_udc_build_ep0_status() and is ACKed instead of stalled.

[ ... ]

> +void crg_udc_handle_setup_pkt(struct crg_gadget_dev *crg_udc,
> +			      struct usb_ctrlrequest *setup_pkt, u8 setup_tag,
> +			      unsigned long *flags)
> +{
> +	u16 w_value = le16_to_cpu(setup_pkt->wValue);
> +	u16 w_index = le16_to_cpu(setup_pkt->wIndex);
> +	u16 w_length = le16_to_cpu(setup_pkt->wLength);
> +	struct usb_gadget_driver *driver;
> +	u32 tmp;
> +
> +	trace_crg_udc_ctrl_req(setup_pkt);
> +
> +	/* EP0 come backs to running when new setup packet comes */
> +	crg_udc->udc_ep[0].ep_state = EP_STATE_RUNNING;
> +
> +	crg_udc->setup_tag = setup_tag;
> +	crg_udc->setup_status = SETUP_PKT_PROCESS_IN_PROGRESS;
> +	crg_udc->setup_fn_call_back = NULL;
> +
> +	if ((setup_pkt->bRequestType & USB_TYPE_MASK) == USB_TYPE_STANDARD) {
> +		switch (setup_pkt->bRequest) {
> +		case USB_REQ_GET_STATUS:

[ ... ]

> +		case USB_REQ_SET_ADDRESS:
> +			if (setup_pkt->bRequestType !=
> +			    (USB_DIR_OUT | USB_RECIP_DEVICE |
> +			     USB_TYPE_STANDARD)) {
> +				crg_udc_set_ep0_halt(crg_udc);
> +				return;
> +			}
> +
> +			setaddressrequest(crg_udc, w_value, w_index, w_length);
> +			return;

[ ... ]

> +	driver = crg_udc->async_callbacks ? crg_udc->gadget_driver : NULL;
> +	spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
> +
> +	/*
> +	 * Make sure any reset notification queued by
> +	 * crg_udc_handle_port_status() has actually run before this SETUP
> +	 * (which may be SET_CONFIGURATION) reaches the function driver.
> +	 * Otherwise udc_reset_work's usb_gadget_udc_reset() could run after
> +	 * the new configuration is already active and tear it back down.
> +	 */
> +	flush_work(&crg_udc->udc_reset_work);
> +
> +	if (!driver || driver->setup(&crg_udc->gadget, setup_pkt) < 0) {

[Severity: Medium]
Can udc_reset_work race with USB_REQ_SET_ADDRESS or with disabling
async_callbacks in crg_udc_handle_setup_pkt()?

Because USB_REQ_SET_ADDRESS (and other UDC-handled standard requests)
returns early before flush_work(&crg_udc->udc_reset_work), a post-reset
SET_ADDRESS can complete and transition crg_udc->device_state and
gadget->state to USB_STATE_ADDRESS via set_address_cmpl() while
udc_reset_work is still pending. When udc_reset_work subsequently executes
usb_gadget_udc_reset(), it resets gadget->state back to USB_STATE_DEFAULT,
leaving gadget->state out of sync with crg_udc->device_state.

Also, driver is read from crg_udc->async_callbacks ? crg_udc->gadget_driver
: NULL before dropping udc_lock and calling
flush_work(&crg_udc->udc_reset_work). If crg_gadget_async_callbacks() sets
async_callbacks to false while flush_work() is sleeping, driver->setup()
will still be called using the stale local pointer.

[ ... ]

> +int crg_udc_queue_ctrl(struct crg_udc_ep *udc_ep_ptr,
> +		       struct crg_udc_request *udc_req_ptr,
> +		       u32 num_of_trbs_needed)
> +{

[ ... ]

> +			u8 original_pcs = udc_ep_ptr->pcs;
> +
> +			for (i = 0; i < num_of_trbs_needed; i++) {
> +				p_trb = enq_pt;
> +				transfer_length = crg_udc_trb_chunk_len(
> +					trb_dma, remaining);
> +				ioc = (i == num_of_trbs_needed - 1);
> +				azp = ioc && need_zlp;
> +				crg_udc_setup_datastage_trb(
> +					p_trb, trb_dma,
> +					i == 0 ? (original_pcs ^ 1) :
> +						 udc_ep_ptr->pcs,
> +					transfer_length,
> +					num_of_trbs_needed - i - 1, ioc, azp,
> +					dir, setup_tag);
> +				udc_req_ptr->all_trbs_queued = 1;
> +				trb_dma += transfer_length;
> +				remaining -= transfer_length;
> +				enq_pt++;

[Severity: Medium]
When num_of_trbs_needed > 1 in crg_udc_queue_ctrl() (for example, when an
EP0 DMA buffer crosses a 64 KB CRG_UDC_TRB_MAX_BUFFER_SIZE boundary),
shouldn't TRB_CHAIN_BIT_MASK be set on the non-final TRBs (!ioc)?

crg_udc_setup_datastage_trb() never sets TRB_CHAIN_BIT_MASK in p_trb->dw3,
unlike crg_udc_queue_trbs() which sets chain_bit = (i < count - 1), so the
controller will treat each TRB chunk as an unchained standalone TD rather
than a single chained data-stage TD.

[ ... ]

> +				}
> +			}

[ ... ]

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261020-axiado-ax3000-usb-device-controller-v3-0-3012ecfc0a93@axiado.com?part=4