* [PATCH v5 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC
2026-11-16 19:06 [PATCH v5 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
@ 2026-11-16 19:06 ` Petar Stepanovic
2026-11-16 19:12 ` sashiko-bot
2026-11-16 19:06 ` [PATCH v5 2/6] usb: gadget: udc: crg: Add hardware definitions and structs Petar Stepanovic
` (4 subsequent siblings)
5 siblings, 1 reply; 13+ messages in thread
From: Petar Stepanovic @ 2026-11-16 19:06 UTC (permalink / raw)
To: Prasad Bolisetty, Greg Kroah-Hartman, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Tzu-Hao Wei,
Karthikeyan Mitran, Philipp Zabel, Neill Kapron
Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
Petar Stepanovic, Krutik Shah, Naveen Kumar Rajgiri Bassappa,
Krzysztof Kozlowski
Add a Devicetree binding for the USB device controller integrated into
the Axiado AX3000 and AX3005 SoCs.
The controller is based on a Corigine USB 3.1 IP core that is
dual-role capable (device and host), used here in device role. It
provides SuperSpeed (5 Gb/s) or High-Speed (480 Mb/s) operation, and
supports control, bulk, interrupt, and isochronous transfer types
across multiple configurable endpoints.
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Tzu-Hao Wei <twei@axiado.com>
Signed-off-by: Tzu-Hao Wei <twei@axiado.com>
Reviewed-by: Krzysztof Kozlowski <krzysztof.kozlowski@oss.qualcomm.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
.../devicetree/bindings/usb/axiado,ax3000-udc.yaml | 69 ++++++++++++++++++++++
MAINTAINERS | 8 +++
2 files changed, 77 insertions(+)
diff --git a/Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml b/Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
new file mode 100644
index 000000000000..8b9acdb81907
--- /dev/null
+++ b/Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
@@ -0,0 +1,69 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/usb/axiado,ax3000-udc.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Axiado AX3000/AX3005 USB Device Controller
+
+maintainers:
+ - Petar Stepanovic <pstepanovic@axiado.com>
+ - Tzu-Hao Wei <twei@axiado.com>
+ - Prasad Bolisetty <pbolisetty@axiado.com>
+
+description:
+ Axiado USB dual-role controller based on the Corigine USB 3.1 IP core.
+ Supports USB 3.1 SuperSpeed and USB 2.0 High-Speed device operation
+ with control, bulk, interrupt, and isochronous transfers.
+
+allOf:
+ - $ref: usb.yaml#
+
+properties:
+ compatible:
+ oneOf:
+ - const: axiado,ax3000-udc
+ - items:
+ - const: axiado,ax3005-udc
+ - const: axiado,ax3000-udc
+
+ reg:
+ maxItems: 1
+
+ interrupts:
+ maxItems: 1
+
+ clocks:
+ maxItems: 1
+
+ resets:
+ maxItems: 1
+
+ phys:
+ maxItems: 1
+
+required:
+ - compatible
+ - reg
+ - interrupts
+
+unevaluatedProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/interrupt-controller/arm-gic.h>
+ #include <dt-bindings/interrupt-controller/irq.h>
+
+ soc {
+ #address-cells = <2>;
+ #size-cells = <2>;
+
+ usb@81200000 {
+ compatible = "axiado,ax3000-udc";
+ reg = <0x0 0x81200000 0x0 0x100000>;
+ interrupt-parent = <&gic500>;
+ interrupts = <GIC_SPI 21 IRQ_TYPE_LEVEL_HIGH>;
+ resets = <&reset 0>;
+ phys = <&usb_phy>;
+ };
+ };
diff --git a/MAINTAINERS b/MAINTAINERS
index 222b7fc2c3aa..1bf0408e00f3 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4662,6 +4662,14 @@ F: Documentation/devicetree/bindings/hwmon/axiado,ax3000-tsadc.yaml
F: Documentation/hwmon/axiado-tsadc.rst
F: drivers/hwmon/axiado-tsadc.c
+AXIADO USB UDC DRIVER
+M: Petar Stepanovic <pstepanovic@axiado.com>
+M: Tzu-Hao Wei <twei@axiado.com>
+M: Prasad Bolisetty <pbolisetty@axiado.com>
+L: linux-usb@vger.kernel.org
+S: Supported
+F: Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
+
AYANEO PLATFORM EC DRIVER
M: Antheas Kapenekakis <lkml@antheas.dev>
L: platform-driver-x86@vger.kernel.org
--
2.34.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH v5 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC
2026-11-16 19:06 ` [PATCH v5 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC Petar Stepanovic
@ 2026-11-16 19:12 ` sashiko-bot
0 siblings, 0 replies; 13+ messages in thread
From: sashiko-bot @ 2026-11-16 19:12 UTC (permalink / raw)
To: Petar Stepanovic; +Cc: linux-usb
> Add a Devicetree binding for the USB device controller integrated into
> the Axiado AX3000 and AX3005 SoCs.
>
> The controller is based on a Corigine USB 3.1 IP core that is
> dual-role capable (device and host), used here in device role. It
> provides SuperSpeed (5 Gb/s) or High-Speed (480 Mb/s) operation, and
> supports control, bulk, interrupt, and isochronous transfer types
> across multiple configurable endpoints.
>
> Co-developed-by: Krutik Shah <krutikshah@axiado.com>
> Signed-off-by: Krutik Shah <krutikshah@axiado.com>
> Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
> Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
> Co-developed-by: Tzu-Hao Wei <twei@axiado.com>
> Signed-off-by: Tzu-Hao Wei <twei@axiado.com>
> Reviewed-by: Krzysztof Kozlowski <krzysztof.kozlowski@oss.qualcomm.com>
> Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261116-axiado-ax3000-usb-device-controller-v5-0-49271ed5c793@axiado.com?part=1
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v5 2/6] usb: gadget: udc: crg: Add hardware definitions and structs
2026-11-16 19:06 [PATCH v5 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
2026-11-16 19:06 ` [PATCH v5 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC Petar Stepanovic
@ 2026-11-16 19:06 ` Petar Stepanovic
2026-11-16 19:24 ` sashiko-bot
2026-11-16 19:06 ` [PATCH v5 3/6] usb: gadget: udc: crg: Add ring and transfer management Petar Stepanovic
` (3 subsequent siblings)
5 siblings, 1 reply; 13+ messages in thread
From: Petar Stepanovic @ 2026-11-16 19:06 UTC (permalink / raw)
To: Prasad Bolisetty, Greg Kroah-Hartman, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Tzu-Hao Wei,
Karthikeyan Mitran, Philipp Zabel, Neill Kapron
Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
Petar Stepanovic, Krutik Shah, Naveen Kumar Rajgiri Bassappa
This patch introduces the hardware definitions, register offsets, and
fundamental data structures for the Axiado AX3000/AX3005 USB Device
Controller driver.
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Tzu-Hao Wei <twei@axiado.com>
Signed-off-by: Tzu-Hao Wei <twei@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
MAINTAINERS | 1 +
drivers/usb/gadget/udc/crg/crg-udc-debug.h | 143 +++++
drivers/usb/gadget/udc/crg/crg-udc-trace.h | 164 ++++++
drivers/usb/gadget/udc/crg/crg-udc.h | 910 +++++++++++++++++++++++++++++
4 files changed, 1218 insertions(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index 1bf0408e00f3..a3bbf6a752a2 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4669,6 +4669,7 @@ M: Prasad Bolisetty <pbolisetty@axiado.com>
L: linux-usb@vger.kernel.org
S: Supported
F: Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
+F: drivers/usb/gadget/udc/crg/
AYANEO PLATFORM EC DRIVER
M: Antheas Kapenekakis <lkml@antheas.dev>
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-debug.h b/drivers/usb/gadget/udc/crg/crg-udc-debug.h
new file mode 100644
index 000000000000..f73c60a79ade
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-debug.h
@@ -0,0 +1,143 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ * Debug/trace decode helpers.
+ *
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+
+#ifndef __CRG_UDC_DEBUG_H
+#define __CRG_UDC_DEBUG_H
+
+#include "crg-udc.h"
+
+static inline const char *crg_udc_trb_type_string(u32 type)
+{
+ switch (type) {
+ case TRB_TYPE_XFER_NORMAL:
+ return "Normal";
+ case TRB_TYPE_XFER_DATA_STAGE:
+ return "Data Stage";
+ case TRB_TYPE_XFER_STATUS_STAGE:
+ return "Status Stage";
+ case TRB_TYPE_XFER_DATA_ISOCH:
+ return "Isoch";
+ case TRB_TYPE_LINK:
+ return "Link";
+ case TRB_TYPE_NO_OP:
+ return "No-Op";
+ case TRB_TYPE_EVT_TRANSFER:
+ return "Transfer Event";
+ case TRB_TYPE_EVT_CMD_COMPLETION:
+ return "Command Completion Event";
+ case TRB_TYPE_EVT_PORT_STATUS_CHANGE:
+ return "Port Status Change Event";
+ case TRB_TYPE_EVT_MFINDEX_WRAP:
+ return "MFINDEX Wrap Event";
+ case TRB_TYPE_EVT_SETUP_PKT:
+ return "Setup Packet Event";
+ default:
+ return "Reserved";
+ }
+}
+
+static inline const char *crg_udc_cmpl_code_string(u32 code)
+{
+ switch (code) {
+ case CMPL_CODE_INVALID:
+ return "Invalid";
+ case CMPL_CODE_SUCCESS:
+ return "Success";
+ case CMPL_CODE_DATA_BUFFER_ERR:
+ return "Data Buffer Error";
+ case CMPL_CODE_BABBLE_DETECTED_ERR:
+ return "Babble Detected";
+ case CMPL_CODE_USB_TRANS_ERR:
+ return "USB Transaction Error";
+ case CMPL_CODE_TRB_ERR:
+ return "TRB Error";
+ case CMPL_CODE_TRB_STALL:
+ return "Stall";
+ case CMPL_CODE_INVALID_STREAM_TYPE_ERR:
+ return "Invalid Stream Type";
+ case CMPL_CODE_SHORT_PKT:
+ return "Short Packet";
+ case CMPL_CODE_RING_UNDERRUN:
+ return "Ring Underrun";
+ case CMPL_CODE_RING_OVERRUN:
+ return "Ring Overrun";
+ case CMPL_CODE_EVENT_RING_FULL_ERR:
+ return "Event Ring Full";
+ case CMPL_CODE_STOPPED:
+ return "Stopped";
+ case CMPL_CODE_STOPPED_LENGTH_INVALID:
+ return "Stopped - Length Invalid";
+ case CMPL_CODE_ISOCH_BUFFER_OVERRUN:
+ return "Isoch Buffer Overrun";
+ case CMPL_CODE_PROTOCOL_STALL:
+ return "Protocol Stall";
+ case CMPL_CODE_SETUP_TAG_MISMATCH:
+ return "Setup Tag Mismatch";
+ case CMPL_CODE_HALTED:
+ return "Halted";
+ case CMPL_CODE_HALTED_LENGTH_INVALID:
+ return "Halted - Length Invalid";
+ case CMPL_CODE_DISABLED:
+ return "Disabled";
+ case CMPL_CODE_DISABLED_LENGTH_INVALID:
+ return "Disabled - Length Invalid";
+ default:
+ return "Unknown";
+ }
+}
+
+static inline const char *crg_udc_decode_portsc(char *str, size_t size,
+ u32 portsc)
+{
+ scnprintf(str, size, "0x%08x, PLS: %lu, Speed: %lu, %s%s%s%s",
+ portsc, FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, portsc),
+ FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, portsc),
+ (portsc & CRG_U3DC_PORTSC_CCS) ? "CCS " : "",
+ (portsc & CRG_U3DC_PORTSC_PED) ? "PED " : "",
+ (portsc & CRG_U3DC_PORTSC_PR) ? "PR " : "",
+ (portsc & CRG_U3DC_PORTSC_PP) ? "PP " : "");
+
+ return str;
+}
+
+static inline const char *crg_udc_decode_trb(char *str, size_t size,
+ struct crg_transfer_trb *trb)
+{
+ u32 dw2 = le32_to_cpu(trb->dw2);
+ u32 dw3 = le32_to_cpu(trb->dw3);
+ u32 type = FIELD_GET(TRB_TYPE_MASK, dw3);
+
+ scnprintf(str, size,
+ "type: %s, len: %lu, cycle: %lu, chain: %lu, ioc: %lu",
+ crg_udc_trb_type_string(type),
+ FIELD_GET(TRB_TRANSFER_LEN_MASK, dw2),
+ dw3 & TRB_CYCLE_BIT_MASK ? 1UL : 0UL,
+ dw3 & TRB_CHAIN_BIT_MASK ? 1UL : 0UL,
+ dw3 & TRB_INTR_ON_COMPLETION_MASK ? 1UL : 0UL);
+
+ return str;
+}
+
+static inline const char *crg_udc_decode_event_trb(char *str, size_t size,
+ struct crg_event_trb *trb)
+{
+ u32 dw2 = le32_to_cpu(trb->dw2);
+ u32 dw3 = le32_to_cpu(trb->dw3);
+ u32 type = FIELD_GET(EVE_TRB_TYPE_MASK, dw3);
+
+ scnprintf(str, size, "type: %s, cmpl: %s, len: %lu, dci: %lu",
+ crg_udc_trb_type_string(type),
+ crg_udc_cmpl_code_string(
+ FIELD_GET(EVE_TRB_COMPL_CODE_MASK, dw2)),
+ FIELD_GET(EVE_TRB_TRAN_LEN_MASK, dw2),
+ FIELD_GET(EVE_TRB_ENDPOINT_ID_MASK, dw3));
+
+ return str;
+}
+
+#endif /* __CRG_UDC_DEBUG_H */
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-trace.h b/drivers/usb/gadget/udc/crg/crg-udc-trace.h
new file mode 100644
index 000000000000..672a719a041b
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-trace.h
@@ -0,0 +1,164 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ * Trace support header file.
+ *
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM crg-udc
+
+#undef TRACE_SYSTEM_VAR
+#define TRACE_SYSTEM_VAR crg_udc
+
+#if !defined(__CRG_UDC_TRACE_H) || defined(TRACE_HEADER_MULTI_READ)
+#define __CRG_UDC_TRACE_H
+
+#include <linux/types.h>
+#include <linux/tracepoint.h>
+#include <linux/usb/ch9.h>
+#include "crg-udc.h"
+#include "crg-udc-debug.h"
+
+#define CRG_MSG_MAX 200
+
+DECLARE_EVENT_CLASS(crg_udc_log_enable_disable, TP_PROTO(int set), TP_ARGS(set),
+ TP_STRUCT__entry(__field(int, set)),
+ TP_fast_assign(__entry->set = set;),
+ TP_printk("%s", __entry->set ? "enabled" : "disabled"));
+
+DEFINE_EVENT(crg_udc_log_enable_disable, crg_udc_pullup, TP_PROTO(int set),
+ TP_ARGS(set));
+
+TRACE_EVENT(crg_udc_no_room_on_ring, TP_PROTO(char *msg), TP_ARGS(msg),
+ TP_STRUCT__entry(__string(text, msg)),
+ TP_fast_assign(__assign_str(text);),
+ TP_printk("%s", __get_str(text)));
+
+TRACE_EVENT(crg_udc_ep_halt,
+ TP_PROTO(struct crg_udc_ep *udc_ep, u8 halt, u8 wedge),
+ TP_ARGS(udc_ep, halt, wedge),
+ TP_STRUCT__entry(__string(name, udc_ep->usb_ep.name)
+ __field(u8, halt) __field(u8, wedge)),
+ TP_fast_assign(__assign_str(name); __entry->halt = halt;
+ __entry->wedge = wedge;),
+ TP_printk("%s: %s%s", __get_str(name),
+ __entry->halt ? "halted" : "cleared",
+ __entry->wedge ? " (wedged)" : ""));
+
+TRACE_EVENT(crg_udc_portsc, TP_PROTO(u32 portsc), TP_ARGS(portsc),
+ TP_STRUCT__entry(__field(u32, portsc)),
+ TP_fast_assign(__entry->portsc = portsc;),
+ TP_printk("%s",
+ crg_udc_decode_portsc(__get_buf(CRG_MSG_MAX), CRG_MSG_MAX,
+ __entry->portsc)));
+
+TRACE_EVENT(crg_udc_doorbell, TP_PROTO(struct crg_gadget_dev *crg_udc, int dci),
+ TP_ARGS(crg_udc, dci),
+ TP_STRUCT__entry(__string(name, crg_udc->udc_ep[dci].usb_ep.name)
+ __field(int, dci)),
+ TP_fast_assign(__assign_str(name); __entry->dci = dci;),
+ TP_printk("%s, dci %d", __get_str(name), __entry->dci));
+
+DECLARE_EVENT_CLASS(
+ crg_udc_log_ctrl, TP_PROTO(struct usb_ctrlrequest *ctrl), TP_ARGS(ctrl),
+ TP_STRUCT__entry(__field(u8, bRequestType) __field(u8, bRequest)
+ __field(u16, wValue) __field(u16, wIndex)
+ __field(u16, wLength)),
+ TP_fast_assign(__entry->bRequestType = ctrl->bRequestType;
+ __entry->bRequest = ctrl->bRequest;
+ __entry->wValue = le16_to_cpu(ctrl->wValue);
+ __entry->wIndex = le16_to_cpu(ctrl->wIndex);
+ __entry->wLength = le16_to_cpu(ctrl->wLength);),
+ TP_printk("%s", usb_decode_ctrl(__get_buf(CRG_MSG_MAX), CRG_MSG_MAX,
+ __entry->bRequestType,
+ __entry->bRequest, __entry->wValue,
+ __entry->wIndex, __entry->wLength)));
+
+DEFINE_EVENT(crg_udc_log_ctrl, crg_udc_ctrl_req,
+ TP_PROTO(struct usb_ctrlrequest *ctrl), TP_ARGS(ctrl));
+
+DECLARE_EVENT_CLASS(
+ crg_udc_log_trb,
+ TP_PROTO(struct crg_udc_ep *udc_ep, struct crg_transfer_trb *trb),
+ TP_ARGS(udc_ep, trb),
+ TP_STRUCT__entry(__string(name, udc_ep->usb_ep.name)
+ __field(struct crg_transfer_trb *, trb)
+ __dynamic_array(char, decoded,
+ CRG_MSG_MAX)),
+ TP_fast_assign(__assign_str(name); __entry->trb = trb;
+ crg_udc_decode_trb(__get_dynamic_array(decoded),
+ CRG_MSG_MAX, trb);),
+ TP_printk("%s: trb %p, %s", __get_str(name), __entry->trb,
+ __get_str(decoded)));
+
+DEFINE_EVENT(crg_udc_log_trb, crg_udc_queue_trb,
+ TP_PROTO(struct crg_udc_ep *udc_ep, struct crg_transfer_trb *trb),
+ TP_ARGS(udc_ep, trb));
+
+TRACE_EVENT(
+ crg_udc_event_trb, TP_PROTO(struct crg_event_trb *trb), TP_ARGS(trb),
+ TP_STRUCT__entry(__field(struct crg_event_trb *, trb)
+ __dynamic_array(char, decoded, CRG_MSG_MAX)),
+ TP_fast_assign(__entry->trb = trb;
+ crg_udc_decode_event_trb(__get_dynamic_array(decoded),
+ CRG_MSG_MAX, trb);),
+ TP_printk("trb %p, %s", __entry->trb, __get_str(decoded)));
+
+DECLARE_EVENT_CLASS(
+ crg_udc_log_request,
+ TP_PROTO(struct crg_udc_ep *udc_ep, struct crg_udc_request *udc_req),
+ TP_ARGS(udc_ep, udc_req),
+ TP_STRUCT__entry(
+ __string(name, udc_ep->usb_ep.name) __field(
+ struct usb_request *, request) __field(void *, buf)
+ __field(unsigned int, actual) __field(unsigned int,
+ length)
+ __field(int, status) __field(dma_addr_t, dma)
+ __field(int, zero) __field(int,
+ short_not_ok)
+ __field(int, no_interrupt)),
+ TP_fast_assign(__assign_str(name); __entry->request = &udc_req->usb_req;
+ __entry->buf = udc_req->usb_req.buf;
+ __entry->actual = udc_req->usb_req.actual;
+ __entry->length = udc_req->usb_req.length;
+ __entry->status = udc_req->usb_req.status;
+ __entry->dma = udc_req->usb_req.dma;
+ __entry->zero = udc_req->usb_req.zero;
+ __entry->short_not_ok = udc_req->usb_req.short_not_ok;
+ __entry->no_interrupt = udc_req->usb_req.no_interrupt;),
+ TP_printk(
+ "%s: req: %p, buf: %p, length: %u/%u, status: %d, dma: %pad, %s%s%s",
+ __get_str(name), __entry->request, __entry->buf,
+ __entry->actual, __entry->length, __entry->status,
+ &__entry->dma, __entry->zero ? "Z" : "z",
+ __entry->short_not_ok ? "S" : "s",
+ __entry->no_interrupt ? "I" : "i"));
+
+DEFINE_EVENT(crg_udc_log_request, crg_udc_alloc_request,
+ TP_PROTO(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req),
+ TP_ARGS(udc_ep, udc_req));
+
+DEFINE_EVENT(crg_udc_log_request, crg_udc_free_request,
+ TP_PROTO(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req),
+ TP_ARGS(udc_ep, udc_req));
+
+DEFINE_EVENT(crg_udc_log_request, crg_udc_request_giveback,
+ TP_PROTO(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req),
+ TP_ARGS(udc_ep, udc_req));
+
+#endif /* __CRG_UDC_TRACE_H */
+
+/* This part must be outside header guard. */
+
+#undef TRACE_INCLUDE_PATH
+#define TRACE_INCLUDE_PATH .
+
+#undef TRACE_INCLUDE_FILE
+#define TRACE_INCLUDE_FILE crg-udc-trace
+
+#include <trace/define_trace.h>
diff --git a/drivers/usb/gadget/udc/crg/crg-udc.h b/drivers/usb/gadget/udc/crg/crg-udc.h
new file mode 100644
index 000000000000..afc81a303f07
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc.h
@@ -0,0 +1,910 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+
+#ifndef CRG_UDC_H
+#define CRG_UDC_H
+
+#include <linux/bitfield.h>
+#include <linux/bits.h>
+#include <linux/dma-mapping.h>
+#include <linux/io.h>
+#include <linux/kernel.h>
+#include <linux/scatterlist.h>
+#include <linux/types.h>
+#include <linux/wait.h>
+#include <linux/workqueue.h>
+
+#include <linux/usb/ch9.h>
+#include <linux/usb/gadget.h>
+
+#define CRG_UCCR_OFFSET 0x2400
+#define CRG_UICR_OFFSET 0x2500
+#define CRG_UICR_STRIDE 0x20
+
+/* Corigine USB 3.1 device core register layout. */
+struct crg_uccr {
+ u32 capability;
+ u32 resv0[3];
+
+ u32 config0;
+ u32 config1;
+ u32 resv1[2];
+
+ u32 control;
+ u32 status;
+ u32 dcbaplo;
+ u32 dcbaphi;
+ u32 portsc;
+ u32 u3portpmsc;
+ u32 u2portpmsc;
+ u32 u3portli;
+
+ u32 doorbell;
+ u32 mfindex;
+ u32 ptmctrl;
+ u32 resv3[5];
+
+ u32 ep_enable;
+ u32 ep_running;
+ u32 resv4[2];
+
+ u32 cmd_param0;
+ u32 cmd_param1;
+ u32 cmd_control;
+ u32 resv5[1];
+
+ u32 odb_capability;
+ u32 resv6[3];
+ u32 odb_config[8];
+
+ u32 debug0;
+};
+
+struct crg_uicr {
+ u32 iman;
+ u32 imod;
+ u32 erstsz;
+ u32 resv0;
+
+ u32 erstbalo;
+ u32 erstbahi;
+ u32 erdplo;
+ u32 erdphi;
+};
+
+/* config0/config1 registers */
+#define CRG_U3DC_CFG0_MAXSPEED_MASK GENMASK(3, 0)
+#define CRG_U3DC_CFG0_MAXSPEED_FS 1
+#define CRG_U3DC_CFG0_MAXSPEED_HS 3
+#define CRG_U3DC_CFG0_MAXSPEED_SS 4
+#define CRG_U3DC_CFG0_MAXSPEED_SSP 5
+
+#define CRG_U3DC_CFG1_CSC_EVENT_EN BIT(0)
+#define CRG_U3DC_CFG1_PEC_EVENT_EN BIT(1)
+#define CRG_U3DC_CFG1_PPC_EVENT_EN BIT(3)
+#define CRG_U3DC_CFG1_PRC_EVENT_EN BIT(4)
+#define CRG_U3DC_CFG1_PLC_EVENT_EN BIT(5)
+#define CRG_U3DC_CFG1_CEC_EVENT_EN BIT(6)
+
+#define CRG_U3DC_CFG1_U3_ENTRY_EN BIT(8)
+#define CRG_U3DC_CFG1_L1_ENTRY_EN BIT(9)
+#define CRG_U3DC_CFG1_U3_RESUME_EN BIT(10)
+#define CRG_U3DC_CFG1_L1_RESUME_EN BIT(11)
+#define CRG_U3DC_CFG1_INACTIVE_PLC_EN BIT(12)
+#define CRG_U3DC_CFG1_U3_RESUME_NORESP_PLC_EN BIT(13)
+#define CRG_U3DC_CFG1_U2_RESUME_NORESP_PLC_EN BIT(14)
+
+#define CRG_U3DC_CFG1_SETUP_EVENT_EN BIT(16)
+
+/* control register */
+#define CRG_U3DC_CTRL_RUN BIT(0)
+#define CRG_U3DC_CTRL_STOP 0
+#define CRG_U3DC_CTRL_SWRST BIT(1)
+#define CRG_U3DC_CTRL_INT_EN BIT(2)
+#define CRG_U3DC_CTRL_SYSERR_EN BIT(3)
+#define CRG_U3DC_CTRL_EWE BIT(10)
+#define CRG_U3DC_CTRL_KP_CNCT BIT(11)
+
+/* status register */
+#define CRG_U3DC_STATUS_DEV_CTRL_HALT BIT(0)
+#define CRG_U3DC_STATUS_SYS_ERR BIT(2)
+#define CRG_U3DC_STATUS_EINT BIT(3)
+
+/* portsc register */
+#define CRG_U3DC_PORTSC_CCS BIT(0)
+#define CRG_U3DC_PORTSC_PED BIT(1)
+#define CRG_U3DC_PORTSC_PP BIT(3)
+#define CRG_U3DC_PORTSC_PR BIT(4)
+#define CRG_U3DC_PORTSC_PLS_MASK GENMASK(8, 5)
+#define CRG_U3DC_PORTSC_PLS_U3 3
+
+#define CRG_U3DC_PORTSC_SPEED_MASK GENMASK(13, 10)
+#define CRG_U3DC_PORTSC_SPEED_FS 0x1
+#define CRG_U3DC_PORTSC_SPEED_LS 0x2
+#define CRG_U3DC_PORTSC_SPEED_HS 0x3
+#define CRG_U3DC_PORTSC_SPEED_SS 0x4
+#define CRG_U3DC_PORTSC_SPEED_SSP 0x5
+
+#define CRG_U3DC_PORTSC_LWS BIT(16)
+#define CRG_U3DC_PORTSC_CSC BIT(17)
+#define CRG_U3DC_PORTSC_PEC BIT(18)
+#define CRG_U3DC_PORTSC_PPC BIT(20)
+#define CRG_U3DC_PORTSC_PRC BIT(21)
+#define CRG_U3DC_PORTSC_PLC BIT(22)
+#define CRG_U3DC_PORTSC_CEC BIT(23)
+#define CRG_U3DC_PORTSC_WCE BIT(25)
+#define CRG_U3DC_PORTSC_WDE BIT(26)
+#define CRG_U3DC_PORTSC_WPR BIT(31)
+
+#define CRG_U3DC_PORTSC_W1C_MASK \
+ (CRG_U3DC_PORTSC_CSC | CRG_U3DC_PORTSC_PEC | CRG_U3DC_PORTSC_PPC | \
+ CRG_U3DC_PORTSC_PRC | CRG_U3DC_PORTSC_PLC | CRG_U3DC_PORTSC_CEC)
+
+/* mfindex register */
+#define CRG_U3DC_MFINDEX_CURRENT_MASK GENMASK(17, 4)
+#define CRG_U3DC_MFINDEX_FRAME_MASK GENMASK(17, 7)
+
+/* ptmctrl register */
+#define CRG_U3DC_PTMCTRL_ISOCH_DELAY_MASK GENMASK(13, 0)
+
+/* u3portpmsc */
+#define CRG_U3DC_U3PORTPM_U1TMOUT_MASK GENMASK(7, 0)
+#define CRG_U3DC_U3PORTPM_U2TMOUT_MASK GENMASK(15, 8)
+#define CRG_U3DC_U3PORTPM_FLA BIT(16)
+#define CRG_U3DC_U3PORTPM_U1IEN BIT(20)
+#define CRG_U3DC_U3PORTPM_U2IEN BIT(21)
+#define CRG_U3DC_U3PORTPM_U1AEN BIT(22)
+#define CRG_U3DC_U3PORTPM_U2AEN BIT(23)
+
+#define CRG_U3DC_U3PORTPM_U1U2TMOUT_MASK GENMASK(31, 24)
+
+/* u2portpmsc */
+#define CRG_U3DC_U2PORTPM_RJ_TH_MASK GENMASK(3, 0)
+#define CRG_U3DC_U2PORTPM_DS_TH_MASK GENMASK(7, 4)
+#define CRG_U3DC_U2PORTPM_LPM_EN BIT(8)
+#define CRG_U3DC_U2PORTPM_RJ_TH_EN BIT(9)
+#define CRG_U3DC_U2PORTPM_DS_EN BIT(10)
+#define CRG_U3DC_U2PORTPM_SLP_EN BIT(11)
+#define CRG_U3DC_U2PORTPM_LPM_FACK BIT(12)
+#define CRG_U3DC_U2PORTPM_L1_AEX BIT(13)
+#define CRG_U3DC_U2PORTPM_H_B_MASK GENMASK(19, 16)
+#define CRG_U3DC_U2PORTPM_RWE BIT(20)
+#define CRG_U3DC_U2PORTPM_TM_MASK GENMASK(31, 28)
+
+/* doorbell register */
+#define CRG_U3DC_DB_TARGET_MASK GENMASK(4, 0)
+
+/* odb registers */
+#define CRG_U3DC_ODBCFG_2N_OFFSET_MASK GENMASK(9, 0)
+#define CRG_U3DC_ODBCFG_2N_SIZE_MASK GENMASK(12, 10)
+#define CRG_U3DC_ODBCFG_2N1_OFFSET_MASK GENMASK(25, 16)
+#define CRG_U3DC_ODBCFG_2N1_SIZE_MASK GENMASK(28, 26)
+
+/* command control register */
+#define CRG_U3DC_CMD_CTRL_ACTIVE BIT(0)
+#define CRG_U3DC_CMD_CTRL_IOC_EN BIT(1)
+#define CRG_U3DC_CMD_CTRL_TYPE_MASK GENMASK(7, 4)
+#define CRG_U3DC_CMD_CTRL_STATUS_MASK GENMASK(19, 16)
+
+/* interrupt register / IMAN bits */
+#define CRG_U3DC_IMAN_INT_PEND BIT(0)
+#define CRG_U3DC_IMAN_INT_EN BIT(1)
+
+/* erdp bits */
+#define CRG_U3DC_ERDPLO_EHB BIT(3)
+
+/* command parameters: init EP0 */
+#define CRG_CMD0_0_DQPTRLO_MASK GENMASK(31, 4)
+#define CRG_CMD0_0_DCS_MASK BIT(0)
+
+/* command parameters: update EP0 */
+#define CRG_CMD1_0_MPS_MASK GENMASK(31, 16)
+
+/* command parameters: set address */
+#define CRG_CMD2_0_DEV_ADDR_MASK GENMASK(7, 0)
+
+#define CRG_UDC_MODE_REG 0x20fc
+#define CRG_UDC_ROLE_DEVICE 0x1
+
+#define CRG_UDC_VENDOR_REG 0x210c
+#define CRG_UDC_VBUS_DETECT BIT(26)
+
+/*
+ * Device-visible data structures. The controller reads and writes these
+ * over DMA, so the layouts and field encodings below are fixed by the
+ * hardware in the same way as the register definitions above.
+ */
+
+/* Transfer and event TRB types */
+enum crg_trb_type {
+ TRB_TYPE_RSVD = 0,
+ TRB_TYPE_XFER_NORMAL,
+ TRB_TYPE_RSVD2,
+ TRB_TYPE_XFER_DATA_STAGE,
+ TRB_TYPE_XFER_STATUS_STAGE,
+ TRB_TYPE_XFER_DATA_ISOCH,
+ TRB_TYPE_LINK,
+ TRB_TYPE_RSVD7,
+ TRB_TYPE_NO_OP,
+
+ TRB_TYPE_EVT_TRANSFER = 32,
+ TRB_TYPE_EVT_CMD_COMPLETION = 33,
+ TRB_TYPE_EVT_PORT_STATUS_CHANGE = 34,
+ TRB_TYPE_EVT_MFINDEX_WRAP = 39,
+ TRB_TYPE_EVT_SETUP_PKT = 40,
+};
+
+/* Event TRB completion codes */
+enum crg_trb_cmpl_code {
+ CMPL_CODE_INVALID = 0,
+ CMPL_CODE_SUCCESS,
+ CMPL_CODE_DATA_BUFFER_ERR,
+ CMPL_CODE_BABBLE_DETECTED_ERR,
+ CMPL_CODE_USB_TRANS_ERR,
+ CMPL_CODE_TRB_ERR,
+ CMPL_CODE_TRB_STALL,
+ CMPL_CODE_INVALID_STREAM_TYPE_ERR = 10,
+ CMPL_CODE_SHORT_PKT = 13,
+ CMPL_CODE_RING_UNDERRUN,
+ CMPL_CODE_RING_OVERRUN,
+ CMPL_CODE_EVENT_RING_FULL_ERR = 21,
+ CMPL_CODE_STOPPED = 26,
+ CMPL_CODE_STOPPED_LENGTH_INVALID = 27,
+ CMPL_CODE_ISOCH_BUFFER_OVERRUN = 31,
+
+ CMPL_CODE_PROTOCOL_STALL = 192,
+ CMPL_CODE_SETUP_TAG_MISMATCH = 193,
+ CMPL_CODE_HALTED = 194,
+ CMPL_CODE_HALTED_LENGTH_INVALID = 195,
+ CMPL_CODE_DISABLED = 196,
+ CMPL_CODE_DISABLED_LENGTH_INVALID = 197,
+};
+
+/* Endpoint type encoding in the endpoint context */
+enum crg_ep_type {
+ EP_TYPE_INVALID = 0,
+ EP_TYPE_ISOCH_OUTBOUND,
+ EP_TYPE_BULK_OUTBOUND,
+ EP_TYPE_INTR_OUTBOUND,
+ EP_TYPE_INVALID2,
+ EP_TYPE_ISOCH_INBOUND,
+ EP_TYPE_BULK_INBOUND,
+ EP_TYPE_INTR_INBOUND,
+};
+
+/* crg_transfer_trb dw2 fields */
+#define TRB_TRANSFER_LEN_MASK GENMASK(16, 0)
+#define TRB_TD_SIZE_MASK GENMASK(21, 17)
+#define TRB_INTR_TARGET_MASK GENMASK(31, 22)
+
+/* crg_transfer_trb dw3 fields */
+#define TRB_CYCLE_BIT_MASK BIT(0)
+#define TRB_LINK_TOGGLE_CYCLE_MASK BIT(1)
+#define TRB_INTR_ON_SHORT_PKT_MASK BIT(2)
+#define TRB_NO_SNOOP_MASK BIT(3)
+#define TRB_CHAIN_BIT_MASK BIT(4)
+#define TRB_INTR_ON_COMPLETION_MASK BIT(5)
+#define TRB_APPEND_ZLP_MASK BIT(7)
+#define TRB_BLOCK_EVENT_INT_MASK BIT(9)
+#define TRB_TYPE_MASK GENMASK(15, 10)
+#define DATA_STAGE_TRB_DIR_MASK BIT(16)
+#define TRB_SETUP_TAG_MASK GENMASK(18, 17)
+#define STATUS_STAGE_TRB_STALL_MASK BIT(19)
+#define STATUS_STAGE_TRB_SET_ADDR_MASK BIT(20)
+#define ISOC_TRB_FRAME_ID_MASK GENMASK(30, 20)
+#define ISOC_TRB_SIA_MASK BIT(31)
+
+struct crg_transfer_trb {
+ __le32 dw0;
+ __le32 dw1;
+ __le32 dw2;
+ __le32 dw3;
+};
+
+/* crg_event_trb dw2 fields */
+#define EVE_TRB_TRAN_LEN_MASK GENMASK(16, 0)
+#define EVE_TRB_COMPL_CODE_MASK GENMASK(31, 24)
+
+/* crg_event_trb dw3 fields */
+#define EVE_TRB_CYCLE_BIT_MASK BIT(0)
+#define EVE_TRB_TYPE_MASK GENMASK(15, 10)
+#define EVE_TRB_ENDPOINT_ID_MASK GENMASK(20, 16)
+#define EVE_TRB_SETUP_TAG_MASK GENMASK(22, 21)
+
+struct crg_event_trb {
+ __le32 dw0;
+ __le32 dw1;
+ __le32 dw2;
+ __le32 dw3;
+};
+
+/* crg_ep_cx dw0 fields */
+#define EP_CX_LOGICAL_EP_NUM_MASK GENMASK(6, 3)
+#define EP_CX_INTERVAL_MASK GENMASK(23, 16)
+
+/* crg_ep_cx dw1 fields */
+#define EP_CX_EP_TYPE_MASK GENMASK(5, 3)
+#define EP_CX_MAX_BURST_SIZE_MASK GENMASK(15, 8)
+#define EP_CX_MAX_PACKET_SIZE_MASK GENMASK(31, 16)
+
+/* crg_ep_cx dw2 fields */
+#define EP_CX_DEQ_CYC_STATE_MASK BIT(0)
+#define EP_CX_TR_DQPT_LO_MASK GENMASK(31, 4)
+
+struct crg_ep_cx {
+ __le32 dw0;
+ __le32 dw1;
+ __le32 dw2;
+ __le32 dw3;
+};
+
+struct crg_erst {
+ /* 64-bit event ring segment address */
+ __le32 seg_addr_lo;
+ __le32 seg_addr_hi;
+ __le32 seg_size;
+ /* Set to zero */
+ __le32 rsvd;
+};
+
+/* command type */
+enum crg_cmd_type {
+ CRG_CMD_INIT_EP0 = 0,
+ CRG_CMD_UPDATE_EP0_CFG = 1,
+ CRG_CMD_SET_ADDR = 2,
+ CRG_CMD_SEND_DEV_NOTIFICATION = 3,
+ CRG_CMD_CONFIG_EP = 4,
+ CRG_CMD_SET_HALT = 5,
+ CRG_CMD_CLEAR_HALT = 6,
+ CRG_CMD_RESET_SEQNUM = 7,
+ CRG_CMD_STOP_EP = 8,
+ CRG_CMD_SET_TR_DQPTR = 9,
+ CRG_CMD_FORCE_FLOW_CONTROL = 10,
+ CRG_CMD_REQ_LDM_EXCHANGE = 11,
+};
+
+#define CRG_MAX_PACKET_SIZE 1024
+
+#define CRG_ERST_SIZE 1
+#define CRG_EVENT_RING_SIZE 256
+#define CRG_NUM_EP_CX 32
+
+/*
+ * Cap on events drained per udc_lock acquisition in crg_udc_process_event_ring().
+ * Command polling and crg_udc_reset()'s hardware-reset poll both run from
+ * crg_udc_handle_event() with udc_lock held, so without a cap a busy event
+ * ring could keep the lock, and local IRQs, held for as long as events
+ * keep arriving.
+ */
+#define CRG_EVENT_RING_BATCH 32
+
+#define CRG_UDC_TRB_MAX_BUFFER_SIZE 65536
+#define CRG_UDC_CONTROL_EP_TD_RING_SIZE 16
+#define CRG_UDC_BULK_EP_TD_RING_SIZE 1024
+#define CRG_UDC_ISOC_EP_TD_RING_SIZE 32
+#define CRG_UDC_INT_EP_TD_RING_SIZE 8
+
+#define CRG_UDC_U1_TIMEOUT_VAL 0x70
+#define CRG_UDC_U2_TIMEOUT_VAL 0x70
+
+enum crg_ep_state {
+ EP_STATE_DISABLED = 0,
+ EP_STATE_RUNNING = 1,
+ EP_STATE_HALTED = 2,
+ EP_STATE_STOPPED = 3
+};
+
+/* control-transfer setup state machine */
+enum crg_setup_status {
+ WAIT_FOR_SETUP = 0,
+ SETUP_PKT_PROCESS_IN_PROGRESS,
+ DATA_STAGE_XFER,
+ DATA_STAGE_RECV,
+ STATUS_STAGE_XFER,
+ STATUS_STAGE_RECV,
+};
+
+/* interrupt-on-completion moderation rate (in TRBs) per endpoint type */
+enum {
+ CRG_UDC_BULK_EP_INTERRUPT_RATE = 5,
+ CRG_UDC_ISOC_EP_INTERRUPT_RATE = 1,
+};
+
+struct crg_buffer_info {
+ void *cpu_addr;
+ dma_addr_t dma_addr;
+ u32 len;
+};
+
+struct crg_sel_value {
+ u16 u2_pel_value;
+ u16 u2_sel_value;
+ u8 u1_pel_value;
+ u8 u1_sel_value;
+};
+
+/* May outlive the request if hardware still owns its DMA addresses. */
+struct crg_udc_bounce {
+ struct list_head quarantine;
+ struct list_head pages;
+ struct sg_table sgt;
+ void *buf;
+ dma_addr_t dma;
+ size_t len;
+ enum dma_data_direction dir;
+ bool mapped;
+};
+
+struct crg_udc_request {
+ struct usb_request usb_req;
+ struct list_head queue;
+ struct crg_udc_bounce *bounce;
+ struct scatterlist *sg;
+ unsigned int sg_index;
+ u32 sg_offset;
+ u64 buff_len_left;
+ u32 trbs_needed;
+ struct crg_transfer_trb *first_trb;
+ struct crg_transfer_trb *last_trb;
+ /* Bytes queued in TDs that completed before the current TD. */
+ u32 td_start_offset;
+ bool all_trbs_queued;
+ /* A TD has been published and its completion event is outstanding. */
+ bool td_pending;
+};
+
+struct crg_udc_ep {
+ struct usb_ep usb_ep;
+
+ struct crg_buffer_info tran_ring_alloc;
+ struct crg_buffer_info tran_ring_info;
+ struct crg_transfer_trb *first_trb;
+ struct crg_transfer_trb *last_trb;
+
+ struct crg_transfer_trb *enq_pt;
+ struct crg_transfer_trb *deq_pt;
+ u8 pcs;
+
+ char name[10];
+ u8 dci;
+ struct list_head queue;
+ const struct usb_endpoint_descriptor *desc;
+ const struct usb_ss_ep_comp_descriptor *comp_desc;
+ bool tran_ring_full;
+ bool configuring;
+ /* HALT/STOP did not confirm DMA stopped; keep requests mapped. */
+ bool halt_failed;
+ /* A failed disable must not be recovered by SET_HALT/CLEAR_HALT. */
+ bool disable_failed;
+ /* STOP_EP completed in ep_dequeue(); giveback callbacks may queue,
+ * but must not publish TRBs until the original dequeue resumes.
+ */
+ bool dequeue_paused;
+ /* Keep giveback callbacks from repopulating a queue being drained. */
+ bool nuking;
+ bool clear_enabled_on_nuke;
+ u32 generation;
+ struct crg_gadget_dev *crg_udc;
+
+ enum crg_ep_state ep_state;
+
+ bool wedge;
+};
+
+#define CRG_RING_NUM 1
+
+struct crg_udc_event {
+ struct crg_buffer_info erst;
+ struct crg_erst *p_erst;
+ struct crg_buffer_info event_ring;
+ struct crg_event_trb *evt_dq_pt;
+ /* Changed on every ring reinitialization under udc_lock. */
+ u32 generation;
+ u8 ccs;
+ struct crg_event_trb *evt_seg0_last_trb;
+};
+
+enum crg_ep0_state {
+ EP0_STATE_IDLE = 0,
+ EP0_STATE_ABORTING,
+ EP0_STATE_ERROR,
+};
+
+struct crg_gadget_dev {
+ void __iomem *mmio_virt_base;
+ struct crg_uccr __iomem *uccr;
+ struct crg_uicr __iomem *uicr[CRG_RING_NUM];
+
+ /* udc_lock device lock */
+ spinlock_t udc_lock;
+
+ struct device *dev;
+ struct clk *clk;
+ struct phy *phy;
+ struct reset_control *rst;
+ bool hw_reset_asserted;
+ bool dma_quiesce_failed;
+ bool external_reset_pending;
+ /* Endpoint command failure needs SWRST and giveback in process context. */
+ bool endpoint_recovery_pending;
+ /* Set RUN after composite drops the pre-reset configuration. */
+ bool restart_after_reset;
+ /* Set when DMA addresses are programmed; cleared by a confirmed reset. */
+ bool hw_dma_armed;
+ bool transfer_ring_bank;
+ bool transfer_rings_live;
+ /* A failed reset/reconstruction prevents all new controller activity. */
+ bool controller_faulted;
+ bool gadget_stopping;
+ wait_queue_head_t teardown_wait;
+ unsigned int queueing;
+ struct list_head dma_quarantine;
+
+ struct usb_gadget gadget;
+ struct usb_gadget_driver *gadget_driver;
+ /* Speed selected for the currently bound gadget driver. */
+ enum usb_device_speed max_speed;
+
+ int irq;
+
+ struct usb_endpoint_descriptor ep0_desc;
+
+ struct crg_udc_ep udc_ep[CRG_NUM_EP_CX];
+ struct crg_buffer_info ep_cx;
+ struct crg_ep_cx *p_epcx;
+
+ struct crg_udc_event udc_event[CRG_RING_NUM];
+
+ struct crg_udc_request *status_req;
+ /* Protected by udc_lock, remains set during a data-stage giveback. */
+ bool status_req_callback_pending;
+ struct crg_sel_value sel_value;
+ void (*setup_fn_call_back)(struct crg_gadget_dev *a);
+
+ enum crg_setup_status setup_status;
+ /* EP0 rejects new requests while the previous control transfer is
+ * being aborted and its completion callbacks are being given back.
+ */
+ enum crg_ep0_state ep0_state;
+
+ u8 device_state;
+ u16 dev_addr;
+ u8 setup_tag;
+ u8 set_tm;
+ /* Latest SETUP event received before post-reset EP0 initialization. */
+ struct usb_ctrlrequest pending_setup;
+ u8 pending_setup_tag;
+ bool pending_setup_valid;
+
+ u32 num_enabled_eps;
+
+ bool connected;
+ bool pullup_on;
+ bool async_callbacks;
+ /*
+ * Set while crg_udc_reset() has udc_lock dropped and is polling for
+ * SWRST to clear. Checked by crg_udc_ep_queue() so a request queued
+ * on another endpoint during that window is rejected instead of
+ * being built into a ring and doorbell-rung while the controller is
+ * still mid-reset.
+ */
+ bool resetting;
+ /* Device lifecycle gates held across IRQ and reset-work callbacks. */
+ bool suspended;
+ bool shutting_down;
+ /* RUN was stopped while an endpoint still had DMA-visible TRBs. */
+ bool stopped_needs_reset;
+ /* Covers SWRST, callback giveback, and context/EP0 reconstruction. */
+ bool controller_recovering;
+ /* Quiesced hardware; composite still needs to disable its endpoints. */
+ bool bus_reset_pending;
+ /* Invalidates an older reset worker after a newer reset/disconnect. */
+ u32 bus_reset_generation;
+ /* Serialize the gadget reset callback against new port recovery. */
+ bool reset_callback_active;
+ /* External gadget callbacks may drop udc_lock and invoke ep_disable(). */
+ unsigned int gadget_callbacks_active;
+ /* Keep core endpoint-enabled state until composite reset runs. */
+ bool preserve_ep_enabled_on_reset;
+ /* System errors are recovered after the gadget reset callback. */
+ bool deferred_system_error;
+ /* W1C port events acknowledged while that callback was running. */
+ u32 deferred_port_events;
+ /* PRC was acknowledged before the port had a valid enabled link. */
+ bool port_reset_pending;
+ /* Retry acknowledged PRC when no further port change IRQ arrives. */
+ struct delayed_work port_retry_work;
+ u8 port_reset_retry_count;
+
+ bool u2_rwe;
+ bool feature_u1_enable;
+ bool feature_u2_enable;
+
+ bool portsc_on_reconnecting;
+ bool u12_forbidden;
+ bool u12_initiate_forbidden;
+ bool reinit_ep0_on_bus_reset;
+
+ /*
+ * usb_gadget_udc_reset() re-enters the gadget/function drivers
+ * synchronously. Calling it directly from the port-status IRQ path
+ * would require dropping udc_lock mid-event-processing, opening a
+ * window for a second port-status event to run crg_udc_reinit()
+ * again while the first event's SET_CONFIGURATION/set_alt() work is
+ * still landing on another thread, tearing the just-enabled
+ * endpoints back down. Defer the notification to this work item so
+ * the IRQ-thread's event processing never has to drop the lock.
+ */
+ struct work_struct udc_reset_work;
+ /* EP0 flushes udc_reset_work, so use separate work to drain the IRQ. */
+ struct work_struct recovery_work;
+};
+
+/*
+ * Caller must hold udc_lock. These helpers describe the current USB session;
+ * they do not replace endpoint, SETUP-tag, or generation validation.
+ */
+static inline bool crg_udc_session_ready(const struct crg_gadget_dev *udc)
+{
+ return !udc->controller_faulted && !udc->resetting &&
+ !udc->controller_recovering && !udc->endpoint_recovery_pending &&
+ !udc->shutting_down && !udc->gadget_stopping && !udc->suspended &&
+ udc->pullup_on && !udc->stopped_needs_reset;
+}
+
+static inline bool crg_udc_gadget_ready(const struct crg_gadget_dev *udc)
+{
+ return crg_udc_session_ready(udc) && udc->gadget_driver &&
+ udc->async_callbacks;
+}
+
+/* Keep usb_ep_disable() in the driver until the last giveback returns. */
+static inline void crg_udc_ep_clear_enabled(struct crg_udc_ep *ep)
+{
+ if (ep->nuking)
+ ep->clear_enabled_on_nuke = true;
+ else
+ ep->usb_ep.enabled = false;
+}
+
+/*
+ * TD Size is the number of USB packets remaining in this TD after this TRB,
+ * not the number of TRBs left. See Corigine USB 3.1 device spec, 8.2.2.
+ * A partially consumed packet is counted until the last TRB of the TD.
+ */
+static inline u32 crg_udc_td_size(u32 td_bytes, u32 bytes_through_trb,
+ u32 maxpacket, bool last_trb)
+{
+ u32 packets;
+
+ if (last_trb || !td_bytes)
+ return 0;
+
+ packets = DIV_ROUND_UP(td_bytes, maxpacket) -
+ bytes_through_trb / maxpacket;
+ return min(packets, 31U);
+}
+
+/*
+ * Controller recovery and bus-reset reinitialization use the same gate.
+ * Call these with udc_lock held at the outermost operation.
+ */
+static inline void crg_udc_recovery_begin(struct crg_gadget_dev *udc)
+{
+ udc->controller_recovering = true;
+}
+
+/*
+ * Called under udc_lock. Never run composite reset concurrently with
+ * ->setup(), ->disconnect() or a request completion callback.
+ */
+static inline void crg_udc_kick_reset_work(struct crg_gadget_dev *udc)
+{
+ if (udc->gadget_callbacks_active || udc->reset_callback_active ||
+ udc->controller_recovering || udc->shutting_down ||
+ udc->gadget_stopping || udc->suspended)
+ return;
+
+ if (udc->endpoint_recovery_pending || udc->external_reset_pending)
+ schedule_work(&udc->recovery_work);
+ else if (udc->bus_reset_pending && !udc->resetting &&
+ udc->async_callbacks && udc->gadget_driver)
+ schedule_work(&udc->udc_reset_work);
+}
+
+static inline void crg_udc_recovery_end(struct crg_gadget_dev *udc)
+{
+ udc->controller_recovering = false;
+ if (udc->gadget_stopping || udc->shutting_down)
+ wake_up(&udc->teardown_wait);
+ if (udc->endpoint_recovery_pending)
+ crg_udc_kick_reset_work(udc);
+}
+
+/* Called with udc_lock held. Leave DMA-visible memory owned by the worker. */
+static inline void crg_udc_schedule_endpoint_recovery(struct crg_gadget_dev *udc)
+{
+ udc->endpoint_recovery_pending = true;
+ crg_udc_kick_reset_work(udc);
+}
+
+static inline void crg_udc_callback_enter(struct crg_gadget_dev *udc)
+{
+ udc->gadget_callbacks_active++;
+}
+
+static inline void crg_udc_callback_exit(struct crg_gadget_dev *udc)
+{
+ udc->gadget_callbacks_active--;
+ if (udc->gadget_stopping && !udc->gadget_callbacks_active)
+ wake_up(&udc->teardown_wait);
+ crg_udc_kick_reset_work(udc);
+}
+
+/* Shared prototypes */
+int crg_udc_build_ep0_status(struct crg_udc_ep *udc_ep_ptr, bool default_value,
+ u32 status, struct crg_udc_request *udc_req_ptr,
+ u8 set_addr, u8 stall);
+/*
+ * Abort the current EP0 control transfer when a new SETUP packet arrives.
+ *
+ * A SETUP packet is allowed to abort any in-progress control transfer. Do
+ * not defer the new SETUP until the old transfer completes: doing so leaves
+ * the old request on the EP0 software queue and subsequent gadget requests
+ * are rejected as "queue not empty".
+ *
+ * The controller's EP0 ring is reinitialized at the ring start, making all
+ * TRBs belonging to the aborted control transfer stale. The new SETUP is
+ * then handled immediately with a fresh setup tag.
+ */
+int crg_udc_abort_ep0_for_setup(struct crg_gadget_dev *crg_udc);
+int crg_udc_enable_setup(struct crg_gadget_dev *crg_udc);
+void crg_udc_ep0_req_complete(struct crg_udc_ep *udc_ep_ptr,
+ unsigned long *flags);
+int crg_udc_prepare_for_setup(struct crg_gadget_dev *crg_udc);
+int crg_udc_set_ep0_halt(struct crg_gadget_dev *crg_udc);
+int crg_udc_update_ep0_maxpacketsize(struct crg_gadget_dev *crg_udc);
+void crg_udc_handle_setup_pkt(struct crg_gadget_dev *crg_udc,
+ struct usb_ctrlrequest *setup_pkt, u8 setup_tag,
+ unsigned long *flags);
+int crg_udc_init_ep0(struct crg_gadget_dev *crg_udc);
+int crg_udc_queue_ctrl(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr,
+ u32 num_of_trbs_needed);
+void crg_udc_advance_dequeue_pt(struct crg_udc_ep *udc_ep);
+unsigned int crg_udc_count_trbs_linear(dma_addr_t addr, u64 len);
+unsigned int crg_udc_count_trbs_sg(struct crg_udc_request *udc_req);
+int crg_udc_init_event_ring(struct crg_gadget_dev *crg_udc, int index);
+bool crg_udc_is_event_rings_empty(struct crg_gadget_dev *crg_udc);
+bool crg_udc_is_pointer_less_than(struct crg_transfer_trb *a,
+ struct crg_transfer_trb *b,
+ struct crg_udc_ep *udc_ep);
+int crg_udc_process_event_ring(struct crg_gadget_dev *crg_udc, int index,
+ unsigned long *flags);
+/* This function will go through the list of the USB requests for the
+ * given endpoint and schedule any unscheduled trb's to the xfer ring
+ */
+int crg_udc_queue_pending_trbs(struct crg_udc_ep *udc_ep_ptr);
+/* num_trbs here is the size of the ring. */
+u32 crg_udc_room_on_ring(struct crg_gadget_dev *crg_udc, u32 num_trbs,
+ struct crg_transfer_trb *p_ring,
+ struct crg_transfer_trb *enq_pt,
+ struct crg_transfer_trb *dq_pt);
+void crg_udc_setup_datastage_trb(struct crg_transfer_trb *p_trb, dma_addr_t dma,
+ u8 pcs, u32 transfer_length, u32 td_size,
+ bool ioc, bool chain, bool azp, u8 dir, u8 setup_tag);
+void crg_udc_setup_link_trb(struct crg_transfer_trb *link_trb, bool toggle,
+ dma_addr_t next_trb);
+void crg_udc_setup_status_trb(struct crg_gadget_dev *crg_udc,
+ struct crg_transfer_trb *p_trb,
+ struct usb_request *usb_req, u8 pcs, u8 set_addr,
+ u8 stall);
+void crg_udc_setup_trb(struct crg_transfer_trb *p_trb, u32 xfer_len,
+ dma_addr_t xfer_buf_addr, u8 td_size, u8 pcs,
+ u8 trb_type, bool short_pkt, bool chain_bit,
+ bool intr_on_compl, bool b_setup_stage, u8 usb_dir,
+ bool b_isoc, u16 frame_id, bool sia, bool azp);
+void crg_udc_squeeze_xfer_ring(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr);
+struct crg_transfer_trb *crg_udc_tran_trb_dma_to_virt(struct crg_udc_ep *udc_ep,
+ dma_addr_t address);
+dma_addr_t crg_udc_tran_trb_virt_to_dma(struct crg_udc_ep *udc_ep,
+ struct crg_transfer_trb *trb);
+void crg_udc_update_dequeue_pt(struct crg_event_trb *event,
+ struct crg_udc_ep *udc_ep);
+u32 crg_udc_trb_chunk_len(dma_addr_t dma, u64 remaining);
+u32 crg_udc_actual_before(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req,
+ struct crg_transfer_trb *pause);
+/*
+ * command related ops
+ */
+int crg_udc_issue_command(struct crg_gadget_dev *crg_udc,
+ enum crg_cmd_type type, u32 param0, u32 param1);
+int crg_udc_stop_gadget(struct crg_gadget_dev *crg_udc);
+int crg_udc_restart_gadget(struct crg_gadget_dev *crg_udc,
+ unsigned long *flags);
+void crg_udc_release_quarantine(struct crg_gadget_dev *crg_udc);
+int crg_udc_program_speed(struct crg_gadget_dev *crg_udc);
+int crg_udc_reset_data_struct(struct crg_gadget_dev *crg_udc);
+int crg_udc_recover(struct crg_gadget_dev *crg_udc,
+ unsigned long *flags);
+int crg_udc_recover_link(struct crg_gadget_dev *crg_udc, unsigned long *flags,
+ bool notify_reset);
+int crg_udc_recover_endpoints(struct crg_gadget_dev *crg_udc,
+ unsigned long *flags);
+struct usb_request *crg_udc_alloc_request(struct usb_ep *_ep, gfp_t gfp_flags);
+int crg_udc_build_td(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr);
+void crg_udc_clear_portpm(struct crg_gadget_dev *crg_udc);
+void crg_udc_ep_nuke(struct crg_udc_ep *udc_ep, int status);
+void crg_udc_free_request(struct usb_ep *_ep, struct usb_request *_req);
+void crg_udc_set_pullup(struct crg_gadget_dev *crg_udc, bool is_on);
+void crg_udc_reset_work(struct work_struct *work);
+void crg_udc_port_retry_work(struct work_struct *work);
+void crg_udc_set_vbus_detect(struct crg_gadget_dev *crg_udc, bool enable);
+int crg_udc_bounce_map_request(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req,
+ enum dma_data_direction dir, gfp_t gfp);
+/* Unmap and free the private bounce buffer after hardware DMA stops. */
+void crg_udc_unmap_request(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req, size_t received);
+int crg_udc_handle_port_status(struct crg_gadget_dev *crg_udc,
+ unsigned long *flags);
+int crg_udc_handle_xfer_event(struct crg_gadget_dev *crg_udc,
+ struct crg_event_trb *event, unsigned long *flags);
+void crg_udc_dispatch_setup(struct crg_gadget_dev *crg_udc,
+ struct usb_ctrlrequest *setup, u8 setup_tag,
+ unsigned long *flags);
+int crg_udc_handle_event(struct crg_gadget_dev *crg_udc,
+ struct crg_event_trb *event, unsigned long *flags);
+/*
+ * Invoke the bound gadget driver's ->disconnect() callback, if any.
+ *
+ * The gadget disconnect callback may be called in interrupt context.
+ * Release only udc_lock while keeping local IRQs disabled.
+ */
+void crg_udc_notify_disconnect(struct crg_gadget_dev *crg_udc,
+ unsigned long *flags);
+int crg_udc_ep_halt(struct crg_udc_ep *udc_ep_ptr, int halt, int ignore_wedge,
+ unsigned long *flags);
+enum crg_ep_state crg_udc_get_ep_state(struct crg_gadget_dev *crg_udc, int dci);
+u32 crg_udc_index2dci(u16 index);
+int crg_udc_init_device_context(struct crg_gadget_dev *crg_udc);
+void crg_udc_init_ep_info(struct crg_gadget_dev *crg_udc);
+void crg_udc_knock_doorbell(struct crg_gadget_dev *crg_udc, int dci);
+/*
+ * Complete a request and give it back to the gadget layer via the
+ * standard usb_gadget_giveback_request() helper.
+ *
+ * Must be called with udc_lock held and interrupts disabled.
+ *
+ * The request's ->complete() callback must not sleep. crg_udc_req_done() drops
+ * udc_lock before the giveback and re-acquires it before returning.
+ * Callers therefore cannot assume state protected by udc_lock is
+ * unchanged across this call.
+ */
+void crg_udc_req_done(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req, int status);
+void crg_udc_clear_req_container(struct crg_udc_request *udc_req_ptr);
+
+#define CRG_QUIRK_DIS_U1U2_ENTRY BIT(0)
+#define CRG_QUIRK_DIS_U1U2_INITIATION BIT(1)
+#define CRG_QUIRK_REINIT_EP0_ON_RESET BIT(2)
+
+#define CRG_UDC_QUIRKS_DEFAULT \
+ (CRG_QUIRK_DIS_U1U2_ENTRY | CRG_QUIRK_DIS_U1U2_INITIATION | \
+ CRG_QUIRK_REINIT_EP0_ON_RESET)
+
+struct crg_udc_priv {
+ u32 quirks;
+};
+
+extern const struct usb_gadget_ops crg_gadget_ops;
+extern const struct usb_ep_ops crg_udc_ep_ops;
+#define gadget_to_udc(g) (container_of(g, struct crg_gadget_dev, gadget))
+
+#endif /* CRG_UDC_H */
--
2.34.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* [PATCH v5 3/6] usb: gadget: udc: crg: Add ring and transfer management
2026-11-16 19:06 [PATCH v5 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
2026-11-16 19:06 ` [PATCH v5 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC Petar Stepanovic
2026-11-16 19:06 ` [PATCH v5 2/6] usb: gadget: udc: crg: Add hardware definitions and structs Petar Stepanovic
@ 2026-11-16 19:06 ` Petar Stepanovic
2026-11-16 19:46 ` sashiko-bot
2026-11-16 19:06 ` [PATCH v5 4/6] usb: gadget: udc: crg: Add endpoint 0 handling Petar Stepanovic
` (2 subsequent siblings)
5 siblings, 1 reply; 13+ messages in thread
From: Petar Stepanovic @ 2026-11-16 19:06 UTC (permalink / raw)
To: Prasad Bolisetty, Greg Kroah-Hartman, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Tzu-Hao Wei,
Karthikeyan Mitran, Philipp Zabel, Neill Kapron
Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
Petar Stepanovic, Krutik Shah, Naveen Kumar Rajgiri Bassappa
This patch introduces the memory management and ring buffer logic
(TRBs and event rings) required by the Axiado AX3000/AX3005 UDC.
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Tzu-Hao Wei <twei@axiado.com>
Signed-off-by: Tzu-Hao Wei <twei@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
drivers/usb/gadget/udc/crg/crg-udc-ring.c | 696 ++++++++++++++++++++++++++++++
1 file changed, 696 insertions(+)
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-ring.c b/drivers/usb/gadget/udc/crg/crg-udc-ring.c
new file mode 100644
index 000000000000..348d2b611a33
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-ring.c
@@ -0,0 +1,696 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/sched.h>
+#include <linux/scatterlist.h>
+
+#include "crg-udc.h"
+
+void crg_udc_advance_dequeue_pt(struct crg_udc_ep *udc_ep)
+{
+ struct crg_udc_request *udc_req;
+
+ /* The completed request remains at the queue head until giveback. */
+ if (!list_empty(&udc_ep->queue)) {
+ udc_req = list_first_entry(&udc_ep->queue,
+ struct crg_udc_request, queue);
+ if (!list_is_last(&udc_req->queue, &udc_ep->queue)) {
+ udc_req = list_next_entry(udc_req, queue);
+ if (udc_req->first_trb) {
+ udc_ep->deq_pt = udc_req->first_trb;
+ return;
+ }
+ }
+ }
+
+ udc_ep->deq_pt = udc_ep->enq_pt;
+}
+
+unsigned int crg_udc_count_trbs_linear(dma_addr_t addr, u64 len)
+{
+ unsigned int num_trbs = 0;
+
+ while (len) {
+ u32 chunk = crg_udc_trb_chunk_len(addr, len);
+
+ if (!chunk)
+ return 0;
+ num_trbs++;
+ addr += chunk;
+ len -= chunk;
+ }
+
+ return num_trbs;
+}
+
+unsigned int crg_udc_count_trbs_sg(struct crg_udc_request *udc_req)
+{
+ struct scatterlist *sg;
+ u64 remaining = udc_req->usb_req.length;
+ unsigned int i;
+ unsigned int num_trbs = 0;
+
+ for_each_sgtable_dma_sg(&udc_req->bounce->sgt, sg, i) {
+ dma_addr_t addr;
+ u64 len;
+
+ if (!sg_dma_len(sg))
+ continue;
+ if (!remaining)
+ break;
+
+ len = min_t(u64, sg_dma_len(sg), remaining);
+ addr = sg_dma_address(sg);
+ while (len) {
+ u32 chunk = crg_udc_trb_chunk_len(addr, len);
+
+ if (!chunk)
+ return 0;
+ num_trbs++;
+ addr += chunk;
+ len -= chunk;
+ remaining -= chunk;
+ }
+ }
+
+ return remaining ? 0 : num_trbs;
+}
+
+static dma_addr_t event_trb_virt_to_dma(struct crg_udc_event *udc_event,
+ struct crg_event_trb *event)
+{
+ dma_addr_t dma_addr = 0;
+ unsigned long seg_offset;
+
+ if (!udc_event || !event)
+ return DMA_MAPPING_ERROR;
+
+ if ((void *)event < udc_event->event_ring.cpu_addr ||
+ (void *)event >=
+ udc_event->event_ring.cpu_addr + udc_event->event_ring.len)
+ return DMA_MAPPING_ERROR;
+
+ seg_offset = (void *)event - udc_event->event_ring.cpu_addr;
+ dma_addr = udc_event->event_ring.dma_addr + seg_offset;
+
+ return dma_addr;
+}
+
+int crg_udc_init_event_ring(struct crg_gadget_dev *crg_udc, int index)
+{
+ struct crg_uicr __iomem *uicr = crg_udc->uicr[index];
+ struct crg_udc_event *udc_event = &crg_udc->udc_event[index];
+
+ int ret;
+ u32 buff_length;
+ dma_addr_t mapping;
+ bool erst_allocated = false;
+
+ buff_length = CRG_ERST_SIZE * sizeof(struct crg_erst);
+ if (!udc_event->erst.cpu_addr) {
+ udc_event->erst.cpu_addr = dma_alloc_coherent(
+ crg_udc->dev, buff_length, &mapping, GFP_ATOMIC);
+
+ if (!udc_event->erst.cpu_addr)
+ return -ENOMEM;
+ erst_allocated = true;
+ } else {
+ mapping = udc_event->erst.dma_addr;
+ }
+
+ udc_event->erst.len = buff_length;
+ udc_event->erst.dma_addr = mapping;
+ udc_event->p_erst = udc_event->erst.cpu_addr;
+
+ buff_length = CRG_EVENT_RING_SIZE * sizeof(struct crg_event_trb);
+ if (!udc_event->event_ring.cpu_addr) {
+ udc_event->event_ring.cpu_addr = dma_alloc_coherent(
+ crg_udc->dev, buff_length, &mapping, GFP_ATOMIC);
+
+ if (!udc_event->event_ring.cpu_addr) {
+ ret = -ENOMEM;
+ goto err_free_erst;
+ }
+ } else {
+ mapping = udc_event->event_ring.dma_addr;
+ }
+
+ udc_event->event_ring.len = buff_length;
+ udc_event->event_ring.dma_addr = mapping;
+ /* Invalidate event handlers which dropped udc_lock during reset. */
+ udc_event->generation++;
+ udc_event->evt_dq_pt = udc_event->event_ring.cpu_addr;
+
+ udc_event->evt_seg0_last_trb =
+ (struct crg_event_trb *)(udc_event->event_ring.cpu_addr) +
+ (CRG_EVENT_RING_SIZE - 1);
+
+ udc_event->ccs = 1;
+
+ udc_event->p_erst->seg_addr_lo =
+ cpu_to_le32(lower_32_bits(udc_event->event_ring.dma_addr));
+ udc_event->p_erst->seg_addr_hi =
+ cpu_to_le32(upper_32_bits(udc_event->event_ring.dma_addr));
+ udc_event->p_erst->seg_size = cpu_to_le32(CRG_EVENT_RING_SIZE);
+ udc_event->p_erst->rsvd = cpu_to_le32(0);
+ /* Ensure that event ring is updated */
+ dma_wmb();
+ /* clear the event ring, to avoid hw unexpected ops
+ * because of dirty data
+ */
+ memset(udc_event->event_ring.cpu_addr, 0, buff_length);
+
+ /* Ensure that event ring is clear before programming
+ * ERSTBA/ERSTSZ/ERDP
+ */
+ dma_wmb();
+
+ crg_udc->hw_dma_armed = true;
+ writel(CRG_ERST_SIZE, &uicr->erstsz);
+ writel(lower_32_bits(udc_event->erst.dma_addr), &uicr->erstbalo);
+ writel(upper_32_bits(udc_event->erst.dma_addr), &uicr->erstbahi);
+ writel(lower_32_bits(udc_event->event_ring.dma_addr) |
+ CRG_U3DC_ERDPLO_EHB,
+ &uicr->erdplo);
+ writel(upper_32_bits(udc_event->event_ring.dma_addr), &uicr->erdphi);
+
+ writel((CRG_U3DC_IMAN_INT_EN | CRG_U3DC_IMAN_INT_PEND), &uicr->iman);
+ /* Interrupt moderation: 4000 * 250ns = 1ms */
+ writel(4000, &uicr->imod);
+ return 0;
+
+err_free_erst:
+ /* Only free what this call allocated; a pre-existing erst is owned
+ * by an earlier successful init and released by crg_udc_free_resources().
+ */
+ if (erst_allocated) {
+ dma_free_coherent(crg_udc->dev, udc_event->erst.len,
+ udc_event->erst.cpu_addr,
+ udc_event->erst.dma_addr);
+ udc_event->erst.cpu_addr = NULL;
+ }
+ return ret;
+}
+
+static bool is_event_ring_x_empty(struct crg_gadget_dev *crg_udc, int index)
+{
+ struct crg_event_trb *event;
+ struct crg_udc_event *udc_event;
+
+ udc_event = &crg_udc->udc_event[index];
+ if (udc_event->evt_dq_pt) {
+ event = (struct crg_event_trb *)udc_event->evt_dq_pt;
+
+ if (FIELD_GET(EVE_TRB_CYCLE_BIT_MASK,
+ le32_to_cpu(event->dw3)) != udc_event->ccs)
+ return true;
+ }
+
+ return false;
+}
+
+bool crg_udc_is_event_rings_empty(struct crg_gadget_dev *crg_udc)
+{
+ int i;
+
+ for (i = 0; i < CRG_RING_NUM; i++) {
+ if (!is_event_ring_x_empty(crg_udc, i)) {
+ dev_err(crg_udc->dev, "event ring not empty\n");
+ return false;
+ }
+ }
+ return true;
+}
+
+bool crg_udc_is_pointer_less_than(struct crg_transfer_trb *a,
+ struct crg_transfer_trb *b,
+ struct crg_udc_ep *udc_ep)
+{
+ if (!a || !b || !udc_ep->first_trb || !udc_ep->last_trb)
+ return false;
+ if (b > a && (udc_ep->enq_pt >= b || udc_ep->enq_pt < a))
+ return true;
+ if (b < a && (udc_ep->enq_pt >= b && udc_ep->enq_pt < a))
+ return true;
+ return false;
+}
+
+int crg_udc_process_event_ring(struct crg_gadget_dev *crg_udc, int index,
+ unsigned long *flags)
+{
+ struct crg_event_trb *event;
+ struct crg_udc_event *udc_event;
+ struct crg_uicr __iomem *uicr = crg_udc->uicr[index];
+ u32 val;
+ dma_addr_t erdp;
+ int ret;
+ u32 generation;
+ unsigned int processed = 0;
+
+ if (!uicr)
+ return -ENODEV;
+
+ /* Do not touch the interrupter while the controller is resetting. */
+ if (unlikely(crg_udc->resetting))
+ return 0;
+
+ val = readl(&uicr->iman);
+ val |= CRG_U3DC_IMAN_INT_PEND;
+ writel(val, &uicr->iman);
+
+ udc_event = &crg_udc->udc_event[index];
+ generation = udc_event->generation;
+ while (udc_event->evt_dq_pt) {
+ event = (struct crg_event_trb *)udc_event->evt_dq_pt;
+
+ if (FIELD_GET(EVE_TRB_CYCLE_BIT_MASK,
+ le32_to_cpu(event->dw3)) != udc_event->ccs)
+ break;
+ /* Read the payload only after observing device ownership. */
+ dma_rmb();
+
+ ret = crg_udc_handle_event(crg_udc, event, flags);
+ /* The giveback callback may release the lock and reset the ring. */
+ if (unlikely(generation != udc_event->generation ||
+ crg_udc->resetting))
+ return 0;
+ if (ret == -ECONNRESET) {
+ /* The disconnect path rebuilt and reprogrammed the ring. */
+ return 0;
+ }
+ if (ret < 0)
+ dev_dbg(crg_udc->dev, "failed to process event: %d\n",
+ ret);
+
+ /* Consume malformed events so they cannot pin the ring head. */
+ if (event == udc_event->evt_seg0_last_trb) {
+ udc_event->ccs = udc_event->ccs ? 0 : 1;
+ udc_event->evt_dq_pt = udc_event->event_ring.cpu_addr;
+ } else {
+ udc_event->evt_dq_pt++;
+ }
+
+ /*
+ * udc_event state is only re-read from crg_udc, never cached
+ * across this drop, so releasing and reacquiring the lock
+ * here is safe and lets other udc_lock users, and this CPU's
+ * other interrupts, make progress during a busy event ring.
+ */
+ if (++processed % CRG_EVENT_RING_BATCH == 0) {
+ erdp = event_trb_virt_to_dma(udc_event,
+ udc_event->evt_dq_pt);
+ if (!dma_mapping_error(crg_udc->dev, erdp)) {
+ writel(upper_32_bits(erdp), &uicr->erdphi);
+ writel(lower_32_bits(erdp) |
+ CRG_U3DC_ERDPLO_EHB,
+ &uicr->erdplo);
+ }
+ spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+ cond_resched();
+ spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+ if (unlikely(generation != udc_event->generation ||
+ crg_udc->resetting))
+ return 0;
+ }
+ }
+
+ /* update dequeue pointer */
+ erdp = event_trb_virt_to_dma(udc_event, udc_event->evt_dq_pt);
+ if (dma_mapping_error(crg_udc->dev, erdp))
+ return -EINVAL;
+ writel(upper_32_bits(erdp), &uicr->erdphi);
+ writel(lower_32_bits(erdp) | CRG_U3DC_ERDPLO_EHB, &uicr->erdplo);
+
+ return 0;
+}
+
+/* This function will go through the list of the USB requests for the
+ * given endpoint and schedule any unscheduled trb's to the xfer ring
+ */
+int crg_udc_queue_pending_trbs(struct crg_udc_ep *udc_ep_ptr)
+{
+ struct crg_udc_request *req;
+ struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+ u32 generation = crg_udc->udc_event[0].generation;
+ u32 ep_generation = udc_ep_ptr->generation;
+ int ret;
+
+restart:
+ /* A giveback can reset/disable the endpoint while udc_lock is dropped. */
+ if (crg_udc->resetting || crg_udc->port_reset_pending ||
+ crg_udc->endpoint_recovery_pending ||
+ !udc_ep_ptr->desc || udc_ep_ptr->halt_failed || udc_ep_ptr->nuking ||
+ udc_ep_ptr->ep_state != EP_STATE_RUNNING)
+ return 0;
+
+ list_for_each_entry(req, &udc_ep_ptr->queue, queue) {
+ if (req->all_trbs_queued)
+ continue;
+
+ /* A large request must finish its current TD before the next
+ * TD (or the next USB request) can be submitted: the event
+ * handler uses all_trbs_queued to complete the USB request.
+ */
+ if (req->td_pending)
+ return 0;
+
+ ret = crg_udc_build_td(udc_ep_ptr, req);
+ if (ret == -ENOSPC || ret == -EBUSY)
+ return ret;
+ if (ret) {
+ dev_err(crg_udc->dev,
+ "failed to build pending transfer: %d\n", ret);
+ /* Giveback removes req and drops udc_lock: callbacks can
+ * remove or free every saved request-list iterator.
+ */
+ crg_udc_req_done(udc_ep_ptr, req, ret);
+ if (generation != crg_udc->udc_event[0].generation ||
+ ep_generation != udc_ep_ptr->generation)
+ return 0;
+ goto restart;
+ }
+
+ if (udc_ep_ptr->tran_ring_full || !req->all_trbs_queued)
+ return 0;
+ }
+ return 0;
+}
+
+/* num_trbs here is the size of the ring. */
+u32 crg_udc_room_on_ring(struct crg_gadget_dev *crg_udc, u32 num_trbs,
+ struct crg_transfer_trb *p_ring,
+ struct crg_transfer_trb *enq_pt,
+ struct crg_transfer_trb *dq_pt)
+{
+ u32 capacity, enq_idx, dq_idx;
+
+ /* One slot is the Link TRB and one data slot remains unused. */
+ if (num_trbs < 3 || !p_ring || !enq_pt || !dq_pt)
+ return 0;
+
+ capacity = num_trbs - 1;
+ /* Never dereference a pointer derived from an invalid/stale event. */
+ if ((unsigned long)enq_pt < (unsigned long)p_ring ||
+ (unsigned long)dq_pt < (unsigned long)p_ring ||
+ (unsigned long)enq_pt >= (unsigned long)(p_ring + capacity) ||
+ (unsigned long)dq_pt >= (unsigned long)(p_ring + capacity)) {
+ WARN_ON_ONCE(1);
+ return 0;
+ }
+
+ enq_idx = enq_pt - p_ring;
+ dq_idx = dq_pt - p_ring;
+ if (enq_idx == dq_idx)
+ return capacity - 1;
+
+ /* Data slots strictly between enqueue and dequeue, modulo Link. */
+ return (dq_idx + capacity - enq_idx) % capacity - 1;
+}
+
+void crg_udc_setup_datastage_trb(struct crg_transfer_trb *p_trb, dma_addr_t dma,
+ u8 pcs, u32 transfer_length, u32 td_size,
+ bool ioc, bool chain, bool azp, u8 dir, u8 setup_tag)
+{
+ u32 tmp;
+
+ p_trb->dw0 = cpu_to_le32(lower_32_bits(dma));
+ p_trb->dw1 = cpu_to_le32(upper_32_bits(dma));
+
+ /* TRB_Transfer_Length
+ * For USB_DIR_OUT, this field is the number of data bytes expected
+ * from xhc. For USB_DIR_IN, this field is the number of data bytes
+ * the device will send.
+ */
+ tmp = 0;
+ FIELD_MODIFY(TRB_TRANSFER_LEN_MASK, &tmp, transfer_length);
+ FIELD_MODIFY(TRB_TD_SIZE_MASK, &tmp, td_size);
+ FIELD_MODIFY(TRB_INTR_TARGET_MASK, &tmp, 0);
+ p_trb->dw2 = cpu_to_le32(tmp);
+
+ tmp = 0;
+ FIELD_MODIFY(TRB_CYCLE_BIT_MASK, &tmp, pcs);
+ FIELD_MODIFY(TRB_INTR_ON_SHORT_PKT_MASK, &tmp, 1);
+ FIELD_MODIFY(TRB_INTR_ON_COMPLETION_MASK, &tmp, ioc);
+ FIELD_MODIFY(TRB_CHAIN_BIT_MASK, &tmp, chain);
+ FIELD_MODIFY(TRB_TYPE_MASK, &tmp, TRB_TYPE_XFER_DATA_STAGE);
+ FIELD_MODIFY(TRB_APPEND_ZLP_MASK, &tmp, azp);
+ FIELD_MODIFY(DATA_STAGE_TRB_DIR_MASK, &tmp, dir);
+ FIELD_MODIFY(TRB_SETUP_TAG_MASK, &tmp, setup_tag);
+
+ /* Publish ownership only after the TRB payload is visible. */
+ dma_wmb();
+ p_trb->dw3 = cpu_to_le32(tmp);
+}
+
+void crg_udc_setup_link_trb(struct crg_transfer_trb *link_trb, bool toggle,
+ dma_addr_t next_trb)
+{
+ u32 dw = 0;
+
+ link_trb->dw0 = cpu_to_le32(lower_32_bits(next_trb));
+ link_trb->dw1 = cpu_to_le32(upper_32_bits(next_trb));
+ link_trb->dw2 = cpu_to_le32(0);
+
+ FIELD_MODIFY(TRB_TYPE_MASK, &dw, TRB_TYPE_LINK);
+ if (toggle)
+ FIELD_MODIFY(TRB_LINK_TOGGLE_CYCLE_MASK, &dw, 1);
+ else
+ FIELD_MODIFY(TRB_LINK_TOGGLE_CYCLE_MASK, &dw, 0);
+
+ /* Publish ownership only after the link address is visible. */
+ dma_wmb();
+ link_trb->dw3 = cpu_to_le32(dw);
+}
+
+void crg_udc_setup_status_trb(struct crg_gadget_dev *crg_udc,
+ struct crg_transfer_trb *p_trb,
+ struct usb_request *usb_req, u8 pcs, u8 set_addr,
+ u8 stall)
+{
+ u32 tmp, dir = 0;
+
+ /* There are some cases where crg_udc_setup_status_trb() is called with
+ * usb_req set to NULL.
+ */
+ p_trb->dw0 = cpu_to_le32(0);
+ p_trb->dw1 = cpu_to_le32(0);
+
+ tmp = 0;
+ FIELD_MODIFY(TRB_INTR_TARGET_MASK, &tmp, 0);
+ p_trb->dw2 = cpu_to_le32(tmp);
+
+ tmp = 0;
+ FIELD_MODIFY(TRB_CYCLE_BIT_MASK, &tmp, pcs);
+ FIELD_MODIFY(TRB_INTR_ON_COMPLETION_MASK, &tmp, 1);
+ FIELD_MODIFY(TRB_TYPE_MASK, &tmp, TRB_TYPE_XFER_STATUS_STAGE);
+
+ dir = (crg_udc->setup_status == STATUS_STAGE_XFER) ? 0 : 1;
+ FIELD_MODIFY(DATA_STAGE_TRB_DIR_MASK, &tmp, dir);
+
+ FIELD_MODIFY(TRB_SETUP_TAG_MASK, &tmp, crg_udc->setup_tag);
+ FIELD_MODIFY(STATUS_STAGE_TRB_STALL_MASK, &tmp, stall);
+ FIELD_MODIFY(STATUS_STAGE_TRB_SET_ADDR_MASK, &tmp, set_addr);
+
+ /* Publish ownership only after the TRB payload is visible. */
+ dma_wmb();
+ p_trb->dw3 = cpu_to_le32(tmp);
+}
+
+void crg_udc_setup_trb(struct crg_transfer_trb *p_trb, u32 xfer_len,
+ dma_addr_t xfer_buf_addr, u8 td_size, u8 pcs,
+ u8 trb_type, bool short_pkt, bool chain_bit,
+ bool intr_on_compl, bool b_setup_stage, u8 usb_dir,
+ bool b_isoc, u16 frame_id, bool sia, bool azp)
+{
+ u32 tmp;
+
+ p_trb->dw0 = cpu_to_le32(lower_32_bits(xfer_buf_addr));
+ p_trb->dw1 = cpu_to_le32(upper_32_bits(xfer_buf_addr));
+
+ tmp = 0;
+ FIELD_MODIFY(TRB_TRANSFER_LEN_MASK, &tmp, xfer_len);
+ FIELD_MODIFY(TRB_TD_SIZE_MASK, &tmp, td_size);
+ FIELD_MODIFY(TRB_INTR_TARGET_MASK, &tmp, 0);
+
+ p_trb->dw2 = cpu_to_le32(tmp);
+
+ tmp = 0;
+ FIELD_MODIFY(TRB_CYCLE_BIT_MASK, &tmp, pcs);
+ FIELD_MODIFY(TRB_INTR_ON_SHORT_PKT_MASK, &tmp, short_pkt);
+ FIELD_MODIFY(TRB_CHAIN_BIT_MASK, &tmp, chain_bit);
+ FIELD_MODIFY(TRB_INTR_ON_COMPLETION_MASK, &tmp, intr_on_compl);
+ FIELD_MODIFY(TRB_APPEND_ZLP_MASK, &tmp, azp);
+ FIELD_MODIFY(TRB_TYPE_MASK, &tmp, trb_type);
+
+ if (b_setup_stage)
+ FIELD_MODIFY(DATA_STAGE_TRB_DIR_MASK, &tmp, usb_dir);
+
+ if (b_isoc) {
+ FIELD_MODIFY(ISOC_TRB_FRAME_ID_MASK, &tmp, frame_id);
+ FIELD_MODIFY(ISOC_TRB_SIA_MASK, &tmp, sia);
+ }
+
+ /* Publish ownership only after the TRB payload is visible. */
+ dma_wmb();
+ p_trb->dw3 = cpu_to_le32(tmp);
+}
+
+void crg_udc_squeeze_xfer_ring(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr)
+{
+ struct crg_transfer_trb *trb = udc_req_ptr->first_trb;
+ struct crg_udc_request *next_req;
+ u8 new_pcs, clear_pcs;
+
+ if (!trb)
+ return;
+
+ /* Rewind the producer to the cancelled request. These TRBs and
+ * everything queued after them will be discarded and rebuilt.
+ */
+ new_pcs = FIELD_GET(TRB_CYCLE_BIT_MASK, le32_to_cpu(trb->dw3));
+ clear_pcs = new_pcs;
+ while (trb != udc_ep_ptr->enq_pt) {
+ /* Zero is not always invalid: after a wrap the consumer
+ * expects cycle 0. Invalidate each slot for its cycle pass.
+ */
+ le32p_replace_bits(&trb->dw3, clear_pcs ^ 1,
+ TRB_CYCLE_BIT_MASK);
+ trb++;
+ if (trb == udc_ep_ptr->last_trb) {
+ trb = udc_ep_ptr->first_trb;
+ clear_pcs ^= 1;
+ }
+ }
+
+ udc_ep_ptr->enq_pt = udc_req_ptr->first_trb;
+ udc_ep_ptr->pcs = new_pcs;
+ udc_ep_ptr->tran_ring_full = false;
+ /* Do not overwrite the Link TRB: a previous active TD may still
+ * require its cycle/chain bits to traverse the ring.
+ */
+ dma_wmb();
+
+ /* Invalidate the bookkeeping of ALL subsequent requests before
+ * attempting to rebuild any: rebuilding may stop when space runs out.
+ * The caller gives back the cancelled request before rebuilding.
+ */
+ next_req = udc_req_ptr;
+ list_for_each_entry_continue(next_req, &udc_ep_ptr->queue, queue) {
+ next_req->usb_req.status = -EINPROGRESS;
+ next_req->usb_req.actual = 0;
+ crg_udc_clear_req_container(next_req);
+ }
+}
+
+struct crg_transfer_trb *crg_udc_tran_trb_dma_to_virt(struct crg_udc_ep *udc_ep,
+ dma_addr_t address)
+{
+ dma_addr_t offset;
+ struct crg_transfer_trb *trb_virt;
+
+ if (lower_32_bits(address) & 0xf)
+ return NULL;
+
+ offset = address - udc_ep->tran_ring_info.dma_addr;
+ if (unlikely(offset >= udc_ep->tran_ring_info.len))
+ return NULL;
+ offset = offset / sizeof(struct crg_transfer_trb);
+ trb_virt = udc_ep->first_trb + offset;
+ if (unlikely(trb_virt > udc_ep->last_trb))
+ return NULL;
+ return trb_virt;
+}
+
+dma_addr_t crg_udc_tran_trb_virt_to_dma(struct crg_udc_ep *udc_ep,
+ struct crg_transfer_trb *trb)
+{
+ unsigned long offset;
+ int trb_idx;
+ dma_addr_t dma_addr = 0;
+
+ trb_idx = trb - udc_ep->first_trb;
+ if (unlikely(trb_idx < 0 || trb > udc_ep->last_trb))
+ return DMA_MAPPING_ERROR;
+
+ offset = trb_idx * sizeof(*trb);
+ if (unlikely(offset >= udc_ep->tran_ring_info.len))
+ return DMA_MAPPING_ERROR;
+ dma_addr = udc_ep->tran_ring_info.dma_addr + offset;
+ return dma_addr;
+}
+
+void crg_udc_update_dequeue_pt(struct crg_event_trb *event,
+ struct crg_udc_ep *udc_ep)
+{
+ u32 deq_pt_lo = le32_to_cpu(event->dw0);
+ u32 deq_pt_hi = le32_to_cpu(event->dw1);
+ u64 dq_pt_addr = (u64)deq_pt_lo | ((u64)deq_pt_hi << 32);
+ struct crg_transfer_trb *deq_pt;
+
+ deq_pt = crg_udc_tran_trb_dma_to_virt(udc_ep, dq_pt_addr);
+ if (!deq_pt)
+ return;
+
+ /*
+ * The reported dequeue pointer can be the Link TRB itself, so wrap
+ * before advancing: last_trb is the final entry of the ring buffer
+ * and last_trb + 1 is outside the coherent allocation.
+ */
+ if (deq_pt == udc_ep->last_trb)
+ deq_pt = udc_ep->first_trb;
+ else
+ deq_pt++;
+
+ if (deq_pt == udc_ep->last_trb ||
+ FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(deq_pt->dw3)) == TRB_TYPE_LINK)
+ deq_pt = udc_ep->first_trb;
+
+ udc_ep->deq_pt = deq_pt;
+}
+
+u32 crg_udc_trb_chunk_len(dma_addr_t dma, u64 remaining)
+{
+ u32 boundary = CRG_UDC_TRB_MAX_BUFFER_SIZE -
+ (lower_32_bits(dma) & (CRG_UDC_TRB_MAX_BUFFER_SIZE - 1));
+
+ return min_t(u64, remaining, boundary);
+}
+
+u32 crg_udc_actual_before(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req,
+ struct crg_transfer_trb *pause)
+{
+ struct crg_transfer_trb *trb = udc_req->first_trb;
+ u64 actual = udc_req->td_start_offset;
+ u32 guard = 0;
+ u32 max_iter;
+
+ if (!trb || !pause || !udc_ep->last_trb)
+ return 0;
+ max_iter = udc_ep->last_trb - udc_ep->first_trb + 1;
+
+ while (trb != pause && trb != udc_ep->enq_pt && guard++ < max_iter) {
+ u32 len;
+
+ /*
+ * Wrap here rather than after the increment below: pause can
+ * be the Link TRB, and jumping straight from last_trb - 1 to
+ * first_trb would step over it and sum the whole ring.
+ */
+ if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(trb->dw3)) ==
+ TRB_TYPE_LINK) {
+ trb = udc_ep->first_trb;
+ continue;
+ }
+ len = FIELD_GET(TRB_TRANSFER_LEN_MASK, le32_to_cpu(trb->dw2));
+ actual += len;
+ trb++;
+ }
+
+ return min_t(u64, actual, udc_req->usb_req.length);
+}
--
2.34.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH v5 3/6] usb: gadget: udc: crg: Add ring and transfer management
2026-11-16 19:06 ` [PATCH v5 3/6] usb: gadget: udc: crg: Add ring and transfer management Petar Stepanovic
@ 2026-11-16 19:46 ` sashiko-bot
0 siblings, 0 replies; 13+ messages in thread
From: sashiko-bot @ 2026-11-16 19:46 UTC (permalink / raw)
To: Petar Stepanovic; +Cc: linux-usb, devicetree, robh, conor+dt
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] usb: gadget: udc: crg: missing session check in
crg_udc_queue_pending_trbs()
--
commit 4959c938e3393a398d380564ec6cf72791b96459
Author: Petar Stepanovic <pstepanovic@axiado.com>
usb: gadget: udc: crg: Add ring and transfer management
This patch adds event ring initialization, event processing, and transfer
ring TRB management for the Axiado AX3000/AX3005 UDC driver.
> diff --git a/drivers/usb/gadget/udc/crg/crg-udc-ring.c b/drivers/usb/gadget/udc/crg/crg-udc-ring.c
> new file mode 100644
> index 0000000000000..348d2b611a332
> --- /dev/null
> +++ b/drivers/usb/gadget/udc/crg/crg-udc-ring.c
[ ... ]
> +int crg_udc_queue_pending_trbs(struct crg_udc_ep *udc_ep_ptr)
> +{
> + struct crg_udc_request *req;
> + struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
> + u32 generation = crg_udc->udc_event[0].generation;
> + u32 ep_generation = udc_ep_ptr->generation;
> + int ret;
> +
> +restart:
> + /* A giveback can reset/disable the endpoint while udc_lock is dropped. */
> + if (crg_udc->resetting || crg_udc->port_reset_pending ||
> + crg_udc->endpoint_recovery_pending ||
> + !udc_ep_ptr->desc || udc_ep_ptr->halt_failed || udc_ep_ptr->nuking ||
> + udc_ep_ptr->ep_state != EP_STATE_RUNNING)
> + return 0;
> +
> + list_for_each_entry(req, &udc_ep_ptr->queue, queue) {
> + if (req->all_trbs_queued)
> + continue;
> +
> + /* A large request must finish its current TD before the next
> + * TD (or the next USB request) can be submitted: the event
> + * handler uses all_trbs_queued to complete the USB request.
> + */
> + if (req->td_pending)
> + return 0;
> +
> + ret = crg_udc_build_td(udc_ep_ptr, req);
> + if (ret == -ENOSPC || ret == -EBUSY)
> + return ret;
> + if (ret) {
> + dev_err(crg_udc->dev,
> + "failed to build pending transfer: %d\n", ret);
> + /* Giveback removes req and drops udc_lock: callbacks can
> + * remove or free every saved request-list iterator.
> + */
> + crg_udc_req_done(udc_ep_ptr, req, ret);
> + if (generation != crg_udc->udc_event[0].generation ||
> + ep_generation != udc_ep_ptr->generation)
> + return 0;
> + goto restart;
> + }
[Severity: Medium]
Can crg_udc_queue_pending_trbs() publish TRBs and ring the doorbell after a
concurrent suspend or teardown has stopped the controller?
When crg_udc_req_done() drops udc_lock (either inside this loop or in a
caller right before invoking crg_udc_queue_pending_trbs()), another context
can stop the session before controller reset runs:
CPU0 CPU1
crg_udc_handle_xfer_event()
crg_udc_req_done()
spin_unlock(&crg_udc->udc_lock)
crg_udc_suspend()
crg_udc->suspended = true;
crg_udc_set_pullup(..., false)
/* clears CRG_U3DC_CTRL_RUN */
spin_unlock(&crg_udc->udc_lock)
synchronize_irq()
spin_lock(&crg_udc->udc_lock)
crg_udc_queue_pending_trbs()
crg_udc_build_td() <-- publishes TRBs and rings doorbell
During this window, crg_udc->resetting is still false, generations have not
changed, and data endpoints remain in EP_STATE_RUNNING, so the check at
restart passes even though crg_udc_session_ready(crg_udc) is false.
Checking !crg_udc_session_ready(crg_udc) at restart would prevent queueing
TRBs once the session is no longer ready.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261116-axiado-ax3000-usb-device-controller-v5-0-49271ed5c793@axiado.com?part=3
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v5 4/6] usb: gadget: udc: crg: Add endpoint 0 handling
2026-11-16 19:06 [PATCH v5 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
` (2 preceding siblings ...)
2026-11-16 19:06 ` [PATCH v5 3/6] usb: gadget: udc: crg: Add ring and transfer management Petar Stepanovic
@ 2026-11-16 19:06 ` Petar Stepanovic
2026-11-16 20:16 ` sashiko-bot
2026-11-16 19:06 ` [PATCH v5 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations Petar Stepanovic
2026-11-16 19:06 ` [PATCH v5 6/6] usb: gadget: udc: crg: Add core driver and build integration Petar Stepanovic
5 siblings, 1 reply; 13+ messages in thread
From: Petar Stepanovic @ 2026-11-16 19:06 UTC (permalink / raw)
To: Prasad Bolisetty, Greg Kroah-Hartman, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Tzu-Hao Wei,
Karthikeyan Mitran, Philipp Zabel, Neill Kapron
Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
Petar Stepanovic, Krutik Shah, Naveen Kumar Rajgiri Bassappa
This patch adds the endpoint 0 setup packet, status, and control
transfer handling for the Axiado AX3000/AX3005 UDC.
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Tzu-Hao Wei <twei@axiado.com>
Signed-off-by: Tzu-Hao Wei <twei@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
drivers/usb/gadget/udc/crg/crg-udc-ep0.c | 1264 ++++++++++++++++++++++++++++++
1 file changed, 1264 insertions(+)
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-ep0.c b/drivers/usb/gadget/udc/crg/crg-udc-ep0.c
new file mode 100644
index 000000000000..4bd0dd4da8c6
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-ep0.c
@@ -0,0 +1,1264 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/iopoll.h>
+
+#include "crg-udc.h"
+#include "crg-udc-trace.h"
+
+static void enable_setup_event(struct crg_gadget_dev *crg_udc);
+
+int crg_udc_build_ep0_status(struct crg_udc_ep *udc_ep_ptr, bool default_value,
+ u32 status, struct crg_udc_request *udc_req_ptr,
+ u8 set_addr, u8 stall)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+ struct crg_transfer_trb *enq_pt = udc_ep_ptr->enq_pt;
+
+ /* Internal status requests bypass usb_ep_queue(), so they must also
+ * respect a teardown that is giving back the old EP0 queue.
+ */
+ if (udc_ep_ptr->nuking) {
+ crg_udc->ep0_state = EP0_STATE_ERROR;
+ return -EBUSY;
+ }
+
+ /* Reserve a free transfer-ring slot before touching request state or
+ * publishing the STATUS TRB. crg_udc_room_on_ring() treats equal
+ * pointers as empty, so check tran_ring_full separately.
+ */
+ if (!udc_ep_ptr->first_trb || !udc_ep_ptr->last_trb ||
+ !udc_ep_ptr->enq_pt || !udc_ep_ptr->deq_pt)
+ return -ENODEV;
+ if (udc_ep_ptr->ep_state != EP_STATE_RUNNING)
+ return -ESHUTDOWN;
+ if (enq_pt < udc_ep_ptr->first_trb ||
+ enq_pt >= udc_ep_ptr->last_trb ||
+ udc_ep_ptr->deq_pt < udc_ep_ptr->first_trb ||
+ udc_ep_ptr->deq_pt >= udc_ep_ptr->last_trb)
+ return -EIO;
+ if (udc_ep_ptr->tran_ring_full ||
+ !crg_udc_room_on_ring(crg_udc, CRG_UDC_CONTROL_EP_TD_RING_SIZE,
+ udc_ep_ptr->first_trb, enq_pt,
+ udc_ep_ptr->deq_pt))
+ return -ENOSPC;
+
+ if (default_value) {
+ udc_req_ptr = crg_udc->status_req;
+ /* The shared request may still be in its data-stage callback. */
+ if (!udc_req_ptr || crg_udc->status_req_callback_pending ||
+ !list_empty(&udc_req_ptr->queue)) {
+ dev_err(crg_udc->dev, "EP0 status request unavailable\n");
+ crg_udc->ep0_state = EP0_STATE_ERROR;
+ return udc_req_ptr ? -EBUSY : -ENODEV;
+ }
+ udc_req_ptr->usb_req.length = 0;
+ udc_req_ptr->usb_req.status = status;
+ udc_req_ptr->usb_req.actual = 0;
+ udc_req_ptr->usb_req.complete = NULL;
+ } else {
+ if (!udc_req_ptr) {
+ crg_udc->ep0_state = EP0_STATE_ERROR;
+ return -EINVAL;
+ }
+ udc_req_ptr->usb_req.status = status;
+ udc_req_ptr->usb_req.actual = 0;
+ }
+
+ crg_udc_setup_status_trb(crg_udc, enq_pt, &udc_req_ptr->usb_req,
+ udc_ep_ptr->pcs, set_addr, stall);
+
+ /* STATUS requests must be matched against their own TRB too. */
+ udc_req_ptr->first_trb = enq_pt;
+ udc_req_ptr->last_trb = enq_pt;
+
+ enq_pt++;
+
+ /* check if we are at end of trb segment. If so, update
+ * pcs and enq for next segment
+ */
+ if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(enq_pt->dw3)) ==
+ TRB_TYPE_LINK) {
+ if (FIELD_GET(TRB_LINK_TOGGLE_CYCLE_MASK,
+ le32_to_cpu(enq_pt->dw3))) {
+ dma_wmb();
+ le32p_replace_bits(&enq_pt->dw3, udc_ep_ptr->pcs,
+ TRB_CYCLE_BIT_MASK);
+ udc_ep_ptr->pcs ^= 0x1;
+ }
+ enq_pt = udc_ep_ptr->first_trb;
+ }
+ udc_ep_ptr->enq_pt = enq_pt;
+
+ crg_udc_knock_doorbell(crg_udc, 0);
+
+ list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+ return 0;
+}
+
+/* Report status-stage publication errors consistently to all callers. */
+static void crg_udc_submit_ep0_status(struct crg_gadget_dev *crg_udc,
+ int status, u8 set_addr)
+{
+ int ret;
+
+ ret = crg_udc_build_ep0_status(&crg_udc->udc_ep[0], true,
+ status, NULL, set_addr, 0);
+ if (ret) {
+ /* A new SETUP must stop and reset EP0 before reuse. */
+ crg_udc->ep0_state = EP0_STATE_ERROR;
+ crg_udc->setup_fn_call_back = NULL;
+ dev_err(crg_udc->dev,
+ "failed to queue EP0 status stage: %d\n", ret);
+ }
+}
+
+/*
+ * Abort the current EP0 control transfer when a new SETUP packet arrives.
+ *
+ * A SETUP packet is allowed to abort any in-progress control transfer. Do
+ * not defer the new SETUP until the old transfer completes: doing so leaves
+ * the old request on the EP0 software queue and subsequent gadget requests
+ * are rejected as "queue not empty".
+ *
+ * The controller's EP0 ring is reinitialized at the ring start, making all
+ * TRBs belonging to the aborted control transfer stale. The new SETUP is
+ * then handled immediately with a fresh setup tag.
+ */
+int crg_udc_abort_ep0_for_setup(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_udc_ep *udc_ep0 = &crg_udc->udc_ep[0];
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ dma_addr_t dqptaddr;
+ u32 cmd_param0;
+ u32 cmd_param1;
+ u32 generation;
+ u32 tmp;
+ int ret;
+
+ /* A disconnect has stopped RUN. Reconnection rebuilds EP0 first. */
+ if (!crg_udc_session_ready(crg_udc))
+ return -ESHUTDOWN;
+
+ /*
+ * A new SETUP is allowed to abort the current control transfer.
+ * Keep EP0 in ABORTING for the complete stop/nuke/reinitialize
+ * sequence. crg_udc_req_done() drops udc_lock while invoking the gadget
+ * callback, so the callback may re-enter the UDC during the nuke.
+ */
+ udc_ep0->generation++;
+ crg_udc->setup_status = WAIT_FOR_SETUP;
+ crg_udc->ep0_state = EP0_STATE_ABORTING;
+
+ /*
+ * Stop EP0 before touching the transfer ring. The controller may still
+ * own the old TRBs even when no EP0 request is running from the gadget's
+ * point of view. Do not clear the DMA-visible ring until the hardware
+ * reports that EP0 is no longer running.
+ */
+ ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP, BIT(0), 0);
+ if (ret) {
+ dev_err(crg_udc->dev,
+ "failed to stop EP0 for SETUP abort: %d\n", ret);
+ goto recover;
+ }
+
+ ret = readl_poll_timeout_atomic(&uccr->ep_running, tmp, !(tmp & BIT(0)),
+ 10, 10000);
+ if (ret) {
+ dev_err(crg_udc->dev, "timeout stopping EP0 for SETUP abort\n");
+ goto recover;
+ }
+
+ udc_ep0->ep_state = EP_STATE_STOPPED;
+ /* Complete requests belonging to the aborted control transfer. */
+ generation = udc_ep0->generation;
+ crg_udc_ep_nuke(udc_ep0, -ECONNRESET);
+ /*
+ * Giveback drops udc_lock. A disconnect can stop RUN without
+ * changing the endpoint generation, so check the USB session too.
+ */
+ if (!crg_udc_gadget_ready(crg_udc) ||
+ generation != udc_ep0->generation)
+ return -ECONNRESET;
+
+ if (!udc_ep0->first_trb || !udc_ep0->tran_ring_info.cpu_addr) {
+ ret = -ENODEV;
+ goto recover;
+ }
+
+ /*
+ * The controller has stopped consuming EP0 TRBs. It is now safe to
+ * discard the old transfer descriptors and restart the ring from its
+ * beginning. No ring memory is freed here.
+ */
+ memset(udc_ep0->first_trb, 0, udc_ep0->tran_ring_info.len);
+ udc_ep0->enq_pt = udc_ep0->first_trb;
+ udc_ep0->deq_pt = udc_ep0->first_trb;
+ udc_ep0->pcs = 1;
+ udc_ep0->tran_ring_full = false;
+ crg_udc_setup_link_trb(udc_ep0->last_trb, true,
+ udc_ep0->tran_ring_info.dma_addr);
+
+ /* Make the new ring contents visible before handing the ring to HW. */
+ dma_wmb();
+
+ dqptaddr = crg_udc_tran_trb_virt_to_dma(udc_ep0, udc_ep0->first_trb);
+ if (dma_mapping_error(crg_udc->dev, dqptaddr)) {
+ ret = -EIO;
+ goto recover;
+ }
+
+ cmd_param0 = (lower_32_bits(dqptaddr) & CRG_CMD0_0_DQPTRLO_MASK) |
+ FIELD_PREP(CRG_CMD0_0_DCS_MASK, udc_ep0->pcs);
+ cmd_param1 = upper_32_bits(dqptaddr);
+
+ ret = crg_udc_issue_command(crg_udc, CRG_CMD_INIT_EP0, cmd_param0,
+ cmd_param1);
+ if (ret) {
+ dev_err(crg_udc->dev,
+ "failed to reinitialize EP0 for SETUP: %d\n", ret);
+ goto recover;
+ }
+
+ udc_ep0->ep_state = EP_STATE_RUNNING;
+ crg_udc->ep0_state = EP0_STATE_IDLE;
+ return 0;
+
+recover:
+ crg_udc->ep0_state = EP0_STATE_ERROR;
+ crg_udc_schedule_endpoint_recovery(crg_udc);
+ return ret;
+}
+
+int crg_udc_enable_setup(struct crg_gadget_dev *crg_udc)
+{
+ enable_setup_event(crg_udc);
+ crg_udc->device_state = USB_STATE_DEFAULT;
+ usb_gadget_set_state(&crg_udc->gadget, USB_STATE_DEFAULT);
+ crg_udc->setup_status = WAIT_FOR_SETUP;
+
+ return 0;
+}
+
+static void enable_setup_event(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 val;
+
+ val = readl(&uccr->config1);
+ val |= CRG_U3DC_CFG1_SETUP_EVENT_EN;
+ writel(val, &uccr->config1);
+}
+
+void crg_udc_ep0_req_complete(struct crg_udc_ep *udc_ep_ptr,
+ unsigned long *flags)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+
+ switch (crg_udc->setup_status) {
+ case DATA_STAGE_XFER:
+ crg_udc->setup_status = STATUS_STAGE_RECV;
+ crg_udc_submit_ep0_status(crg_udc, -EINPROGRESS, 0);
+ break;
+ case DATA_STAGE_RECV:
+ crg_udc->setup_status = STATUS_STAGE_XFER;
+ crg_udc_submit_ep0_status(crg_udc, -EINPROGRESS, 0);
+ break;
+ default:
+ if (crg_udc->setup_fn_call_back) {
+ crg_udc->setup_fn_call_back(crg_udc);
+ crg_udc->setup_fn_call_back = NULL;
+ }
+ crg_udc->setup_status = WAIT_FOR_SETUP;
+ break;
+ }
+}
+
+static void get_status_cmpl(struct usb_ep *ep, struct usb_request *req)
+{
+ struct crg_udc_ep *udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+ struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+ unsigned long flags;
+ void *buf;
+
+ /* The data-stage completion handler queues the status stage. */
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ buf = req->buf;
+ req->buf = NULL;
+ crg_udc->status_req_callback_pending = false;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ kfree(buf);
+}
+
+static void getstatusrequest(struct crg_gadget_dev *crg_udc, u8 request_type,
+ u16 value, u16 index, u16 length)
+{
+ struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+ struct crg_udc_request *udc_req_ptr = crg_udc->status_req;
+ u16 status_val = 0;
+ u32 dci;
+ int ret;
+
+ if (!udc_req_ptr || crg_udc->status_req_callback_pending ||
+ !list_empty(&udc_req_ptr->queue) || value ||
+ length != sizeof(u16)) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ switch (request_type & USB_RECIP_MASK) {
+ case USB_RECIP_DEVICE:
+ if (index)
+ goto invalid;
+ if (crg_udc->gadget.speed == USB_SPEED_HIGH ||
+ crg_udc->gadget.speed == USB_SPEED_FULL) {
+ if (crg_udc->u2_rwe)
+ status_val |= BIT(USB_DEVICE_REMOTE_WAKEUP);
+ } else if (crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+ if (crg_udc->feature_u1_enable)
+ status_val |= BIT(USB_DEV_STAT_U1_ENABLED);
+ if (crg_udc->feature_u2_enable)
+ status_val |= BIT(USB_DEV_STAT_U2_ENABLED);
+ }
+ if (crg_udc->gadget.is_selfpowered)
+ status_val |= BIT(USB_DEVICE_SELF_POWERED);
+ break;
+
+ case USB_RECIP_INTERFACE:
+ if (crg_udc->device_state != USB_STATE_CONFIGURED ||
+ (index & 0xff00))
+ goto invalid;
+ break;
+
+ case USB_RECIP_ENDPOINT:
+ if (index & ~(USB_ENDPOINT_NUMBER_MASK | USB_DIR_IN))
+ goto invalid;
+ dci = crg_udc_index2dci(index);
+ if (dci == 1 || dci >= CRG_NUM_EP_CX)
+ goto invalid;
+ if (crg_udc->device_state < USB_STATE_ADDRESS && dci != 0)
+ goto invalid;
+ if (crg_udc->device_state == USB_STATE_ADDRESS && dci != 0)
+ goto invalid;
+ if (crg_udc->device_state == USB_STATE_CONFIGURED &&
+ crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_DISABLED)
+ goto invalid;
+ if (crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_HALTED)
+ status_val = BIT(USB_ENDPOINT_HALT);
+ break;
+
+ default:
+ goto invalid;
+ }
+
+ kfree(udc_req_ptr->usb_req.buf);
+
+ udc_req_ptr->usb_req.buf = kzalloc_obj(status_val, GFP_ATOMIC);
+ if (!udc_req_ptr->usb_req.buf)
+ goto invalid;
+
+ *(__le16 *)udc_req_ptr->usb_req.buf = cpu_to_le16(status_val);
+ udc_req_ptr->usb_req.length = sizeof(status_val);
+ udc_req_ptr->usb_req.status = -EINPROGRESS;
+ udc_req_ptr->usb_req.actual = 0;
+ udc_req_ptr->usb_req.complete = get_status_cmpl;
+ crg_udc->status_req_callback_pending = true;
+
+ ret = crg_udc_bounce_map_request(udc_ep_ptr, udc_req_ptr,
+ DMA_TO_DEVICE, GFP_ATOMIC);
+ if (ret) {
+ kfree(udc_req_ptr->usb_req.buf);
+ udc_req_ptr->usb_req.buf = NULL;
+ crg_udc->status_req_callback_pending = false;
+ goto invalid;
+ }
+
+ crg_udc->setup_status = DATA_STAGE_XFER;
+
+ ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+ if (ret) {
+ crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr, 0);
+ kfree(udc_req_ptr->usb_req.buf);
+ udc_req_ptr->usb_req.buf = NULL;
+ crg_udc->status_req_callback_pending = false;
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+ list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+ return;
+
+invalid:
+ crg_udc_set_ep0_halt(crg_udc);
+}
+
+int crg_udc_prepare_for_setup(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_udc_ep *udc_ep0_ptr;
+ dma_addr_t dqptaddr;
+ u32 cmd_param0;
+ u32 cmd_param1;
+ int ret;
+
+ if (!crg_udc_gadget_ready(crg_udc))
+ return -ESHUTDOWN;
+
+ if (!crg_udc_is_event_rings_empty(crg_udc) ||
+ crg_udc->portsc_on_reconnecting == 1)
+ return -EBUSY;
+
+ udc_ep0_ptr = &crg_udc->udc_ep[0];
+ udc_ep0_ptr->generation++;
+ /* If we reinit ep0 on bus reset, we just make ep0 dequeue pointer
+ * align with enqueue pointer, all remaining xfer trbs became dumb once
+ * which will not produce xfer event anymore.
+ *
+ * If we considering the opposite solution, we should wait all ep0 xfer
+ * trbs be completed(with some err complete code)
+ */
+ if (crg_udc->reinit_ep0_on_bus_reset) {
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ enum crg_ep_state ep0_ep_state;
+ u32 generation;
+ u32 running;
+
+ /* Block requeue from giveback callbacks, then quiesce DMA. */
+ crg_udc->ep0_state = EP0_STATE_ABORTING;
+ /* A HALTED EP0 may still own a queued STALL TRB. */
+ ep0_ep_state = crg_udc_get_ep_state(crg_udc, 0);
+ if (ep0_ep_state != EP_STATE_DISABLED &&
+ ep0_ep_state != EP_STATE_STOPPED) {
+ ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP,
+ BIT(0), 0);
+ if (ret)
+ goto reset_error;
+
+ ret = readl_poll_timeout_atomic(&uccr->ep_running, running,
+ !(running & BIT(0)), 10, 10000);
+ if (ret)
+ goto reset_error;
+ }
+ udc_ep0_ptr->ep_state = EP_STATE_STOPPED;
+ generation = udc_ep0_ptr->generation;
+ crg_udc_ep_nuke(udc_ep0_ptr, -ESHUTDOWN);
+ /* The giveback callback may have disconnected the gadget. */
+ if (!crg_udc_gadget_ready(crg_udc) ||
+ generation != udc_ep0_ptr->generation)
+ return -ECONNRESET;
+
+ /* context related ops */
+ dqptaddr = crg_udc_tran_trb_virt_to_dma(udc_ep0_ptr,
+ udc_ep0_ptr->enq_pt);
+ if (dma_mapping_error(crg_udc->dev, dqptaddr)) {
+ ret = -EIO;
+ goto reset_error;
+ }
+ cmd_param0 =
+ (lower_32_bits(dqptaddr) & CRG_CMD0_0_DQPTRLO_MASK) |
+ FIELD_PREP(CRG_CMD0_0_DCS_MASK, udc_ep0_ptr->pcs);
+ cmd_param1 = upper_32_bits(dqptaddr);
+ ret = crg_udc_issue_command(crg_udc, CRG_CMD_INIT_EP0,
+ cmd_param0, cmd_param1);
+ if (ret)
+ goto reset_error;
+
+ udc_ep0_ptr->deq_pt = udc_ep0_ptr->enq_pt;
+ udc_ep0_ptr->tran_ring_full = false;
+ udc_ep0_ptr->ep_state = EP_STATE_RUNNING;
+ crg_udc->ep0_state = EP0_STATE_IDLE;
+ } else {
+ if (!list_empty(&udc_ep0_ptr->queue)) {
+ dev_err(crg_udc->dev,
+ "remaining ep0 xfer trbs on reset\n");
+ return -EBUSY;
+ }
+ }
+
+ crg_udc_enable_setup(crg_udc);
+
+ return 0;
+
+reset_error:
+ crg_udc->ep0_state = EP0_STATE_ERROR;
+ crg_udc_schedule_endpoint_recovery(crg_udc);
+ return ret;
+}
+
+static void set_address_cmpl(struct crg_gadget_dev *crg_udc)
+{
+ if (crg_udc->device_state == USB_STATE_DEFAULT &&
+ crg_udc->dev_addr != 0) {
+ crg_udc->device_state = USB_STATE_ADDRESS;
+ usb_gadget_set_state(&crg_udc->gadget, USB_STATE_ADDRESS);
+ } else if (crg_udc->device_state == USB_STATE_ADDRESS) {
+ if (crg_udc->dev_addr == 0) {
+ crg_udc->device_state = USB_STATE_DEFAULT;
+ usb_gadget_set_state(&crg_udc->gadget,
+ USB_STATE_DEFAULT);
+ }
+
+ }
+}
+
+int crg_udc_set_ep0_halt(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+ enum crg_ep_state ep_state;
+ int ret;
+
+ ep_state = crg_udc_get_ep_state(crg_udc, udc_ep_ptr->dci);
+ if (ep_state == EP_STATE_HALTED)
+ return 0;
+ if (ep_state == EP_STATE_DISABLED)
+ return -ESHUTDOWN;
+
+ /* STALL terminates the current DATA/STATUS sequence. */
+ crg_udc->setup_status = STATUS_STAGE_XFER;
+ crg_udc->setup_fn_call_back = NULL;
+
+ ret = crg_udc_build_ep0_status(udc_ep_ptr, true, -EINVAL,
+ NULL, 0, 1);
+ if (ret) {
+ /* A failed STALL publication leaves EP0 needing recovery. */
+ crg_udc->ep0_state = EP0_STATE_ERROR;
+ return ret;
+ }
+
+ udc_ep_ptr->ep_state = EP_STATE_HALTED;
+
+ return 0;
+}
+
+static void set_sel_cmpl(struct usb_ep *ep, struct usb_request *req)
+{
+ struct crg_udc_ep *udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+ struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+ const struct usb_set_sel_req *sel;
+ unsigned long flags;
+ void *buf;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ buf = req->buf;
+ if (buf && !req->status && req->actual >= sizeof(*sel)) {
+ sel = buf;
+ crg_udc->sel_value.u1_sel_value = sel->u1_sel;
+ crg_udc->sel_value.u1_pel_value = sel->u1_pel;
+ crg_udc->sel_value.u2_sel_value = le16_to_cpu(sel->u2_sel);
+ crg_udc->sel_value.u2_pel_value = le16_to_cpu(sel->u2_pel);
+ }
+ req->buf = NULL;
+ crg_udc->status_req_callback_pending = false;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ kfree(buf);
+}
+
+static void set_test_mode_cmpl(struct crg_gadget_dev *crg_udc)
+{
+ if (crg_udc->set_tm != 0) {
+ u32 tmp;
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+
+ tmp = readl(&uccr->u2portpmsc);
+ FIELD_MODIFY(CRG_U3DC_U2PORTPM_TM_MASK, &tmp, crg_udc->set_tm);
+ writel(tmp, &uccr->u2portpmsc);
+
+ crg_udc->set_tm = 0;
+ }
+}
+
+static void setaddressrequest(struct crg_gadget_dev *crg_udc, u16 value,
+ u16 index, u16 length)
+{
+ u32 param0;
+
+ if (value > 127 || index || length ||
+ (crg_udc->device_state != USB_STATE_DEFAULT &&
+ crg_udc->device_state != USB_STATE_ADDRESS))
+ goto invalid;
+
+ /* SET_ADDRESS(0) is legal in the Default state. */
+ param0 = FIELD_PREP(CRG_CMD2_0_DEV_ADDR_MASK, value);
+ if (crg_udc_issue_command(crg_udc, CRG_CMD_SET_ADDR, param0, 0))
+ goto invalid;
+
+ crg_udc->dev_addr = value;
+ crg_udc->setup_status = STATUS_STAGE_XFER;
+ crg_udc->setup_fn_call_back = set_address_cmpl;
+ crg_udc_submit_ep0_status(crg_udc, -EINPROGRESS, 1);
+ return;
+
+invalid:
+ crg_udc->setup_fn_call_back = NULL;
+ crg_udc_set_ep0_halt(crg_udc);
+}
+
+static bool setfeaturesrequest(struct crg_gadget_dev *crg_udc, u8 request_type,
+ u8 b_request, u16 value, u16 index, u16 length,
+ unsigned long *flags)
+{
+ int status = -EINPROGRESS;
+ u8 dci;
+ struct crg_udc_ep *udc_ep_ptr;
+ u32 tmp;
+ u32 set_feat = 0;
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+
+ if (length || (request_type & USB_DIR_IN))
+ goto set_feature_error;
+
+ /* TEST_MODE is permitted in the Default state; other features aren't. */
+ if (crg_udc->device_state == USB_STATE_DEFAULT &&
+ !((request_type & USB_RECIP_MASK) == USB_RECIP_DEVICE &&
+ value == USB_DEVICE_TEST_MODE &&
+ b_request == USB_REQ_SET_FEATURE))
+ goto set_feature_error;
+
+ set_feat = (b_request == USB_REQ_SET_FEATURE) ? 1 : 0;
+ if ((request_type & (USB_RECIP_MASK | USB_TYPE_MASK)) ==
+ (USB_RECIP_ENDPOINT | USB_TYPE_STANDARD)) {
+ dev_dbg(crg_udc->dev, "Halt/Unhalt EP\n");
+ if (crg_udc->device_state == USB_STATE_ADDRESS) {
+ if (index != 0) {
+ status = -EINVAL;
+ goto set_feature_error;
+ }
+ }
+
+ if (value != USB_ENDPOINT_HALT)
+ goto set_feature_error;
+
+ if (index & ~(USB_ENDPOINT_NUMBER_MASK | USB_DIR_IN))
+ goto set_feature_error;
+
+ dci = crg_udc_index2dci(index);
+ if (dci == 1 || dci >= CRG_NUM_EP_CX)
+ goto set_feature_error;
+ if (dci != 0 && crg_udc->device_state == USB_STATE_CONFIGURED &&
+ crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_DISABLED)
+ goto set_feature_error;
+
+ udc_ep_ptr = &crg_udc->udc_ep[dci];
+ status = crg_udc_ep_halt(
+ udc_ep_ptr, (b_request == USB_REQ_SET_FEATURE) ? 1 : 0,
+ 0, flags);
+
+ if (status < 0)
+ goto set_feature_error;
+ } else if ((request_type & (USB_RECIP_MASK | USB_TYPE_MASK)) ==
+ (USB_RECIP_DEVICE | USB_TYPE_STANDARD)) {
+ switch (value) {
+ case USB_DEVICE_REMOTE_WAKEUP:
+ /* REMOTE_WAKEUP selector is not used by USB3.0 */
+ if (index ||
+ crg_udc->device_state < USB_STATE_DEFAULT ||
+ crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+ status = -EINVAL;
+ goto set_feature_error;
+ }
+
+ crg_udc->u2_rwe = set_feat;
+ crg_udc->gadget.wakeup_armed = set_feat;
+
+ break;
+ case USB_DEVICE_U1_ENABLE:
+ case USB_DEVICE_U2_ENABLE: {
+ u32 timeout_val;
+
+ if (crg_udc->device_state != USB_STATE_CONFIGURED) {
+ dev_err(crg_udc->dev,
+ "u12 enable fail, usb state=%d\n",
+ crg_udc->device_state);
+ status = -EINVAL;
+ goto set_feature_error;
+ }
+
+ if (index != 0 ||
+ crg_udc->gadget.speed < USB_SPEED_SUPER) {
+ status = -EINVAL;
+ goto set_feature_error;
+ }
+
+ if (set_feat == 1 && crg_udc->u12_forbidden) {
+ status = -EINVAL;
+ goto set_feature_error;
+ }
+
+ tmp = readl(&uccr->u3portpmsc);
+ if (value == USB_DEVICE_U1_ENABLE) {
+ if (!crg_udc->u12_initiate_forbidden) {
+ tmp &= (~CRG_U3DC_U3PORTPM_U1IEN);
+ tmp |= FIELD_PREP(
+ CRG_U3DC_U3PORTPM_U1IEN,
+ set_feat);
+ }
+ if (!crg_udc->u12_forbidden)
+ timeout_val =
+ set_feat ? CRG_UDC_U1_TIMEOUT_VAL : 0;
+ else
+ timeout_val = 0;
+
+ FIELD_MODIFY(CRG_U3DC_U3PORTPM_U1TMOUT_MASK,
+ &tmp, timeout_val);
+
+ crg_udc->feature_u1_enable = set_feat;
+ }
+
+ if (value == USB_DEVICE_U2_ENABLE) {
+ if (!crg_udc->u12_initiate_forbidden) {
+ tmp &= (~CRG_U3DC_U3PORTPM_U2IEN);
+ tmp |= FIELD_PREP(
+ CRG_U3DC_U3PORTPM_U2IEN,
+ set_feat);
+ }
+ if (!crg_udc->u12_forbidden)
+ timeout_val =
+ set_feat ? CRG_UDC_U2_TIMEOUT_VAL : 0;
+ else
+ timeout_val = 0;
+
+ FIELD_MODIFY(CRG_U3DC_U3PORTPM_U2TMOUT_MASK,
+ &tmp, timeout_val);
+
+ crg_udc->feature_u2_enable = set_feat;
+ }
+ writel(tmp, &uccr->u3portpmsc);
+
+ break;
+ }
+ case USB_DEVICE_TEST_MODE: {
+ u32 u_pattern;
+
+ if (b_request != USB_REQ_SET_FEATURE ||
+ crg_udc->gadget.speed != USB_SPEED_HIGH ||
+ crg_udc->device_state < USB_STATE_DEFAULT ||
+ (index & 0xff) || (index >> 8) == 0 ||
+ (index >> 8) > 6)
+ goto set_feature_error;
+ u_pattern = index >> 8;
+ crg_udc->set_tm = u_pattern;
+ crg_udc->setup_fn_call_back = &set_test_mode_cmpl;
+
+ break;
+ }
+
+ default:
+ goto set_feature_error;
+ }
+
+ } else if ((request_type & (USB_RECIP_MASK | USB_TYPE_MASK)) ==
+ (USB_RECIP_INTERFACE | USB_TYPE_STANDARD)) {
+ if (crg_udc->device_state != USB_STATE_CONFIGURED) {
+ dev_err(crg_udc->dev,
+ "interface u12 enable fail, usb state=%d\n",
+ crg_udc->device_state);
+ status = -EINVAL;
+ goto set_feature_error;
+ }
+
+ /* Suspend Option */
+ if (value != USB_INTRF_FUNC_SUSPEND) {
+ status = -EINVAL;
+ goto set_feature_error;
+ }
+
+ /* Both cleared LP/RW bits and either suspend option are valid. */
+ if (index & ~(0xff | USB_INTRF_FUNC_SUSPEND_LP |
+ USB_INTRF_FUNC_SUSPEND_RW))
+ goto set_feature_error;
+
+ if (index & USB_INTRF_FUNC_SUSPEND_RW)
+ dev_dbg(crg_udc->dev, "Interface En Remote Wakeup\n");
+ else
+ dev_dbg(crg_udc->dev, "Interface Dis RemoteWakeup\n");
+
+ /* Do not need to return status stage here
+ * Pass to composite gadget driver to process
+ * the request
+ */
+ return false;
+ } else {
+ /* Unknown recipient must not receive a successful status stage. */
+ goto set_feature_error;
+ }
+
+ crg_udc->setup_status = STATUS_STAGE_XFER;
+ crg_udc_submit_ep0_status(crg_udc, status, 0);
+ return true;
+
+set_feature_error:
+ crg_udc_set_ep0_halt(crg_udc);
+ return true;
+}
+
+static void setselrequest(struct crg_gadget_dev *crg_udc, u16 value, u16 index,
+ u16 length)
+{
+ struct crg_udc_request *udc_req_ptr = crg_udc->status_req;
+ struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+ struct usb_set_sel_req *sel;
+ int ret;
+
+ if (!udc_req_ptr || crg_udc->status_req_callback_pending ||
+ !list_empty(&udc_req_ptr->queue) ||
+ crg_udc->device_state == USB_STATE_DEFAULT || value || index ||
+ length != sizeof(struct usb_set_sel_req)) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ kfree(udc_req_ptr->usb_req.buf);
+
+ udc_req_ptr->usb_req.buf = kzalloc_obj(*sel, GFP_ATOMIC);
+ if (!udc_req_ptr->usb_req.buf) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ sel = udc_req_ptr->usb_req.buf;
+ udc_req_ptr->usb_req.length = sizeof(*sel);
+ udc_req_ptr->usb_req.status = -EINPROGRESS;
+ udc_req_ptr->usb_req.actual = 0;
+ udc_req_ptr->usb_req.complete = set_sel_cmpl;
+ crg_udc->status_req_callback_pending = true;
+ ret = crg_udc_bounce_map_request(udc_ep_ptr, udc_req_ptr,
+ DMA_FROM_DEVICE, GFP_ATOMIC);
+ if (ret) {
+ kfree(udc_req_ptr->usb_req.buf);
+ udc_req_ptr->usb_req.buf = NULL;
+ crg_udc->status_req_callback_pending = false;
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ crg_udc->setup_status = DATA_STAGE_RECV;
+
+ ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+ if (ret) {
+ crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr, 0);
+ kfree(udc_req_ptr->usb_req.buf);
+ udc_req_ptr->usb_req.buf = NULL;
+ crg_udc->status_req_callback_pending = false;
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+ list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+}
+
+int crg_udc_update_ep0_maxpacketsize(struct crg_gadget_dev *crg_udc)
+{
+ u16 maxpacketsize = 0;
+ struct crg_udc_ep *udc_ep0 = &crg_udc->udc_ep[0];
+ u32 param0;
+
+ if (crg_udc->gadget.speed >= USB_SPEED_SUPER)
+ maxpacketsize = 512;
+ else
+ maxpacketsize = 64;
+
+ param0 = FIELD_PREP(CRG_CMD1_0_MPS_MASK, maxpacketsize);
+ if (crg_udc_issue_command(crg_udc, CRG_CMD_UPDATE_EP0_CFG, param0, 0))
+ return -EIO;
+
+ crg_udc->ep0_desc.wMaxPacketSize = cpu_to_le16(maxpacketsize);
+ udc_ep0->usb_ep.maxpacket = maxpacketsize;
+
+ return 0;
+}
+
+void crg_udc_handle_setup_pkt(struct crg_gadget_dev *crg_udc,
+ struct usb_ctrlrequest *setup_pkt, u8 setup_tag,
+ unsigned long *flags)
+{
+ u16 w_value = le16_to_cpu(setup_pkt->wValue);
+ u16 w_index = le16_to_cpu(setup_pkt->wIndex);
+ u16 w_length = le16_to_cpu(setup_pkt->wLength);
+ struct usb_gadget_driver *driver, *setup_wait_driver;
+ u32 ep0_generation, event_generation;
+ u32 wait_ep0_generation, wait_event_generation;
+ u8 wait_setup_tag, active_setup_tag;
+ int setup_ret;
+ u32 tmp;
+
+ /* Guard EP0 across flush_work(), which temporarily drops udc_lock. */
+ if (!crg_udc_gadget_ready(crg_udc) ||
+ crg_udc->udc_ep[0].ep_state != EP_STATE_RUNNING ||
+ crg_udc->ep0_state != EP0_STATE_IDLE)
+ return;
+ wait_ep0_generation = crg_udc->udc_ep[0].generation;
+ wait_event_generation = crg_udc->udc_event[0].generation;
+ setup_wait_driver = crg_udc->gadget_driver;
+ wait_setup_tag = crg_udc->setup_tag;
+ crg_udc->ep0_state = EP0_STATE_ABORTING;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+ flush_work(&crg_udc->udc_reset_work);
+ spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+ /* Another SETUP or reset may have taken ownership of EP0. */
+ if (wait_ep0_generation != crg_udc->udc_ep[0].generation ||
+ wait_event_generation != crg_udc->udc_event[0].generation ||
+ wait_setup_tag != crg_udc->setup_tag ||
+ crg_udc->ep0_state != EP0_STATE_ABORTING)
+ return;
+
+ if (!crg_udc_gadget_ready(crg_udc) ||
+ crg_udc->device_state == USB_STATE_RECONNECTING ||
+ crg_udc->bus_reset_pending || crg_udc->port_reset_pending ||
+ crg_udc->udc_ep[0].ep_state != EP_STATE_RUNNING ||
+ crg_udc->gadget_driver != setup_wait_driver) {
+ /*
+ * Leave EP0 blocked: reset/reinit or a new SETUP must reclaim it.
+ * Returning to IDLE here could allow queues during teardown.
+ */
+ return;
+ }
+ crg_udc->ep0_state = EP0_STATE_IDLE;
+
+ trace_crg_udc_ctrl_req(setup_pkt);
+
+ /* New SETUP supersedes any mapping of the previous transaction. */
+ crg_udc->udc_ep[0].generation++;
+ /* EP0 come backs to running when new setup packet comes */
+ crg_udc->udc_ep[0].ep_state = EP_STATE_RUNNING;
+
+ crg_udc->setup_tag = setup_tag;
+ crg_udc->setup_status = SETUP_PKT_PROCESS_IN_PROGRESS;
+ crg_udc->setup_fn_call_back = NULL;
+
+ if ((setup_pkt->bRequestType & USB_TYPE_MASK) == USB_TYPE_STANDARD) {
+ switch (setup_pkt->bRequest) {
+ case USB_REQ_GET_STATUS:
+ if (setup_pkt->bRequestType !=
+ (USB_DIR_IN | USB_RECIP_DEVICE |
+ USB_TYPE_STANDARD) &&
+ setup_pkt->bRequestType !=
+ (USB_DIR_IN | USB_RECIP_INTERFACE |
+ USB_TYPE_STANDARD) &&
+ setup_pkt->bRequestType !=
+ (USB_DIR_IN | USB_RECIP_ENDPOINT |
+ USB_TYPE_STANDARD)) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ getstatusrequest(crg_udc, setup_pkt->bRequestType,
+ w_value, w_index, w_length);
+ return;
+ case USB_REQ_SET_ADDRESS:
+ if (setup_pkt->bRequestType !=
+ (USB_DIR_OUT | USB_RECIP_DEVICE |
+ USB_TYPE_STANDARD)) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ setaddressrequest(crg_udc, w_value, w_index, w_length);
+ return;
+ case USB_REQ_SET_SEL:
+ if (setup_pkt->bRequestType !=
+ (USB_DIR_OUT | USB_RECIP_DEVICE |
+ USB_TYPE_STANDARD)) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+ crg_udc->setup_status = DATA_STAGE_RECV;
+ setselrequest(crg_udc, w_value, w_index, w_length);
+ return;
+ case USB_REQ_SET_ISOCH_DELAY:
+ if (setup_pkt->bRequestType !=
+ (USB_DIR_OUT | USB_RECIP_DEVICE |
+ USB_TYPE_STANDARD) ||
+ w_index || w_length ||
+ w_value >
+ FIELD_MAX(
+ CRG_U3DC_PTMCTRL_ISOCH_DELAY_MASK)) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ tmp = readl(&crg_udc->uccr->ptmctrl);
+ FIELD_MODIFY(CRG_U3DC_PTMCTRL_ISOCH_DELAY_MASK, &tmp,
+ w_value);
+ writel(tmp, &crg_udc->uccr->ptmctrl);
+ crg_udc->setup_status = STATUS_STAGE_XFER;
+ crg_udc_submit_ep0_status(crg_udc, -EINPROGRESS, 0);
+ return;
+
+ case USB_REQ_CLEAR_FEATURE:
+ case USB_REQ_SET_FEATURE:
+ /* Need composite gadget driver
+ * to process the function remote wakeup request
+ */
+ if (setfeaturesrequest(crg_udc, setup_pkt->bRequestType,
+ setup_pkt->bRequest, w_value,
+ w_index, w_length, flags)) {
+ /* Get here if request has been processed.*/
+ return;
+ }
+ break;
+ case USB_REQ_SET_CONFIGURATION:
+ if (setup_pkt->bRequestType !=
+ (USB_DIR_OUT | USB_RECIP_DEVICE |
+ USB_TYPE_STANDARD) ||
+ w_length || w_value > 0xff) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+ /* In theory we need to clear RUN bit before
+ * status stage of deconfig request sent.
+ * But seeing problem if we do it before all the
+ * endpoints belonging to the configuration
+ * disabled.
+ */
+ if (crg_udc->device_state <= USB_STATE_DEFAULT) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ break;
+ default:
+ dev_dbg(crg_udc->dev,
+ "USB_REQ default bRequest=%d, bRequestType=%d\n",
+ setup_pkt->bRequest, setup_pkt->bRequestType);
+ }
+ }
+
+ if (w_length) {
+ /* data phase from gadget like GET_CONFIGURATION
+ * call the setup routine of gadget driver.
+ * remember the request direction.
+ */
+ crg_udc->setup_status = (setup_pkt->bRequestType & USB_DIR_IN) ?
+ DATA_STAGE_XFER :
+ DATA_STAGE_RECV;
+ }
+ /* The callback drops udc_lock; a newer SETUP or reset can supersede us. */
+ ep0_generation = crg_udc->udc_ep[0].generation;
+ event_generation = crg_udc->udc_event[0].generation;
+ active_setup_tag = crg_udc->setup_tag;
+ driver = crg_udc->async_callbacks ? crg_udc->gadget_driver : NULL;
+ crg_udc_callback_enter(crg_udc);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+
+ if (!driver || !READ_ONCE(crg_udc->async_callbacks) ||
+ READ_ONCE(crg_udc->gadget_driver) != driver)
+ setup_ret = -ESHUTDOWN;
+ else
+ setup_ret = driver->setup(&crg_udc->gadget, setup_pkt);
+
+ spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+ crg_udc_callback_exit(crg_udc);
+ /* Never STALL or update device state for an obsolete SETUP. */
+ if (!crg_udc_gadget_ready(crg_udc) ||
+ crg_udc->bus_reset_pending || crg_udc->port_reset_pending ||
+ crg_udc->device_state == USB_STATE_RECONNECTING ||
+ !driver || crg_udc->gadget_driver != driver ||
+ crg_udc->udc_ep[0].generation != ep0_generation ||
+ crg_udc->udc_event[0].generation != event_generation ||
+ crg_udc->setup_tag != active_setup_tag ||
+ crg_udc->ep0_state != EP0_STATE_IDLE ||
+ crg_udc->udc_ep[0].ep_state != EP_STATE_RUNNING)
+ return;
+
+ if (setup_ret < 0) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ if (setup_pkt->bRequest == USB_REQ_SET_CONFIGURATION &&
+ (setup_pkt->bRequestType & (USB_TYPE_MASK | USB_RECIP_MASK)) ==
+ (USB_TYPE_STANDARD | USB_RECIP_DEVICE)) {
+ if (w_value) {
+ crg_udc->device_state = USB_STATE_CONFIGURED;
+ usb_gadget_set_state(&crg_udc->gadget,
+ USB_STATE_CONFIGURED);
+ } else {
+ /* Endpoint disable may already have updated device_state. */
+ crg_udc->device_state = USB_STATE_ADDRESS;
+ usb_gadget_set_state(&crg_udc->gadget,
+ USB_STATE_ADDRESS);
+ }
+ }
+}
+
+int crg_udc_init_ep0(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+ u32 cmd_param0;
+ u32 cmd_param1;
+
+ if (!udc_ep_ptr->tran_ring_info.cpu_addr)
+ return -ENOMEM;
+
+ udc_ep_ptr->generation++;
+ udc_ep_ptr->first_trb = udc_ep_ptr->tran_ring_info.cpu_addr;
+ udc_ep_ptr->last_trb = udc_ep_ptr->first_trb +
+ CRG_UDC_CONTROL_EP_TD_RING_SIZE - 1;
+ memset(udc_ep_ptr->first_trb, 0, udc_ep_ptr->tran_ring_info.len);
+ /* Ensure that transfer ring is updated */
+ dma_wmb();
+ udc_ep_ptr->enq_pt = udc_ep_ptr->first_trb;
+ udc_ep_ptr->deq_pt = udc_ep_ptr->first_trb;
+ udc_ep_ptr->pcs = 1;
+ udc_ep_ptr->tran_ring_full = false;
+
+ crg_udc_setup_link_trb(udc_ep_ptr->last_trb, true,
+ udc_ep_ptr->tran_ring_info.dma_addr);
+
+ /* context related ops */
+ cmd_param0 = (lower_32_bits(udc_ep_ptr->tran_ring_info.dma_addr) &
+ CRG_CMD0_0_DQPTRLO_MASK) |
+ FIELD_PREP(CRG_CMD0_0_DCS_MASK, udc_ep_ptr->pcs);
+ cmd_param1 = upper_32_bits(udc_ep_ptr->tran_ring_info.dma_addr);
+
+ if (crg_udc_issue_command(crg_udc, CRG_CMD_INIT_EP0, cmd_param0,
+ cmd_param1))
+ return -EIO;
+
+ udc_ep_ptr->desc = &crg_udc->ep0_desc;
+ udc_ep_ptr->ep_state = EP_STATE_RUNNING;
+ udc_ep_ptr->pcs = 1;
+
+ return 0;
+}
+
+int crg_udc_queue_ctrl(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr,
+ u32 num_of_trbs_needed)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+ enum crg_ep_state ep_state;
+ struct crg_transfer_trb *enq_pt = udc_ep_ptr->enq_pt;
+ struct crg_transfer_trb *dq_pt = udc_ep_ptr->deq_pt;
+ struct usb_request *usb_req = &udc_req_ptr->usb_req;
+ struct crg_transfer_trb *p_trb;
+ u32 transfer_length;
+ bool ioc;
+ bool azp;
+ u8 dir = 0;
+ u8 setup_tag = crg_udc->setup_tag;
+
+ /* A new SETUP aborts the previous EP0 transaction. The completion
+ * callback of the aborted request is allowed to run, but it must not
+ * requeue the old request before the new SETUP has been handed to the
+ * gadget driver.
+ */
+ if (crg_udc->ep0_state == EP0_STATE_ABORTING)
+ return -ECONNRESET;
+
+ ep_state = crg_udc_get_ep_state(crg_udc, 0);
+ /* Need to queue the request even ep is paused or halted */
+ if (ep_state != EP_STATE_RUNNING) {
+ dev_dbg(crg_udc->dev, "EP State = 0x%x\n", ep_state);
+ return -EINVAL;
+ }
+ if (list_empty(&udc_ep_ptr->queue)) {
+ /* For control endpoint, we can handle one setup request at a
+ * time. so if there are TD pending in the transfer ring.
+ * wait for the sequence number error event. Then put the new
+ * request to transfer ring
+ */
+ if (enq_pt == dq_pt && !udc_ep_ptr->tran_ring_full) {
+ u32 i, td_done = 0;
+ bool need_zlp = false;
+ struct crg_udc_bounce *bounce = udc_req_ptr->bounce;
+ dma_addr_t trb_dma = bounce ? bounce->dma : 0;
+ u64 remaining = usb_req->length;
+ /* Transfer ring is empty
+ * setup data stage TRBs
+ */
+ udc_req_ptr->first_trb = udc_ep_ptr->enq_pt;
+ if (crg_udc->setup_status == DATA_STAGE_XFER)
+ dir = 0;
+ else if (crg_udc->setup_status == DATA_STAGE_RECV)
+ dir = 1;
+ else
+ dev_dbg(crg_udc->dev,
+ "unexpected setup_status!%d\n",
+ crg_udc->setup_status);
+ if (!udc_ep_ptr->usb_ep.maxpacket)
+ return -EINVAL;
+ if (udc_req_ptr->usb_req.zero == 1 &&
+ udc_req_ptr->usb_req.length != 0 &&
+ ((udc_req_ptr->usb_req.length %
+ udc_ep_ptr->usb_ep.maxpacket) == 0))
+ need_zlp = true; /* zero length packet */
+ if (num_of_trbs_needed > 32)
+ return -EMSGSIZE;
+
+ if (num_of_trbs_needed >
+ crg_udc_room_on_ring(
+ crg_udc, CRG_UDC_CONTROL_EP_TD_RING_SIZE,
+ udc_ep_ptr->first_trb, enq_pt, dq_pt))
+ return -ENOSPC;
+
+ u8 original_pcs = udc_ep_ptr->pcs;
+
+ for (i = 0; i < num_of_trbs_needed; i++) {
+ p_trb = enq_pt;
+ transfer_length = crg_udc_trb_chunk_len(
+ trb_dma, remaining);
+ ioc = (i == num_of_trbs_needed - 1);
+ azp = ioc && need_zlp;
+ td_done += transfer_length;
+ crg_udc_setup_datastage_trb(
+ p_trb, trb_dma,
+ i == 0 ? (original_pcs ^ 1) :
+ udc_ep_ptr->pcs,
+ transfer_length,
+ crg_udc_td_size(usb_req->length, td_done,
+ udc_ep_ptr->usb_ep.maxpacket, ioc),
+ ioc,
+ !ioc, azp, dir, setup_tag);
+ udc_req_ptr->all_trbs_queued = 1;
+ trb_dma += transfer_length;
+ remaining -= transfer_length;
+ enq_pt++;
+ if (FIELD_GET(TRB_TYPE_MASK,
+ le32_to_cpu(enq_pt->dw3)) ==
+ TRB_TYPE_LINK) {
+ if (FIELD_GET(
+ TRB_LINK_TOGGLE_CYCLE_MASK,
+ le32_to_cpu(enq_pt->dw3))) {
+ dma_wmb();
+ le32p_replace_bits(
+ &enq_pt->dw3,
+ udc_ep_ptr->pcs,
+ TRB_CYCLE_BIT_MASK);
+ udc_ep_ptr->pcs ^= 0x1;
+ }
+ enq_pt = udc_ep_ptr->first_trb;
+ }
+ }
+ dma_wmb();
+ le32p_replace_bits(&udc_req_ptr->first_trb->dw3,
+ original_pcs, TRB_CYCLE_BIT_MASK);
+ udc_ep_ptr->enq_pt = enq_pt;
+ crg_udc_knock_doorbell(crg_udc, 0);
+ if (udc_ep_ptr->enq_pt == udc_ep_ptr->first_trb)
+ udc_req_ptr->last_trb =
+ udc_ep_ptr->last_trb - 1;
+ else
+ udc_req_ptr->last_trb = udc_ep_ptr->enq_pt - 1;
+ } else {
+ /* The SETUP handler normally flushes stale control TRBs.
+ * Do not accept an unscheduled request if EP0 is still busy:
+ * SETUP_TAG_MISMATCH events are intentionally ignored.
+ */
+ dev_err(crg_udc->dev,
+ "ep0 ring not idle on new setup\n");
+ return -EBUSY;
+ }
+ } else {
+ dev_err(crg_udc->dev, "udc_ep_ptr->queue not empty\n");
+ /* New setup packet came
+ * Drop this request.
+ */
+ return -EINVAL;
+ }
+ return 0;
+}
--
2.34.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH v5 4/6] usb: gadget: udc: crg: Add endpoint 0 handling
2026-11-16 19:06 ` [PATCH v5 4/6] usb: gadget: udc: crg: Add endpoint 0 handling Petar Stepanovic
@ 2026-11-16 20:16 ` sashiko-bot
0 siblings, 0 replies; 13+ messages in thread
From: sashiko-bot @ 2026-11-16 20:16 UTC (permalink / raw)
To: Petar Stepanovic; +Cc: linux-usb, devicetree, robh, conor+dt
Thank you for your contribution! Sashiko AI review found 5 potential
issues to consider:
- [High] usb: gadget: udc: crg: broken EP0 halt and lock-drop handling
in setfeaturesrequest()
- [Medium] usb: gadget: udc: crg: clear U1 and U2 LPM state on
SET_CONFIGURATION(0)
- [Medium] usb: gadget: udc: crg: mishandled interface GET_STATUS and
FUNC_SUSPEND on EP0
- [Medium] usb: gadget: crg: missing setup_status error return and
ep0_state checks on EP0
- [Medium] usb: gadget: crg: missing ep0_state checks in EP0 status and
halt paths
--
commit 38b91eca05d2bb158aa1aa4be9dbbe69096a9ef9
Author: Petar Stepanovic <pstepanovic@axiado.com>
usb: gadget: udc: crg: Add endpoint 0 handling
This commit adds endpoint 0 setup packet, status stage, and control transfer
handling for the Axiado AX3000/AX3005 UDC driver.
> diff --git a/drivers/usb/gadget/udc/crg/crg-udc-ep0.c b/drivers/usb/gadget/udc/crg/crg-udc-ep0.c
> new file mode 100644
> index 0000000000000..4bd0dd4da8c67
> --- /dev/null
> +++ b/drivers/usb/gadget/udc/crg/crg-udc-ep0.c
> @@ -0,0 +1,1264 @@
[ ... ]
> +int crg_udc_build_ep0_status(struct crg_udc_ep *udc_ep_ptr, bool default_value,
> + u32 status, struct crg_udc_request *udc_req_ptr,
> + u8 set_addr, u8 stall)
> +{
> + struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
> + struct crg_transfer_trb *enq_pt = udc_ep_ptr->enq_pt;
> +
> + /* Internal status requests bypass usb_ep_queue(), so they must also
> + * respect a teardown that is giving back the old EP0 queue.
> + */
> + if (udc_ep_ptr->nuking) {
> + crg_udc->ep0_state = EP0_STATE_ERROR;
> + return -EBUSY;
> + }
> +
> + /* Reserve a free transfer-ring slot before touching request state or
> + * publishing the STATUS TRB. crg_udc_room_on_ring() treats equal
> + * pointers as empty, so check tran_ring_full separately.
> + */
> + if (!udc_ep_ptr->first_trb || !udc_ep_ptr->last_trb ||
> + !udc_ep_ptr->enq_pt || !udc_ep_ptr->deq_pt)
> + return -ENODEV;
> + if (udc_ep_ptr->ep_state != EP_STATE_RUNNING)
> + return -ESHUTDOWN;
[Severity: Medium]
Should crg_udc_build_ep0_status() and crg_udc_set_ep0_halt() check that
crg_udc->ep0_state is EP0_STATE_IDLE before queuing a status or stall TRB?
crg_udc_handle_setup_pkt() sets crg_udc->ep0_state = EP0_STATE_ABORTING
while udc_ep[0].ep_state is still EP_STATE_RUNNING, then drops udc_lock
across flush_work(&crg_udc->udc_reset_work):
CPU0 CPU1
crg_udc_handle_setup_pkt()
ep0_state = EP0_STATE_ABORTING
spin_unlock(&udc_lock)
flush_work(&udc_reset_work)
crg_udc_ep_set_halt(ep0)
...
crg_udc_set_ep0_halt()
crg_udc_build_ep0_status()
ep_state = EP_STATE_HALTED
spin_lock(&udc_lock)
if (ep_state != EP_STATE_RUNNING)
return; <-- drops new SETUP with ep0_state stuck in EP0_STATE_ABORTING
Because neither crg_udc_set_ep0_halt() nor crg_udc_build_ep0_status() checks
crg_udc->ep0_state, a concurrent halt or status submission while ep0_state
is EP0_STATE_ABORTING or EP0_STATE_ERROR queues status_req with the previous
setup_tag, rings doorbell 0, and sets ep_state to EP_STATE_HALTED without
bumping generation, causing crg_udc_handle_setup_pkt() to abort the incoming
SETUP and leave EP0 blocked in EP0_STATE_ABORTING.
[ ... ]
> +static bool setfeaturesrequest(struct crg_gadget_dev *crg_udc, u8 request_type,
> + u8 b_request, u16 value, u16 index, u16 length,
> + unsigned long *flags)
> +{
> + int status = -EINPROGRESS;
> + u8 dci;
> + struct crg_udc_ep *udc_ep_ptr;
> + u32 tmp;
> + u32 set_feat = 0;
> + struct crg_uccr __iomem *uccr = crg_udc->uccr;
[ ... ]
> + udc_ep_ptr = &crg_udc->udc_ep[dci];
> + status = crg_udc_ep_halt(
> + udc_ep_ptr, (b_request == USB_REQ_SET_FEATURE) ? 1 : 0,
> + 0, flags);
> +
> + if (status < 0)
> + goto set_feature_error;
[ ... ]
> + } else if ((request_type & (USB_RECIP_MASK | USB_TYPE_MASK)) ==
> + (USB_RECIP_INTERFACE | USB_TYPE_STANDARD)) {
[ ... ]
> + /* Do not need to return status stage here
> + * Pass to composite gadget driver to process
> + * the request
> + */
> + return false;
> + } else {
> + /* Unknown recipient must not receive a successful status stage. */
> + goto set_feature_error;
> + }
> +
> + crg_udc->setup_status = STATUS_STAGE_XFER;
> + crg_udc_submit_ep0_status(crg_udc, status, 0);
> + return true;
> +
> +set_feature_error:
> + crg_udc_set_ep0_halt(crg_udc);
> + return true;
> +}
[Severity: High]
Does the USB_RECIP_ENDPOINT path in setfeaturesrequest() mishandle EP0
targets, status-stage request state, and lock drops across
crg_udc_ep_halt()?
First, when SET_FEATURE(USB_ENDPOINT_HALT) targets EP0 (dci == 0),
crg_udc_ep_halt() calls crg_udc_set_ep0_halt(), which already queues
status_req with stall = 1 and sets udc_ep[0].ep_state = EP_STATE_HALTED;
falling through to crg_udc_submit_ep0_status() then fails with -ESHUTDOWN
and forces crg_udc->ep0_state into EP0_STATE_ERROR.
Second, crg_udc_ep_halt() can drop udc_lock via flags during request
givebacks in crg_udc_ep_nuke() or crg_udc_queue_pending_trbs(), but
setfeaturesrequest() does not revalidate EP0/event generations or reset
state before calling crg_udc_submit_ep0_status() or crg_udc_set_ep0_halt(),
which can publish a status or stall TRB and ring doorbell 0 during a reset
or on a reinitialized EP0 ring.
Third, assigning status = crg_udc_ep_halt(...) overwrites status from
-EINPROGRESS to 0 on success, so crg_udc_submit_ep0_status() queues the
in-flight status_req with usb_req.status = 0 before hardware completion.
[ ... ]
> + if ((setup_pkt->bRequestType & USB_TYPE_MASK) == USB_TYPE_STANDARD) {
> + switch (setup_pkt->bRequest) {
> + case USB_REQ_GET_STATUS:
> + if (setup_pkt->bRequestType !=
> + (USB_DIR_IN | USB_RECIP_DEVICE |
> + USB_TYPE_STANDARD) &&
> + setup_pkt->bRequestType !=
> + (USB_DIR_IN | USB_RECIP_INTERFACE |
> + USB_TYPE_STANDARD) &&
> + setup_pkt->bRequestType !=
> + (USB_DIR_IN | USB_RECIP_ENDPOINT |
> + USB_TYPE_STANDARD)) {
> + crg_udc_set_ep0_halt(crg_udc);
> + return;
> + }
> +
> + getstatusrequest(crg_udc, setup_pkt->bRequestType,
> + w_value, w_index, w_length);
> + return;
[Severity: Medium]
Should SuperSpeed USB_REQ_GET_STATUS for USB_RECIP_INTERFACE be delegated to
driver->setup(), and should setfeaturesrequest() reject
USB_INTRF_FUNC_SUSPEND below SuperSpeed?
setfeaturesrequest() delegates USB_INTRF_FUNC_SUSPEND to driver->setup() so
the composite driver can update function wakeup state, but
crg_udc_handle_setup_pkt() intercepts USB_REQ_GET_STATUS for
USB_RECIP_INTERFACE and getstatusrequest() always replies with 0 instead of
letting composite_setup() report USB_INTRF_STAT_FUNC_RW_CAP and
USB_INTRF_STAT_FUNC_RW. In addition, the USB_RECIP_INTERFACE branch in
setfeaturesrequest() does not check crg_udc->gadget.speed >= USB_SPEED_SUPER,
and composite_setup() only checks gadget_is_superspeed() (which tests
max_speed), so USB_INTRF_FUNC_SUSPEND is accepted at High-Speed and
Full-Speed instead of stalling EP0.
[ ... ]
> + if (setup_pkt->bRequest == USB_REQ_SET_CONFIGURATION &&
> + (setup_pkt->bRequestType & (USB_TYPE_MASK | USB_RECIP_MASK)) ==
> + (USB_TYPE_STANDARD | USB_RECIP_DEVICE)) {
> + if (w_value) {
> + crg_udc->device_state = USB_STATE_CONFIGURED;
> + usb_gadget_set_state(&crg_udc->gadget,
> + USB_STATE_CONFIGURED);
> + } else {
> + /* Endpoint disable may already have updated device_state. */
> + crg_udc->device_state = USB_STATE_ADDRESS;
> + usb_gadget_set_state(&crg_udc->gadget,
> + USB_STATE_ADDRESS);
> + }
> + }
> +}
[Severity: Medium]
When USB_REQ_SET_CONFIGURATION with w_value == 0 transitions the device back
to USB_STATE_ADDRESS, should it also clear U1/U2 link power management state
via crg_udc_clear_portpm()?
Without clearing uccr->u3portpmsc, feature_u1_enable, and feature_u2_enable
on deconfiguration, hardware U1/U2 initiation and timeouts stay active in
USB_STATE_ADDRESS, getstatusrequest() continues reporting
USB_DEV_STAT_U1_ENABLED and USB_DEV_STAT_U2_ENABLED, and the host cannot
clear them via CLEAR_FEATURE because setfeaturesrequest() rejects U1/U2
requests outside USB_STATE_CONFIGURED.
[ ... ]
> +int crg_udc_queue_ctrl(struct crg_udc_ep *udc_ep_ptr,
> + struct crg_udc_request *udc_req_ptr,
> + u32 num_of_trbs_needed)
> +{
[ ... ]
> + if (crg_udc->ep0_state == EP0_STATE_ABORTING)
> + return -ECONNRESET;
> +
> + ep_state = crg_udc_get_ep_state(crg_udc, 0);
> + /* Need to queue the request even ep is paused or halted */
> + if (ep_state != EP_STATE_RUNNING) {
> + dev_dbg(crg_udc->dev, "EP State = 0x%x\n", ep_state);
> + return -EINVAL;
> + }
> + if (list_empty(&udc_ep_ptr->queue)) {
> + /* For control endpoint, we can handle one setup request at a
> + * time. so if there are TD pending in the transfer ring.
> + * wait for the sequence number error event. Then put the new
> + * request to transfer ring
> + */
> + if (enq_pt == dq_pt && !udc_ep_ptr->tran_ring_full) {
> + u32 i, td_done = 0;
> + bool need_zlp = false;
> + struct crg_udc_bounce *bounce = udc_req_ptr->bounce;
> + dma_addr_t trb_dma = bounce ? bounce->dma : 0;
> + u64 remaining = usb_req->length;
> + /* Transfer ring is empty
> + * setup data stage TRBs
> + */
> + udc_req_ptr->first_trb = udc_ep_ptr->enq_pt;
> + if (crg_udc->setup_status == DATA_STAGE_XFER)
> + dir = 0;
> + else if (crg_udc->setup_status == DATA_STAGE_RECV)
> + dir = 1;
> + else
> + dev_dbg(crg_udc->dev,
> + "unexpected setup_status!%d\n",
> + crg_udc->setup_status);
[Severity: Medium]
Should crg_udc_queue_ctrl() return -EINVAL when setup_status is neither
DATA_STAGE_XFER nor DATA_STAGE_RECV, and also reject requests when
crg_udc->ep0_state != EP0_STATE_IDLE?
In crg_udc_ep_queue(), a non-zero-length EP0 request queued while
setup_status is STATUS_STAGE_XFER or STATUS_STAGE_RECV (for example, from a
completion callback during status-stage giveback before setup_status is
reset to WAIT_FOR_SETUP) passes the !_req->length check and reaches
crg_udc_queue_ctrl(). Because the else branch only logs dev_dbg(),
crg_udc_queue_ctrl() proceeds to queue TRB_TYPE_XFER_DATA_STAGE TRBs with
dir = 0 and ring the EP0 doorbell. Similarly, checking only
ep0_state == EP0_STATE_ABORTING at entry allows control requests to be
queued while ep0_state is EP0_STATE_ERROR.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261116-axiado-ax3000-usb-device-controller-v5-0-49271ed5c793@axiado.com?part=4
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v5 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations
2026-11-16 19:06 [PATCH v5 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
` (3 preceding siblings ...)
2026-11-16 19:06 ` [PATCH v5 4/6] usb: gadget: udc: crg: Add endpoint 0 handling Petar Stepanovic
@ 2026-11-16 19:06 ` Petar Stepanovic
2026-11-16 20:47 ` sashiko-bot
2026-11-16 19:06 ` [PATCH v5 6/6] usb: gadget: udc: crg: Add core driver and build integration Petar Stepanovic
5 siblings, 1 reply; 13+ messages in thread
From: Petar Stepanovic @ 2026-11-16 19:06 UTC (permalink / raw)
To: Prasad Bolisetty, Greg Kroah-Hartman, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Tzu-Hao Wei,
Karthikeyan Mitran, Philipp Zabel, Neill Kapron
Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
Petar Stepanovic, Krutik Shah, Naveen Kumar Rajgiri Bassappa
This patch implements the gadget and endpoint operations for the
Axiado AX3000/AX3005 UDC driver.
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Tzu-Hao Wei <twei@axiado.com>
Signed-off-by: Tzu-Hao Wei <twei@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
drivers/usb/gadget/udc/crg/crg-udc-gadget.c | 3255 +++++++++++++++++++++++++++
drivers/usb/gadget/udc/crg/crg-udc-trace.c | 9 +
2 files changed, 3264 insertions(+)
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-gadget.c b/drivers/usb/gadget/udc/crg/crg-udc-gadget.c
new file mode 100644
index 000000000000..dddfc5cf3d3e
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-gadget.c
@@ -0,0 +1,3255 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/gfp.h>
+#include <linux/highmem.h>
+#include <linux/interrupt.h>
+#include <linux/iopoll.h>
+#include <linux/jiffies.h>
+#include <linux/slab.h>
+
+#include "crg-udc.h"
+#include "crg-udc-trace.h"
+
+/* Forward declarations for functions used before their definition. */
+static int crg_udc_epcx_setup(struct crg_udc_ep *udc_ep);
+static int crg_udc_epcx_update_dqptr(struct crg_udc_ep *udc_ep);
+static int crg_udc_queue_trbs(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr, bool b_isoc,
+ u32 xfer_ring_size, u32 num_trbs_needed,
+ u64 buffer_length);
+static int crg_udc_reinit(struct crg_gadget_dev *crg_udc, bool notify_reset);
+static bool crg_udc_sg_advance(struct crg_udc_request *udc_req, u32 bytes);
+static bool crg_udc_sg_current(struct crg_udc_request *udc_req, dma_addr_t *dma,
+ u32 *len);
+static int crg_udc_sg_init(struct crg_udc_request *udc_req);
+static int crg_udc_validate_ep_desc(struct crg_udc_ep *udc_ep,
+ const struct usb_endpoint_descriptor *desc);
+static bool is_request_dequeued(struct crg_gadget_dev *crg_udc,
+ struct crg_udc_ep *udc_ep,
+ struct crg_event_trb *event);
+static int crg_udc_stop_error_ep(struct crg_gadget_dev *crg_udc, u8 dci);
+
+static int crg_gadget_get_frame(struct usb_gadget *g)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+ unsigned long flags;
+ int frame = -ESHUTDOWN;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (!crg_udc->gadget_stopping && !crg_udc->shutting_down &&
+ !crg_udc->resetting && !crg_udc->controller_recovering &&
+ !crg_udc->suspended)
+ frame = FIELD_GET(CRG_U3DC_MFINDEX_FRAME_MASK,
+ readl(&crg_udc->uccr->mfindex));
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return frame;
+}
+
+static int crg_gadget_pullup(struct usb_gadget *g, int is_on)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+ unsigned long flags;
+ int ret = 0;
+
+ /*
+ * The core can call ->disconnect() as soon as pullup(0) returns.
+ * Return the requests even if hardware stop required quarantine.
+ */
+ if (!is_on) {
+ ret = crg_udc_stop_gadget(crg_udc);
+ if (!ret)
+ trace_crg_udc_pullup(false);
+ return ret;
+ }
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (crg_udc->shutting_down || crg_udc->controller_recovering ||
+ (!crg_udc->gadget_stopping &&
+ (crg_udc->resetting || crg_udc->controller_faulted))) {
+ ret = -ESHUTDOWN;
+ goto out;
+ }
+ if (crg_udc->gadget_stopping) {
+ ret = crg_udc_restart_gadget(crg_udc, &flags);
+ if (ret)
+ goto out;
+ } else if (crg_udc->stopped_needs_reset) {
+ ret = crg_udc_recover(crg_udc, &flags);
+ if (ret)
+ goto out;
+ }
+
+ crg_udc->pullup_on = true;
+ if (!crg_udc->suspended)
+ crg_udc_set_pullup(crg_udc, true);
+out:
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ if (!ret)
+ trace_crg_udc_pullup(true);
+ return ret;
+}
+
+static int crg_gadget_set_remote_wakeup(struct usb_gadget *g, int set)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+ unsigned long flags;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ crg_udc->u2_rwe = !!set;
+ g->wakeup_armed = !!set;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ return 0;
+}
+
+static int crg_gadget_set_selfpowered(struct usb_gadget *g, int set)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+ unsigned long flags;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ g->is_selfpowered = !!set;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ return 0;
+}
+
+static void crg_udc_set_speed(struct usb_gadget *gadget,
+ enum usb_device_speed speed)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(gadget);
+
+ switch (speed) {
+ case USB_SPEED_FULL:
+ case USB_SPEED_HIGH:
+ case USB_SPEED_SUPER:
+ break;
+ case USB_SPEED_LOW:
+ speed = USB_SPEED_FULL;
+ break;
+ default:
+ speed = gadget->max_speed;
+ break;
+ }
+ crg_udc->max_speed = min(speed, gadget->max_speed);
+}
+
+static int crg_gadget_start(struct usb_gadget *g,
+ struct usb_gadget_driver *driver)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+ unsigned long flags;
+ int ret;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (crg_udc->shutting_down || crg_udc->controller_recovering ||
+ (!crg_udc->gadget_stopping &&
+ (crg_udc->resetting || crg_udc->controller_faulted))) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EBUSY;
+ }
+ if (crg_udc->gadget_stopping) {
+ ret = crg_udc_restart_gadget(crg_udc, &flags);
+ } else {
+ ret = crg_udc_program_speed(crg_udc);
+ if (!ret)
+ crg_udc_set_vbus_detect(crg_udc, true);
+ }
+ if (!ret)
+ crg_udc->gadget_driver = driver;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return ret;
+}
+
+static int crg_gadget_stop(struct usb_gadget *g)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+ unsigned long flags;
+ int i, ret;
+
+ ret = crg_udc_stop_gadget(crg_udc);
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ crg_udc->gadget_driver = NULL;
+ for (i = 2; i < CRG_NUM_EP_CX; i++)
+ crg_udc_ep_clear_enabled(&crg_udc->udc_ep[i]);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return ret;
+}
+
+static int crg_gadget_wakeup(struct usb_gadget *g)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ unsigned long flags;
+ u32 portsc;
+ int ret = 0;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (crg_udc->resetting) {
+ ret = -ESHUTDOWN;
+ goto out;
+ }
+ if (!crg_udc->connected || g->speed > USB_SPEED_HIGH) {
+ ret = -EOPNOTSUPP;
+ goto out;
+ }
+ if (!crg_udc->u2_rwe) {
+ ret = -EACCES;
+ goto out;
+ }
+
+ portsc = readl(&uccr->portsc);
+ if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, portsc) !=
+ CRG_U3DC_PORTSC_PLS_U3) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ portsc &= ~(CRG_U3DC_PORTSC_W1C_MASK | CRG_U3DC_PORTSC_PLS_MASK);
+ portsc |= CRG_U3DC_PORTSC_LWS;
+ writel(portsc, &uccr->portsc);
+out:
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return ret;
+}
+
+static void crg_gadget_async_callbacks(struct usb_gadget *g, bool enable)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+ unsigned long flags;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ crg_udc->async_callbacks = enable;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ /* Also cover unbind paths that skip pullup(0). */
+ if (!enable)
+ crg_udc_stop_gadget(crg_udc);
+}
+
+struct usb_request *crg_udc_alloc_request(struct usb_ep *_ep, gfp_t gfp_flags)
+{
+ struct crg_udc_request *udc_req_ptr;
+
+ udc_req_ptr = kzalloc_obj(*udc_req_ptr, gfp_flags);
+
+ if (!udc_req_ptr)
+ return NULL;
+
+ udc_req_ptr->usb_req.dma = DMA_MAPPING_ERROR;
+ INIT_LIST_HEAD(&udc_req_ptr->queue);
+
+ trace_crg_udc_alloc_request(
+ container_of(_ep, struct crg_udc_ep, usb_ep), udc_req_ptr);
+
+ return &udc_req_ptr->usb_req;
+}
+
+int crg_udc_build_td(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr)
+{
+ int status = 0;
+ struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+ u32 num_trbs_needed;
+ u64 buffer_length;
+
+ if (udc_req_ptr->trbs_needed) {
+ /* If this is called to complete pending TRB transfers
+ * of previous Request
+ */
+ buffer_length = udc_req_ptr->buff_len_left;
+ num_trbs_needed = udc_req_ptr->trbs_needed;
+ } else {
+ buffer_length = udc_req_ptr->usb_req.length;
+ if (udc_req_ptr->bounce && udc_req_ptr->bounce->sgt.nents)
+ num_trbs_needed = crg_udc_count_trbs_sg(udc_req_ptr);
+ else
+ num_trbs_needed = crg_udc_count_trbs_linear(
+ udc_req_ptr->bounce ? udc_req_ptr->bounce->dma : 0,
+ buffer_length);
+ if (!num_trbs_needed && buffer_length)
+ return -EINVAL;
+ if (!num_trbs_needed)
+ num_trbs_needed = 1;
+ }
+
+ if (usb_endpoint_xfer_control(udc_ep_ptr->desc))
+ status = crg_udc_queue_ctrl(udc_ep_ptr, udc_req_ptr,
+ num_trbs_needed);
+ else if (usb_endpoint_xfer_isoc(udc_ep_ptr->desc))
+ status = crg_udc_queue_trbs(udc_ep_ptr, udc_req_ptr, true,
+ CRG_UDC_ISOC_EP_TD_RING_SIZE,
+ num_trbs_needed, buffer_length);
+ else if (usb_endpoint_xfer_bulk(udc_ep_ptr->desc))
+ status = crg_udc_queue_trbs(udc_ep_ptr, udc_req_ptr, false,
+ CRG_UDC_BULK_EP_TD_RING_SIZE,
+ num_trbs_needed, buffer_length);
+ else
+ /* interrupt endpoint */
+ status = crg_udc_queue_trbs(udc_ep_ptr, udc_req_ptr, false,
+ CRG_UDC_INT_EP_TD_RING_SIZE,
+ num_trbs_needed, buffer_length);
+
+ if (!status) {
+ udc_req_ptr->td_pending = true;
+ if (!usb_endpoint_xfer_control(udc_ep_ptr->desc))
+ crg_udc_knock_doorbell(crg_udc, udc_ep_ptr->dci);
+ }
+
+ return status;
+}
+
+void crg_udc_clear_portpm(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 tmp;
+
+ tmp = readl(&uccr->u3portpmsc);
+
+ tmp &= (~CRG_U3DC_U3PORTPM_U1IEN);
+ FIELD_MODIFY(CRG_U3DC_U3PORTPM_U1TMOUT_MASK, &tmp, 0);
+ tmp &= (~CRG_U3DC_U3PORTPM_U2IEN);
+ FIELD_MODIFY(CRG_U3DC_U3PORTPM_U2TMOUT_MASK, &tmp, 0);
+
+ writel(tmp, &uccr->u3portpmsc);
+
+ crg_udc->feature_u1_enable = 0;
+ crg_udc->feature_u2_enable = 0;
+}
+
+/* HALTED endpoints may hold requests for CLEAR_FEATURE(ENDPOINT_HALT).
+ * STOPPED endpoints may accept a requeue only while ep_dequeue() has
+ * confirmed STOP_EP and is giving a cancelled request back.
+ * Neither state may publish new TRBs until RUNNING again.
+ * Called with udc_lock held.
+ */
+static bool crg_udc_ep_can_queue(struct crg_udc_ep *ep)
+{
+ if (ep->nuking)
+ return false;
+
+ if (ep->ep_state == EP_STATE_RUNNING)
+ return true;
+
+ if (ep->dci == 0)
+ return false;
+
+ if (ep->ep_state == EP_STATE_HALTED)
+ return true;
+
+ return ep->ep_state == EP_STATE_STOPPED && ep->dequeue_paused;
+}
+
+static int crg_udc_ep_dequeue(struct usb_ep *_ep, struct usb_request *_req)
+{
+ struct crg_udc_ep *udc_ep_ptr;
+ struct crg_gadget_dev *crg_udc;
+ struct crg_uccr __iomem *uccr;
+ struct crg_udc_request *udc_req = NULL;
+ struct crg_udc_request *iter;
+ struct crg_ep_cx *p_ep_cx;
+ struct crg_transfer_trb *pause_pt;
+ enum crg_ep_state old_ep_state;
+ u32 tmp, deq_pt_lo, deq_pt_hi;
+ dma_addr_t dq_pt_addr;
+ u8 dci;
+ u32 generation;
+ unsigned long flags;
+ const struct usb_endpoint_descriptor *old_desc;
+ u32 event_generation;
+ int ret;
+
+ if (!_ep || !_req)
+ return -EINVAL;
+
+ udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+ crg_udc = udc_ep_ptr->crg_udc;
+ uccr = crg_udc->uccr;
+ dci = udc_ep_ptr->dci;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ list_for_each_entry(iter, &udc_ep_ptr->queue, queue) {
+ if (&iter->usb_req == _req) {
+ udc_req = iter;
+ break;
+ }
+ }
+
+ if (!udc_req) {
+ ret = -EINVAL;
+ goto out;
+ }
+ if (crg_udc->shutting_down) {
+ ret = -ESHUTDOWN;
+ goto out;
+ }
+ if (crg_udc->resetting || crg_udc->controller_recovering ||
+ crg_udc->gadget_stopping || crg_udc->endpoint_recovery_pending ||
+ udc_ep_ptr->nuking || udc_ep_ptr->ep_state == EP_STATE_DISABLED) {
+ /* The pending teardown will give back this queued request. */
+ ret = 0;
+ goto out;
+ }
+ if (dci == 0) {
+ /* The abort schedules recovery on hardware failure. */
+ if (crg_udc->port_reset_pending || crg_udc->bus_reset_pending ||
+ crg_udc_abort_ep0_for_setup(crg_udc) == -ESHUTDOWN)
+ crg_udc_schedule_endpoint_recovery(crg_udc);
+ ret = 0;
+ goto out;
+ }
+ if (crg_udc->port_reset_pending || crg_udc->bus_reset_pending) {
+ ret = 0;
+ goto out;
+ }
+ if (udc_ep_ptr->halt_failed) {
+ ret = -EIO;
+ goto recover;
+ }
+
+ old_ep_state = udc_ep_ptr->ep_state;
+ old_desc = udc_ep_ptr->desc;
+ event_generation = crg_udc->udc_event[0].generation;
+ generation = udc_ep_ptr->generation;
+
+ if (!udc_req->first_trb) {
+ /* This request was queued during HALT / STOP; no live TRB. */
+ crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return 0;
+ }
+
+ /*
+ * RUN=0 quiesces endpoint DMA but prevents STOP_EP / SET_TR_DQPTR.
+ * Give the request back without touching the stopped hardware ring.
+ * Force a full controller rebuild before RUN can be asserted again.
+ */
+ if (!(readl(&uccr->control) & CRG_U3DC_CTRL_RUN)) {
+ ret = readl_poll_timeout_atomic(&uccr->ep_running, tmp,
+ !(tmp & BIT(dci)), 10, 10000);
+ if (ret)
+ goto recover;
+ crg_udc->stopped_needs_reset = true;
+ crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return 0;
+ }
+
+ /* A stopped ring may still contain published requests. */
+ if (old_ep_state == EP_STATE_STOPPED) {
+ ret = -EBUSY;
+ goto recover;
+ }
+
+ if (old_ep_state == EP_STATE_RUNNING) {
+ ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP,
+ BIT(dci), 0);
+
+ if (ret)
+ goto recover;
+ ret = readl_poll_timeout_atomic(&uccr->ep_running, tmp,
+ !(tmp & BIT(dci)), 10, 10000);
+ if (ret)
+ goto recover;
+ udc_ep_ptr->ep_state = EP_STATE_STOPPED;
+ udc_ep_ptr->dequeue_paused = true;
+ }
+
+ p_ep_cx = &crg_udc->p_epcx[dci - 2];
+ deq_pt_lo = le32_to_cpu(p_ep_cx->dw2) & EP_CX_TR_DQPT_LO_MASK;
+ deq_pt_hi = le32_to_cpu(p_ep_cx->dw3);
+ dq_pt_addr = (u64)deq_pt_lo | ((u64)deq_pt_hi << 32);
+ pause_pt = crg_udc_tran_trb_dma_to_virt(udc_ep_ptr, dq_pt_addr);
+ if (!pause_pt) {
+ ret = -EIO;
+ goto recover;
+ }
+
+ if (crg_udc_is_pointer_less_than(pause_pt, udc_req->first_trb,
+ udc_ep_ptr)) {
+ crg_udc_squeeze_xfer_ring(udc_ep_ptr, udc_req);
+ /* Give back the cancelled request BEFORE scheduling the tail:
+ * its callback may add/remove requests, and dropping udc_lock
+ * while traversing the old list is unsafe.
+ */
+ crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+ } else if (udc_req->last_trb &&
+ crg_udc_is_pointer_less_than(udc_req->last_trb, pause_pt,
+ udc_ep_ptr)) {
+ crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+ } else {
+ struct crg_udc_request *next_req = NULL;
+
+ udc_req->usb_req.actual =
+ crg_udc_actual_before(udc_ep_ptr, udc_req, pause_pt);
+
+ if (!list_is_last(&udc_req->queue, &udc_ep_ptr->queue))
+ next_req = list_entry(udc_req->queue.next,
+ struct crg_udc_request, queue);
+
+ if (next_req && next_req->first_trb)
+ udc_ep_ptr->deq_pt = next_req->first_trb;
+ else
+ udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
+
+ ret = crg_udc_epcx_update_dqptr(udc_ep_ptr);
+ if (ret)
+ goto recover;
+ udc_ep_ptr->tran_ring_full = false;
+ crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+ }
+
+ /* Stop accepting deferred queues before resuming the endpoint. */
+ if (udc_ep_ptr->generation == generation &&
+ udc_ep_ptr->desc == old_desc)
+ udc_ep_ptr->dequeue_paused = false;
+
+ if (old_ep_state == EP_STATE_RUNNING && !crg_udc->resetting &&
+ !crg_udc->port_reset_pending &&
+ !crg_udc->endpoint_recovery_pending &&
+ !udc_ep_ptr->halt_failed &&
+ udc_ep_ptr->desc == old_desc &&
+ event_generation == crg_udc->udc_event[0].generation &&
+ udc_ep_ptr->generation == generation &&
+ udc_ep_ptr->ep_state == EP_STATE_STOPPED) {
+ udc_ep_ptr->ep_state = EP_STATE_RUNNING;
+ udc_ep_ptr->tran_ring_full = false;
+ if (crg_udc_queue_pending_trbs(udc_ep_ptr))
+ udc_ep_ptr->tran_ring_full = true;
+ /* Queue processing can give back requests and drop udc_lock. */
+ if (!crg_udc->resetting && !crg_udc->port_reset_pending &&
+ !crg_udc->endpoint_recovery_pending &&
+ !udc_ep_ptr->halt_failed && udc_ep_ptr->desc == old_desc &&
+ udc_ep_ptr->generation == generation &&
+ udc_ep_ptr->ep_state == EP_STATE_RUNNING)
+ crg_udc_knock_doorbell(crg_udc, dci);
+ }
+
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return 0;
+
+recover:
+ /*
+ * A gadget may hold its own lock across dequeue. Leave requests
+ * mapped and let the worker reset the controller and give them back.
+ */
+ dev_err(crg_udc->dev, "EP%d dequeue failed (%d); scheduling recovery\n",
+ dci, ret);
+ udc_ep_ptr->dequeue_paused = false;
+ udc_ep_ptr->halt_failed = true;
+ udc_ep_ptr->ep_state = EP_STATE_STOPPED;
+ crg_udc_schedule_endpoint_recovery(crg_udc);
+ ret = 0;
+out:
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return ret;
+}
+
+/* endpoint operations */
+
+static int crg_udc_ep_disable(struct usb_ep *ep)
+{
+ struct crg_udc_ep *udc_ep;
+ struct crg_gadget_dev *crg_udc;
+ struct crg_ep_cx *p_ep_cx;
+ struct crg_uccr __iomem *uccr;
+ unsigned long flags;
+ u32 running;
+ int ret;
+
+ if (!ep)
+ return -EINVAL;
+ udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+ crg_udc = udc_ep->crg_udc;
+ if (udc_ep->dci == 0)
+ return 0;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ /* A completion callback must not restart disable during giveback. */
+ if (udc_ep->nuking) {
+ ret = -EBUSY;
+ goto out;
+ }
+ if (udc_ep->configuring) {
+ ret = -EBUSY;
+ goto out;
+ }
+ if (udc_ep->ep_state == EP_STATE_DISABLED) {
+ /*
+ * A reset can mark this endpoint disabled before reaching its
+ * queue. A callback from another endpoint may disable it now.
+ * Give back its requests before returning success.
+ */
+ crg_udc_ep_nuke(udc_ep, -ESHUTDOWN);
+ udc_ep->desc = NULL;
+ udc_ep->comp_desc = NULL;
+ udc_ep->usb_ep.enabled = false;
+ ret = 0;
+ goto out;
+ }
+ if (crg_udc->resetting || crg_udc->controller_recovering) {
+ ret = -ESHUTDOWN;
+ goto out;
+ }
+
+ uccr = crg_udc->uccr;
+ p_ep_cx = &crg_udc->p_epcx[udc_ep->dci - 2];
+ /* The disable attempt owns this endpoint until DMA is quiescent. */
+ udc_ep->ep_state = EP_STATE_STOPPED;
+ udc_ep->dequeue_paused = false;
+ writel(BIT(udc_ep->dci), &uccr->ep_enable);
+ /*
+ * Wait once for each hardware state, with a bounded timeout. DWC3's
+ * delayed-stop retry uses a controller-specific SETUP event which CRG
+ * does not currently have. An asynchronous retry must not outlive
+ * usb_ep_disable() while Gadget owns the original request buffers.
+ */
+ ret = readl_poll_timeout_atomic(&uccr->ep_enable, running,
+ !(running & BIT(udc_ep->dci)), 10, 10000);
+ if (ret)
+ goto disable_failed;
+ /* A cleared enable bit alone does not prove DMA is idle. */
+ ret = readl_poll_timeout_atomic(&uccr->ep_running, running,
+ !(running & BIT(udc_ep->dci)), 10, 10000);
+ if (ret)
+ goto disable_failed;
+
+ /* Endpoint DMA is stopped: mappings can now be released. */
+ udc_ep->disable_failed = false;
+ udc_ep->halt_failed = false;
+ udc_ep->dequeue_paused = false;
+ udc_ep->generation++;
+ udc_ep->desc = NULL;
+ udc_ep->comp_desc = NULL;
+ udc_ep->ep_state = EP_STATE_DISABLED;
+ udc_ep->tran_ring_full = false;
+ if (crg_udc->num_enabled_eps)
+ crg_udc->num_enabled_eps--;
+ memset(p_ep_cx, 0, sizeof(*p_ep_cx));
+
+ if (!crg_udc->num_enabled_eps &&
+ crg_udc->device_state == USB_STATE_CONFIGURED)
+ crg_udc->device_state = USB_STATE_ADDRESS;
+
+ /* crg_udc_req_done() drops the lock during giveback. All endpoint
+ * state has to be finalized before that can happen.
+ */
+ crg_udc_ep_nuke(udc_ep, -ESHUTDOWN);
+ udc_ep->usb_ep.enabled = false;
+ ret = 0;
+ goto out;
+
+disable_failed:
+ /*
+ * usb_ep_disable() may be called with IRQs disabled, and a gadget may
+ * release its requests as soon as this call returns. A background
+ * reset alone cannot make the outstanding DMA buffers safe to free.
+ * Block the endpoint first, then quiesce the entire controller now.
+ */
+ udc_ep->disable_failed = true;
+ udc_ep->halt_failed = true;
+ dev_err(crg_udc->dev,
+ "EP%d disable failed (%d); attempting controller reset\n",
+ udc_ep->dci, ret);
+
+ ret = crg_udc_recover_endpoints(crg_udc, &flags);
+ if (ret && udc_ep->ep_state == EP_STATE_DISABLED && !udc_ep->nuking &&
+ list_empty(&udc_ep->queue))
+ udc_ep->usb_ep.enabled = false;
+
+out:
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return ret;
+}
+
+static int crg_udc_ep_enable(struct usb_ep *ep,
+ const struct usb_endpoint_descriptor *desc)
+{
+ struct crg_udc_ep *udc_ep;
+ struct crg_gadget_dev *crg_udc;
+ size_t len;
+ unsigned long flags;
+ u32 ring_size, param0;
+ int ret;
+
+ if (!ep || !desc || desc->bDescriptorType != USB_DT_ENDPOINT)
+ return -EINVAL;
+ udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+ if (udc_ep->dci == 0)
+ return 0;
+ crg_udc = udc_ep->crg_udc;
+
+ if (usb_endpoint_xfer_bulk(desc))
+ ring_size = CRG_UDC_BULK_EP_TD_RING_SIZE;
+ else if (usb_endpoint_xfer_isoc(desc))
+ ring_size = CRG_UDC_ISOC_EP_TD_RING_SIZE;
+ else if (usb_endpoint_xfer_int(desc))
+ ring_size = CRG_UDC_INT_EP_TD_RING_SIZE;
+ else
+ return -EINVAL;
+ len = ring_size * sizeof(struct crg_transfer_trb);
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (!crg_udc->gadget_driver || crg_udc->controller_faulted ||
+ crg_udc->resetting || crg_udc->gadget_stopping ||
+ crg_udc->controller_recovering ||
+ crg_udc->endpoint_recovery_pending ||
+ crg_udc->shutting_down || crg_udc->bus_reset_pending ||
+ crg_udc->port_reset_pending) {
+ ret = -ESHUTDOWN;
+ goto out;
+ }
+ if (udc_ep->nuking || udc_ep->configuring ||
+ udc_ep->disable_failed ||
+ udc_ep->ep_state != EP_STATE_DISABLED) {
+ ret = -EBUSY;
+ goto out;
+ }
+ udc_ep->configuring = true;
+ udc_ep->desc = desc;
+ udc_ep->comp_desc = ep->comp_desc;
+ ret = crg_udc_validate_ep_desc(udc_ep, desc);
+ if (ret)
+ goto err;
+
+ /* Rings are allocated at probe, outside the atomic enable path. */
+ if (!udc_ep->tran_ring_info.cpu_addr ||
+ udc_ep->tran_ring_info.len < len) {
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ udc_ep->first_trb = udc_ep->tran_ring_info.cpu_addr;
+ udc_ep->last_trb = udc_ep->first_trb + ring_size - 1;
+ memset(udc_ep->first_trb, 0, len);
+ dma_wmb();
+ crg_udc_setup_link_trb(udc_ep->last_trb, true,
+ udc_ep->tran_ring_info.dma_addr);
+ udc_ep->enq_pt = udc_ep->first_trb;
+ udc_ep->deq_pt = udc_ep->first_trb;
+ udc_ep->pcs = 1;
+ udc_ep->tran_ring_full = false;
+ ret = crg_udc_epcx_setup(udc_ep);
+ if (ret)
+ goto err;
+ param0 = BIT(udc_ep->dci);
+ ret = crg_udc_issue_command(crg_udc, CRG_CMD_CONFIG_EP, param0, 0);
+ if (ret)
+ goto err;
+
+ udc_ep->generation++;
+ /* usb_ep.maxpacket must reflect the descriptor actually configured. */
+ udc_ep->usb_ep.maxpacket = usb_endpoint_maxp(desc);
+ udc_ep->wedge = false;
+ udc_ep->halt_failed = false;
+ udc_ep->disable_failed = false;
+ udc_ep->dequeue_paused = false;
+ udc_ep->usb_ep.enabled = 1;
+ udc_ep->ep_state = EP_STATE_RUNNING;
+ udc_ep->configuring = false;
+ crg_udc->num_enabled_eps++;
+ if (crg_udc->device_state == USB_STATE_ADDRESS)
+ crg_udc->device_state = USB_STATE_CONFIGURED;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return 0;
+
+err:
+ udc_ep->desc = NULL;
+ udc_ep->comp_desc = NULL;
+ udc_ep->configuring = false;
+ /* Keep the preallocated ring for a later retry. */
+ udc_ep->ep_state = EP_STATE_DISABLED;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return ret;
+out:
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return ret;
+}
+
+void crg_udc_ep_nuke(struct crg_udc_ep *udc_ep, int status)
+{
+ struct crg_udc_request *req;
+
+ /*
+ * A giveback may re-enter ep_nuke() via disable or reset. The outer
+ * drain still owns all originally queued requests, so do not nest it.
+ * Also reject new queue/enable operations until the drain is finished.
+ */
+ if (udc_ep->nuking)
+ return;
+
+ udc_ep->nuking = true;
+ while (!list_empty(&udc_ep->queue)) {
+ req = list_first_entry(&udc_ep->queue,
+ struct crg_udc_request, queue);
+ crg_udc_req_done(udc_ep, req, status);
+ }
+ udc_ep->nuking = false;
+ if (udc_ep->clear_enabled_on_nuke) {
+ udc_ep->clear_enabled_on_nuke = false;
+ udc_ep->usb_ep.enabled = false;
+ }
+}
+
+/* The CPU page list stays intact even if dma_map_sgtable() merges entries. */
+static size_t crg_udc_copy_bounce_pages(struct crg_udc_bounce *bounce,
+ struct usb_request *req, size_t len)
+{
+ bool to_bounce = bounce->dir == DMA_TO_DEVICE;
+ struct sg_mapping_iter miter;
+ struct page *page;
+ unsigned int sg_flags = to_bounce ? SG_MITER_FROM_SG : SG_MITER_TO_SG;
+ size_t done = 0;
+
+ if (req->num_sgs)
+ sg_miter_start(&miter, req->sg, req->num_sgs,
+ SG_MITER_ATOMIC | sg_flags);
+
+ list_for_each_entry(page, &bounce->pages, lru) {
+ size_t count = min_t(size_t, PAGE_SIZE, len - done);
+ size_t offset = 0;
+ void *vaddr;
+
+ if (!count)
+ break;
+
+ vaddr = kmap_local_page(page);
+ if (req->num_sgs) {
+ while (offset < count && sg_miter_next(&miter)) {
+ size_t chunk = min(miter.length, count - offset);
+
+ if (to_bounce)
+ memcpy(vaddr + offset, miter.addr, chunk);
+ else
+ memcpy(miter.addr, vaddr + offset, chunk);
+ miter.consumed = chunk;
+ offset += chunk;
+ }
+ sg_miter_stop(&miter);
+ } else {
+ if (to_bounce)
+ memcpy(vaddr, req->buf + done, count);
+ else
+ memcpy(req->buf + done, vaddr, count);
+ offset = count;
+ }
+ kunmap_local(vaddr);
+
+ done += offset;
+ if (offset != count)
+ break;
+ }
+ return done;
+}
+
+static int crg_udc_bounce_map_pages(struct device *dev,
+ struct crg_udc_bounce *bounce,
+ struct usb_request *req, gfp_t gfp)
+{
+ unsigned int nr_pages = DIV_ROUND_UP_ULL(req->length, PAGE_SIZE);
+ size_t remaining = req->length;
+ struct scatterlist *sg;
+ unsigned int i;
+ int ret;
+
+ if (bounce->dir == DMA_FROM_DEVICE)
+ gfp |= __GFP_ZERO;
+ ret = sg_alloc_table(&bounce->sgt, nr_pages, gfp);
+ if (ret)
+ return ret;
+
+ for_each_sgtable_sg(&bounce->sgt, sg, i) {
+ size_t len = min_t(size_t, remaining, PAGE_SIZE);
+ struct page *page = alloc_page(gfp);
+
+ if (!page)
+ return -ENOMEM;
+
+ list_add_tail(&page->lru, &bounce->pages);
+ sg_set_page(sg, page, len, 0);
+ remaining -= len;
+ }
+
+ if (bounce->dir == DMA_TO_DEVICE &&
+ crg_udc_copy_bounce_pages(bounce, req, req->length) != req->length)
+ return -EINVAL;
+
+ return dma_map_sgtable(dev, &bounce->sgt, bounce->dir, 0);
+}
+
+static int crg_udc_bounce_map_ep0(struct device *dev,
+ struct crg_udc_bounce *bounce,
+ struct usb_request *req, gfp_t gfp)
+{
+ bounce->buf = kzalloc(req->length, gfp);
+ if (!bounce->buf)
+ return -ENOMEM;
+
+ if (bounce->dir == DMA_TO_DEVICE)
+ memcpy(bounce->buf, req->buf, req->length);
+
+ bounce->dma = dma_map_single(dev, bounce->buf, bounce->len, bounce->dir);
+ return dma_mapping_error(dev, bounce->dma) ? -EIO : 0;
+}
+
+int crg_udc_bounce_map_request(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req,
+ enum dma_data_direction dir, gfp_t gfp)
+{
+ struct device *dev = udc_ep->crg_udc->dev;
+ struct usb_request *req = &udc_req->usb_req;
+ struct crg_udc_bounce *bounce;
+ int nents, ret;
+
+ if (!req->length)
+ return 0;
+ if (udc_req->bounce)
+ return -EBUSY;
+
+ /* CPU copies require buffers the UDC is allowed to access. */
+ if (req->sg_was_mapped || req->dma_mapped)
+ return -EOPNOTSUPP;
+ if (req->num_sgs) {
+ if (!req->sg)
+ return -EINVAL;
+ nents = sg_nents_for_len(req->sg, req->length);
+ if (nents < 0 || nents > req->num_sgs)
+ return -EINVAL;
+ } else if (!req->buf) {
+ return -EINVAL;
+ }
+
+ bounce = kzalloc(sizeof(*bounce), gfp);
+ if (!bounce)
+ return -ENOMEM;
+ INIT_LIST_HEAD(&bounce->pages);
+ bounce->dir = dir;
+ bounce->len = req->length;
+ udc_req->bounce = bounce;
+
+ if (udc_ep->dci)
+ ret = crg_udc_bounce_map_pages(dev, bounce, req, gfp);
+ else
+ ret = crg_udc_bounce_map_ep0(dev, bounce, req, gfp);
+
+ if (ret) {
+ crg_udc_unmap_request(udc_ep, udc_req, 0);
+ return ret;
+ }
+
+ bounce->mapped = true;
+ return 0;
+}
+
+static int crg_udc_ep_queue(struct usb_ep *_ep, struct usb_request *_req,
+ gfp_t gfp_flags)
+{
+ struct crg_udc_request *udc_req_ptr;
+ struct crg_udc_ep *udc_ep_ptr;
+ struct crg_gadget_dev *crg_udc;
+ enum dma_data_direction dma_data_dir;
+ const struct usb_endpoint_descriptor *desc;
+ unsigned long flags;
+ int status;
+ bool is_sg;
+ u8 setup_tag = 0;
+ enum crg_setup_status setup_status = WAIT_FOR_SETUP;
+ u32 generation = 0;
+
+ if (!_ep || !_req)
+ return -EINVAL;
+
+ udc_req_ptr = container_of(_req, struct crg_udc_request, usb_req);
+ udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+ crg_udc = udc_ep_ptr->crg_udc;
+ is_sg = _req->num_sgs != 0 && _req->length != 0;
+
+ if (is_sg && !_req->sg)
+ return -EINVAL;
+ if (_req->stream_id)
+ return -EOPNOTSUPP;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ /*
+ * A completion callback for an aborted EP0 transfer may requeue the
+ * same request. Reject EP0 requests while the previous control
+ * transfer is being aborted or EP0 is in an error state. EP0 becomes
+ * available again only after the new SETUP has been accepted and
+ * the EP0 transfer ring has been successfully reinitialized.
+ */
+ if (udc_ep_ptr->dci == 0 && crg_udc->ep0_state != EP0_STATE_IDLE) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -ECONNRESET;
+ }
+ if (!udc_ep_ptr->first_trb || !_req->complete ||
+ (_req->length && !is_sg && !_req->buf) ||
+ !list_empty(&udc_req_ptr->queue)) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EINVAL;
+ }
+ /*
+ * pullup(false) stops RUN without necessarily changing the EP
+ * generation. Do not accept requests into a stopped USB session.
+ */
+ if (!crg_udc_gadget_ready(crg_udc) ||
+ crg_udc->bus_reset_pending || crg_udc->port_reset_pending) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -ESHUTDOWN;
+ }
+ if (!udc_ep_ptr->desc) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EINVAL;
+ }
+ desc = udc_ep_ptr->desc;
+ if (udc_ep_ptr->halt_failed ||
+ !crg_udc_ep_can_queue(udc_ep_ptr) ||
+ udc_ep_ptr->configuring) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -ESHUTDOWN;
+ }
+ setup_tag = crg_udc->setup_tag;
+ if (udc_ep_ptr->dci == 0)
+ setup_status = crg_udc->setup_status;
+ generation = udc_ep_ptr->generation;
+
+ if (usb_endpoint_xfer_control(udc_ep_ptr->desc) && is_sg) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EOPNOTSUPP;
+ }
+
+ if (usb_endpoint_xfer_control(udc_ep_ptr->desc) && !_req->length) {
+ if (crg_udc->setup_status == SETUP_PKT_PROCESS_IN_PROGRESS) {
+ /* Only a control request without DATA needs STATUS now. */
+ crg_udc_clear_req_container(udc_req_ptr);
+ crg_udc->setup_status = STATUS_STAGE_XFER;
+ status = crg_udc_build_ep0_status(udc_ep_ptr, false,
+ -EINPROGRESS, udc_req_ptr, 0, 0);
+ if (status)
+ crg_udc->setup_status = SETUP_PKT_PROCESS_IN_PROGRESS;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return status;
+ }
+ /* A zero-length DATA stage still needs a DATA-stage TRB. */
+ if (crg_udc->setup_status != DATA_STAGE_XFER &&
+ crg_udc->setup_status != DATA_STAGE_RECV) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EINVAL;
+ }
+ }
+
+ if (usb_endpoint_xfer_control(udc_ep_ptr->desc)) {
+ if (crg_udc->setup_status == DATA_STAGE_XFER ||
+ crg_udc->setup_status == STATUS_STAGE_XFER) {
+ dma_data_dir = DMA_TO_DEVICE;
+ } else if (crg_udc->setup_status == DATA_STAGE_RECV ||
+ crg_udc->setup_status == STATUS_STAGE_RECV) {
+ dma_data_dir = DMA_FROM_DEVICE;
+ } else {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EINVAL;
+ }
+ } else {
+ dma_data_dir = usb_endpoint_dir_in(udc_ep_ptr->desc) ?
+ DMA_TO_DEVICE :
+ DMA_FROM_DEVICE;
+ }
+ crg_udc->queueing++;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ crg_udc_clear_req_container(udc_req_ptr);
+
+ status = crg_udc_bounce_map_request(udc_ep_ptr, udc_req_ptr,
+ dma_data_dir, gfp_flags);
+ if (status)
+ goto out_queueing;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (udc_ep_ptr->desc != desc || !udc_req_ptr->usb_req.complete ||
+ !list_empty(&udc_req_ptr->queue) ||
+ !crg_udc_gadget_ready(crg_udc) ||
+ crg_udc->bus_reset_pending || crg_udc->port_reset_pending ||
+ udc_ep_ptr->configuring || udc_ep_ptr->halt_failed ||
+ !crg_udc_ep_can_queue(udc_ep_ptr) ||
+ udc_ep_ptr->generation != generation ||
+ (udc_ep_ptr->dci == 0 &&
+ (crg_udc->ep0_state != EP0_STATE_IDLE ||
+ crg_udc->setup_status != setup_status ||
+ crg_udc->setup_tag != setup_tag))) {
+ status = -ESHUTDOWN;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ goto unmap;
+ }
+
+ _req->status = -EINPROGRESS;
+ _req->actual = 0;
+
+ /* A halted or temporarily stopped endpoint may accept requests,
+ * but no TRBs may be published until it resumes.
+ */
+ if (udc_ep_ptr->ep_state != EP_STATE_RUNNING) {
+ status = 0;
+ } else if (!list_empty(&udc_ep_ptr->queue)) {
+ struct crg_udc_request *tail;
+
+ tail = list_last_entry(&udc_ep_ptr->queue,
+ struct crg_udc_request, queue);
+ status = (!tail->all_trbs_queued ||
+ udc_ep_ptr->tran_ring_full) ? 0 :
+ crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+ } else if (udc_ep_ptr->tran_ring_full) {
+ status = 0;
+ } else {
+ status = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+ }
+
+ if (!status) {
+ list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+ goto out_locked;
+ }
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+unmap:
+ crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr, 0);
+out_queueing:
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+out_locked:
+ crg_udc->queueing--;
+ if (crg_udc->gadget_stopping && !crg_udc->queueing)
+ wake_up(&crg_udc->teardown_wait);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return status;
+}
+
+static int crg_udc_ep_set_halt(struct usb_ep *_ep, int value)
+{
+ struct crg_udc_ep *udc_ep_ptr;
+ int status;
+ unsigned long flags = 0;
+ struct crg_gadget_dev *crg_udc;
+
+ if (!_ep)
+ return -EINVAL;
+
+ udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+ crg_udc = udc_ep_ptr->crg_udc;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (!udc_ep_ptr->desc) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EINVAL;
+ }
+ /*
+ * A previous HALT/STOP failure retained the queue for DMA safety.
+ * Do not reject the HALT retry merely because requests remain queued.
+ */
+ if (value && !udc_ep_ptr->halt_failed &&
+ usb_endpoint_dir_in(udc_ep_ptr->desc) &&
+ !list_empty(&udc_ep_ptr->queue)) {
+ dev_err(crg_udc->dev, "set_halt: list not empty\n");
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EAGAIN;
+ }
+
+ status = crg_udc_ep_halt(udc_ep_ptr, value, 1, &flags);
+
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ return status;
+}
+
+static int crg_udc_ep_set_wedge(struct usb_ep *_ep)
+{
+ struct crg_udc_ep *udc_ep_ptr;
+ bool was_wedged;
+ int status;
+ unsigned long flags = 0;
+ struct crg_gadget_dev *crg_udc;
+
+ if (!_ep)
+ return -EINVAL;
+
+ udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+ crg_udc = udc_ep_ptr->crg_udc;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+
+ if (!udc_ep_ptr->desc) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EINVAL;
+ }
+
+ /* Like set_halt(), do not cancel queued IN requests. A failed
+ * HALT/STOP may be retried even though its queue remains populated.
+ */
+ if (!udc_ep_ptr->halt_failed &&
+ usb_endpoint_dir_in(udc_ep_ptr->desc) &&
+ !list_empty(&udc_ep_ptr->queue)) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EAGAIN;
+ }
+
+ was_wedged = udc_ep_ptr->wedge;
+ udc_ep_ptr->wedge = true;
+ status = crg_udc_ep_halt(udc_ep_ptr, 1, 1, &flags);
+ if (status)
+ udc_ep_ptr->wedge = was_wedged;
+
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ return status;
+}
+
+static int crg_udc_epcx_setup(struct crg_udc_ep *udc_ep)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+ const struct usb_endpoint_descriptor *desc = udc_ep->desc;
+ const struct usb_ss_ep_comp_descriptor *comp_desc = udc_ep->comp_desc;
+ u8 dci = udc_ep->dci;
+ struct crg_ep_cx *epcx = &crg_udc->p_epcx[dci - 2];
+ enum crg_ep_type ep_type;
+ u16 maxburst = 0;
+ u8 maxstreams = 0;
+ u16 maxsize;
+ u32 dw;
+
+ /* corigine gadget dir should be opposite to host dir */
+ if (usb_endpoint_dir_out(desc))
+ ep_type = usb_endpoint_type(desc) + EP_TYPE_INVALID2;
+ else
+ ep_type = usb_endpoint_type(desc);
+
+ maxsize = usb_endpoint_maxp(desc) & 0x07ff;
+
+ if (crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+ if (!comp_desc)
+ return -EINVAL;
+ maxburst = comp_desc->bMaxBurst;
+
+ if (usb_endpoint_xfer_bulk(udc_ep->desc))
+ maxstreams = comp_desc->bmAttributes & 0x1f;
+
+ } else if ((crg_udc->gadget.speed == USB_SPEED_HIGH ||
+ crg_udc->gadget.speed == USB_SPEED_FULL) &&
+ (usb_endpoint_xfer_int(udc_ep->desc) ||
+ usb_endpoint_xfer_isoc(udc_ep->desc))) {
+ if (crg_udc->gadget.speed == USB_SPEED_HIGH)
+ maxburst = usb_endpoint_maxp_mult(desc) - 1;
+ if (maxburst == 0x3) {
+ dev_err(crg_udc->dev, "invalid maxburst\n");
+ maxburst = 0x2;
+ }
+ }
+
+ /* fill ep_dw0 */
+ dw = 0;
+ FIELD_MODIFY(EP_CX_LOGICAL_EP_NUM_MASK, &dw, udc_ep->dci / 2);
+ FIELD_MODIFY(EP_CX_INTERVAL_MASK, &dw, desc->bInterval);
+ if (maxstreams) {
+ dev_err(crg_udc->dev, "maxstream=%d is not expected\n",
+ maxstreams);
+ }
+ epcx->dw0 = cpu_to_le32(dw);
+
+ /* fill ep_dw1 */
+ dw = 0;
+ FIELD_MODIFY(EP_CX_EP_TYPE_MASK, &dw, ep_type);
+ FIELD_MODIFY(EP_CX_MAX_PACKET_SIZE_MASK, &dw, maxsize);
+ FIELD_MODIFY(EP_CX_MAX_BURST_SIZE_MASK, &dw, maxburst);
+ epcx->dw1 = cpu_to_le32(dw);
+
+ /* fill ep_dw2 */
+ dw = lower_32_bits(udc_ep->tran_ring_info.dma_addr) &
+ EP_CX_TR_DQPT_LO_MASK;
+ FIELD_MODIFY(EP_CX_DEQ_CYC_STATE_MASK, &dw, udc_ep->pcs);
+ epcx->dw2 = cpu_to_le32(dw);
+
+ /* fill ep_dw3 */
+ dw = upper_32_bits(udc_ep->tran_ring_info.dma_addr);
+ epcx->dw3 = cpu_to_le32(dw);
+ /* Ensure that epcx is updated */
+ dma_wmb();
+ return 0;
+}
+
+static int crg_udc_epcx_update_dqptr(struct crg_udc_ep *udc_ep)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+ u8 dci = udc_ep->dci;
+ struct crg_ep_cx *epcx;
+ u32 dw;
+ dma_addr_t dqptaddr;
+ u32 cmd_param0;
+
+ if (dci < 2 || dci >= CRG_NUM_EP_CX) {
+ dev_err(crg_udc->dev, "invalid DCI %u for dequeue pointer\n", dci);
+ return -EINVAL;
+ }
+
+ epcx = &crg_udc->p_epcx[dci - 2];
+ dqptaddr = crg_udc_tran_trb_virt_to_dma(udc_ep, udc_ep->deq_pt);
+
+ if (dqptaddr == DMA_MAPPING_ERROR)
+ return -EINVAL;
+
+ /* fill ep_dw2 */
+ dw = lower_32_bits(dqptaddr) & EP_CX_TR_DQPT_LO_MASK;
+ /* The consumer cycle can lag the producer by one ring wrap. */
+ FIELD_MODIFY(EP_CX_DEQ_CYC_STATE_MASK, &dw,
+ udc_ep->deq_pt > udc_ep->enq_pt ?
+ udc_ep->pcs ^ 1 : udc_ep->pcs);
+ epcx->dw2 = cpu_to_le32(dw);
+
+ /* fill ep_dw3 */
+ dw = upper_32_bits(dqptaddr);
+ epcx->dw3 = cpu_to_le32(dw);
+
+ cmd_param0 = BIT(udc_ep->dci);
+ /* Ensure that dqptr is updated */
+ dma_wmb();
+
+ return crg_udc_issue_command(crg_udc, CRG_CMD_SET_TR_DQPTR, cmd_param0, 0);
+}
+
+void crg_udc_free_request(struct usb_ep *_ep, struct usb_request *_req)
+{
+ struct crg_udc_request *udc_req_ptr;
+
+ if (!_ep || !_req)
+ return;
+
+ udc_req_ptr = container_of(_req, struct crg_udc_request, usb_req);
+
+ trace_crg_udc_free_request(container_of(_ep, struct crg_udc_ep, usb_ep),
+ udc_req_ptr);
+
+ /* A queued request is returned through its completion callback. */
+ if (!list_empty(&udc_req_ptr->queue) || udc_req_ptr->bounce) {
+ struct crg_udc_ep *ep = container_of(_ep, struct crg_udc_ep, usb_ep);
+
+ dev_err_ratelimited(ep->crg_udc->dev, "freeing active request on %s\n",
+ _ep->name);
+ return;
+ }
+
+ kfree(udc_req_ptr);
+}
+
+static int crg_udc_queue_trbs(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr, bool b_isoc,
+ u32 xfer_ring_size, u32 num_trbs_needed,
+ u64 buffer_length)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+ struct crg_transfer_trb *p_xfer_ring = udc_ep_ptr->first_trb;
+ struct crg_transfer_trb *enq_pt = udc_ep_ptr->enq_pt;
+ struct usb_request *usb_req = &udc_req_ptr->usb_req;
+ /* Plan the whole TD before touching hardware-visible TRBs. */
+ struct {
+ dma_addr_t dma;
+ u32 len;
+ } plan[32];
+ struct crg_udc_request sg_cursor = {
+ .bounce = udc_req_ptr->bounce,
+ .sg = udc_req_ptr->sg,
+ .sg_index = udc_req_ptr->sg_index,
+ .sg_offset = udc_req_ptr->sg_offset,
+ };
+ u64 remaining = buffer_length;
+ u32 num_trbs_ava, count, i;
+ u32 td_trbs, td_size, td_bytes, td_done = 0;
+ u32 td_start_offset = usb_req->length - buffer_length;
+ u32 j = 1;
+ u32 intr_rate = b_isoc ? CRG_UDC_ISOC_EP_INTERRUPT_RATE :
+ CRG_UDC_BULK_EP_INTERRUPT_RATE;
+ bool full_td = true;
+ bool need_zlp = false;
+ bool is_sg = udc_req_ptr->bounce && udc_req_ptr->bounce->sgt.nents;
+ u8 original_pcs = udc_ep_ptr->pcs;
+
+ if (!udc_ep_ptr->usb_ep.maxpacket)
+ return -EINVAL;
+
+ if (!b_isoc && usb_req->zero && usb_req->length &&
+ !(usb_req->length % udc_ep_ptr->usb_ep.maxpacket))
+ need_zlp = true;
+
+ if (!num_trbs_needed)
+ return -EINVAL;
+
+ /* TD_SIZE is five bits. A request may span multiple 32-TRB TDs. */
+ td_trbs = min_t(u32, num_trbs_needed, ARRAY_SIZE(plan));
+ num_trbs_ava = crg_udc_room_on_ring(crg_udc, xfer_ring_size,
+ p_xfer_ring, udc_ep_ptr->enq_pt,
+ udc_ep_ptr->deq_pt);
+
+ if (num_trbs_ava < td_trbs) {
+ if (b_isoc)
+ return -EBUSY;
+ trace_crg_udc_no_room_on_ring(
+ "deferring TRBs to next doorbell");
+ count = num_trbs_ava;
+ full_td = false;
+ need_zlp = false;
+ } else {
+ count = td_trbs;
+ }
+
+ if (!count) {
+ /* There really is no room, not a malformed SG list. */
+ udc_ep_ptr->tran_ring_full = true;
+ return -ENOSPC;
+ }
+
+ /* Preflight all buffers, including every SG boundary. No request SG
+ * state or hardware ring state changes if any entry is invalid.
+ */
+ if (is_sg && !sg_cursor.sg && crg_udc_sg_init(&sg_cursor))
+ return -EINVAL;
+
+ for (i = 0; i < count; i++) {
+ dma_addr_t dma;
+ u32 len, sg_len = 0;
+
+ if (is_sg) {
+ if (!crg_udc_sg_current(&sg_cursor, &dma, &sg_len) ||
+ !sg_len || dma_mapping_error(crg_udc->dev, dma))
+ return -EINVAL;
+ len = min_t(u64, remaining,
+ min_t(u32, sg_len,
+ crg_udc_trb_chunk_len(dma, remaining)));
+ } else {
+ if (remaining)
+ return -EINVAL;
+ dma = 0;
+ len = crg_udc_trb_chunk_len(dma, remaining);
+ }
+
+ if (!len && remaining)
+ return -EINVAL;
+ if (is_sg && !crg_udc_sg_advance(&sg_cursor, len))
+ return -EINVAL;
+
+ plan[i].dma = dma;
+ plan[i].len = len;
+ remaining -= len;
+ }
+
+ /* All validation is complete. The following loop cannot fail; the
+ * first TRB's cycle bit is published last, after the complete TD.
+ */
+ td_bytes = buffer_length - remaining;
+ remaining = buffer_length;
+ for (i = 0; i < count; i++) {
+ u32 trb_len = plan[i].len;
+ bool chain_bit = i != count - 1;
+ bool ioc = !chain_bit;
+ bool short_pkt = usb_endpoint_dir_out(udc_ep_ptr->desc);
+
+ remaining -= trb_len;
+ td_done += trb_len;
+ td_size = crg_udc_td_size(td_bytes, td_done,
+ udc_ep_ptr->usb_ep.maxpacket, !chain_bit);
+ if (!full_td && j == intr_rate) {
+ ioc = true;
+ j = 0;
+ }
+
+ if (b_isoc) {
+ crg_udc_setup_trb(
+ enq_pt, trb_len, plan[i].dma, td_size,
+ i == 0 ? (original_pcs ^ 1) : udc_ep_ptr->pcs,
+ i == 0 ? TRB_TYPE_XFER_DATA_ISOCH :
+ TRB_TYPE_XFER_NORMAL,
+ short_pkt, chain_bit, ioc, false, 0,
+ i == 0, 0, i == 0, 0);
+ } else {
+ bool azp = ioc && !chain_bit && !remaining && need_zlp;
+
+ crg_udc_setup_trb(
+ enq_pt, trb_len, plan[i].dma, td_size,
+ i == 0 ? (original_pcs ^ 1) : udc_ep_ptr->pcs,
+ TRB_TYPE_XFER_NORMAL, short_pkt, chain_bit, ioc,
+ false, 0, false, 0, 0, azp);
+ }
+
+ trace_crg_udc_queue_trb(udc_ep_ptr, enq_pt);
+ enq_pt++;
+ j++;
+
+ if (enq_pt == udc_ep_ptr->last_trb) {
+ dma_wmb();
+ le32p_replace_bits(&enq_pt->dw3, udc_ep_ptr->pcs,
+ TRB_CYCLE_BIT_MASK);
+ udc_ep_ptr->pcs ^= 1;
+ enq_pt = udc_ep_ptr->first_trb;
+ }
+ }
+
+ dma_wmb();
+ le32p_replace_bits(&udc_ep_ptr->enq_pt->dw3, original_pcs,
+ TRB_CYCLE_BIT_MASK);
+
+ /* Commit request bookkeeping only after publishing the whole TD. */
+ udc_req_ptr->first_trb = udc_ep_ptr->enq_pt;
+ udc_req_ptr->td_start_offset = td_start_offset;
+ udc_ep_ptr->enq_pt = enq_pt;
+ udc_ep_ptr->tran_ring_full = count == num_trbs_ava;
+ udc_req_ptr->buff_len_left = remaining;
+ udc_req_ptr->td_pending = true;
+ udc_req_ptr->trbs_needed = remaining ? num_trbs_needed - count : 0;
+ if (!remaining)
+ udc_req_ptr->all_trbs_queued = true;
+ if (is_sg) {
+ udc_req_ptr->sg = sg_cursor.sg;
+ udc_req_ptr->sg_index = sg_cursor.sg_index;
+ udc_req_ptr->sg_offset = sg_cursor.sg_offset;
+ }
+
+ if (udc_ep_ptr->enq_pt == udc_ep_ptr->first_trb)
+ udc_req_ptr->last_trb = udc_ep_ptr->last_trb - 1;
+ else
+ udc_req_ptr->last_trb = udc_ep_ptr->enq_pt - 1;
+
+ return 0;
+}
+
+static int crg_udc_reinit(struct crg_gadget_dev *crg_udc, bool notify_reset)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ struct crg_udc_ep *ep;
+ u32 i, tmp;
+ int ret;
+
+ if (crg_udc->shutting_down || crg_udc->gadget_stopping)
+ return -ESHUTDOWN;
+ if (crg_udc->controller_recovering)
+ return -EBUSY;
+ crg_udc_recovery_begin(crg_udc);
+
+ /* A newer bus reset invalidates any already-running reset worker. */
+ crg_udc->bus_reset_generation++;
+ /* Invalidate EP0 mappings and SETUPs belonging to the old bus state. */
+ crg_udc->udc_ep[0].generation++;
+ crg_udc->pending_setup_valid = false;
+ crg_udc->setup_status = WAIT_FOR_SETUP;
+ crg_udc->device_state = USB_STATE_RECONNECTING;
+ crg_udc->port_reset_pending = false;
+
+ /*
+ * The controller datasheet defines EP Enabled only in bits 31:2
+ * (RW1C). Bits 1:0 are reserved and may read as nonzero (bit 0
+ * stayed set on AX3000), so neither write them nor wait for them
+ * to clear when disabling endpoints before DMA cleanup.
+ */
+ tmp = readl(&uccr->ep_enable) & GENMASK(31, 2);
+ writel(tmp, &uccr->ep_enable);
+ if (readl_poll_timeout_atomic(&uccr->ep_enable, tmp,
+ !(tmp & GENMASK(31, 2)), 10, 10000)) {
+ /* Still DMA-visible: only SWRST can safely release these TRBs. */
+ dev_err(crg_udc->dev,
+ "bus reset: EP_ENABLE did not clear (0x%x)\n", tmp);
+ crg_udc->bus_reset_pending = false;
+ crg_udc->resetting = true;
+ ret = -ETIMEDOUT;
+ goto out;
+ }
+
+ /*
+ * Disabling an endpoint does not itself prove that DMA stopped.
+ * Before unmapping any requests, verify every data endpoint is idle.
+ */
+ if (readl_poll_timeout_atomic(&uccr->ep_running, tmp,
+ !(tmp & GENMASK(CRG_NUM_EP_CX - 1, 2)),
+ 10, 10000)) {
+ dev_err(crg_udc->dev,
+ "bus reset: EP_RUNNING did not clear (0x%x)\n", tmp);
+ crg_udc->bus_reset_pending = false;
+ crg_udc->resetting = true;
+ ret = -ETIMEDOUT;
+ goto out;
+ }
+
+ /*
+ * No callbacks until every endpoint has been marked inactive. When
+ * notifying composite, keep usb_ep.enabled and descriptors intact:
+ * its ->disable() callbacks must still reach usb_ep_disable().
+ */
+ crg_udc->bus_reset_pending = notify_reset;
+ for (i = 2; i < CRG_NUM_EP_CX; i++) {
+ ep = &crg_udc->udc_ep[i];
+ ep->generation++;
+ ep->ep_state = EP_STATE_DISABLED;
+ ep->halt_failed = false;
+ ep->disable_failed = false;
+ ep->dequeue_paused = false;
+ ep->tran_ring_full = false;
+ if (!notify_reset) {
+ ep->desc = NULL;
+ ep->comp_desc = NULL;
+ }
+ }
+ crg_udc->num_enabled_eps = 0;
+
+ for (i = 2; i < CRG_NUM_EP_CX; i++)
+ crg_udc_ep_nuke(&crg_udc->udc_ep[i], -ESHUTDOWN);
+ if (!notify_reset)
+ for (i = 2; i < CRG_NUM_EP_CX; i++)
+ crg_udc_ep_clear_enabled(&crg_udc->udc_ep[i]);
+
+ /*
+ * Givebacks release udc_lock. A system suspend/shutdown may have
+ * taken ownership of the controller during that window.
+ */
+ if (crg_udc->suspended || crg_udc->shutting_down || crg_udc->resetting) {
+ ret = -ESHUTDOWN;
+ goto out;
+ }
+ /* A gadget disconnect may have stopped RUN during giveback. */
+ if (crg_udc->stopped_needs_reset) {
+ ret = -ECONNRESET;
+ goto out;
+ }
+
+ /* EP0 is reinitialized after the event ring has been drained. */
+ if (crg_udc->dev_addr != 0) {
+ u32 param0 = FIELD_PREP(CRG_CMD2_0_DEV_ADDR_MASK, 0);
+
+ if (crg_udc_issue_command(crg_udc, CRG_CMD_SET_ADDR,
+ param0, 0)) {
+ /* Cleanup must still run, but no new transfers may start. */
+ crg_udc->resetting = true;
+ ret = -EIO;
+ goto out;
+ }
+ crg_udc->dev_addr = 0;
+ }
+
+ crg_udc_clear_portpm(crg_udc);
+ ret = 0;
+out:
+ crg_udc_recovery_end(crg_udc);
+ return ret;
+}
+
+void crg_udc_set_pullup(struct crg_gadget_dev *crg_udc, bool is_on)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 val;
+
+ if (is_on) {
+ if (crg_udc->controller_faulted || crg_udc->resetting ||
+ crg_udc->controller_recovering || crg_udc->bus_reset_pending ||
+ crg_udc->endpoint_recovery_pending ||
+ crg_udc->shutting_down || crg_udc->gadget_stopping)
+ return;
+ /* Enable event reporting and interrupts, then assert RUN */
+ val = readl(&uccr->config1);
+ val |= (CRG_U3DC_CFG1_CSC_EVENT_EN |
+ CRG_U3DC_CFG1_PEC_EVENT_EN |
+ CRG_U3DC_CFG1_PPC_EVENT_EN |
+ CRG_U3DC_CFG1_PRC_EVENT_EN |
+ CRG_U3DC_CFG1_PLC_EVENT_EN |
+ CRG_U3DC_CFG1_CEC_EVENT_EN);
+ writel(val, &uccr->config1);
+
+ val = readl(&uccr->control);
+ val |= (CRG_U3DC_CTRL_SYSERR_EN | CRG_U3DC_CTRL_INT_EN |
+ CRG_U3DC_CTRL_RUN);
+ crg_udc->hw_dma_armed = true;
+ crg_udc->transfer_rings_live = true;
+ writel(val, &uccr->control);
+ } else {
+ val = readl(&uccr->control);
+ val &= ~CRG_U3DC_CTRL_RUN;
+ writel(val, &uccr->control);
+ }
+}
+
+void crg_udc_set_vbus_detect(struct crg_gadget_dev *crg_udc, bool enable)
+{
+ u32 val = readl(crg_udc->mmio_virt_base + CRG_UDC_VENDOR_REG);
+
+ if (enable)
+ val |= CRG_UDC_VBUS_DETECT;
+ else
+ val &= ~CRG_UDC_VBUS_DETECT;
+
+ writel(val, crg_udc->mmio_virt_base + CRG_UDC_VENDOR_REG);
+}
+
+static bool crg_udc_sg_advance(struct crg_udc_request *udc_req, u32 bytes)
+{
+ struct scatterlist *sg = udc_req->sg;
+ unsigned int i = udc_req->sg_index;
+
+ while (bytes) {
+ u32 len;
+
+ if (!sg || i >= udc_req->bounce->sgt.nents)
+ return false;
+
+ len = sg_dma_len(sg);
+ if (!len || udc_req->sg_offset == len) {
+ sg = sg_next(sg);
+ i++;
+ udc_req->sg_offset = 0;
+ continue;
+ }
+
+ len -= udc_req->sg_offset;
+ if (bytes < len) {
+ udc_req->sg_offset += bytes;
+ bytes = 0;
+ break;
+ }
+
+ bytes -= len;
+ sg = sg_next(sg);
+ i++;
+ udc_req->sg_offset = 0;
+ }
+
+ udc_req->sg = sg;
+ udc_req->sg_index = i;
+ return true;
+}
+
+static bool crg_udc_sg_current(struct crg_udc_request *udc_req, dma_addr_t *dma,
+ u32 *len)
+{
+ struct scatterlist *sg = udc_req->sg;
+
+ while (sg && udc_req->sg_index < udc_req->bounce->sgt.nents &&
+ !sg_dma_len(sg)) {
+ sg = sg_next(sg);
+ udc_req->sg_index++;
+ udc_req->sg_offset = 0;
+ }
+ udc_req->sg = sg;
+
+ if (!sg || udc_req->sg_index >= udc_req->bounce->sgt.nents)
+ return false;
+
+ if (udc_req->sg_offset >= sg_dma_len(sg))
+ return false;
+
+ *dma = sg_dma_address(sg) + udc_req->sg_offset;
+ *len = sg_dma_len(sg) - udc_req->sg_offset;
+ return true;
+}
+
+static int crg_udc_sg_init(struct crg_udc_request *udc_req)
+{
+ struct scatterlist *sg;
+ unsigned int i;
+
+ for_each_sgtable_dma_sg(&udc_req->bounce->sgt, sg, i) {
+ if (sg_dma_len(sg)) {
+ udc_req->sg = sg;
+ udc_req->sg_index = i;
+ udc_req->sg_offset = 0;
+ return 0;
+ }
+ }
+
+ return -EINVAL;
+}
+
+static void crg_udc_bounce_unmap(struct device *dev,
+ struct crg_udc_bounce *bounce)
+{
+ if (!bounce->mapped)
+ return;
+
+ if (bounce->buf)
+ dma_unmap_single(dev, bounce->dma, bounce->len, bounce->dir);
+ else
+ dma_unmap_sgtable(dev, &bounce->sgt, bounce->dir, 0);
+}
+
+static void crg_udc_bounce_free(struct crg_udc_bounce *bounce)
+{
+ struct page *page, *next;
+
+ list_for_each_entry_safe(page, next, &bounce->pages, lru) {
+ list_del(&page->lru);
+ __free_page(page);
+ }
+ sg_free_table(&bounce->sgt);
+ kfree(bounce->buf);
+ kfree(bounce);
+}
+
+/* Call only after hardware can no longer access the retained mappings. */
+void crg_udc_release_quarantine(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_udc_bounce *bounce, *next;
+
+ list_for_each_entry_safe(bounce, next, &crg_udc->dma_quarantine,
+ quarantine) {
+ list_del(&bounce->quarantine);
+ crg_udc_bounce_unmap(crg_udc->dev, bounce);
+ crg_udc_bounce_free(bounce);
+ }
+}
+
+void crg_udc_unmap_request(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req, size_t received)
+{
+ struct crg_udc_bounce *bounce = udc_req->bounce;
+ struct usb_request *req = &udc_req->usb_req;
+
+ if (!bounce)
+ return;
+
+ udc_req->bounce = NULL;
+ crg_udc_bounce_unmap(udc_ep->crg_udc->dev, bounce);
+ if (bounce->dir == DMA_FROM_DEVICE && received) {
+ size_t len = min(received, bounce->len);
+
+ if (bounce->buf) {
+ memcpy(req->buf, bounce->buf, len);
+ } else {
+ size_t copied = crg_udc_copy_bounce_pages(bounce, req, len);
+
+ if (copied != len) {
+ req->status = -EFAULT;
+ req->actual = copied;
+ }
+ }
+ }
+ crg_udc_bounce_free(bounce);
+}
+
+static int crg_udc_validate_ep_desc(struct crg_udc_ep *udc_ep,
+ const struct usb_endpoint_descriptor *desc)
+{
+ const struct usb_ss_ep_comp_descriptor *comp = udc_ep->comp_desc;
+ u16 maxpacket = usb_endpoint_maxp(desc);
+
+ if (usb_endpoint_xfer_control(desc) || !maxpacket ||
+ maxpacket > CRG_MAX_PACKET_SIZE)
+ return -EINVAL;
+
+ if (crg_udc_index2dci(desc->bEndpointAddress) != udc_ep->dci)
+ return -EINVAL;
+
+ if (usb_endpoint_xfer_bulk(desc) && comp && usb_ss_max_streams(comp))
+ return -EOPNOTSUPP;
+
+ if (usb_endpoint_xfer_isoc(desc) && desc->bInterval == 0)
+ return -EINVAL;
+
+ if (udc_ep->crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+ if (!comp || comp->bMaxBurst > 15)
+ return -EINVAL;
+ if (usb_endpoint_xfer_isoc(desc) &&
+ ((comp->bmAttributes & 0x3) > 2))
+ return -EINVAL;
+ } else if (udc_ep->crg_udc->gadget.speed == USB_SPEED_HIGH) {
+ if (desc->bInterval > 16 && (usb_endpoint_xfer_int(desc) ||
+ usb_endpoint_xfer_isoc(desc)))
+ return -EINVAL;
+ if (usb_endpoint_xfer_isoc(desc) &&
+ usb_endpoint_maxp_mult(desc) > 3)
+ return -EINVAL;
+ } else if (udc_ep->crg_udc->gadget.speed == USB_SPEED_FULL) {
+ if (maxpacket > 64 && !usb_endpoint_xfer_isoc(desc))
+ return -EINVAL;
+ if (usb_endpoint_xfer_isoc(desc) && maxpacket > 1023)
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static bool crg_udc_vbus_connected(struct crg_gadget_dev *crg_udc)
+{
+ return !!(readl(&crg_udc->uccr->portsc) & CRG_U3DC_PORTSC_PP);
+}
+
+static void crg_ep_struct_setup(struct crg_gadget_dev *crg_udc, u32 dci,
+ const char *name)
+{
+ struct crg_udc_ep *ep = &crg_udc->udc_ep[dci];
+
+ ep->dci = dci;
+
+ if (ep->dci > 1) {
+ strscpy_pad(ep->name, name, sizeof(ep->name));
+ ep->usb_ep.name = ep->name;
+
+ ep->usb_ep.caps.type_iso = 1;
+ ep->usb_ep.caps.type_bulk = 1;
+ ep->usb_ep.caps.type_int = 1;
+ if (ep->dci % 2)
+ ep->usb_ep.caps.dir_out = 1;
+ else
+ ep->usb_ep.caps.dir_in = 1;
+ usb_ep_set_maxpacket_limit(&ep->usb_ep, CRG_MAX_PACKET_SIZE);
+ } else {
+ strscpy(ep->name, "ep0", sizeof(ep->name));
+ ep->usb_ep.name = ep->name;
+ ep->usb_ep.maxpacket = 512;
+ ep->usb_ep.caps.type_control = 1;
+ ep->usb_ep.caps.dir_in = 1;
+ ep->usb_ep.caps.dir_out = 1;
+ }
+
+ ep->usb_ep.ops = &crg_udc_ep_ops;
+ ep->crg_udc = crg_udc;
+
+ INIT_LIST_HEAD(&ep->queue);
+ if (ep->dci > 1)
+ list_add_tail(&ep->usb_ep.ep_list, &crg_udc->gadget.ep_list);
+}
+
+void crg_udc_reset_work(struct work_struct *work)
+{
+ struct crg_gadget_dev *crg_udc =
+ container_of(work, struct crg_gadget_dev, udc_reset_work);
+ struct usb_gadget_driver *driver;
+ unsigned long flags;
+ u32 generation;
+ bool requeue;
+ int i, ret;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (crg_udc->resetting || crg_udc->controller_recovering ||
+ crg_udc->shutting_down || crg_udc->gadget_stopping ||
+ crg_udc->suspended)
+ goto unlock;
+ generation = crg_udc->bus_reset_generation;
+ driver = crg_udc->bus_reset_pending && crg_udc->async_callbacks ?
+ crg_udc->gadget_driver : NULL;
+ /* Defer until callbacks initiated before the reset have returned. */
+ if (driver && crg_udc->gadget_callbacks_active)
+ goto unlock;
+ if (driver)
+ crg_udc->reset_callback_active = true;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ if (driver)
+ usb_gadget_udc_reset(&crg_udc->gadget, driver);
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (generation == crg_udc->bus_reset_generation &&
+ crg_udc->bus_reset_pending) {
+ /*
+ * DMA was stopped by crg_udc_reinit() before scheduling this work.
+ * Some gadget functions do not call usb_ep_disable() when reset.
+ * Finalize these endpoints under the lock and generation check;
+ * calling usb_ep_disable() after unlocking would let a newer
+ * reset reuse the endpoint before the disable operation.
+ */
+ for (i = 2; i < CRG_NUM_EP_CX; i++) {
+ struct crg_udc_ep *ep = &crg_udc->udc_ep[i];
+ struct crg_ep_cx *epcx;
+
+ if (ep->ep_state != EP_STATE_DISABLED ||
+ !ep->usb_ep.enabled)
+ continue;
+
+ dev_warn(crg_udc->dev,
+ "EP%d remained enabled after gadget reset\n", i);
+ epcx = &crg_udc->p_epcx[i - 2];
+ ep->generation++;
+ ep->desc = NULL;
+ ep->comp_desc = NULL;
+ ep->tran_ring_full = false;
+ ep->halt_failed = false;
+ ep->disable_failed = false;
+ ep->dequeue_paused = false;
+ memset(epcx, 0, sizeof(*epcx));
+ /*
+ * The controller disabled this endpoint and drained the
+ * requests; reflect that in the USB core's enabled flag.
+ */
+ crg_udc_ep_clear_enabled(ep);
+ }
+ if (!crg_udc->resetting)
+ crg_udc->bus_reset_pending = false;
+ }
+
+ /*
+ * Keep port reset events deferred through endpoint cleanup, not just
+ * through the gadget callback. Replay after clearing the guard.
+ */
+ if (driver)
+ crg_udc->reset_callback_active = false;
+ if (!crg_udc->resetting && !crg_udc->suspended &&
+ !crg_udc->shutting_down && crg_udc->deferred_system_error) {
+ crg_udc->deferred_system_error = false;
+ crg_udc_recover_link(crg_udc, &flags, false);
+ }
+ if (!crg_udc->resetting && !crg_udc->suspended &&
+ !crg_udc->shutting_down && crg_udc->deferred_port_events) {
+ ret = crg_udc_handle_port_status(crg_udc, &flags);
+ if (ret && ret != -ECONNRESET)
+ dev_err(crg_udc->dev,
+ "deferred port status failed: %d\n", ret);
+ }
+
+ /* Port-event replay may have initiated another bus reset. */
+ requeue = generation != crg_udc->bus_reset_generation &&
+ crg_udc->bus_reset_pending && crg_udc->async_callbacks &&
+ crg_udc->gadget_driver;
+ /* A failed configured-session reset defers RUN until the gadget's
+ * reset callback has dropped the old composite configuration.
+ */
+ if (!requeue && !crg_udc->bus_reset_pending && !crg_udc->connected &&
+ !crg_udc->resetting && !crg_udc->suspended &&
+ !crg_udc->shutting_down && crg_udc->pullup_on &&
+ crg_udc->gadget_driver && crg_udc->async_callbacks &&
+ (crg_udc->restart_after_reset || crg_udc_vbus_connected(crg_udc))) {
+ crg_udc->restart_after_reset = false;
+ crg_udc_set_pullup(crg_udc, true);
+ }
+
+ if (requeue || crg_udc->external_reset_pending ||
+ crg_udc->endpoint_recovery_pending)
+ crg_udc_kick_reset_work(crg_udc);
+unlock:
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+}
+
+/*
+ * Called with udc_lock held from the port IRQ or its retry worker.
+ * The old bus-reset path may have failed to quiesce endpoint DMA. Never
+ * unmap requests or restart RUN based on a failed EP_ENABLE/EP_RUNNING
+ * poll; SWRST must succeed and rebuild all rings first.
+ *
+ * A successful full reset invalidates the current PORTSC snapshot/event
+ * ring, so callers must stop processing that event (-ECONNRESET).
+ */
+static int crg_udc_recover_port_failure(struct crg_gadget_dev *crg_udc,
+ unsigned long *flags, bool notify_reset)
+{
+ int ret;
+
+ if (crg_udc->controller_recovering || crg_udc->suspended ||
+ crg_udc->shutting_down || !crg_udc->gadget_driver)
+ return -ESHUTDOWN;
+
+ ret = crg_udc_recover_link(crg_udc, flags, notify_reset);
+ return ret ?: -ECONNRESET;
+}
+
+/* PRC is W1C at detection, not a completion notification. */
+#define CRG_PORT_RESET_RETRY_MS 50
+#define CRG_PORT_RESET_MAX_RETRIES 40
+
+void crg_udc_port_retry_work(struct work_struct *work)
+{
+ struct crg_gadget_dev *crg_udc =
+ container_of(to_delayed_work(work), struct crg_gadget_dev,
+ port_retry_work);
+ unsigned long flags;
+ int ret;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (!crg_udc->port_reset_pending && !crg_udc->deferred_port_events)
+ goto out;
+ if (crg_udc->resetting || crg_udc->suspended ||
+ crg_udc->shutting_down || !crg_udc->pullup_on ||
+ !crg_udc->gadget_driver || !crg_udc->async_callbacks)
+ goto out;
+ /* Reset work replays deferred events after the callback. A controller
+ * reconstruction is different: retry once its register state is stable.
+ */
+ if (crg_udc->controller_recovering) {
+ mod_delayed_work(system_wq, &crg_udc->port_retry_work,
+ msecs_to_jiffies(CRG_PORT_RESET_RETRY_MS));
+ goto out;
+ }
+ if (crg_udc->port_reset_pending &&
+ crg_udc->port_reset_retry_count++ >= CRG_PORT_RESET_MAX_RETRIES) {
+ bool notify_reset = crg_udc->device_state == USB_STATE_CONFIGURED;
+
+ dev_err(crg_udc->dev,
+ "port reset did not become ready; resetting controller\n");
+ ret = crg_udc_recover_port_failure(crg_udc, &flags,
+ notify_reset);
+ if (ret != -ECONNRESET)
+ dev_err(crg_udc->dev,
+ "port-reset recovery returned %d\n", ret);
+ goto out;
+ }
+ ret = crg_udc_handle_port_status(crg_udc, &flags);
+ if (ret && ret != -ECONNRESET)
+ dev_dbg(crg_udc->dev, "port reset poll: %d\n", ret);
+out:
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+}
+
+int crg_udc_handle_port_status(struct crg_gadget_dev *crg_udc,
+ unsigned long *flags)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ bool was_connected = crg_udc->connected;
+ u32 portsc_val;
+ u32 tmp;
+ u32 speed;
+ int ret;
+
+ /* Snapshot and acknowledge every W1C status bit exactly once. */
+ portsc_val = readl(&uccr->portsc);
+ trace_crg_udc_portsc(portsc_val);
+ writel(portsc_val & CRG_U3DC_PORTSC_W1C_MASK, &uccr->portsc);
+
+ /* A new port reset must not tear down endpoints from inside a gadget
+ * reset callback. Preserve the acknowledged W1C events for replay.
+ */
+ if (crg_udc->reset_callback_active || crg_udc->suspended ||
+ crg_udc->shutting_down || crg_udc->controller_recovering) {
+ if (crg_udc->suspended || crg_udc->shutting_down)
+ return -ESHUTDOWN;
+ crg_udc->deferred_port_events |=
+ portsc_val & CRG_U3DC_PORTSC_W1C_MASK;
+ if (portsc_val & CRG_U3DC_PORTSC_PRC)
+ crg_udc->port_reset_pending = true;
+ return 0;
+ }
+
+ portsc_val |= crg_udc->deferred_port_events;
+ crg_udc->deferred_port_events = 0;
+
+ /* Process link changes before any disconnect path can reset the UDC. */
+ if (portsc_val & CRG_U3DC_PORTSC_PLC) {
+ tmp = readl(&uccr->portsc);
+ if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) == 0xf) {
+ tmp &= ~CRG_U3DC_PORTSC_W1C_MASK;
+ FIELD_MODIFY(CRG_U3DC_PORTSC_PLS_MASK, &tmp, 0x0);
+ tmp |= CRG_U3DC_PORTSC_LWS;
+ writel(tmp, &uccr->portsc);
+ }
+ }
+
+ /* A reset may finish before the port is ready; retry on later changes. */
+ if (portsc_val & CRG_U3DC_PORTSC_PRC) {
+ if (!crg_udc->port_reset_pending)
+ crg_udc->port_reset_retry_count = 0;
+ crg_udc->port_reset_pending = true;
+ }
+
+ /* Retry even when no new W1C event was raised: PRC describes when
+ * the reset was detected, not when PED becomes ready.
+ */
+ if (crg_udc->port_reset_pending) {
+ tmp = readl(&uccr->portsc);
+ if (tmp & CRG_U3DC_PORTSC_PRC) {
+ dev_dbg(crg_udc->dev, "PRC is still set\n");
+ } else if (tmp & CRG_U3DC_PORTSC_PR) {
+ /* first port status change event for port reset */
+ dev_dbg(crg_udc->dev,
+ "PRC is not set, but PR is set\n");
+ } else {
+ if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) != 0 ||
+ !(tmp & CRG_U3DC_PORTSC_PED)) {
+ dev_dbg(crg_udc->dev, "portsc=0x%x no PED\n",
+ tmp);
+ goto prc_done;
+ }
+
+ switch (FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, tmp)) {
+ case CRG_U3DC_PORTSC_SPEED_SSP:
+ speed = USB_SPEED_SUPER_PLUS;
+ break;
+ case CRG_U3DC_PORTSC_SPEED_SS:
+ speed = USB_SPEED_SUPER;
+ break;
+ case CRG_U3DC_PORTSC_SPEED_HS:
+ speed = USB_SPEED_HIGH;
+ break;
+
+ case CRG_U3DC_PORTSC_SPEED_FS:
+ speed = USB_SPEED_FULL;
+ break;
+
+ case CRG_U3DC_PORTSC_SPEED_LS:
+ speed = USB_SPEED_LOW;
+ break;
+ default:
+ speed = USB_SPEED_UNKNOWN;
+ goto prc_done;
+ }
+
+ /* All prerequisites are met; the deferred reset is consumed. */
+ crg_udc->port_reset_pending = false;
+
+ if (crg_udc->device_state >= USB_STATE_DEFAULT) {
+ /*
+ * A single enumeration commonly involves two
+ * bus resets (an initial one, then another
+ * after SET_ADDRESS before SET_CONFIGURATION).
+ * By the second reset device_state is already
+ * USB_STATE_ADDRESS, which is >= DEFAULT but
+ * has no active configuration to tear down.
+ * Only notify composite when a configuration
+ * was actually active: otherwise this races
+ * with the SET_CONFIGURATION that's about to
+ * arrive, since the deferred work below can
+ * run after fsg_set_alt() has already raised
+ * its own FSG_STATE_CONFIG_CHANGE exception
+ * and reset_config()'s ->disable() would
+ * overwrite that exception's argument back to
+ * NULL before the function's thread consumes
+ * it, silently skipping the real enable.
+ */
+ bool was_configured = crg_udc->device_state ==
+ USB_STATE_CONFIGURED;
+ bool notify_reset = was_configured &&
+ crg_udc->async_callbacks &&
+ crg_udc->gadget_driver;
+ int reinit_ret;
+
+ reinit_ret = crg_udc_reinit(crg_udc, notify_reset);
+ if (reinit_ret) {
+ dev_err(crg_udc->dev,
+ "bus reset reinit failed: %d\n",
+ reinit_ret);
+ /* SWRST is required before releasing DMA buffers.
+ * Do not resume processing the stale port event.
+ */
+ if (reinit_ret == -ETIMEDOUT ||
+ reinit_ret == -EIO)
+ return crg_udc_recover_port_failure(
+ crg_udc, flags, notify_reset);
+ return reinit_ret;
+ }
+ if (notify_reset && crg_udc->bus_reset_pending)
+ schedule_work(&crg_udc->udc_reset_work);
+ }
+
+ crg_udc->gadget.speed = speed;
+
+ ret = crg_udc_update_ep0_maxpacketsize(crg_udc);
+ if (ret) {
+ dev_err(crg_udc->dev,
+ "failed to configure EP0 maxpacket: %d\n", ret);
+ return crg_udc_recover_port_failure(
+ crg_udc, flags,
+ crg_udc->bus_reset_pending ||
+ crg_udc->device_state == USB_STATE_CONFIGURED);
+ }
+
+ crg_udc->connected = 1;
+
+ if (crg_udc->device_state < USB_STATE_RECONNECTING)
+ crg_udc_enable_setup(crg_udc);
+ }
+ }
+prc_done:
+ /* handle Port Reset end */
+
+ /* Port-enable and power changes can also complete initialization. */
+ if (portsc_val & (CRG_U3DC_PORTSC_CSC |
+ CRG_U3DC_PORTSC_PEC |
+ CRG_U3DC_PORTSC_PPC)) {
+ tmp = readl(&uccr->portsc);
+ if ((tmp & (CRG_U3DC_PORTSC_CCS | CRG_U3DC_PORTSC_PP)) ==
+ (CRG_U3DC_PORTSC_CCS | CRG_U3DC_PORTSC_PP)) {
+ /* Do not configure EP0 until the pending port reset is handled. */
+ if (crg_udc->port_reset_pending)
+ goto csc_done;
+ if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) != 0 ||
+ (!(tmp & CRG_U3DC_PORTSC_PED)))
+ goto csc_done;
+
+ switch (FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, tmp)) {
+ case CRG_U3DC_PORTSC_SPEED_SSP:
+ speed = USB_SPEED_SUPER_PLUS;
+ break;
+ case CRG_U3DC_PORTSC_SPEED_SS:
+ speed = USB_SPEED_SUPER;
+ break;
+ case CRG_U3DC_PORTSC_SPEED_HS:
+ speed = USB_SPEED_HIGH;
+ break;
+
+ case CRG_U3DC_PORTSC_SPEED_FS:
+ speed = USB_SPEED_FULL;
+ break;
+
+ case CRG_U3DC_PORTSC_SPEED_LS:
+ speed = USB_SPEED_LOW;
+ break;
+ default:
+ speed = USB_SPEED_UNKNOWN;
+ goto csc_done;
+ }
+ crg_udc->gadget.speed = speed;
+
+ ret = crg_udc_update_ep0_maxpacketsize(crg_udc);
+ if (ret) {
+ dev_err(crg_udc->dev,
+ "failed to configure EP0 maxpacket: %d\n", ret);
+ return crg_udc_recover_port_failure(
+ crg_udc, flags,
+ crg_udc->bus_reset_pending ||
+ crg_udc->device_state == USB_STATE_CONFIGURED);
+ }
+
+ crg_udc->connected = 1;
+
+ if (crg_udc->device_state < USB_STATE_RECONNECTING)
+ crg_udc_enable_setup(crg_udc);
+
+ } else if ((portsc_val & (CRG_U3DC_PORTSC_CSC |
+ CRG_U3DC_PORTSC_PPC)) &&
+ (!(tmp & CRG_U3DC_PORTSC_CCS) ||
+ !(tmp & CRG_U3DC_PORTSC_PP))) {
+ bool cable_connected;
+ int ccs_drop_ignore = 0;
+
+ if ((FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) == 0x0) &&
+ (FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, tmp) <
+ CRG_U3DC_PORTSC_SPEED_SS)) {
+ ccs_drop_ignore = 1;
+ dev_err(crg_udc->dev, "ccs glitch detect on HS/FS!\n");
+ }
+
+ cable_connected = crg_udc_vbus_connected(crg_udc);
+ /* A missing port-power bit overrides the HS/FS CCS glitch rule. */
+ if (!cable_connected)
+ ccs_drop_ignore = 0;
+ if (!ccs_drop_ignore)
+ crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+ if ((!ccs_drop_ignore || !cable_connected) &&
+ was_connected && crg_udc->connected) {
+ u32 generation = crg_udc->udc_event[0].generation;
+
+ /* The callback drops udc_lock: mark link down first. */
+ crg_udc->connected = 0;
+ crg_udc_notify_disconnect(crg_udc, flags);
+ if (crg_udc->resetting ||
+ generation != crg_udc->udc_event[0].generation)
+ return -ECONNRESET;
+
+ /* Another thread may have handled a new connection. */
+ if (crg_udc->connected)
+ goto csc_done;
+
+ /* A new CCS=1 alone is not proof that software was reset. */
+ cable_connected = crg_udc_vbus_connected(crg_udc);
+ }
+ if (cable_connected && !ccs_drop_ignore) {
+ bool notify_reset = crg_udc->device_state ==
+ USB_STATE_CONFIGURED;
+ int ret;
+
+ ret = crg_udc_reinit(crg_udc, false);
+ if (ret == -ETIMEDOUT || ret == -EIO)
+ return crg_udc_recover_port_failure(
+ crg_udc, flags, notify_reset);
+ if (ret)
+ return ret;
+ crg_udc->connected = 0;
+ } else if (!cable_connected) {
+ dev_dbg(crg_udc->dev,
+ "cable disconnected, rst controller\n");
+ /* Keep RUN and port interrupts enabled for the next cable. */
+ crg_udc_recover_link(crg_udc, flags, false);
+ return -ECONNRESET;
+ }
+ }
+ }
+csc_done:
+ /* PRC was already acknowledged. It may never interrupt us again if
+ * PED becomes ready without a further CSC/PLC transition.
+ */
+ if (crg_udc->port_reset_pending && !crg_udc->resetting &&
+ !crg_udc->shutting_down && !crg_udc->suspended &&
+ crg_udc->pullup_on && crg_udc->gadget_driver &&
+ crg_udc->async_callbacks &&
+ crg_udc->port_reset_retry_count < CRG_PORT_RESET_MAX_RETRIES)
+ mod_delayed_work(system_wq, &crg_udc->port_retry_work,
+ msecs_to_jiffies(CRG_PORT_RESET_RETRY_MS));
+ return 0;
+}
+
+/*
+ * A giveback drops udc_lock; its callback can reset, stall, or replace
+ * the control transfer. Never advance an old EP0 transaction afterwards.
+ */
+static void crg_udc_ep0_complete_if_current(struct crg_gadget_dev *crg_udc,
+ struct crg_udc_ep *ep, u32 generation,
+ u32 event_generation, u8 setup_tag,
+ enum crg_setup_status setup_status,
+ unsigned long *flags)
+{
+ if (crg_udc->resetting || crg_udc->bus_reset_pending ||
+ crg_udc->port_reset_pending ||
+ crg_udc->device_state == USB_STATE_RECONNECTING ||
+ crg_udc->ep0_state != EP0_STATE_IDLE ||
+ ep->ep_state != EP_STATE_RUNNING || ep->nuking || !ep->desc ||
+ !usb_endpoint_xfer_control(ep->desc) ||
+ ep->generation != generation ||
+ crg_udc->udc_event[0].generation != event_generation ||
+ crg_udc->setup_tag != setup_tag ||
+ crg_udc->setup_status != setup_status)
+ return;
+
+ crg_udc_ep0_req_complete(ep, flags);
+}
+
+int crg_udc_handle_xfer_event(struct crg_gadget_dev *crg_udc,
+ struct crg_event_trb *event, unsigned long *flags)
+{
+ u32 dw3 = le32_to_cpu(event->dw3);
+ u8 dci = FIELD_GET(EVE_TRB_ENDPOINT_ID_MASK, dw3);
+ u16 comp_code =
+ FIELD_GET(EVE_TRB_COMPL_CODE_MASK, le32_to_cpu(event->dw2));
+ struct crg_udc_ep *udc_ep_ptr;
+ struct crg_udc_request *udc_req_ptr;
+ struct crg_transfer_trb *trb;
+ bool trbs_dequeued = false;
+ u32 ep_generation = 0, event_generation = 0;
+ enum crg_setup_status setup_status = WAIT_FOR_SETUP;
+ u8 setup_tag = 0;
+ u32 residual;
+
+ if (dci >= CRG_NUM_EP_CX || dci == 1)
+ return -EINVAL;
+
+ udc_ep_ptr = &crg_udc->udc_ep[dci];
+ if (dci == 0) {
+ ep_generation = udc_ep_ptr->generation;
+ event_generation = crg_udc->udc_event[0].generation;
+ setup_tag = crg_udc->setup_tag;
+ setup_status = crg_udc->setup_status;
+ }
+ /* Recovery owns the mappings while DMA state is uncertain. */
+ if (crg_udc->endpoint_recovery_pending || crg_udc->port_reset_pending ||
+ udc_ep_ptr->halt_failed)
+ return 0;
+ if (!udc_ep_ptr->first_trb ||
+ crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_DISABLED)
+ return -ENODEV;
+
+ /* EP0 transfer events from an abort must not affect the new ring. */
+ if (dci == 0 && crg_udc->ep0_state != EP0_STATE_IDLE)
+ return 0;
+
+ if (comp_code != CMPL_CODE_STOPPED &&
+ comp_code != CMPL_CODE_STOPPED_LENGTH_INVALID &&
+ comp_code != CMPL_CODE_DISABLED &&
+ comp_code != CMPL_CODE_DISABLED_LENGTH_INVALID &&
+ comp_code != CMPL_CODE_HALTED &&
+ comp_code != CMPL_CODE_HALTED_LENGTH_INVALID &&
+ comp_code != CMPL_CODE_SETUP_TAG_MISMATCH) {
+ /*
+ * Only live requests may advance the transfer-ring dequeue
+ * pointer. EP0 data and status requests both track their TRBs.
+ */
+ if (list_empty(&udc_ep_ptr->queue) ||
+ is_request_dequeued(crg_udc, udc_ep_ptr, event))
+ return 0;
+
+ crg_udc_update_dequeue_pt(event, udc_ep_ptr);
+ }
+
+ switch (comp_code) {
+ case CMPL_CODE_SUCCESS:
+ trb = crg_udc_tran_trb_dma_to_virt(
+ udc_ep_ptr,
+ (u64)le32_to_cpu(event->dw0) |
+ ((u64)le32_to_cpu(event->dw1) << 32));
+ if (!trb || list_empty(&udc_ep_ptr->queue))
+ break;
+
+ /*
+ * A control transfer has separate data-stage and status-stage
+ * TRBs. The status request is an internal zero-length request
+ * and its successful completion must not go through
+ * crg_udc_build_td(). Doing that attempts to queue another
+ * control TD while the status request is already on the EP0
+ * software queue, leaving it stuck in -EINPROGRESS. The next
+ * SETUP then sees a non-empty queue and enumeration fails.
+ */
+ if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(trb->dw3)) ==
+ TRB_TYPE_XFER_STATUS_STAGE) {
+ udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+ struct crg_udc_request,
+ queue);
+ if (dci == 0) {
+ udc_req_ptr->usb_req.actual = 0;
+ trbs_dequeued = true;
+ crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
+ crg_udc_ep0_complete_if_current(crg_udc, udc_ep_ptr,
+ ep_generation, event_generation,
+ setup_tag, setup_status, flags);
+ } else {
+ dev_err(crg_udc->dev,
+ "status-stage TRB on endpoint %u\n",
+ dci);
+ }
+ break;
+ }
+
+ if (!FIELD_GET(TRB_CHAIN_BIT_MASK, le32_to_cpu(trb->dw3))) {
+ udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+ struct crg_udc_request,
+ queue);
+ trbs_dequeued = true;
+ udc_req_ptr->td_pending = false;
+
+ if (udc_req_ptr->all_trbs_queued) {
+ udc_req_ptr->usb_req.actual =
+ udc_req_ptr->usb_req.length -
+ udc_req_ptr->buff_len_left;
+ residual = FIELD_GET(EVE_TRB_TRAN_LEN_MASK,
+ le32_to_cpu(event->dw2));
+ if (residual > udc_req_ptr->usb_req.actual)
+ residual = 0;
+ udc_req_ptr->usb_req.actual -= residual;
+ crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
+
+ if (dci == 0)
+ crg_udc_ep0_complete_if_current(crg_udc, udc_ep_ptr,
+ ep_generation, event_generation,
+ setup_tag, setup_status, flags);
+ } else {
+ int ret;
+
+ ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+ if (ret == -ENOSPC || ret == -EBUSY) {
+ /* A previous TD completed, but this next TD
+ * could not yet be published. Retry later.
+ */
+ udc_ep_ptr->tran_ring_full = true;
+ } else if (ret) {
+ udc_req_ptr->usb_req.actual =
+ udc_req_ptr->usb_req.length -
+ udc_req_ptr->buff_len_left;
+ crg_udc_req_done(udc_ep_ptr, udc_req_ptr,
+ ret);
+ }
+ }
+ }
+ break;
+
+ case CMPL_CODE_SHORT_PKT:
+ if (list_empty(&udc_ep_ptr->queue))
+ break;
+
+ udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+ struct crg_udc_request, queue);
+ trb = crg_udc_tran_trb_dma_to_virt(
+ udc_ep_ptr,
+ (u64)le32_to_cpu(event->dw0) |
+ ((u64)le32_to_cpu(event->dw1) << 32));
+ if (!trb)
+ break;
+
+ residual = FIELD_GET(EVE_TRB_TRAN_LEN_MASK,
+ le32_to_cpu(event->dw2));
+ udc_req_ptr->usb_req.actual =
+ crg_udc_actual_before(udc_ep_ptr, udc_req_ptr, trb) +
+ FIELD_GET(TRB_TRANSFER_LEN_MASK,
+ le32_to_cpu(trb->dw2)) -
+ min_t(u32, residual,
+ FIELD_GET(TRB_TRANSFER_LEN_MASK,
+ le32_to_cpu(trb->dw2)));
+ udc_req_ptr->usb_req.actual = min(udc_req_ptr->usb_req.actual,
+ udc_req_ptr->usb_req.length);
+
+ /*
+ * Advance past the completed request before giveback drops
+ * udc_lock and a completion callback can modify the queue.
+ */
+ trbs_dequeued = true;
+ crg_udc_advance_dequeue_pt(udc_ep_ptr);
+ if (udc_req_ptr->usb_req.short_not_ok &&
+ usb_endpoint_dir_out(udc_ep_ptr->desc))
+ crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EREMOTEIO);
+ else
+ crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
+ if (dci == 0)
+ crg_udc_ep0_complete_if_current(crg_udc, udc_ep_ptr,
+ ep_generation, event_generation,
+ setup_tag, setup_status, flags);
+ break;
+
+ case CMPL_CODE_PROTOCOL_STALL:
+ case CMPL_CODE_TRB_STALL:
+ if (!list_empty(&udc_ep_ptr->queue)) {
+ udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+ struct crg_udc_request,
+ queue);
+ crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EPIPE);
+ trbs_dequeued = true;
+ }
+ if (dci == 0)
+ crg_udc->setup_status = WAIT_FOR_SETUP;
+ break;
+
+ case CMPL_CODE_SETUP_TAG_MISMATCH:
+ /*
+ * EP0 SETUP changes are serialized by the SETUP event handler.
+ * When a new SETUP arrives, crg_udc_abort_ep0_for_setup() stops
+ * EP0, gives back the old request, resets the EP0 ring, and then
+ * accepts the new SETUP. A SETUP_TAG_MISMATCH event generated
+ * afterwards therefore belongs to the aborted control transfer.
+ *
+ * Do not complete the current EP0 request or change setup_status
+ * here: doing so could operate on the new control transfer and
+ * recreate the EP0 queue/state race this path is intended to avoid.
+ */
+ if (dci == 0) {
+ dev_dbg(crg_udc->dev,
+ "ignoring stale EP0 SETUP_TAG_MISMATCH\n");
+ break;
+ }
+
+ /* This completion code is meaningful only for EP0. */
+ dev_warn(crg_udc->dev,
+ "SETUP_TAG_MISMATCH on non-EP0 endpoint %u\n", dci);
+ break;
+
+ case CMPL_CODE_BABBLE_DETECTED_ERR:
+ case CMPL_CODE_DATA_BUFFER_ERR:
+ case CMPL_CODE_INVALID_STREAM_TYPE_ERR:
+ case CMPL_CODE_RING_UNDERRUN:
+ case CMPL_CODE_RING_OVERRUN:
+ case CMPL_CODE_EVENT_RING_FULL_ERR:
+ case CMPL_CODE_ISOCH_BUFFER_OVERRUN:
+ case CMPL_CODE_USB_TRANS_ERR:
+ case CMPL_CODE_TRB_ERR:
+ default:
+ /* Unknown codes are not proof that endpoint DMA has stopped. */
+ dev_err(crg_udc->dev, "transfer error, completion code 0x%x\n",
+ comp_code);
+ if (dci == 0) {
+ int ret;
+
+ /* EP0 must stop DMA before completing a failed transfer. */
+ if (!list_empty(&udc_ep_ptr->queue)) {
+ udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+ struct crg_udc_request,
+ queue);
+ if (udc_req_ptr->usb_req.status == -EINPROGRESS)
+ udc_req_ptr->usb_req.status = -EIO;
+ }
+ ret = crg_udc_abort_ep0_for_setup(crg_udc);
+ if (ret)
+ dev_err(crg_udc->dev,
+ "failed to stop EP0 after transfer error: %d\n",
+ ret);
+ } else {
+ int ret = crg_udc_stop_error_ep(crg_udc, dci);
+
+ if (ret)
+ dev_err(crg_udc->dev,
+ "failed to stop EP%u after transfer error: %d\n",
+ dci, ret);
+ }
+ break;
+
+ case CMPL_CODE_STOPPED:
+ case CMPL_CODE_STOPPED_LENGTH_INVALID:
+ case CMPL_CODE_HALTED:
+ case CMPL_CODE_HALTED_LENGTH_INVALID:
+ case CMPL_CODE_DISABLED:
+ case CMPL_CODE_DISABLED_LENGTH_INVALID:
+ break;
+ }
+
+ /* A completed TD frees room, but this event may also have published
+ * the next TD and filled the ring again. Recompute from the current
+ * pointers: unconditionally clearing tran_ring_full loses that state.
+ */
+ if (trbs_dequeued && dci != 0 &&
+ udc_ep_ptr->ep_state == EP_STATE_RUNNING &&
+ !crg_udc->resetting && !crg_udc->port_reset_pending &&
+ !udc_ep_ptr->halt_failed) {
+ u32 ring_size = udc_ep_ptr->last_trb -
+ udc_ep_ptr->first_trb + 1;
+ u32 room = crg_udc_room_on_ring(crg_udc, ring_size,
+ udc_ep_ptr->first_trb,
+ udc_ep_ptr->enq_pt,
+ udc_ep_ptr->deq_pt);
+
+ udc_ep_ptr->tran_ring_full = !room;
+ if (room && crg_udc_queue_pending_trbs(udc_ep_ptr))
+ udc_ep_ptr->tran_ring_full = true;
+ }
+
+ return 0;
+}
+
+void crg_udc_dispatch_setup(struct crg_gadget_dev *crg_udc,
+ struct usb_ctrlrequest *setup, u8 setup_tag,
+ unsigned long *flags)
+{
+ struct crg_udc_ep *ep0 = &crg_udc->udc_ep[0];
+
+ /* A completed control transfer can leave old TRBs on the ring even
+ * after its software queue becomes empty. A new SETUP aborts those
+ * TRBs: stop and reinitialize EP0 before accepting another request.
+ */
+ if (crg_udc->setup_status != WAIT_FOR_SETUP ||
+ crg_udc->ep0_state != EP0_STATE_IDLE ||
+ ep0->ep_state != EP_STATE_RUNNING ||
+ !list_empty(&ep0->queue) || ep0->enq_pt != ep0->deq_pt ||
+ ep0->tran_ring_full) {
+ /* Do not submit TRBs if STOP_EP failed. */
+ if (crg_udc_abort_ep0_for_setup(crg_udc))
+ return;
+ }
+
+ crg_udc_handle_setup_pkt(crg_udc, setup, setup_tag, flags);
+}
+
+int crg_udc_handle_event(struct crg_gadget_dev *crg_udc,
+ struct crg_event_trb *event, unsigned long *flags)
+{
+ int ret;
+
+ trace_crg_udc_event_trb(event);
+
+ switch (FIELD_GET(EVE_TRB_TYPE_MASK, le32_to_cpu(event->dw3))) {
+ case TRB_TYPE_EVT_PORT_STATUS_CHANGE:
+ if (crg_udc->device_state == USB_STATE_RECONNECTING) {
+ crg_udc->portsc_on_reconnecting = 1;
+ break;
+ }
+ /* 1.Port Reset 2.Port Connection Change 3.Port Link Change */
+ ret = crg_udc_handle_port_status(crg_udc, flags);
+ if (ret)
+ return ret;
+
+ break;
+ case TRB_TYPE_EVT_TRANSFER:
+ if (crg_udc->device_state < USB_STATE_RECONNECTING) {
+ dev_err(crg_udc->dev,
+ "Xfer compl event rcved when dev state=%d !\n",
+ crg_udc->device_state);
+ break;
+ }
+
+ ret = crg_udc_handle_xfer_event(crg_udc, event, flags);
+ if (ret)
+ return ret;
+ break;
+ case TRB_TYPE_EVT_SETUP_PKT: {
+ struct usb_ctrlrequest setup;
+ u32 dw0 = le32_to_cpu(event->dw0);
+ u32 dw1 = le32_to_cpu(event->dw1);
+ u8 setup_tag;
+
+ BUILD_BUG_ON(sizeof(struct usb_ctrlrequest) != 8);
+ setup.bRequestType = dw0 & 0xff;
+ setup.bRequest = (dw0 >> 8) & 0xff;
+ setup.wValue = cpu_to_le16((dw0 >> 16) & 0xffff);
+ setup.wIndex = cpu_to_le16(dw1 & 0xffff);
+ setup.wLength = cpu_to_le16((dw1 >> 16) & 0xffff);
+
+ setup_tag = FIELD_GET(EVE_TRB_SETUP_TAG_MASK,
+ le32_to_cpu(event->dw3));
+ if (crg_udc->device_state == USB_STATE_RECONNECTING) {
+ /* EP0 is not ready until the old event ring is drained. */
+ crg_udc->pending_setup = setup;
+ crg_udc->pending_setup_tag = setup_tag;
+ crg_udc->pending_setup_valid = true;
+ break;
+ }
+ crg_udc_dispatch_setup(crg_udc, &setup, setup_tag, flags);
+
+ break;
+ }
+ default:
+ dev_dbg(crg_udc->dev, "unexpected TRB_TYPE = 0x%lx",
+ FIELD_GET(EVE_TRB_TYPE_MASK, le32_to_cpu(event->dw3)));
+ break;
+ }
+
+ return 0;
+}
+
+/*
+ * Invoke the bound gadget driver's ->disconnect() callback, if any.
+ *
+ * The gadget disconnect callback may be called in interrupt context.
+ * Release only udc_lock while keeping local IRQs disabled.
+ */
+void crg_udc_notify_disconnect(struct crg_gadget_dev *crg_udc,
+ unsigned long *flags)
+{
+ struct usb_gadget_driver *driver = crg_udc->gadget_driver;
+
+ /* Report link loss even when there is no gadget callback. */
+ usb_gadget_set_state(&crg_udc->gadget, USB_STATE_NOTATTACHED);
+
+ if (!crg_udc->async_callbacks || !driver || !driver->disconnect)
+ return;
+
+ /*
+ * The unbind path disables async callbacks and synchronizes the IRQ
+ * before freeing the gadget driver. Keep this pointer across unlock.
+ */
+ crg_udc_callback_enter(crg_udc);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+ driver->disconnect(&crg_udc->gadget);
+ spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+ crg_udc_callback_exit(crg_udc);
+}
+
+int crg_udc_ep_halt(struct crg_udc_ep *udc_ep_ptr, int halt, int ignore_wedge,
+ unsigned long *flags)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ enum crg_ep_state ep_state;
+ bool reset_seq_only = false;
+ bool recovering;
+ int do_halt, ret;
+ u32 param0;
+ u32 tmp;
+
+ if (crg_udc->resetting || crg_udc->endpoint_recovery_pending)
+ return -ESHUTDOWN;
+ if (!udc_ep_ptr->desc) {
+ dev_err(crg_udc->dev, "NULL desc\n");
+ return -EINVAL;
+ }
+ /* Failed disable is only recoverable by disable retry or reset. */
+ if (udc_ep_ptr->disable_failed)
+ return -ESHUTDOWN;
+
+ if (usb_endpoint_xfer_isoc(udc_ep_ptr->desc)) {
+ dev_err(crg_udc->dev, "Isoc ep, halt not supported\n");
+ return -EOPNOTSUPP;
+ }
+
+ if (udc_ep_ptr->dci == 0) {
+ /* EP0 stalls with a STATUS-stage TRB, not SET_HALT. */
+ if (!halt)
+ return -EOPNOTSUPP;
+
+ return crg_udc_set_ep0_halt(crg_udc);
+ }
+
+ /* A giveback can drop udc_lock while ep_nuke() still owns requests.
+ * Never resume DMA until the entire cancellation drain is finished.
+ */
+ if (udc_ep_ptr->nuking && !halt)
+ return -EBUSY;
+
+ /* Do not resume hardware while dequeue() still owns its ring. */
+ if (udc_ep_ptr->dequeue_paused && !halt)
+ return -EBUSY;
+
+ /* A failed halt cannot be cleared until DMA is safely stopped. */
+ if (udc_ep_ptr->halt_failed && !halt)
+ return -EIO;
+ recovering = udc_ep_ptr->halt_failed;
+
+ ep_state = crg_udc_get_ep_state(crg_udc, udc_ep_ptr->dci);
+
+ if (ep_state == EP_STATE_DISABLED)
+ return 0;
+
+ if (ep_state == EP_STATE_HALTED) {
+ if (halt != 0)
+ return 0;
+
+ /* want unhalt an halted ep */
+ if (udc_ep_ptr->wedge && !ignore_wedge) {
+ do_halt = -1;
+ reset_seq_only = true;
+ } else {
+ do_halt = 0;
+ }
+
+ } else {
+ /* ep state == running or stopped */
+ if (halt != 0) {
+ /* want halt a running ep */
+ do_halt = 1;
+ } else {
+ /* reset a running ep */
+ do_halt = 0;
+ reset_seq_only = true;
+ }
+ }
+
+ trace_crg_udc_ep_halt(udc_ep_ptr, do_halt == 1, udc_ep_ptr->wedge);
+
+ param0 = BIT(udc_ep_ptr->dci);
+ if (do_halt == 1) {
+ /* Stop hardware DMA before updating the ring or giving back. */
+ ret = crg_udc_issue_command(crg_udc, CRG_CMD_SET_HALT,
+ param0, 0);
+ if (!ret)
+ ret = readl_poll_timeout_atomic(&uccr->ep_running, tmp,
+ !(tmp & param0), 10, 10000);
+ if (ret) {
+ /* The controller might still own the mapped buffers. */
+ udc_ep_ptr->halt_failed = true;
+ udc_ep_ptr->ep_state = EP_STATE_STOPPED;
+ dev_err(crg_udc->dev,
+ "EP%d halt failed (%d); DMA mappings retained\n",
+ udc_ep_ptr->dci, ret);
+ return ret;
+ }
+ udc_ep_ptr->halt_failed = false;
+ udc_ep_ptr->dequeue_paused = false;
+
+ if (recovering || crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+ udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
+ udc_ep_ptr->tran_ring_full = false;
+ }
+ udc_ep_ptr->ep_state = EP_STATE_HALTED;
+ /*
+ * A failed stop retained old DMA mappings even at HS/FS.
+ * After a confirmed retry, all those requests can be given back.
+ */
+ if (recovering || crg_udc->gadget.speed >= USB_SPEED_SUPER)
+ crg_udc_ep_nuke(udc_ep_ptr, -ECONNRESET);
+ } else if (do_halt == 0) {
+ /* clearing ep halt state */
+ /* reset sequence number */
+ if (crg_udc_issue_command(crg_udc, CRG_CMD_RESET_SEQNUM, param0,
+ 0))
+ return -EIO;
+
+ if (!reset_seq_only) {
+ /* Clear halt for a halted EP. */
+ /* NOTE: we must CLEAR_HALT first,
+ * then SET_TR_DQPTR
+ */
+ ret = crg_udc_issue_command(crg_udc,
+ CRG_CMD_CLEAR_HALT, param0, 0);
+ if (ret)
+ goto clear_halt_failed;
+ ret = crg_udc_epcx_update_dqptr(udc_ep_ptr);
+ if (ret)
+ goto clear_halt_failed;
+ }
+
+ udc_ep_ptr->wedge = 0;
+ udc_ep_ptr->ep_state = EP_STATE_RUNNING;
+ /* Requests queued during HALT have DMA mappings but no TRBs.
+ * Build them before notifying the resumed controller.
+ */
+ if (!list_empty(&udc_ep_ptr->queue)) {
+ u32 generation = udc_ep_ptr->generation;
+
+ ret = crg_udc_queue_pending_trbs(udc_ep_ptr);
+ if (ret == -ENOSPC || ret == -EBUSY)
+ udc_ep_ptr->tran_ring_full = true;
+ else if (ret)
+ return ret;
+
+ /* Pending-request giveback can drop udc_lock. */
+ if (crg_udc->resetting || crg_udc->port_reset_pending ||
+ udc_ep_ptr->halt_failed ||
+ udc_ep_ptr->generation != generation ||
+ udc_ep_ptr->ep_state != EP_STATE_RUNNING ||
+ !udc_ep_ptr->desc)
+ return -ESHUTDOWN;
+ crg_udc_knock_doorbell(crg_udc, udc_ep_ptr->dci);
+ }
+ } else {
+ /* wedged EP deny CLEAR HALT */
+ dev_dbg(crg_udc->dev, "wedged EP deny CLEAR HALT dci = %d\n",
+ udc_ep_ptr->dci);
+ /* reset sequence number */
+ if (reset_seq_only &&
+ crg_udc_issue_command(crg_udc, CRG_CMD_RESET_SEQNUM, param0,
+ 0))
+ return -EIO;
+ }
+
+ return 0;
+
+clear_halt_failed:
+ /*
+ * CLEAR_HALT may have taken effect even if the following command
+ * failed. Block software queues and retain DMA mappings until a
+ * confirmed SET_HALT retry, endpoint disable, or controller reset.
+ */
+ udc_ep_ptr->halt_failed = true;
+ udc_ep_ptr->dequeue_paused = false;
+ udc_ep_ptr->ep_state = EP_STATE_STOPPED;
+ dev_err(crg_udc->dev,
+ "EP%d CLEAR_HALT recovery failed (%d); DMA mappings retained\n",
+ udc_ep_ptr->dci, ret);
+ return ret;
+}
+
+enum crg_ep_state crg_udc_get_ep_state(struct crg_gadget_dev *crg_udc, int dci)
+{
+ struct crg_udc_ep *udc_ep_ptr;
+
+ /*
+ * DCI 0 is ep0; DCI 1 is unused in the Corigine ep-context layout
+ * (ep contexts start from DCI 2 for ep1in/ep1out). This is why the
+ * ep-context array is indexed as p_epcx[dci - 2] elsewhere.
+ */
+ if (dci < 0 || dci >= CRG_NUM_EP_CX || dci == 1)
+ return EP_STATE_DISABLED;
+
+ udc_ep_ptr = &crg_udc->udc_ep[dci];
+
+ return udc_ep_ptr->ep_state;
+}
+
+u32 crg_udc_index2dci(u16 index)
+{
+ if (index == 0)
+ return 0;
+
+ return (index & USB_ENDPOINT_NUMBER_MASK) * 2 +
+ ((index & USB_DIR_IN) ? 0 : 1);
+}
+
+int crg_udc_init_device_context(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ int ret;
+ u32 buff_length;
+ dma_addr_t mapping;
+
+ /* ep0 is not included in ep contexts in crg udc */
+ buff_length = (CRG_NUM_EP_CX - 2) * sizeof(struct crg_ep_cx);
+
+ if (!crg_udc->ep_cx.cpu_addr) {
+ crg_udc->ep_cx.cpu_addr = dma_alloc_coherent(
+ crg_udc->dev, buff_length, &mapping, GFP_ATOMIC);
+ if (!crg_udc->ep_cx.cpu_addr) {
+ ret = -ENOMEM;
+ return ret;
+ }
+ } else {
+ mapping = crg_udc->ep_cx.dma_addr;
+ }
+
+ crg_udc->p_epcx = crg_udc->ep_cx.cpu_addr;
+ crg_udc->ep_cx.len = buff_length;
+ crg_udc->ep_cx.dma_addr = mapping;
+ memset(crg_udc->p_epcx, 0, buff_length);
+ /*
+ * Ensure the endpoint contexts are initialized before making their
+ * DMA address visible to the controller.
+ */
+ dma_wmb();
+ /* hw ops DCBAPLO DCBAPHI */
+ writel(lower_32_bits(crg_udc->ep_cx.dma_addr), &uccr->dcbaplo);
+ writel(upper_32_bits(crg_udc->ep_cx.dma_addr), &uccr->dcbaphi);
+
+ return 0;
+}
+
+void crg_udc_init_ep_info(struct crg_gadget_dev *crg_udc)
+{
+ int i;
+
+ /* udc_ep[0] is reserved */
+ crg_ep_struct_setup(crg_udc, 0, NULL);
+
+ for (i = 1; i < CRG_NUM_EP_CX / 2; i++) {
+ char name[14];
+
+ snprintf(name, sizeof(name), "ep%din", i);
+ crg_ep_struct_setup(crg_udc, i * 2, name);
+ snprintf(name, sizeof(name), "ep%dout", i);
+ crg_ep_struct_setup(crg_udc, i * 2 + 1, name);
+ }
+}
+
+static bool is_request_dequeued(struct crg_gadget_dev *crg_udc,
+ struct crg_udc_ep *udc_ep,
+ struct crg_event_trb *event)
+{
+ struct crg_udc_request *udc_req;
+ u32 trb_pt_lo = le32_to_cpu(event->dw0);
+ u32 trb_pt_hi = le32_to_cpu(event->dw1);
+ u64 trb_addr = (u64)trb_pt_lo | ((u64)trb_pt_hi << 32);
+ struct crg_transfer_trb *trb_pt;
+
+ if (list_empty(&udc_ep->queue))
+ return true;
+
+ trb_pt = crg_udc_tran_trb_dma_to_virt(udc_ep, trb_addr);
+ if (!trb_pt)
+ return true;
+
+ /* The transfer event handler consumes the queue head only. Do not
+ * accept an event for a later request and attribute it to the head.
+ */
+ udc_req = list_first_entry(&udc_ep->queue, struct crg_udc_request, queue);
+ if (trb_pt == udc_req->first_trb || trb_pt == udc_req->last_trb)
+ return false;
+
+ return !(crg_udc_is_pointer_less_than(trb_pt, udc_req->last_trb,
+ udc_ep) &&
+ crg_udc_is_pointer_less_than(udc_req->first_trb, trb_pt,
+ udc_ep));
+}
+
+void crg_udc_knock_doorbell(struct crg_gadget_dev *crg_udc, int dci)
+{
+ u32 tmp;
+ struct crg_uccr __iomem *uccr;
+
+ uccr = crg_udc->uccr;
+ /* Ensure everything is written before notifying the HW */
+ dma_wmb();
+
+ tmp = FIELD_PREP(CRG_U3DC_DB_TARGET_MASK, dci);
+ writel(tmp, &uccr->doorbell);
+
+ trace_crg_udc_doorbell(crg_udc, dci);
+}
+
+/*
+ * Complete a request and give it back to the gadget layer via the
+ * standard usb_gadget_giveback_request() helper.
+ *
+ * Must be called with udc_lock held and interrupts disabled.
+ *
+ * The request's ->complete() callback must not sleep. crg_udc_req_done() drops
+ * udc_lock before the giveback and re-acquires it before returning.
+ * Callers therefore cannot assume state protected by udc_lock is
+ * unchanged across this call.
+ */
+void crg_udc_req_done(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req, int status)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+
+ if (likely(udc_req->usb_req.status == -EINPROGRESS))
+ udc_req->usb_req.status = status;
+
+ trace_crg_udc_request_giveback(udc_ep, udc_req);
+
+ list_del_init(&udc_req->queue);
+
+ /*
+ * Only published payloads can still be owned by the DMA master.
+ * Move the mapping off the request before giving it back. No CPU
+ * access to the bounce pages is allowed while they are quarantined.
+ */
+ if (crg_udc->dma_quiesce_failed && udc_req->bounce &&
+ udc_req->first_trb) {
+ list_add_tail(&udc_req->bounce->quarantine,
+ &crg_udc->dma_quarantine);
+ udc_req->bounce = NULL;
+ udc_req->usb_req.status = -ESHUTDOWN;
+ udc_req->usb_req.actual = 0;
+ } else {
+ crg_udc_unmap_request(udc_ep, udc_req, udc_req->usb_req.actual);
+ }
+
+ if (udc_req->usb_req.complete) {
+ crg_udc_callback_enter(crg_udc);
+ spin_unlock(&crg_udc->udc_lock);
+ usb_gadget_giveback_request(&udc_ep->usb_ep, &udc_req->usb_req);
+ spin_lock(&crg_udc->udc_lock);
+ crg_udc_callback_exit(crg_udc);
+ }
+}
+
+/*
+ * A transfer error must stop hardware DMA before giving requests back.
+ * Keep the endpoint blocked if the stop command fails, so callers cannot
+ * queue more TRBs while old requests still own DMA mappings.
+ */
+static int crg_udc_stop_error_ep(struct crg_gadget_dev *crg_udc, u8 dci)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[dci];
+ struct crg_udc_request *req;
+ bool isoc;
+ u32 tmp;
+ int ret;
+
+ if (crg_udc->resetting || !udc_ep_ptr->desc ||
+ udc_ep_ptr->ep_state == EP_STATE_DISABLED)
+ return -ESHUTDOWN;
+
+ /* Preserve -EIO for the failed request, including after recovery. */
+ if (!list_empty(&udc_ep_ptr->queue)) {
+ req = list_first_entry(&udc_ep_ptr->queue,
+ struct crg_udc_request, queue);
+ if (req->usb_req.status == -EINPROGRESS)
+ req->usb_req.status = -EIO;
+ }
+
+ isoc = usb_endpoint_xfer_isoc(udc_ep_ptr->desc);
+ if (udc_ep_ptr->ep_state != EP_STATE_HALTED) {
+ /* Block queues while stopping, including on command failure. */
+ udc_ep_ptr->ep_state = EP_STATE_STOPPED;
+ udc_ep_ptr->dequeue_paused = false;
+ ret = crg_udc_issue_command(crg_udc,
+ isoc ? CRG_CMD_STOP_EP : CRG_CMD_SET_HALT,
+ BIT(dci), 0);
+ if (ret)
+ goto stop_failed;
+ }
+
+ ret = readl_poll_timeout_atomic(&uccr->ep_running, tmp,
+ !(tmp & BIT(dci)), 10, 10000);
+ if (ret)
+ goto stop_failed;
+
+ /* The controller is no longer reading request buffers. */
+ udc_ep_ptr->halt_failed = false;
+ udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
+ udc_ep_ptr->tran_ring_full = false;
+ udc_ep_ptr->ep_state = isoc ? EP_STATE_STOPPED : EP_STATE_HALTED;
+
+ crg_udc_ep_nuke(udc_ep_ptr, -ECONNRESET);
+
+ return 0;
+
+stop_failed:
+ udc_ep_ptr->halt_failed = true;
+ udc_ep_ptr->ep_state = EP_STATE_STOPPED;
+ crg_udc_schedule_endpoint_recovery(crg_udc);
+ return ret;
+}
+
+void crg_udc_clear_req_container(struct crg_udc_request *udc_req_ptr)
+{
+ udc_req_ptr->buff_len_left = 0;
+ udc_req_ptr->trbs_needed = 0;
+ udc_req_ptr->all_trbs_queued = 0;
+ udc_req_ptr->td_pending = false;
+ udc_req_ptr->first_trb = NULL;
+ udc_req_ptr->last_trb = NULL;
+ udc_req_ptr->td_start_offset = 0;
+ udc_req_ptr->sg = NULL;
+ udc_req_ptr->sg_index = 0;
+ udc_req_ptr->sg_offset = 0;
+}
+
+const struct usb_ep_ops crg_udc_ep_ops = {
+ .enable = crg_udc_ep_enable,
+ .disable = crg_udc_ep_disable,
+ .alloc_request = crg_udc_alloc_request,
+ .free_request = crg_udc_free_request,
+ .queue = crg_udc_ep_queue,
+ .dequeue = crg_udc_ep_dequeue,
+ .set_halt = crg_udc_ep_set_halt,
+ .set_wedge = crg_udc_ep_set_wedge,
+};
+
+const struct usb_gadget_ops crg_gadget_ops = {
+ .get_frame = crg_gadget_get_frame,
+ .wakeup = crg_gadget_wakeup,
+ .set_remote_wakeup = crg_gadget_set_remote_wakeup,
+ .set_selfpowered = crg_gadget_set_selfpowered,
+ .pullup = crg_gadget_pullup,
+ .udc_start = crg_gadget_start,
+ .udc_stop = crg_gadget_stop,
+ .udc_set_speed = crg_udc_set_speed,
+ .udc_async_callbacks = crg_gadget_async_callbacks,
+};
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-trace.c b/drivers/usb/gadget/udc/crg/crg-udc-trace.c
new file mode 100644
index 000000000000..ee091708dbe0
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-trace.c
@@ -0,0 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ * Trace Support.
+ *
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#define CREATE_TRACE_POINTS
+#include "crg-udc-trace.h"
--
2.34.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH v5 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations
2026-11-16 19:06 ` [PATCH v5 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations Petar Stepanovic
@ 2026-11-16 20:47 ` sashiko-bot
0 siblings, 0 replies; 13+ messages in thread
From: sashiko-bot @ 2026-11-16 20:47 UTC (permalink / raw)
To: Petar Stepanovic; +Cc: linux-usb, devicetree, robh, conor+dt
Thank you for your contribution! Sashiko AI review found 10 potential
issues to consider:
- [High] usb: gadget: crg: stale deq_pt in crg_udc_ep_dequeue() when
pause_pt passed TD
- [High] usb: gadget: crg: stuck USB_STATE_RECONNECTING in worker
port-status paths
- [High] usb: gadget: crg: isoc endpoints wedged in EP_STATE_STOPPED on
xfer error
- [Medium] usb: gadget: crg: broken EP0 USB_ENDPOINT_HALT handling in
crg_udc_ep_halt()
- [Medium] usb: gadget: crg: request rejected on ring-full in
crg_udc_ep_queue()
- [Medium] usb: gadget: crg: premature USB_STATE_ADDRESS in
crg_udc_ep_disable()
- [Medium] usb: gadget: udc: crg: lost port reset poll in
crg_udc_port_retry_work()
- [Medium] usb: gadget: udc: crg: controller reset on nested
crg_udc_ep_dequeue()
- [Medium] usb: gadget: udc: crg: dequeue pointer rewind in
crg_udc_ep_halt()
- [Low] usb: gadget: crg: incomplete cleanup in crg_udc_ep_disable()
when disabled
--
commit 4ce545406bad41a031bcbc10a89cb66370b3d324
Author: Petar Stepanovic <pstepanovic@axiado.com>
usb: gadget: udc: crg: Add gadget and endpoint operations
This commit adds the USB gadget and endpoint operations, transfer ring
management, port status handling, and error recovery for the Axiado
AX3000/AX3005 UDC driver.
> diff --git a/drivers/usb/gadget/udc/crg/crg-udc-gadget.c b/drivers/usb/gadget/udc/crg-udc-gadget.c
> new file mode 100644
> index 0000000000000..dddfc5cf3d3ec
> --- /dev/null
> +++ b/drivers/usb/gadget/udc/crg/crg-udc-gadget.c
[ ... ]
> +static int crg_udc_ep_dequeue(struct usb_ep *_ep, struct usb_request *_req)
> +{
[ ... ]
> + /* A stopped ring may still contain published requests. */
> + if (old_ep_state == EP_STATE_STOPPED) {
> + ret = -EBUSY;
> + goto recover;
> + }
> +
> + if (old_ep_state == EP_STATE_RUNNING) {
> + ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP,
> + BIT(dci), 0);
> +
> + if (ret)
> + goto recover;
> + ret = readl_poll_timeout_atomic(&uccr->ep_running, tmp,
> + !(tmp & BIT(dci)), 10, 10000);
> + if (ret)
> + goto recover;
> + udc_ep_ptr->ep_state = EP_STATE_STOPPED;
> + udc_ep_ptr->dequeue_paused = true;
> + }
[Severity: Medium]
Does a nested or concurrent crg_udc_ep_dequeue() trigger a full controller
recovery reset when a second request with live TRBs is dequeued during
giveback?
crg_udc_ep_dequeue(ep, req1)
ep->ep_state = EP_STATE_STOPPED
ep->dequeue_paused = true
crg_udc_req_done(ep, req1, -ECONNRESET)
spin_unlock(&crg_udc->udc_lock)
usb_gadget_giveback_request()
req1->complete()
usb_ep_dequeue(ep, req2)
crg_udc_ep_dequeue(ep, req2)
old_ep_state == EP_STATE_STOPPED
goto recover <-- schedules controller reset
When req2 already has first_trb set on the stopped ring, the inner
crg_udc_ep_dequeue() sees old_ep_state == EP_STATE_STOPPED without checking
udc_ep_ptr->dequeue_paused, marks halt_failed = true, and schedules a full
controller reset that drops the USB connection.
> + p_ep_cx = &crg_udc->p_epcx[dci - 2];
> + deq_pt_lo = le32_to_cpu(p_ep_cx->dw2) & EP_CX_TR_DQPT_LO_MASK;
> + deq_pt_hi = le32_to_cpu(p_ep_cx->dw3);
> + dq_pt_addr = (u64)deq_pt_lo | ((u64)deq_pt_hi << 32);
> + pause_pt = crg_udc_tran_trb_dma_to_virt(udc_ep_ptr, dq_pt_addr);
> + if (!pause_pt) {
> + ret = -EIO;
> + goto recover;
> + }
> +
> + if (crg_udc_is_pointer_less_than(pause_pt, udc_req->first_trb,
> + udc_ep_ptr)) {
> + crg_udc_squeeze_xfer_ring(udc_ep_ptr, udc_req);
> + /* Give back the cancelled request BEFORE scheduling the tail:
> + * its callback may add/remove requests, and dropping udc_lock
> + * while traversing the old list is unsafe.
> + */
> + crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
> + } else if (udc_req->last_trb &&
> + crg_udc_is_pointer_less_than(udc_req->last_trb, pause_pt,
> + udc_ep_ptr)) {
> + crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
> + } else {
[Severity: High]
Does the else-if branch leave udc_ep_ptr->deq_pt stuck at udc_req->first_trb
when pause_pt has already advanced past udc_req->last_trb?
When hardware has consumed the TD's TRBs and its Transfer Event is still
pending on the event ring, crg_udc_req_done() unlinks udc_req from
udc_ep_ptr->queue and unmaps udc_req->bounce with usb_req.actual still 0
(discarding received OUT data). When crg_udc_handle_xfer_event() later
processes the pending event, is_request_dequeued() sees that udc_req is no
longer on the queue and returns before calling crg_udc_update_dequeue_pt().
That leaks the request's TRB slots in crg_udc_room_on_ring(), and if the
endpoint is later halted and unhalted at High or Full Speed,
crg_udc_ep_halt() calls crg_udc_epcx_update_dqptr() and reprograms the
hardware dequeue pointer back to the unmapped TRBs.
[ ... ]
> +static int crg_udc_ep_disable(struct usb_ep *ep)
> +{
[ ... ]
> + if (udc_ep->ep_state == EP_STATE_DISABLED) {
> + /*
> + * A reset can mark this endpoint disabled before reaching its
> + * queue. A callback from another endpoint may disable it now.
> + * Give back its requests before returning success.
> + */
> + crg_udc_ep_nuke(udc_ep, -ESHUTDOWN);
> + udc_ep->desc = NULL;
> + udc_ep->comp_desc = NULL;
> + udc_ep->usb_ep.enabled = false;
> + ret = 0;
> + goto out;
> + }
[Severity: Low]
Should this path clear udc_ep->desc, udc_ep->comp_desc, and
crg_udc->p_epcx[udc_ep->dci - 2] before calling crg_udc_ep_nuke()?
When crg_udc_reinit() disables endpoints for a bus reset with notify_reset
set, it preserves desc, comp_desc, usb_ep.enabled, and p_epcx for the
gadget's usb_ep_disable() call. Taking this early branch leaves p_epcx
unzeroed (since crg_udc_reset_work() skips endpoints once usb_ep.enabled is
false) and exposes non-NULL descriptors while crg_udc_ep_nuke() drops
udc_lock for request givebacks.
[ ... ]
> + if (!crg_udc->num_enabled_eps &&
> + crg_udc->device_state == USB_STATE_CONFIGURED)
> + crg_udc->device_state = USB_STATE_ADDRESS;
[Severity: Medium]
Should disabling the last non-EP0 endpoint downgrade crg_udc->device_state
to USB_STATE_ADDRESS when the USB configuration is still active?
If a configured gadget disables all data endpoints (for example, when
switching an interface to alternate setting 0), the device is still in the
Configured state until SET_CONFIGURATION(0) or a bus reset occurs.
Setting USB_STATE_ADDRESS here causes getstatusrequest() and
setfeaturesrequest() to stall valid Configured-state control requests, and
causes crg_udc_handle_port_status() to see was_configured == false on a
subsequent bus reset and skip usb_gadget_udc_reset().
[ ... ]
> +static int crg_udc_ep_queue(struct usb_ep *_ep, struct usb_request *_req,
> + gfp_t gfp_flags)
> +{
[ ... ]
> + if (udc_ep_ptr->ep_state != EP_STATE_RUNNING) {
> + status = 0;
> + } else if (!list_empty(&udc_ep_ptr->queue)) {
> + struct crg_udc_request *tail;
> +
> + tail = list_last_entry(&udc_ep_ptr->queue,
> + struct crg_udc_request, queue);
> + status = (!tail->all_trbs_queued ||
> + udc_ep_ptr->tran_ring_full) ? 0 :
> + crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
> + } else if (udc_ep_ptr->tran_ring_full) {
> + status = 0;
> + } else {
> + status = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
> + }
> +
> + if (!status) {
> + list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
> + goto out_locked;
> + }
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +
> +unmap:
> + crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr, 0);
[Severity: Medium]
When crg_udc_build_td() returns -EBUSY or -ENOSPC, should crg_udc_ep_queue()
set tran_ring_full = true and keep the request on udc_ep_ptr->queue instead
of unmapping and failing it?
On an isochronous endpoint with fewer free ring slots than td_trbs,
crg_udc_queue_trbs() returns -EBUSY without setting tran_ring_full.
Similarly, crg_udc_ep_dequeue() clears tran_ring_full = false before calling
crg_udc_queue_pending_trbs(), which returns 0 without calling
crg_udc_build_td() if all queued requests already have all_trbs_queued or
td_pending set; the next crg_udc_ep_queue() call on that full ring then
invokes crg_udc_build_td() and fails with -ENOSPC. All other callers of
crg_udc_build_td() treat -ENOSPC and -EBUSY as ring-full conditions that
defer the request.
[ ... ]
> +void crg_udc_port_retry_work(struct work_struct *work)
> +{
> + struct crg_gadget_dev *crg_udc =
> + container_of(to_delayed_work(work), struct crg_gadget_dev,
> + port_retry_work);
> + unsigned long flags;
> + int ret;
> +
> + spin_lock_irqsave(&crg_udc->udc_lock, flags);
> + if (!crg_udc->port_reset_pending && !crg_udc->deferred_port_events)
> + goto out;
> + if (crg_udc->resetting || crg_udc->suspended ||
> + crg_udc->shutting_down || !crg_udc->pullup_on ||
> + !crg_udc->gadget_driver || !crg_udc->async_callbacks)
> + goto out;
> + /* Reset work replays deferred events after the callback. A controller
> + * reconstruction is different: retry once its register state is stable.
> + */
> + if (crg_udc->controller_recovering) {
> + mod_delayed_work(system_wq, &crg_udc->port_retry_work,
> + msecs_to_jiffies(CRG_PORT_RESET_RETRY_MS));
> + goto out;
> + }
> + if (crg_udc->port_reset_pending &&
> + crg_udc->port_reset_retry_count++ >= CRG_PORT_RESET_MAX_RETRIES) {
> + bool notify_reset = crg_udc->device_state == USB_STATE_CONFIGURED;
> +
> + dev_err(crg_udc->dev,
> + "port reset did not become ready; resetting controller\n");
> + ret = crg_udc_recover_port_failure(crg_udc, &flags,
> + notify_reset);
> + if (ret != -ECONNRESET)
> + dev_err(crg_udc->dev,
> + "port-reset recovery returned %d\n", ret);
> + goto out;
> + }
> + ret = crg_udc_handle_port_status(crg_udc, &flags);
> + if (ret && ret != -ECONNRESET)
> + dev_dbg(crg_udc->dev, "port reset poll: %d\n", ret);
> +out:
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +}
[Severity: Medium]
Can a port reset poll be permanently lost if crg_udc_port_retry_work() runs
while crg_udc->reset_callback_active is true?
kworker (udc_reset_work) kworker (port_retry_work)
crg_udc_reset_work()
reset_callback_active = true
spin_unlock(&udc_lock)
usb_gadget_udc_reset()
crg_udc_port_retry_work()
crg_udc_handle_port_status()
portsc_val & W1C_MASK == 0
if (reset_callback_active)
deferred_port_events |= 0
return 0
spin_lock(&udc_lock)
reset_callback_active = false
if (deferred_port_events) <-- false; port_reset_pending stays true
Because CRG_U3DC_PORTSC_PRC was already W1C-cleared when port_reset_pending
was first set, crg_udc_handle_port_status() sees no new W1C bits during
reset_callback_active and returns early without rescheduling
port_retry_work. crg_udc_reset_work() then only checks
deferred_port_events, leaving port_reset_pending stuck true and blocking EP0
setup handling, endpoint enablement, and request queueing. In addition, if
port_reset_retry_count reaches CRG_PORT_RESET_MAX_RETRIES while
reset_callback_active is true, crg_udc_port_retry_work() runs
crg_udc_recover_port_failure() concurrently with usb_gadget_udc_reset().
[ ... ]
> +int crg_udc_handle_port_status(struct crg_gadget_dev *crg_udc,
> + unsigned long *flags)
> +{
[ ... ]
> + if (crg_udc->device_state >= USB_STATE_DEFAULT) {
[ ... ]
> + reinit_ret = crg_udc_reinit(crg_udc, notify_reset);
[ ... ]
> + }
> +
> + crg_udc->gadget.speed = speed;
> +
> + ret = crg_udc_update_ep0_maxpacketsize(crg_udc);
> + if (ret) {
> + dev_err(crg_udc->dev,
> + "failed to configure EP0 maxpacket: %d\n", ret);
> + return crg_udc_recover_port_failure(
> + crg_udc, flags,
> + crg_udc->bus_reset_pending ||
> + crg_udc->device_state == USB_STATE_CONFIGURED);
> + }
> +
> + crg_udc->connected = 1;
> +
> + if (crg_udc->device_state < USB_STATE_RECONNECTING)
> + crg_udc_enable_setup(crg_udc);
> + }
> + }
[Severity: High]
Does crg_udc->device_state remain stuck in USB_STATE_RECONNECTING when a
port reset completes via crg_udc_port_retry_work() or crg_udc_reset_work(),
or after a link drop with VBUS present?
When device_state >= USB_STATE_DEFAULT, crg_udc_reinit() sets device_state
to USB_STATE_RECONNECTING, so crg_udc_handle_port_status() skips
crg_udc_enable_setup() expecting the caller to drain the event ring and call
crg_udc_prepare_for_setup(). Neither crg_udc_port_retry_work() nor
crg_udc_reset_work() calls crg_udc_prepare_for_setup() after
crg_udc_handle_port_status(), and the link-drop path (cable_connected &&
!ccs_drop_ignore) also calls crg_udc_reinit() with connected = 0 so a later
CSC/PEC/PPC reconnect skips crg_udc_enable_setup() as well.
Also, while port_reset_pending is true before crg_udc_reinit() runs,
device_state is not yet USB_STATE_RECONNECTING; if a new SETUP event arrives
before port_retry_work polls PED, crg_udc_handle_event() dispatches it to
crg_udc_handle_setup_pkt(), which sets ep0_state = EP0_STATE_ABORTING and
returns early on port_reset_pending, dropping the SETUP packet and wedging
EP0.
[ ... ]
> +int crg_udc_ep_halt(struct crg_udc_ep *udc_ep_ptr, int halt, int ignore_wedge,
> + unsigned long *flags)
> +{
[ ... ]
> + if (udc_ep_ptr->dci == 0) {
> + /* EP0 stalls with a STATUS-stage TRB, not SET_HALT. */
> + if (!halt)
> + return -EOPNOTSUPP;
> +
> + return crg_udc_set_ep0_halt(crg_udc);
> + }
[Severity: Medium]
Does this break USB_REQ_SET_FEATURE and USB_REQ_CLEAR_FEATURE(ENDPOINT_HALT)
targeting EP0 in setfeaturesrequest()?
drivers/usb/gadget/udc/crg/crg-udc-ep0.c:setfeaturesrequest() {
...
udc_ep_ptr = &crg_udc->udc_ep[dci];
status = crg_udc_ep_halt(
udc_ep_ptr, (b_request == USB_REQ_SET_FEATURE) ? 1 : 0,
0, flags);
if (status < 0)
goto set_feature_error;
...
crg_udc->setup_status = STATUS_STAGE_XFER;
crg_udc_submit_ep0_status(crg_udc, status, 0);
}
On CLEAR_FEATURE(USB_ENDPOINT_HALT) for EP0, returning -EOPNOTSUPP causes
setfeaturesrequest() to stall EP0 instead of completing the status stage.
On SET_FEATURE(USB_ENDPOINT_HALT) for EP0, crg_udc_set_ep0_halt() queues a
STALL status TRB using crg_udc->status_req, sets ep_state = EP_STATE_HALTED,
and returns 0; setfeaturesrequest() then calls crg_udc_submit_ep0_status() a
second time, which fails on the halted endpoint and sets ep0_state =
EP0_STATE_ERROR so crg_udc_handle_xfer_event() ignores the STALL completion.
[ ... ]
> + if (recovering || crg_udc->gadget.speed >= USB_SPEED_SUPER) {
> + udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
> + udc_ep_ptr->tran_ring_full = false;
> + }
> + udc_ep_ptr->ep_state = EP_STATE_HALTED;
> + /*
> + * A failed stop retained old DMA mappings even at HS/FS.
> + * After a confirmed retry, all those requests can be given back.
> + */
> + if (recovering || crg_udc->gadget.speed >= USB_SPEED_SUPER)
> + crg_udc_ep_nuke(udc_ep_ptr, -ECONNRESET);
> + } else if (do_halt == 0) {
> + /* clearing ep halt state */
> + /* reset sequence number */
> + if (crg_udc_issue_command(crg_udc, CRG_CMD_RESET_SEQNUM, param0,
> + 0))
> + return -EIO;
> +
> + if (!reset_seq_only) {
> + /* Clear halt for a halted EP. */
> + /* NOTE: we must CLEAR_HALT first,
> + * then SET_TR_DQPTR
> + */
> + ret = crg_udc_issue_command(crg_udc,
> + CRG_CMD_CLEAR_HALT, param0, 0);
> + if (ret)
> + goto clear_halt_failed;
> + ret = crg_udc_epcx_update_dqptr(udc_ep_ptr);
> + if (ret)
> + goto clear_halt_failed;
> + }
[Severity: Medium]
Can clearing a halt at High or Full Speed rewind the hardware dequeue
pointer to the start of a partially completed multi-TRB request?
Chained TRBs before the final TRB of a TD have ioc = false, so hardware
advances TR_DQPT in p_epcx as each TRB completes while udc_ep_ptr->deq_pt
remains at first_trb. At High or Full Speed, halting the endpoint retains
the in-flight request without updating udc_ep_ptr->deq_pt, and clearing the
halt then calls crg_udc_epcx_update_dqptr(), overwriting the hardware
dequeue pointer with udc_ep_ptr->deq_pt and re-transferring the already
completed TRBs.
[ ... ]
> +static int crg_udc_stop_error_ep(struct crg_gadget_dev *crg_udc, u8 dci)
> +{
[ ... ]
> + /* The controller is no longer reading request buffers. */
> + udc_ep_ptr->halt_failed = false;
> + udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
> + udc_ep_ptr->tran_ring_full = false;
> + udc_ep_ptr->ep_state = isoc ? EP_STATE_STOPPED : EP_STATE_HALTED;
> +
> + crg_udc_ep_nuke(udc_ep_ptr, -ECONNRESET);
> +
> + return 0;
[Severity: High]
Does setting ep_state = EP_STATE_STOPPED here permanently wedge isochronous
endpoints after any transfer error event?
With ep_state == EP_STATE_STOPPED and dequeue_paused == false,
crg_udc_ep_can_queue() returns false and crg_udc_ep_queue() rejects all
subsequent requests with -ESHUTDOWN. Unlike bulk and interrupt endpoints,
crg_udc_ep_halt() rejects isochronous endpoints with -EOPNOTSUPP, and
udc_ep_ptr->deq_pt is advanced to enq_pt in software without calling
crg_udc_epcx_update_dqptr(), leaving no way for the gadget to resume the
isochronous stream without disabling and re-enabling the endpoint.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261116-axiado-ax3000-usb-device-controller-v5-0-49271ed5c793@axiado.com?part=5
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v5 6/6] usb: gadget: udc: crg: Add core driver and build integration
2026-11-16 19:06 [PATCH v5 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
` (4 preceding siblings ...)
2026-11-16 19:06 ` [PATCH v5 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations Petar Stepanovic
@ 2026-11-16 19:06 ` Petar Stepanovic
2026-11-16 21:15 ` sashiko-bot
5 siblings, 1 reply; 13+ messages in thread
From: Petar Stepanovic @ 2026-11-16 19:06 UTC (permalink / raw)
To: Prasad Bolisetty, Greg Kroah-Hartman, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Tzu-Hao Wei,
Karthikeyan Mitran, Philipp Zabel, Neill Kapron
Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
Petar Stepanovic, Krutik Shah, Naveen Kumar Rajgiri Bassappa
This patch introduces the core probe/remove functions for the Axiado
AX3000/AX3005 UDC driver, and wires up the directory into the build
system via Kconfig and Makefile.
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Tzu-Hao Wei <twei@axiado.com>
Signed-off-by: Tzu-Hao Wei <twei@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
drivers/usb/gadget/udc/Kconfig | 2 +
drivers/usb/gadget/udc/Makefile | 1 +
drivers/usb/gadget/udc/crg/Kconfig | 16 +
drivers/usb/gadget/udc/crg/Makefile | 9 +
drivers/usb/gadget/udc/crg/crg-udc-core.c | 1376 +++++++++++++++++++++++++++++
5 files changed, 1404 insertions(+)
diff --git a/drivers/usb/gadget/udc/Kconfig b/drivers/usb/gadget/udc/Kconfig
index 7ddbc2606686..72a5a8a04982 100644
--- a/drivers/usb/gadget/udc/Kconfig
+++ b/drivers/usb/gadget/udc/Kconfig
@@ -421,6 +421,8 @@ source "drivers/usb/gadget/udc/aspeed-vhub/Kconfig"
source "drivers/usb/gadget/udc/cdns2/Kconfig"
+source "drivers/usb/gadget/udc/crg/Kconfig"
+
#
# LAST -- dummy/emulated controller
#
diff --git a/drivers/usb/gadget/udc/Makefile b/drivers/usb/gadget/udc/Makefile
index 1b9b1a4f9c57..227d2a0b525b 100644
--- a/drivers/usb/gadget/udc/Makefile
+++ b/drivers/usb/gadget/udc/Makefile
@@ -38,3 +38,4 @@ obj-$(CONFIG_USB_ASPEED_UDC) += aspeed_udc.o
obj-$(CONFIG_USB_BDC_UDC) += bdc/
obj-$(CONFIG_USB_MAX3420_UDC) += max3420_udc.o
obj-$(CONFIG_USB_CDNS2_UDC) += cdns2/
+obj-$(CONFIG_USB_CRG_UDC) += crg/
diff --git a/drivers/usb/gadget/udc/crg/Kconfig b/drivers/usb/gadget/udc/crg/Kconfig
new file mode 100644
index 000000000000..7167c4943389
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/Kconfig
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: GPL-2.0
+config USB_CRG_UDC
+ tristate "Corigine / Axiado USB Device Controller"
+ depends on USB_GADGET
+ depends on HAS_DMA
+ depends on ARCH_AXIADO || COMPILE_TEST
+ help
+ Say Y here to support the USB device controller found on Axiado AX3000
+ and AX3005 SoCs, which is based on the Corigine USB 3.1 IP.
+
+ This controller supports High-Speed and SuperSpeed.
+
+ Say "y" to link the driver statically, or "m" to build a
+ dynamically linked module called "crg-udc.ko".
+
+ If unsure, say N.
diff --git a/drivers/usb/gadget/udc/crg/Makefile b/drivers/usb/gadget/udc/crg/Makefile
new file mode 100644
index 000000000000..96b209fb59fc
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/Makefile
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: GPL-2.0
+
+# define_trace.h needs to know how to find our trace header
+CFLAGS_crg-udc-trace.o := -I$(src)
+
+obj-$(CONFIG_USB_CRG_UDC) += crg-udc.o
+
+crg-udc-y := crg-udc-core.o crg-udc-ring.o crg-udc-ep0.o crg-udc-gadget.o
+crg-udc-$(CONFIG_TRACING) += crg-udc-trace.o
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-core.c b/drivers/usb/gadget/udc/crg/crg-udc-core.c
new file mode 100644
index 000000000000..e508dbd45339
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-core.c
@@ -0,0 +1,1376 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/clk.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/iopoll.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/phy/phy.h>
+#include <linux/platform_device.h>
+#include <linux/pm.h>
+#include <linux/property.h>
+#include <linux/reset.h>
+#include <linux/slab.h>
+
+#include "crg-udc.h"
+
+static int crg_gadget_irq_init(struct platform_device *pdev,
+ struct crg_gadget_dev *crg_udc);
+static void crg_udc_phy_exit(void *data);
+static void crg_udc_recovery_work(struct work_struct *work);
+static int crg_udc_reset(struct crg_gadget_dev *crg_udc, unsigned long *flags);
+
+static int crg_udc_alloc_ep_rings(struct crg_gadget_dev *crg_udc)
+{
+ int i;
+
+ /*
+ * Preallocate two disjoint rings per endpoint. Endpoint enable and
+ * controller recovery can run in atomic context.
+ */
+ for (i = 0; i < CRG_NUM_EP_CX; i++) {
+ struct crg_buffer_info *ring = &crg_udc->udc_ep[i].tran_ring_alloc;
+ size_t len = (i ? CRG_UDC_BULK_EP_TD_RING_SIZE :
+ CRG_UDC_CONTROL_EP_TD_RING_SIZE) *
+ sizeof(struct crg_transfer_trb);
+
+ if (i == 1)
+ continue;
+
+ ring->cpu_addr = dma_alloc_coherent(crg_udc->dev, 2 * len,
+ &ring->dma_addr, GFP_KERNEL);
+ if (!ring->cpu_addr)
+ return -ENOMEM;
+ ring->len = 2 * len;
+ }
+
+ return 0;
+}
+
+static void crg_udc_select_transfer_rings(struct crg_gadget_dev *crg_udc)
+{
+ int i;
+
+ /*
+ * Transfer events identify TRBs by DMA address. After confirmed
+ * quiescence, use the other bank for the next session so a completion
+ * for the previous bank cannot match a new request. Keep the selected
+ * bank if a rebuild failed before RUN was set. Reusing a bank from
+ * older sessions relies on the halt/reset and event-ring clear.
+ */
+ if (crg_udc->transfer_rings_live) {
+ crg_udc->transfer_ring_bank = !crg_udc->transfer_ring_bank;
+ crg_udc->transfer_rings_live = false;
+ }
+
+ for (i = 0; i < CRG_NUM_EP_CX; i++) {
+ struct crg_udc_ep *ep = &crg_udc->udc_ep[i];
+ struct crg_buffer_info *alloc = &ep->tran_ring_alloc;
+ u32 len = alloc->len / 2;
+ u32 offset = crg_udc->transfer_ring_bank ? len : 0;
+ void *addr;
+
+ if (!alloc->cpu_addr)
+ continue;
+
+ addr = (u8 *)alloc->cpu_addr + offset;
+ if (ep->tran_ring_info.cpu_addr == addr)
+ continue;
+
+ ep->tran_ring_info.cpu_addr = addr;
+ ep->tran_ring_info.dma_addr = alloc->dma_addr + offset;
+ ep->tran_ring_info.len = len;
+ ep->first_trb = NULL;
+ ep->last_trb = NULL;
+ ep->enq_pt = NULL;
+ ep->deq_pt = NULL;
+ }
+}
+
+static void crg_udc_gadget_release(struct device *dev)
+{
+ struct crg_gadget_dev *crg_udc;
+
+ crg_udc = container_of(dev, struct crg_gadget_dev, gadget.dev);
+ kfree(crg_udc);
+}
+
+/*
+ * command related ops
+ */
+int crg_udc_issue_command(struct crg_gadget_dev *crg_udc,
+ enum crg_cmd_type type, u32 param0, u32 param1)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 status;
+ u32 tmp;
+ bool running;
+
+ running = !!(readl(&uccr->control) & CRG_U3DC_CTRL_RUN);
+ /* Only INIT_EP0 may be staged while the controller is stopped. */
+ if (!running && type != CRG_CMD_INIT_EP0)
+ return -ESHUTDOWN;
+
+ /*
+ * A staged command must complete before the command registers can
+ * be reused. Hardware does not consume commands while RUN is clear.
+ */
+ tmp = readl(&uccr->cmd_control);
+ if (tmp & CRG_U3DC_CMD_CTRL_ACTIVE) {
+ if (!running)
+ return -EBUSY;
+ if (readl_poll_timeout_atomic(&uccr->cmd_control, tmp,
+ !(tmp & CRG_U3DC_CMD_CTRL_ACTIVE),
+ 1, 10000))
+ return -ETIMEDOUT;
+ }
+ /* Ensure that everything is written before issuing new command */
+ dma_wmb();
+
+ writel(param0, &uccr->cmd_param0);
+ writel(param1, &uccr->cmd_param1);
+
+ tmp = CRG_U3DC_CMD_CTRL_ACTIVE |
+ FIELD_PREP(CRG_U3DC_CMD_CTRL_TYPE_MASK, type);
+ writel(tmp, &uccr->cmd_control);
+
+ if (!running)
+ return 0;
+
+ if (readl_poll_timeout_atomic(&uccr->cmd_control, tmp,
+ !(tmp & CRG_U3DC_CMD_CTRL_ACTIVE), 1,
+ 1000)) {
+ dev_dbg(crg_udc->dev, "command timed out\n");
+ return -ETIMEDOUT;
+ }
+
+ status = FIELD_GET(CRG_U3DC_CMD_CTRL_STATUS_MASK, tmp);
+ if (status) {
+ dev_dbg(crg_udc->dev, "command failed with status %u\n",
+ status);
+ return -EIO;
+ }
+
+ return 0;
+}
+
+static void crg_udc_clk_disable(void *data)
+{
+ clk_disable_unprepare(data);
+}
+
+/*
+ * Stop the controller and wait for pending event writes before resetting it.
+ * Corigine USB 3.1 DRD v1.5, sections 8.3.3 and 8.3.4: Controller Halted
+ * requires Controller Idle, at which point queued events have reached the
+ * event ring. Discard those events when rebuilding the ring after reset.
+ */
+static int crg_udc_soft_reset(struct crg_gadget_dev *crg_udc, bool atomic)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 ctrl, val;
+ int ret;
+
+ ctrl = readl(&uccr->control);
+ ctrl &= ~(CRG_U3DC_CTRL_RUN | CRG_U3DC_CTRL_INT_EN |
+ CRG_U3DC_CTRL_SYSERR_EN | CRG_U3DC_CTRL_SWRST);
+ writel(ctrl, &uccr->control);
+
+ if (atomic)
+ ret = readl_poll_timeout_atomic(&uccr->status, val,
+ val & CRG_U3DC_STATUS_DEV_CTRL_HALT,
+ 10, 10000);
+ else
+ ret = readl_poll_timeout(&uccr->status, val,
+ val & CRG_U3DC_STATUS_DEV_CTRL_HALT,
+ 10, 10000);
+ if (ret) {
+ dev_err(crg_udc->dev, "controller halt timed out before reset\n");
+ return ret;
+ }
+
+ writel(ctrl | CRG_U3DC_CTRL_SWRST, &uccr->control);
+ if (atomic)
+ ret = readl_poll_timeout_atomic(&uccr->control, val,
+ !(val & CRG_U3DC_CTRL_SWRST), 10, 10000);
+ else
+ ret = readl_poll_timeout(&uccr->control, val,
+ !(val & CRG_U3DC_CTRL_SWRST), 10, 10000);
+ if (ret)
+ dev_err(crg_udc->dev, "software reset timed out\n");
+
+ return ret;
+}
+
+/*
+ * Stop bus-master DMA with new submissions blocked. This can sleep.
+ * An asserted SoC reset is a fallback when halting or SWRST times out.
+ */
+static int crg_udc_quiesce_hw(struct crg_gadget_dev *crg_udc)
+{
+ int ret;
+
+ if (crg_udc->hw_reset_asserted || !crg_udc->hw_dma_armed)
+ return 0;
+ if (!crg_udc->uccr)
+ return -ENODEV;
+
+ ret = crg_udc_soft_reset(crg_udc, false);
+ if (!ret) {
+ crg_udc->hw_dma_armed = false;
+ return 0;
+ }
+
+ if (!crg_udc->rst)
+ return ret;
+
+ ret = reset_control_assert(crg_udc->rst);
+ if (!ret) {
+ crg_udc->hw_reset_asserted = true;
+ crg_udc->hw_dma_armed = false;
+ }
+
+ return ret;
+}
+
+static void crg_udc_free_resources(struct crg_gadget_dev *crg_udc)
+{
+ unsigned long flags;
+ int i, ret;
+
+ /* Block any in-flight threaded handler before releasing DMA memory. */
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ crg_udc->shutting_down = true;
+ if (crg_udc->uccr && !crg_udc->hw_reset_asserted &&
+ crg_udc->hw_dma_armed) {
+ u32 val = readl(&crg_udc->uccr->control);
+
+ val &= ~(CRG_U3DC_CTRL_INT_EN | CRG_U3DC_CTRL_RUN |
+ CRG_U3DC_CTRL_SYSERR_EN);
+ writel(val, &crg_udc->uccr->control);
+ }
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ if (crg_udc->irq >= 0) {
+ synchronize_irq(crg_udc->irq);
+ devm_free_irq(crg_udc->dev, crg_udc->irq, crg_udc);
+ crg_udc->irq = -1;
+ crg_udc->gadget.irq = 0;
+ }
+
+ if (crg_udc->status_req) {
+ kfree(crg_udc->status_req->usb_req.buf);
+ crg_udc_free_request(&crg_udc->udc_ep[0].usb_ep,
+ &crg_udc->status_req->usb_req);
+ crg_udc->status_req = NULL;
+ }
+
+ ret = crg_udc_quiesce_hw(crg_udc);
+ if (ret) {
+ crg_udc->dma_quiesce_failed = true;
+ /*
+ * Retain potentially live DMA mappings and coherent allocations.
+ * A platform with detachable DMA translations must isolate the
+ * master independently before tearing down its DMA domain.
+ */
+ dev_crit(crg_udc->dev,
+ "DMA stop failed (%d); retaining allocations, platform isolation required\n",
+ ret);
+ return;
+ }
+ crg_udc->dma_quiesce_failed = false;
+ crg_udc_release_quarantine(crg_udc);
+
+ for (i = 0; i < CRG_RING_NUM; i++) {
+ struct crg_udc_event *udc_event = &crg_udc->udc_event[i];
+
+ if (udc_event->event_ring.cpu_addr) {
+ dma_free_coherent(crg_udc->dev,
+ udc_event->event_ring.len,
+ udc_event->event_ring.cpu_addr,
+ udc_event->event_ring.dma_addr);
+ udc_event->event_ring.cpu_addr = NULL;
+ udc_event->event_ring.dma_addr = DMA_MAPPING_ERROR;
+ udc_event->event_ring.len = 0;
+ udc_event->evt_dq_pt = NULL;
+ }
+ if (udc_event->erst.cpu_addr) {
+ dma_free_coherent(crg_udc->dev, udc_event->erst.len,
+ udc_event->erst.cpu_addr,
+ udc_event->erst.dma_addr);
+ udc_event->erst.cpu_addr = NULL;
+ udc_event->erst.dma_addr = DMA_MAPPING_ERROR;
+ udc_event->erst.len = 0;
+ udc_event->p_erst = NULL;
+ }
+ }
+
+ if (crg_udc->ep_cx.cpu_addr) {
+ dma_free_coherent(crg_udc->dev, crg_udc->ep_cx.len,
+ crg_udc->ep_cx.cpu_addr,
+ crg_udc->ep_cx.dma_addr);
+ crg_udc->ep_cx.cpu_addr = NULL;
+ crg_udc->ep_cx.dma_addr = DMA_MAPPING_ERROR;
+ crg_udc->ep_cx.len = 0;
+ crg_udc->p_epcx = NULL;
+ }
+
+ for (i = 0; i < CRG_NUM_EP_CX; i++) {
+ struct crg_udc_ep *udc_ep = &crg_udc->udc_ep[i];
+ struct crg_buffer_info *alloc = &udc_ep->tran_ring_alloc;
+
+ if (alloc->cpu_addr) {
+ dma_free_coherent(crg_udc->dev, alloc->len,
+ alloc->cpu_addr, alloc->dma_addr);
+ *alloc = (struct crg_buffer_info){};
+ udc_ep->tran_ring_info.cpu_addr = NULL;
+ udc_ep->tran_ring_info.dma_addr = DMA_MAPPING_ERROR;
+ udc_ep->tran_ring_info.len = 0;
+ udc_ep->first_trb = NULL;
+ udc_ep->last_trb = NULL;
+ udc_ep->enq_pt = NULL;
+ udc_ep->deq_pt = NULL;
+ }
+ }
+}
+
+static int crg_udc_init_hw(struct crg_gadget_dev *crg_udc)
+{
+ struct device *dev = crg_udc->dev;
+ struct reset_control *rst;
+ int ret;
+
+ crg_udc->clk = devm_clk_get_optional(dev, NULL);
+ if (IS_ERR(crg_udc->clk))
+ return dev_err_probe(dev, PTR_ERR(crg_udc->clk),
+ "failed to get clock\n");
+
+ ret = clk_prepare_enable(crg_udc->clk);
+ if (ret)
+ return dev_err_probe(dev, ret, "failed to enable clock\n");
+
+ ret = devm_add_action_or_reset(dev, crg_udc_clk_disable, crg_udc->clk);
+ if (ret)
+ return ret;
+
+ rst = devm_reset_control_get_optional_exclusive_deasserted(dev, NULL);
+ if (IS_ERR(rst))
+ return dev_err_probe(dev, PTR_ERR(rst),
+ "failed to get reset\n");
+ crg_udc->rst = rst;
+
+ crg_udc->phy = devm_phy_optional_get(dev, NULL);
+ if (IS_ERR(crg_udc->phy))
+ return dev_err_probe(dev, PTR_ERR(crg_udc->phy),
+ "failed to get phy\n");
+
+ ret = phy_init(crg_udc->phy);
+ if (ret)
+ return dev_err_probe(dev, ret, "failed to init phy\n");
+
+ ret = phy_power_on(crg_udc->phy);
+ if (ret) {
+ phy_exit(crg_udc->phy);
+ return dev_err_probe(dev, ret, "failed to power on phy\n");
+ }
+
+ return devm_add_action_or_reset(dev, crg_udc_phy_exit, crg_udc->phy);
+}
+
+static void crg_udc_phy_exit(void *data)
+{
+ struct phy *phy = data;
+
+ phy_power_off(phy);
+ phy_exit(phy);
+}
+
+static int crg_udc_probe(struct platform_device *pdev)
+{
+ unsigned long flags = 0;
+ int ret;
+ int i;
+ struct crg_gadget_dev *crg_udc;
+ const struct crg_udc_priv *priv;
+
+ priv = device_get_match_data(&pdev->dev);
+ if (!priv)
+ return -ENODEV;
+
+ crg_udc = kzalloc_obj(*crg_udc);
+ if (!crg_udc)
+ return -ENOMEM;
+ usb_initialize_gadget(&pdev->dev, &crg_udc->gadget,
+ crg_udc_gadget_release);
+
+ crg_udc->dev = &pdev->dev;
+ crg_udc->irq = -1;
+ /* Probe errors can call free_resources() before hardware init. */
+ spin_lock_init(&crg_udc->udc_lock);
+ init_waitqueue_head(&crg_udc->teardown_wait);
+ INIT_LIST_HEAD(&crg_udc->dma_quarantine);
+
+ ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64));
+ if (ret) {
+ ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32));
+ if (ret) {
+ dev_err(&pdev->dev, "no usable DMA mask\n");
+ goto err_free_resources;
+ }
+ }
+
+ ret = crg_udc_init_hw(crg_udc);
+ if (ret)
+ goto err_free_resources;
+
+ crg_udc->ep0_desc.bLength = USB_DT_ENDPOINT_SIZE;
+ crg_udc->ep0_desc.bDescriptorType = USB_DT_ENDPOINT;
+ crg_udc->ep0_desc.bEndpointAddress = 0;
+ crg_udc->ep0_desc.bmAttributes = USB_ENDPOINT_XFER_CONTROL;
+ crg_udc->ep0_desc.wMaxPacketSize = cpu_to_le16(64);
+
+ crg_udc->u12_forbidden = priv->quirks & CRG_QUIRK_DIS_U1U2_ENTRY;
+ crg_udc->u12_initiate_forbidden = priv->quirks &
+ CRG_QUIRK_DIS_U1U2_INITIATION;
+ crg_udc->reinit_ep0_on_bus_reset = priv->quirks &
+ CRG_QUIRK_REINIT_EP0_ON_RESET;
+
+ INIT_WORK(&crg_udc->udc_reset_work, crg_udc_reset_work);
+ INIT_WORK(&crg_udc->recovery_work, crg_udc_recovery_work);
+ INIT_DELAYED_WORK(&crg_udc->port_retry_work, crg_udc_port_retry_work);
+ platform_set_drvdata(pdev, crg_udc);
+
+ crg_udc->gadget.ops = &crg_gadget_ops;
+ crg_udc->gadget.ep0 = &crg_udc->udc_ep[0].usb_ep;
+ INIT_LIST_HEAD(&crg_udc->gadget.ep_list);
+ /*
+ * The controller supports up to SuperSpeed. Honour the standard
+ * "maximum-speed" property to cap it; default to the hardware
+ * maximum when the property is absent.
+ */
+ crg_udc->gadget.max_speed = usb_get_maximum_speed(&pdev->dev);
+ /* Hardware only supports Full-, High-, and SuperSpeed. */
+ switch (crg_udc->gadget.max_speed) {
+ case USB_SPEED_FULL:
+ case USB_SPEED_HIGH:
+ case USB_SPEED_SUPER:
+ break;
+ case USB_SPEED_LOW:
+ crg_udc->gadget.max_speed = USB_SPEED_FULL;
+ break;
+ default:
+ crg_udc->gadget.max_speed = USB_SPEED_SUPER;
+ break;
+ }
+ crg_udc->max_speed = crg_udc->gadget.max_speed;
+ crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+ crg_udc->gadget.name = KBUILD_MODNAME;
+ crg_udc->gadget.sg_supported = true;
+ crg_udc->gadget.wakeup_capable = true;
+ crg_udc->connected = 0;
+ crg_udc->dev_addr = 0;
+
+ crg_udc->mmio_virt_base = devm_platform_ioremap_resource(pdev, 0);
+ if (IS_ERR(crg_udc->mmio_virt_base)) {
+ ret = dev_err_probe(&pdev->dev,
+ PTR_ERR(crg_udc->mmio_virt_base),
+ "failed to map registers\n");
+ goto err_put_gadget;
+ }
+
+ /* set controller device role */
+ writel((readl(crg_udc->mmio_virt_base + CRG_UDC_MODE_REG) |
+ CRG_UDC_ROLE_DEVICE),
+ crg_udc->mmio_virt_base + CRG_UDC_MODE_REG);
+ for (i = 0; i < CRG_RING_NUM; i++) {
+ crg_udc->uicr[i] = crg_udc->mmio_virt_base + CRG_UICR_OFFSET +
+ i * CRG_UICR_STRIDE;
+ }
+ crg_udc->uccr = crg_udc->mmio_virt_base + CRG_UCCR_OFFSET;
+
+ /* Initialize endpoint objects before reset, DMA setup or any IRQ. */
+ crg_udc_init_ep_info(crg_udc);
+
+ ret = crg_udc_alloc_ep_rings(crg_udc);
+ if (ret)
+ goto err_free_resources;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ ret = crg_udc_reset(crg_udc, &flags);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ if (ret)
+ goto err_free_resources;
+
+ crg_udc_clear_portpm(crg_udc);
+
+ ret = crg_udc_reset_data_struct(crg_udc);
+ if (ret) {
+ dev_err(crg_udc->dev, "crg_udc_reset_data_struct error\n");
+ goto err_free_resources;
+ }
+
+ ret = crg_udc_init_ep0(crg_udc);
+ if (ret)
+ goto err_free_resources;
+
+ crg_udc->udc_ep[0].desc = &crg_udc->ep0_desc;
+ /* Release the reset gate only after EP0 and all event rings exist. */
+ crg_udc->resetting = false;
+
+ ret = crg_gadget_irq_init(pdev, crg_udc);
+ if (ret)
+ goto err_free_resources;
+
+ ret = usb_add_gadget(&crg_udc->gadget);
+ if (ret)
+ goto err_free_resources;
+
+ return 0;
+
+err_free_resources:
+ crg_udc_free_resources(crg_udc);
+err_put_gadget:
+ usb_put_gadget(&crg_udc->gadget);
+ return ret;
+}
+
+static void crg_udc_remove(struct platform_device *pdev)
+{
+ struct crg_gadget_dev *crg_udc = platform_get_drvdata(pdev);
+ unsigned long flags;
+ u32 tmp;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ crg_udc->shutting_down = true;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ /* The gadget core calls pullup(0) before the disconnect callback. */
+ usb_del_gadget(&crg_udc->gadget);
+ cancel_delayed_work_sync(&crg_udc->port_retry_work);
+ cancel_work_sync(&crg_udc->recovery_work);
+ cancel_work_sync(&crg_udc->udc_reset_work);
+ crg_udc_free_resources(crg_udc);
+
+ /* Do not access registers while the external reset is asserted. */
+ if (!crg_udc->hw_reset_asserted && !crg_udc->dma_quiesce_failed) {
+ tmp = readl(crg_udc->mmio_virt_base + CRG_UDC_MODE_REG) & ~0x1;
+ writel(tmp, crg_udc->mmio_virt_base + CRG_UDC_MODE_REG);
+ }
+
+ platform_set_drvdata(pdev, NULL);
+ usb_put_gadget(&crg_udc->gadget);
+}
+
+static void crg_udc_shutdown(struct platform_device *pdev)
+{
+ struct crg_gadget_dev *crg_udc = platform_get_drvdata(pdev);
+ unsigned long flags;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ crg_udc->pullup_on = false;
+ crg_udc->shutting_down = true;
+ crg_udc->resetting = true;
+ crg_udc->device_state = USB_STATE_ATTACHED;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ if (crg_udc->irq >= 0)
+ synchronize_irq(crg_udc->irq);
+ cancel_delayed_work_sync(&crg_udc->port_retry_work);
+ cancel_work_sync(&crg_udc->recovery_work);
+ cancel_work_sync(&crg_udc->udc_reset_work);
+
+ /* Reset and gadget stop may own the registers across a lock drop. */
+ spin_lock_irq(&crg_udc->udc_lock);
+ wait_event_lock_irq(crg_udc->teardown_wait,
+ !crg_udc->controller_recovering, crg_udc->udc_lock);
+ if (crg_udc->uccr && !crg_udc->hw_reset_asserted) {
+ u32 val = readl(&crg_udc->uccr->control);
+
+ crg_udc_set_vbus_detect(crg_udc, false);
+ val &= ~(CRG_U3DC_CTRL_INT_EN | CRG_U3DC_CTRL_RUN |
+ CRG_U3DC_CTRL_SYSERR_EN);
+ writel(val, &crg_udc->uccr->control);
+ }
+ spin_unlock_irq(&crg_udc->udc_lock);
+}
+
+static int crg_udc_suspend(struct device *dev)
+{
+ struct crg_gadget_dev *crg_udc = dev_get_drvdata(dev);
+ unsigned long flags;
+ u32 val;
+ int ret;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (crg_udc->controller_recovering || crg_udc->shutting_down ||
+ crg_udc->external_reset_pending ||
+ crg_udc->endpoint_recovery_pending ||
+ (crg_udc->hw_reset_asserted && !crg_udc->gadget_stopping)) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EBUSY;
+ }
+ crg_udc->suspended = true;
+ if (crg_udc->gadget_stopping) {
+ if (crg_udc->hw_dma_armed) {
+ crg_udc->suspended = false;
+ ret = -EBUSY;
+ } else {
+ ret = 0;
+ }
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return ret;
+ }
+ crg_udc_set_pullup(crg_udc, false);
+ val = readl(&crg_udc->uccr->control);
+ val &= ~CRG_U3DC_CTRL_INT_EN;
+ writel(val, &crg_udc->uccr->control);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ if (crg_udc->irq >= 0)
+ synchronize_irq(crg_udc->irq);
+ cancel_delayed_work_sync(&crg_udc->port_retry_work);
+ cancel_work_sync(&crg_udc->recovery_work);
+ cancel_work_sync(&crg_udc->udc_reset_work);
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (crg_udc->gadget_stopping || crg_udc->shutting_down ||
+ crg_udc->controller_recovering || crg_udc->external_reset_pending ||
+ crg_udc->endpoint_recovery_pending) {
+ ret = -EBUSY;
+ goto out;
+ }
+ /*
+ * A reset may already have marked the link disconnected while its
+ * gadget notification was still pending. Deliver disconnect before
+ * resetting the software state, including that cancelled work.
+ */
+ if (crg_udc->connected || crg_udc->bus_reset_pending)
+ crg_udc_notify_disconnect(crg_udc, &flags);
+ crg_udc->connected = false;
+ crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+ if (crg_udc->external_reset_pending || crg_udc->endpoint_recovery_pending) {
+ ret = -EBUSY;
+ goto out;
+ }
+ ret = crg_udc_recover(crg_udc, &flags);
+ if (ret)
+ dev_err(crg_udc->dev, "suspend controller reset failed: %d\n", ret);
+out:
+ if (ret) {
+ crg_udc->suspended = false;
+ crg_udc_kick_reset_work(crg_udc);
+ }
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return ret;
+}
+
+static int crg_udc_resume(struct device *dev)
+{
+ struct crg_gadget_dev *crg_udc = dev_get_drvdata(dev);
+ unsigned long flags;
+ int ret = 0;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ crg_udc->suspended = false;
+ if (!crg_udc->gadget_stopping) {
+ if (crg_udc->shutting_down || crg_udc->resetting)
+ ret = -ESHUTDOWN;
+ else if (crg_udc->gadget_driver && crg_udc->pullup_on)
+ crg_udc_set_pullup(crg_udc, true);
+ }
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return ret;
+}
+
+/* Program the speed limit while the controller is stopped. */
+int crg_udc_program_speed(struct crg_gadget_dev *crg_udc)
+{
+ u32 cfg, speed;
+
+ switch (crg_udc->max_speed) {
+ case USB_SPEED_FULL:
+ speed = CRG_U3DC_CFG0_MAXSPEED_FS;
+ break;
+ case USB_SPEED_HIGH:
+ speed = CRG_U3DC_CFG0_MAXSPEED_HS;
+ break;
+ case USB_SPEED_SUPER:
+ speed = CRG_U3DC_CFG0_MAXSPEED_SS;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ cfg = readl(&crg_udc->uccr->config0);
+ cfg &= ~CRG_U3DC_CFG0_MAXSPEED_MASK;
+ cfg |= FIELD_PREP(CRG_U3DC_CFG0_MAXSPEED_MASK, speed);
+ writel(cfg, &crg_udc->uccr->config0);
+
+ return 0;
+}
+
+int crg_udc_reset_data_struct(struct crg_gadget_dev *crg_udc)
+{
+ int i;
+ int ret;
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 val;
+
+ crg_udc->ep0_state = EP0_STATE_IDLE;
+
+ val = readl(&uccr->control);
+ val &= (~(CRG_U3DC_CTRL_INT_EN | CRG_U3DC_CTRL_RUN));
+ writel(val, &uccr->control);
+
+ ret = crg_udc_program_speed(crg_udc);
+ if (ret)
+ return ret;
+
+ crg_udc_select_transfer_rings(crg_udc);
+
+ for (i = 0; i < CRG_RING_NUM; i++) {
+ ret = crg_udc_init_event_ring(crg_udc, i);
+ if (ret)
+ return ret;
+ }
+
+ ret = crg_udc_init_device_context(crg_udc);
+ if (ret)
+ return ret;
+
+ if (!crg_udc->status_req) {
+ struct usb_request *req;
+
+ req = crg_udc_alloc_request(&crg_udc->udc_ep[0].usb_ep,
+ GFP_ATOMIC);
+ if (!req)
+ return -ENOMEM;
+
+ crg_udc->status_req =
+ container_of(req, struct crg_udc_request, usb_req);
+ }
+
+ /* other hw ops */
+ return 0;
+}
+
+static void crg_gadget_handle_interrupt(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ unsigned long flags = 0;
+ u32 tmp_status;
+ int ret;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+
+ /*
+ * A reset started on another CPU owns the controller registers and
+ * the event rings until it has rebuilt them. It drops udc_lock while
+ * polling SWRST and during givebacks, so this handler can get in.
+ */
+ if (crg_udc->resetting || crg_udc->suspended ||
+ crg_udc->shutting_down || crg_udc->controller_recovering) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return;
+ }
+
+ tmp_status = readl(&uccr->status);
+
+ if (tmp_status & CRG_U3DC_STATUS_SYS_ERR) {
+ bool was_connected = crg_udc->connected;
+ u32 generation = crg_udc->udc_event[0].generation;
+
+ /* A controller reset must not overlap composite's ->reset(). */
+ if (crg_udc->reset_callback_active) {
+ writel(CRG_U3DC_STATUS_SYS_ERR, &uccr->status);
+ crg_udc->deferred_system_error = true;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return;
+ }
+
+ dev_err(crg_udc->dev, "controller system error\n");
+ writel(CRG_U3DC_STATUS_SYS_ERR, &uccr->status);
+ /* Function drivers must see enabled endpoints on disconnect. */
+ if (was_connected)
+ crg_udc_notify_disconnect(crg_udc, &flags);
+ /*
+ * The disconnect callback drops udc_lock: another CPU may be
+ * resetting now, or may have reset and rebuilt the rings
+ * already, in which case this error has been recovered.
+ */
+ if (crg_udc->resetting || crg_udc->suspended ||
+ crg_udc->shutting_down ||
+ generation != crg_udc->udc_event[0].generation) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return;
+ }
+ crg_udc_recover_link(crg_udc, &flags, false);
+
+ /*
+ * tmp_status was captured before the reset above, which tore
+ * down and rebuilt the event rings. It no longer reflects
+ * the controller's current state, and if recovery failed the
+ * rings may not be consistent at all. Skip the rest of this
+ * interrupt's processing; any event still pending will raise
+ * a fresh interrupt once the controller is running again.
+ */
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return;
+ }
+
+ if (tmp_status & CRG_U3DC_STATUS_EINT) {
+ int i;
+
+ writel(CRG_U3DC_STATUS_EINT, &uccr->status);
+
+ for (i = 0; i < CRG_RING_NUM; i++) {
+ u32 generation = crg_udc->udc_event[i].generation;
+
+ ret = crg_udc_process_event_ring(crg_udc, i, &flags);
+ /*
+ * The event handlers drop udc_lock, so a reset may have
+ * started (resetting) or already completed and rebuilt
+ * the rings (generation) while they ran. Either way
+ * tmp_status and the checks below are stale.
+ */
+ if (unlikely(crg_udc->resetting || crg_udc->suspended ||
+ crg_udc->shutting_down ||
+ generation != crg_udc->udc_event[i].generation)) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return;
+ }
+ if (ret)
+ dev_err(crg_udc->dev,
+ "process event ring error: %d\n", ret);
+ }
+ }
+
+ if (crg_udc->device_state == USB_STATE_RECONNECTING &&
+ crg_udc->portsc_on_reconnecting == 1 &&
+ crg_udc_is_event_rings_empty(crg_udc)) {
+ crg_udc->portsc_on_reconnecting = 0;
+ crg_udc_handle_port_status(crg_udc, &flags);
+ /* The port handler drops udc_lock. */
+ if (crg_udc->resetting) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return;
+ }
+ }
+
+ if (crg_udc->device_state == USB_STATE_RECONNECTING &&
+ crg_udc->connected == 1) {
+ ret = crg_udc_prepare_for_setup(crg_udc);
+ if (!ret && crg_udc->pending_setup_valid &&
+ !crg_udc->resetting && crg_udc->connected &&
+ crg_udc->device_state == USB_STATE_DEFAULT) {
+ struct usb_ctrlrequest setup = crg_udc->pending_setup;
+ u8 tag = crg_udc->pending_setup_tag;
+
+ crg_udc->pending_setup_valid = false;
+ crg_udc_dispatch_setup(crg_udc, &setup, tag, &flags);
+ } else if (ret && ret != -EBUSY) {
+ dev_err(crg_udc->dev,
+ "failed to prepare EP0 after reset: %d\n", ret);
+ }
+ }
+
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+}
+
+/*
+ * Primary handler only checks whether the shared interrupt belongs to the
+ * controller. All controller work, including reset polling, runs in the
+ * threaded handler and therefore never busy-waits in hard IRQ context.
+ */
+static irqreturn_t crg_udc_common_irq(int irq, void *dev)
+{
+ struct crg_gadget_dev *crg_udc = dev;
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 status;
+
+ if (READ_ONCE(crg_udc->gadget_stopping) ||
+ READ_ONCE(crg_udc->shutting_down) ||
+ READ_ONCE(crg_udc->resetting))
+ return IRQ_NONE;
+
+ status = readl(&uccr->status);
+ if (!(status & (CRG_U3DC_STATUS_EINT | CRG_U3DC_STATUS_SYS_ERR)))
+ return IRQ_NONE;
+
+ return IRQ_WAKE_THREAD;
+}
+
+static irqreturn_t crg_udc_irq_thread(int irq, void *dev)
+{
+ struct crg_gadget_dev *crg_udc = dev;
+
+ crg_gadget_handle_interrupt(crg_udc);
+
+ return IRQ_HANDLED;
+}
+
+static int crg_gadget_irq_init(struct platform_device *pdev,
+ struct crg_gadget_dev *crg_udc)
+{
+ int irq;
+ int ret;
+
+ irq = platform_get_irq(pdev, 0);
+ if (irq < 0)
+ return irq;
+
+ /*
+ * IRQF_ONESHOT keeps this (level-triggered) line masked until
+ * crg_udc_irq_thread() has cleared the status bits, which is the
+ * correct pairing for IRQF_SHARED with a threaded handler.
+ */
+ ret = devm_request_threaded_irq(&pdev->dev, irq, crg_udc_common_irq,
+ crg_udc_irq_thread,
+ IRQF_SHARED | IRQF_ONESHOT, "crg_udc",
+ crg_udc);
+ if (ret) {
+ dev_err(crg_udc->dev, "request of irq%d failed\n", irq);
+ } else {
+ crg_udc->irq = irq;
+ crg_udc->gadget.irq = irq;
+ }
+
+ return ret;
+}
+
+/* Invalidate the old software session before dropping udc_lock. */
+static void crg_udc_reset_session(struct crg_gadget_dev *crg_udc)
+{
+ crg_udc->resetting = true;
+ crg_udc->bus_reset_generation++;
+ crg_udc->bus_reset_pending = false;
+ crg_udc->external_reset_pending = false;
+ crg_udc->endpoint_recovery_pending = false;
+ crg_udc->restart_after_reset = false;
+ crg_udc->deferred_port_events = 0;
+ crg_udc->deferred_system_error = false;
+ crg_udc->portsc_on_reconnecting = false;
+ crg_udc->setup_fn_call_back = NULL;
+ crg_udc->set_tm = 0;
+ crg_udc->udc_ep[0].generation++;
+ crg_udc->pending_setup_valid = false;
+}
+
+static void crg_udc_reset_queues(struct crg_gadget_dev *crg_udc,
+ bool preserve_enabled)
+{
+ int i;
+
+ crg_udc->setup_status = WAIT_FOR_SETUP;
+ crg_udc->device_state = USB_STATE_ATTACHED;
+ crg_udc->dev_addr = 0;
+ crg_udc->port_reset_pending = false;
+ crg_udc->num_enabled_eps = 0;
+
+ crg_udc->udc_ep[0].desc = NULL;
+ crg_udc->udc_ep[0].ep_state = EP_STATE_DISABLED;
+
+ /* Data endpoint contexts start at DCI 2. DCI 1 has no queue. */
+ for (i = 2; i < CRG_NUM_EP_CX; i++) {
+ struct crg_udc_ep *ep = &crg_udc->udc_ep[i];
+
+ ep->generation++;
+ ep->desc = NULL;
+ ep->comp_desc = NULL;
+ ep->tran_ring_full = false;
+ ep->halt_failed = false;
+ ep->disable_failed = false;
+ ep->dequeue_paused = false;
+ ep->ep_state = EP_STATE_DISABLED;
+ }
+
+ for (i = 2; i < CRG_NUM_EP_CX; i++)
+ crg_udc_ep_nuke(&crg_udc->udc_ep[i], -ESHUTDOWN);
+ crg_udc_ep_nuke(&crg_udc->udc_ep[0], -ESHUTDOWN);
+
+ /* Preserve Gadget's endpoint state through all givebacks. */
+ if (!preserve_enabled)
+ for (i = 2; i < CRG_NUM_EP_CX; i++)
+ crg_udc_ep_clear_enabled(&crg_udc->udc_ep[i]);
+}
+
+static int crg_udc_reset(struct crg_gadget_dev *crg_udc, unsigned long *flags)
+{
+ int ret;
+
+ crg_udc_reset_session(crg_udc);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+ ret = crg_udc_soft_reset(crg_udc, true);
+ spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+ if (ret)
+ return ret;
+
+ crg_udc->hw_dma_armed = false;
+ crg_udc->dma_quiesce_failed = false;
+ crg_udc_release_quarantine(crg_udc);
+ crg_udc_clear_portpm(crg_udc);
+ crg_udc_reset_queues(crg_udc, crg_udc->preserve_ep_enabled_on_reset);
+ return 0;
+}
+
+/*
+ * The UDC core serializes pullup, async-callback and stop operations.
+ * Return requests before disconnect or unbind can free their storage.
+ */
+int crg_udc_stop_gadget(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 val;
+ int ret;
+
+ spin_lock_irq(&crg_udc->udc_lock);
+ if (crg_udc->gadget_stopping) {
+ /* A previous stop or a failed restart can leave DMA armed. */
+ bool retry = crg_udc->hw_dma_armed;
+
+ if (!retry) {
+ spin_unlock_irq(&crg_udc->udc_lock);
+ return 0;
+ }
+ goto quiesce;
+ }
+ crg_udc->gadget_stopping = true;
+ crg_udc->pullup_on = false;
+ crg_udc_reset_session(crg_udc);
+ if (!crg_udc->controller_recovering && crg_udc->hw_dma_armed) {
+ val = readl(&uccr->control);
+ val &= ~(CRG_U3DC_CTRL_RUN | CRG_U3DC_CTRL_INT_EN |
+ CRG_U3DC_CTRL_SYSERR_EN);
+ writel(val, &uccr->control);
+ }
+ spin_unlock_irq(&crg_udc->udc_lock);
+
+ if (crg_udc->irq >= 0)
+ synchronize_irq(crg_udc->irq);
+ cancel_delayed_work_sync(&crg_udc->port_retry_work);
+ cancel_work_sync(&crg_udc->recovery_work);
+ cancel_work_sync(&crg_udc->udc_reset_work);
+
+ spin_lock_irq(&crg_udc->udc_lock);
+ wait_event_lock_irq(crg_udc->teardown_wait,
+ !crg_udc->controller_recovering &&
+ !crg_udc->gadget_callbacks_active && !crg_udc->queueing,
+ crg_udc->udc_lock);
+quiesce:
+ crg_udc_recovery_begin(crg_udc);
+ spin_unlock_irq(&crg_udc->udc_lock);
+ ret = crg_udc_quiesce_hw(crg_udc);
+ if (ret)
+ dev_err(crg_udc->dev,
+ "cannot stop DMA during gadget teardown: %d\n", ret);
+
+ spin_lock_irq(&crg_udc->udc_lock);
+ crg_udc->dma_quiesce_failed = !!ret;
+ crg_udc->controller_faulted = !!ret;
+ if (!ret) {
+ if (!crg_udc->hw_reset_asserted)
+ crg_udc_set_vbus_detect(crg_udc, false);
+ crg_udc_release_quarantine(crg_udc);
+ }
+ /* Function drivers can inspect usb_ep.enabled during disconnect. */
+ crg_udc_reset_queues(crg_udc, true);
+ crg_udc->connected = false;
+ crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+ crg_udc_recovery_end(crg_udc);
+ spin_unlock_irq(&crg_udc->udc_lock);
+
+ usb_gadget_set_state(&crg_udc->gadget, USB_STATE_NOTATTACHED);
+ return ret;
+}
+
+/* Called with udc_lock held and IRQ and queue entry blocked. */
+static int crg_udc_deassert_reset(struct crg_gadget_dev *crg_udc,
+ unsigned long *flags)
+{
+ int ret;
+
+ spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+ ret = reset_control_deassert(crg_udc->rst);
+ spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+ if (!ret)
+ crg_udc->hw_reset_asserted = false;
+ return ret;
+}
+
+static int crg_udc_rebuild_after_stop(struct crg_gadget_dev *crg_udc)
+{
+ void __iomem *mode = crg_udc->mmio_virt_base + CRG_UDC_MODE_REG;
+ int ret;
+
+ writel(readl(mode) | CRG_UDC_ROLE_DEVICE, mode);
+ crg_udc_clear_portpm(crg_udc);
+ ret = crg_udc_reset_data_struct(crg_udc);
+ if (!ret)
+ ret = crg_udc_init_ep0(crg_udc);
+ if (!ret) {
+ crg_udc_set_vbus_detect(crg_udc, true);
+ crg_udc->stopped_needs_reset = false;
+ crg_udc->resetting = false;
+ }
+ return ret;
+}
+
+/*
+ * The gadget core serializes start and pullup with stop. Keep the IRQ and
+ * queue paths gated by gadget_stopping while the external reset may sleep.
+ */
+int crg_udc_restart_gadget(struct crg_gadget_dev *crg_udc,
+ unsigned long *flags)
+{
+ int i, ret;
+
+ if (crg_udc->suspended)
+ return -EBUSY;
+
+ crg_udc_recovery_begin(crg_udc);
+ /*
+ * A previous teardown or failed reconstruction may have left DMA
+ * armed. Retry quiescing it before releasing any retained mappings.
+ */
+ spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+ ret = crg_udc_quiesce_hw(crg_udc);
+ spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+ crg_udc->dma_quiesce_failed = !!ret;
+ if (ret)
+ goto out;
+
+ crg_udc_release_quarantine(crg_udc);
+ if (crg_udc->shutting_down) {
+ ret = -ESHUTDOWN;
+ goto out;
+ }
+ if (crg_udc->hw_reset_asserted) {
+ ret = crg_udc_deassert_reset(crg_udc, flags);
+ if (ret)
+ goto out;
+ }
+ if (crg_udc->shutting_down) {
+ ret = -ESHUTDOWN;
+ goto out;
+ }
+ ret = crg_udc_rebuild_after_stop(crg_udc);
+ if (ret)
+ goto out;
+
+ for (i = 2; i < CRG_NUM_EP_CX; i++)
+ crg_udc_ep_clear_enabled(&crg_udc->udc_ep[i]);
+ crg_udc->gadget_stopping = false;
+out:
+ crg_udc->controller_faulted = !!ret;
+ crg_udc_recovery_end(crg_udc);
+ return ret;
+}
+
+/*
+ * Run the optional reset in process context after the atomic SWRST path
+ * has returned the requests and quarantined any DMA-visible mappings.
+ */
+static int crg_udc_recover_external(struct crg_gadget_dev *crg_udc,
+ unsigned long *flags)
+{
+ int ret;
+
+ crg_udc->external_reset_pending = false;
+ crg_udc_recovery_begin(crg_udc);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+ if (crg_udc->irq >= 0)
+ synchronize_irq(crg_udc->irq);
+ ret = reset_control_assert(crg_udc->rst);
+ spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+ if (ret)
+ goto out;
+
+ crg_udc->hw_reset_asserted = true;
+ crg_udc->hw_dma_armed = false;
+ crg_udc->dma_quiesce_failed = false;
+ crg_udc_release_quarantine(crg_udc);
+
+ if (crg_udc->shutting_down || crg_udc->gadget_stopping) {
+ ret = -ESHUTDOWN;
+ goto out;
+ }
+ ret = crg_udc_deassert_reset(crg_udc, flags);
+ if (ret)
+ goto out;
+ if (crg_udc->shutting_down || crg_udc->gadget_stopping) {
+ ret = -ESHUTDOWN;
+ goto out;
+ }
+ ret = crg_udc_rebuild_after_stop(crg_udc);
+out:
+ if (ret)
+ crg_udc->bus_reset_pending = false;
+ crg_udc->controller_faulted = !!ret;
+ crg_udc_recovery_end(crg_udc);
+ return ret;
+}
+
+static void crg_udc_restart_link(struct crg_gadget_dev *crg_udc,
+ bool notify_reset)
+{
+ if (!crg_udc->async_callbacks || !crg_udc->gadget_driver ||
+ crg_udc->reset_callback_active)
+ return;
+
+ if (notify_reset) {
+ crg_udc->bus_reset_pending = true;
+ crg_udc->restart_after_reset = true;
+ crg_udc_kick_reset_work(crg_udc);
+ } else if (!crg_udc->suspended && crg_udc->pullup_on) {
+ crg_udc_set_pullup(crg_udc, true);
+ }
+}
+
+static void crg_udc_recovery_work(struct work_struct *work)
+{
+ struct crg_gadget_dev *crg_udc =
+ container_of(work, struct crg_gadget_dev, recovery_work);
+ unsigned long flags;
+ int ret;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (crg_udc->controller_recovering || crg_udc->gadget_callbacks_active ||
+ crg_udc->reset_callback_active ||
+ crg_udc->suspended || crg_udc->shutting_down || crg_udc->gadget_stopping)
+ goto unlock;
+
+ if (crg_udc->endpoint_recovery_pending) {
+ crg_udc_recover_endpoints(crg_udc, &flags);
+ goto unlock;
+ }
+
+ if (!crg_udc->external_reset_pending)
+ goto unlock;
+
+ ret = crg_udc_recover_external(crg_udc, &flags);
+ if (ret) {
+ if (ret != -ESHUTDOWN)
+ dev_err(crg_udc->dev, "external reset recovery failed: %d\n", ret);
+ goto unlock;
+ }
+
+ crg_udc_restart_link(crg_udc, crg_udc->bus_reset_pending);
+unlock:
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+}
+
+/*
+ * Called with udc_lock held. Keep the entire controller reconstruction
+ * serialized, not just the SWRST register poll. In particular, request
+ * callbacks drop udc_lock, allowing stop/suspend and IRQs to re-enter.
+ */
+int crg_udc_recover(struct crg_gadget_dev *crg_udc, unsigned long *flags)
+{
+ int ret;
+
+ if (crg_udc->controller_recovering || crg_udc->shutting_down ||
+ crg_udc->gadget_stopping || crg_udc->hw_reset_asserted)
+ return -EBUSY;
+
+ crg_udc_recovery_begin(crg_udc);
+ ret = crg_udc_reset(crg_udc, flags);
+ if (ret) {
+ crg_udc->controller_faulted = true;
+ crg_udc->dma_quiesce_failed = crg_udc->hw_dma_armed;
+ crg_udc_reset_queues(crg_udc, crg_udc->preserve_ep_enabled_on_reset);
+ }
+ if (!ret && !crg_udc->shutting_down && !crg_udc->gadget_stopping)
+ ret = crg_udc_reset_data_struct(crg_udc);
+ if (!ret && !crg_udc->shutting_down && !crg_udc->gadget_stopping)
+ ret = crg_udc_init_ep0(crg_udc);
+ if (!ret && (crg_udc->shutting_down || crg_udc->gadget_stopping))
+ ret = -ESHUTDOWN;
+
+ crg_udc->controller_faulted = !!ret;
+ crg_udc->resetting = !!ret;
+ if (!ret)
+ crg_udc->stopped_needs_reset = false;
+ crg_udc_recovery_end(crg_udc);
+ return ret;
+}
+
+/* Called with udc_lock held for recovery of a live USB session. */
+int crg_udc_recover_link(struct crg_gadget_dev *crg_udc, unsigned long *flags,
+ bool notify_reset)
+{
+ int ret;
+
+ if (crg_udc->controller_recovering || crg_udc->shutting_down ||
+ crg_udc->gadget_stopping || crg_udc->suspended ||
+ crg_udc->hw_reset_asserted)
+ return -EBUSY;
+
+ notify_reset = notify_reset && crg_udc->async_callbacks &&
+ crg_udc->gadget_driver && !crg_udc->reset_callback_active;
+ crg_udc->connected = false;
+ crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+ crg_udc->preserve_ep_enabled_on_reset = notify_reset;
+ ret = crg_udc_recover(crg_udc, flags);
+ crg_udc->preserve_ep_enabled_on_reset = false;
+
+ if (crg_udc->shutting_down || crg_udc->gadget_stopping)
+ return ret;
+
+ if (ret) {
+ dev_err(crg_udc->dev, "link recovery failed: %d\n", ret);
+ if (ret == -ETIMEDOUT && crg_udc->dma_quiesce_failed &&
+ crg_udc->rst) {
+ crg_udc->external_reset_pending = true;
+ crg_udc->bus_reset_pending = notify_reset;
+ crg_udc_kick_reset_work(crg_udc);
+ }
+ return ret;
+ }
+
+ crg_udc_restart_link(crg_udc, notify_reset);
+ return 0;
+}
+
+/* A full endpoint recovery also invalidates the gadget configuration. */
+int crg_udc_recover_endpoints(struct crg_gadget_dev *crg_udc,
+ unsigned long *flags)
+{
+ bool notify_reset = crg_udc->device_state == USB_STATE_CONFIGURED ||
+ crg_udc->bus_reset_pending ||
+ crg_udc->gadget_callbacks_active;
+
+ return crg_udc_recover_link(crg_udc, flags, notify_reset);
+}
+
+static const struct crg_udc_priv crg_udc_data = {
+ .quirks = CRG_UDC_QUIRKS_DEFAULT,
+};
+
+static const struct of_device_id of_crg_udc_match[] = {
+ { .compatible = "axiado,ax3000-udc", .data = &crg_udc_data },
+ { },
+};
+MODULE_DEVICE_TABLE(of, of_crg_udc_match);
+
+static DEFINE_SIMPLE_DEV_PM_OPS(crg_udc_pm_ops, crg_udc_suspend,
+ crg_udc_resume);
+
+static struct platform_driver crg_udc_driver = {
+ .probe = crg_udc_probe,
+ .remove = crg_udc_remove,
+ .shutdown = crg_udc_shutdown,
+ .driver = {
+ .name = KBUILD_MODNAME,
+ .of_match_table = of_crg_udc_match,
+ .pm = pm_sleep_ptr(&crg_udc_pm_ops),
+ },
+};
+module_platform_driver(crg_udc_driver);
+
+MODULE_AUTHOR("Axiado Corporation");
+MODULE_DESCRIPTION("Axiado AX3000/AX3005 USB Device Controller");
+MODULE_LICENSE("GPL");
--
2.34.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH v5 6/6] usb: gadget: udc: crg: Add core driver and build integration
2026-11-16 19:06 ` [PATCH v5 6/6] usb: gadget: udc: crg: Add core driver and build integration Petar Stepanovic
@ 2026-11-16 21:15 ` sashiko-bot
0 siblings, 0 replies; 13+ messages in thread
From: sashiko-bot @ 2026-11-16 21:15 UTC (permalink / raw)
To: Petar Stepanovic; +Cc: linux-usb, devicetree, robh, conor+dt
Thank you for your contribution! Sashiko AI review found 10 potential
issues to consider:
- [High] usb: gadget: crg: unhandled level IRQ storm in
crg_udc_common_irq()
- [High] usb: gadget: crg: broken endpoint teardown and UAF in
crg_udc_suspend()
- [Medium] usb: gadget: crg: missing crg_udc_prepare_for_setup() in port
reset workers
- [Medium] usb: gadget: crg: unscheduled external reset on
crg_udc_recover() timeout
- [Medium] usb: gadget: crg: lost link restart in crg_udc_restart_link()
during reset
- [Medium] usb: gadget: crg: lost disconnect on SYS_ERR when
bus_reset_pending is set
- [Medium] usb: gadget: crg: uncancelled work items on probe error and
remove
- [Medium] usb: gadget: crg: uncleared connected flag before disconnect
on SYS_ERR
- [Medium] usb: gadget: crg: missing DMA quiescence wait in
crg_udc_shutdown()
- [Medium] usb: gadget: udc: crg: lost reset notification in
crg_udc_handle_port_status()
--
commit 808e1b308d74053a13eba452fa900f8bf2bf60a5
Author: Petar Stepanovic <pstepanovic@axiado.com>
usb: gadget: udc: crg: Add core driver and build integration
This patch adds the core platform driver, interrupt handling, reset and
recovery logic, and Kconfig/Makefile integration for the Corigine/Axiado
AX3000/AX3005 USB Device Controller.
> diff --git a/drivers/usb/gadget/udc/crg/crg-udc-core.c b/drivers/usb/gadget/udc/crg/crg-udc-core.c
> new file mode 100644
> index 0000000000000..e508dbd45339f
> --- /dev/null
> +++ b/drivers/usb/gadget/udc/crg/crg-udc-core.c
[ ... ]
> + crg_udc->resetting = false;
> +
> + ret = crg_gadget_irq_init(pdev, crg_udc);
> + if (ret)
> + goto err_free_resources;
> +
> + ret = usb_add_gadget(&crg_udc->gadget);
> + if (ret)
> + goto err_free_resources;
> +
> + return 0;
> +
> +err_free_resources:
> + crg_udc_free_resources(crg_udc);
> +err_put_gadget:
> + usb_put_gadget(&crg_udc->gadget);
> + return ret;
> +}
> +
> +static void crg_udc_remove(struct platform_device *pdev)
> +{
> + struct crg_gadget_dev *crg_udc = platform_get_drvdata(pdev);
> + unsigned long flags;
> + u32 tmp;
> +
> + spin_lock_irqsave(&crg_udc->udc_lock, flags);
> + crg_udc->shutting_down = true;
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +
> + /* The gadget core calls pullup(0) before the disconnect callback. */
> + usb_del_gadget(&crg_udc->gadget);
> + cancel_delayed_work_sync(&crg_udc->port_retry_work);
> + cancel_work_sync(&crg_udc->recovery_work);
> + cancel_work_sync(&crg_udc->udc_reset_work);
> + crg_udc_free_resources(crg_udc);
[Severity: Medium]
Can an interrupt scheduled between crg_gadget_irq_init() and a failing
usb_add_gadget() leave port_retry_work, recovery_work, or udc_reset_work
queued after crg_udc is freed?
crg_udc_free_resources() synchronizes and frees the IRQ, then frees the DMA
rings, without cancelling those three work items before usb_put_gadget()
frees crg_udc. In crg_udc_remove(), the work items are also cancelled
before crg_udc_free_resources() synchronizes and frees the IRQ rather than
after.
> +
> + /* Do not access registers while the external reset is asserted. */
> + if (!crg_udc->hw_reset_asserted && !crg_udc->dma_quiesce_failed) {
> + tmp = readl(crg_udc->mmio_virt_base + CRG_UDC_MODE_REG) & ~0x1;
> + writel(tmp, crg_udc->mmio_virt_base + CRG_UDC_MODE_REG);
> + }
> +
> + platform_set_drvdata(pdev, NULL);
> + usb_put_gadget(&crg_udc->gadget);
> +}
> +
> +static void crg_udc_shutdown(struct platform_device *pdev)
> +{
> + struct crg_gadget_dev *crg_udc = platform_get_drvdata(pdev);
> + unsigned long flags;
> +
> + spin_lock_irqsave(&crg_udc->udc_lock, flags);
> + crg_udc->pullup_on = false;
> + crg_udc->shutting_down = true;
> + crg_udc->resetting = true;
> + crg_udc->device_state = USB_STATE_ATTACHED;
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + if (crg_udc->irq >= 0)
> + synchronize_irq(crg_udc->irq);
> + cancel_delayed_work_sync(&crg_udc->port_retry_work);
> + cancel_work_sync(&crg_udc->recovery_work);
> + cancel_work_sync(&crg_udc->udc_reset_work);
> +
> + /* Reset and gadget stop may own the registers across a lock drop. */
> + spin_lock_irq(&crg_udc->udc_lock);
> + wait_event_lock_irq(crg_udc->teardown_wait,
> + !crg_udc->controller_recovering, crg_udc->udc_lock);
> + if (crg_udc->uccr && !crg_udc->hw_reset_asserted) {
> + u32 val = readl(&crg_udc->uccr->control);
> +
> + crg_udc_set_vbus_detect(crg_udc, false);
> + val &= ~(CRG_U3DC_CTRL_INT_EN | CRG_U3DC_CTRL_RUN |
> + CRG_U3DC_CTRL_SYSERR_EN);
> + writel(val, &crg_udc->uccr->control);
> + }
> + spin_unlock_irq(&crg_udc->udc_lock);
> +}
[Severity: Medium]
Does crg_udc_shutdown() return before bus-master DMA has actually stopped?
As noted in crg_udc_soft_reset(), clearing CRG_U3DC_CTRL_RUN does not halt
DMA or pending event-ring writes until CRG_U3DC_STATUS_DEV_CTRL_HALT
asserts. Returning without waiting for halt or calling crg_udc_quiesce_hw()
leaves DMA active across shutdown or kexec.
> +
> +static int crg_udc_suspend(struct device *dev)
> +{
> + struct crg_gadget_dev *crg_udc = dev_get_drvdata(dev);
> + unsigned long flags;
> + u32 val;
> + int ret;
> +
> + spin_lock_irqsave(&crg_udc->udc_lock, flags);
> + if (crg_udc->controller_recovering || crg_udc->shutting_down ||
> + crg_udc->external_reset_pending ||
> + crg_udc->endpoint_recovery_pending ||
> + (crg_udc->hw_reset_asserted && !crg_udc->gadget_stopping)) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -EBUSY;
> + }
> + crg_udc->suspended = true;
> + if (crg_udc->gadget_stopping) {
> + if (crg_udc->hw_dma_armed) {
> + crg_udc->suspended = false;
> + ret = -EBUSY;
> + } else {
> + ret = 0;
> + }
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return ret;
> + }
> + crg_udc_set_pullup(crg_udc, false);
> + val = readl(&crg_udc->uccr->control);
> + val &= ~CRG_U3DC_CTRL_INT_EN;
> + writel(val, &crg_udc->uccr->control);
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +
> + if (crg_udc->irq >= 0)
> + synchronize_irq(crg_udc->irq);
> + cancel_delayed_work_sync(&crg_udc->port_retry_work);
> + cancel_work_sync(&crg_udc->recovery_work);
> + cancel_work_sync(&crg_udc->udc_reset_work);
> +
> + spin_lock_irqsave(&crg_udc->udc_lock, flags);
> + if (crg_udc->gadget_stopping || crg_udc->shutting_down ||
> + crg_udc->controller_recovering || crg_udc->external_reset_pending ||
> + crg_udc->endpoint_recovery_pending) {
> + ret = -EBUSY;
> + goto out;
> + }
> + /*
> + * A reset may already have marked the link disconnected while its
> + * gadget notification was still pending. Deliver disconnect before
> + * resetting the software state, including that cancelled work.
> + */
> + if (crg_udc->connected || crg_udc->bus_reset_pending)
> + crg_udc_notify_disconnect(crg_udc, &flags);
> + crg_udc->connected = false;
> + crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
> + if (crg_udc->external_reset_pending || crg_udc->endpoint_recovery_pending) {
> + ret = -EBUSY;
> + goto out;
> + }
> + ret = crg_udc_recover(crg_udc, &flags);
> + if (ret)
> + dev_err(crg_udc->dev, "suspend controller reset failed: %d\n", ret);
> +out:
> + if (ret) {
> + crg_udc->suspended = false;
> + crg_udc_kick_reset_work(crg_udc);
> + }
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return ret;
> +}
[Severity: High]
Can crg_udc_suspend() cause a use-after-free on queued usb_requests when
crg_udc_notify_disconnect() disables endpoints?
crg_udc_suspend() sets crg_udc->suspended = true and clears CRG_U3DC_CTRL_RUN
via crg_udc_set_pullup(crg_udc, false) before calling
crg_udc_notify_disconnect() and crg_udc_recover(). When the gadget driver's
->disconnect() callback invokes usb_ep_disable() -> crg_udc_ep_disable() on
an active endpoint, polling uccr->ep_enable or uccr->ep_running can time out
with RUN cleared and call crg_udc_recover_endpoints() ->
crg_udc_recover_link(). Because crg_udc_recover_link() rejects
crg_udc->suspended with -EBUSY, usb_ep_disable() returns without stopping DMA
or draining udc_ep->queue, and the function driver frees its usb_requests
before crg_udc_recover() -> crg_udc_reset_queues() -> crg_udc_ep_nuke() gives
them back.
In addition, if usb_ep_dequeue() runs during that window,
crg_udc_abort_ep0_for_setup() or CRG_CMD_STOP_EP fails because suspended is
set and RUN is clear, setting endpoint_recovery_pending = true and causing
crg_udc_suspend() to abort with -EBUSY instead of running crg_udc_recover().
And if crg_udc_suspend() takes the early goto out before crg_udc_recover(),
CRG_U3DC_CTRL_RUN and CRG_U3DC_CTRL_INT_EN remain cleared while
crg_udc->suspended is reset to false.
[ ... ]
> + if (tmp_status & CRG_U3DC_STATUS_SYS_ERR) {
> + bool was_connected = crg_udc->connected;
> + u32 generation = crg_udc->udc_event[0].generation;
> +
> + /* A controller reset must not overlap composite's ->reset(). */
> + if (crg_udc->reset_callback_active) {
> + writel(CRG_U3DC_STATUS_SYS_ERR, &uccr->status);
> + crg_udc->deferred_system_error = true;
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return;
> + }
> +
> + dev_err(crg_udc->dev, "controller system error\n");
> + writel(CRG_U3DC_STATUS_SYS_ERR, &uccr->status);
> + /* Function drivers must see enabled endpoints on disconnect. */
> + if (was_connected)
> + crg_udc_notify_disconnect(crg_udc, &flags);
[Severity: Medium]
If CRG_U3DC_STATUS_SYS_ERR fires after a bus reset has already cleared
crg_udc->connected while crg_udc->bus_reset_pending is still true, does this
drop the gadget reset/disconnect notification?
Unlike crg_udc_suspend(), which checks crg_udc->connected ||
crg_udc->bus_reset_pending, this path only checks crg_udc->connected and
then calls crg_udc_recover_link(crg_udc, &flags, false), which clears
bus_reset_pending and marks all endpoints disabled without notifying the
gadget driver.
> + /*
> + * The disconnect callback drops udc_lock: another CPU may be
> + * resetting now, or may have reset and rebuilt the rings
> + * already, in which case this error has been recovered.
> + */
> + if (crg_udc->resetting || crg_udc->suspended ||
> + crg_udc->shutting_down ||
> + generation != crg_udc->udc_event[0].generation) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return;
> + }
> + crg_udc_recover_link(crg_udc, &flags, false);
[Severity: Medium]
Should crg_udc->connected (and device_state) be cleared before calling
crg_udc_notify_disconnect() here?
Because crg_udc_notify_disconnect() drops udc_lock while crg_udc->connected
is still true and crg_udc->device_state is still USB_STATE_CONFIGURED, a
concurrent crg_udc_suspend() or port status handler can deliver a second
disconnect notification. Also, if usb_ep_disable() inside ->disconnect()
times out on the faulted controller and calls crg_udc_recover_endpoints(),
notify_reset evaluates to true (via device_state == USB_STATE_CONFIGURED and
gadget_callbacks_active) and queues a bus reset on top of the disconnect.
[ ... ]
> + if (crg_udc->device_state == USB_STATE_RECONNECTING &&
> + crg_udc->portsc_on_reconnecting == 1 &&
> + crg_udc_is_event_rings_empty(crg_udc)) {
> + crg_udc->portsc_on_reconnecting = 0;
> + crg_udc_handle_port_status(crg_udc, &flags);
> + /* The port handler drops udc_lock. */
> + if (crg_udc->resetting) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return;
> + }
> + }
[Severity: Medium]
Can a second port event while crg_udc->bus_reset_pending is true clear
bus_reset_pending without notifying the gadget driver?
When an initial bus reset on a configured gadget runs crg_udc_reinit(crg_udc,
true), crg_udc->device_state becomes USB_STATE_RECONNECTING and
crg_udc->bus_reset_pending is set to true until udc_reset_work runs. If a
second port reset, a port-reset timeout in crg_udc_port_retry_work(), or a
CCS drop with VBUS present arrives before udc_reset_work runs, those paths
only check crg_udc->device_state == USB_STATE_CONFIGURED (or pass false to
crg_udc_reinit()):
drivers/usb/gadget/udc/crg/crg-udc-gadget.c:crg_udc_handle_port_status() {
...
bool was_configured = crg_udc->device_state ==
USB_STATE_CONFIGURED;
bool notify_reset = was_configured &&
crg_udc->async_callbacks &&
crg_udc->gadget_driver;
int reinit_ret;
reinit_ret = crg_udc_reinit(crg_udc, notify_reset);
...
}
crg_udc_reinit(crg_udc, false) then overwrites crg_udc->bus_reset_pending
with false and clears usb_ep.enabled on all endpoints without running
usb_gadget_udc_reset().
> +
> + if (crg_udc->device_state == USB_STATE_RECONNECTING &&
> + crg_udc->connected == 1) {
> + ret = crg_udc_prepare_for_setup(crg_udc);
> + if (!ret && crg_udc->pending_setup_valid &&
> + !crg_udc->resetting && crg_udc->connected &&
> + crg_udc->device_state == USB_STATE_DEFAULT) {
> + struct usb_ctrlrequest setup = crg_udc->pending_setup;
> + u8 tag = crg_udc->pending_setup_tag;
> +
> + crg_udc->pending_setup_valid = false;
> + crg_udc_dispatch_setup(crg_udc, &setup, tag, &flags);
> + } else if (ret && ret != -EBUSY) {
> + dev_err(crg_udc->dev,
> + "failed to prepare EP0 after reset: %d\n", ret);
> + }
> + }
[Severity: Medium]
What prepares EP0 and transitions device_state out of USB_STATE_RECONNECTING
when crg_udc_handle_port_status() is called from crg_udc_port_retry_work() or
crg_udc_reset_work()?
When a port reset on a session with device_state >= USB_STATE_DEFAULT
completes inside crg_udc_handle_port_status(), crg_udc_reinit() sets
device_state = USB_STATE_RECONNECTING and crg_udc_handle_port_status() sets
connected = 1 without calling crg_udc_enable_setup(). Only
crg_gadget_handle_interrupt() runs the USB_STATE_RECONNECTING block above to
call crg_udc_prepare_for_setup() and dispatch pending_setup; the two worker
callers of crg_udc_handle_port_status() return with the controller stuck in
USB_STATE_RECONNECTING.
> +
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +}
> +
> +/*
> + * Primary handler only checks whether the shared interrupt belongs to the
> + * controller. All controller work, including reset polling, runs in the
> + * threaded handler and therefore never busy-waits in hard IRQ context.
> + */
> +static irqreturn_t crg_udc_common_irq(int irq, void *dev)
> +{
> + struct crg_gadget_dev *crg_udc = dev;
> + struct crg_uccr __iomem *uccr = crg_udc->uccr;
> + u32 status;
> +
> + if (READ_ONCE(crg_udc->gadget_stopping) ||
> + READ_ONCE(crg_udc->shutting_down) ||
> + READ_ONCE(crg_udc->resetting))
> + return IRQ_NONE;
> +
> + status = readl(&uccr->status);
> + if (!(status & (CRG_U3DC_STATUS_EINT | CRG_U3DC_STATUS_SYS_ERR)))
> + return IRQ_NONE;
> +
> + return IRQ_WAKE_THREAD;
> +}
[Severity: High]
Can returning IRQ_NONE here while the level-triggered interrupt is still
asserted cause a hard-IRQ storm that disables the shared IRQ line?
The IRQ is registered with IRQF_SHARED | IRQF_ONESHOT, so the line is only
masked when crg_udc_common_irq() returns IRQ_WAKE_THREAD. Several paths set
resetting, shutting_down, or gadget_stopping and re-enable local interrupts
before CRG_U3DC_CTRL_INT_EN and CRG_U3DC_CTRL_SYSERR_EN are cleared in
uccr->control:
crg_udc_reset()
crg_udc_reset_session()
crg_udc->resetting = true;
spin_unlock_irqrestore(&crg_udc->udc_lock, *flags)
<hardirq>
crg_udc_common_irq()
if (READ_ONCE(crg_udc->resetting))
return IRQ_NONE; <-- level IRQ still asserted; storms
The same window exists in crg_udc_remove() during usb_del_gadget(), in
crg_udc_shutdown() before synchronize_irq(), and in crg_udc_stop_gadget()
when controller_recovering is true. Similarly, when suspended or
controller_recovering is true while resetting is false (such as during
crg_udc_reinit()), crg_udc_common_irq() wakes crg_udc_irq_thread(), which
returns early in crg_gadget_handle_interrupt() without clearing
uccr->status and unmasks the still-asserted level interrupt in a tight loop.
[ ... ]
> +static void crg_udc_restart_link(struct crg_gadget_dev *crg_udc,
> + bool notify_reset)
> +{
> + if (!crg_udc->async_callbacks || !crg_udc->gadget_driver ||
> + crg_udc->reset_callback_active)
> + return;
> +
> + if (notify_reset) {
> + crg_udc->bus_reset_pending = true;
> + crg_udc->restart_after_reset = true;
> + crg_udc_kick_reset_work(crg_udc);
> + } else if (!crg_udc->suspended && crg_udc->pullup_on) {
> + crg_udc_set_pullup(crg_udc, true);
> + }
> +}
[Severity: Medium]
If crg_udc_recover_endpoints() runs while crg_udc->reset_callback_active is
true, does this lose the post-reset pullup restart?
When usb_ep_disable() times out inside usb_gadget_udc_reset() and calls
crg_udc_recover_endpoints() -> crg_udc_recover_link(), crg_udc_reset() clears
crg_udc->restart_after_reset = false and resets the controller (clearing
CRG_U3DC_CTRL_RUN). crg_udc_restart_link() then returns early because
reset_callback_active is true, leaving restart_after_reset false when
crg_udc_reset_work() finishes.
[ ... ]
> +int crg_udc_recover(struct crg_gadget_dev *crg_udc, unsigned long *flags)
> +{
> + int ret;
> +
> + if (crg_udc->controller_recovering || crg_udc->shutting_down ||
> + crg_udc->gadget_stopping || crg_udc->hw_reset_asserted)
> + return -EBUSY;
> +
> + crg_udc_recovery_begin(crg_udc);
> + ret = crg_udc_reset(crg_udc, flags);
> + if (ret) {
> + crg_udc->controller_faulted = true;
> + crg_udc->dma_quiesce_failed = crg_udc->hw_dma_armed;
> + crg_udc_reset_queues(crg_udc, crg_udc->preserve_ep_enabled_on_reset);
> + }
> + if (!ret && !crg_udc->shutting_down && !crg_udc->gadget_stopping)
> + ret = crg_udc_reset_data_struct(crg_udc);
> + if (!ret && !crg_udc->shutting_down && !crg_udc->gadget_stopping)
> + ret = crg_udc_init_ep0(crg_udc);
> + if (!ret && (crg_udc->shutting_down || crg_udc->gadget_stopping))
> + ret = -ESHUTDOWN;
> +
> + crg_udc->controller_faulted = !!ret;
> + crg_udc->resetting = !!ret;
> + if (!ret)
> + crg_udc->stopped_needs_reset = false;
> + crg_udc_recovery_end(crg_udc);
> + return ret;
> +}
[Severity: Medium]
When crg_udc_reset() times out in crg_udc_recover(), should this set
crg_udc->external_reset_pending when crg_udc->dma_quiesce_failed &&
crg_udc->rst, like crg_udc_recover_link() does?
crg_udc_reset_session() clears external_reset_pending, and on timeout
crg_udc_reset_queues() quarantines any active DMA bounce buffers into
crg_udc->dma_quarantine. When crg_udc_recover() is called directly from
crg_udc_suspend() or crg_gadget_pullup(), external_reset_pending is never
set, so crg_udc_kick_reset_work() does not schedule recovery_work to assert
the external reset and release the quarantined buffers.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261116-axiado-ax3000-usb-device-controller-v5-0-49271ed5c793@axiado.com?part=6
^ permalink raw reply [flat|nested] 13+ messages in thread