* [PATCH v4 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC
2026-10-30 8:10 [PATCH v4 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
@ 2026-10-30 8:10 ` Petar Stepanovic
2026-10-30 8:14 ` sashiko-bot
2026-10-30 8:10 ` [PATCH v4 2/6] usb: gadget: udc: crg: Add hardware definitions and structs Petar Stepanovic
` (4 subsequent siblings)
5 siblings, 1 reply; 14+ messages in thread
From: Petar Stepanovic @ 2026-10-30 8:10 UTC (permalink / raw)
To: Naveen Kumar Rajgiri Bassappa, Prasad Bolisetty,
Greg Kroah-Hartman, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Tzu-Hao Wei, Karthikeyan Mitran, Philipp Zabel
Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
Petar Stepanovic, Krutik Shah
Add a Devicetree binding for the USB device controller integrated into
the Axiado AX3000 and AX3005 SoCs.
The controller is based on a Corigine USB 3.1 IP core that is
dual-role capable (device and host), used here in device role. It
provides SuperSpeed (5 Gb/s) or High-Speed (480 Mb/s) operation, and
supports control, bulk, interrupt, and isochronous transfer types
across multiple configurable endpoints.
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
.../devicetree/bindings/usb/axiado,ax3000-udc.yaml | 69 ++++++++++++++++++++++
MAINTAINERS | 8 +++
2 files changed, 77 insertions(+)
diff --git a/Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml b/Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
new file mode 100644
index 000000000000..b9f7f4879ee0
--- /dev/null
+++ b/Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
@@ -0,0 +1,69 @@
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/usb/axiado,ax3000-udc.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Axiado AX3000/AX3005 USB Device Controller
+
+maintainers:
+ - Petar Stepanovic <pstepanovic@axiado.com>
+ - Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
+ - Prasad Bolisetty <pbolisetty@axiado.com>
+
+description:
+ Axiado USB dual-role controller based on the Corigine USB 3.1 IP core.
+ Supports USB 3.1 SuperSpeed and USB 2.0 High-Speed device operation
+ with control, bulk, interrupt, and isochronous transfers.
+
+allOf:
+ - $ref: usb.yaml#
+
+properties:
+ compatible:
+ oneOf:
+ - const: axiado,ax3000-udc
+ - items:
+ - const: axiado,ax3005-udc
+ - const: axiado,ax3000-udc
+
+ reg:
+ maxItems: 1
+
+ interrupts:
+ maxItems: 1
+
+ clocks:
+ maxItems: 1
+
+ resets:
+ maxItems: 1
+
+ phys:
+ maxItems: 1
+
+required:
+ - compatible
+ - reg
+ - interrupts
+
+unevaluatedProperties: false
+
+examples:
+ - |
+ #include <dt-bindings/interrupt-controller/arm-gic.h>
+ #include <dt-bindings/interrupt-controller/irq.h>
+
+ soc {
+ #address-cells = <2>;
+ #size-cells = <2>;
+
+ usb@81200000 {
+ compatible = "axiado,ax3000-udc";
+ reg = <0x0 0x81200000 0x0 0x100000>;
+ interrupt-parent = <&gic500>;
+ interrupts = <GIC_SPI 21 IRQ_TYPE_LEVEL_HIGH>;
+ resets = <&reset 0>;
+ phys = <&usb_phy>;
+ };
+ };
diff --git a/MAINTAINERS b/MAINTAINERS
index 65e8a4b5c90b..58c5c8e00870 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4508,6 +4508,14 @@ F: Documentation/devicetree/bindings/spi/axiado,ax3000-spi.yaml
F: drivers/spi/spi-axiado.c
F: drivers/spi/spi-axiado.h
+AXIADO USB UDC DRIVER
+M: Petar Stepanovic <pstepanovic@axiado.com>
+M: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
+M: Prasad Bolisetty <pbolisetty@axiado.com>
+L: linux-usb@vger.kernel.org
+S: Supported
+F: Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
+
AYANEO PLATFORM EC DRIVER
M: Antheas Kapenekakis <lkml@antheas.dev>
L: platform-driver-x86@vger.kernel.org
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH v4 2/6] usb: gadget: udc: crg: Add hardware definitions and structs
2026-10-30 8:10 [PATCH v4 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
2026-10-30 8:10 ` [PATCH v4 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC Petar Stepanovic
@ 2026-10-30 8:10 ` Petar Stepanovic
2026-10-30 8:21 ` sashiko-bot
2026-10-30 8:10 ` [PATCH v4 3/6] usb: gadget: udc: crg: Add ring and transfer management Petar Stepanovic
` (3 subsequent siblings)
5 siblings, 1 reply; 14+ messages in thread
From: Petar Stepanovic @ 2026-10-30 8:10 UTC (permalink / raw)
To: Naveen Kumar Rajgiri Bassappa, Prasad Bolisetty,
Greg Kroah-Hartman, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Tzu-Hao Wei, Karthikeyan Mitran, Philipp Zabel
Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
Petar Stepanovic, Krutik Shah
This patch introduces the hardware definitions, register offsets, and
fundamental data structures for the Axiado AX3000/AX3005 USB Device
Controller driver.
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
MAINTAINERS | 1 +
drivers/usb/gadget/udc/crg/crg-udc-debug.h | 143 ++++++
drivers/usb/gadget/udc/crg/crg-udc-trace.h | 164 +++++++
drivers/usb/gadget/udc/crg/crg-udc.h | 729 +++++++++++++++++++++++++++++
4 files changed, 1037 insertions(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index 58c5c8e00870..d5b6a04a95ad 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4515,6 +4515,7 @@ M: Prasad Bolisetty <pbolisetty@axiado.com>
L: linux-usb@vger.kernel.org
S: Supported
F: Documentation/devicetree/bindings/usb/axiado,ax3000-udc.yaml
+F: drivers/usb/gadget/udc/crg/
AYANEO PLATFORM EC DRIVER
M: Antheas Kapenekakis <lkml@antheas.dev>
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-debug.h b/drivers/usb/gadget/udc/crg/crg-udc-debug.h
new file mode 100644
index 000000000000..f73c60a79ade
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-debug.h
@@ -0,0 +1,143 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ * Debug/trace decode helpers.
+ *
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+
+#ifndef __CRG_UDC_DEBUG_H
+#define __CRG_UDC_DEBUG_H
+
+#include "crg-udc.h"
+
+static inline const char *crg_udc_trb_type_string(u32 type)
+{
+ switch (type) {
+ case TRB_TYPE_XFER_NORMAL:
+ return "Normal";
+ case TRB_TYPE_XFER_DATA_STAGE:
+ return "Data Stage";
+ case TRB_TYPE_XFER_STATUS_STAGE:
+ return "Status Stage";
+ case TRB_TYPE_XFER_DATA_ISOCH:
+ return "Isoch";
+ case TRB_TYPE_LINK:
+ return "Link";
+ case TRB_TYPE_NO_OP:
+ return "No-Op";
+ case TRB_TYPE_EVT_TRANSFER:
+ return "Transfer Event";
+ case TRB_TYPE_EVT_CMD_COMPLETION:
+ return "Command Completion Event";
+ case TRB_TYPE_EVT_PORT_STATUS_CHANGE:
+ return "Port Status Change Event";
+ case TRB_TYPE_EVT_MFINDEX_WRAP:
+ return "MFINDEX Wrap Event";
+ case TRB_TYPE_EVT_SETUP_PKT:
+ return "Setup Packet Event";
+ default:
+ return "Reserved";
+ }
+}
+
+static inline const char *crg_udc_cmpl_code_string(u32 code)
+{
+ switch (code) {
+ case CMPL_CODE_INVALID:
+ return "Invalid";
+ case CMPL_CODE_SUCCESS:
+ return "Success";
+ case CMPL_CODE_DATA_BUFFER_ERR:
+ return "Data Buffer Error";
+ case CMPL_CODE_BABBLE_DETECTED_ERR:
+ return "Babble Detected";
+ case CMPL_CODE_USB_TRANS_ERR:
+ return "USB Transaction Error";
+ case CMPL_CODE_TRB_ERR:
+ return "TRB Error";
+ case CMPL_CODE_TRB_STALL:
+ return "Stall";
+ case CMPL_CODE_INVALID_STREAM_TYPE_ERR:
+ return "Invalid Stream Type";
+ case CMPL_CODE_SHORT_PKT:
+ return "Short Packet";
+ case CMPL_CODE_RING_UNDERRUN:
+ return "Ring Underrun";
+ case CMPL_CODE_RING_OVERRUN:
+ return "Ring Overrun";
+ case CMPL_CODE_EVENT_RING_FULL_ERR:
+ return "Event Ring Full";
+ case CMPL_CODE_STOPPED:
+ return "Stopped";
+ case CMPL_CODE_STOPPED_LENGTH_INVALID:
+ return "Stopped - Length Invalid";
+ case CMPL_CODE_ISOCH_BUFFER_OVERRUN:
+ return "Isoch Buffer Overrun";
+ case CMPL_CODE_PROTOCOL_STALL:
+ return "Protocol Stall";
+ case CMPL_CODE_SETUP_TAG_MISMATCH:
+ return "Setup Tag Mismatch";
+ case CMPL_CODE_HALTED:
+ return "Halted";
+ case CMPL_CODE_HALTED_LENGTH_INVALID:
+ return "Halted - Length Invalid";
+ case CMPL_CODE_DISABLED:
+ return "Disabled";
+ case CMPL_CODE_DISABLED_LENGTH_INVALID:
+ return "Disabled - Length Invalid";
+ default:
+ return "Unknown";
+ }
+}
+
+static inline const char *crg_udc_decode_portsc(char *str, size_t size,
+ u32 portsc)
+{
+ scnprintf(str, size, "0x%08x, PLS: %lu, Speed: %lu, %s%s%s%s",
+ portsc, FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, portsc),
+ FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, portsc),
+ (portsc & CRG_U3DC_PORTSC_CCS) ? "CCS " : "",
+ (portsc & CRG_U3DC_PORTSC_PED) ? "PED " : "",
+ (portsc & CRG_U3DC_PORTSC_PR) ? "PR " : "",
+ (portsc & CRG_U3DC_PORTSC_PP) ? "PP " : "");
+
+ return str;
+}
+
+static inline const char *crg_udc_decode_trb(char *str, size_t size,
+ struct crg_transfer_trb *trb)
+{
+ u32 dw2 = le32_to_cpu(trb->dw2);
+ u32 dw3 = le32_to_cpu(trb->dw3);
+ u32 type = FIELD_GET(TRB_TYPE_MASK, dw3);
+
+ scnprintf(str, size,
+ "type: %s, len: %lu, cycle: %lu, chain: %lu, ioc: %lu",
+ crg_udc_trb_type_string(type),
+ FIELD_GET(TRB_TRANSFER_LEN_MASK, dw2),
+ dw3 & TRB_CYCLE_BIT_MASK ? 1UL : 0UL,
+ dw3 & TRB_CHAIN_BIT_MASK ? 1UL : 0UL,
+ dw3 & TRB_INTR_ON_COMPLETION_MASK ? 1UL : 0UL);
+
+ return str;
+}
+
+static inline const char *crg_udc_decode_event_trb(char *str, size_t size,
+ struct crg_event_trb *trb)
+{
+ u32 dw2 = le32_to_cpu(trb->dw2);
+ u32 dw3 = le32_to_cpu(trb->dw3);
+ u32 type = FIELD_GET(EVE_TRB_TYPE_MASK, dw3);
+
+ scnprintf(str, size, "type: %s, cmpl: %s, len: %lu, dci: %lu",
+ crg_udc_trb_type_string(type),
+ crg_udc_cmpl_code_string(
+ FIELD_GET(EVE_TRB_COMPL_CODE_MASK, dw2)),
+ FIELD_GET(EVE_TRB_TRAN_LEN_MASK, dw2),
+ FIELD_GET(EVE_TRB_ENDPOINT_ID_MASK, dw3));
+
+ return str;
+}
+
+#endif /* __CRG_UDC_DEBUG_H */
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-trace.h b/drivers/usb/gadget/udc/crg/crg-udc-trace.h
new file mode 100644
index 000000000000..672a719a041b
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-trace.h
@@ -0,0 +1,164 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ * Trace support header file.
+ *
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM crg-udc
+
+#undef TRACE_SYSTEM_VAR
+#define TRACE_SYSTEM_VAR crg_udc
+
+#if !defined(__CRG_UDC_TRACE_H) || defined(TRACE_HEADER_MULTI_READ)
+#define __CRG_UDC_TRACE_H
+
+#include <linux/types.h>
+#include <linux/tracepoint.h>
+#include <linux/usb/ch9.h>
+#include "crg-udc.h"
+#include "crg-udc-debug.h"
+
+#define CRG_MSG_MAX 200
+
+DECLARE_EVENT_CLASS(crg_udc_log_enable_disable, TP_PROTO(int set), TP_ARGS(set),
+ TP_STRUCT__entry(__field(int, set)),
+ TP_fast_assign(__entry->set = set;),
+ TP_printk("%s", __entry->set ? "enabled" : "disabled"));
+
+DEFINE_EVENT(crg_udc_log_enable_disable, crg_udc_pullup, TP_PROTO(int set),
+ TP_ARGS(set));
+
+TRACE_EVENT(crg_udc_no_room_on_ring, TP_PROTO(char *msg), TP_ARGS(msg),
+ TP_STRUCT__entry(__string(text, msg)),
+ TP_fast_assign(__assign_str(text);),
+ TP_printk("%s", __get_str(text)));
+
+TRACE_EVENT(crg_udc_ep_halt,
+ TP_PROTO(struct crg_udc_ep *udc_ep, u8 halt, u8 wedge),
+ TP_ARGS(udc_ep, halt, wedge),
+ TP_STRUCT__entry(__string(name, udc_ep->usb_ep.name)
+ __field(u8, halt) __field(u8, wedge)),
+ TP_fast_assign(__assign_str(name); __entry->halt = halt;
+ __entry->wedge = wedge;),
+ TP_printk("%s: %s%s", __get_str(name),
+ __entry->halt ? "halted" : "cleared",
+ __entry->wedge ? " (wedged)" : ""));
+
+TRACE_EVENT(crg_udc_portsc, TP_PROTO(u32 portsc), TP_ARGS(portsc),
+ TP_STRUCT__entry(__field(u32, portsc)),
+ TP_fast_assign(__entry->portsc = portsc;),
+ TP_printk("%s",
+ crg_udc_decode_portsc(__get_buf(CRG_MSG_MAX), CRG_MSG_MAX,
+ __entry->portsc)));
+
+TRACE_EVENT(crg_udc_doorbell, TP_PROTO(struct crg_gadget_dev *crg_udc, int dci),
+ TP_ARGS(crg_udc, dci),
+ TP_STRUCT__entry(__string(name, crg_udc->udc_ep[dci].usb_ep.name)
+ __field(int, dci)),
+ TP_fast_assign(__assign_str(name); __entry->dci = dci;),
+ TP_printk("%s, dci %d", __get_str(name), __entry->dci));
+
+DECLARE_EVENT_CLASS(
+ crg_udc_log_ctrl, TP_PROTO(struct usb_ctrlrequest *ctrl), TP_ARGS(ctrl),
+ TP_STRUCT__entry(__field(u8, bRequestType) __field(u8, bRequest)
+ __field(u16, wValue) __field(u16, wIndex)
+ __field(u16, wLength)),
+ TP_fast_assign(__entry->bRequestType = ctrl->bRequestType;
+ __entry->bRequest = ctrl->bRequest;
+ __entry->wValue = le16_to_cpu(ctrl->wValue);
+ __entry->wIndex = le16_to_cpu(ctrl->wIndex);
+ __entry->wLength = le16_to_cpu(ctrl->wLength);),
+ TP_printk("%s", usb_decode_ctrl(__get_buf(CRG_MSG_MAX), CRG_MSG_MAX,
+ __entry->bRequestType,
+ __entry->bRequest, __entry->wValue,
+ __entry->wIndex, __entry->wLength)));
+
+DEFINE_EVENT(crg_udc_log_ctrl, crg_udc_ctrl_req,
+ TP_PROTO(struct usb_ctrlrequest *ctrl), TP_ARGS(ctrl));
+
+DECLARE_EVENT_CLASS(
+ crg_udc_log_trb,
+ TP_PROTO(struct crg_udc_ep *udc_ep, struct crg_transfer_trb *trb),
+ TP_ARGS(udc_ep, trb),
+ TP_STRUCT__entry(__string(name, udc_ep->usb_ep.name)
+ __field(struct crg_transfer_trb *, trb)
+ __dynamic_array(char, decoded,
+ CRG_MSG_MAX)),
+ TP_fast_assign(__assign_str(name); __entry->trb = trb;
+ crg_udc_decode_trb(__get_dynamic_array(decoded),
+ CRG_MSG_MAX, trb);),
+ TP_printk("%s: trb %p, %s", __get_str(name), __entry->trb,
+ __get_str(decoded)));
+
+DEFINE_EVENT(crg_udc_log_trb, crg_udc_queue_trb,
+ TP_PROTO(struct crg_udc_ep *udc_ep, struct crg_transfer_trb *trb),
+ TP_ARGS(udc_ep, trb));
+
+TRACE_EVENT(
+ crg_udc_event_trb, TP_PROTO(struct crg_event_trb *trb), TP_ARGS(trb),
+ TP_STRUCT__entry(__field(struct crg_event_trb *, trb)
+ __dynamic_array(char, decoded, CRG_MSG_MAX)),
+ TP_fast_assign(__entry->trb = trb;
+ crg_udc_decode_event_trb(__get_dynamic_array(decoded),
+ CRG_MSG_MAX, trb);),
+ TP_printk("trb %p, %s", __entry->trb, __get_str(decoded)));
+
+DECLARE_EVENT_CLASS(
+ crg_udc_log_request,
+ TP_PROTO(struct crg_udc_ep *udc_ep, struct crg_udc_request *udc_req),
+ TP_ARGS(udc_ep, udc_req),
+ TP_STRUCT__entry(
+ __string(name, udc_ep->usb_ep.name) __field(
+ struct usb_request *, request) __field(void *, buf)
+ __field(unsigned int, actual) __field(unsigned int,
+ length)
+ __field(int, status) __field(dma_addr_t, dma)
+ __field(int, zero) __field(int,
+ short_not_ok)
+ __field(int, no_interrupt)),
+ TP_fast_assign(__assign_str(name); __entry->request = &udc_req->usb_req;
+ __entry->buf = udc_req->usb_req.buf;
+ __entry->actual = udc_req->usb_req.actual;
+ __entry->length = udc_req->usb_req.length;
+ __entry->status = udc_req->usb_req.status;
+ __entry->dma = udc_req->usb_req.dma;
+ __entry->zero = udc_req->usb_req.zero;
+ __entry->short_not_ok = udc_req->usb_req.short_not_ok;
+ __entry->no_interrupt = udc_req->usb_req.no_interrupt;),
+ TP_printk(
+ "%s: req: %p, buf: %p, length: %u/%u, status: %d, dma: %pad, %s%s%s",
+ __get_str(name), __entry->request, __entry->buf,
+ __entry->actual, __entry->length, __entry->status,
+ &__entry->dma, __entry->zero ? "Z" : "z",
+ __entry->short_not_ok ? "S" : "s",
+ __entry->no_interrupt ? "I" : "i"));
+
+DEFINE_EVENT(crg_udc_log_request, crg_udc_alloc_request,
+ TP_PROTO(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req),
+ TP_ARGS(udc_ep, udc_req));
+
+DEFINE_EVENT(crg_udc_log_request, crg_udc_free_request,
+ TP_PROTO(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req),
+ TP_ARGS(udc_ep, udc_req));
+
+DEFINE_EVENT(crg_udc_log_request, crg_udc_request_giveback,
+ TP_PROTO(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req),
+ TP_ARGS(udc_ep, udc_req));
+
+#endif /* __CRG_UDC_TRACE_H */
+
+/* This part must be outside header guard. */
+
+#undef TRACE_INCLUDE_PATH
+#define TRACE_INCLUDE_PATH .
+
+#undef TRACE_INCLUDE_FILE
+#define TRACE_INCLUDE_FILE crg-udc-trace
+
+#include <trace/define_trace.h>
diff --git a/drivers/usb/gadget/udc/crg/crg-udc.h b/drivers/usb/gadget/udc/crg/crg-udc.h
new file mode 100644
index 000000000000..246822219a8a
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc.h
@@ -0,0 +1,729 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+
+#ifndef CRG_UDC_H
+#define CRG_UDC_H
+
+#include <linux/bitfield.h>
+#include <linux/bits.h>
+#include <linux/dma-mapping.h>
+#include <linux/io.h>
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/workqueue.h>
+
+#include <linux/usb/ch9.h>
+#include <linux/usb/gadget.h>
+
+#define CRG_UCCR_OFFSET 0x2400
+#define CRG_UICR_OFFSET 0x2500
+#define CRG_UICR_STRIDE 0x20
+
+/* Corigine USB 3.1 device core register layout. */
+struct crg_uccr {
+ u32 capability;
+ u32 resv0[3];
+
+ u32 config0;
+ u32 config1;
+ u32 resv1[2];
+
+ u32 control;
+ u32 status;
+ u32 dcbaplo;
+ u32 dcbaphi;
+ u32 portsc;
+ u32 u3portpmsc;
+ u32 u2portpmsc;
+ u32 u3portli;
+
+ u32 doorbell;
+ u32 mfindex;
+ u32 ptmctrl;
+ u32 resv3[5];
+
+ u32 ep_enable;
+ u32 ep_running;
+ u32 resv4[2];
+
+ u32 cmd_param0;
+ u32 cmd_param1;
+ u32 cmd_control;
+ u32 resv5[1];
+
+ u32 odb_capability;
+ u32 resv6[3];
+ u32 odb_config[8];
+
+ u32 debug0;
+};
+
+struct crg_uicr {
+ u32 iman;
+ u32 imod;
+ u32 erstsz;
+ u32 resv0;
+
+ u32 erstbalo;
+ u32 erstbahi;
+ u32 erdplo;
+ u32 erdphi;
+};
+
+/* config0/config1 registers */
+#define CRG_U3DC_CFG0_MAXSPEED_MASK GENMASK(3, 0)
+#define CRG_U3DC_CFG0_MAXSPEED_FS 1
+#define CRG_U3DC_CFG0_MAXSPEED_HS 3
+#define CRG_U3DC_CFG0_MAXSPEED_SS 4
+#define CRG_U3DC_CFG0_MAXSPEED_SSP 5
+
+#define CRG_U3DC_CFG1_CSC_EVENT_EN BIT(0)
+#define CRG_U3DC_CFG1_PEC_EVENT_EN BIT(1)
+#define CRG_U3DC_CFG1_PPC_EVENT_EN BIT(3)
+#define CRG_U3DC_CFG1_PRC_EVENT_EN BIT(4)
+#define CRG_U3DC_CFG1_PLC_EVENT_EN BIT(5)
+#define CRG_U3DC_CFG1_CEC_EVENT_EN BIT(6)
+
+#define CRG_U3DC_CFG1_U3_ENTRY_EN BIT(8)
+#define CRG_U3DC_CFG1_L1_ENTRY_EN BIT(9)
+#define CRG_U3DC_CFG1_U3_RESUME_EN BIT(10)
+#define CRG_U3DC_CFG1_L1_RESUME_EN BIT(11)
+#define CRG_U3DC_CFG1_INACTIVE_PLC_EN BIT(12)
+#define CRG_U3DC_CFG1_U3_RESUME_NORESP_PLC_EN BIT(13)
+#define CRG_U3DC_CFG1_U2_RESUME_NORESP_PLC_EN BIT(14)
+
+#define CRG_U3DC_CFG1_SETUP_EVENT_EN BIT(16)
+
+/* control register */
+#define CRG_U3DC_CTRL_RUN BIT(0)
+#define CRG_U3DC_CTRL_STOP 0
+#define CRG_U3DC_CTRL_SWRST BIT(1)
+#define CRG_U3DC_CTRL_INT_EN BIT(2)
+#define CRG_U3DC_CTRL_SYSERR_EN BIT(3)
+#define CRG_U3DC_CTRL_EWE BIT(10)
+#define CRG_U3DC_CTRL_KP_CNCT BIT(11)
+
+/* status register */
+#define CRG_U3DC_STATUS_DEV_CTRL_HALT BIT(0)
+#define CRG_U3DC_STATUS_SYS_ERR BIT(2)
+#define CRG_U3DC_STATUS_EINT BIT(3)
+
+/* portsc register */
+#define CRG_U3DC_PORTSC_CCS BIT(0)
+#define CRG_U3DC_PORTSC_PED BIT(1)
+#define CRG_U3DC_PORTSC_PP BIT(3)
+#define CRG_U3DC_PORTSC_PR BIT(4)
+#define CRG_U3DC_PORTSC_PLS_MASK GENMASK(8, 5)
+#define CRG_U3DC_PORTSC_PLS_U3 3
+
+#define CRG_U3DC_PORTSC_SPEED_MASK GENMASK(13, 10)
+#define CRG_U3DC_PORTSC_SPEED_FS 0x1
+#define CRG_U3DC_PORTSC_SPEED_LS 0x2
+#define CRG_U3DC_PORTSC_SPEED_HS 0x3
+#define CRG_U3DC_PORTSC_SPEED_SS 0x4
+#define CRG_U3DC_PORTSC_SPEED_SSP 0x5
+
+#define CRG_U3DC_PORTSC_LWS BIT(16)
+#define CRG_U3DC_PORTSC_CSC BIT(17)
+#define CRG_U3DC_PORTSC_PEC BIT(18)
+#define CRG_U3DC_PORTSC_PPC BIT(20)
+#define CRG_U3DC_PORTSC_PRC BIT(21)
+#define CRG_U3DC_PORTSC_PLC BIT(22)
+#define CRG_U3DC_PORTSC_CEC BIT(23)
+#define CRG_U3DC_PORTSC_WCE BIT(25)
+#define CRG_U3DC_PORTSC_WDE BIT(26)
+#define CRG_U3DC_PORTSC_WPR BIT(31)
+
+#define CRG_U3DC_PORTSC_W1C_MASK \
+ (CRG_U3DC_PORTSC_CSC | CRG_U3DC_PORTSC_PEC | CRG_U3DC_PORTSC_PPC | \
+ CRG_U3DC_PORTSC_PRC | CRG_U3DC_PORTSC_PLC | CRG_U3DC_PORTSC_CEC)
+
+/* mfindex register */
+#define CRG_U3DC_MFINDEX_CURRENT_MASK GENMASK(17, 4)
+#define CRG_U3DC_MFINDEX_FRAME_MASK GENMASK(17, 7)
+
+/* ptmctrl register */
+#define CRG_U3DC_PTMCTRL_ISOCH_DELAY_MASK GENMASK(13, 0)
+
+/* u3portpmsc */
+#define CRG_U3DC_U3PORTPM_U1TMOUT_MASK GENMASK(7, 0)
+#define CRG_U3DC_U3PORTPM_U2TMOUT_MASK GENMASK(15, 8)
+#define CRG_U3DC_U3PORTPM_FLA BIT(16)
+#define CRG_U3DC_U3PORTPM_U1IEN BIT(20)
+#define CRG_U3DC_U3PORTPM_U2IEN BIT(21)
+#define CRG_U3DC_U3PORTPM_U1AEN BIT(22)
+#define CRG_U3DC_U3PORTPM_U2AEN BIT(23)
+
+#define CRG_U3DC_U3PORTPM_U1U2TMOUT_MASK GENMASK(31, 24)
+
+/* u2portpmsc */
+#define CRG_U3DC_U2PORTPM_RJ_TH_MASK GENMASK(3, 0)
+#define CRG_U3DC_U2PORTPM_DS_TH_MASK GENMASK(7, 4)
+#define CRG_U3DC_U2PORTPM_LPM_EN BIT(8)
+#define CRG_U3DC_U2PORTPM_RJ_TH_EN BIT(9)
+#define CRG_U3DC_U2PORTPM_DS_EN BIT(10)
+#define CRG_U3DC_U2PORTPM_SLP_EN BIT(11)
+#define CRG_U3DC_U2PORTPM_LPM_FACK BIT(12)
+#define CRG_U3DC_U2PORTPM_L1_AEX BIT(13)
+#define CRG_U3DC_U2PORTPM_H_B_MASK GENMASK(19, 16)
+#define CRG_U3DC_U2PORTPM_RWE BIT(20)
+#define CRG_U3DC_U2PORTPM_TM_MASK GENMASK(31, 28)
+
+/* doorbell register */
+#define CRG_U3DC_DB_TARGET_MASK GENMASK(4, 0)
+
+/* odb registers */
+#define CRG_U3DC_ODBCFG_2N_OFFSET_MASK GENMASK(9, 0)
+#define CRG_U3DC_ODBCFG_2N_SIZE_MASK GENMASK(12, 10)
+#define CRG_U3DC_ODBCFG_2N1_OFFSET_MASK GENMASK(25, 16)
+#define CRG_U3DC_ODBCFG_2N1_SIZE_MASK GENMASK(28, 26)
+
+/* command control register */
+#define CRG_U3DC_CMD_CTRL_ACTIVE BIT(0)
+#define CRG_U3DC_CMD_CTRL_IOC_EN BIT(1)
+#define CRG_U3DC_CMD_CTRL_TYPE_MASK GENMASK(7, 4)
+#define CRG_U3DC_CMD_CTRL_STATUS_MASK GENMASK(19, 16)
+
+/* interrupt register / IMAN bits */
+#define CRG_U3DC_IMAN_INT_PEND BIT(0)
+#define CRG_U3DC_IMAN_INT_EN BIT(1)
+
+/* erdp bits */
+#define CRG_U3DC_ERDPLO_EHB BIT(3)
+
+/* command parameters: init EP0 */
+#define CRG_CMD0_0_DQPTRLO_MASK GENMASK(31, 4)
+#define CRG_CMD0_0_DCS_MASK BIT(0)
+
+/* command parameters: update EP0 */
+#define CRG_CMD1_0_MPS_MASK GENMASK(31, 16)
+
+/* command parameters: set address */
+#define CRG_CMD2_0_DEV_ADDR_MASK GENMASK(7, 0)
+
+#define CRG_UDC_MODE_REG 0x20fc
+#define CRG_UDC_ROLE_DEVICE 0x1
+
+#define CRG_UDC_VENDOR_REG 0x210c
+#define CRG_UDC_VBUS_DETECT BIT(26)
+
+/*
+ * Device-visible data structures. The controller reads and writes these
+ * over DMA, so the layouts and field encodings below are fixed by the
+ * hardware in the same way as the register definitions above.
+ */
+
+/* Transfer and event TRB types */
+enum crg_trb_type {
+ TRB_TYPE_RSVD = 0,
+ TRB_TYPE_XFER_NORMAL,
+ TRB_TYPE_RSVD2,
+ TRB_TYPE_XFER_DATA_STAGE,
+ TRB_TYPE_XFER_STATUS_STAGE,
+ TRB_TYPE_XFER_DATA_ISOCH,
+ TRB_TYPE_LINK,
+ TRB_TYPE_RSVD7,
+ TRB_TYPE_NO_OP,
+
+ TRB_TYPE_EVT_TRANSFER = 32,
+ TRB_TYPE_EVT_CMD_COMPLETION = 33,
+ TRB_TYPE_EVT_PORT_STATUS_CHANGE = 34,
+ TRB_TYPE_EVT_MFINDEX_WRAP = 39,
+ TRB_TYPE_EVT_SETUP_PKT = 40,
+};
+
+/* Event TRB completion codes */
+enum crg_trb_cmpl_code {
+ CMPL_CODE_INVALID = 0,
+ CMPL_CODE_SUCCESS,
+ CMPL_CODE_DATA_BUFFER_ERR,
+ CMPL_CODE_BABBLE_DETECTED_ERR,
+ CMPL_CODE_USB_TRANS_ERR,
+ CMPL_CODE_TRB_ERR,
+ CMPL_CODE_TRB_STALL,
+ CMPL_CODE_INVALID_STREAM_TYPE_ERR = 10,
+ CMPL_CODE_SHORT_PKT = 13,
+ CMPL_CODE_RING_UNDERRUN,
+ CMPL_CODE_RING_OVERRUN,
+ CMPL_CODE_EVENT_RING_FULL_ERR = 21,
+ CMPL_CODE_STOPPED = 26,
+ CMPL_CODE_STOPPED_LENGTH_INVALID = 27,
+ CMPL_CODE_ISOCH_BUFFER_OVERRUN = 31,
+
+ CMPL_CODE_PROTOCOL_STALL = 192,
+ CMPL_CODE_SETUP_TAG_MISMATCH = 193,
+ CMPL_CODE_HALTED = 194,
+ CMPL_CODE_HALTED_LENGTH_INVALID = 195,
+ CMPL_CODE_DISABLED = 196,
+ CMPL_CODE_DISABLED_LENGTH_INVALID = 197,
+};
+
+/* Endpoint type encoding in the endpoint context */
+enum crg_ep_type {
+ EP_TYPE_INVALID = 0,
+ EP_TYPE_ISOCH_OUTBOUND,
+ EP_TYPE_BULK_OUTBOUND,
+ EP_TYPE_INTR_OUTBOUND,
+ EP_TYPE_INVALID2,
+ EP_TYPE_ISOCH_INBOUND,
+ EP_TYPE_BULK_INBOUND,
+ EP_TYPE_INTR_INBOUND,
+};
+
+/* crg_transfer_trb dw2 fields */
+#define TRB_TRANSFER_LEN_MASK GENMASK(16, 0)
+#define TRB_TD_SIZE_MASK GENMASK(21, 17)
+#define TRB_INTR_TARGET_MASK GENMASK(31, 22)
+
+/* crg_transfer_trb dw3 fields */
+#define TRB_CYCLE_BIT_MASK BIT(0)
+#define TRB_LINK_TOGGLE_CYCLE_MASK BIT(1)
+#define TRB_INTR_ON_SHORT_PKT_MASK BIT(2)
+#define TRB_NO_SNOOP_MASK BIT(3)
+#define TRB_CHAIN_BIT_MASK BIT(4)
+#define TRB_INTR_ON_COMPLETION_MASK BIT(5)
+#define TRB_APPEND_ZLP_MASK BIT(7)
+#define TRB_BLOCK_EVENT_INT_MASK BIT(9)
+#define TRB_TYPE_MASK GENMASK(15, 10)
+#define DATA_STAGE_TRB_DIR_MASK BIT(16)
+#define TRB_SETUP_TAG_MASK GENMASK(18, 17)
+#define STATUS_STAGE_TRB_STALL_MASK BIT(19)
+#define STATUS_STAGE_TRB_SET_ADDR_MASK BIT(20)
+#define ISOC_TRB_FRAME_ID_MASK GENMASK(30, 20)
+#define ISOC_TRB_SIA_MASK BIT(31)
+
+struct crg_transfer_trb {
+ __le32 dw0;
+ __le32 dw1;
+ __le32 dw2;
+ __le32 dw3;
+};
+
+/* crg_event_trb dw2 fields */
+#define EVE_TRB_TRAN_LEN_MASK GENMASK(16, 0)
+#define EVE_TRB_COMPL_CODE_MASK GENMASK(31, 24)
+
+/* crg_event_trb dw3 fields */
+#define EVE_TRB_CYCLE_BIT_MASK BIT(0)
+#define EVE_TRB_TYPE_MASK GENMASK(15, 10)
+#define EVE_TRB_ENDPOINT_ID_MASK GENMASK(20, 16)
+#define EVE_TRB_SETUP_TAG_MASK GENMASK(22, 21)
+
+struct crg_event_trb {
+ __le32 dw0;
+ __le32 dw1;
+ __le32 dw2;
+ __le32 dw3;
+};
+
+/* crg_ep_cx dw0 fields */
+#define EP_CX_LOGICAL_EP_NUM_MASK GENMASK(6, 3)
+#define EP_CX_INTERVAL_MASK GENMASK(23, 16)
+
+/* crg_ep_cx dw1 fields */
+#define EP_CX_EP_TYPE_MASK GENMASK(5, 3)
+#define EP_CX_MAX_BURST_SIZE_MASK GENMASK(15, 8)
+#define EP_CX_MAX_PACKET_SIZE_MASK GENMASK(31, 16)
+
+/* crg_ep_cx dw2 fields */
+#define EP_CX_DEQ_CYC_STATE_MASK BIT(0)
+#define EP_CX_TR_DQPT_LO_MASK GENMASK(31, 4)
+
+struct crg_ep_cx {
+ __le32 dw0;
+ __le32 dw1;
+ __le32 dw2;
+ __le32 dw3;
+};
+
+struct crg_erst {
+ /* 64-bit event ring segment address */
+ __le32 seg_addr_lo;
+ __le32 seg_addr_hi;
+ __le32 seg_size;
+ /* Set to zero */
+ __le32 rsvd;
+};
+
+/* command type */
+enum crg_cmd_type {
+ CRG_CMD_INIT_EP0 = 0,
+ CRG_CMD_UPDATE_EP0_CFG = 1,
+ CRG_CMD_SET_ADDR = 2,
+ CRG_CMD_SEND_DEV_NOTIFICATION = 3,
+ CRG_CMD_CONFIG_EP = 4,
+ CRG_CMD_SET_HALT = 5,
+ CRG_CMD_CLEAR_HALT = 6,
+ CRG_CMD_RESET_SEQNUM = 7,
+ CRG_CMD_STOP_EP = 8,
+ CRG_CMD_SET_TR_DQPTR = 9,
+ CRG_CMD_FORCE_FLOW_CONTROL = 10,
+ CRG_CMD_REQ_LDM_EXCHANGE = 11,
+};
+
+#define CRG_MAX_PACKET_SIZE 1024
+
+#define CRG_ERST_SIZE 1
+#define CRG_EVENT_RING_SIZE 256
+#define CRG_NUM_EP_CX 32
+
+/*
+ * Cap on events drained per udc_lock acquisition in crg_udc_process_event_ring().
+ * Command polling and crg_udc_reset()'s hardware-reset poll both run from
+ * crg_udc_handle_event() with udc_lock held, so without a cap a busy event
+ * ring could keep the lock, and local IRQs, held for as long as events
+ * keep arriving.
+ */
+#define CRG_EVENT_RING_BATCH 32
+
+#define CRG_UDC_TRB_MAX_BUFFER_SIZE 65536
+#define CRG_UDC_CONTROL_EP_TD_RING_SIZE 16
+#define CRG_UDC_BULK_EP_TD_RING_SIZE 1024
+#define CRG_UDC_ISOC_EP_TD_RING_SIZE 32
+#define CRG_UDC_INT_EP_TD_RING_SIZE 8
+
+#define CRG_UDC_U1_TIMEOUT_VAL 0x70
+#define CRG_UDC_U2_TIMEOUT_VAL 0x70
+
+enum crg_ep_state {
+ EP_STATE_DISABLED = 0,
+ EP_STATE_RUNNING = 1,
+ EP_STATE_HALTED = 2,
+ EP_STATE_STOPPED = 3
+};
+
+/* control-transfer setup state machine */
+enum crg_setup_status {
+ WAIT_FOR_SETUP = 0,
+ SETUP_PKT_PROCESS_IN_PROGRESS,
+ DATA_STAGE_XFER,
+ DATA_STAGE_RECV,
+ STATUS_STAGE_XFER,
+ STATUS_STAGE_RECV,
+};
+
+/* interrupt-on-completion moderation rate (in TRBs) per endpoint type */
+enum {
+ CRG_UDC_BULK_EP_INTERRUPT_RATE = 5,
+ CRG_UDC_ISOC_EP_INTERRUPT_RATE = 1,
+};
+
+struct crg_buffer_info {
+ void *cpu_addr;
+ dma_addr_t dma_addr;
+ u32 len;
+};
+
+struct crg_sel_value {
+ u16 u2_pel_value;
+ u16 u2_sel_value;
+ u8 u1_pel_value;
+ u8 u1_sel_value;
+};
+
+struct crg_udc_request {
+ struct usb_request usb_req;
+ struct list_head queue;
+ bool mapped;
+ /*
+ * Direction used for the DMA mapping call that produced the mapping
+ * tracked above. ep0 is bidirectional and its
+ * descriptor address is always 0, so usb_endpoint_dir_in() on the
+ * endpoint cannot tell an unmap apart from a control IN or OUT
+ * transfer; the direction actually used to map is recorded here
+ * instead and reused as-is at unmap time.
+ */
+ enum dma_data_direction map_dir;
+ struct scatterlist *sg;
+ unsigned int sg_index;
+ u32 sg_offset;
+ u64 buff_len_left;
+ u32 trbs_needed;
+ struct crg_transfer_trb *first_trb;
+ struct crg_transfer_trb *last_trb;
+ bool all_trbs_queued;
+ /* A TD has been published and its completion event is outstanding. */
+ bool td_pending;
+};
+
+struct crg_udc_ep {
+ struct usb_ep usb_ep;
+
+ struct crg_buffer_info tran_ring_info;
+ struct crg_transfer_trb *first_trb;
+ struct crg_transfer_trb *last_trb;
+
+ struct crg_transfer_trb *enq_pt;
+ struct crg_transfer_trb *deq_pt;
+ u8 pcs;
+
+ char name[10];
+ u8 dci;
+ struct list_head queue;
+ const struct usb_endpoint_descriptor *desc;
+ const struct usb_ss_ep_comp_descriptor *comp_desc;
+ bool tran_ring_full;
+ bool configuring;
+ u32 generation;
+ struct crg_gadget_dev *crg_udc;
+
+ enum crg_ep_state ep_state;
+
+ bool wedge;
+};
+
+#define CRG_RING_NUM 1
+
+struct crg_udc_event {
+ struct crg_buffer_info erst;
+ struct crg_erst *p_erst;
+ struct crg_buffer_info event_ring;
+ struct crg_event_trb *evt_dq_pt;
+ /* Changed on every ring reinitialization under udc_lock. */
+ u32 generation;
+ u8 ccs;
+ struct crg_event_trb *evt_seg0_last_trb;
+};
+
+enum crg_ep0_state {
+ EP0_STATE_IDLE = 0,
+ EP0_STATE_ABORTING,
+ EP0_STATE_ERROR,
+};
+
+struct crg_gadget_dev {
+ void __iomem *mmio_virt_base;
+ struct crg_uccr __iomem *uccr;
+ struct crg_uicr __iomem *uicr[CRG_RING_NUM];
+
+ /* udc_lock device lock */
+ spinlock_t udc_lock;
+
+ struct device *dev;
+ struct clk *clk;
+ struct phy *phy;
+
+ struct usb_gadget gadget;
+ struct usb_gadget_driver *gadget_driver;
+ /* Speed selected for the currently bound gadget driver. */
+ enum usb_device_speed max_speed;
+
+ int irq;
+
+ struct usb_endpoint_descriptor ep0_desc;
+
+ struct crg_udc_ep udc_ep[CRG_NUM_EP_CX];
+ struct crg_buffer_info ep_cx;
+ struct crg_ep_cx *p_epcx;
+
+ struct crg_udc_event udc_event[CRG_RING_NUM];
+
+ struct crg_udc_request *status_req;
+ /* Protected by udc_lock, remains set during a data-stage giveback. */
+ bool status_req_callback_pending;
+ struct crg_sel_value sel_value;
+ void (*setup_fn_call_back)(struct crg_gadget_dev *a);
+
+ enum crg_setup_status setup_status;
+ /* EP0 rejects new requests while the previous control transfer is
+ * being aborted and its completion callbacks are being given back.
+ */
+ enum crg_ep0_state ep0_state;
+
+ u8 device_state;
+ u16 dev_addr;
+ u8 setup_tag;
+ u8 set_tm;
+
+ u32 num_enabled_eps;
+
+ bool connected;
+ bool pullup_on;
+ bool async_callbacks;
+ /*
+ * Set while crg_udc_reset() has udc_lock dropped and is polling for
+ * SWRST to clear. Checked by crg_udc_ep_queue() so a request queued
+ * on another endpoint during that window is rejected instead of
+ * being built into a ring and doorbell-rung while the controller is
+ * still mid-reset.
+ */
+ bool resetting;
+
+ bool u2_rwe;
+ bool feature_u1_enable;
+ bool feature_u2_enable;
+
+ bool portsc_on_reconnecting;
+ bool u12_forbidden;
+ bool u12_initiate_forbidden;
+ bool reinit_ep0_on_bus_reset;
+
+ /*
+ * usb_gadget_udc_reset() re-enters the gadget/function drivers
+ * synchronously. Calling it directly from the port-status IRQ path
+ * would require dropping udc_lock mid-event-processing, opening a
+ * window for a second port-status event to run crg_udc_reinit()
+ * again while the first event's SET_CONFIGURATION/set_alt() work is
+ * still landing on another thread, tearing the just-enabled
+ * endpoints back down. Defer the notification to this work item so
+ * the IRQ-thread's event processing never has to drop the lock.
+ */
+ struct work_struct udc_reset_work;
+};
+
+/* Shared prototypes */
+void crg_udc_build_ep0_status(struct crg_udc_ep *udc_ep_ptr, bool default_value,
+ u32 status, struct crg_udc_request *udc_req_ptr,
+ u8 set_addr, u8 stall);
+/*
+ * Abort the current EP0 control transfer when a new SETUP packet arrives.
+ *
+ * A SETUP packet is allowed to abort any in-progress control transfer. Do
+ * not defer the new SETUP until the old transfer completes: doing so leaves
+ * the old request on the EP0 software queue and subsequent gadget requests
+ * are rejected as "queue not empty".
+ *
+ * The controller's EP0 ring is reinitialized at the ring start, making all
+ * TRBs belonging to the aborted control transfer stale. The new SETUP is
+ * then handled immediately with a fresh setup tag.
+ */
+int crg_udc_abort_ep0_for_setup(struct crg_gadget_dev *crg_udc);
+int crg_udc_enable_setup(struct crg_gadget_dev *crg_udc);
+void crg_udc_ep0_req_complete(struct crg_udc_ep *udc_ep_ptr,
+ unsigned long *flags);
+int crg_udc_prepare_for_setup(struct crg_gadget_dev *crg_udc);
+int crg_udc_set_ep0_halt(struct crg_gadget_dev *crg_udc);
+int crg_udc_update_ep0_maxpacketsize(struct crg_gadget_dev *crg_udc);
+void crg_udc_handle_setup_pkt(struct crg_gadget_dev *crg_udc,
+ struct usb_ctrlrequest *setup_pkt, u8 setup_tag,
+ unsigned long *flags);
+int crg_udc_init_ep0(struct crg_gadget_dev *crg_udc);
+int crg_udc_queue_ctrl(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr,
+ u32 num_of_trbs_needed);
+void crg_udc_advance_dequeue_pt(struct crg_udc_ep *udc_ep);
+unsigned int crg_udc_count_trbs_linear(dma_addr_t addr, u64 len);
+unsigned int crg_udc_count_trbs_sg(struct crg_udc_request *udc_req);
+int crg_udc_init_event_ring(struct crg_gadget_dev *crg_udc, int index);
+bool crg_udc_is_event_rings_empty(struct crg_gadget_dev *crg_udc);
+bool crg_udc_is_pointer_less_than(struct crg_transfer_trb *a,
+ struct crg_transfer_trb *b,
+ struct crg_udc_ep *udc_ep);
+int crg_udc_process_event_ring(struct crg_gadget_dev *crg_udc, int index,
+ unsigned long *flags);
+/* This function will go through the list of the USB requests for the
+ * given endpoint and schedule any unscheduled trb's to the xfer ring
+ */
+int crg_udc_queue_pending_trbs(struct crg_udc_ep *udc_ep_ptr);
+/* num_trbs here is the size of the ring. */
+u32 crg_udc_room_on_ring(struct crg_gadget_dev *crg_udc, u32 num_trbs,
+ struct crg_transfer_trb *p_ring,
+ struct crg_transfer_trb *enq_pt,
+ struct crg_transfer_trb *dq_pt);
+void crg_udc_setup_datastage_trb(struct crg_transfer_trb *p_trb, dma_addr_t dma,
+ u8 pcs, u32 transfer_length, u32 td_size,
+ bool ioc, bool chain, bool azp, u8 dir, u8 setup_tag);
+void crg_udc_setup_link_trb(struct crg_transfer_trb *link_trb, bool toggle,
+ dma_addr_t next_trb);
+void crg_udc_setup_status_trb(struct crg_gadget_dev *crg_udc,
+ struct crg_transfer_trb *p_trb,
+ struct usb_request *usb_req, u8 pcs, u8 set_addr,
+ u8 stall);
+void crg_udc_setup_trb(struct crg_transfer_trb *p_trb, u32 xfer_len,
+ dma_addr_t xfer_buf_addr, u8 td_size, u8 pcs,
+ u8 trb_type, bool short_pkt, bool chain_bit,
+ bool intr_on_compl, bool b_setup_stage, u8 usb_dir,
+ bool b_isoc, u16 frame_id, bool sia, bool azp);
+void crg_udc_squeeze_xfer_ring(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr);
+struct crg_transfer_trb *crg_udc_tran_trb_dma_to_virt(struct crg_udc_ep *udc_ep,
+ dma_addr_t address);
+dma_addr_t crg_udc_tran_trb_virt_to_dma(struct crg_udc_ep *udc_ep,
+ struct crg_transfer_trb *trb);
+void crg_udc_update_dequeue_pt(struct crg_event_trb *event,
+ struct crg_udc_ep *udc_ep);
+u32 crg_udc_trb_chunk_len(dma_addr_t dma, u64 remaining);
+u32 crg_udc_actual_before(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req,
+ struct crg_transfer_trb *pause);
+/*
+ * command related ops
+ */
+int crg_udc_issue_command(struct crg_gadget_dev *crg_udc,
+ enum crg_cmd_type type, u32 param0, u32 param1);
+void crg_udc_free_resources(struct crg_gadget_dev *crg_udc);
+int crg_udc_program_speed(struct crg_gadget_dev *crg_udc);
+int crg_udc_reset_data_struct(struct crg_gadget_dev *crg_udc);
+int crg_udc_reset(struct crg_gadget_dev *crg_udc, unsigned long *flags);
+struct usb_request *crg_udc_alloc_request(struct usb_ep *_ep, gfp_t gfp_flags);
+int crg_udc_build_td(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr);
+void crg_udc_clear_portpm(struct crg_gadget_dev *crg_udc);
+void crg_udc_ep_nuke(struct crg_udc_ep *udc_ep, int status);
+void crg_udc_free_request(struct usb_ep *_ep, struct usb_request *_req);
+void crg_udc_set_pullup(struct crg_gadget_dev *crg_udc, bool is_on);
+void crg_udc_reset_work(struct work_struct *work);
+void crg_udc_set_vbus_detect(struct crg_gadget_dev *crg_udc, bool enable);
+/*
+ * Undo whatever crg_udc_ep_queue() mapped for this request: the merged
+ * SG list, the single buffer, or neither if the request carried a
+ * pre-mapped DMA address.
+ */
+void crg_udc_unmap_request(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req);
+int crg_udc_handle_port_status(struct crg_gadget_dev *crg_udc,
+ unsigned long *flags);
+int crg_udc_handle_xfer_event(struct crg_gadget_dev *crg_udc,
+ struct crg_event_trb *event, unsigned long *flags);
+int crg_udc_handle_event(struct crg_gadget_dev *crg_udc,
+ struct crg_event_trb *event, unsigned long *flags);
+/*
+ * Invoke the bound gadget driver's ->disconnect() callback, if any.
+ *
+ * The gadget disconnect callback may be called in interrupt context.
+ * Release only udc_lock while keeping local IRQs disabled.
+ */
+void crg_udc_notify_disconnect(struct crg_gadget_dev *crg_udc,
+ unsigned long *flags);
+int crg_udc_ep_halt(struct crg_udc_ep *udc_ep_ptr, int halt, int ignore_wedge,
+ unsigned long *flags);
+enum crg_ep_state crg_udc_get_ep_state(struct crg_gadget_dev *crg_udc, int dci);
+u32 crg_udc_index2dci(u16 index);
+int crg_udc_init_device_context(struct crg_gadget_dev *crg_udc);
+void crg_udc_init_ep_info(struct crg_gadget_dev *crg_udc);
+void crg_udc_knock_doorbell(struct crg_gadget_dev *crg_udc, int dci);
+/*
+ * Complete a request and give it back to the gadget layer via the
+ * standard usb_gadget_giveback_request() helper.
+ *
+ * Must be called with udc_lock held and interrupts disabled.
+ *
+ * The request's ->complete() callback must not sleep. crg_udc_req_done() drops
+ * udc_lock before the giveback and re-acquires it before returning.
+ * Callers therefore cannot assume state protected by udc_lock is
+ * unchanged across this call.
+ */
+void crg_udc_req_done(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req, int status);
+void crg_udc_clear_req_container(struct crg_udc_request *udc_req_ptr);
+
+#define CRG_QUIRK_DIS_U1U2_ENTRY BIT(0)
+#define CRG_QUIRK_DIS_U1U2_INITIATION BIT(1)
+#define CRG_QUIRK_REINIT_EP0_ON_RESET BIT(2)
+
+#define CRG_UDC_QUIRKS_DEFAULT \
+ (CRG_QUIRK_DIS_U1U2_ENTRY | CRG_QUIRK_DIS_U1U2_INITIATION | \
+ CRG_QUIRK_REINIT_EP0_ON_RESET)
+
+struct crg_udc_priv {
+ u32 quirks;
+};
+
+extern const struct usb_gadget_ops crg_gadget_ops;
+extern const struct usb_ep_ops crg_udc_ep_ops;
+#define gadget_to_udc(g) (container_of(g, struct crg_gadget_dev, gadget))
+
+#endif /* CRG_UDC_H */
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH v4 3/6] usb: gadget: udc: crg: Add ring and transfer management
2026-10-30 8:10 [PATCH v4 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
2026-10-30 8:10 ` [PATCH v4 1/6] dt-bindings: usb: gadget: udc: add Axiado AX3000 and AX3005 UDC Petar Stepanovic
2026-10-30 8:10 ` [PATCH v4 2/6] usb: gadget: udc: crg: Add hardware definitions and structs Petar Stepanovic
@ 2026-10-30 8:10 ` Petar Stepanovic
2026-10-30 8:29 ` sashiko-bot
2026-10-30 8:10 ` [PATCH v4 4/6] usb: gadget: udc: crg: Add endpoint 0 handling Petar Stepanovic
` (2 subsequent siblings)
5 siblings, 1 reply; 14+ messages in thread
From: Petar Stepanovic @ 2026-10-30 8:10 UTC (permalink / raw)
To: Naveen Kumar Rajgiri Bassappa, Prasad Bolisetty,
Greg Kroah-Hartman, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Tzu-Hao Wei, Karthikeyan Mitran, Philipp Zabel
Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
Petar Stepanovic, Krutik Shah
This patch introduces the memory management and ring buffer logic
(TRBs and event rings) required by the Axiado AX3000/AX3005 UDC.
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
drivers/usb/gadget/udc/crg/crg-udc-ring.c | 668 ++++++++++++++++++++++++++++++
1 file changed, 668 insertions(+)
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-ring.c b/drivers/usb/gadget/udc/crg/crg-udc-ring.c
new file mode 100644
index 000000000000..f1de8893b790
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-ring.c
@@ -0,0 +1,668 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/sched.h>
+#include <linux/scatterlist.h>
+
+#include "crg-udc.h"
+
+void crg_udc_advance_dequeue_pt(struct crg_udc_ep *udc_ep)
+{
+ struct crg_udc_request *udc_req;
+
+ if (!list_empty(&udc_ep->queue)) {
+ udc_req = list_entry(udc_ep->queue.next, struct crg_udc_request,
+ queue);
+
+ if (udc_req->first_trb)
+ udc_ep->deq_pt = udc_req->first_trb;
+ else
+ udc_ep->deq_pt = udc_ep->enq_pt;
+ } else {
+ udc_ep->deq_pt = udc_ep->enq_pt;
+ }
+}
+
+unsigned int crg_udc_count_trbs_linear(dma_addr_t addr, u64 len)
+{
+ unsigned int num_trbs = 0;
+
+ while (len) {
+ u32 chunk = crg_udc_trb_chunk_len(addr, len);
+
+ if (!chunk)
+ return 0;
+ num_trbs++;
+ addr += chunk;
+ len -= chunk;
+ }
+
+ return num_trbs;
+}
+
+unsigned int crg_udc_count_trbs_sg(struct crg_udc_request *udc_req)
+{
+ struct scatterlist *sg;
+ u64 remaining = udc_req->usb_req.length;
+ unsigned int i;
+ unsigned int num_trbs = 0;
+
+ for_each_sg(udc_req->usb_req.sg, sg, udc_req->usb_req.num_sgs, i) {
+ dma_addr_t addr;
+ u64 len;
+
+ if (!sg_dma_len(sg))
+ continue;
+ if (!remaining)
+ break;
+
+ len = min_t(u64, sg_dma_len(sg), remaining);
+ addr = sg_dma_address(sg);
+ while (len) {
+ u32 chunk = crg_udc_trb_chunk_len(addr, len);
+
+ if (!chunk)
+ return 0;
+ num_trbs++;
+ addr += chunk;
+ len -= chunk;
+ remaining -= chunk;
+ }
+ }
+
+ return remaining ? 0 : num_trbs;
+}
+
+static dma_addr_t event_trb_virt_to_dma(struct crg_udc_event *udc_event,
+ struct crg_event_trb *event)
+{
+ dma_addr_t dma_addr = 0;
+ unsigned long seg_offset;
+
+ if (!udc_event || !event)
+ return DMA_MAPPING_ERROR;
+
+ if ((void *)event < udc_event->event_ring.cpu_addr ||
+ (void *)event >=
+ udc_event->event_ring.cpu_addr + udc_event->event_ring.len)
+ return DMA_MAPPING_ERROR;
+
+ seg_offset = (void *)event - udc_event->event_ring.cpu_addr;
+ dma_addr = udc_event->event_ring.dma_addr + seg_offset;
+
+ return dma_addr;
+}
+
+int crg_udc_init_event_ring(struct crg_gadget_dev *crg_udc, int index)
+{
+ struct crg_uicr __iomem *uicr = crg_udc->uicr[index];
+ struct crg_udc_event *udc_event = &crg_udc->udc_event[index];
+
+ int ret;
+ u32 buff_length;
+ dma_addr_t mapping;
+ bool erst_allocated = false;
+
+ buff_length = CRG_ERST_SIZE * sizeof(struct crg_erst);
+ if (!udc_event->erst.cpu_addr) {
+ udc_event->erst.cpu_addr = dma_alloc_coherent(
+ crg_udc->dev, buff_length, &mapping, GFP_ATOMIC);
+
+ if (!udc_event->erst.cpu_addr)
+ return -ENOMEM;
+ erst_allocated = true;
+ } else {
+ mapping = udc_event->erst.dma_addr;
+ }
+
+ udc_event->erst.len = buff_length;
+ udc_event->erst.dma_addr = mapping;
+ udc_event->p_erst = udc_event->erst.cpu_addr;
+
+ buff_length = CRG_EVENT_RING_SIZE * sizeof(struct crg_event_trb);
+ if (!udc_event->event_ring.cpu_addr) {
+ udc_event->event_ring.cpu_addr = dma_alloc_coherent(
+ crg_udc->dev, buff_length, &mapping, GFP_ATOMIC);
+
+ if (!udc_event->event_ring.cpu_addr) {
+ ret = -ENOMEM;
+ goto err_free_erst;
+ }
+ } else {
+ mapping = udc_event->event_ring.dma_addr;
+ }
+
+ udc_event->event_ring.len = buff_length;
+ udc_event->event_ring.dma_addr = mapping;
+ /* Invalidate event handlers which dropped udc_lock during reset. */
+ udc_event->generation++;
+ udc_event->evt_dq_pt = udc_event->event_ring.cpu_addr;
+
+ udc_event->evt_seg0_last_trb =
+ (struct crg_event_trb *)(udc_event->event_ring.cpu_addr) +
+ (CRG_EVENT_RING_SIZE - 1);
+
+ udc_event->ccs = 1;
+
+ udc_event->p_erst->seg_addr_lo =
+ cpu_to_le32(lower_32_bits(udc_event->event_ring.dma_addr));
+ udc_event->p_erst->seg_addr_hi =
+ cpu_to_le32(upper_32_bits(udc_event->event_ring.dma_addr));
+ udc_event->p_erst->seg_size = cpu_to_le32(CRG_EVENT_RING_SIZE);
+ udc_event->p_erst->rsvd = cpu_to_le32(0);
+ /* Ensure that event ring is updated */
+ dma_wmb();
+ /* clear the event ring, to avoid hw unexpected ops
+ * because of dirty data
+ */
+ memset(udc_event->event_ring.cpu_addr, 0, buff_length);
+
+ /* Ensure that event ring is clear before programming
+ * ERSTBA/ERSTSZ/ERDP
+ */
+ dma_wmb();
+
+ writel(CRG_ERST_SIZE, &uicr->erstsz);
+ writel(lower_32_bits(udc_event->erst.dma_addr), &uicr->erstbalo);
+ writel(upper_32_bits(udc_event->erst.dma_addr), &uicr->erstbahi);
+ writel(lower_32_bits(udc_event->event_ring.dma_addr) |
+ CRG_U3DC_ERDPLO_EHB,
+ &uicr->erdplo);
+ writel(upper_32_bits(udc_event->event_ring.dma_addr), &uicr->erdphi);
+
+ writel((CRG_U3DC_IMAN_INT_EN | CRG_U3DC_IMAN_INT_PEND), &uicr->iman);
+ /* Interrupt moderation: 4000 * 250ns = 1ms */
+ writel(4000, &uicr->imod);
+ return 0;
+
+err_free_erst:
+ /* Only free what this call allocated; a pre-existing erst is owned
+ * by an earlier successful init and released by crg_udc_free_resources().
+ */
+ if (erst_allocated) {
+ dma_free_coherent(crg_udc->dev, udc_event->erst.len,
+ udc_event->erst.cpu_addr,
+ udc_event->erst.dma_addr);
+ udc_event->erst.cpu_addr = NULL;
+ }
+ return ret;
+}
+
+static bool is_event_ring_x_empty(struct crg_gadget_dev *crg_udc, int index)
+{
+ struct crg_event_trb *event;
+ struct crg_udc_event *udc_event;
+
+ udc_event = &crg_udc->udc_event[index];
+ if (udc_event->evt_dq_pt) {
+ event = (struct crg_event_trb *)udc_event->evt_dq_pt;
+
+ if (FIELD_GET(EVE_TRB_CYCLE_BIT_MASK,
+ le32_to_cpu(event->dw3)) != udc_event->ccs)
+ return true;
+ }
+
+ return false;
+}
+
+bool crg_udc_is_event_rings_empty(struct crg_gadget_dev *crg_udc)
+{
+ int i;
+
+ for (i = 0; i < CRG_RING_NUM; i++) {
+ if (!is_event_ring_x_empty(crg_udc, i)) {
+ dev_err(crg_udc->dev, "event ring not empty\n");
+ return false;
+ }
+ }
+ return true;
+}
+
+bool crg_udc_is_pointer_less_than(struct crg_transfer_trb *a,
+ struct crg_transfer_trb *b,
+ struct crg_udc_ep *udc_ep)
+{
+ if (!a || !b || !udc_ep->first_trb || !udc_ep->last_trb)
+ return false;
+ if (b > a && (udc_ep->enq_pt >= b || udc_ep->enq_pt < a))
+ return true;
+ if (b < a && (udc_ep->enq_pt >= b && udc_ep->enq_pt < a))
+ return true;
+ return false;
+}
+
+int crg_udc_process_event_ring(struct crg_gadget_dev *crg_udc, int index,
+ unsigned long *flags)
+{
+ struct crg_event_trb *event;
+ struct crg_udc_event *udc_event;
+ struct crg_uicr __iomem *uicr = crg_udc->uicr[index];
+ u32 val;
+ dma_addr_t erdp;
+ int ret;
+ u32 generation;
+ unsigned int processed = 0;
+
+ if (!uicr)
+ return -ENODEV;
+
+ val = readl(&uicr->iman);
+ val |= CRG_U3DC_IMAN_INT_PEND;
+ writel(val, &uicr->iman);
+
+ udc_event = &crg_udc->udc_event[index];
+ generation = udc_event->generation;
+ while (udc_event->evt_dq_pt) {
+ event = (struct crg_event_trb *)udc_event->evt_dq_pt;
+
+ if (FIELD_GET(EVE_TRB_CYCLE_BIT_MASK,
+ le32_to_cpu(event->dw3)) != udc_event->ccs)
+ break;
+ /* Read the payload only after observing device ownership. */
+ dma_rmb();
+
+ ret = crg_udc_handle_event(crg_udc, event, flags);
+ /* The giveback callback may release the lock and reset the ring. */
+ if (unlikely(generation != udc_event->generation))
+ return 0;
+ if (ret == -ECONNRESET) {
+ /* The disconnect path rebuilt and reprogrammed the ring. */
+ return 0;
+ }
+ if (ret < 0)
+ dev_dbg(crg_udc->dev, "failed to process event: %d\n",
+ ret);
+
+ /* Consume malformed events so they cannot pin the ring head. */
+ if (event == udc_event->evt_seg0_last_trb) {
+ udc_event->ccs = udc_event->ccs ? 0 : 1;
+ udc_event->evt_dq_pt = udc_event->event_ring.cpu_addr;
+ } else {
+ udc_event->evt_dq_pt++;
+ }
+
+ /*
+ * udc_event state is only re-read from crg_udc, never cached
+ * across this drop, so releasing and reacquiring the lock
+ * here is safe and lets other udc_lock users, and this CPU's
+ * other interrupts, make progress during a busy event ring.
+ */
+ if (++processed % CRG_EVENT_RING_BATCH == 0) {
+ erdp = event_trb_virt_to_dma(udc_event,
+ udc_event->evt_dq_pt);
+ if (!dma_mapping_error(crg_udc->dev, erdp)) {
+ writel(upper_32_bits(erdp), &uicr->erdphi);
+ writel(lower_32_bits(erdp) |
+ CRG_U3DC_ERDPLO_EHB,
+ &uicr->erdplo);
+ }
+ spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+ cond_resched();
+ spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+ if (unlikely(generation != udc_event->generation))
+ return 0;
+ }
+ }
+
+ /* update dequeue pointer */
+ erdp = event_trb_virt_to_dma(udc_event, udc_event->evt_dq_pt);
+ if (dma_mapping_error(crg_udc->dev, erdp))
+ return -EINVAL;
+ writel(upper_32_bits(erdp), &uicr->erdphi);
+ writel(lower_32_bits(erdp) | CRG_U3DC_ERDPLO_EHB, &uicr->erdplo);
+
+ return 0;
+}
+
+/* This function will go through the list of the USB requests for the
+ * given endpoint and schedule any unscheduled trb's to the xfer ring
+ */
+int crg_udc_queue_pending_trbs(struct crg_udc_ep *udc_ep_ptr)
+{
+ struct crg_udc_request *req;
+ struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+ u32 generation = crg_udc->udc_event[0].generation;
+ u32 ep_generation = udc_ep_ptr->generation;
+ int ret;
+
+restart:
+ /* A giveback can reset/disable the endpoint while udc_lock is dropped. */
+ if (crg_udc->resetting || !udc_ep_ptr->desc ||
+ udc_ep_ptr->ep_state == EP_STATE_DISABLED)
+ return 0;
+
+ list_for_each_entry(req, &udc_ep_ptr->queue, queue) {
+ if (req->all_trbs_queued)
+ continue;
+
+ /* A large request must finish its current TD before the next
+ * TD (or the next USB request) can be submitted: the event
+ * handler uses all_trbs_queued to complete the USB request.
+ */
+ if (req->td_pending)
+ return 0;
+
+ ret = crg_udc_build_td(udc_ep_ptr, req);
+ if (ret == -ENOSPC || ret == -EBUSY)
+ return ret;
+ if (ret) {
+ dev_err(crg_udc->dev,
+ "failed to build pending transfer: %d\n", ret);
+ /* Giveback removes req and drops udc_lock: callbacks can
+ * remove or free every saved request-list iterator.
+ */
+ crg_udc_req_done(udc_ep_ptr, req, ret);
+ if (generation != crg_udc->udc_event[0].generation ||
+ ep_generation != udc_ep_ptr->generation)
+ return 0;
+ goto restart;
+ }
+
+ if (udc_ep_ptr->tran_ring_full || !req->all_trbs_queued)
+ return 0;
+ }
+ return 0;
+}
+
+/* num_trbs here is the size of the ring. */
+u32 crg_udc_room_on_ring(struct crg_gadget_dev *crg_udc, u32 num_trbs,
+ struct crg_transfer_trb *p_ring,
+ struct crg_transfer_trb *enq_pt,
+ struct crg_transfer_trb *dq_pt)
+{
+ u32 i = 0;
+
+ if (enq_pt == dq_pt) {
+ /* ring is empty */
+ return num_trbs - 2;
+ }
+
+ while (enq_pt != dq_pt) {
+ i++;
+
+ enq_pt++;
+
+ if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(enq_pt->dw3)) ==
+ TRB_TYPE_LINK)
+ enq_pt = p_ring;
+
+ if (i > num_trbs)
+ break;
+ }
+
+ return i - 1;
+}
+
+void crg_udc_setup_datastage_trb(struct crg_transfer_trb *p_trb, dma_addr_t dma,
+ u8 pcs, u32 transfer_length, u32 td_size,
+ bool ioc, bool chain, bool azp, u8 dir, u8 setup_tag)
+{
+ u32 tmp;
+
+ p_trb->dw0 = cpu_to_le32(lower_32_bits(dma));
+ p_trb->dw1 = cpu_to_le32(upper_32_bits(dma));
+
+ /* TRB_Transfer_Length
+ * For USB_DIR_OUT, this field is the number of data bytes expected
+ * from xhc. For USB_DIR_IN, this field is the number of data bytes
+ * the device will send.
+ */
+ tmp = 0;
+ FIELD_MODIFY(TRB_TRANSFER_LEN_MASK, &tmp, transfer_length);
+ FIELD_MODIFY(TRB_TD_SIZE_MASK, &tmp, td_size);
+ FIELD_MODIFY(TRB_INTR_TARGET_MASK, &tmp, 0);
+ p_trb->dw2 = cpu_to_le32(tmp);
+
+ tmp = 0;
+ FIELD_MODIFY(TRB_CYCLE_BIT_MASK, &tmp, pcs);
+ FIELD_MODIFY(TRB_INTR_ON_SHORT_PKT_MASK, &tmp, 1);
+ FIELD_MODIFY(TRB_INTR_ON_COMPLETION_MASK, &tmp, ioc);
+ FIELD_MODIFY(TRB_CHAIN_BIT_MASK, &tmp, chain);
+ FIELD_MODIFY(TRB_TYPE_MASK, &tmp, TRB_TYPE_XFER_DATA_STAGE);
+ FIELD_MODIFY(TRB_APPEND_ZLP_MASK, &tmp, azp);
+ FIELD_MODIFY(DATA_STAGE_TRB_DIR_MASK, &tmp, dir);
+ FIELD_MODIFY(TRB_SETUP_TAG_MASK, &tmp, setup_tag);
+
+ /* Publish ownership only after the TRB payload is visible. */
+ dma_wmb();
+ p_trb->dw3 = cpu_to_le32(tmp);
+}
+
+void crg_udc_setup_link_trb(struct crg_transfer_trb *link_trb, bool toggle,
+ dma_addr_t next_trb)
+{
+ u32 dw = 0;
+
+ link_trb->dw0 = cpu_to_le32(lower_32_bits(next_trb));
+ link_trb->dw1 = cpu_to_le32(upper_32_bits(next_trb));
+ link_trb->dw2 = cpu_to_le32(0);
+
+ FIELD_MODIFY(TRB_TYPE_MASK, &dw, TRB_TYPE_LINK);
+ if (toggle)
+ FIELD_MODIFY(TRB_LINK_TOGGLE_CYCLE_MASK, &dw, 1);
+ else
+ FIELD_MODIFY(TRB_LINK_TOGGLE_CYCLE_MASK, &dw, 0);
+
+ /* Publish ownership only after the link address is visible. */
+ dma_wmb();
+ link_trb->dw3 = cpu_to_le32(dw);
+}
+
+void crg_udc_setup_status_trb(struct crg_gadget_dev *crg_udc,
+ struct crg_transfer_trb *p_trb,
+ struct usb_request *usb_req, u8 pcs, u8 set_addr,
+ u8 stall)
+{
+ u32 tmp, dir = 0;
+
+ /* There are some cases where crg_udc_setup_status_trb() is called with
+ * usb_req set to NULL.
+ */
+ p_trb->dw0 = cpu_to_le32(0);
+ p_trb->dw1 = cpu_to_le32(0);
+
+ tmp = 0;
+ FIELD_MODIFY(TRB_INTR_TARGET_MASK, &tmp, 0);
+ p_trb->dw2 = cpu_to_le32(tmp);
+
+ tmp = 0;
+ FIELD_MODIFY(TRB_CYCLE_BIT_MASK, &tmp, pcs);
+ FIELD_MODIFY(TRB_INTR_ON_COMPLETION_MASK, &tmp, 1);
+ FIELD_MODIFY(TRB_TYPE_MASK, &tmp, TRB_TYPE_XFER_STATUS_STAGE);
+
+ dir = (crg_udc->setup_status == STATUS_STAGE_XFER) ? 0 : 1;
+ FIELD_MODIFY(DATA_STAGE_TRB_DIR_MASK, &tmp, dir);
+
+ FIELD_MODIFY(TRB_SETUP_TAG_MASK, &tmp, crg_udc->setup_tag);
+ FIELD_MODIFY(STATUS_STAGE_TRB_STALL_MASK, &tmp, stall);
+ FIELD_MODIFY(STATUS_STAGE_TRB_SET_ADDR_MASK, &tmp, set_addr);
+
+ /* Publish ownership only after the TRB payload is visible. */
+ dma_wmb();
+ p_trb->dw3 = cpu_to_le32(tmp);
+}
+
+void crg_udc_setup_trb(struct crg_transfer_trb *p_trb, u32 xfer_len,
+ dma_addr_t xfer_buf_addr, u8 td_size, u8 pcs,
+ u8 trb_type, bool short_pkt, bool chain_bit,
+ bool intr_on_compl, bool b_setup_stage, u8 usb_dir,
+ bool b_isoc, u16 frame_id, bool sia, bool azp)
+{
+ u32 tmp;
+
+ p_trb->dw0 = cpu_to_le32(lower_32_bits(xfer_buf_addr));
+ p_trb->dw1 = cpu_to_le32(upper_32_bits(xfer_buf_addr));
+
+ tmp = 0;
+ FIELD_MODIFY(TRB_TRANSFER_LEN_MASK, &tmp, xfer_len);
+ FIELD_MODIFY(TRB_TD_SIZE_MASK, &tmp, td_size);
+ FIELD_MODIFY(TRB_INTR_TARGET_MASK, &tmp, 0);
+
+ p_trb->dw2 = cpu_to_le32(tmp);
+
+ tmp = 0;
+ FIELD_MODIFY(TRB_CYCLE_BIT_MASK, &tmp, pcs);
+ FIELD_MODIFY(TRB_INTR_ON_SHORT_PKT_MASK, &tmp, short_pkt);
+ FIELD_MODIFY(TRB_CHAIN_BIT_MASK, &tmp, chain_bit);
+ FIELD_MODIFY(TRB_INTR_ON_COMPLETION_MASK, &tmp, intr_on_compl);
+ FIELD_MODIFY(TRB_APPEND_ZLP_MASK, &tmp, azp);
+ FIELD_MODIFY(TRB_TYPE_MASK, &tmp, trb_type);
+
+ if (b_setup_stage)
+ FIELD_MODIFY(DATA_STAGE_TRB_DIR_MASK, &tmp, usb_dir);
+
+ if (b_isoc) {
+ FIELD_MODIFY(ISOC_TRB_FRAME_ID_MASK, &tmp, frame_id);
+ FIELD_MODIFY(ISOC_TRB_SIA_MASK, &tmp, sia);
+ }
+
+ /* Publish ownership only after the TRB payload is visible. */
+ dma_wmb();
+ p_trb->dw3 = cpu_to_le32(tmp);
+}
+
+void crg_udc_squeeze_xfer_ring(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr)
+{
+ struct crg_transfer_trb *trb = udc_req_ptr->first_trb;
+ struct crg_udc_request *next_req;
+ u8 new_pcs, clear_pcs;
+
+ if (!trb)
+ return;
+
+ /* Rewind the producer to the cancelled request. These TRBs and
+ * everything queued after them will be discarded and rebuilt.
+ */
+ new_pcs = FIELD_GET(TRB_CYCLE_BIT_MASK, le32_to_cpu(trb->dw3));
+ clear_pcs = new_pcs;
+ while (trb != udc_ep_ptr->enq_pt) {
+ /* Zero is not always invalid: after a wrap the consumer
+ * expects cycle 0. Invalidate each slot for its cycle pass.
+ */
+ le32p_replace_bits(&trb->dw3, clear_pcs ^ 1,
+ TRB_CYCLE_BIT_MASK);
+ trb++;
+ if (trb == udc_ep_ptr->last_trb) {
+ trb = udc_ep_ptr->first_trb;
+ clear_pcs ^= 1;
+ }
+ }
+
+ udc_ep_ptr->enq_pt = udc_req_ptr->first_trb;
+ udc_ep_ptr->pcs = new_pcs;
+ udc_ep_ptr->tran_ring_full = false;
+ /* Do not overwrite the Link TRB: a previous active TD may still
+ * require its cycle/chain bits to traverse the ring.
+ */
+ dma_wmb();
+
+ /* Invalidate the bookkeeping of ALL subsequent requests before
+ * attempting to rebuild any: rebuilding may stop when space runs out.
+ * The caller gives back the cancelled request before rebuilding.
+ */
+ next_req = udc_req_ptr;
+ list_for_each_entry_continue(next_req, &udc_ep_ptr->queue, queue) {
+ next_req->usb_req.status = -EINPROGRESS;
+ next_req->usb_req.actual = 0;
+ crg_udc_clear_req_container(next_req);
+ }
+}
+
+struct crg_transfer_trb *crg_udc_tran_trb_dma_to_virt(struct crg_udc_ep *udc_ep,
+ dma_addr_t address)
+{
+ dma_addr_t offset;
+ struct crg_transfer_trb *trb_virt;
+
+ if (lower_32_bits(address) & 0xf)
+ return NULL;
+
+ offset = address - udc_ep->tran_ring_info.dma_addr;
+ if (unlikely(offset >= udc_ep->tran_ring_info.len))
+ return NULL;
+ offset = offset / sizeof(struct crg_transfer_trb);
+ trb_virt = udc_ep->first_trb + offset;
+ return trb_virt;
+}
+
+dma_addr_t crg_udc_tran_trb_virt_to_dma(struct crg_udc_ep *udc_ep,
+ struct crg_transfer_trb *trb)
+{
+ unsigned long offset;
+ int trb_idx;
+ dma_addr_t dma_addr = 0;
+
+ trb_idx = trb - udc_ep->first_trb;
+ if (unlikely(trb_idx < 0))
+ return DMA_MAPPING_ERROR;
+
+ offset = trb_idx * sizeof(*trb);
+ if (unlikely(offset >= udc_ep->tran_ring_info.len))
+ return DMA_MAPPING_ERROR;
+ dma_addr = udc_ep->tran_ring_info.dma_addr + offset;
+ return dma_addr;
+}
+
+void crg_udc_update_dequeue_pt(struct crg_event_trb *event,
+ struct crg_udc_ep *udc_ep)
+{
+ u32 deq_pt_lo = le32_to_cpu(event->dw0);
+ u32 deq_pt_hi = le32_to_cpu(event->dw1);
+ u64 dq_pt_addr = (u64)deq_pt_lo | ((u64)deq_pt_hi << 32);
+ struct crg_transfer_trb *deq_pt;
+
+ deq_pt = crg_udc_tran_trb_dma_to_virt(udc_ep, dq_pt_addr);
+ if (!deq_pt)
+ return;
+ deq_pt++;
+ if (deq_pt == udc_ep->last_trb)
+ deq_pt = udc_ep->first_trb;
+
+ if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(deq_pt->dw3)) == TRB_TYPE_LINK)
+ deq_pt = udc_ep->first_trb;
+
+ udc_ep->deq_pt = deq_pt;
+}
+
+u32 crg_udc_trb_chunk_len(dma_addr_t dma, u64 remaining)
+{
+ u32 boundary = CRG_UDC_TRB_MAX_BUFFER_SIZE -
+ (lower_32_bits(dma) & (CRG_UDC_TRB_MAX_BUFFER_SIZE - 1));
+
+ return min_t(u64, remaining, boundary);
+}
+
+u32 crg_udc_actual_before(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req,
+ struct crg_transfer_trb *pause)
+{
+ struct crg_transfer_trb *trb = udc_req->first_trb;
+ u32 actual = 0;
+ u32 guard = 0;
+ u32 max_iter = udc_ep->tran_ring_info.len / sizeof(*trb);
+
+ if (!trb || !pause)
+ return 0;
+
+ while (trb != pause && trb != udc_ep->enq_pt && guard++ < max_iter) {
+ u32 len;
+
+ if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(trb->dw3)) ==
+ TRB_TYPE_LINK) {
+ trb = udc_ep->first_trb;
+ continue;
+ }
+ len = FIELD_GET(TRB_TRANSFER_LEN_MASK, le32_to_cpu(trb->dw2));
+ actual += len;
+ trb++;
+ if (trb == udc_ep->last_trb)
+ trb = udc_ep->first_trb;
+ }
+
+ return min(actual, udc_req->usb_req.length);
+}
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [PATCH v4 3/6] usb: gadget: udc: crg: Add ring and transfer management
2026-10-30 8:10 ` [PATCH v4 3/6] usb: gadget: udc: crg: Add ring and transfer management Petar Stepanovic
@ 2026-10-30 8:29 ` sashiko-bot
0 siblings, 0 replies; 14+ messages in thread
From: sashiko-bot @ 2026-10-30 8:29 UTC (permalink / raw)
To: Petar Stepanovic; +Cc: linux-usb, devicetree, robh, conor+dt
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] usb: gadget: crg: out-of-bounds TRB read in crg_udc_update_dequeue_pt()
- [High] usb: gadget: crg: premature wrap skips pause in crg_udc_actual_before()
- [High] usb: gadget: crg: crg_udc_process_event_ring() misses in-progress reset after lock drop
--
commit 050a1d6b53db454c24d8712a72246d62e4e494d9
Author: Petar Stepanovic <pstepanovic@axiado.com>
usb: gadget: udc: crg: Add ring and transfer management
This commit introduces memory management and ring buffer logic for TRBs and
event rings required by the Axiado AX3000/AX3005 UDC driver.
> diff --git a/drivers/usb/gadget/udc/crg/crg-udc-ring.c b/drivers/usb/gadget/udc/crg/crg-udc-ring.c
> new file mode 100644
> index 0000000000000..f1de8893b790f
> --- /dev/null
> +++ b/drivers/usb/gadget/udc/crg/crg-udc-ring.c
[ ... ]
> +int crg_udc_process_event_ring(struct crg_gadget_dev *crg_udc, int index,
> + unsigned long *flags)
> +{
> + struct crg_event_trb *event;
> + struct crg_udc_event *udc_event;
> + struct crg_uicr __iomem *uicr = crg_udc->uicr[index];
> + u32 val;
> + dma_addr_t erdp;
> + int ret;
> + u32 generation;
> + unsigned int processed = 0;
> +
> + if (!uicr)
> + return -ENODEV;
> +
> + val = readl(&uicr->iman);
> + val |= CRG_U3DC_IMAN_INT_PEND;
> + writel(val, &uicr->iman);
> +
> + udc_event = &crg_udc->udc_event[index];
> + generation = udc_event->generation;
> + while (udc_event->evt_dq_pt) {
> + event = (struct crg_event_trb *)udc_event->evt_dq_pt;
> +
> + if (FIELD_GET(EVE_TRB_CYCLE_BIT_MASK,
> + le32_to_cpu(event->dw3)) != udc_event->ccs)
> + break;
> + /* Read the payload only after observing device ownership. */
> + dma_rmb();
> +
> + ret = crg_udc_handle_event(crg_udc, event, flags);
> + /* The giveback callback may release the lock and reset the ring. */
> + if (unlikely(generation != udc_event->generation))
> + return 0;
> + if (ret == -ECONNRESET) {
> + /* The disconnect path rebuilt and reprogrammed the ring. */
> + return 0;
> + }
> + if (ret < 0)
> + dev_dbg(crg_udc->dev, "failed to process event: %d\n",
> + ret);
> +
> + /* Consume malformed events so they cannot pin the ring head. */
> + if (event == udc_event->evt_seg0_last_trb) {
> + udc_event->ccs = udc_event->ccs ? 0 : 1;
> + udc_event->evt_dq_pt = udc_event->event_ring.cpu_addr;
> + } else {
> + udc_event->evt_dq_pt++;
> + }
> +
> + /*
> + * udc_event state is only re-read from crg_udc, never cached
> + * across this drop, so releasing and reacquiring the lock
> + * here is safe and lets other udc_lock users, and this CPU's
> + * other interrupts, make progress during a busy event ring.
> + */
> + if (++processed % CRG_EVENT_RING_BATCH == 0) {
> + erdp = event_trb_virt_to_dma(udc_event,
> + udc_event->evt_dq_pt);
> + if (!dma_mapping_error(crg_udc->dev, erdp)) {
> + writel(upper_32_bits(erdp), &uicr->erdphi);
> + writel(lower_32_bits(erdp) |
> + CRG_U3DC_ERDPLO_EHB,
> + &uicr->erdplo);
> + }
> + spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
> + cond_resched();
> + spin_lock_irqsave(&crg_udc->udc_lock, *flags);
> + if (unlikely(generation != udc_event->generation))
> + return 0;
> + }
> + }
[Severity: High]
As reported by Sashiko on v3 [1], can crg_udc_process_event_ring() miss an
in-progress reset when crg_udc->udc_lock is dropped inside
crg_udc_handle_event() or across the cond_resched() batch boundary?
When crg_gadget_stop() -> crg_udc_reset() runs on another CPU while
crg_udc->udc_lock is released, it sets crg_udc->resetting = true, asserts
CRG_U3DC_CTRL_SWRST, and drops crg_udc->udc_lock while polling for SWRST to
clear and again during crg_udc_ep_nuke() -> crg_udc_req_done() givebacks.
Because udc_event->generation is only incremented later when
crg_udc_reset_data_struct() -> crg_udc_init_event_ring() runs (and is not
incremented at all if crg_udc_reset() times out),
crg_udc_process_event_ring() can re-acquire crg_udc->udc_lock during those
windows with generation == udc_event->generation still true while
crg_udc->resetting is set.
Would this allow crg_udc_process_event_ring() to continue advancing
udc_event->evt_dq_pt, processing stale pre-reset events, and writing
uicr->erdphi and uicr->erdplo while the controller is mid-reset? Should the
post-lock checks in crg_udc_process_event_ring() also test
crg_udc->resetting, similar to crg_udc_queue_pending_trbs() and
crg_udc_handle_setup_pkt()?
[ ... ]
> +struct crg_transfer_trb *crg_udc_tran_trb_dma_to_virt(struct crg_udc_ep *udc_ep,
> + dma_addr_t address)
> +{
> + dma_addr_t offset;
> + struct crg_transfer_trb *trb_virt;
> +
> + if (lower_32_bits(address) & 0xf)
> + return NULL;
> +
> + offset = address - udc_ep->tran_ring_info.dma_addr;
> + if (unlikely(offset >= udc_ep->tran_ring_info.len))
> + return NULL;
> + offset = offset / sizeof(struct crg_transfer_trb);
> + trb_virt = udc_ep->first_trb + offset;
> + return trb_virt;
> +}
> +
> +dma_addr_t crg_udc_tran_trb_virt_to_dma(struct crg_udc_ep *udc_ep,
> + struct crg_transfer_trb *trb)
> +{
> + unsigned long offset;
> + int trb_idx;
> + dma_addr_t dma_addr = 0;
> +
> + trb_idx = trb - udc_ep->first_trb;
> + if (unlikely(trb_idx < 0))
> + return DMA_MAPPING_ERROR;
> +
> + offset = trb_idx * sizeof(*trb);
> + if (unlikely(offset >= udc_ep->tran_ring_info.len))
> + return DMA_MAPPING_ERROR;
> + dma_addr = udc_ep->tran_ring_info.dma_addr + offset;
> + return dma_addr;
> +}
> +
> +void crg_udc_update_dequeue_pt(struct crg_event_trb *event,
> + struct crg_udc_ep *udc_ep)
> +{
> + u32 deq_pt_lo = le32_to_cpu(event->dw0);
> + u32 deq_pt_hi = le32_to_cpu(event->dw1);
> + u64 dq_pt_addr = (u64)deq_pt_lo | ((u64)deq_pt_hi << 32);
> + struct crg_transfer_trb *deq_pt;
> +
> + deq_pt = crg_udc_tran_trb_dma_to_virt(udc_ep, dq_pt_addr);
> + if (!deq_pt)
> + return;
> + deq_pt++;
> + if (deq_pt == udc_ep->last_trb)
> + deq_pt = udc_ep->first_trb;
> +
> + if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(deq_pt->dw3)) == TRB_TYPE_LINK)
> + deq_pt = udc_ep->first_trb;
> +
> + udc_ep->deq_pt = deq_pt;
> +}
[Severity: High]
What happens in crg_udc_update_dequeue_pt() if the event TRB's DMA address
points to the Link TRB at udc_ep->last_trb (for example, on a
CMPL_CODE_TRB_ERR event)?
Because udc_ep->last_trb sits at &udc_ep->first_trb[ring_size - 1],
crg_udc_tran_trb_dma_to_virt() accepts its DMA address and returns
deq_pt == udc_ep->last_trb.
Advancing deq_pt++ before checking against udc_ep->last_trb moves deq_pt to
udc_ep->last_trb + 1, which is one element past the end of the
udc_ep->tran_ring_info coherent DMA buffer.
Doesn't that cause the subsequent read of deq_pt->dw3 to access memory out of
bounds and potentially store udc_ep->last_trb + 1 into udc_ep->deq_pt, which
in turn causes crg_udc_room_on_ring() to hit its loop guard and treat the
entire ring as free?
> +
> +u32 crg_udc_trb_chunk_len(dma_addr_t dma, u64 remaining)
> +{
> + u32 boundary = CRG_UDC_TRB_MAX_BUFFER_SIZE -
> + (lower_32_bits(dma) & (CRG_UDC_TRB_MAX_BUFFER_SIZE - 1));
> +
> + return min_t(u64, remaining, boundary);
> +}
> +
> +u32 crg_udc_actual_before(struct crg_udc_ep *udc_ep,
> + struct crg_udc_request *udc_req,
> + struct crg_transfer_trb *pause)
> +{
> + struct crg_transfer_trb *trb = udc_req->first_trb;
> + u32 actual = 0;
> + u32 guard = 0;
> + u32 max_iter = udc_ep->tran_ring_info.len / sizeof(*trb);
> +
> + if (!trb || !pause)
> + return 0;
> +
> + while (trb != pause && trb != udc_ep->enq_pt && guard++ < max_iter) {
> + u32 len;
> +
> + if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(trb->dw3)) ==
> + TRB_TYPE_LINK) {
> + trb = udc_ep->first_trb;
> + continue;
> + }
> + len = FIELD_GET(TRB_TRANSFER_LEN_MASK, le32_to_cpu(trb->dw2));
> + actual += len;
> + trb++;
> + if (trb == udc_ep->last_trb)
> + trb = udc_ep->first_trb;
> + }
> +
> + return min(actual, udc_req->usb_req.length);
> +}
[Severity: High]
In crg_udc_actual_before(), if crg_udc_ep_dequeue() stops an active endpoint
while a multi-TRB request wraps around the transfer ring and the hardware
dequeue pointer (pause) is at the Link TRB (udc_ep->last_trb), will this loop
skip right past pause?
At the end of the iteration for udc_ep->last_trb - 1, trb++ reaches
udc_ep->last_trb and is immediately wrapped to udc_ep->first_trb before the
while (trb != pause && ...) condition is evaluated:
trb++;
if (trb == udc_ep->last_trb)
trb = udc_ep->first_trb;
Because trb jumps from udc_ep->last_trb - 1 directly to udc_ep->first_trb, it
never compares equal to pause == udc_ep->last_trb (and the TRB_TYPE_LINK
check at the top of the loop is also bypassed), causing
crg_udc_actual_before() to sum all remaining queued TRBs up to
udc_ep->enq_pt into udc_req->usb_req.actual.
[1] https://lore.kernel.org/r/20261020-axiado-ax3000-usb-device-controller-v3-3-3012ecfc0a93@axiado.com
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261030-axiado-ax3000-usb-device-controller-v4-0-c5267b36a777@axiado.com?part=3
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v4 4/6] usb: gadget: udc: crg: Add endpoint 0 handling
2026-10-30 8:10 [PATCH v4 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
` (2 preceding siblings ...)
2026-10-30 8:10 ` [PATCH v4 3/6] usb: gadget: udc: crg: Add ring and transfer management Petar Stepanovic
@ 2026-10-30 8:10 ` Petar Stepanovic
2026-10-30 8:40 ` sashiko-bot
2026-10-30 8:10 ` [PATCH v4 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations Petar Stepanovic
2026-10-30 8:10 ` [PATCH v4 6/6] usb: gadget: udc: crg: Add core driver and build integration Petar Stepanovic
5 siblings, 1 reply; 14+ messages in thread
From: Petar Stepanovic @ 2026-10-30 8:10 UTC (permalink / raw)
To: Naveen Kumar Rajgiri Bassappa, Prasad Bolisetty,
Greg Kroah-Hartman, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Tzu-Hao Wei, Karthikeyan Mitran, Philipp Zabel
Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
Petar Stepanovic, Krutik Shah
This patch adds the endpoint 0 setup packet, status, and control
transfer handling for the Axiado AX3000/AX3005 UDC.
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
drivers/usb/gadget/udc/crg/crg-udc-ep0.c | 1133 ++++++++++++++++++++++++++++++
1 file changed, 1133 insertions(+)
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-ep0.c b/drivers/usb/gadget/udc/crg/crg-udc-ep0.c
new file mode 100644
index 000000000000..073d9090dd07
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-ep0.c
@@ -0,0 +1,1133 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/iopoll.h>
+
+#include "crg-udc.h"
+#include "crg-udc-trace.h"
+
+static void enable_setup_event(struct crg_gadget_dev *crg_udc);
+
+void crg_udc_build_ep0_status(struct crg_udc_ep *udc_ep_ptr, bool default_value,
+ u32 status, struct crg_udc_request *udc_req_ptr,
+ u8 set_addr, u8 stall)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+ struct crg_transfer_trb *enq_pt = udc_ep_ptr->enq_pt;
+
+ if (default_value) {
+ udc_req_ptr = crg_udc->status_req;
+ /* The shared request may still be in its data-stage callback. */
+ if (!udc_req_ptr || crg_udc->status_req_callback_pending ||
+ !list_empty(&udc_req_ptr->queue)) {
+ dev_err(crg_udc->dev, "EP0 status request unavailable\n");
+ crg_udc->ep0_state = EP0_STATE_ERROR;
+ return;
+ }
+ udc_req_ptr->usb_req.length = 0;
+ udc_req_ptr->usb_req.status = status;
+ udc_req_ptr->usb_req.actual = 0;
+ udc_req_ptr->usb_req.complete = NULL;
+ } else {
+ udc_req_ptr->usb_req.status = status;
+ udc_req_ptr->usb_req.actual = 0;
+ }
+
+ crg_udc_setup_status_trb(crg_udc, enq_pt, &udc_req_ptr->usb_req,
+ udc_ep_ptr->pcs, set_addr, stall);
+
+ enq_pt++;
+
+ /* check if we are at end of trb segment. If so, update
+ * pcs and enq for next segment
+ */
+ if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(enq_pt->dw3)) ==
+ TRB_TYPE_LINK) {
+ if (FIELD_GET(TRB_LINK_TOGGLE_CYCLE_MASK,
+ le32_to_cpu(enq_pt->dw3))) {
+ dma_wmb();
+ le32p_replace_bits(&enq_pt->dw3, udc_ep_ptr->pcs,
+ TRB_CYCLE_BIT_MASK);
+ udc_ep_ptr->pcs ^= 0x1;
+ }
+ enq_pt = udc_ep_ptr->first_trb;
+ }
+ udc_ep_ptr->enq_pt = enq_pt;
+
+ crg_udc_knock_doorbell(crg_udc, 0);
+
+ list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+}
+
+/*
+ * Abort the current EP0 control transfer when a new SETUP packet arrives.
+ *
+ * A SETUP packet is allowed to abort any in-progress control transfer. Do
+ * not defer the new SETUP until the old transfer completes: doing so leaves
+ * the old request on the EP0 software queue and subsequent gadget requests
+ * are rejected as "queue not empty".
+ *
+ * The controller's EP0 ring is reinitialized at the ring start, making all
+ * TRBs belonging to the aborted control transfer stale. The new SETUP is
+ * then handled immediately with a fresh setup tag.
+ */
+int crg_udc_abort_ep0_for_setup(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_udc_ep *udc_ep0 = &crg_udc->udc_ep[0];
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ dma_addr_t dqptaddr;
+ u32 cmd_param0;
+ u32 cmd_param1;
+ u32 tmp;
+ int ret;
+
+ /*
+ * A new SETUP is allowed to abort the current control transfer.
+ * Keep EP0 in ABORTING for the complete stop/nuke/reinitialize
+ * sequence. crg_udc_req_done() drops udc_lock while invoking the gadget
+ * callback, so the callback may re-enter the UDC during the nuke.
+ */
+ crg_udc->setup_status = WAIT_FOR_SETUP;
+ crg_udc->ep0_state = EP0_STATE_ABORTING;
+
+ /*
+ * Stop EP0 before touching the transfer ring. The controller may still
+ * own the old TRBs even when no EP0 request is running from the gadget's
+ * point of view. Do not clear the DMA-visible ring until the hardware
+ * reports that EP0 is no longer running.
+ */
+ ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP, BIT(0), 0);
+ if (ret) {
+ dev_err(crg_udc->dev,
+ "failed to stop EP0 for SETUP abort: %d\n", ret);
+ crg_udc->ep0_state = EP0_STATE_ERROR;
+ return ret;
+ }
+
+ ret = readl_poll_timeout_atomic(&uccr->ep_running, tmp, !(tmp & BIT(0)),
+ 10, 10000);
+ if (ret) {
+ dev_err(crg_udc->dev, "timeout stopping EP0 for SETUP abort\n");
+ crg_udc->ep0_state = EP0_STATE_ERROR;
+ return ret;
+ }
+
+ udc_ep0->ep_state = EP_STATE_STOPPED;
+ /* Complete requests belonging to the aborted control transfer. */
+ crg_udc_ep_nuke(udc_ep0, -ECONNRESET);
+
+ if (!udc_ep0->first_trb || !udc_ep0->tran_ring_info.cpu_addr) {
+ crg_udc->ep0_state = EP0_STATE_ERROR;
+ return -ENODEV;
+ }
+
+ /*
+ * The controller has stopped consuming EP0 TRBs. It is now safe to
+ * discard the old transfer descriptors and restart the ring from its
+ * beginning. No ring memory is freed here.
+ */
+ memset(udc_ep0->first_trb, 0, udc_ep0->tran_ring_info.len);
+ udc_ep0->enq_pt = udc_ep0->first_trb;
+ udc_ep0->deq_pt = udc_ep0->first_trb;
+ udc_ep0->pcs = 1;
+ udc_ep0->tran_ring_full = false;
+ crg_udc_setup_link_trb(udc_ep0->last_trb, true,
+ udc_ep0->tran_ring_info.dma_addr);
+
+ /* Make the new ring contents visible before handing the ring to HW. */
+ dma_wmb();
+
+ dqptaddr = crg_udc_tran_trb_virt_to_dma(udc_ep0, udc_ep0->first_trb);
+ if (dma_mapping_error(crg_udc->dev, dqptaddr)) {
+ crg_udc->ep0_state = EP0_STATE_ERROR;
+ return -EIO;
+ }
+
+ cmd_param0 = (lower_32_bits(dqptaddr) & CRG_CMD0_0_DQPTRLO_MASK) |
+ FIELD_PREP(CRG_CMD0_0_DCS_MASK, udc_ep0->pcs);
+ cmd_param1 = upper_32_bits(dqptaddr);
+
+ ret = crg_udc_issue_command(crg_udc, CRG_CMD_INIT_EP0, cmd_param0,
+ cmd_param1);
+ if (ret) {
+ dev_err(crg_udc->dev,
+ "failed to reinitialize EP0 for SETUP: %d\n", ret);
+ crg_udc->ep0_state = EP0_STATE_ERROR;
+ return ret;
+ }
+
+ udc_ep0->ep_state = EP_STATE_RUNNING;
+ crg_udc->ep0_state = EP0_STATE_IDLE;
+ return 0;
+}
+
+int crg_udc_enable_setup(struct crg_gadget_dev *crg_udc)
+{
+ enable_setup_event(crg_udc);
+ crg_udc->device_state = USB_STATE_DEFAULT;
+ usb_gadget_set_state(&crg_udc->gadget, USB_STATE_DEFAULT);
+ crg_udc->setup_status = WAIT_FOR_SETUP;
+
+ return 0;
+}
+
+static void enable_setup_event(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 val;
+
+ val = readl(&uccr->config1);
+ val |= CRG_U3DC_CFG1_SETUP_EVENT_EN;
+ writel(val, &uccr->config1);
+}
+
+void crg_udc_ep0_req_complete(struct crg_udc_ep *udc_ep_ptr,
+ unsigned long *flags)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+
+ switch (crg_udc->setup_status) {
+ case DATA_STAGE_XFER:
+ crg_udc->setup_status = STATUS_STAGE_RECV;
+ crg_udc_build_ep0_status(udc_ep_ptr, true, -EINPROGRESS, NULL,
+ 0, 0);
+ break;
+ case DATA_STAGE_RECV:
+ crg_udc->setup_status = STATUS_STAGE_XFER;
+ crg_udc_build_ep0_status(udc_ep_ptr, true, -EINPROGRESS, NULL,
+ 0, 0);
+ break;
+ default:
+ if (crg_udc->setup_fn_call_back) {
+ crg_udc->setup_fn_call_back(crg_udc);
+ crg_udc->setup_fn_call_back = NULL;
+ }
+ crg_udc->setup_status = WAIT_FOR_SETUP;
+ break;
+ }
+}
+
+static void get_status_cmpl(struct usb_ep *ep, struct usb_request *req)
+{
+ struct crg_udc_ep *udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+ struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+ unsigned long flags;
+ void *buf;
+
+ /* The data-stage completion handler queues the status stage. */
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ buf = req->buf;
+ req->buf = NULL;
+ crg_udc->status_req_callback_pending = false;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ kfree(buf);
+}
+
+static void getstatusrequest(struct crg_gadget_dev *crg_udc, u8 request_type,
+ u16 value, u16 index, u16 length)
+{
+ struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+ struct crg_udc_request *udc_req_ptr = crg_udc->status_req;
+ u16 status_val = 0;
+ u32 dci;
+ int ret;
+
+ if (!udc_req_ptr || crg_udc->status_req_callback_pending ||
+ !list_empty(&udc_req_ptr->queue) || value ||
+ length != sizeof(u16)) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ switch (request_type & USB_RECIP_MASK) {
+ case USB_RECIP_DEVICE:
+ if (index)
+ goto invalid;
+ if (crg_udc->gadget.speed == USB_SPEED_HIGH ||
+ crg_udc->gadget.speed == USB_SPEED_FULL) {
+ if (crg_udc->u2_rwe)
+ status_val |= BIT(USB_DEVICE_REMOTE_WAKEUP);
+ } else if (crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+ if (crg_udc->feature_u1_enable)
+ status_val |= BIT(USB_DEV_STAT_U1_ENABLED);
+ if (crg_udc->feature_u2_enable)
+ status_val |= BIT(USB_DEV_STAT_U2_ENABLED);
+ }
+ if (crg_udc->gadget.is_selfpowered)
+ status_val |= BIT(USB_DEVICE_SELF_POWERED);
+ break;
+
+ case USB_RECIP_INTERFACE:
+ if (crg_udc->device_state != USB_STATE_CONFIGURED ||
+ (index & 0xff00))
+ goto invalid;
+ break;
+
+ case USB_RECIP_ENDPOINT:
+ if (index & ~(USB_ENDPOINT_NUMBER_MASK | USB_DIR_IN))
+ goto invalid;
+ dci = crg_udc_index2dci(index);
+ if (dci == 1 || dci >= CRG_NUM_EP_CX)
+ goto invalid;
+ if (crg_udc->device_state < USB_STATE_ADDRESS && dci != 0)
+ goto invalid;
+ if (crg_udc->device_state == USB_STATE_ADDRESS && dci != 0)
+ goto invalid;
+ if (crg_udc->device_state == USB_STATE_CONFIGURED &&
+ crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_DISABLED)
+ goto invalid;
+ if (crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_HALTED)
+ status_val = BIT(USB_ENDPOINT_HALT);
+ break;
+
+ default:
+ goto invalid;
+ }
+
+ kfree(udc_req_ptr->usb_req.buf);
+
+ udc_req_ptr->usb_req.buf = kzalloc_obj(status_val, GFP_ATOMIC);
+ if (!udc_req_ptr->usb_req.buf)
+ goto invalid;
+
+ *(__le16 *)udc_req_ptr->usb_req.buf = cpu_to_le16(status_val);
+ udc_req_ptr->usb_req.length = sizeof(status_val);
+ udc_req_ptr->usb_req.status = -EINPROGRESS;
+ udc_req_ptr->usb_req.actual = 0;
+ udc_req_ptr->usb_req.complete = get_status_cmpl;
+ crg_udc->status_req_callback_pending = true;
+
+ udc_req_ptr->map_dir = DMA_TO_DEVICE;
+ ret = usb_gadget_map_request_by_dev(crg_udc->dev, &udc_req_ptr->usb_req,
+ true);
+ if (ret) {
+ kfree(udc_req_ptr->usb_req.buf);
+ udc_req_ptr->usb_req.buf = NULL;
+ crg_udc->status_req_callback_pending = false;
+ goto invalid;
+ }
+
+ crg_udc->setup_status = DATA_STAGE_XFER;
+
+ ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+ if (ret) {
+ crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr);
+ kfree(udc_req_ptr->usb_req.buf);
+ udc_req_ptr->usb_req.buf = NULL;
+ crg_udc->status_req_callback_pending = false;
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+ list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+ return;
+
+invalid:
+ crg_udc_set_ep0_halt(crg_udc);
+}
+
+int crg_udc_prepare_for_setup(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_udc_ep *udc_ep0_ptr;
+ dma_addr_t dqptaddr;
+ u32 cmd_param0;
+ u32 cmd_param1;
+ int ret;
+
+ if (!crg_udc_is_event_rings_empty(crg_udc) ||
+ crg_udc->portsc_on_reconnecting == 1)
+ return -EBUSY;
+
+ udc_ep0_ptr = &crg_udc->udc_ep[0];
+ /* If we reinit ep0 on bus reset, we just make ep0 dequeue pointer
+ * align with enqueue pointer, all remaining xfer trbs became dumb once
+ * which will not produce xfer event anymore.
+ *
+ * If we considering the opposite solution, we should wait all ep0 xfer
+ * trbs be completed(with some err complete code)
+ */
+ if (crg_udc->reinit_ep0_on_bus_reset) {
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 running;
+
+ /* Block requeue from giveback callbacks, then quiesce DMA. */
+ crg_udc->ep0_state = EP0_STATE_ABORTING;
+ if (crg_udc_get_ep_state(crg_udc, 0) == EP_STATE_RUNNING) {
+ ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP,
+ BIT(0), 0);
+ if (ret)
+ goto reset_error;
+
+ ret = readl_poll_timeout_atomic(&uccr->ep_running, running,
+ !(running & BIT(0)), 10, 10000);
+ if (ret)
+ goto reset_error;
+ }
+ udc_ep0_ptr->ep_state = EP_STATE_STOPPED;
+ crg_udc_ep_nuke(udc_ep0_ptr, -ESHUTDOWN);
+
+ /* context related ops */
+ dqptaddr = crg_udc_tran_trb_virt_to_dma(udc_ep0_ptr,
+ udc_ep0_ptr->enq_pt);
+ if (dma_mapping_error(crg_udc->dev, dqptaddr)) {
+ ret = -EIO;
+ goto reset_error;
+ }
+ cmd_param0 =
+ (lower_32_bits(dqptaddr) & CRG_CMD0_0_DQPTRLO_MASK) |
+ FIELD_PREP(CRG_CMD0_0_DCS_MASK, udc_ep0_ptr->pcs);
+ cmd_param1 = upper_32_bits(dqptaddr);
+ ret = crg_udc_issue_command(crg_udc, CRG_CMD_INIT_EP0,
+ cmd_param0, cmd_param1);
+ if (ret)
+ goto reset_error;
+
+ udc_ep0_ptr->deq_pt = udc_ep0_ptr->enq_pt;
+ udc_ep0_ptr->tran_ring_full = false;
+ udc_ep0_ptr->ep_state = EP_STATE_RUNNING;
+ crg_udc->ep0_state = EP0_STATE_IDLE;
+ } else {
+ if (!list_empty(&udc_ep0_ptr->queue)) {
+ dev_err(crg_udc->dev,
+ "remaining ep0 xfer trbs on reset\n");
+ return -EBUSY;
+ }
+ }
+
+ crg_udc_enable_setup(crg_udc);
+
+ return 0;
+
+reset_error:
+ crg_udc->ep0_state = EP0_STATE_ERROR;
+ return ret;
+}
+
+static void set_address_cmpl(struct crg_gadget_dev *crg_udc)
+{
+ if (crg_udc->device_state == USB_STATE_DEFAULT &&
+ crg_udc->dev_addr != 0) {
+ crg_udc->device_state = USB_STATE_ADDRESS;
+ usb_gadget_set_state(&crg_udc->gadget, USB_STATE_ADDRESS);
+ } else if (crg_udc->device_state == USB_STATE_ADDRESS) {
+ if (crg_udc->dev_addr == 0) {
+ crg_udc->device_state = USB_STATE_DEFAULT;
+ usb_gadget_set_state(&crg_udc->gadget,
+ USB_STATE_DEFAULT);
+ }
+
+ }
+}
+
+int crg_udc_set_ep0_halt(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+ enum crg_ep_state ep_state;
+
+ ep_state = crg_udc_get_ep_state(crg_udc, udc_ep_ptr->dci);
+ if (ep_state == EP_STATE_HALTED || ep_state == EP_STATE_DISABLED)
+ return 0;
+
+ crg_udc_build_ep0_status(udc_ep_ptr, true, -EINVAL, NULL, 0, 1);
+
+ udc_ep_ptr->ep_state = EP_STATE_HALTED;
+
+ return 0;
+}
+
+static void set_sel_cmpl(struct usb_ep *ep, struct usb_request *req)
+{
+ struct crg_udc_ep *udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+ struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+ const struct usb_set_sel_req *sel;
+ unsigned long flags;
+ void *buf;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ buf = req->buf;
+ if (buf && !req->status && req->actual >= sizeof(*sel)) {
+ sel = buf;
+ crg_udc->sel_value.u1_sel_value = sel->u1_sel;
+ crg_udc->sel_value.u1_pel_value = sel->u1_pel;
+ crg_udc->sel_value.u2_sel_value = le16_to_cpu(sel->u2_sel);
+ crg_udc->sel_value.u2_pel_value = le16_to_cpu(sel->u2_pel);
+ }
+ req->buf = NULL;
+ crg_udc->status_req_callback_pending = false;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ kfree(buf);
+}
+
+static void set_test_mode_cmpl(struct crg_gadget_dev *crg_udc)
+{
+ if (crg_udc->set_tm != 0) {
+ u32 tmp;
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+
+ tmp = readl(&uccr->u2portpmsc);
+ FIELD_MODIFY(CRG_U3DC_U2PORTPM_TM_MASK, &tmp, crg_udc->set_tm);
+ writel(tmp, &uccr->u2portpmsc);
+
+ crg_udc->set_tm = 0;
+ }
+}
+
+static void setaddressrequest(struct crg_gadget_dev *crg_udc, u16 value,
+ u16 index, u16 length)
+{
+ u32 param0;
+
+ if (value > 127 || index || length ||
+ (crg_udc->device_state != USB_STATE_DEFAULT &&
+ crg_udc->device_state != USB_STATE_ADDRESS))
+ goto invalid;
+
+ /* SET_ADDRESS(0) is legal in the Default state. */
+ param0 = FIELD_PREP(CRG_CMD2_0_DEV_ADDR_MASK, value);
+ if (crg_udc_issue_command(crg_udc, CRG_CMD_SET_ADDR, param0, 0))
+ goto invalid;
+
+ crg_udc->dev_addr = value;
+ crg_udc->setup_status = STATUS_STAGE_XFER;
+ crg_udc->setup_fn_call_back = set_address_cmpl;
+ crg_udc_build_ep0_status(&crg_udc->udc_ep[0], true,
+ -EINPROGRESS, NULL, 1, 0);
+ return;
+
+invalid:
+ crg_udc->setup_fn_call_back = NULL;
+ crg_udc_set_ep0_halt(crg_udc);
+}
+
+static bool setfeaturesrequest(struct crg_gadget_dev *crg_udc, u8 request_type,
+ u8 b_request, u16 value, u16 index, u16 length,
+ unsigned long *flags)
+{
+ int status = -EINPROGRESS;
+ u8 dci;
+ struct crg_udc_ep *udc_ep_ptr;
+ u32 tmp;
+ u32 set_feat = 0;
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+
+ if (length || (request_type & USB_DIR_IN))
+ goto set_feature_error;
+
+ /* TEST_MODE is permitted in the Default state; other features aren't. */
+ if (crg_udc->device_state == USB_STATE_DEFAULT &&
+ !((request_type & USB_RECIP_MASK) == USB_RECIP_DEVICE &&
+ value == USB_DEVICE_TEST_MODE &&
+ b_request == USB_REQ_SET_FEATURE))
+ goto set_feature_error;
+
+ set_feat = (b_request == USB_REQ_SET_FEATURE) ? 1 : 0;
+ if ((request_type & (USB_RECIP_MASK | USB_TYPE_MASK)) ==
+ (USB_RECIP_ENDPOINT | USB_TYPE_STANDARD)) {
+ dev_dbg(crg_udc->dev, "Halt/Unhalt EP\n");
+ if (crg_udc->device_state == USB_STATE_ADDRESS) {
+ if (index != 0) {
+ status = -EINVAL;
+ goto set_feature_error;
+ }
+ }
+
+ if (value != USB_ENDPOINT_HALT)
+ goto set_feature_error;
+
+ if (index & ~(USB_ENDPOINT_NUMBER_MASK | USB_DIR_IN))
+ goto set_feature_error;
+
+ dci = crg_udc_index2dci(index);
+ if (dci == 1 || dci >= CRG_NUM_EP_CX)
+ goto set_feature_error;
+ if (dci != 0 && crg_udc->device_state == USB_STATE_CONFIGURED &&
+ crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_DISABLED)
+ goto set_feature_error;
+
+ udc_ep_ptr = &crg_udc->udc_ep[dci];
+ status = crg_udc_ep_halt(
+ udc_ep_ptr, (b_request == USB_REQ_SET_FEATURE) ? 1 : 0,
+ 0, flags);
+
+ if (status < 0)
+ goto set_feature_error;
+ } else if ((request_type & (USB_RECIP_MASK | USB_TYPE_MASK)) ==
+ (USB_RECIP_DEVICE | USB_TYPE_STANDARD)) {
+ switch (value) {
+ case USB_DEVICE_REMOTE_WAKEUP:
+ /* REMOTE_WAKEUP selector is not used by USB3.0 */
+ if (index ||
+ crg_udc->device_state < USB_STATE_DEFAULT ||
+ crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+ status = -EINVAL;
+ goto set_feature_error;
+ }
+
+ crg_udc->u2_rwe = set_feat;
+ crg_udc->gadget.wakeup_armed = set_feat;
+
+ break;
+ case USB_DEVICE_U1_ENABLE:
+ case USB_DEVICE_U2_ENABLE: {
+ u32 timeout_val;
+
+ if (crg_udc->device_state != USB_STATE_CONFIGURED) {
+ dev_err(crg_udc->dev,
+ "u12 enable fail, usb state=%d\n",
+ crg_udc->device_state);
+ status = -EINVAL;
+ goto set_feature_error;
+ }
+
+ if (index != 0 ||
+ crg_udc->gadget.speed < USB_SPEED_SUPER) {
+ status = -EINVAL;
+ goto set_feature_error;
+ }
+
+ if (set_feat == 1 && crg_udc->u12_forbidden) {
+ status = -EINVAL;
+ goto set_feature_error;
+ }
+
+ tmp = readl(&uccr->u3portpmsc);
+ if (value == USB_DEVICE_U1_ENABLE) {
+ if (!crg_udc->u12_initiate_forbidden) {
+ tmp &= (~CRG_U3DC_U3PORTPM_U1IEN);
+ tmp |= FIELD_PREP(
+ CRG_U3DC_U3PORTPM_U1IEN,
+ set_feat);
+ }
+ if (!crg_udc->u12_forbidden)
+ timeout_val =
+ set_feat ? CRG_UDC_U1_TIMEOUT_VAL : 0;
+ else
+ timeout_val = 0;
+
+ FIELD_MODIFY(CRG_U3DC_U3PORTPM_U1TMOUT_MASK,
+ &tmp, timeout_val);
+
+ crg_udc->feature_u1_enable = set_feat;
+ }
+
+ if (value == USB_DEVICE_U2_ENABLE) {
+ if (!crg_udc->u12_initiate_forbidden) {
+ tmp &= (~CRG_U3DC_U3PORTPM_U2IEN);
+ tmp |= FIELD_PREP(
+ CRG_U3DC_U3PORTPM_U2IEN,
+ set_feat);
+ }
+ if (!crg_udc->u12_forbidden)
+ timeout_val =
+ set_feat ? CRG_UDC_U2_TIMEOUT_VAL : 0;
+ else
+ timeout_val = 0;
+
+ FIELD_MODIFY(CRG_U3DC_U3PORTPM_U2TMOUT_MASK,
+ &tmp, timeout_val);
+
+ crg_udc->feature_u2_enable = set_feat;
+ }
+ writel(tmp, &uccr->u3portpmsc);
+
+ break;
+ }
+ case USB_DEVICE_TEST_MODE: {
+ u32 u_pattern;
+
+ if (b_request != USB_REQ_SET_FEATURE ||
+ crg_udc->gadget.speed != USB_SPEED_HIGH ||
+ crg_udc->device_state < USB_STATE_DEFAULT ||
+ (index & 0xff) || (index >> 8) == 0 ||
+ (index >> 8) > 6)
+ goto set_feature_error;
+ u_pattern = index >> 8;
+ crg_udc->set_tm = u_pattern;
+ crg_udc->setup_fn_call_back = &set_test_mode_cmpl;
+
+ break;
+ }
+
+ default:
+ goto set_feature_error;
+ }
+
+ } else if ((request_type & (USB_RECIP_MASK | USB_TYPE_MASK)) ==
+ (USB_RECIP_INTERFACE | USB_TYPE_STANDARD)) {
+ if (crg_udc->device_state != USB_STATE_CONFIGURED) {
+ dev_err(crg_udc->dev,
+ "interface u12 enable fail, usb state=%d\n",
+ crg_udc->device_state);
+ status = -EINVAL;
+ goto set_feature_error;
+ }
+
+ /* Suspend Option */
+ if (value != USB_INTRF_FUNC_SUSPEND) {
+ status = -EINVAL;
+ goto set_feature_error;
+ }
+
+ /* Both cleared LP/RW bits and either suspend option are valid. */
+ if (index & ~(0xff | USB_INTRF_FUNC_SUSPEND_LP |
+ USB_INTRF_FUNC_SUSPEND_RW))
+ goto set_feature_error;
+
+ if (index & USB_INTRF_FUNC_SUSPEND_RW)
+ dev_dbg(crg_udc->dev, "Interface En Remote Wakeup\n");
+ else
+ dev_dbg(crg_udc->dev, "Interface Dis RemoteWakeup\n");
+
+ /* Do not need to return status stage here
+ * Pass to composite gadget driver to process
+ * the request
+ */
+ return false;
+ } else {
+ /* Unknown recipient must not receive a successful status stage. */
+ goto set_feature_error;
+ }
+
+ crg_udc->setup_status = STATUS_STAGE_XFER;
+ crg_udc_build_ep0_status(&crg_udc->udc_ep[0], true, status, NULL, 0, 0);
+ return true;
+
+set_feature_error:
+ crg_udc_set_ep0_halt(crg_udc);
+ return true;
+}
+
+static void setselrequest(struct crg_gadget_dev *crg_udc, u16 value, u16 index,
+ u16 length)
+{
+ struct crg_udc_request *udc_req_ptr = crg_udc->status_req;
+ struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+ struct usb_set_sel_req *sel;
+ int ret;
+
+ if (!udc_req_ptr || crg_udc->status_req_callback_pending ||
+ !list_empty(&udc_req_ptr->queue) ||
+ crg_udc->device_state == USB_STATE_DEFAULT || value || index ||
+ length != sizeof(struct usb_set_sel_req)) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ kfree(udc_req_ptr->usb_req.buf);
+
+ udc_req_ptr->usb_req.buf = kzalloc_obj(*sel, GFP_ATOMIC);
+ if (!udc_req_ptr->usb_req.buf) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ sel = udc_req_ptr->usb_req.buf;
+ udc_req_ptr->usb_req.length = sizeof(*sel);
+ udc_req_ptr->usb_req.status = -EINPROGRESS;
+ udc_req_ptr->usb_req.actual = 0;
+ udc_req_ptr->usb_req.complete = set_sel_cmpl;
+ crg_udc->status_req_callback_pending = true;
+ udc_req_ptr->map_dir = DMA_FROM_DEVICE;
+ ret = usb_gadget_map_request_by_dev(crg_udc->dev, &udc_req_ptr->usb_req,
+ false);
+ if (ret) {
+ kfree(udc_req_ptr->usb_req.buf);
+ udc_req_ptr->usb_req.buf = NULL;
+ crg_udc->status_req_callback_pending = false;
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ crg_udc->setup_status = DATA_STAGE_RECV;
+
+ ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+ if (ret) {
+ crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr);
+ kfree(udc_req_ptr->usb_req.buf);
+ udc_req_ptr->usb_req.buf = NULL;
+ crg_udc->status_req_callback_pending = false;
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+ list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+}
+
+int crg_udc_update_ep0_maxpacketsize(struct crg_gadget_dev *crg_udc)
+{
+ u16 maxpacketsize = 0;
+ struct crg_udc_ep *udc_ep0 = &crg_udc->udc_ep[0];
+ u32 param0;
+
+ if (crg_udc->gadget.speed >= USB_SPEED_SUPER)
+ maxpacketsize = 512;
+ else
+ maxpacketsize = 64;
+
+ param0 = FIELD_PREP(CRG_CMD1_0_MPS_MASK, maxpacketsize);
+ if (crg_udc_issue_command(crg_udc, CRG_CMD_UPDATE_EP0_CFG, param0, 0))
+ return -EIO;
+
+ crg_udc->ep0_desc.wMaxPacketSize = cpu_to_le16(maxpacketsize);
+ udc_ep0->usb_ep.maxpacket = maxpacketsize;
+
+ return 0;
+}
+
+void crg_udc_handle_setup_pkt(struct crg_gadget_dev *crg_udc,
+ struct usb_ctrlrequest *setup_pkt, u8 setup_tag,
+ unsigned long *flags)
+{
+ u16 w_value = le16_to_cpu(setup_pkt->wValue);
+ u16 w_index = le16_to_cpu(setup_pkt->wIndex);
+ u16 w_length = le16_to_cpu(setup_pkt->wLength);
+ struct usb_gadget_driver *driver;
+ u32 tmp;
+
+ /* No SETUP (including SET_ADDRESS) may overtake a pending reset. */
+ spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+ flush_work(&crg_udc->udc_reset_work);
+ spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+ if (crg_udc->resetting || !crg_udc->gadget_driver ||
+ crg_udc->ep0_state == EP0_STATE_ABORTING)
+ return;
+
+ trace_crg_udc_ctrl_req(setup_pkt);
+
+ /* EP0 come backs to running when new setup packet comes */
+ crg_udc->udc_ep[0].ep_state = EP_STATE_RUNNING;
+
+ crg_udc->setup_tag = setup_tag;
+ crg_udc->setup_status = SETUP_PKT_PROCESS_IN_PROGRESS;
+ crg_udc->setup_fn_call_back = NULL;
+
+ if ((setup_pkt->bRequestType & USB_TYPE_MASK) == USB_TYPE_STANDARD) {
+ switch (setup_pkt->bRequest) {
+ case USB_REQ_GET_STATUS:
+ if (setup_pkt->bRequestType !=
+ (USB_DIR_IN | USB_RECIP_DEVICE |
+ USB_TYPE_STANDARD) &&
+ setup_pkt->bRequestType !=
+ (USB_DIR_IN | USB_RECIP_INTERFACE |
+ USB_TYPE_STANDARD) &&
+ setup_pkt->bRequestType !=
+ (USB_DIR_IN | USB_RECIP_ENDPOINT |
+ USB_TYPE_STANDARD)) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ getstatusrequest(crg_udc, setup_pkt->bRequestType,
+ w_value, w_index, w_length);
+ return;
+ case USB_REQ_SET_ADDRESS:
+ if (setup_pkt->bRequestType !=
+ (USB_DIR_OUT | USB_RECIP_DEVICE |
+ USB_TYPE_STANDARD)) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ setaddressrequest(crg_udc, w_value, w_index, w_length);
+ return;
+ case USB_REQ_SET_SEL:
+ if (setup_pkt->bRequestType !=
+ (USB_DIR_OUT | USB_RECIP_DEVICE |
+ USB_TYPE_STANDARD)) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+ crg_udc->setup_status = DATA_STAGE_RECV;
+ setselrequest(crg_udc, w_value, w_index, w_length);
+ return;
+ case USB_REQ_SET_ISOCH_DELAY:
+ if (setup_pkt->bRequestType !=
+ (USB_DIR_OUT | USB_RECIP_DEVICE |
+ USB_TYPE_STANDARD) ||
+ w_index || w_length ||
+ w_value >
+ FIELD_MAX(
+ CRG_U3DC_PTMCTRL_ISOCH_DELAY_MASK)) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ tmp = readl(&crg_udc->uccr->ptmctrl);
+ FIELD_MODIFY(CRG_U3DC_PTMCTRL_ISOCH_DELAY_MASK, &tmp,
+ w_value);
+ writel(tmp, &crg_udc->uccr->ptmctrl);
+ crg_udc->setup_status = STATUS_STAGE_XFER;
+ crg_udc_build_ep0_status(&crg_udc->udc_ep[0], true,
+ -EINPROGRESS, NULL, 0, 0);
+ return;
+
+ case USB_REQ_CLEAR_FEATURE:
+ case USB_REQ_SET_FEATURE:
+ /* Need composite gadget driver
+ * to process the function remote wakeup request
+ */
+ if (setfeaturesrequest(crg_udc, setup_pkt->bRequestType,
+ setup_pkt->bRequest, w_value,
+ w_index, w_length, flags)) {
+ /* Get here if request has been processed.*/
+ return;
+ }
+ break;
+ case USB_REQ_SET_CONFIGURATION:
+ if (setup_pkt->bRequestType !=
+ (USB_DIR_OUT | USB_RECIP_DEVICE |
+ USB_TYPE_STANDARD) ||
+ w_length || w_value > 0xff) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+ /* In theory we need to clear RUN bit before
+ * status stage of deconfig request sent.
+ * But seeing problem if we do it before all the
+ * endpoints belonging to the configuration
+ * disabled.
+ */
+ if (crg_udc->device_state <= USB_STATE_DEFAULT) {
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+
+ break;
+ default:
+ dev_dbg(crg_udc->dev,
+ "USB_REQ default bRequest=%d, bRequestType=%d\n",
+ setup_pkt->bRequest, setup_pkt->bRequestType);
+ }
+ }
+
+ if (w_length) {
+ /* data phase from gadget like GET_CONFIGURATION
+ * call the setup routine of gadget driver.
+ * remember the request direction.
+ */
+ crg_udc->setup_status = (setup_pkt->bRequestType & USB_DIR_IN) ?
+ DATA_STAGE_XFER :
+ DATA_STAGE_RECV;
+ }
+ driver = crg_udc->async_callbacks ? crg_udc->gadget_driver : NULL;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+
+ /* Recheck callback enablement after the locked snapshot. */
+ if (!driver || !READ_ONCE(crg_udc->async_callbacks) ||
+ READ_ONCE(crg_udc->gadget_driver) != driver ||
+ driver->setup(&crg_udc->gadget, setup_pkt) < 0) {
+ spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+ crg_udc_set_ep0_halt(crg_udc);
+ return;
+ }
+ spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+ if (setup_pkt->bRequest == USB_REQ_SET_CONFIGURATION &&
+ (setup_pkt->bRequestType & (USB_TYPE_MASK | USB_RECIP_MASK)) ==
+ (USB_TYPE_STANDARD | USB_RECIP_DEVICE)) {
+ if (w_value) {
+ crg_udc->device_state = USB_STATE_CONFIGURED;
+ usb_gadget_set_state(&crg_udc->gadget,
+ USB_STATE_CONFIGURED);
+ } else {
+ /* Endpoint disable may already have updated device_state. */
+ crg_udc->device_state = USB_STATE_ADDRESS;
+ usb_gadget_set_state(&crg_udc->gadget,
+ USB_STATE_ADDRESS);
+ }
+ }
+}
+
+int crg_udc_init_ep0(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
+ u32 cmd_param0;
+ u32 cmd_param1;
+
+ /* setup transfer ring */
+ if (!udc_ep_ptr->tran_ring_info.cpu_addr) {
+ dma_addr_t dma;
+ u32 ring_size = CRG_UDC_CONTROL_EP_TD_RING_SIZE;
+ void *vaddr;
+ size_t len;
+
+ len = ring_size * sizeof(struct crg_transfer_trb);
+ vaddr = dma_alloc_coherent(crg_udc->dev, len, &dma, GFP_KERNEL);
+ if (!vaddr)
+ return -ENOMEM;
+
+ udc_ep_ptr->tran_ring_info.cpu_addr = vaddr;
+ udc_ep_ptr->tran_ring_info.dma_addr = dma;
+ udc_ep_ptr->tran_ring_info.len = len;
+ udc_ep_ptr->first_trb = vaddr;
+ udc_ep_ptr->last_trb = udc_ep_ptr->first_trb + ring_size - 1;
+ }
+
+ memset(udc_ep_ptr->first_trb, 0, udc_ep_ptr->tran_ring_info.len);
+ /* Ensure that transfer ring is updated */
+ dma_wmb();
+ udc_ep_ptr->enq_pt = udc_ep_ptr->first_trb;
+ udc_ep_ptr->deq_pt = udc_ep_ptr->first_trb;
+ udc_ep_ptr->pcs = 1;
+ udc_ep_ptr->tran_ring_full = false;
+
+ crg_udc_setup_link_trb(udc_ep_ptr->last_trb, true,
+ udc_ep_ptr->tran_ring_info.dma_addr);
+
+ /* context related ops */
+ cmd_param0 = (lower_32_bits(udc_ep_ptr->tran_ring_info.dma_addr) &
+ CRG_CMD0_0_DQPTRLO_MASK) |
+ FIELD_PREP(CRG_CMD0_0_DCS_MASK, udc_ep_ptr->pcs);
+ cmd_param1 = upper_32_bits(udc_ep_ptr->tran_ring_info.dma_addr);
+
+ if (crg_udc_issue_command(crg_udc, CRG_CMD_INIT_EP0, cmd_param0,
+ cmd_param1))
+ return -EIO;
+
+ udc_ep_ptr->desc = &crg_udc->ep0_desc;
+ udc_ep_ptr->ep_state = EP_STATE_RUNNING;
+ udc_ep_ptr->pcs = 1;
+
+ return 0;
+}
+
+int crg_udc_queue_ctrl(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr,
+ u32 num_of_trbs_needed)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+ enum crg_ep_state ep_state;
+ struct crg_transfer_trb *enq_pt = udc_ep_ptr->enq_pt;
+ struct crg_transfer_trb *dq_pt = udc_ep_ptr->deq_pt;
+ struct usb_request *usb_req = &udc_req_ptr->usb_req;
+ struct crg_transfer_trb *p_trb;
+ u32 transfer_length;
+ bool ioc;
+ bool azp;
+ u8 dir = 0;
+ u8 setup_tag = crg_udc->setup_tag;
+
+ /* A new SETUP aborts the previous EP0 transaction. The completion
+ * callback of the aborted request is allowed to run, but it must not
+ * requeue the old request before the new SETUP has been handed to the
+ * gadget driver.
+ */
+ if (crg_udc->ep0_state == EP0_STATE_ABORTING)
+ return -ECONNRESET;
+
+ ep_state = crg_udc_get_ep_state(crg_udc, 0);
+ /* Need to queue the request even ep is paused or halted */
+ if (ep_state != EP_STATE_RUNNING) {
+ dev_dbg(crg_udc->dev, "EP State = 0x%x\n", ep_state);
+ return -EINVAL;
+ }
+ if (list_empty(&udc_ep_ptr->queue)) {
+ /* For control endpoint, we can handle one setup request at a
+ * time. so if there are TD pending in the transfer ring.
+ * wait for the sequence number error event. Then put the new
+ * request to transfer ring
+ */
+ if (enq_pt == dq_pt) {
+ u32 i;
+ bool need_zlp = false;
+ dma_addr_t trb_dma = usb_req->dma;
+ u64 remaining = usb_req->length;
+ /* Transfer ring is empty
+ * setup data stage TRBs
+ */
+ udc_req_ptr->first_trb = udc_ep_ptr->enq_pt;
+ if (crg_udc->setup_status == DATA_STAGE_XFER)
+ dir = 0;
+ else if (crg_udc->setup_status == DATA_STAGE_RECV)
+ dir = 1;
+ else
+ dev_dbg(crg_udc->dev,
+ "unexpected setup_status!%d\n",
+ crg_udc->setup_status);
+ if (udc_req_ptr->usb_req.zero == 1 &&
+ udc_req_ptr->usb_req.length != 0 &&
+ ((udc_req_ptr->usb_req.length %
+ udc_ep_ptr->usb_ep.maxpacket) == 0))
+ need_zlp = true; /* zero length packet */
+ if (num_of_trbs_needed > 32)
+ return -EMSGSIZE;
+
+ if (num_of_trbs_needed >
+ crg_udc_room_on_ring(
+ crg_udc, CRG_UDC_CONTROL_EP_TD_RING_SIZE,
+ udc_ep_ptr->first_trb, enq_pt, dq_pt))
+ return -ENOSPC;
+
+ u8 original_pcs = udc_ep_ptr->pcs;
+
+ for (i = 0; i < num_of_trbs_needed; i++) {
+ p_trb = enq_pt;
+ transfer_length = crg_udc_trb_chunk_len(
+ trb_dma, remaining);
+ ioc = (i == num_of_trbs_needed - 1);
+ azp = ioc && need_zlp;
+ crg_udc_setup_datastage_trb(
+ p_trb, trb_dma,
+ i == 0 ? (original_pcs ^ 1) :
+ udc_ep_ptr->pcs,
+ transfer_length,
+ num_of_trbs_needed - i - 1, ioc,
+ !ioc, azp, dir, setup_tag);
+ udc_req_ptr->all_trbs_queued = 1;
+ trb_dma += transfer_length;
+ remaining -= transfer_length;
+ enq_pt++;
+ if (FIELD_GET(TRB_TYPE_MASK,
+ le32_to_cpu(enq_pt->dw3)) ==
+ TRB_TYPE_LINK) {
+ if (FIELD_GET(
+ TRB_LINK_TOGGLE_CYCLE_MASK,
+ le32_to_cpu(enq_pt->dw3))) {
+ dma_wmb();
+ le32p_replace_bits(
+ &enq_pt->dw3,
+ udc_ep_ptr->pcs,
+ TRB_CYCLE_BIT_MASK);
+ udc_ep_ptr->pcs ^= 0x1;
+ }
+ enq_pt = udc_ep_ptr->first_trb;
+ }
+ }
+ dma_wmb();
+ le32p_replace_bits(&udc_req_ptr->first_trb->dw3,
+ original_pcs, TRB_CYCLE_BIT_MASK);
+ udc_ep_ptr->enq_pt = enq_pt;
+ crg_udc_knock_doorbell(crg_udc, 0);
+ if (udc_ep_ptr->enq_pt == udc_ep_ptr->first_trb)
+ udc_req_ptr->last_trb =
+ udc_ep_ptr->last_trb - 1;
+ else
+ udc_req_ptr->last_trb = udc_ep_ptr->enq_pt - 1;
+ } else {
+ /*
+ * The software queue is empty but the transfer ring
+ * still holds pending TDs from a previous control
+ * transfer (the host issued a new setup before the
+ * old one drained). We process one setup at a time,
+ * so we intentionally do not queue TRBs here.
+ * Returning success leaves this request on the
+ * endpoint queue; the stale TDs and this request are
+ * flushed and completed (with -EINVAL) by the
+ * sequence-number-error / setup-tag-mismatch event in
+ * crg_udc_handle_xfer_event().
+ */
+ dev_err(crg_udc->dev,
+ "ep0 ring not idle on new setup\n");
+ }
+ } else {
+ dev_err(crg_udc->dev, "udc_ep_ptr->queue not empty\n");
+ /* New setup packet came
+ * Drop this request.
+ */
+ return -EINVAL;
+ }
+ return 0;
+}
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [PATCH v4 4/6] usb: gadget: udc: crg: Add endpoint 0 handling
2026-10-30 8:10 ` [PATCH v4 4/6] usb: gadget: udc: crg: Add endpoint 0 handling Petar Stepanovic
@ 2026-10-30 8:40 ` sashiko-bot
0 siblings, 0 replies; 14+ messages in thread
From: sashiko-bot @ 2026-10-30 8:40 UTC (permalink / raw)
To: Petar Stepanovic; +Cc: linux-usb, devicetree, robh, conor+dt
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [High] usb: gadget: udc: crg: stale setup_status on EP0 halt in crg_udc_set_ep0_halt()
- [High] usb: gadget: udc: crg: missing state checks after relocking in crg_udc_handle_setup_pkt()
- [High] usb: gadget: crg: missing reset check after ep0 nuke in setup paths
- [Medium] usb: gadget: crg: crg_udc_prepare_for_setup() skips stopping halted EP0
--
commit 0dcda6f8ceb35327942fb48b7002136244fccc5c
Author: Petar Stepanovic <pstepanovic@axiado.com>
usb: gadget: udc: crg: Add endpoint 0 handling
This patch adds endpoint 0 setup packet, status stage, and control transfer
handling for the Axiado AX3000/AX3005 UDC driver.
> diff --git a/drivers/usb/gadget/udc/crg/crg-udc-ep0.c b/drivers/usb/gadget/udc/crg/crg-udc-ep0.c
> new file mode 100644
> index 0000000000000..073d9090dd07c
> --- /dev/null
> +++ b/drivers/usb/gadget/udc/crg/crg-udc-ep0.c
[ ... ]
> +int crg_udc_abort_ep0_for_setup(struct crg_gadget_dev *crg_udc)
> +{
> + struct crg_udc_ep *udc_ep0 = &crg_udc->udc_ep[0];
> + struct crg_uccr __iomem *uccr = crg_udc->uccr;
> + dma_addr_t dqptaddr;
> + u32 cmd_param0;
> + u32 cmd_param1;
> + u32 tmp;
> + int ret;
[ ... ]
> + udc_ep0->ep_state = EP_STATE_STOPPED;
> + /* Complete requests belonging to the aborted control transfer. */
> + crg_udc_ep_nuke(udc_ep0, -ECONNRESET);
> +
> + if (!udc_ep0->first_trb || !udc_ep0->tran_ring_info.cpu_addr) {
> + crg_udc->ep0_state = EP0_STATE_ERROR;
> + return -ENODEV;
> + }
[Severity: High]
When crg_udc_ep_nuke() runs here in crg_udc_abort_ep0_for_setup(), and
similarly in crg_udc_prepare_for_setup(), crg_udc_req_done() drops
crg_udc->udc_lock to invoke the request completion callback.
While udc_lock is dropped, another CPU running crg_gadget_stop() or
crg_udc_shutdown() can acquire udc_lock and call crg_udc_reset(), setting
crg_udc->resetting = true, incrementing udc_ep[0].generation, and dropping
udc_lock while polling for CRG_U3DC_CTRL_SWRST to clear, followed by
crg_udc_reset_data_struct() and crg_udc_init_ep0().
After crg_udc_ep_nuke() returns, neither crg_udc_abort_ep0_for_setup() nor
crg_udc_prepare_for_setup() checks crg_udc->resetting, udc_ep[0].generation,
or udc_event[0].generation before touching controller registers and issuing
CRG_CMD_INIT_EP0. If crg_udc_reset() is still polling SWRST, this issues
MMIO commands mid-reset; if crg_gadget_stop() has already staged
CRG_CMD_INIT_EP0 with CRG_U3DC_CMD_CTRL_ACTIVE set while RUN is clear, the
subsequent CRG_CMD_INIT_EP0 fails with -EBUSY, leaving ep0_state in
EP0_STATE_ERROR and causing crg_udc_handle_event() to call
crg_udc_set_ep0_halt() and queue status_req after the gadget has stopped.
Should both functions snapshot the generation counter before
crg_udc_ep_nuke() and abort if crg_udc->resetting is set or the generation
changed while udc_lock was dropped?
[ ... ]
> +int crg_udc_prepare_for_setup(struct crg_gadget_dev *crg_udc)
> +{
> + struct crg_udc_ep *udc_ep0_ptr;
> + dma_addr_t dqptaddr;
> + u32 cmd_param0;
> + u32 cmd_param1;
> + int ret;
> +
> + if (!crg_udc_is_event_rings_empty(crg_udc) ||
> + crg_udc->portsc_on_reconnecting == 1)
> + return -EBUSY;
> +
> + udc_ep0_ptr = &crg_udc->udc_ep[0];
> + /* If we reinit ep0 on bus reset, we just make ep0 dequeue pointer
> + * align with enqueue pointer, all remaining xfer trbs became dumb once
> + * which will not produce xfer event anymore.
> + *
> + * If we considering the opposite solution, we should wait all ep0 xfer
> + * trbs be completed(with some err complete code)
> + */
> + if (crg_udc->reinit_ep0_on_bus_reset) {
> + struct crg_uccr __iomem *uccr = crg_udc->uccr;
> + u32 running;
> +
> + /* Block requeue from giveback callbacks, then quiesce DMA. */
> + crg_udc->ep0_state = EP0_STATE_ABORTING;
> + if (crg_udc_get_ep_state(crg_udc, 0) == EP_STATE_RUNNING) {
> + ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP,
> + BIT(0), 0);
> + if (ret)
> + goto reset_error;
> +
> + ret = readl_poll_timeout_atomic(&uccr->ep_running, running,
> + !(running & BIT(0)), 10, 10000);
> + if (ret)
> + goto reset_error;
> + }
> + udc_ep0_ptr->ep_state = EP_STATE_STOPPED;
> + crg_udc_ep_nuke(udc_ep0_ptr, -ESHUTDOWN);
[Severity: Medium]
As reported by Sashiko on v3 [1], what happens if a USB bus reset occurs
after a control transfer has stalled?
When crg_udc_set_ep0_halt() stalls EP0, it calls crg_udc_build_ep0_status()
to queue a status-stage stall TRB, rings doorbell 0, places status_req on
udc_ep[0].queue, and immediately sets udc_ep[0].ep_state = EP_STATE_HALTED
in software while the hardware is still processing the TRB.
If a bus reset then triggers crg_udc_prepare_for_setup() while ep_state is
EP_STATE_HALTED, checking only for EP_STATE_RUNNING skips issuing
CRG_CMD_STOP_EP and polling uccr->ep_running before crg_udc_ep_nuke() and
CRG_CMD_INIT_EP0. Should crg_udc_prepare_for_setup() stop EP0 whenever it
is not disabled or stopped, matching crg_udc_abort_ep0_for_setup()?
[ ... ]
> +int crg_udc_set_ep0_halt(struct crg_gadget_dev *crg_udc)
> +{
> + struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[0];
> + enum crg_ep_state ep_state;
> +
> + ep_state = crg_udc_get_ep_state(crg_udc, udc_ep_ptr->dci);
> + if (ep_state == EP_STATE_HALTED || ep_state == EP_STATE_DISABLED)
> + return 0;
> +
> + crg_udc_build_ep0_status(udc_ep_ptr, true, -EINVAL, NULL, 0, 1);
> +
> + udc_ep_ptr->ep_state = EP_STATE_HALTED;
> +
> + return 0;
> +}
[Severity: High]
As reported by Sashiko on v3 [2], does crg_udc_set_ep0_halt() leave
crg_udc->setup_status in a stale data-stage state when stalling a control
transfer with wLength > 0?
In crg_udc_handle_setup_pkt() and getstatusrequest(), crg_udc->setup_status
is set to DATA_STAGE_XFER or DATA_STAGE_RECV before calling setselrequest(),
crg_udc_build_td(), or driver->setup():
crg_udc_handle_setup_pkt()
crg_udc->setup_status = DATA_STAGE_XFER / DATA_STAGE_RECV;
driver->setup() -> fails
crg_udc_set_ep0_halt()
crg_udc_build_ep0_status(..., stall = 1)
Because crg_udc_set_ep0_halt() does not update crg_udc->setup_status to
STATUS_STAGE_XFER (or clear crg_udc->setup_fn_call_back), two problems
follow:
1. crg_udc_setup_status_trb() computes dir as
(crg_udc->setup_status == STATUS_STAGE_XFER) ? 0 : 1, which evaluates
to 1 (OUT) instead of 0 (IN).
2. When the stall status TRB completes and invokes
crg_udc_ep0_req_complete(), switch (crg_udc->setup_status) matches
DATA_STAGE_XFER or DATA_STAGE_RECV instead of the default branch:
crg_udc_ep0_req_complete() {
switch (crg_udc->setup_status) {
case DATA_STAGE_XFER:
crg_udc->setup_status = STATUS_STAGE_RECV;
crg_udc_build_ep0_status(udc_ep_ptr, true, -EINPROGRESS, NULL,
0, 0);
break;
case DATA_STAGE_RECV:
crg_udc->setup_status = STATUS_STAGE_XFER;
crg_udc_build_ep0_status(udc_ep_ptr, true, -EINPROGRESS, NULL,
0, 0);
break;
This queues a second, unrequested, non-stalled status-stage TRB on EP0 and
rings doorbell 0 on the halted endpoint. Should crg_udc_set_ep0_halt() set
crg_udc->setup_status = STATUS_STAGE_XFER and clear
crg_udc->setup_fn_call_back before calling crg_udc_build_ep0_status()?
[ ... ]
> +void crg_udc_handle_setup_pkt(struct crg_gadget_dev *crg_udc,
> + struct usb_ctrlrequest *setup_pkt, u8 setup_tag,
> + unsigned long *flags)
> +{
> + u16 w_value = le16_to_cpu(setup_pkt->wValue);
> + u16 w_index = le16_to_cpu(setup_pkt->wIndex);
> + u16 w_length = le16_to_cpu(setup_pkt->wLength);
> + struct usb_gadget_driver *driver;
> + u32 tmp;
> +
> + /* No SETUP (including SET_ADDRESS) may overtake a pending reset. */
> + spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
> + flush_work(&crg_udc->udc_reset_work);
> + spin_lock_irqsave(&crg_udc->udc_lock, *flags);
> + if (crg_udc->resetting || !crg_udc->gadget_driver ||
> + crg_udc->ep0_state == EP0_STATE_ABORTING)
> + return;
[ ... ]
> + if (w_length) {
> + /* data phase from gadget like GET_CONFIGURATION
> + * call the setup routine of gadget driver.
> + * remember the request direction.
> + */
> + crg_udc->setup_status = (setup_pkt->bRequestType & USB_DIR_IN) ?
> + DATA_STAGE_XFER :
> + DATA_STAGE_RECV;
> + }
> + driver = crg_udc->async_callbacks ? crg_udc->gadget_driver : NULL;
> + spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
> +
> + /* Recheck callback enablement after the locked snapshot. */
> + if (!driver || !READ_ONCE(crg_udc->async_callbacks) ||
> + READ_ONCE(crg_udc->gadget_driver) != driver ||
> + driver->setup(&crg_udc->gadget, setup_pkt) < 0) {
> + spin_lock_irqsave(&crg_udc->udc_lock, *flags);
> + crg_udc_set_ep0_halt(crg_udc);
> + return;
> + }
> + spin_lock_irqsave(&crg_udc->udc_lock, *flags);
> + if (setup_pkt->bRequest == USB_REQ_SET_CONFIGURATION &&
> + (setup_pkt->bRequestType & (USB_TYPE_MASK | USB_RECIP_MASK)) ==
> + (USB_TYPE_STANDARD | USB_RECIP_DEVICE)) {
> + if (w_value) {
> + crg_udc->device_state = USB_STATE_CONFIGURED;
> + usb_gadget_set_state(&crg_udc->gadget,
> + USB_STATE_CONFIGURED);
> + } else {
> + /* Endpoint disable may already have updated device_state. */
> + crg_udc->device_state = USB_STATE_ADDRESS;
> + usb_gadget_set_state(&crg_udc->gadget,
> + USB_STATE_ADDRESS);
> + }
> + }
> +}
[Severity: High]
Can dropping crg_udc->udc_lock in crg_udc_handle_setup_pkt() race with a
concurrent reset, unbind, or new setup packet?
First, around flush_work(&crg_udc->udc_reset_work), if a new setup packet
aborted an in-flight control transfer, crg_udc_abort_ep0_for_setup() has
already set ep0_state = EP0_STATE_IDLE before udc_lock is dropped for
flush_work(), allowing a concurrent usb_ep_queue() on EP0 to bypass the
EP0_STATE_ABORTING check in crg_udc_queue_ctrl(). In addition, if a
controller reset (crg_udc_reset() and crg_udc_reset_data_struct())
completes while udc_lock is dropped, crg_udc->resetting is false again and
udc_event[0].generation has been incremented, but crg_udc_handle_setup_pkt()
does not verify udc_event[0].generation and continues processing the stale
pre-reset setup packet.
Second, after udc_lock is dropped for driver->setup() and reacquired,
neither the error branch nor the USB_REQ_SET_CONFIGURATION branch rechecks
crg_udc->resetting, crg_udc->gadget_driver, crg_udc->async_callbacks,
crg_udc->ep0_state, crg_udc->setup_tag, or udc_event[0].generation:
1. If async_callbacks was cleared during gadget stop/unbind, or if
crg_udc_reset() is currently polling SWRST with udc_lock dropped (where
crg_udc->resetting is true while udc_ep[0].ep_state is still
EP_STATE_RUNNING), the error path unconditionally calls
crg_udc_set_ep0_halt(crg_udc), queueing a status TRB and ringing
doorbell 0 during reset or teardown.
2. If driver->setup() succeeded for USB_REQ_SET_CONFIGURATION while a
concurrent reset or disconnect occurred, the post-lock block overwrites
crg_udc->device_state and the gadget state with USB_STATE_CONFIGURED or
USB_STATE_ADDRESS.
Should crg_udc_handle_setup_pkt() keep ep0_state in EP0_STATE_ABORTING
until flush_work() finishes, and validate the event generation,
crg_udc->resetting, crg_udc->async_callbacks, and crg_udc->setup_tag after
reacquiring udc_lock?
[1] https://lore.kernel.org/r/20261020-axiado-ax3000-usb-device-controller-v3-4-3012ecfc0a93@axiado.com
[2] https://lore.kernel.org/r/20261020-axiado-ax3000-usb-device-controller-v3-4-3012ecfc0a93@axiado.com
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261030-axiado-ax3000-usb-device-controller-v4-0-c5267b36a777@axiado.com?part=4
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v4 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations
2026-10-30 8:10 [PATCH v4 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
` (3 preceding siblings ...)
2026-10-30 8:10 ` [PATCH v4 4/6] usb: gadget: udc: crg: Add endpoint 0 handling Petar Stepanovic
@ 2026-10-30 8:10 ` Petar Stepanovic
2026-10-30 8:50 ` sashiko-bot
2026-10-30 8:10 ` [PATCH v4 6/6] usb: gadget: udc: crg: Add core driver and build integration Petar Stepanovic
5 siblings, 1 reply; 14+ messages in thread
From: Petar Stepanovic @ 2026-10-30 8:10 UTC (permalink / raw)
To: Naveen Kumar Rajgiri Bassappa, Prasad Bolisetty,
Greg Kroah-Hartman, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Tzu-Hao Wei, Karthikeyan Mitran, Philipp Zabel
Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
Petar Stepanovic, Krutik Shah
This patch implements the gadget and endpoint operations for the
Axiado AX3000/AX3005 UDC driver.
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
drivers/usb/gadget/udc/crg/crg-udc-gadget.c | 2377 +++++++++++++++++++++++++++
drivers/usb/gadget/udc/crg/crg-udc-trace.c | 9 +
2 files changed, 2386 insertions(+)
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-gadget.c b/drivers/usb/gadget/udc/crg/crg-udc-gadget.c
new file mode 100644
index 000000000000..8c52f8856490
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-gadget.c
@@ -0,0 +1,2377 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/iopoll.h>
+#include <linux/scatterlist.h>
+
+#include "crg-udc.h"
+#include "crg-udc-trace.h"
+
+/* Forward declarations for functions used before their definition. */
+static int crg_udc_epcx_setup(struct crg_udc_ep *udc_ep);
+static int crg_udc_epcx_update_dqptr(struct crg_udc_ep *udc_ep);
+static int crg_udc_queue_trbs(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr, bool b_isoc,
+ u32 xfer_ring_size, u32 num_trbs_needed,
+ u64 buffer_length);
+static int crg_udc_reinit(struct crg_gadget_dev *crg_udc);
+static bool crg_udc_sg_advance(struct crg_udc_request *udc_req, u32 bytes);
+static bool crg_udc_sg_current(struct crg_udc_request *udc_req, dma_addr_t *dma,
+ u32 *len);
+static int crg_udc_sg_init(struct crg_udc_request *udc_req);
+static int crg_udc_validate_ep_desc(struct crg_udc_ep *udc_ep,
+ const struct usb_endpoint_descriptor *desc);
+static bool is_request_dequeued(struct crg_gadget_dev *crg_udc,
+ struct crg_udc_ep *udc_ep,
+ struct crg_event_trb *event);
+static int set_ep_halt(struct crg_gadget_dev *crg_udc, int dci,
+ unsigned long *flags);
+
+static int crg_gadget_get_frame(struct usb_gadget *g)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+
+ return FIELD_GET(CRG_U3DC_MFINDEX_FRAME_MASK,
+ readl(&crg_udc->uccr->mfindex));
+}
+
+static int crg_gadget_pullup(struct usb_gadget *g, int is_on)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+ unsigned long flags;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (is_on && crg_udc->resetting) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -ESHUTDOWN;
+ }
+ crg_udc->pullup_on = !!is_on;
+ crg_udc_set_pullup(crg_udc, crg_udc->pullup_on);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ trace_crg_udc_pullup(crg_udc->pullup_on);
+ return 0;
+}
+
+static int crg_gadget_set_remote_wakeup(struct usb_gadget *g, int set)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+ unsigned long flags;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ crg_udc->u2_rwe = !!set;
+ g->wakeup_armed = !!set;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ return 0;
+}
+
+static int crg_gadget_set_selfpowered(struct usb_gadget *g, int set)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+ unsigned long flags;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ g->is_selfpowered = !!set;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ return 0;
+}
+
+static void crg_udc_set_speed(struct usb_gadget *gadget,
+ enum usb_device_speed speed)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(gadget);
+
+ crg_udc->max_speed = speed;
+}
+
+static int crg_gadget_start(struct usb_gadget *g,
+ struct usb_gadget_driver *driver)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+ unsigned long flags;
+ int ret;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ /* Do not reopen a controller whose previous reset failed. */
+ if (crg_udc->resetting) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EBUSY;
+ }
+ /* The UDC core calls udc_set_speed() before udc_start(). */
+ ret = crg_udc_program_speed(crg_udc);
+ if (!ret) {
+ crg_udc->gadget_driver = driver;
+ crg_udc_set_vbus_detect(crg_udc, true);
+ }
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return ret;
+}
+
+static int crg_gadget_stop(struct usb_gadget *g)
+{
+ struct crg_gadget_dev *crg_udc;
+ unsigned long flags = 0;
+
+ crg_udc = gadget_to_udc(g);
+
+ /*
+ * Clear gadget_driver first so crg_udc_handle_port_status() can no
+ * longer schedule udc_reset_work (it only does so while
+ * gadget_driver is set), then cancel_work_sync() is guaranteed to
+ * be the last word: no new reset notification can be queued after
+ * it returns.
+ */
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ crg_udc->gadget_driver = NULL;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ cancel_work_sync(&crg_udc->udc_reset_work);
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ crg_udc->device_state = USB_STATE_ATTACHED;
+ crg_udc_set_vbus_detect(crg_udc, false);
+
+ /*
+ * crg_udc_reset() completes any queued requests via
+ * crg_udc_ep_nuke() -> crg_udc_req_done(). crg_udc_req_done() drops udc_lock
+ * (IRQ-safe) around each ->complete() callback and re-acquires it
+ * afterwards, so gadget completion callbacks are never invoked with
+ * the spinlock held. flags is threaded through for that purpose.
+ */
+ if (crg_udc_reset(crg_udc, &flags)) {
+ dev_err(crg_udc->dev, "controller reset failed during stop\n");
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EIO;
+ }
+ if (crg_udc_reset_data_struct(crg_udc)) {
+ dev_err(crg_udc->dev,
+ "controller data reset failed during stop\n");
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EIO;
+ }
+
+ crg_udc->connected = 0;
+ crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+
+ if (crg_udc_init_ep0(crg_udc)) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EIO;
+ }
+
+ /* The hardware and EP0 have been reinitialized successfully. */
+ crg_udc->resetting = false;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ return 0;
+}
+
+static int crg_gadget_wakeup(struct usb_gadget *g)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ unsigned long flags;
+ u32 portsc;
+ int ret = 0;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (crg_udc->resetting) {
+ ret = -ESHUTDOWN;
+ goto out;
+ }
+ if (!crg_udc->connected || g->speed > USB_SPEED_HIGH) {
+ ret = -EOPNOTSUPP;
+ goto out;
+ }
+ if (!crg_udc->u2_rwe) {
+ ret = -EACCES;
+ goto out;
+ }
+
+ portsc = readl(&uccr->portsc);
+ if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, portsc) !=
+ CRG_U3DC_PORTSC_PLS_U3) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ portsc &= ~(CRG_U3DC_PORTSC_W1C_MASK | CRG_U3DC_PORTSC_PLS_MASK);
+ portsc |= CRG_U3DC_PORTSC_LWS;
+ writel(portsc, &uccr->portsc);
+out:
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return ret;
+}
+
+static void crg_gadget_async_callbacks(struct usb_gadget *g, bool enable)
+{
+ struct crg_gadget_dev *crg_udc = gadget_to_udc(g);
+ unsigned long flags;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ crg_udc->async_callbacks = enable;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ /* Drain any in-flight notification before the gadget is unbound. */
+ if (!enable)
+ cancel_work_sync(&crg_udc->udc_reset_work);
+}
+
+struct usb_request *crg_udc_alloc_request(struct usb_ep *_ep, gfp_t gfp_flags)
+{
+ struct crg_udc_request *udc_req_ptr;
+
+ udc_req_ptr = kzalloc_obj(*udc_req_ptr, gfp_flags);
+
+ if (!udc_req_ptr)
+ return NULL;
+
+ udc_req_ptr->usb_req.dma = DMA_MAPPING_ERROR;
+ INIT_LIST_HEAD(&udc_req_ptr->queue);
+
+ trace_crg_udc_alloc_request(
+ container_of(_ep, struct crg_udc_ep, usb_ep), udc_req_ptr);
+
+ return &udc_req_ptr->usb_req;
+}
+
+int crg_udc_build_td(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr)
+{
+ int status = 0;
+ struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+ u32 num_trbs_needed;
+ u64 buffer_length;
+
+ if (udc_req_ptr->trbs_needed) {
+ /* If this is called to complete pending TRB transfers
+ * of previous Request
+ */
+ buffer_length = udc_req_ptr->buff_len_left;
+ num_trbs_needed = udc_req_ptr->trbs_needed;
+ } else {
+ buffer_length = udc_req_ptr->usb_req.length;
+ if (udc_req_ptr->usb_req.num_mapped_sgs)
+ num_trbs_needed = crg_udc_count_trbs_sg(udc_req_ptr);
+ else
+ num_trbs_needed = crg_udc_count_trbs_linear(
+ udc_req_ptr->usb_req.dma, buffer_length);
+ if (!num_trbs_needed && buffer_length)
+ return -EINVAL;
+ if (!num_trbs_needed)
+ num_trbs_needed = 1;
+ }
+
+ if (usb_endpoint_xfer_control(udc_ep_ptr->desc))
+ status = crg_udc_queue_ctrl(udc_ep_ptr, udc_req_ptr,
+ num_trbs_needed);
+ else if (usb_endpoint_xfer_isoc(udc_ep_ptr->desc))
+ status = crg_udc_queue_trbs(udc_ep_ptr, udc_req_ptr, true,
+ CRG_UDC_ISOC_EP_TD_RING_SIZE,
+ num_trbs_needed, buffer_length);
+ else if (usb_endpoint_xfer_bulk(udc_ep_ptr->desc))
+ status = crg_udc_queue_trbs(udc_ep_ptr, udc_req_ptr, false,
+ CRG_UDC_BULK_EP_TD_RING_SIZE,
+ num_trbs_needed, buffer_length);
+ else
+ /* interrupt endpoint */
+ status = crg_udc_queue_trbs(udc_ep_ptr, udc_req_ptr, false,
+ CRG_UDC_INT_EP_TD_RING_SIZE,
+ num_trbs_needed, buffer_length);
+
+ if (!status) {
+ udc_req_ptr->td_pending = true;
+ if (!usb_endpoint_xfer_control(udc_ep_ptr->desc))
+ crg_udc_knock_doorbell(crg_udc, udc_ep_ptr->dci);
+ }
+
+ return status;
+}
+
+void crg_udc_clear_portpm(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 tmp;
+
+ tmp = readl(&uccr->u3portpmsc);
+
+ tmp &= (~CRG_U3DC_U3PORTPM_U1IEN);
+ FIELD_MODIFY(CRG_U3DC_U3PORTPM_U1TMOUT_MASK, &tmp, 0);
+ tmp &= (~CRG_U3DC_U3PORTPM_U2IEN);
+ FIELD_MODIFY(CRG_U3DC_U3PORTPM_U2TMOUT_MASK, &tmp, 0);
+
+ writel(tmp, &uccr->u3portpmsc);
+
+ crg_udc->feature_u1_enable = 0;
+ crg_udc->feature_u2_enable = 0;
+}
+
+static int crg_udc_ep_dequeue(struct usb_ep *_ep, struct usb_request *_req)
+{
+ struct crg_udc_ep *udc_ep_ptr;
+ struct crg_gadget_dev *crg_udc;
+ struct crg_uccr __iomem *uccr;
+ struct crg_udc_request *udc_req = NULL;
+ struct crg_udc_request *iter;
+ struct crg_ep_cx *p_ep_cx;
+ struct crg_transfer_trb *pause_pt;
+ enum crg_ep_state old_ep_state;
+ u32 tmp, deq_pt_lo, deq_pt_hi;
+ dma_addr_t dq_pt_addr;
+ u8 dci;
+ bool found = false;
+ u32 generation;
+ unsigned long flags;
+ const struct usb_endpoint_descriptor *old_desc;
+ u32 event_generation;
+
+ if (!_ep || !_req)
+ return -EINVAL;
+
+ udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+ crg_udc = udc_ep_ptr->crg_udc;
+ uccr = crg_udc->uccr;
+ dci = udc_ep_ptr->dci;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (crg_udc->resetting) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -ESHUTDOWN;
+ }
+ old_ep_state = udc_ep_ptr->ep_state;
+ old_desc = udc_ep_ptr->desc;
+ event_generation = crg_udc->udc_event[0].generation;
+ generation = udc_ep_ptr->generation;
+
+ list_for_each_entry(iter, &udc_ep_ptr->queue, queue) {
+ if (&iter->usb_req == _req) {
+ udc_req = iter;
+ found = true;
+ break;
+ }
+ }
+
+ if (!found) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EINVAL;
+ }
+
+ if (!udc_req->first_trb) {
+ crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return 0;
+ }
+
+ if (dci == 0) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EINVAL;
+ }
+
+ if (old_ep_state == EP_STATE_RUNNING) {
+ int ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP,
+ BIT(dci), 0);
+
+ if (ret) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return ret;
+ }
+ if (readl_poll_timeout_atomic(&uccr->ep_running, tmp,
+ !(tmp & BIT(dci)), 10, 10000)) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -ETIMEDOUT;
+ }
+ udc_ep_ptr->ep_state = EP_STATE_STOPPED;
+ }
+
+ p_ep_cx = &crg_udc->p_epcx[dci - 2];
+ deq_pt_lo = le32_to_cpu(p_ep_cx->dw2) & EP_CX_TR_DQPT_LO_MASK;
+ deq_pt_hi = le32_to_cpu(p_ep_cx->dw3);
+ dq_pt_addr = (u64)deq_pt_lo | ((u64)deq_pt_hi << 32);
+ pause_pt = crg_udc_tran_trb_dma_to_virt(udc_ep_ptr, dq_pt_addr);
+ if (!pause_pt) {
+ if (old_ep_state == EP_STATE_RUNNING) {
+ udc_ep_ptr->ep_state = EP_STATE_RUNNING;
+ crg_udc_knock_doorbell(crg_udc, dci);
+ }
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EIO;
+ }
+
+ if (crg_udc_is_pointer_less_than(pause_pt, udc_req->first_trb,
+ udc_ep_ptr)) {
+ crg_udc_squeeze_xfer_ring(udc_ep_ptr, udc_req);
+ /* Give back the cancelled request BEFORE scheduling the tail:
+ * its callback may add/remove requests, and dropping udc_lock
+ * while traversing the old list is unsafe.
+ */
+ crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+ } else if (udc_req->last_trb &&
+ crg_udc_is_pointer_less_than(udc_req->last_trb, pause_pt,
+ udc_ep_ptr)) {
+ crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+ } else {
+ struct crg_udc_request *next_req = NULL;
+
+ udc_req->usb_req.actual =
+ crg_udc_actual_before(udc_ep_ptr, udc_req, pause_pt);
+
+ if (!list_is_last(&udc_req->queue, &udc_ep_ptr->queue))
+ next_req = list_entry(udc_req->queue.next,
+ struct crg_udc_request, queue);
+
+ if (next_req && next_req->first_trb)
+ udc_ep_ptr->deq_pt = next_req->first_trb;
+ else
+ udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
+
+ if (crg_udc_epcx_update_dqptr(udc_ep_ptr)) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EIO;
+ }
+ udc_ep_ptr->tran_ring_full = false;
+ crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
+ }
+
+ if (old_ep_state == EP_STATE_RUNNING && !crg_udc->resetting &&
+ udc_ep_ptr->desc == old_desc &&
+ event_generation == crg_udc->udc_event[0].generation &&
+ udc_ep_ptr->generation == generation &&
+ udc_ep_ptr->ep_state == EP_STATE_STOPPED) {
+ udc_ep_ptr->ep_state = EP_STATE_RUNNING;
+ udc_ep_ptr->tran_ring_full = false;
+ if (crg_udc_queue_pending_trbs(udc_ep_ptr))
+ udc_ep_ptr->tran_ring_full = true;
+ crg_udc_knock_doorbell(crg_udc, dci);
+ }
+
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return 0;
+}
+
+/* endpoint operations */
+
+static int crg_udc_ep_disable(struct usb_ep *ep)
+{
+ struct crg_udc_ep *udc_ep;
+ struct crg_gadget_dev *crg_udc;
+ struct crg_ep_cx *p_ep_cx;
+ struct crg_uccr __iomem *uccr;
+ unsigned long flags;
+ u32 running;
+ int ret;
+
+ if (!ep)
+ return -EINVAL;
+ udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+ crg_udc = udc_ep->crg_udc;
+ if (udc_ep->dci == 0)
+ return 0;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (crg_udc->resetting) {
+ ret = -ESHUTDOWN;
+ goto out;
+ }
+ if (udc_ep->configuring) {
+ ret = -EBUSY;
+ goto out;
+ }
+ if (udc_ep->ep_state == EP_STATE_DISABLED) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ uccr = crg_udc->uccr;
+ p_ep_cx = &crg_udc->p_epcx[udc_ep->dci - 2];
+ writel(BIT(udc_ep->dci), &uccr->ep_enable);
+ /* Do not unmap request buffers while the controller still owns them. */
+ ret = readl_poll_timeout_atomic(&uccr->ep_enable, running,
+ !(running & BIT(udc_ep->dci)), 10, 10000);
+ if (ret)
+ goto out;
+
+ udc_ep->generation++;
+ udc_ep->desc = NULL;
+ udc_ep->comp_desc = NULL;
+ udc_ep->ep_state = EP_STATE_DISABLED;
+ udc_ep->usb_ep.enabled = 0;
+ udc_ep->tran_ring_full = false;
+ if (crg_udc->num_enabled_eps)
+ crg_udc->num_enabled_eps--;
+ memset(p_ep_cx, 0, sizeof(*p_ep_cx));
+
+ if (!crg_udc->num_enabled_eps &&
+ crg_udc->device_state == USB_STATE_CONFIGURED)
+ crg_udc->device_state = USB_STATE_ADDRESS;
+
+ /* crg_udc_req_done() drops the lock during giveback. All endpoint
+ * state has to be finalized before that can happen.
+ */
+ crg_udc_ep_nuke(udc_ep, -ESHUTDOWN);
+ ret = 0;
+out:
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return ret;
+}
+
+static int crg_udc_ep_enable(struct usb_ep *ep,
+ const struct usb_endpoint_descriptor *desc)
+{
+ struct crg_udc_ep *udc_ep;
+ struct crg_gadget_dev *crg_udc;
+ struct crg_transfer_trb *new_ring = NULL;
+ void *old_ring = NULL;
+ dma_addr_t new_dma = 0, old_dma = 0;
+ size_t len, old_len = 0;
+ unsigned long flags;
+ u32 ring_size, param0;
+ int ret;
+
+ if (!ep || !desc || desc->bDescriptorType != USB_DT_ENDPOINT)
+ return -EINVAL;
+ udc_ep = container_of(ep, struct crg_udc_ep, usb_ep);
+ if (udc_ep->dci == 0)
+ return 0;
+ crg_udc = udc_ep->crg_udc;
+
+ if (usb_endpoint_xfer_bulk(desc))
+ ring_size = CRG_UDC_BULK_EP_TD_RING_SIZE;
+ else if (usb_endpoint_xfer_isoc(desc))
+ ring_size = CRG_UDC_ISOC_EP_TD_RING_SIZE;
+ else if (usb_endpoint_xfer_int(desc))
+ ring_size = CRG_UDC_INT_EP_TD_RING_SIZE;
+ else
+ return -EINVAL;
+ len = ring_size * sizeof(struct crg_transfer_trb);
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (!crg_udc->gadget_driver || crg_udc->resetting) {
+ ret = -ESHUTDOWN;
+ goto out;
+ }
+ if (udc_ep->configuring || udc_ep->ep_state != EP_STATE_DISABLED) {
+ ret = -EBUSY;
+ goto out;
+ }
+ udc_ep->configuring = true;
+ udc_ep->desc = desc;
+ udc_ep->comp_desc = ep->comp_desc;
+ ret = crg_udc_validate_ep_desc(udc_ep, desc);
+ if (ret)
+ goto err;
+
+ /* Leave the old allocation valid until its replacement is ready.
+ * Other ep operations reject the configuring endpoint.
+ */
+ if (!udc_ep->tran_ring_info.cpu_addr ||
+ udc_ep->tran_ring_info.len != len) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ new_ring = dma_alloc_coherent(crg_udc->dev, len, &new_dma,
+ GFP_KERNEL);
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (!new_ring) {
+ ret = -ENOMEM;
+ goto err;
+ }
+ if (!crg_udc->gadget_driver || crg_udc->resetting ||
+ udc_ep->ep_state != EP_STATE_DISABLED ||
+ udc_ep->desc != desc) {
+ ret = -ESHUTDOWN;
+ goto err;
+ }
+ old_ring = udc_ep->tran_ring_info.cpu_addr;
+ old_dma = udc_ep->tran_ring_info.dma_addr;
+ old_len = udc_ep->tran_ring_info.len;
+ udc_ep->tran_ring_info.cpu_addr = new_ring;
+ udc_ep->tran_ring_info.dma_addr = new_dma;
+ udc_ep->tran_ring_info.len = len;
+ new_ring = NULL;
+ }
+
+ udc_ep->first_trb = udc_ep->tran_ring_info.cpu_addr;
+ udc_ep->last_trb = udc_ep->first_trb + ring_size - 1;
+ memset(udc_ep->first_trb, 0, len);
+ dma_wmb();
+ crg_udc_setup_link_trb(udc_ep->last_trb, true,
+ udc_ep->tran_ring_info.dma_addr);
+ udc_ep->enq_pt = udc_ep->first_trb;
+ udc_ep->deq_pt = udc_ep->first_trb;
+ udc_ep->pcs = 1;
+ udc_ep->tran_ring_full = false;
+ ret = crg_udc_epcx_setup(udc_ep);
+ if (ret)
+ goto err;
+ param0 = BIT(udc_ep->dci);
+ ret = crg_udc_issue_command(crg_udc, CRG_CMD_CONFIG_EP, param0, 0);
+ if (ret)
+ goto err;
+
+ udc_ep->generation++;
+ udc_ep->wedge = false;
+ udc_ep->usb_ep.enabled = 1;
+ udc_ep->ep_state = EP_STATE_RUNNING;
+ udc_ep->configuring = false;
+ crg_udc->num_enabled_eps++;
+ if (crg_udc->device_state == USB_STATE_ADDRESS)
+ crg_udc->device_state = USB_STATE_CONFIGURED;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ if (old_ring)
+ dma_free_coherent(crg_udc->dev, old_len, old_ring, old_dma);
+ return 0;
+
+err:
+ udc_ep->desc = NULL;
+ udc_ep->comp_desc = NULL;
+ udc_ep->configuring = false;
+ /* Keep a valid ring allocation for a later retry. */
+ udc_ep->ep_state = EP_STATE_DISABLED;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ if (new_ring)
+ dma_free_coherent(crg_udc->dev, len, new_ring, new_dma);
+ if (old_ring)
+ dma_free_coherent(crg_udc->dev, old_len, old_ring, old_dma);
+ return ret;
+out:
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return ret;
+}
+
+void crg_udc_ep_nuke(struct crg_udc_ep *udc_ep, int status)
+{
+ struct crg_udc_request *req;
+ struct list_head nuke_list;
+
+ list_replace_init(&udc_ep->queue, &nuke_list);
+
+ while (!list_empty(&nuke_list)) {
+ req = list_first_entry(&nuke_list, struct crg_udc_request,
+ queue);
+
+ crg_udc_req_done(udc_ep, req, status);
+ }
+}
+
+static int crg_udc_ep_queue(struct usb_ep *_ep, struct usb_request *_req,
+ gfp_t gfp_flags)
+{
+ struct crg_udc_request *udc_req_ptr;
+ struct crg_udc_ep *udc_ep_ptr;
+ struct crg_gadget_dev *crg_udc;
+ enum dma_data_direction dma_data_dir;
+ const struct usb_endpoint_descriptor *desc;
+ unsigned long flags;
+ int status;
+ bool is_sg;
+ u8 setup_tag = 0;
+ u32 generation = 0;
+
+ if (!_ep || !_req)
+ return -EINVAL;
+
+ udc_req_ptr = container_of(_req, struct crg_udc_request, usb_req);
+ udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+ crg_udc = udc_ep_ptr->crg_udc;
+ is_sg = _req->num_sgs != 0 && _req->length != 0;
+
+ if (is_sg && !_req->sg)
+ return -EINVAL;
+ if (_req->stream_id)
+ return -EOPNOTSUPP;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ /*
+ * A completion callback for an aborted EP0 transfer may requeue the
+ * same request. Reject EP0 requests while the previous control
+ * transfer is being aborted or EP0 is in an error state. EP0 becomes
+ * available again only after the new SETUP has been accepted and
+ * the EP0 transfer ring has been successfully reinitialized.
+ */
+ if (udc_ep_ptr->dci == 0 && crg_udc->ep0_state != EP0_STATE_IDLE) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -ECONNRESET;
+ }
+ if (!udc_ep_ptr->first_trb || !_req->complete ||
+ (!is_sg && !_req->buf) || !list_empty(&udc_req_ptr->queue)) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EINVAL;
+ }
+ if (crg_udc->resetting) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -ESHUTDOWN;
+ }
+ if (!udc_ep_ptr->desc) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EINVAL;
+ }
+ desc = udc_ep_ptr->desc;
+ if (udc_ep_ptr->ep_state != EP_STATE_RUNNING ||
+ udc_ep_ptr->configuring) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -ESHUTDOWN;
+ }
+ setup_tag = crg_udc->setup_tag;
+ generation = udc_ep_ptr->generation;
+
+ if (usb_endpoint_xfer_control(udc_ep_ptr->desc) && is_sg) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EOPNOTSUPP;
+ }
+
+ if (usb_endpoint_xfer_control(udc_ep_ptr->desc) && !_req->length) {
+ if (crg_udc->setup_status == DATA_STAGE_XFER)
+ crg_udc->setup_status = STATUS_STAGE_RECV;
+ else
+ crg_udc->setup_status = STATUS_STAGE_XFER;
+ crg_udc_build_ep0_status(&crg_udc->udc_ep[0], false,
+ -EINPROGRESS, udc_req_ptr, 0, 0);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return 0;
+ }
+
+ if (usb_endpoint_xfer_control(udc_ep_ptr->desc)) {
+ if (crg_udc->setup_status == DATA_STAGE_XFER ||
+ crg_udc->setup_status == STATUS_STAGE_XFER) {
+ dma_data_dir = DMA_TO_DEVICE;
+ } else if (crg_udc->setup_status == DATA_STAGE_RECV ||
+ crg_udc->setup_status == STATUS_STAGE_RECV) {
+ dma_data_dir = DMA_FROM_DEVICE;
+ } else {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EINVAL;
+ }
+ } else {
+ dma_data_dir = usb_endpoint_dir_in(udc_ep_ptr->desc) ?
+ DMA_TO_DEVICE :
+ DMA_FROM_DEVICE;
+ }
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ crg_udc_clear_req_container(udc_req_ptr);
+ udc_req_ptr->mapped = false;
+
+ if (_req->length) {
+ udc_req_ptr->map_dir = dma_data_dir;
+ status = usb_gadget_map_request_by_dev(
+ crg_udc->dev, _req, dma_data_dir == DMA_TO_DEVICE);
+ if (status)
+ goto err_unmap;
+
+ if (is_sg) {
+ if (!_req->num_mapped_sgs ||
+ crg_udc_sg_init(udc_req_ptr)) {
+ status = -EIO;
+ goto err_unmap;
+ }
+ } else if (_req->dma_mapped) {
+ udc_req_ptr->mapped = true;
+ }
+ }
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (udc_ep_ptr->desc != desc || !udc_req_ptr->usb_req.complete ||
+ !list_empty(&udc_req_ptr->queue) || crg_udc->resetting ||
+ udc_ep_ptr->configuring ||
+ udc_ep_ptr->ep_state != EP_STATE_RUNNING ||
+ udc_ep_ptr->generation != generation ||
+ (udc_ep_ptr->dci == 0 &&
+ (crg_udc->ep0_state != EP0_STATE_IDLE ||
+ crg_udc->setup_tag != setup_tag))) {
+ status = -ESHUTDOWN;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr);
+ return status;
+ }
+
+ _req->status = -EINPROGRESS;
+ _req->actual = 0;
+
+ /* Do not queue a newer request ahead of a partially queued one. */
+ if (!list_empty(&udc_ep_ptr->queue)) {
+ struct crg_udc_request *tail;
+
+ tail = list_last_entry(&udc_ep_ptr->queue,
+ struct crg_udc_request, queue);
+ status = (!tail->all_trbs_queued ||
+ udc_ep_ptr->tran_ring_full) ? 0 :
+ crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+ } else if (udc_ep_ptr->tran_ring_full) {
+ status = 0;
+ } else {
+ status = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+ }
+
+ if (!status)
+ list_add_tail(&udc_req_ptr->queue, &udc_ep_ptr->queue);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ if (status)
+ crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr);
+ return status;
+
+err_unmap:
+ crg_udc_unmap_request(udc_ep_ptr, udc_req_ptr);
+ return status;
+}
+
+static int crg_udc_ep_set_halt(struct usb_ep *_ep, int value)
+{
+ struct crg_udc_ep *udc_ep_ptr;
+ int status;
+ unsigned long flags = 0;
+ struct crg_gadget_dev *crg_udc;
+
+ if (!_ep)
+ return -EINVAL;
+
+ udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+ crg_udc = udc_ep_ptr->crg_udc;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (!udc_ep_ptr->desc) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EINVAL;
+ }
+ if (value && usb_endpoint_dir_in(udc_ep_ptr->desc) &&
+ !list_empty(&udc_ep_ptr->queue)) {
+ dev_err(crg_udc->dev, "set_halt: list not empty\n");
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EAGAIN;
+ }
+
+ status = crg_udc_ep_halt(udc_ep_ptr, value, 1, &flags);
+
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ return status;
+}
+
+static int crg_udc_ep_set_wedge(struct usb_ep *_ep)
+{
+ struct crg_udc_ep *udc_ep_ptr;
+ int status;
+ unsigned long flags = 0;
+ struct crg_gadget_dev *crg_udc;
+
+ if (!_ep)
+ return -EINVAL;
+
+ udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
+ crg_udc = udc_ep_ptr->crg_udc;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+
+ if (!udc_ep_ptr->desc) {
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return -EINVAL;
+ }
+
+ udc_ep_ptr->wedge = 1;
+
+ status = crg_udc_ep_halt(udc_ep_ptr, 1, 1, &flags);
+
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ return status;
+}
+
+static int crg_udc_epcx_setup(struct crg_udc_ep *udc_ep)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+ const struct usb_endpoint_descriptor *desc = udc_ep->desc;
+ const struct usb_ss_ep_comp_descriptor *comp_desc = udc_ep->comp_desc;
+ u8 dci = udc_ep->dci;
+ struct crg_ep_cx *epcx = &crg_udc->p_epcx[dci - 2];
+ enum crg_ep_type ep_type;
+ u16 maxburst = 0;
+ u8 maxstreams = 0;
+ u16 maxsize;
+ u32 dw;
+
+ /* corigine gadget dir should be opposite to host dir */
+ if (usb_endpoint_dir_out(desc))
+ ep_type = usb_endpoint_type(desc) + EP_TYPE_INVALID2;
+ else
+ ep_type = usb_endpoint_type(desc);
+
+ maxsize = usb_endpoint_maxp(desc) & 0x07ff;
+
+ if (crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+ if (!comp_desc)
+ return -EINVAL;
+ maxburst = comp_desc->bMaxBurst;
+
+ if (usb_endpoint_xfer_bulk(udc_ep->desc))
+ maxstreams = comp_desc->bmAttributes & 0x1f;
+
+ } else if ((crg_udc->gadget.speed == USB_SPEED_HIGH ||
+ crg_udc->gadget.speed == USB_SPEED_FULL) &&
+ (usb_endpoint_xfer_int(udc_ep->desc) ||
+ usb_endpoint_xfer_isoc(udc_ep->desc))) {
+ if (crg_udc->gadget.speed == USB_SPEED_HIGH)
+ maxburst = usb_endpoint_maxp_mult(desc) - 1;
+ if (maxburst == 0x3) {
+ dev_err(crg_udc->dev, "invalid maxburst\n");
+ maxburst = 0x2;
+ }
+ }
+
+ /* fill ep_dw0 */
+ dw = 0;
+ FIELD_MODIFY(EP_CX_LOGICAL_EP_NUM_MASK, &dw, udc_ep->dci / 2);
+ FIELD_MODIFY(EP_CX_INTERVAL_MASK, &dw, desc->bInterval);
+ if (maxstreams) {
+ dev_err(crg_udc->dev, "maxstream=%d is not expected\n",
+ maxstreams);
+ }
+ epcx->dw0 = cpu_to_le32(dw);
+
+ /* fill ep_dw1 */
+ dw = 0;
+ FIELD_MODIFY(EP_CX_EP_TYPE_MASK, &dw, ep_type);
+ FIELD_MODIFY(EP_CX_MAX_PACKET_SIZE_MASK, &dw, maxsize);
+ FIELD_MODIFY(EP_CX_MAX_BURST_SIZE_MASK, &dw, maxburst);
+ epcx->dw1 = cpu_to_le32(dw);
+
+ /* fill ep_dw2 */
+ dw = lower_32_bits(udc_ep->tran_ring_info.dma_addr) &
+ EP_CX_TR_DQPT_LO_MASK;
+ FIELD_MODIFY(EP_CX_DEQ_CYC_STATE_MASK, &dw, udc_ep->pcs);
+ epcx->dw2 = cpu_to_le32(dw);
+
+ /* fill ep_dw3 */
+ dw = upper_32_bits(udc_ep->tran_ring_info.dma_addr);
+ epcx->dw3 = cpu_to_le32(dw);
+ /* Ensure that epcx is updated */
+ dma_wmb();
+ return 0;
+}
+
+static int crg_udc_epcx_update_dqptr(struct crg_udc_ep *udc_ep)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+ u8 dci = udc_ep->dci;
+ struct crg_ep_cx *epcx;
+ u32 dw;
+ dma_addr_t dqptaddr;
+ u32 cmd_param0;
+
+ if (dci < 2 || dci >= CRG_NUM_EP_CX) {
+ dev_err(crg_udc->dev, "invalid DCI %u for dequeue pointer\n", dci);
+ return -EINVAL;
+ }
+
+ epcx = &crg_udc->p_epcx[dci - 2];
+ dqptaddr = crg_udc_tran_trb_virt_to_dma(udc_ep, udc_ep->deq_pt);
+
+ if (dqptaddr == DMA_MAPPING_ERROR)
+ return -EINVAL;
+
+ /* fill ep_dw2 */
+ dw = lower_32_bits(dqptaddr) & EP_CX_TR_DQPT_LO_MASK;
+ /* The consumer cycle can lag the producer by one ring wrap. */
+ FIELD_MODIFY(EP_CX_DEQ_CYC_STATE_MASK, &dw,
+ udc_ep->deq_pt > udc_ep->enq_pt ?
+ udc_ep->pcs ^ 1 : udc_ep->pcs);
+ epcx->dw2 = cpu_to_le32(dw);
+
+ /* fill ep_dw3 */
+ dw = upper_32_bits(dqptaddr);
+ epcx->dw3 = cpu_to_le32(dw);
+
+ cmd_param0 = BIT(udc_ep->dci);
+ /* Ensure that dqptr is updated */
+ dma_wmb();
+
+ return crg_udc_issue_command(crg_udc, CRG_CMD_SET_TR_DQPTR, cmd_param0, 0);
+}
+
+void crg_udc_free_request(struct usb_ep *_ep, struct usb_request *_req)
+{
+ struct crg_udc_request *udc_req_ptr;
+
+ if (!_ep || !_req)
+ return;
+
+ udc_req_ptr = container_of(_req, struct crg_udc_request, usb_req);
+
+ trace_crg_udc_free_request(container_of(_ep, struct crg_udc_ep, usb_ep),
+ udc_req_ptr);
+
+ if (udc_req_ptr->mapped || udc_req_ptr->usb_req.num_mapped_sgs)
+ crg_udc_unmap_request(container_of(_ep, struct crg_udc_ep,
+ usb_ep),
+ udc_req_ptr);
+ kfree(udc_req_ptr);
+}
+
+static int crg_udc_queue_trbs(struct crg_udc_ep *udc_ep_ptr,
+ struct crg_udc_request *udc_req_ptr, bool b_isoc,
+ u32 xfer_ring_size, u32 num_trbs_needed,
+ u64 buffer_length)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+ struct crg_transfer_trb *p_xfer_ring = udc_ep_ptr->first_trb;
+ struct crg_transfer_trb *enq_pt = udc_ep_ptr->enq_pt;
+ struct usb_request *usb_req = &udc_req_ptr->usb_req;
+ u32 num_trbs_ava, count, i;
+ u32 td_trbs;
+ u32 td_size;
+ u32 j = 1;
+ u32 intr_rate = b_isoc ? CRG_UDC_ISOC_EP_INTERRUPT_RATE :
+ CRG_UDC_BULK_EP_INTERRUPT_RATE;
+ bool full_td = true;
+ bool need_zlp = false;
+ bool is_sg = usb_req->num_mapped_sgs;
+
+ if (!b_isoc && usb_req->zero && usb_req->length &&
+ !(usb_req->length % udc_ep_ptr->usb_ep.maxpacket))
+ need_zlp = true;
+
+ if (!num_trbs_needed)
+ return -EINVAL;
+
+ /* TD_SIZE is five bits and encodes the number of TRBs remaining
+ * in this TD minus one. A request may contain multiple TDs.
+ */
+ td_trbs = min_t(u32, num_trbs_needed, 32);
+
+ num_trbs_ava = crg_udc_room_on_ring(crg_udc, xfer_ring_size,
+ p_xfer_ring, udc_ep_ptr->enq_pt,
+ udc_ep_ptr->deq_pt);
+
+ if (num_trbs_ava < td_trbs) {
+ if (b_isoc)
+ return -EBUSY;
+ trace_crg_udc_no_room_on_ring(
+ "deferring TRBs to next doorbell");
+ count = num_trbs_ava;
+ full_td = false;
+ udc_ep_ptr->tran_ring_full = true;
+ need_zlp = false;
+ } else {
+ count = td_trbs;
+ }
+
+ if (!count)
+ return -ENOSPC;
+
+ td_size = count - 1;
+
+ if (is_sg && !udc_req_ptr->sg && crg_udc_sg_init(udc_req_ptr))
+ return -EINVAL;
+
+ u8 original_pcs = udc_ep_ptr->pcs;
+
+ for (i = 0; i < count; i++) {
+ dma_addr_t trb_buf_addr;
+ u32 trb_len, sg_len = 0;
+ bool chain_bit = true;
+ bool ioc = false;
+ bool short_pkt = usb_endpoint_dir_out(udc_ep_ptr->desc);
+
+ if (is_sg) {
+ if (!crg_udc_sg_current(udc_req_ptr, &trb_buf_addr,
+ &sg_len))
+ return -EINVAL;
+ if (!sg_len ||
+ dma_mapping_error(crg_udc->dev, trb_buf_addr))
+ return -EINVAL;
+ trb_len = min_t(
+ u64, buffer_length,
+ min_t(u32, sg_len,
+ crg_udc_trb_chunk_len(trb_buf_addr,
+ buffer_length)));
+ } else {
+ if (buffer_length)
+ trb_buf_addr = usb_req->dma + (usb_req->length -
+ buffer_length);
+ else
+ trb_buf_addr = 0;
+ trb_len = crg_udc_trb_chunk_len(trb_buf_addr,
+ buffer_length);
+ }
+
+ if (!trb_len && buffer_length)
+ return -EINVAL;
+
+ buffer_length -= trb_len;
+
+ if (i == count - 1) {
+ chain_bit = false;
+ ioc = true;
+ if (!buffer_length)
+ udc_req_ptr->all_trbs_queued = true;
+ }
+
+ if (!full_td && j == intr_rate) {
+ ioc = true;
+ j = 0;
+ }
+
+ if (b_isoc) {
+ crg_udc_setup_trb(
+ enq_pt, trb_len, trb_buf_addr, td_size,
+ i == 0 ? (original_pcs ^ 1) : udc_ep_ptr->pcs,
+ TRB_TYPE_XFER_DATA_ISOCH, short_pkt, chain_bit,
+ ioc, false, 0, true, 0, 1, 0);
+ } else {
+ bool azp = ioc && !chain_bit && !buffer_length && need_zlp;
+
+ crg_udc_setup_trb(
+ enq_pt, trb_len, trb_buf_addr, td_size,
+ i == 0 ? (original_pcs ^ 1) : udc_ep_ptr->pcs,
+ TRB_TYPE_XFER_NORMAL, short_pkt, chain_bit, ioc,
+ false, 0, false, 0, 0, azp);
+ }
+
+ trace_crg_udc_queue_trb(udc_ep_ptr, enq_pt);
+
+ if (is_sg && !crg_udc_sg_advance(udc_req_ptr, trb_len))
+ return -EINVAL;
+
+ td_size--;
+ enq_pt++;
+ j++;
+
+ if (enq_pt == udc_ep_ptr->last_trb) {
+ dma_wmb();
+ le32p_replace_bits(&enq_pt->dw3, udc_ep_ptr->pcs,
+ TRB_CYCLE_BIT_MASK);
+ udc_ep_ptr->pcs ^= 1;
+ enq_pt = udc_ep_ptr->first_trb;
+ }
+ }
+
+ dma_wmb();
+ le32p_replace_bits(&udc_ep_ptr->enq_pt->dw3, original_pcs,
+ TRB_CYCLE_BIT_MASK);
+
+ if (!udc_req_ptr->trbs_needed)
+ udc_req_ptr->first_trb = udc_ep_ptr->enq_pt;
+
+ udc_ep_ptr->enq_pt = enq_pt;
+ if (count == num_trbs_ava)
+ udc_ep_ptr->tran_ring_full = true;
+ udc_req_ptr->buff_len_left = buffer_length;
+ udc_req_ptr->td_pending = true;
+ if (buffer_length)
+ udc_req_ptr->trbs_needed = num_trbs_needed - count;
+ else
+ udc_req_ptr->trbs_needed = 0;
+
+ /* Track the last TRB of the current TD, even for multi-TD requests. */
+ if (udc_ep_ptr->enq_pt == udc_ep_ptr->first_trb)
+ udc_req_ptr->last_trb = udc_ep_ptr->last_trb - 1;
+ else
+ udc_req_ptr->last_trb = udc_ep_ptr->enq_pt - 1;
+
+ return 0;
+}
+
+static int crg_udc_reinit(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 i, tmp;
+ struct crg_udc_ep *udc_ep_ptr;
+
+ crg_udc->setup_status = WAIT_FOR_SETUP;
+ /* Base on Figure 9-1, default USB_STATE is attached */
+ crg_udc->device_state = USB_STATE_RECONNECTING;
+
+ /* halt all the endpoints */
+
+ /* disable all the endpoints */
+ tmp = readl(&uccr->ep_enable);
+ writel(tmp, &uccr->ep_enable);
+ if (readl_poll_timeout_atomic(&uccr->ep_enable, tmp, !tmp, 1, 50)) {
+ dev_dbg(crg_udc->dev,
+ "timeout waiting for all endpoints to disable\n");
+ return -ETIMEDOUT;
+ }
+
+ for (i = 2; i < CRG_NUM_EP_CX; i++) {
+ udc_ep_ptr = &crg_udc->udc_ep[i];
+ udc_ep_ptr->usb_ep.enabled = 0;
+ udc_ep_ptr->generation++;
+ udc_ep_ptr->ep_state = EP_STATE_DISABLED;
+ if (udc_ep_ptr->desc) {
+ udc_ep_ptr->desc = NULL;
+ udc_ep_ptr->comp_desc = NULL;
+ crg_udc_ep_nuke(udc_ep_ptr, -ESHUTDOWN);
+ }
+ udc_ep_ptr->tran_ring_full = false;
+ udc_ep_ptr->ep_state = EP_STATE_DISABLED;
+ }
+ crg_udc->num_enabled_eps = 0;
+
+ /* we don't handle ep0 here, we crg_udc_init_ep0 when event ring is empty */
+
+ if (crg_udc->dev_addr != 0) {
+ u32 param0;
+
+ param0 = FIELD_PREP(CRG_CMD2_0_DEV_ADDR_MASK, 0);
+ if (crg_udc_issue_command(crg_udc, CRG_CMD_SET_ADDR, param0, 0))
+ return -EIO;
+ crg_udc->dev_addr = 0;
+ }
+
+ crg_udc_clear_portpm(crg_udc);
+
+ return 0;
+}
+
+void crg_udc_set_pullup(struct crg_gadget_dev *crg_udc, bool is_on)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 val;
+
+ if (is_on) {
+ /* Enable event reporting and interrupts, then assert RUN */
+ val = readl(&uccr->config1);
+ val |= (CRG_U3DC_CFG1_CSC_EVENT_EN |
+ CRG_U3DC_CFG1_PEC_EVENT_EN |
+ CRG_U3DC_CFG1_PPC_EVENT_EN |
+ CRG_U3DC_CFG1_PRC_EVENT_EN |
+ CRG_U3DC_CFG1_PLC_EVENT_EN |
+ CRG_U3DC_CFG1_CEC_EVENT_EN);
+ writel(val, &uccr->config1);
+
+ val = readl(&uccr->control);
+ val |= (CRG_U3DC_CTRL_SYSERR_EN | CRG_U3DC_CTRL_INT_EN |
+ CRG_U3DC_CTRL_RUN);
+ writel(val, &uccr->control);
+ } else {
+ val = readl(&uccr->control);
+ val &= ~CRG_U3DC_CTRL_RUN;
+ writel(val, &uccr->control);
+ }
+}
+
+void crg_udc_set_vbus_detect(struct crg_gadget_dev *crg_udc, bool enable)
+{
+ u32 val = readl(crg_udc->mmio_virt_base + CRG_UDC_VENDOR_REG);
+
+ if (enable)
+ val |= CRG_UDC_VBUS_DETECT;
+ else
+ val &= ~CRG_UDC_VBUS_DETECT;
+
+ writel(val, crg_udc->mmio_virt_base + CRG_UDC_VENDOR_REG);
+}
+
+static bool crg_udc_sg_advance(struct crg_udc_request *udc_req, u32 bytes)
+{
+ struct scatterlist *sg = udc_req->sg;
+ unsigned int i = udc_req->sg_index;
+
+ while (bytes) {
+ u32 len;
+
+ if (!sg || i >= udc_req->usb_req.num_sgs)
+ return false;
+
+ len = sg_dma_len(sg);
+ if (!len || udc_req->sg_offset == len) {
+ sg = sg_next(sg);
+ i++;
+ udc_req->sg_offset = 0;
+ continue;
+ }
+
+ len -= udc_req->sg_offset;
+ if (bytes < len) {
+ udc_req->sg_offset += bytes;
+ bytes = 0;
+ break;
+ }
+
+ bytes -= len;
+ sg = sg_next(sg);
+ i++;
+ udc_req->sg_offset = 0;
+ }
+
+ udc_req->sg = sg;
+ udc_req->sg_index = i;
+ return true;
+}
+
+static bool crg_udc_sg_current(struct crg_udc_request *udc_req, dma_addr_t *dma,
+ u32 *len)
+{
+ struct scatterlist *sg = udc_req->sg;
+
+ while (sg && udc_req->sg_index < udc_req->usb_req.num_sgs &&
+ !sg_dma_len(sg)) {
+ sg = sg_next(sg);
+ udc_req->sg_index++;
+ udc_req->sg_offset = 0;
+ }
+ udc_req->sg = sg;
+
+ if (!sg || udc_req->sg_index >= udc_req->usb_req.num_sgs)
+ return false;
+
+ if (udc_req->sg_offset >= sg_dma_len(sg))
+ return false;
+
+ *dma = sg_dma_address(sg) + udc_req->sg_offset;
+ *len = sg_dma_len(sg) - udc_req->sg_offset;
+ return true;
+}
+
+static int crg_udc_sg_init(struct crg_udc_request *udc_req)
+{
+ struct scatterlist *sg;
+ unsigned int i;
+
+ for_each_sg(udc_req->usb_req.sg, sg, udc_req->usb_req.num_sgs, i) {
+ if (sg_dma_len(sg)) {
+ udc_req->sg = sg;
+ udc_req->sg_index = i;
+ udc_req->sg_offset = 0;
+ return 0;
+ }
+ }
+
+ return -EINVAL;
+}
+
+/*
+ * Undo whatever crg_udc_ep_queue() mapped for this request: the merged
+ * SG list, the single buffer, or neither if the request carried a
+ * pre-mapped DMA address.
+ */
+void crg_udc_unmap_request(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+ struct usb_request *req = &udc_req->usb_req;
+ bool is_in = udc_req->map_dir == DMA_TO_DEVICE;
+
+ if (req->num_mapped_sgs || req->dma_mapped)
+ usb_gadget_unmap_request_by_dev(crg_udc->dev, req, is_in);
+
+ if (udc_req->mapped) {
+ req->dma = DMA_MAPPING_ERROR;
+ udc_req->mapped = false;
+ }
+
+ udc_req->sg = NULL;
+ udc_req->sg_index = 0;
+ udc_req->sg_offset = 0;
+}
+
+static int crg_udc_validate_ep_desc(struct crg_udc_ep *udc_ep,
+ const struct usb_endpoint_descriptor *desc)
+{
+ const struct usb_ss_ep_comp_descriptor *comp = udc_ep->comp_desc;
+ u16 maxpacket = usb_endpoint_maxp(desc);
+
+ if (usb_endpoint_xfer_control(desc) || !maxpacket ||
+ maxpacket > CRG_MAX_PACKET_SIZE)
+ return -EINVAL;
+
+ if (crg_udc_index2dci(desc->bEndpointAddress) != udc_ep->dci)
+ return -EINVAL;
+
+ if (usb_endpoint_xfer_bulk(desc) && comp && usb_ss_max_streams(comp))
+ return -EOPNOTSUPP;
+
+ if (usb_endpoint_xfer_isoc(desc) && desc->bInterval == 0)
+ return -EINVAL;
+
+ if (udc_ep->crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+ if (!comp || comp->bMaxBurst > 15)
+ return -EINVAL;
+ if (usb_endpoint_xfer_isoc(desc) &&
+ ((comp->bmAttributes & 0x3) > 2))
+ return -EINVAL;
+ } else if (udc_ep->crg_udc->gadget.speed == USB_SPEED_HIGH) {
+ if (desc->bInterval > 16 && (usb_endpoint_xfer_int(desc) ||
+ usb_endpoint_xfer_isoc(desc)))
+ return -EINVAL;
+ if (usb_endpoint_xfer_isoc(desc) &&
+ usb_endpoint_maxp_mult(desc) > 3)
+ return -EINVAL;
+ } else if (udc_ep->crg_udc->gadget.speed == USB_SPEED_FULL) {
+ if (maxpacket > 64 && !usb_endpoint_xfer_isoc(desc))
+ return -EINVAL;
+ if (usb_endpoint_xfer_isoc(desc) && maxpacket > 1023)
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static bool crg_udc_vbus_connected(struct crg_gadget_dev *crg_udc)
+{
+ return !!(readl(&crg_udc->uccr->portsc) & CRG_U3DC_PORTSC_PP);
+}
+
+static void crg_ep_struct_setup(struct crg_gadget_dev *crg_udc, u32 dci,
+ const char *name)
+{
+ struct crg_udc_ep *ep = &crg_udc->udc_ep[dci];
+
+ ep->dci = dci;
+
+ if (ep->dci > 1) {
+ strscpy_pad(ep->name, name, sizeof(ep->name));
+ ep->usb_ep.name = ep->name;
+
+ ep->usb_ep.caps.type_iso = 1;
+ ep->usb_ep.caps.type_bulk = 1;
+ ep->usb_ep.caps.type_int = 1;
+ if (ep->dci % 2)
+ ep->usb_ep.caps.dir_out = 1;
+ else
+ ep->usb_ep.caps.dir_in = 1;
+ usb_ep_set_maxpacket_limit(&ep->usb_ep, CRG_MAX_PACKET_SIZE);
+ } else {
+ strscpy(ep->name, "ep0", sizeof(ep->name));
+ ep->usb_ep.name = ep->name;
+ ep->usb_ep.maxpacket = 512;
+ ep->usb_ep.caps.type_control = 1;
+ ep->usb_ep.caps.dir_in = 1;
+ ep->usb_ep.caps.dir_out = 1;
+ }
+
+ ep->usb_ep.ops = &crg_udc_ep_ops;
+ ep->crg_udc = crg_udc;
+
+ INIT_LIST_HEAD(&ep->queue);
+ if (ep->dci > 1)
+ list_add_tail(&ep->usb_ep.ep_list, &crg_udc->gadget.ep_list);
+}
+
+void crg_udc_reset_work(struct work_struct *work)
+{
+ struct crg_gadget_dev *crg_udc =
+ container_of(work, struct crg_gadget_dev, udc_reset_work);
+
+ struct usb_gadget_driver *driver;
+ unsigned long flags;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ driver = crg_udc->async_callbacks ? crg_udc->gadget_driver : NULL;
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ /* Disabling async callbacks drains this work before driver unbind. */
+ if (driver)
+ usb_gadget_udc_reset(&crg_udc->gadget, driver);
+}
+
+int crg_udc_handle_port_status(struct crg_gadget_dev *crg_udc,
+ unsigned long *flags)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ bool was_connected = crg_udc->connected;
+ u32 portsc_val;
+ u32 tmp;
+ u32 speed;
+ int ret;
+
+ /* handle Port Reset */
+ portsc_val = readl(&uccr->portsc);
+ trace_crg_udc_portsc(portsc_val);
+ writel(portsc_val & CRG_U3DC_PORTSC_W1C_MASK, &uccr->portsc);
+
+ tmp = readl(&uccr->portsc);
+
+ if (portsc_val & CRG_U3DC_PORTSC_PRC) {
+ tmp = readl(&uccr->portsc);
+ if (tmp & CRG_U3DC_PORTSC_PRC) {
+ dev_dbg(crg_udc->dev, "PRC is still set\n");
+ } else if (tmp & CRG_U3DC_PORTSC_PR) {
+ /* first port status change event for port reset */
+ dev_dbg(crg_udc->dev,
+ "PRC is not set, but PR is set\n");
+ } else {
+ if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) != 0 ||
+ (!(tmp & CRG_U3DC_PORTSC_PED))) {
+ dev_dbg(crg_udc->dev, "portsc=0x%x no PED\n",
+ tmp);
+ return 0;
+ }
+
+ switch (FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, tmp)) {
+ case CRG_U3DC_PORTSC_SPEED_SSP:
+ speed = USB_SPEED_SUPER_PLUS;
+ break;
+ case CRG_U3DC_PORTSC_SPEED_SS:
+ speed = USB_SPEED_SUPER;
+ break;
+ case CRG_U3DC_PORTSC_SPEED_HS:
+ speed = USB_SPEED_HIGH;
+ break;
+
+ case CRG_U3DC_PORTSC_SPEED_FS:
+ speed = USB_SPEED_FULL;
+ break;
+
+ case CRG_U3DC_PORTSC_SPEED_LS:
+ speed = USB_SPEED_LOW;
+ break;
+ default:
+ speed = USB_SPEED_UNKNOWN;
+ return 0;
+ }
+
+ if (crg_udc->device_state >= USB_STATE_DEFAULT) {
+ /*
+ * A single enumeration commonly involves two
+ * bus resets (an initial one, then another
+ * after SET_ADDRESS before SET_CONFIGURATION).
+ * By the second reset device_state is already
+ * USB_STATE_ADDRESS, which is >= DEFAULT but
+ * has no active configuration to tear down.
+ * Only notify composite when a configuration
+ * was actually active: otherwise this races
+ * with the SET_CONFIGURATION that's about to
+ * arrive, since the deferred work below can
+ * run after fsg_set_alt() has already raised
+ * its own FSG_STATE_CONFIG_CHANGE exception
+ * and reset_config()'s ->disable() would
+ * overwrite that exception's argument back to
+ * NULL before the function's thread consumes
+ * it, silently skipping the real enable.
+ */
+ bool was_configured = crg_udc->device_state ==
+ USB_STATE_CONFIGURED;
+ int reinit_ret;
+
+ reinit_ret = crg_udc_reinit(crg_udc);
+ if (reinit_ret) {
+ dev_dbg(crg_udc->dev,
+ "bus reset reinit failed: %d\n",
+ reinit_ret);
+ return reinit_ret;
+ }
+
+ /*
+ * Notify the function/composite driver of the
+ * bus reset so it tears down the active
+ * configuration (usb_gadget_udc_reset() ->
+ * ->reset()). Without this, a host that
+ * re-sends the same SET_CONFIGURATION value
+ * after a reset can see composite treat it as
+ * a no-op and skip re-running set_alt(), even
+ * though every endpoint was just disabled in
+ * software above.
+ *
+ * This must not be called synchronously here:
+ * usb_gadget_udc_reset() re-enters the
+ * function drivers, which requires dropping
+ * udc_lock. Doing that mid-event-processing
+ * would let a second port-status event run
+ * crg_udc_reinit() again while this event's
+ * SET_CONFIGURATION/set_alt() work is still
+ * landing on another thread, tearing the
+ * just-enabled endpoints back down before
+ * they can be used. Defer it to a work item
+ * so this function never has to drop the
+ * lock.
+ */
+ if (was_configured && crg_udc->async_callbacks &&
+ crg_udc->gadget_driver)
+ schedule_work(&crg_udc->udc_reset_work);
+ }
+
+ crg_udc->gadget.speed = speed;
+
+ crg_udc_update_ep0_maxpacketsize(crg_udc);
+
+ crg_udc->connected = 1;
+
+ if (crg_udc->device_state < USB_STATE_RECONNECTING)
+ crg_udc_enable_setup(crg_udc);
+ }
+ }
+ /* handle Port Reset end */
+
+ /* Connection changes must not mask independent link-state events. */
+ if (portsc_val & CRG_U3DC_PORTSC_CSC) {
+ tmp = readl(&uccr->portsc);
+ if ((tmp & (CRG_U3DC_PORTSC_CCS | CRG_U3DC_PORTSC_PP)) ==
+ (CRG_U3DC_PORTSC_CCS | CRG_U3DC_PORTSC_PP)) {
+ if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) != 0 ||
+ (!(tmp & CRG_U3DC_PORTSC_PED)))
+ return 0;
+
+ switch (FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, tmp)) {
+ case CRG_U3DC_PORTSC_SPEED_SSP:
+ speed = USB_SPEED_SUPER_PLUS;
+ break;
+ case CRG_U3DC_PORTSC_SPEED_SS:
+ speed = USB_SPEED_SUPER;
+ break;
+ case CRG_U3DC_PORTSC_SPEED_HS:
+ speed = USB_SPEED_HIGH;
+ break;
+
+ case CRG_U3DC_PORTSC_SPEED_FS:
+ speed = USB_SPEED_FULL;
+ break;
+
+ case CRG_U3DC_PORTSC_SPEED_LS:
+ speed = USB_SPEED_LOW;
+ break;
+ default:
+ speed = USB_SPEED_UNKNOWN;
+ return 0;
+ }
+ crg_udc->gadget.speed = speed;
+
+ crg_udc_update_ep0_maxpacketsize(crg_udc);
+
+ crg_udc->connected = 1;
+
+ if (crg_udc->device_state < USB_STATE_RECONNECTING)
+ crg_udc_enable_setup(crg_udc);
+
+ } else if (!(tmp & CRG_U3DC_PORTSC_CCS)) {
+ bool cable_connected;
+ int ccs_drop_ignore = 0;
+
+ if ((FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) == 0x0) &&
+ (FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, tmp) <
+ CRG_U3DC_PORTSC_SPEED_SS)) {
+ ccs_drop_ignore = 1;
+ dev_err(crg_udc->dev, "ccs glitch detect on HS/FS!\n");
+ }
+
+ if (!ccs_drop_ignore)
+ crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+
+ cable_connected = crg_udc_vbus_connected(crg_udc);
+ if (cable_connected && !ccs_drop_ignore) {
+ int ret;
+
+ if (was_connected)
+ crg_udc_notify_disconnect(crg_udc, flags);
+ ret = crg_udc_reinit(crg_udc);
+ if (ret)
+ return ret;
+ crg_udc->connected = 0;
+ } else if (!cable_connected) {
+ dev_dbg(crg_udc->dev,
+ "cable disconnected, rst controller\n");
+ if (was_connected)
+ crg_udc_notify_disconnect(crg_udc, flags);
+ ret = crg_udc_reset(crg_udc, flags);
+ if (!ret)
+ ret = crg_udc_reset_data_struct(crg_udc);
+ if (!ret)
+ ret = crg_udc_init_ep0(crg_udc);
+ if (ret)
+ dev_err(crg_udc->dev,
+ "controller reset failed on disconnect: %d\n",
+ ret);
+ else
+ crg_udc->resetting = false;
+ crg_udc->connected = 0;
+
+ crg_udc->device_state = USB_STATE_ATTACHED;
+ return -ECONNRESET;
+ }
+ }
+ }
+ /* handle Port Connection Change end */
+
+ if (portsc_val & CRG_U3DC_PORTSC_PLC) {
+ tmp = readl(&uccr->portsc);
+ if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) == 0xf) {
+ tmp &= ~CRG_U3DC_PORTSC_W1C_MASK;
+ FIELD_MODIFY(CRG_U3DC_PORTSC_PLS_MASK, &tmp, 0x0);
+ tmp |= CRG_U3DC_PORTSC_LWS;
+ writel(tmp, &uccr->portsc);
+ }
+ }
+
+ return 0;
+}
+
+int crg_udc_handle_xfer_event(struct crg_gadget_dev *crg_udc,
+ struct crg_event_trb *event, unsigned long *flags)
+{
+ u32 dw3 = le32_to_cpu(event->dw3);
+ u8 dci = FIELD_GET(EVE_TRB_ENDPOINT_ID_MASK, dw3);
+ u16 comp_code =
+ FIELD_GET(EVE_TRB_COMPL_CODE_MASK, le32_to_cpu(event->dw2));
+ struct crg_udc_ep *udc_ep_ptr;
+ struct crg_udc_request *udc_req_ptr;
+ struct crg_transfer_trb *trb;
+ bool trbs_dequeued = false;
+ u32 residual;
+
+ if (dci >= CRG_NUM_EP_CX || dci == 1)
+ return -EINVAL;
+
+ udc_ep_ptr = &crg_udc->udc_ep[dci];
+ if (!udc_ep_ptr->first_trb ||
+ crg_udc_get_ep_state(crg_udc, dci) == EP_STATE_DISABLED)
+ return -ENODEV;
+
+ if (comp_code != CMPL_CODE_STOPPED &&
+ comp_code != CMPL_CODE_STOPPED_LENGTH_INVALID &&
+ comp_code != CMPL_CODE_DISABLED &&
+ comp_code != CMPL_CODE_DISABLED_LENGTH_INVALID &&
+ comp_code != CMPL_CODE_HALTED &&
+ comp_code != CMPL_CODE_HALTED_LENGTH_INVALID)
+ crg_udc_update_dequeue_pt(event, udc_ep_ptr);
+
+ if (is_request_dequeued(crg_udc, udc_ep_ptr, event)) {
+ trbs_dequeued = true;
+ goto queue_more_trbs;
+ }
+
+ switch (comp_code) {
+ case CMPL_CODE_SUCCESS:
+ trb = crg_udc_tran_trb_dma_to_virt(
+ udc_ep_ptr,
+ (u64)le32_to_cpu(event->dw0) |
+ ((u64)le32_to_cpu(event->dw1) << 32));
+ if (!trb || list_empty(&udc_ep_ptr->queue))
+ break;
+
+ /*
+ * A control transfer has separate data-stage and status-stage
+ * TRBs. The status request is an internal zero-length request
+ * and its successful completion must not go through
+ * crg_udc_build_td(). Doing that attempts to queue another
+ * control TD while the status request is already on the EP0
+ * software queue, leaving it stuck in -EINPROGRESS. The next
+ * SETUP then sees a non-empty queue and enumeration fails.
+ */
+ if (FIELD_GET(TRB_TYPE_MASK, le32_to_cpu(trb->dw3)) ==
+ TRB_TYPE_XFER_STATUS_STAGE) {
+ udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+ struct crg_udc_request,
+ queue);
+ if (dci == 0) {
+ udc_req_ptr->usb_req.actual = 0;
+ trbs_dequeued = true;
+ crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
+ crg_udc_ep0_req_complete(udc_ep_ptr, flags);
+ } else {
+ dev_err(crg_udc->dev,
+ "status-stage TRB on endpoint %u\n",
+ dci);
+ }
+ break;
+ }
+
+ if (!FIELD_GET(TRB_CHAIN_BIT_MASK, le32_to_cpu(trb->dw3))) {
+ udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+ struct crg_udc_request,
+ queue);
+ trbs_dequeued = true;
+ udc_req_ptr->td_pending = false;
+
+ if (udc_req_ptr->all_trbs_queued) {
+ udc_req_ptr->usb_req.actual =
+ udc_req_ptr->usb_req.length -
+ udc_req_ptr->buff_len_left;
+ residual = FIELD_GET(EVE_TRB_TRAN_LEN_MASK,
+ le32_to_cpu(event->dw2));
+ if (residual > udc_req_ptr->usb_req.actual)
+ residual = 0;
+ udc_req_ptr->usb_req.actual -= residual;
+ crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
+
+ if (dci == 0 && udc_ep_ptr->desc &&
+ usb_endpoint_xfer_control(udc_ep_ptr->desc))
+ crg_udc_ep0_req_complete(udc_ep_ptr,
+ flags);
+ } else {
+ int ret;
+
+ ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
+ if (ret == -ENOSPC || ret == -EBUSY) {
+ /* A previous TD completed, but this next TD
+ * could not yet be published. Retry later.
+ */
+ udc_ep_ptr->tran_ring_full = true;
+ } else if (ret) {
+ udc_req_ptr->usb_req.actual =
+ udc_req_ptr->usb_req.length -
+ udc_req_ptr->buff_len_left;
+ crg_udc_req_done(udc_ep_ptr, udc_req_ptr,
+ ret);
+ }
+ }
+ }
+ break;
+
+ case CMPL_CODE_SHORT_PKT:
+ if (list_empty(&udc_ep_ptr->queue))
+ break;
+
+ udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+ struct crg_udc_request, queue);
+ trb = crg_udc_tran_trb_dma_to_virt(
+ udc_ep_ptr,
+ (u64)le32_to_cpu(event->dw0) |
+ ((u64)le32_to_cpu(event->dw1) << 32));
+ if (!trb)
+ break;
+
+ residual = FIELD_GET(EVE_TRB_TRAN_LEN_MASK,
+ le32_to_cpu(event->dw2));
+ udc_req_ptr->usb_req.actual =
+ crg_udc_actual_before(udc_ep_ptr, udc_req_ptr, trb) +
+ FIELD_GET(TRB_TRANSFER_LEN_MASK,
+ le32_to_cpu(trb->dw2)) -
+ min_t(u32, residual,
+ FIELD_GET(TRB_TRANSFER_LEN_MASK,
+ le32_to_cpu(trb->dw2)));
+ udc_req_ptr->usb_req.actual = min(udc_req_ptr->usb_req.actual,
+ udc_req_ptr->usb_req.length);
+
+ /* Update the ring before dropping udc_lock for giveback. */
+ trbs_dequeued = true;
+ crg_udc_advance_dequeue_pt(udc_ep_ptr);
+ if (udc_req_ptr->usb_req.short_not_ok &&
+ usb_endpoint_dir_out(udc_ep_ptr->desc))
+ crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EREMOTEIO);
+ else
+ crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
+ if (dci == 0 && udc_ep_ptr->desc &&
+ crg_udc->ep0_state == EP0_STATE_IDLE &&
+ usb_endpoint_xfer_control(udc_ep_ptr->desc))
+ crg_udc_ep0_req_complete(udc_ep_ptr, flags);
+ break;
+
+ case CMPL_CODE_PROTOCOL_STALL:
+ case CMPL_CODE_TRB_STALL:
+ if (!list_empty(&udc_ep_ptr->queue)) {
+ udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+ struct crg_udc_request,
+ queue);
+ crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EPIPE);
+ trbs_dequeued = true;
+ }
+ if (dci == 0)
+ crg_udc->setup_status = WAIT_FOR_SETUP;
+ break;
+
+ case CMPL_CODE_SETUP_TAG_MISMATCH:
+ /*
+ * EP0 SETUP changes are serialized by the SETUP event handler.
+ * When a new SETUP arrives, crg_udc_abort_ep0_for_setup() stops
+ * EP0, gives back the old request, resets the EP0 ring, and then
+ * accepts the new SETUP. A SETUP_TAG_MISMATCH event generated
+ * afterwards therefore belongs to the aborted control transfer.
+ *
+ * Do not complete the current EP0 request or change setup_status
+ * here: doing so could operate on the new control transfer and
+ * recreate the EP0 queue/state race this path is intended to avoid.
+ */
+ if (dci == 0) {
+ dev_dbg(crg_udc->dev,
+ "ignoring stale EP0 SETUP_TAG_MISMATCH\n");
+ break;
+ }
+
+ /* This completion code is meaningful only for EP0. */
+ dev_warn(crg_udc->dev,
+ "SETUP_TAG_MISMATCH on non-EP0 endpoint %u\n", dci);
+ break;
+
+ case CMPL_CODE_BABBLE_DETECTED_ERR:
+ case CMPL_CODE_DATA_BUFFER_ERR:
+ case CMPL_CODE_INVALID_STREAM_TYPE_ERR:
+ case CMPL_CODE_RING_UNDERRUN:
+ case CMPL_CODE_RING_OVERRUN:
+ case CMPL_CODE_EVENT_RING_FULL_ERR:
+ case CMPL_CODE_ISOCH_BUFFER_OVERRUN:
+ case CMPL_CODE_USB_TRANS_ERR:
+ case CMPL_CODE_TRB_ERR:
+ dev_err(crg_udc->dev, "transfer error, completion code 0x%x\n",
+ comp_code);
+ if (!list_empty(&udc_ep_ptr->queue)) {
+ udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+ struct crg_udc_request,
+ queue);
+ crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EIO);
+ trbs_dequeued = true;
+ }
+ if (dci)
+ set_ep_halt(crg_udc, dci, flags);
+ break;
+
+ case CMPL_CODE_STOPPED:
+ case CMPL_CODE_STOPPED_LENGTH_INVALID:
+ case CMPL_CODE_HALTED:
+ case CMPL_CODE_HALTED_LENGTH_INVALID:
+ case CMPL_CODE_DISABLED:
+ case CMPL_CODE_DISABLED_LENGTH_INVALID:
+ break;
+
+ default:
+ dev_err(crg_udc->dev,
+ "unsupported transfer completion code 0x%x\n",
+ comp_code);
+ if (!list_empty(&udc_ep_ptr->queue)) {
+ udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
+ struct crg_udc_request,
+ queue);
+ crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EIO);
+ trbs_dequeued = true;
+ }
+ break;
+ }
+
+queue_more_trbs:
+ /* Even if the ring was not full, a later request may be waiting
+ * behind a TD from an earlier request (32 TRBs per TD).
+ */
+ if (trbs_dequeued && dci != 0 &&
+ udc_ep_ptr->ep_state == EP_STATE_RUNNING &&
+ !crg_udc->resetting) {
+ udc_ep_ptr->tran_ring_full = false;
+ if (crg_udc_queue_pending_trbs(udc_ep_ptr))
+ udc_ep_ptr->tran_ring_full = true;
+ }
+
+ return 0;
+}
+
+int crg_udc_handle_event(struct crg_gadget_dev *crg_udc,
+ struct crg_event_trb *event, unsigned long *flags)
+{
+ int ret;
+
+ trace_crg_udc_event_trb(event);
+
+ switch (FIELD_GET(EVE_TRB_TYPE_MASK, le32_to_cpu(event->dw3))) {
+ case TRB_TYPE_EVT_PORT_STATUS_CHANGE:
+ if (crg_udc->device_state == USB_STATE_RECONNECTING) {
+ crg_udc->portsc_on_reconnecting = 1;
+ break;
+ }
+ /* 1.Port Reset 2.Port Connection Change 3.Port Link Change */
+ ret = crg_udc_handle_port_status(crg_udc, flags);
+ if (ret)
+ return ret;
+
+ break;
+ case TRB_TYPE_EVT_TRANSFER:
+ if (crg_udc->device_state < USB_STATE_RECONNECTING) {
+ dev_err(crg_udc->dev,
+ "Xfer compl event rcved when dev state=%d !\n",
+ crg_udc->device_state);
+ break;
+ }
+
+ ret = crg_udc_handle_xfer_event(crg_udc, event, flags);
+ if (ret)
+ return ret;
+ break;
+ case TRB_TYPE_EVT_SETUP_PKT: {
+ struct usb_ctrlrequest setup;
+ u32 dw0 = le32_to_cpu(event->dw0);
+ u32 dw1 = le32_to_cpu(event->dw1);
+ u8 setup_tag;
+
+ BUILD_BUG_ON(sizeof(struct usb_ctrlrequest) != 8);
+ setup.bRequestType = dw0 & 0xff;
+ setup.bRequest = (dw0 >> 8) & 0xff;
+ setup.wValue = cpu_to_le16((dw0 >> 16) & 0xffff);
+ setup.wIndex = cpu_to_le16(dw1 & 0xffff);
+ setup.wLength = cpu_to_le16((dw1 >> 16) & 0xffff);
+
+ setup_tag = FIELD_GET(EVE_TRB_SETUP_TAG_MASK,
+ le32_to_cpu(event->dw3));
+ if (crg_udc->setup_status != WAIT_FOR_SETUP) {
+ /* A new SETUP aborts the previous control transfer. The old
+ * EP0 request must be completed and its TRBs discarded before
+ * the new setup is handed to the gadget driver.
+ */
+ if (crg_udc_abort_ep0_for_setup(crg_udc)) {
+ crg_udc_set_ep0_halt(crg_udc);
+ break;
+ }
+ }
+
+ crg_udc_handle_setup_pkt(crg_udc, &setup, setup_tag, flags);
+
+ break;
+ }
+ default:
+ dev_dbg(crg_udc->dev, "unexpected TRB_TYPE = 0x%lx",
+ FIELD_GET(EVE_TRB_TYPE_MASK, le32_to_cpu(event->dw3)));
+ break;
+ }
+
+ return 0;
+}
+
+/*
+ * Invoke the bound gadget driver's ->disconnect() callback, if any.
+ *
+ * The gadget disconnect callback may be called in interrupt context.
+ * Release only udc_lock while keeping local IRQs disabled.
+ */
+void crg_udc_notify_disconnect(struct crg_gadget_dev *crg_udc,
+ unsigned long *flags)
+{
+ /* Report link loss even when there is no gadget callback. */
+ usb_gadget_set_state(&crg_udc->gadget, USB_STATE_NOTATTACHED);
+
+ if (!crg_udc->async_callbacks || !crg_udc->gadget_driver ||
+ !crg_udc->gadget_driver->disconnect)
+ return;
+
+ spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+ crg_udc->gadget_driver->disconnect(&crg_udc->gadget);
+ spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+}
+
+int crg_udc_ep_halt(struct crg_udc_ep *udc_ep_ptr, int halt, int ignore_wedge,
+ unsigned long *flags)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ enum crg_ep_state ep_state;
+ bool reset_seq_only = false;
+ int do_halt;
+ u32 param0;
+ u32 tmp;
+
+ if (crg_udc->resetting)
+ return -ESHUTDOWN;
+ if (!udc_ep_ptr->desc) {
+ dev_err(crg_udc->dev, "NULL desc\n");
+ return -EINVAL;
+ }
+
+ if (usb_endpoint_xfer_isoc(udc_ep_ptr->desc)) {
+ dev_err(crg_udc->dev, "Isoc ep, halt not supported\n");
+ return -EOPNOTSUPP;
+ }
+
+ if (udc_ep_ptr->dci == 0)
+ return 0;
+
+ ep_state = crg_udc_get_ep_state(crg_udc, udc_ep_ptr->dci);
+
+ if (ep_state == EP_STATE_DISABLED)
+ return 0;
+
+ if (ep_state == EP_STATE_HALTED) {
+ if (halt != 0)
+ return 0;
+
+ /* want unhalt an halted ep */
+ if (udc_ep_ptr->wedge && !ignore_wedge) {
+ do_halt = -1;
+ reset_seq_only = true;
+ } else {
+ do_halt = 0;
+ }
+
+ } else {
+ /* ep state == running or stopped */
+ if (halt != 0) {
+ /* want halt a running ep */
+ do_halt = 1;
+ } else {
+ /* reset a running ep */
+ do_halt = 0;
+ reset_seq_only = true;
+ }
+ }
+
+ trace_crg_udc_ep_halt(udc_ep_ptr, do_halt == 1, udc_ep_ptr->wedge);
+
+ param0 = BIT(udc_ep_ptr->dci);
+ if (do_halt == 1) {
+ /* setting ep to halt */
+ if (crg_udc_issue_command(crg_udc, CRG_CMD_SET_HALT, param0, 0))
+ return -EIO;
+ if (readl_poll_timeout_atomic(&uccr->ep_running, tmp,
+ !(tmp & param0), 10, 10000))
+ dev_warn(crg_udc->dev, "ep %d halt wait timed out\n",
+ udc_ep_ptr->dci);
+
+ if (crg_udc->gadget.speed >= USB_SPEED_SUPER) {
+ udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
+ udc_ep_ptr->tran_ring_full = false;
+ }
+ udc_ep_ptr->ep_state = EP_STATE_HALTED;
+ if (crg_udc->gadget.speed >= USB_SPEED_SUPER)
+ crg_udc_ep_nuke(udc_ep_ptr, -ECONNRESET);
+ } else if (do_halt == 0) {
+ /* clearing ep halt state */
+ /* reset sequence number */
+ if (crg_udc_issue_command(crg_udc, CRG_CMD_RESET_SEQNUM, param0,
+ 0))
+ return -EIO;
+
+ if (!reset_seq_only) {
+ /* Clear halt for a halted EP. */
+ /* NOTE: we must CLEAR_HALT first,
+ * then SET_TR_DQPTR
+ */
+ if (crg_udc_issue_command(crg_udc, CRG_CMD_CLEAR_HALT,
+ param0, 0))
+ return -EIO;
+ if (crg_udc_epcx_update_dqptr(udc_ep_ptr))
+ return -EIO;
+ }
+
+ udc_ep_ptr->wedge = 0;
+ udc_ep_ptr->ep_state = EP_STATE_RUNNING;
+ /* set endpoint to running state */
+ /* clear pause for the endpoint */
+ if (!list_empty(&udc_ep_ptr->queue))
+ crg_udc_knock_doorbell(crg_udc, udc_ep_ptr->dci);
+ } else {
+ /* wedged EP deny CLEAR HALT */
+ dev_dbg(crg_udc->dev, "wedged EP deny CLEAR HALT dci = %d\n",
+ udc_ep_ptr->dci);
+ /* reset sequence number */
+ if (reset_seq_only &&
+ crg_udc_issue_command(crg_udc, CRG_CMD_RESET_SEQNUM, param0,
+ 0))
+ return -EIO;
+ }
+
+ return 0;
+}
+
+enum crg_ep_state crg_udc_get_ep_state(struct crg_gadget_dev *crg_udc, int dci)
+{
+ struct crg_udc_ep *udc_ep_ptr;
+
+ /*
+ * DCI 0 is ep0; DCI 1 is unused in the Corigine ep-context layout
+ * (ep contexts start from DCI 2 for ep1in/ep1out). This is why the
+ * ep-context array is indexed as p_epcx[dci - 2] elsewhere.
+ */
+ if (dci < 0 || dci >= CRG_NUM_EP_CX || dci == 1)
+ return EP_STATE_DISABLED;
+
+ udc_ep_ptr = &crg_udc->udc_ep[dci];
+
+ return udc_ep_ptr->ep_state;
+}
+
+u32 crg_udc_index2dci(u16 index)
+{
+ if (index == 0)
+ return 0;
+
+ return (index & USB_ENDPOINT_NUMBER_MASK) * 2 +
+ ((index & USB_DIR_IN) ? 0 : 1);
+}
+
+int crg_udc_init_device_context(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ int ret;
+ u32 buff_length;
+ dma_addr_t mapping;
+
+ /* ep0 is not included in ep contexts in crg udc */
+ buff_length = (CRG_NUM_EP_CX - 2) * sizeof(struct crg_ep_cx);
+
+ if (!crg_udc->ep_cx.cpu_addr) {
+ crg_udc->ep_cx.cpu_addr = dma_alloc_coherent(
+ crg_udc->dev, buff_length, &mapping, GFP_ATOMIC);
+ if (!crg_udc->ep_cx.cpu_addr) {
+ ret = -ENOMEM;
+ return ret;
+ }
+ } else {
+ mapping = crg_udc->ep_cx.dma_addr;
+ }
+
+ crg_udc->p_epcx = crg_udc->ep_cx.cpu_addr;
+ crg_udc->ep_cx.len = buff_length;
+ crg_udc->ep_cx.dma_addr = mapping;
+ memset(crg_udc->p_epcx, 0, buff_length);
+ /*
+ * Ensure the endpoint contexts are initialized before making their
+ * DMA address visible to the controller.
+ */
+ dma_wmb();
+ /* hw ops DCBAPLO DCBAPHI */
+ writel(lower_32_bits(crg_udc->ep_cx.dma_addr), &uccr->dcbaplo);
+ writel(upper_32_bits(crg_udc->ep_cx.dma_addr), &uccr->dcbaphi);
+
+ return 0;
+}
+
+void crg_udc_init_ep_info(struct crg_gadget_dev *crg_udc)
+{
+ int i;
+
+ /* udc_ep[0] is reserved */
+ crg_ep_struct_setup(crg_udc, 0, NULL);
+
+ for (i = 1; i < CRG_NUM_EP_CX / 2; i++) {
+ char name[14];
+
+ snprintf(name, sizeof(name), "ep%din", i);
+ crg_ep_struct_setup(crg_udc, i * 2, name);
+ snprintf(name, sizeof(name), "ep%dout", i);
+ crg_ep_struct_setup(crg_udc, i * 2 + 1, name);
+ }
+}
+
+static bool is_request_dequeued(struct crg_gadget_dev *crg_udc,
+ struct crg_udc_ep *udc_ep,
+ struct crg_event_trb *event)
+{
+ struct crg_udc_request *udc_req;
+ u32 trb_pt_lo = le32_to_cpu(event->dw0);
+ u32 trb_pt_hi = le32_to_cpu(event->dw1);
+ u64 trb_addr = (u64)trb_pt_lo | ((u64)trb_pt_hi << 32);
+ struct crg_transfer_trb *trb_pt;
+ bool status = true;
+
+ if (udc_ep->dci == 0 || list_empty(&udc_ep->queue))
+ return false;
+
+ trb_pt = crg_udc_tran_trb_dma_to_virt(udc_ep, trb_addr);
+ if (!trb_pt)
+ return false;
+ list_for_each_entry(udc_req, &udc_ep->queue, queue) {
+ if (trb_pt == udc_req->last_trb ||
+ trb_pt == udc_req->first_trb) {
+ status = false;
+ break;
+ }
+
+ if (crg_udc_is_pointer_less_than(trb_pt, udc_req->last_trb,
+ udc_ep) &&
+ crg_udc_is_pointer_less_than(udc_req->first_trb, trb_pt,
+ udc_ep)) {
+ status = false;
+ break;
+ }
+ }
+
+ return status;
+}
+
+void crg_udc_knock_doorbell(struct crg_gadget_dev *crg_udc, int dci)
+{
+ u32 tmp;
+ struct crg_uccr __iomem *uccr;
+
+ uccr = crg_udc->uccr;
+ /* Ensure everything is written before notifying the HW */
+ dma_wmb();
+
+ tmp = FIELD_PREP(CRG_U3DC_DB_TARGET_MASK, dci);
+ writel(tmp, &uccr->doorbell);
+
+ trace_crg_udc_doorbell(crg_udc, dci);
+}
+
+/*
+ * Complete a request and give it back to the gadget layer via the
+ * standard usb_gadget_giveback_request() helper.
+ *
+ * Must be called with udc_lock held and interrupts disabled.
+ *
+ * The request's ->complete() callback must not sleep. crg_udc_req_done() drops
+ * udc_lock before the giveback and re-acquires it before returning.
+ * Callers therefore cannot assume state protected by udc_lock is
+ * unchanged across this call.
+ */
+void crg_udc_req_done(struct crg_udc_ep *udc_ep,
+ struct crg_udc_request *udc_req, int status)
+{
+ struct crg_gadget_dev *crg_udc = udc_ep->crg_udc;
+
+ if (likely(udc_req->usb_req.status == -EINPROGRESS))
+ udc_req->usb_req.status = status;
+
+ trace_crg_udc_request_giveback(udc_ep, udc_req);
+
+ list_del_init(&udc_req->queue);
+
+ crg_udc_unmap_request(udc_ep, udc_req);
+
+ if (udc_req->usb_req.complete) {
+ spin_unlock(&crg_udc->udc_lock);
+ usb_gadget_giveback_request(&udc_ep->usb_ep, &udc_req->usb_req);
+ spin_lock(&crg_udc->udc_lock);
+ }
+}
+
+static int set_ep_halt(struct crg_gadget_dev *crg_udc, int dci,
+ unsigned long *flags)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ struct crg_udc_ep *udc_ep_ptr = &crg_udc->udc_ep[dci];
+ u32 param0;
+ u32 tmp;
+
+ if (dci == 0)
+ return 0;
+
+ if (udc_ep_ptr->ep_state == EP_STATE_DISABLED ||
+ udc_ep_ptr->ep_state == EP_STATE_HALTED)
+ return 0;
+ if (udc_ep_ptr->desc && usb_endpoint_xfer_isoc(udc_ep_ptr->desc))
+ return -EOPNOTSUPP;
+
+ param0 = BIT(dci);
+ if (crg_udc_issue_command(crg_udc, CRG_CMD_SET_HALT, param0, 0))
+ return -EIO;
+ if (readl_poll_timeout_atomic(&uccr->ep_running, tmp, !(tmp & param0),
+ 10, 10000))
+ return -ETIMEDOUT;
+ udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
+ udc_ep_ptr->tran_ring_full = false;
+ udc_ep_ptr->ep_state = EP_STATE_HALTED;
+ crg_udc_ep_nuke(udc_ep_ptr, -ECONNRESET);
+
+ return 0;
+}
+
+void crg_udc_clear_req_container(struct crg_udc_request *udc_req_ptr)
+{
+ udc_req_ptr->buff_len_left = 0;
+ udc_req_ptr->trbs_needed = 0;
+ udc_req_ptr->all_trbs_queued = 0;
+ udc_req_ptr->td_pending = false;
+ udc_req_ptr->first_trb = NULL;
+ udc_req_ptr->last_trb = NULL;
+ udc_req_ptr->sg = NULL;
+ udc_req_ptr->sg_index = 0;
+ udc_req_ptr->sg_offset = 0;
+}
+
+const struct usb_ep_ops crg_udc_ep_ops = {
+ .enable = crg_udc_ep_enable,
+ .disable = crg_udc_ep_disable,
+ .alloc_request = crg_udc_alloc_request,
+ .free_request = crg_udc_free_request,
+ .queue = crg_udc_ep_queue,
+ .dequeue = crg_udc_ep_dequeue,
+ .set_halt = crg_udc_ep_set_halt,
+ .set_wedge = crg_udc_ep_set_wedge,
+};
+
+const struct usb_gadget_ops crg_gadget_ops = {
+ .get_frame = crg_gadget_get_frame,
+ .wakeup = crg_gadget_wakeup,
+ .set_remote_wakeup = crg_gadget_set_remote_wakeup,
+ .set_selfpowered = crg_gadget_set_selfpowered,
+ .pullup = crg_gadget_pullup,
+ .udc_start = crg_gadget_start,
+ .udc_stop = crg_gadget_stop,
+ .udc_set_speed = crg_udc_set_speed,
+ .udc_async_callbacks = crg_gadget_async_callbacks,
+};
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-trace.c b/drivers/usb/gadget/udc/crg/crg-udc-trace.c
new file mode 100644
index 000000000000..ee091708dbe0
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-trace.c
@@ -0,0 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ * Trace Support.
+ *
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#define CREATE_TRACE_POINTS
+#include "crg-udc-trace.h"
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [PATCH v4 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations
2026-10-30 8:10 ` [PATCH v4 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations Petar Stepanovic
@ 2026-10-30 8:50 ` sashiko-bot
0 siblings, 0 replies; 14+ messages in thread
From: sashiko-bot @ 2026-10-30 8:50 UTC (permalink / raw)
To: Petar Stepanovic; +Cc: linux-usb, devicetree, robh, conor+dt
Thank you for your contribution! Sashiko AI review found 22 potential issue(s) to consider:
- [High] usb: gadget: crg: premature dequeue pointer update in crg_udc_handle_xfer_event()
- [High] usb: gadget: crg: sleeping DMA allocation and free in crg_udc_ep_enable()
- [High] usb: gadget: crg: stale first_trb on multi-TD requests in crg_udc_queue_trbs()
- [High] usb: gadget: crg: broken zero-length EP0 data stage and dequeue handling
- [High] usb: gadget: crg: crg_udc_ep_nuke() hides requests on stack list across unlock
- [High] usb: gadget: crg: corrupt deq_pt on stale events in crg_udc_handle_xfer_event()
- [High] usb: gadget: crg: premature giveback before halt in crg_udc_handle_xfer_event()
- [High] usb: gadget: crg: silent no-op when stalling EP0 in crg_udc_ep_halt()
- [High] usb: gadget: crg: crg_udc_reinit() clears usb_ep.enabled before reset work
- [High] usb: gadget: crg: lock drop mid-loop in crg_udc_reinit() leaves EPs running
- [High] usb: gadget: crg: DMA unmap on ep_running poll timeout in crg_udc_ep_halt()
- [High] usb: gadget: udc: crg: lost port events in crg_udc_handle_port_status()
- [High] usb: gadget: udc: crg: premature SETUP handling in crg_udc_handle_event()
- [High] usb: gadget: udc: crg: stuck EP0_STATE_ERROR in crg_udc_handle_event()
- [High] usb: gadget: udc: crg: stuck EP_STATE_STOPPED in crg_udc_ep_dequeue()
- [High] usb: gadget: udc: crg: rejected halted/stopped queue in crg_udc_ep_queue()
- [Medium] usb: gadget: crg: crg_udc_ep_queue() rejects 0-length requests with NULL buf
- [Medium] usb: gadget: crg: missing IN queue check in crg_udc_ep_set_wedge()
- [Medium] usb: gadget: crg: missing usb_ep.maxpacket update in crg_udc_ep_enable()
- [Medium] usb: gadget: udc: crg: unrolled ring state on error in crg_udc_queue_trbs()
- [Medium] usb: gadget: udc: crg: clobbered tran_ring_full in crg_udc_handle_xfer_event()
- [Medium] usb: gadget: crg: missing EP0 state revalidation in crg_udc_ep_queue()
--
commit c0163bd83b42f6c6c4258f527a05306a12934416
Author: Petar Stepanovic <pstepanovic@axiado.com>
usb: gadget: udc: crg: Add gadget and endpoint operations
This patch implements the usb_gadget_ops and usb_ep_ops callbacks along with
transfer ring, endpoint state, event, and port status handling for the Axiado
AX3000/AX3005 UDC driver.
> diff --git a/drivers/usb/gadget/udc/crg/crg-udc-gadget.c b/drivers/usb/gadget/udc/crg/crg-udc-gadget.c
> new file mode 100644
> index 0000000000000..8c52f8856490a
> --- /dev/null
> +++ b/drivers/usb/gadget/udc/crg/crg-udc-gadget.c
[ ... ]
> +static int crg_udc_ep_dequeue(struct usb_ep *_ep, struct usb_request *_req)
> +{
> + struct crg_udc_ep *udc_ep_ptr;
> + struct crg_gadget_dev *crg_udc;
> + struct crg_uccr __iomem *uccr;
> + struct crg_udc_request *udc_req = NULL;
> + struct crg_udc_request *iter;
> + struct crg_ep_cx *p_ep_cx;
> + struct crg_transfer_trb *pause_pt;
> + enum crg_ep_state old_ep_state;
> + u32 tmp, deq_pt_lo, deq_pt_hi;
> + dma_addr_t dq_pt_addr;
> + u8 dci;
> + bool found = false;
> + u32 generation;
> + unsigned long flags;
> + const struct usb_endpoint_descriptor *old_desc;
> + u32 event_generation;
> +
> + if (!_ep || !_req)
> + return -EINVAL;
> +
> + udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
> + crg_udc = udc_ep_ptr->crg_udc;
> + uccr = crg_udc->uccr;
> + dci = udc_ep_ptr->dci;
> +
> + spin_lock_irqsave(&crg_udc->udc_lock, flags);
> + if (crg_udc->resetting) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -ESHUTDOWN;
> + }
> + old_ep_state = udc_ep_ptr->ep_state;
> + old_desc = udc_ep_ptr->desc;
> + event_generation = crg_udc->udc_event[0].generation;
> + generation = udc_ep_ptr->generation;
> +
> + list_for_each_entry(iter, &udc_ep_ptr->queue, queue) {
> + if (&iter->usb_req == _req) {
> + udc_req = iter;
> + found = true;
> + break;
> + }
> + }
> +
> + if (!found) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -EINVAL;
> + }
> +
> + if (!udc_req->first_trb) {
> + crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return 0;
> + }
> +
> + if (dci == 0) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -EINVAL;
> + }
[Severity: High]
What happens when a gadget driver dequeues its EP0 request in
crg_udc_ep_dequeue(), for instance during __composite_unbind()?
If the request was queued through crg_udc_build_ep0_status(), first_trb is
never assigned. For a newly allocated request where first_trb is NULL,
crg_udc_ep_dequeue() completes and unlinks the request while its status TRB
is still live on the EP0 hardware ring without stopping EP0 DMA.
Conversely, if the EP0 request is a data-stage request (or a reused request
with a stale non-NULL first_trb), crg_udc_ep_dequeue() hits the dci == 0
check and returns -EINVAL without stopping EP0 or removing the request from
udc_ep[0].queue. Doesn't that lead to a use-after-free when the caller
immediately frees the usb_request on unbind?
> + if (old_ep_state == EP_STATE_RUNNING) {
> + int ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP,
> + BIT(dci), 0);
> +
> + if (ret) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return ret;
> + }
> + if (readl_poll_timeout_atomic(&uccr->ep_running, tmp,
> + !(tmp & BIT(dci)), 10, 10000)) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -ETIMEDOUT;
> + }
> + udc_ep_ptr->ep_state = EP_STATE_STOPPED;
> + }
> +
> + p_ep_cx = &crg_udc->p_epcx[dci - 2];
> + deq_pt_lo = le32_to_cpu(p_ep_cx->dw2) & EP_CX_TR_DQPT_LO_MASK;
> + deq_pt_hi = le32_to_cpu(p_ep_cx->dw3);
> + dq_pt_addr = (u64)deq_pt_lo | ((u64)deq_pt_hi << 32);
> + pause_pt = crg_udc_tran_trb_dma_to_virt(udc_ep_ptr, dq_pt_addr);
> + if (!pause_pt) {
> + if (old_ep_state == EP_STATE_RUNNING) {
> + udc_ep_ptr->ep_state = EP_STATE_RUNNING;
> + crg_udc_knock_doorbell(crg_udc, dci);
> + }
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -EIO;
> + }
> +
> + if (crg_udc_is_pointer_less_than(pause_pt, udc_req->first_trb,
> + udc_ep_ptr)) {
> + crg_udc_squeeze_xfer_ring(udc_ep_ptr, udc_req);
> + /* Give back the cancelled request BEFORE scheduling the tail:
> + * its callback may add/remove requests, and dropping udc_lock
> + * while traversing the old list is unsafe.
> + */
> + crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
> + } else if (udc_req->last_trb &&
> + crg_udc_is_pointer_less_than(udc_req->last_trb, pause_pt,
> + udc_ep_ptr)) {
> + crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
> + } else {
> + struct crg_udc_request *next_req = NULL;
> +
> + udc_req->usb_req.actual =
> + crg_udc_actual_before(udc_ep_ptr, udc_req, pause_pt);
> +
> + if (!list_is_last(&udc_req->queue, &udc_ep_ptr->queue))
> + next_req = list_entry(udc_req->queue.next,
> + struct crg_udc_request, queue);
> +
> + if (next_req && next_req->first_trb)
> + udc_ep_ptr->deq_pt = next_req->first_trb;
> + else
> + udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
> +
> + if (crg_udc_epcx_update_dqptr(udc_ep_ptr)) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -EIO;
> + }
> + udc_ep_ptr->tran_ring_full = false;
> + crg_udc_req_done(udc_ep_ptr, udc_req, -ECONNRESET);
> + }
> +
> + if (old_ep_state == EP_STATE_RUNNING && !crg_udc->resetting &&
> + udc_ep_ptr->desc == old_desc &&
> + event_generation == crg_udc->udc_event[0].generation &&
> + udc_ep_ptr->generation == generation &&
> + udc_ep_ptr->ep_state == EP_STATE_STOPPED) {
> + udc_ep_ptr->ep_state = EP_STATE_RUNNING;
> + udc_ep_ptr->tran_ring_full = false;
> + if (crg_udc_queue_pending_trbs(udc_ep_ptr))
> + udc_ep_ptr->tran_ring_full = true;
> + crg_udc_knock_doorbell(crg_udc, dci);
> + }
[Severity: High]
Doesn't this leave udc_ep_ptr->ep_state stuck in EP_STATE_STOPPED if
crg_udc_epcx_update_dqptr() fails, and also reject any requests queued from
the completion callback during crg_udc_req_done()?
First, if crg_udc_epcx_update_dqptr(udc_ep_ptr) fails, crg_udc_ep_dequeue()
unlocks and returns -EIO without restoring udc_ep_ptr->ep_state to
EP_STATE_RUNNING (unlike the !pause_pt error path above), leaving the
endpoint permanently wedged in EP_STATE_STOPPED.
Second, crg_udc_req_done() is called while udc_ep_ptr->ep_state is still
EP_STATE_STOPPED, before EP_STATE_RUNNING is restored. The comment notes that
the completion callback may add requests while udc_lock is dropped, but
crg_udc_ep_queue() rejects any request when ep_state != EP_STATE_RUNNING with
-ESHUTDOWN.
[ ... ]
> +static int crg_udc_ep_enable(struct usb_ep *ep,
> + const struct usb_endpoint_descriptor *desc)
> +{
[ ... ]
> + /* Leave the old allocation valid until its replacement is ready.
> + * Other ep operations reject the configuring endpoint.
> + */
> + if (!udc_ep->tran_ring_info.cpu_addr ||
> + udc_ep->tran_ring_info.len != len) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + new_ring = dma_alloc_coherent(crg_udc->dev, len, &new_dma,
> + GFP_KERNEL);
> + spin_lock_irqsave(&crg_udc->udc_lock, flags);
> + if (!new_ring) {
> + ret = -ENOMEM;
> + goto err;
> + }
> + if (!crg_udc->gadget_driver || crg_udc->resetting ||
> + udc_ep->ep_state != EP_STATE_DISABLED ||
> + udc_ep->desc != desc) {
> + ret = -ESHUTDOWN;
> + goto err;
> + }
> + old_ring = udc_ep->tran_ring_info.cpu_addr;
> + old_dma = udc_ep->tran_ring_info.dma_addr;
> + old_len = udc_ep->tran_ring_info.len;
> + udc_ep->tran_ring_info.cpu_addr = new_ring;
> + udc_ep->tran_ring_info.dma_addr = new_dma;
> + udc_ep->tran_ring_info.len = len;
> + new_ring = NULL;
> + }
> +
> + udc_ep->first_trb = udc_ep->tran_ring_info.cpu_addr;
> + udc_ep->last_trb = udc_ep->first_trb + ring_size - 1;
> + memset(udc_ep->first_trb, 0, len);
> + dma_wmb();
> + crg_udc_setup_link_trb(udc_ep->last_trb, true,
> + udc_ep->tran_ring_info.dma_addr);
> + udc_ep->enq_pt = udc_ep->first_trb;
> + udc_ep->deq_pt = udc_ep->first_trb;
> + udc_ep->pcs = 1;
> + udc_ep->tran_ring_full = false;
> + ret = crg_udc_epcx_setup(udc_ep);
> + if (ret)
> + goto err;
> + param0 = BIT(udc_ep->dci);
> + ret = crg_udc_issue_command(crg_udc, CRG_CMD_CONFIG_EP, param0, 0);
> + if (ret)
> + goto err;
> +
> + udc_ep->generation++;
> + udc_ep->wedge = false;
> + udc_ep->usb_ep.enabled = 1;
> + udc_ep->ep_state = EP_STATE_RUNNING;
> + udc_ep->configuring = false;
> + crg_udc->num_enabled_eps++;
> + if (crg_udc->device_state == USB_STATE_ADDRESS)
> + crg_udc->device_state = USB_STATE_CONFIGURED;
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + if (old_ring)
> + dma_free_coherent(crg_udc->dev, old_len, old_ring, old_dma);
> + return 0;
> +
> +err:
> + udc_ep->desc = NULL;
> + udc_ep->comp_desc = NULL;
> + udc_ep->configuring = false;
> + /* Keep a valid ring allocation for a later retry. */
> + udc_ep->ep_state = EP_STATE_DISABLED;
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + if (new_ring)
> + dma_free_coherent(crg_udc->dev, len, new_ring, new_dma);
> + if (old_ring)
> + dma_free_coherent(crg_udc->dev, old_len, old_ring, old_dma);
> + return ret;
[Severity: High]
Can crg_udc_ep_enable() sleep or call dma_free_coherent() here?
The USB gadget API allows usb_ep_enable() to be called in atomic context. For
example:
composite_setup() [holds spin_lock(&cdev->lock)]
-> set_config() / f->set_alt()
-> gserial_connect() / ffs_func_eps_enable() [holds spin_lock_irqsave()]
-> usb_ep_enable()
-> crg_udc_ep_enable()
Dropping crg_udc->udc_lock with spin_unlock_irqrestore() only restores the
interrupt state at entry to crg_udc_ep_enable() and does not release spinlocks
held by the caller. Calling dma_alloc_coherent(..., GFP_KERNEL) sleeps while
caller spinlocks are held, and calling dma_free_coherent() when interrupts
were disabled by the caller triggers WARN_ON(irqs_disabled()) in
dma_free_attrs().
[Severity: Medium]
Should crg_udc_ep_enable() also update udc_ep->usb_ep.maxpacket to
usb_endpoint_maxp(desc)?
crg_ep_struct_setup() initializes ep->usb_ep.maxpacket to CRG_MAX_PACKET_SIZE
(1024), and usb_ep_enable() in udc/core.c does not update ep->maxpacket. When
a gadget driver enables an endpoint without calling config_ep_by_speed() (such
as gadgetfs, raw_gadget, or f_printer), udc_ep->usb_ep.maxpacket stays at 1024
even for 512-byte HS or 64-byte FS endpoints, causing crg_udc_queue_trbs() to
miscalculate need_zlp via !(usb_req->length % udc_ep_ptr->usb_ep.maxpacket).
> +out:
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return ret;
> +}
> +
> +void crg_udc_ep_nuke(struct crg_udc_ep *udc_ep, int status)
> +{
> + struct crg_udc_request *req;
> + struct list_head nuke_list;
> +
> + list_replace_init(&udc_ep->queue, &nuke_list);
> +
> + while (!list_empty(&nuke_list)) {
> + req = list_first_entry(&nuke_list, struct crg_udc_request,
> + queue);
> +
> + crg_udc_req_done(udc_ep, req, status);
> + }
> +}
[Severity: High]
Can moving all requests onto the stack-local nuke_list before dropping
udc_lock in crg_udc_req_done() cause a use-after-free?
While crg_udc->udc_lock is dropped during the first request's completion
callback, udc_ep->queue is already empty even though the remaining requests on
nuke_list have not been given back yet. If that callback or another thread
calls usb_ep_dequeue() and usb_ep_free_request() on one of those remaining
requests, crg_udc_ep_dequeue() will not find it on udc_ep->queue and will
return -EINVAL immediately, allowing the caller to free the request while
crg_udc_ep_nuke() still holds it on nuke_list and dereferences it on the next
iteration.
> +static int crg_udc_ep_queue(struct usb_ep *_ep, struct usb_request *_req,
> + gfp_t gfp_flags)
> +{
> + struct crg_udc_request *udc_req_ptr;
> + struct crg_udc_ep *udc_ep_ptr;
> + struct crg_gadget_dev *crg_udc;
> + enum dma_data_direction dma_data_dir;
> + const struct usb_endpoint_descriptor *desc;
> + unsigned long flags;
> + int status;
> + bool is_sg;
> + u8 setup_tag = 0;
> + u32 generation = 0;
> +
> + if (!_ep || !_req)
> + return -EINVAL;
> +
> + udc_req_ptr = container_of(_req, struct crg_udc_request, usb_req);
> + udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
> + crg_udc = udc_ep_ptr->crg_udc;
> + is_sg = _req->num_sgs != 0 && _req->length != 0;
> +
> + if (is_sg && !_req->sg)
> + return -EINVAL;
> + if (_req->stream_id)
> + return -EOPNOTSUPP;
> +
> + spin_lock_irqsave(&crg_udc->udc_lock, flags);
> + /*
> + * A completion callback for an aborted EP0 transfer may requeue the
> + * same request. Reject EP0 requests while the previous control
> + * transfer is being aborted or EP0 is in an error state. EP0 becomes
> + * available again only after the new SETUP has been accepted and
> + * the EP0 transfer ring has been successfully reinitialized.
> + */
> + if (udc_ep_ptr->dci == 0 && crg_udc->ep0_state != EP0_STATE_IDLE) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -ECONNRESET;
> + }
> + if (!udc_ep_ptr->first_trb || !_req->complete ||
> + (!is_sg && !_req->buf) || !list_empty(&udc_req_ptr->queue)) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -EINVAL;
> + }
[Severity: Medium]
Does checking (!is_sg && !_req->buf) reject valid zero-length requests when
_req->buf is NULL?
Because is_sg is false whenever _req->length == 0, any 0-length request with
_req->buf == NULL (such as scatter-gather requests recycled for 0-length
submissions by uvcg_video_hw_submit()) will fail here with -EINVAL even though
_req->length == 0 does not dereference or DMA-map _req->buf.
> + if (crg_udc->resetting) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -ESHUTDOWN;
> + }
> + if (!udc_ep_ptr->desc) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -EINVAL;
> + }
> + desc = udc_ep_ptr->desc;
> + if (udc_ep_ptr->ep_state != EP_STATE_RUNNING ||
> + udc_ep_ptr->configuring) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -ESHUTDOWN;
> + }
[Severity: High]
Should crg_udc_ep_queue() reject requests when udc_ep_ptr->ep_state is
EP_STATE_HALTED or EP_STATE_STOPPED?
Standard USB gadget function drivers (such as f_mass_storage) halt the bulk-in
endpoint and immediately call usb_ep_queue() to queue the CSW status packet
while the endpoint is still halted, waiting for the host to issue
CLEAR_FEATURE(ENDPOINT_HALT). In fact, crg_udc_ep_halt() checks
!list_empty(&udc_ep_ptr->queue) when clearing a halt to ring the doorbell for
queued requests, and crg_udc_ep_dequeue() temporarily places the endpoint in
EP_STATE_STOPPED during giveback. Returning -ESHUTDOWN whenever ep_state !=
EP_STATE_RUNNING (here and in the post-mapping check below) prevents queuing
requests onto halted or paused endpoints.
> + setup_tag = crg_udc->setup_tag;
> + generation = udc_ep_ptr->generation;
> +
> + if (usb_endpoint_xfer_control(udc_ep_ptr->desc) && is_sg) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -EOPNOTSUPP;
> + }
> +
> + if (usb_endpoint_xfer_control(udc_ep_ptr->desc) && !_req->length) {
> + if (crg_udc->setup_status == DATA_STAGE_XFER)
> + crg_udc->setup_status = STATUS_STAGE_RECV;
> + else
> + crg_udc->setup_status = STATUS_STAGE_XFER;
> + crg_udc_build_ep0_status(&crg_udc->udc_ep[0], false,
> + -EINPROGRESS, udc_req_ptr, 0, 0);
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return 0;
> + }
[Severity: High]
Doesn't this treat a 0-length EP0 Data Stage response (when the host's SETUP
had wLength > 0) as a Status Stage TRB?
When wLength > 0, crg_udc_handle_setup_pkt() sets crg_udc->setup_status to
DATA_STAGE_XFER or DATA_STAGE_RECV, and crg_udc_ep0_req_complete()
automatically queues the Status Stage after the Data Stage completes. If a
gadget driver queues a 0-byte response for the Data Stage, this branch
overwrites setup_status to STATUS_STAGE_RECV / STATUS_STAGE_XFER and calls
crg_udc_build_ep0_status(), queuing a TRB_TYPE_XFER_STATUS_STAGE TRB instead
of a 0-length TRB_TYPE_XFER_DATA_STAGE TRB.
Returning early here also skips crg_udc_clear_req_container(udc_req_ptr),
leaving stale first_trb/last_trb/all_trbs_queued fields if udc_req_ptr was
reused from a previous transfer.
> + if (usb_endpoint_xfer_control(udc_ep_ptr->desc)) {
> + if (crg_udc->setup_status == DATA_STAGE_XFER ||
> + crg_udc->setup_status == STATUS_STAGE_XFER) {
> + dma_data_dir = DMA_TO_DEVICE;
> + } else if (crg_udc->setup_status == DATA_STAGE_RECV ||
> + crg_udc->setup_status == STATUS_STAGE_RECV) {
> + dma_data_dir = DMA_FROM_DEVICE;
> + } else {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -EINVAL;
> + }
> + } else {
> + dma_data_dir = usb_endpoint_dir_in(udc_ep_ptr->desc) ?
> + DMA_TO_DEVICE :
> + DMA_FROM_DEVICE;
> + }
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +
> + crg_udc_clear_req_container(udc_req_ptr);
> + udc_req_ptr->mapped = false;
> +
> + if (_req->length) {
> + udc_req_ptr->map_dir = dma_data_dir;
> + status = usb_gadget_map_request_by_dev(
> + crg_udc->dev, _req, dma_data_dir == DMA_TO_DEVICE);
> + if (status)
> + goto err_unmap;
> +
> + if (is_sg) {
> + if (!_req->num_mapped_sgs ||
> + crg_udc_sg_init(udc_req_ptr)) {
> + status = -EIO;
> + goto err_unmap;
> + }
> + } else if (_req->dma_mapped) {
> + udc_req_ptr->mapped = true;
> + }
> + }
> +
> + spin_lock_irqsave(&crg_udc->udc_lock, flags);
> + if (udc_ep_ptr->desc != desc || !udc_req_ptr->usb_req.complete ||
> + !list_empty(&udc_req_ptr->queue) || crg_udc->resetting ||
> + udc_ep_ptr->configuring ||
> + udc_ep_ptr->ep_state != EP_STATE_RUNNING ||
> + udc_ep_ptr->generation != generation ||
> + (udc_ep_ptr->dci == 0 &&
> + (crg_udc->ep0_state != EP0_STATE_IDLE ||
> + crg_udc->setup_tag != setup_tag))) {
[Severity: Medium]
As reported by Sashiko on v3 [1], what happens if a USB bus reset occurs while
udc_lock is dropped for DMA mapping of an EP0 request?
During a bus reset, crg_udc_reinit() and crg_udc_prepare_for_setup() reset
crg_udc->setup_status to WAIT_FOR_SETUP and reinitialize the EP0 ring, but
crg_udc_reinit() only increments udc_ep[i].generation for i >= 2 and leaves
udc_ep[0].generation and crg_udc->setup_tag unchanged. When crg_udc_ep_queue()
reacquires udc_lock, these checks pass because setup_status is not
revalidated and udc_ep[0].generation was not bumped, allowing the stale EP0
request to be queued onto the EP0 ring while setup_status is WAIT_FOR_SETUP.
[ ... ]
> +static int crg_udc_ep_set_wedge(struct usb_ep *_ep)
> +{
> + struct crg_udc_ep *udc_ep_ptr;
> + int status;
> + unsigned long flags = 0;
> + struct crg_gadget_dev *crg_udc;
> +
> + if (!_ep)
> + return -EINVAL;
> +
> + udc_ep_ptr = container_of(_ep, struct crg_udc_ep, usb_ep);
> + crg_udc = udc_ep_ptr->crg_udc;
> +
> + spin_lock_irqsave(&crg_udc->udc_lock, flags);
> +
> + if (!udc_ep_ptr->desc) {
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return -EINVAL;
> + }
> +
> + udc_ep_ptr->wedge = 1;
> +
> + status = crg_udc_ep_halt(udc_ep_ptr, 1, 1, &flags);
[Severity: Medium]
Should crg_udc_ep_set_wedge() check for a non-empty queue on IN endpoints and
return -EAGAIN, just like crg_udc_ep_set_halt() does?
Without that check, calling usb_ep_set_wedge() on an IN endpoint with pending
requests sets udc_ep_ptr->wedge = 1 and calls crg_udc_ep_halt(), which halts
the endpoint and (at SuperSpeed) aborts queued IN transfers via
crg_udc_ep_nuke(). Also, if crg_udc_ep_halt() returns an error,
udc_ep_ptr->wedge remains set to 1.
[ ... ]
> +static int crg_udc_queue_trbs(struct crg_udc_ep *udc_ep_ptr,
> + struct crg_udc_request *udc_req_ptr, bool b_isoc,
> + u32 xfer_ring_size, u32 num_trbs_needed,
> + u64 buffer_length)
> +{
[ ... ]
> + if (num_trbs_ava < td_trbs) {
> + if (b_isoc)
> + return -EBUSY;
> + trace_crg_udc_no_room_on_ring(
> + "deferring TRBs to next doorbell");
> + count = num_trbs_ava;
> + full_td = false;
> + udc_ep_ptr->tran_ring_full = true;
> + need_zlp = false;
> + } else {
> + count = td_trbs;
> + }
> +
> + if (!count)
> + return -ENOSPC;
> +
> + td_size = count - 1;
> +
> + if (is_sg && !udc_req_ptr->sg && crg_udc_sg_init(udc_req_ptr))
> + return -EINVAL;
> +
> + u8 original_pcs = udc_ep_ptr->pcs;
> +
> + for (i = 0; i < count; i++) {
[ ... ]
> + trace_crg_udc_queue_trb(udc_ep_ptr, enq_pt);
> +
> + if (is_sg && !crg_udc_sg_advance(udc_req_ptr, trb_len))
> + return -EINVAL;
> +
> + td_size--;
> + enq_pt++;
> + j++;
> +
> + if (enq_pt == udc_ep_ptr->last_trb) {
> + dma_wmb();
> + le32p_replace_bits(&enq_pt->dw3, udc_ep_ptr->pcs,
> + TRB_CYCLE_BIT_MASK);
> + udc_ep_ptr->pcs ^= 1;
> + enq_pt = udc_ep_ptr->first_trb;
> + }
> + }
[Severity: Medium]
If crg_udc_queue_trbs() returns an error after setting
udc_ep_ptr->tran_ring_full = true or partway through the TRB loop, doesn't
that leave the ring state inconsistent?
When 0 < num_trbs_ava < td_trbs, udc_ep_ptr->tran_ring_full is set to true
before crg_udc_sg_init() and before the loop; if an error occurs while the
ring was previously empty, tran_ring_full stays true with no TRBs queued, so
subsequent crg_udc_ep_queue() calls will never build TDs or ring the doorbell.
Similarly, if a later loop iteration fails after writing TRBs with active
cycle bits or toggling the Link TRB cycle bit and udc_ep_ptr->pcs,
udc_ep_ptr->enq_pt is not updated and those changes are not rolled back.
> + dma_wmb();
> + le32p_replace_bits(&udc_ep_ptr->enq_pt->dw3, original_pcs,
> + TRB_CYCLE_BIT_MASK);
> +
> + if (!udc_req_ptr->trbs_needed)
> + udc_req_ptr->first_trb = udc_ep_ptr->enq_pt;
> +
> + udc_ep_ptr->enq_pt = enq_pt;
> + if (count == num_trbs_ava)
> + udc_ep_ptr->tran_ring_full = true;
> + udc_req_ptr->buff_len_left = buffer_length;
> + udc_req_ptr->td_pending = true;
> + if (buffer_length)
> + udc_req_ptr->trbs_needed = num_trbs_needed - count;
> + else
> + udc_req_ptr->trbs_needed = 0;
> +
> + /* Track the last TRB of the current TD, even for multi-TD requests. */
> + if (udc_ep_ptr->enq_pt == udc_ep_ptr->first_trb)
> + udc_req_ptr->last_trb = udc_ep_ptr->last_trb - 1;
> + else
> + udc_req_ptr->last_trb = udc_ep_ptr->enq_pt - 1;
[Severity: High]
For a multi-TD request, why is udc_req_ptr->last_trb updated on every TD while
udc_req_ptr->first_trb is only set when !udc_req_ptr->trbs_needed?
When the first TD of a multi-TD request completes in crg_udc_handle_xfer_event()
and crg_udc_build_td() queues the next TD, udc_req_ptr->trbs_needed is
non-zero, so udc_req_ptr->first_trb remains pointing at the retired first TD
while udc_req_ptr->last_trb advances to the new TD.
Once udc_ep_ptr->enq_pt wraps around the transfer ring to or past that stale
first_trb, crg_udc_is_pointer_less_than(udc_req->first_trb, trb_pt, udc_ep) in
is_request_dequeued() returns false for active TRBs, causing valid transfer
completion events to be treated as dequeued and dropped. It also causes
crg_udc_ep_dequeue() and crg_udc_actual_before() to misclassify the active TD
range.
> + return 0;
> +}
> +
> +static int crg_udc_reinit(struct crg_gadget_dev *crg_udc)
> +{
> + struct crg_uccr __iomem *uccr = crg_udc->uccr;
> + u32 i, tmp;
> + struct crg_udc_ep *udc_ep_ptr;
> +
> + crg_udc->setup_status = WAIT_FOR_SETUP;
> + /* Base on Figure 9-1, default USB_STATE is attached */
> + crg_udc->device_state = USB_STATE_RECONNECTING;
> +
> + /* halt all the endpoints */
> +
> + /* disable all the endpoints */
> + tmp = readl(&uccr->ep_enable);
> + writel(tmp, &uccr->ep_enable);
> + if (readl_poll_timeout_atomic(&uccr->ep_enable, tmp, !tmp, 1, 50)) {
> + dev_dbg(crg_udc->dev,
> + "timeout waiting for all endpoints to disable\n");
> + return -ETIMEDOUT;
> + }
> +
> + for (i = 2; i < CRG_NUM_EP_CX; i++) {
> + udc_ep_ptr = &crg_udc->udc_ep[i];
> + udc_ep_ptr->usb_ep.enabled = 0;
> + udc_ep_ptr->generation++;
> + udc_ep_ptr->ep_state = EP_STATE_DISABLED;
> + if (udc_ep_ptr->desc) {
> + udc_ep_ptr->desc = NULL;
> + udc_ep_ptr->comp_desc = NULL;
> + crg_udc_ep_nuke(udc_ep_ptr, -ESHUTDOWN);
> + }
> + udc_ep_ptr->tran_ring_full = false;
> + udc_ep_ptr->ep_state = EP_STATE_DISABLED;
> + }
> + crg_udc->num_enabled_eps = 0;
[Severity: High]
Doesn't clearing udc_ep_ptr->usb_ep.enabled = 0 and setting ep_state =
EP_STATE_DISABLED here in crg_udc_reinit() break gadget function teardown when
udc_reset_work runs on bus reset?
When a bus reset occurs while configured, crg_udc_handle_port_status() calls
crg_udc_reinit() first and then schedules udc_reset_work, which invokes
usb_gadget_udc_reset() -> composite_reset() -> reset_config() -> f->disable().
Standard USB function drivers (such as f_ecm, f_eem, f_rndis, f_ncm, f_acm,
and f_serial) check if (ep->enabled) in their ->disable() callbacks before
tearing down state and freeing requests, and crg_udc_ep_disable() also returns
-EINVAL if ep_state is already EP_STATE_DISABLED. Because usb_ep.enabled was
already cleared in crg_udc_reinit(), those function drivers will skip teardown
and leak requests across bus resets.
[Severity: High]
Also in this loop, crg_udc_ep_nuke() drops crg_udc->udc_lock around each
request's completion callback for endpoint i before endpoints i + 1 ..
CRG_NUM_EP_CX - 1 have been marked disabled in software.
While udc_lock is dropped during crg_udc_ep_nuke() on endpoint i, higher-
numbered endpoints still have ep_state == EP_STATE_RUNNING and desc != NULL
even though their hardware enable bits in uccr->ep_enable were just cleared,
allowing callbacks or concurrent threads to queue TRBs and ring doorbells on
disabled hardware endpoints. And if endpoint i is re-enabled while udc_lock is
dropped, the assignments after crg_udc_ep_nuke() will overwrite its state back
to EP_STATE_DISABLED and reset crg_udc->num_enabled_eps to 0.
[ ... ]
> +int crg_udc_handle_port_status(struct crg_gadget_dev *crg_udc,
> + unsigned long *flags)
> +{
> + struct crg_uccr __iomem *uccr = crg_udc->uccr;
> + bool was_connected = crg_udc->connected;
> + u32 portsc_val;
> + u32 tmp;
> + u32 speed;
> + int ret;
> +
> + /* handle Port Reset */
> + portsc_val = readl(&uccr->portsc);
> + trace_crg_udc_portsc(portsc_val);
> + writel(portsc_val & CRG_U3DC_PORTSC_W1C_MASK, &uccr->portsc);
> +
> + tmp = readl(&uccr->portsc);
> +
> + if (portsc_val & CRG_U3DC_PORTSC_PRC) {
> + tmp = readl(&uccr->portsc);
> + if (tmp & CRG_U3DC_PORTSC_PRC) {
> + dev_dbg(crg_udc->dev, "PRC is still set\n");
> + } else if (tmp & CRG_U3DC_PORTSC_PR) {
> + /* first port status change event for port reset */
> + dev_dbg(crg_udc->dev,
> + "PRC is not set, but PR is set\n");
> + } else {
> + if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) != 0 ||
> + (!(tmp & CRG_U3DC_PORTSC_PED))) {
> + dev_dbg(crg_udc->dev, "portsc=0x%x no PED\n",
> + tmp);
> + return 0;
> + }
[ ... ]
> + /* Connection changes must not mask independent link-state events. */
> + if (portsc_val & CRG_U3DC_PORTSC_CSC) {
> + tmp = readl(&uccr->portsc);
> + if ((tmp & (CRG_U3DC_PORTSC_CCS | CRG_U3DC_PORTSC_PP)) ==
> + (CRG_U3DC_PORTSC_CCS | CRG_U3DC_PORTSC_PP)) {
> + if (FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) != 0 ||
> + (!(tmp & CRG_U3DC_PORTSC_PED)))
> + return 0;
> +
> + switch (FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, tmp)) {
> + case CRG_U3DC_PORTSC_SPEED_SSP:
> + speed = USB_SPEED_SUPER_PLUS;
> + break;
> + case CRG_U3DC_PORTSC_SPEED_SS:
> + speed = USB_SPEED_SUPER;
> + break;
> + case CRG_U3DC_PORTSC_SPEED_HS:
> + speed = USB_SPEED_HIGH;
> + break;
> +
> + case CRG_U3DC_PORTSC_SPEED_FS:
> + speed = USB_SPEED_FULL;
> + break;
> +
> + case CRG_U3DC_PORTSC_SPEED_LS:
> + speed = USB_SPEED_LOW;
> + break;
> + default:
> + speed = USB_SPEED_UNKNOWN;
> + return 0;
> + }
> + crg_udc->gadget.speed = speed;
> +
> + crg_udc_update_ep0_maxpacketsize(crg_udc);
> +
> + crg_udc->connected = 1;
> +
> + if (crg_udc->device_state < USB_STATE_RECONNECTING)
> + crg_udc_enable_setup(crg_udc);
> +
> + } else if (!(tmp & CRG_U3DC_PORTSC_CCS)) {
> + bool cable_connected;
> + int ccs_drop_ignore = 0;
> +
> + if ((FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) == 0x0) &&
> + (FIELD_GET(CRG_U3DC_PORTSC_SPEED_MASK, tmp) <
> + CRG_U3DC_PORTSC_SPEED_SS)) {
> + ccs_drop_ignore = 1;
> + dev_err(crg_udc->dev, "ccs glitch detect on HS/FS!\n");
> + }
> +
> + if (!ccs_drop_ignore)
> + crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
> +
> + cable_connected = crg_udc_vbus_connected(crg_udc);
> + if (cable_connected && !ccs_drop_ignore) {
> + int ret;
> +
> + if (was_connected)
> + crg_udc_notify_disconnect(crg_udc, flags);
> + ret = crg_udc_reinit(crg_udc);
> + if (ret)
> + return ret;
> + crg_udc->connected = 0;
> + } else if (!cable_connected) {
> + dev_dbg(crg_udc->dev,
> + "cable disconnected, rst controller\n");
> + if (was_connected)
> + crg_udc_notify_disconnect(crg_udc, flags);
> + ret = crg_udc_reset(crg_udc, flags);
> + if (!ret)
> + ret = crg_udc_reset_data_struct(crg_udc);
> + if (!ret)
> + ret = crg_udc_init_ep0(crg_udc);
> + if (ret)
> + dev_err(crg_udc->dev,
> + "controller reset failed on disconnect: %d\n",
> + ret);
> + else
> + crg_udc->resetting = false;
> + crg_udc->connected = 0;
> +
> + crg_udc->device_state = USB_STATE_ATTACHED;
> + return -ECONNRESET;
> + }
> + }
> + }
> + /* handle Port Connection Change end */
> +
> + if (portsc_val & CRG_U3DC_PORTSC_PLC) {
[Severity: High]
As reported by Sashiko on v3 [1], can early returns in the PRC and CSC blocks
drop concurrent CSC or PLC events that were already cleared in uccr->portsc?
At the top of crg_udc_handle_port_status(), writel(portsc_val &
CRG_U3DC_PORTSC_W1C_MASK, &uccr->portsc) clears all W1C bits (including CSC
and PLC) in hardware. However, if FIELD_GET(CRG_U3DC_PORTSC_PLS_MASK, tmp) != 0
or !(tmp & CRG_U3DC_PORTSC_PED) (or speed is unknown) in the PRC block or the
CSC connect block, the function executes return 0 instead of falling through
to the remaining status-change checks. For example, if a disconnect or link
error aborts a port reset so both PRC and CSC/PLC are set and PED is 0, or if
both CSC and PLC (with PLS == 0xf) are set, returning 0 skips the CSC and PLC
handlers after their hardware status bits have already been cleared.
Also, in the CSC disconnect branches, crg_udc_notify_disconnect() drops
udc_lock while crg_udc->connected is still 1 and before crg_udc_reinit() or
crg_udc_reset() runs, without revalidating controller state after reacquiring
udc_lock.
[ ... ]
> +int crg_udc_handle_xfer_event(struct crg_gadget_dev *crg_udc,
> + struct crg_event_trb *event, unsigned long *flags)
> +{
[ ... ]
> + if (comp_code != CMPL_CODE_STOPPED &&
> + comp_code != CMPL_CODE_STOPPED_LENGTH_INVALID &&
> + comp_code != CMPL_CODE_DISABLED &&
> + comp_code != CMPL_CODE_DISABLED_LENGTH_INVALID &&
> + comp_code != CMPL_CODE_HALTED &&
> + comp_code != CMPL_CODE_HALTED_LENGTH_INVALID)
> + crg_udc_update_dequeue_pt(event, udc_ep_ptr);
> +
> + if (is_request_dequeued(crg_udc, udc_ep_ptr, event)) {
> + trbs_dequeued = true;
> + goto queue_more_trbs;\n> + }
[Severity: High]
Doesn't calling crg_udc_update_dequeue_pt() before is_request_dequeued()
corrupt udc_ep_ptr->deq_pt when a stale event arrives for an already-dequeued
request or an aborted EP0 transfer?
1. When crg_udc_ep_dequeue() cancels an active request, it advances
udc_ep_ptr->deq_pt to next_req->first_trb or udc_ep_ptr->enq_pt. If the
hardware had already posted a transfer event for the cancelled request before
CRG_CMD_STOP_EP completed, calling crg_udc_update_dequeue_pt() before
is_request_dequeued() rewinds udc_ep_ptr->deq_pt back to the stale event's
TRB + 1.
2. In is_request_dequeued(), if udc_ep->queue is empty (because the cancelled
request was the only request on the endpoint), is_request_dequeued() returns
false instead of true:
if (udc_ep->dci == 0 || list_empty(&udc_ep->queue))
return false;
That leaves udc_ep_ptr->deq_pt pointing into the middle of an empty ring and
allows a stale error event to fall through and halt the endpoint via
set_ep_halt().
3. On EP0, CMPL_CODE_SETUP_TAG_MISMATCH is not excluded from
crg_udc_update_dequeue_pt(), so a stale mismatch event arriving after
crg_udc_abort_ep0_for_setup() resets the EP0 ring will overwrite
udc_ep[0].deq_pt and break the enq_pt == dq_pt check in crg_udc_queue_ctrl().
> + switch (comp_code) {
> + case CMPL_CODE_SUCCESS:
[ ... ]
> + if (!FIELD_GET(TRB_CHAIN_BIT_MASK, le32_to_cpu(trb->dw3))) {
> + udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
> + struct crg_udc_request,
> + queue);
> + trbs_dequeued = true;
> + udc_req_ptr->td_pending = false;
> +
> + if (udc_req_ptr->all_trbs_queued) {
> + udc_req_ptr->usb_req.actual =
> + udc_req_ptr->usb_req.length -
> + udc_req_ptr->buff_len_left;
> + residual = FIELD_GET(EVE_TRB_TRAN_LEN_MASK,
> + le32_to_cpu(event->dw2));
> + if (residual > udc_req_ptr->usb_req.actual)
> + residual = 0;
> + udc_req_ptr->usb_req.actual -= residual;
> + crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
> +
> + if (dci == 0 && udc_ep_ptr->desc &&
> + usb_endpoint_xfer_control(udc_ep_ptr->desc))
> + crg_udc_ep0_req_complete(udc_ep_ptr,
> + flags);
> + } else {
> + int ret;
> +
> + ret = crg_udc_build_td(udc_ep_ptr, udc_req_ptr);
> + if (ret == -ENOSPC || ret == -EBUSY) {
> + /* A previous TD completed, but this next TD
> + * could not yet be published. Retry later.
> + */
> + udc_ep_ptr->tran_ring_full = true;
> + } else if (ret) {
> + udc_req_ptr->usb_req.actual =
> + udc_req_ptr->usb_req.length -
> + udc_req_ptr->buff_len_left;
> + crg_udc_req_done(udc_ep_ptr, udc_req_ptr,
> + ret);
> + }
> + }
> + }
> + break;
> +
> + case CMPL_CODE_SHORT_PKT:
> + if (list_empty(&udc_ep_ptr->queue))
> + break;
> +
> + udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
> + struct crg_udc_request, queue);
> + trb = crg_udc_tran_trb_dma_to_virt(
> + udc_ep_ptr,
> + (u64)le32_to_cpu(event->dw0) |
> + ((u64)le32_to_cpu(event->dw1) << 32));
> + if (!trb)
> + break;
> +
> + residual = FIELD_GET(EVE_TRB_TRAN_LEN_MASK,
> + le32_to_cpu(event->dw2));
> + udc_req_ptr->usb_req.actual =
> + crg_udc_actual_before(udc_ep_ptr, udc_req_ptr, trb) +
> + FIELD_GET(TRB_TRANSFER_LEN_MASK,
> + le32_to_cpu(trb->dw2)) -
> + min_t(u32, residual,
> + FIELD_GET(TRB_TRANSFER_LEN_MASK,
> + le32_to_cpu(trb->dw2)));
> + udc_req_ptr->usb_req.actual = min(udc_req_ptr->usb_req.actual,
> + udc_req_ptr->usb_req.length);
> +
> + /* Update the ring before dropping udc_lock for giveback. */
> + trbs_dequeued = true;
> + crg_udc_advance_dequeue_pt(udc_ep_ptr);
> + if (udc_req_ptr->usb_req.short_not_ok &&
> + usb_endpoint_dir_out(udc_ep_ptr->desc))
> + crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EREMOTEIO);
> + else
> + crg_udc_req_done(udc_ep_ptr, udc_req_ptr, 0);
[Severity: High]
Isn't crg_udc_advance_dequeue_pt(udc_ep_ptr) called here while udc_req_ptr is
still at the head of udc_ep_ptr->queue?
In drivers/usb/gadget/udc/crg/crg-udc-ring.c, crg_udc_advance_dequeue_pt()
sets udc_ep->deq_pt from udc_ep->queue.next:
if (!list_empty(&udc_ep->queue)) {
udc_req = list_entry(udc_ep->queue.next, struct crg_udc_request,
queue);
if (udc_req->first_trb)
udc_ep->deq_pt = udc_req->first_trb;
Because udc_req_ptr is not unlinked from udc_ep_ptr->queue until
list_del_init(&udc_req->queue) inside crg_udc_req_done(), calling
crg_udc_advance_dequeue_pt() before udc_req_ptr is removed rewinds
udc_ep_ptr->deq_pt back to the completing request's own first_trb instead of
advancing it to the next request or udc_ep_ptr->enq_pt. That leaks TRB slots
in crg_udc_room_on_ring() on every short packet until the ring wedges with
-ENOSPC, and breaks subsequent EP0 transfers that require enq_pt == dq_pt in
crg_udc_queue_ctrl().
> + if (dci == 0 && udc_ep_ptr->desc &&
> + crg_udc->ep0_state == EP0_STATE_IDLE &&
> + usb_endpoint_xfer_control(udc_ep_ptr->desc))
> + crg_udc_ep0_req_complete(udc_ep_ptr, flags);
> + break;
[ ... ]
> + case CMPL_CODE_BABBLE_DETECTED_ERR:
> + case CMPL_CODE_DATA_BUFFER_ERR:
> + case CMPL_CODE_INVALID_STREAM_TYPE_ERR:
> + case CMPL_CODE_RING_UNDERRUN:
> + case CMPL_CODE_RING_OVERRUN:
> + case CMPL_CODE_EVENT_RING_FULL_ERR:
> + case CMPL_CODE_ISOCH_BUFFER_OVERRUN:
> + case CMPL_CODE_USB_TRANS_ERR:
> + case CMPL_CODE_TRB_ERR:
> + dev_err(crg_udc->dev, "transfer error, completion code 0x%x\n",
> + comp_code);
> + if (!list_empty(&udc_ep_ptr->queue)) {
> + udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
> + struct crg_udc_request,
> + queue);
> + crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EIO);
> + trbs_dequeued = true;
> + }
> + if (dci)
> + set_ep_halt(crg_udc, dci, flags);
> + break;
[Severity: High]
Doesn't calling crg_udc_req_done() before set_ep_halt() unmap the request's
DMA buffer and drop udc_lock while the endpoint is still running?
If the completion callback (or another CPU while udc_lock is dropped in
crg_udc_req_done()) queues a new request on udc_ep_ptr, crg_udc_ep_queue()
sees ep_state == EP_STATE_RUNNING and rings the doorbell; then when
crg_udc_req_done() returns, set_ep_halt() immediately halts the endpoint and
nukes the newly queued request with -ECONNRESET. Furthermore, set_ep_halt()
does not check whether crg_udc->resetting or udc_ep_ptr->generation changed
while udc_lock was dropped.
> +
> + case CMPL_CODE_STOPPED:
> + case CMPL_CODE_STOPPED_LENGTH_INVALID:
> + case CMPL_CODE_HALTED:
> + case CMPL_CODE_HALTED_LENGTH_INVALID:
> + case CMPL_CODE_DISABLED:
> + case CMPL_CODE_DISABLED_LENGTH_INVALID:
> + break;
> +
> + default:
> + dev_err(crg_udc->dev,
> + "unsupported transfer completion code 0x%x\n",
> + comp_code);
> + if (!list_empty(&udc_ep_ptr->queue)) {
> + udc_req_ptr = list_first_entry(&udc_ep_ptr->queue,
> + struct crg_udc_request,
> + queue);
> + crg_udc_req_done(udc_ep_ptr, udc_req_ptr, -EIO);
> + trbs_dequeued = true;
> + }
> + break;
> + }
> +
> +queue_more_trbs:
> + /* Even if the ring was not full, a later request may be waiting
> + * behind a TD from an earlier request (32 TRBs per TD).
> + */
> + if (trbs_dequeued && dci != 0 &&
> + udc_ep_ptr->ep_state == EP_STATE_RUNNING &&
> + !crg_udc->resetting) {
> + udc_ep_ptr->tran_ring_full = false;
> + if (crg_udc_queue_pending_trbs(udc_ep_ptr))
> + udc_ep_ptr->tran_ring_full = true;
> + }
[Severity: Medium]
As reported by Sashiko on v3 [1], doesn't queue_more_trbs clobber
udc_ep_ptr->tran_ring_full when the !udc_req_ptr->all_trbs_queued branch under
CMPL_CODE_SUCCESS just filled the ring?
When an intermediate TD of a multi-TD request completes, that branch sets
trbs_dequeued = true and calls crg_udc_build_td(udc_ep_ptr, udc_req_ptr). If
that TD consumes the remaining space on the ring (or returns -ENOSPC/-EBUSY
when td_pending is already set), udc_ep_ptr->tran_ring_full is set to true.
Execution then reaches queue_more_trbs, which sets udc_ep_ptr->tran_ring_full
= false and calls crg_udc_queue_pending_trbs(udc_ep_ptr). Because udc_req_ptr
now has td_pending == true or all_trbs_queued == true,
crg_udc_queue_pending_trbs() returns 0 without re-detecting the full ring,
leaving tran_ring_full false so the next crg_udc_ep_queue() call attempts
crg_udc_build_td() and fails with -ENOSPC.
> +
> + return 0;
> +}
> +
> +int crg_udc_handle_event(struct crg_gadget_dev *crg_udc,
> + struct crg_event_trb *event, unsigned long *flags)
> +{
> + int ret;
> +
> + trace_crg_udc_event_trb(event);
> +
> + switch (FIELD_GET(EVE_TRB_TYPE_MASK, le32_to_cpu(event->dw3))) {
> + case TRB_TYPE_EVT_PORT_STATUS_CHANGE:
> + if (crg_udc->device_state == USB_STATE_RECONNECTING) {
> + crg_udc->portsc_on_reconnecting = 1;
> + break;
> + }
> + /* 1.Port Reset 2.Port Connection Change 3.Port Link Change */
> + ret = crg_udc_handle_port_status(crg_udc, flags);
> + if (ret)
> + return ret;
> +
> + break;
> + case TRB_TYPE_EVT_TRANSFER:
> + if (crg_udc->device_state < USB_STATE_RECONNECTING) {
> + dev_err(crg_udc->dev,
> + "Xfer compl event rcved when dev state=%d !\n",
> + crg_udc->device_state);
> + break;
> + }
> +
> + ret = crg_udc_handle_xfer_event(crg_udc, event, flags);
> + if (ret)
> + return ret;
> + break;
> + case TRB_TYPE_EVT_SETUP_PKT: {
> + struct usb_ctrlrequest setup;
> + u32 dw0 = le32_to_cpu(event->dw0);
> + u32 dw1 = le32_to_cpu(event->dw1);
> + u8 setup_tag;
> +
> + BUILD_BUG_ON(sizeof(struct usb_ctrlrequest) != 8);
> + setup.bRequestType = dw0 & 0xff;
> + setup.bRequest = (dw0 >> 8) & 0xff;
> + setup.wValue = cpu_to_le16((dw0 >> 16) & 0xffff);
> + setup.wIndex = cpu_to_le16(dw1 & 0xffff);
> + setup.wLength = cpu_to_le16((dw1 >> 16) & 0xffff);
> +
> + setup_tag = FIELD_GET(EVE_TRB_SETUP_TAG_MASK,
> + le32_to_cpu(event->dw3));
> + if (crg_udc->setup_status != WAIT_FOR_SETUP) {
> + /* A new SETUP aborts the previous control transfer. The old
> + * EP0 request must be completed and its TRBs discarded before
> + * the new setup is handed to the gadget driver.
> + */
> + if (crg_udc_abort_ep0_for_setup(crg_udc)) {
> + crg_udc_set_ep0_halt(crg_udc);
> + break;
> + }
> + }
> +
> + crg_udc_handle_setup_pkt(crg_udc, &setup, setup_tag, flags);
> +
> + break;
> + }
[Severity: High]
What happens if a TRB_TYPE_EVT_SETUP_PKT event is processed here while
crg_udc->device_state == USB_STATE_RECONNECTING?
When a bus reset occurs after CRG_U3DC_CFG1_SETUP_EVENT_EN has been enabled,
crg_udc_reinit() sets crg_udc->device_state = USB_STATE_RECONNECTING and
defers EP0 reinitialization (crg_udc_prepare_for_setup()) until the event ring
becomes empty. If a SETUP packet is already in the event ring during the same
drain loop, crg_udc_handle_setup_pkt() runs while device_state is still
USB_STATE_RECONNECTING: setaddressrequest() rejects USB_REQ_SET_ADDRESS and
stalls EP0 because device_state is neither USB_STATE_DEFAULT nor
USB_STATE_ADDRESS, and any control transfer queued for GET_DESCRIPTOR will be
immediately stopped and nuked with -ESHUTDOWN when crg_udc_prepare_for_setup()
runs at the end of the interrupt.
[Severity: High]
In addition, shouldn't the abort condition here also check
crg_udc->ep0_state != EP0_STATE_IDLE (not just
crg_udc->setup_status != WAIT_FOR_SETUP)?
crg_udc_abort_ep0_for_setup() sets crg_udc->setup_status = WAIT_FOR_SETUP at
the very beginning of the function. If crg_udc_abort_ep0_for_setup(),
crg_udc_prepare_for_setup(), or crg_udc_build_ep0_status() fails,
crg_udc->ep0_state is set to EP0_STATE_ERROR while setup_status is
WAIT_FOR_SETUP. On the next SETUP packet, checking only
setup_status != WAIT_FOR_SETUP skips crg_udc_abort_ep0_for_setup(), leaving
ep0_state permanently stuck in EP0_STATE_ERROR and causing crg_udc_ep_queue()
to reject all future EP0 requests with -ECONNRESET.
[ ... ]
> +int crg_udc_ep_halt(struct crg_udc_ep *udc_ep_ptr, int halt, int ignore_wedge,
> + unsigned long *flags)
> +{
> + struct crg_gadget_dev *crg_udc = udc_ep_ptr->crg_udc;
> + struct crg_uccr __iomem *uccr = crg_udc->uccr;
> + enum crg_ep_state ep_state;
> + bool reset_seq_only = false;
> + int do_halt;
> + u32 param0;
> + u32 tmp;
> +
> + if (crg_udc->resetting)
> + return -ESHUTDOWN;
> + if (!udc_ep_ptr->desc) {
> + dev_err(crg_udc->dev, "NULL desc\n");
> + return -EINVAL;
> + }
> +
> + if (usb_endpoint_xfer_isoc(udc_ep_ptr->desc)) {
> + dev_err(crg_udc->dev, "Isoc ep, halt not supported\n");
> + return -EOPNOTSUPP;
> + }
> +
> + if (udc_ep_ptr->dci == 0)
> + return 0;
[Severity: High]
When a gadget driver calls usb_ep_set_halt(gadget->ep0, 1) to stall EP0 (for
example from __ffs_ep0_stall(), uvc_send_response(), or a Control OUT
data-stage completion callback), doesn't returning 0 for dci == 0 turn the
EP0 stall into a silent no-op instead of calling crg_udc_set_ep0_halt()?
Furthermore, in crg_udc_handle_xfer_event() under CMPL_CODE_SUCCESS, after
crg_udc_req_done() runs the EP0 data-stage completion callback, it calls
crg_udc_ep0_req_complete(udc_ep_ptr, flags) without checking that
crg_udc->ep0_state == EP0_STATE_IDLE and udc_ep_ptr->ep_state ==
EP_STATE_RUNNING, which queues an ACK status stage even if the callback
stalled or aborted EP0.
> + ep_state = crg_udc_get_ep_state(crg_udc, udc_ep_ptr->dci);
> +
> + if (ep_state == EP_STATE_DISABLED)
> + return 0;
> +
> + if (ep_state == EP_STATE_HALTED) {
> + if (halt != 0)
> + return 0;
> +
> + /* want unhalt an halted ep */
> + if (udc_ep_ptr->wedge && !ignore_wedge) {
> + do_halt = -1;
> + reset_seq_only = true;
> + } else {
> + do_halt = 0;
> + }
> +
> + } else {
> + /* ep state == running or stopped */
> + if (halt != 0) {
> + /* want halt a running ep */
> + do_halt = 1;
> + } else {
> + /* reset a running ep */
> + do_halt = 0;
> + reset_seq_only = true;
> + }
> + }
> +
> + trace_crg_udc_ep_halt(udc_ep_ptr, do_halt == 1, udc_ep_ptr->wedge);
> +
> + param0 = BIT(udc_ep_ptr->dci);
> + if (do_halt == 1) {
> + /* setting ep to halt */
> + if (crg_udc_issue_command(crg_udc, CRG_CMD_SET_HALT, param0, 0))
> + return -EIO;
> + if (readl_poll_timeout_atomic(&uccr->ep_running, tmp,
> + !(tmp & param0), 10, 10000))
> + dev_warn(crg_udc->dev, "ep %d halt wait timed out\n",
> + udc_ep_ptr->dci);
> +
> + if (crg_udc->gadget.speed >= USB_SPEED_SUPER) {
> + udc_ep_ptr->deq_pt = udc_ep_ptr->enq_pt;
> + udc_ep_ptr->tran_ring_full = false;
> + }
> + udc_ep_ptr->ep_state = EP_STATE_HALTED;
> + if (crg_udc->gadget.speed >= USB_SPEED_SUPER)
> + crg_udc_ep_nuke(udc_ep_ptr, -ECONNRESET);
[Severity: High]
If readl_poll_timeout_atomic(&uccr->ep_running, ...) times out here, shouldn't
crg_udc_ep_halt() return -ETIMEDOUT (just like set_ep_halt() and
crg_udc_ep_dequeue() do) instead of proceeding to crg_udc_ep_nuke()?
Unmapping request DMA buffers and giving requests back to the gadget driver
via crg_udc_ep_nuke() while the endpoint bit in uccr->ep_running is still set
risks DMA use-after-free while the controller still owns the buffers.
[1] https://lore.kernel.org/r/20261020-axiado-ax3000-usb-device-controller-v3-5-3012ecfc0a93@axiado.com
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261030-axiado-ax3000-usb-device-controller-v4-0-c5267b36a777@axiado.com?part=5
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v4 6/6] usb: gadget: udc: crg: Add core driver and build integration
2026-10-30 8:10 [PATCH v4 0/6] Axiado AX3000 and AX3005 USB Device Controller (UDC) Petar Stepanovic
` (4 preceding siblings ...)
2026-10-30 8:10 ` [PATCH v4 5/6] usb: gadget: udc: crg: Add gadget and endpoint operations Petar Stepanovic
@ 2026-10-30 8:10 ` Petar Stepanovic
2026-10-30 8:46 ` sashiko-bot
2026-10-30 9:34 ` Philipp Zabel
5 siblings, 2 replies; 14+ messages in thread
From: Petar Stepanovic @ 2026-10-30 8:10 UTC (permalink / raw)
To: Naveen Kumar Rajgiri Bassappa, Prasad Bolisetty,
Greg Kroah-Hartman, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Tzu-Hao Wei, Karthikeyan Mitran, Philipp Zabel
Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
Petar Stepanovic, Krutik Shah
This patch introduces the core probe/remove functions for the Axiado
AX3000/AX3005 UDC driver, and wires up the directory into the build
system via Kconfig and Makefile.
Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
Co-developed-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Krutik Shah <krutikshah@axiado.com>
Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
---
drivers/usb/gadget/udc/Kconfig | 2 +
drivers/usb/gadget/udc/Makefile | 1 +
drivers/usb/gadget/udc/crg/Kconfig | 16 +
drivers/usb/gadget/udc/crg/Makefile | 9 +
drivers/usb/gadget/udc/crg/crg-udc-core.c | 723 ++++++++++++++++++++++++++++++
5 files changed, 751 insertions(+)
diff --git a/drivers/usb/gadget/udc/Kconfig b/drivers/usb/gadget/udc/Kconfig
index 7ddbc2606686..72a5a8a04982 100644
--- a/drivers/usb/gadget/udc/Kconfig
+++ b/drivers/usb/gadget/udc/Kconfig
@@ -421,6 +421,8 @@ source "drivers/usb/gadget/udc/aspeed-vhub/Kconfig"
source "drivers/usb/gadget/udc/cdns2/Kconfig"
+source "drivers/usb/gadget/udc/crg/Kconfig"
+
#
# LAST -- dummy/emulated controller
#
diff --git a/drivers/usb/gadget/udc/Makefile b/drivers/usb/gadget/udc/Makefile
index 1b9b1a4f9c57..227d2a0b525b 100644
--- a/drivers/usb/gadget/udc/Makefile
+++ b/drivers/usb/gadget/udc/Makefile
@@ -38,3 +38,4 @@ obj-$(CONFIG_USB_ASPEED_UDC) += aspeed_udc.o
obj-$(CONFIG_USB_BDC_UDC) += bdc/
obj-$(CONFIG_USB_MAX3420_UDC) += max3420_udc.o
obj-$(CONFIG_USB_CDNS2_UDC) += cdns2/
+obj-$(CONFIG_USB_CRG_UDC) += crg/
diff --git a/drivers/usb/gadget/udc/crg/Kconfig b/drivers/usb/gadget/udc/crg/Kconfig
new file mode 100644
index 000000000000..7167c4943389
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/Kconfig
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: GPL-2.0
+config USB_CRG_UDC
+ tristate "Corigine / Axiado USB Device Controller"
+ depends on USB_GADGET
+ depends on HAS_DMA
+ depends on ARCH_AXIADO || COMPILE_TEST
+ help
+ Say Y here to support the USB device controller found on Axiado AX3000
+ and AX3005 SoCs, which is based on the Corigine USB 3.1 IP.
+
+ This controller supports High-Speed and SuperSpeed.
+
+ Say "y" to link the driver statically, or "m" to build a
+ dynamically linked module called "crg-udc.ko".
+
+ If unsure, say N.
diff --git a/drivers/usb/gadget/udc/crg/Makefile b/drivers/usb/gadget/udc/crg/Makefile
new file mode 100644
index 000000000000..96b209fb59fc
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/Makefile
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: GPL-2.0
+
+# define_trace.h needs to know how to find our trace header
+CFLAGS_crg-udc-trace.o := -I$(src)
+
+obj-$(CONFIG_USB_CRG_UDC) += crg-udc.o
+
+crg-udc-y := crg-udc-core.o crg-udc-ring.o crg-udc-ep0.o crg-udc-gadget.o
+crg-udc-$(CONFIG_TRACING) += crg-udc-trace.o
diff --git a/drivers/usb/gadget/udc/crg/crg-udc-core.c b/drivers/usb/gadget/udc/crg/crg-udc-core.c
new file mode 100644
index 000000000000..33be5c920ebe
--- /dev/null
+++ b/drivers/usb/gadget/udc/crg/crg-udc-core.c
@@ -0,0 +1,723 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * Axiado AX3000/AX3005 USB Device Controller driver.
+ *
+ * Copyright (c) 2019 Corigine Inc.
+ * Copyright (c) 2021-2026 Axiado Corporation.
+ */
+#include <linux/clk.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/iopoll.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/phy/phy.h>
+#include <linux/platform_device.h>
+#include <linux/pm.h>
+#include <linux/property.h>
+#include <linux/reset.h>
+#include <linux/slab.h>
+
+#include "crg-udc.h"
+
+static int crg_gadget_irq_init(struct platform_device *pdev,
+ struct crg_gadget_dev *crg_udc);
+static void crg_udc_phy_exit(void *data);
+
+static void crg_udc_gadget_release(struct device *dev)
+{
+ struct crg_gadget_dev *crg_udc;
+
+ crg_udc = container_of(dev, struct crg_gadget_dev, gadget.dev);
+ kfree(crg_udc);
+}
+
+/*
+ * command related ops
+ */
+int crg_udc_issue_command(struct crg_gadget_dev *crg_udc,
+ enum crg_cmd_type type, u32 param0, u32 param1)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 status;
+ u32 tmp;
+ bool running;
+
+ running = !!(readl(&uccr->control) & CRG_U3DC_CTRL_RUN);
+ /* Only INIT_EP0 may be staged while the controller is stopped. */
+ if (!running && type != CRG_CMD_INIT_EP0)
+ return -ESHUTDOWN;
+
+ /*
+ * A staged command must complete before the command registers can
+ * be reused. Hardware does not consume commands while RUN is clear.
+ */
+ tmp = readl(&uccr->cmd_control);
+ if (tmp & CRG_U3DC_CMD_CTRL_ACTIVE) {
+ if (!running)
+ return -EBUSY;
+ if (readl_poll_timeout_atomic(&uccr->cmd_control, tmp,
+ !(tmp & CRG_U3DC_CMD_CTRL_ACTIVE),
+ 1, 10000))
+ return -ETIMEDOUT;
+ }
+ /* Ensure that everything is written before issuing new command */
+ dma_wmb();
+
+ writel(param0, &uccr->cmd_param0);
+ writel(param1, &uccr->cmd_param1);
+
+ tmp = CRG_U3DC_CMD_CTRL_ACTIVE |
+ FIELD_PREP(CRG_U3DC_CMD_CTRL_TYPE_MASK, type);
+ writel(tmp, &uccr->cmd_control);
+
+ if (!running)
+ return 0;
+
+ if (readl_poll_timeout_atomic(&uccr->cmd_control, tmp,
+ !(tmp & CRG_U3DC_CMD_CTRL_ACTIVE), 1,
+ 1000)) {
+ dev_dbg(crg_udc->dev, "command timed out\n");
+ return -ETIMEDOUT;
+ }
+
+ status = FIELD_GET(CRG_U3DC_CMD_CTRL_STATUS_MASK, tmp);
+ if (status) {
+ dev_dbg(crg_udc->dev, "command failed with status %u\n",
+ status);
+ return -EIO;
+ }
+
+ return 0;
+}
+
+static void crg_udc_clk_disable(void *data)
+{
+ clk_disable_unprepare(data);
+}
+
+void crg_udc_free_resources(struct crg_gadget_dev *crg_udc)
+{
+ int i;
+
+ if (crg_udc->uccr) {
+ u32 val = readl(&crg_udc->uccr->control);
+
+ val &= ~(CRG_U3DC_CTRL_INT_EN | CRG_U3DC_CTRL_RUN);
+ writel(val, &crg_udc->uccr->control);
+ }
+ if (crg_udc->irq >= 0) {
+ devm_free_irq(crg_udc->dev, crg_udc->irq, crg_udc);
+ crg_udc->irq = -1;
+ crg_udc->gadget.irq = 0;
+ }
+
+ for (i = 0; i < CRG_RING_NUM; i++) {
+ struct crg_udc_event *udc_event = &crg_udc->udc_event[i];
+
+ if (udc_event->event_ring.cpu_addr) {
+ dma_free_coherent(crg_udc->dev,
+ udc_event->event_ring.len,
+ udc_event->event_ring.cpu_addr,
+ udc_event->event_ring.dma_addr);
+ udc_event->event_ring.cpu_addr = NULL;
+ udc_event->event_ring.dma_addr = DMA_MAPPING_ERROR;
+ udc_event->event_ring.len = 0;
+ udc_event->evt_dq_pt = NULL;
+ }
+ if (udc_event->erst.cpu_addr) {
+ dma_free_coherent(crg_udc->dev, udc_event->erst.len,
+ udc_event->erst.cpu_addr,
+ udc_event->erst.dma_addr);
+ udc_event->erst.cpu_addr = NULL;
+ udc_event->erst.dma_addr = DMA_MAPPING_ERROR;
+ udc_event->erst.len = 0;
+ udc_event->p_erst = NULL;
+ }
+ }
+
+ if (crg_udc->ep_cx.cpu_addr) {
+ dma_free_coherent(crg_udc->dev, crg_udc->ep_cx.len,
+ crg_udc->ep_cx.cpu_addr,
+ crg_udc->ep_cx.dma_addr);
+ crg_udc->ep_cx.cpu_addr = NULL;
+ crg_udc->ep_cx.dma_addr = DMA_MAPPING_ERROR;
+ crg_udc->ep_cx.len = 0;
+ crg_udc->p_epcx = NULL;
+ }
+
+ for (i = 0; i < CRG_NUM_EP_CX; i++) {
+ struct crg_udc_ep *udc_ep = &crg_udc->udc_ep[i];
+
+ if (udc_ep->tran_ring_info.cpu_addr) {
+ dma_free_coherent(crg_udc->dev,
+ udc_ep->tran_ring_info.len,
+ udc_ep->tran_ring_info.cpu_addr,
+ udc_ep->tran_ring_info.dma_addr);
+ udc_ep->tran_ring_info.cpu_addr = NULL;
+ udc_ep->tran_ring_info.dma_addr = DMA_MAPPING_ERROR;
+ udc_ep->tran_ring_info.len = 0;
+ udc_ep->first_trb = NULL;
+ udc_ep->last_trb = NULL;
+ udc_ep->enq_pt = NULL;
+ udc_ep->deq_pt = NULL;
+ }
+ }
+
+ if (crg_udc->status_req) {
+ crg_udc_free_request(&crg_udc->udc_ep[0].usb_ep,
+ &crg_udc->status_req->usb_req);
+ crg_udc->status_req = NULL;
+ }
+}
+
+static int crg_udc_init_hw(struct crg_gadget_dev *crg_udc)
+{
+ struct device *dev = crg_udc->dev;
+ struct reset_control *rst;
+ int ret;
+
+ crg_udc->clk = devm_clk_get_optional(dev, NULL);
+ if (IS_ERR(crg_udc->clk))
+ return dev_err_probe(dev, PTR_ERR(crg_udc->clk),
+ "failed to get clock\n");
+
+ ret = clk_prepare_enable(crg_udc->clk);
+ if (ret)
+ return dev_err_probe(dev, ret, "failed to enable clock\n");
+
+ ret = devm_add_action_or_reset(dev, crg_udc_clk_disable, crg_udc->clk);
+ if (ret)
+ return ret;
+
+ rst =
+ devm_reset_control_get_optional_exclusive_deasserted(dev, NULL);
+ if (IS_ERR(rst))
+ return dev_err_probe(dev, PTR_ERR(rst),
+ "failed to get reset\n");
+
+ crg_udc->phy = devm_phy_optional_get(dev, NULL);
+ if (IS_ERR(crg_udc->phy))
+ return dev_err_probe(dev, PTR_ERR(crg_udc->phy),
+ "failed to get phy\n");
+
+ ret = phy_init(crg_udc->phy);
+ if (ret)
+ return dev_err_probe(dev, ret, "failed to init phy\n");
+
+ ret = phy_power_on(crg_udc->phy);
+ if (ret) {
+ phy_exit(crg_udc->phy);
+ return dev_err_probe(dev, ret, "failed to power on phy\n");
+ }
+
+ return devm_add_action_or_reset(dev, crg_udc_phy_exit, crg_udc->phy);
+}
+
+static void crg_udc_phy_exit(void *data)
+{
+ struct phy *phy = data;
+
+ phy_power_off(phy);
+ phy_exit(phy);
+}
+
+static int crg_udc_probe(struct platform_device *pdev)
+{
+ unsigned long flags = 0;
+ int ret;
+ int i;
+ struct crg_gadget_dev *crg_udc;
+ const struct crg_udc_priv *priv;
+
+ priv = device_get_match_data(&pdev->dev);
+ if (!priv)
+ return -ENODEV;
+
+ crg_udc = kzalloc_obj(*crg_udc, GFP_KERNEL);
+ if (!crg_udc)
+ return -ENOMEM;
+ usb_initialize_gadget(&pdev->dev, &crg_udc->gadget,
+ crg_udc_gadget_release);
+
+ crg_udc->dev = &pdev->dev;
+ crg_udc->irq = -1;
+
+ ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(64));
+ if (ret) {
+ ret = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32));
+ if (ret) {
+ dev_err(&pdev->dev, "no usable DMA mask\n");
+ goto err_free_resources;
+ }
+ }
+
+ ret = crg_udc_init_hw(crg_udc);
+ if (ret)
+ goto err_free_resources;
+
+ crg_udc->ep0_desc.bLength = USB_DT_ENDPOINT_SIZE;
+ crg_udc->ep0_desc.bDescriptorType = USB_DT_ENDPOINT;
+ crg_udc->ep0_desc.bEndpointAddress = 0;
+ crg_udc->ep0_desc.bmAttributes = USB_ENDPOINT_XFER_CONTROL;
+ crg_udc->ep0_desc.wMaxPacketSize = cpu_to_le16(64);
+
+ crg_udc->u12_forbidden = priv->quirks & CRG_QUIRK_DIS_U1U2_ENTRY;
+ crg_udc->u12_initiate_forbidden = priv->quirks &
+ CRG_QUIRK_DIS_U1U2_INITIATION;
+ crg_udc->reinit_ep0_on_bus_reset = priv->quirks &
+ CRG_QUIRK_REINIT_EP0_ON_RESET;
+
+ spin_lock_init(&crg_udc->udc_lock);
+ INIT_WORK(&crg_udc->udc_reset_work, crg_udc_reset_work);
+ platform_set_drvdata(pdev, crg_udc);
+
+ crg_udc->gadget.ops = &crg_gadget_ops;
+ crg_udc->gadget.ep0 = &crg_udc->udc_ep[0].usb_ep;
+ INIT_LIST_HEAD(&crg_udc->gadget.ep_list);
+ /*
+ * The controller supports up to SuperSpeed. Honour the standard
+ * "maximum-speed" property to cap it; default to the hardware
+ * maximum when the property is absent.
+ */
+ crg_udc->gadget.max_speed = usb_get_maximum_speed(&pdev->dev);
+ if (crg_udc->gadget.max_speed == USB_SPEED_UNKNOWN)
+ crg_udc->gadget.max_speed = USB_SPEED_SUPER;
+ crg_udc->max_speed = crg_udc->gadget.max_speed;
+ crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+ crg_udc->gadget.name = KBUILD_MODNAME;
+ crg_udc->gadget.sg_supported = true;
+ crg_udc->gadget.wakeup_capable = true;
+ crg_udc->connected = 0;
+ crg_udc->dev_addr = 0;
+
+ crg_udc->mmio_virt_base = devm_platform_ioremap_resource(pdev, 0);
+ if (IS_ERR(crg_udc->mmio_virt_base)) {
+ ret = dev_err_probe(&pdev->dev,
+ PTR_ERR(crg_udc->mmio_virt_base),
+ "failed to map registers\n");
+ goto err_put_gadget;
+ }
+
+ /* set controller device role */
+ writel((readl(crg_udc->mmio_virt_base + CRG_UDC_MODE_REG) |
+ CRG_UDC_ROLE_DEVICE),
+ crg_udc->mmio_virt_base + CRG_UDC_MODE_REG);
+ for (i = 0; i < CRG_RING_NUM; i++) {
+ crg_udc->uicr[i] = crg_udc->mmio_virt_base + CRG_UICR_OFFSET +
+ i * CRG_UICR_STRIDE;
+ }
+ crg_udc->uccr = crg_udc->mmio_virt_base + CRG_UCCR_OFFSET;
+
+ /* Initialize endpoint objects before reset, DMA setup or any IRQ. */
+ crg_udc_init_ep_info(crg_udc);
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ ret = crg_udc_reset(crg_udc, &flags);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ if (ret)
+ goto err_free_resources;
+
+ crg_udc_clear_portpm(crg_udc);
+
+ ret = crg_udc_reset_data_struct(crg_udc);
+ if (ret) {
+ dev_err(crg_udc->dev, "crg_udc_reset_data_struct error\n");
+ goto err_free_resources;
+ }
+
+ ret = crg_udc_init_ep0(crg_udc);
+ if (ret)
+ goto err_free_resources;
+
+ crg_udc->udc_ep[0].desc = &crg_udc->ep0_desc;
+ /* Release the reset gate only after EP0 and all event rings exist. */
+ crg_udc->resetting = false;
+
+ ret = crg_gadget_irq_init(pdev, crg_udc);
+ if (ret)
+ goto err_free_resources;
+
+ ret = usb_add_gadget(&crg_udc->gadget);
+ if (ret)
+ goto err_free_resources;
+
+ return 0;
+
+err_free_resources:
+ crg_udc_free_resources(crg_udc);
+err_put_gadget:
+ usb_put_gadget(&crg_udc->gadget);
+ return ret;
+}
+
+static void crg_udc_remove(struct platform_device *pdev)
+{
+ struct crg_gadget_dev *crg_udc = platform_get_drvdata(pdev);
+ u32 tmp;
+
+ /*
+ * The gadget core must notify/unbind the function driver before we
+ * change the controller's visible state or disable VBUS detection.
+ */
+ usb_del_gadget(&crg_udc->gadget);
+ cancel_work_sync(&crg_udc->udc_reset_work);
+ crg_udc_free_resources(crg_udc);
+
+ /* set controller host role */
+ tmp = readl(crg_udc->mmio_virt_base + CRG_UDC_MODE_REG) & ~0x1;
+ writel(tmp, crg_udc->mmio_virt_base + CRG_UDC_MODE_REG);
+
+ platform_set_drvdata(pdev, NULL);
+ usb_put_gadget(&crg_udc->gadget);
+}
+
+static void crg_udc_shutdown(struct platform_device *pdev)
+{
+ struct crg_gadget_dev *crg_udc = platform_get_drvdata(pdev);
+ unsigned long flags;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ crg_udc->pullup_on = false;
+ crg_udc->resetting = true;
+ crg_udc->device_state = USB_STATE_ATTACHED;
+ crg_udc_set_pullup(crg_udc, false);
+ crg_udc_set_vbus_detect(crg_udc, false);
+ if (crg_udc->uccr) {
+ u32 val = readl(&crg_udc->uccr->control);
+
+ val &= ~(CRG_U3DC_CTRL_INT_EN | CRG_U3DC_CTRL_RUN);
+ writel(val, &crg_udc->uccr->control);
+ }
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ if (crg_udc->irq >= 0)
+ synchronize_irq(crg_udc->irq);
+ cancel_work_sync(&crg_udc->udc_reset_work);
+}
+
+static int crg_udc_suspend(struct device *dev)
+{
+ struct crg_gadget_dev *crg_udc = dev_get_drvdata(dev);
+ unsigned long flags;
+
+ /* Stop the pullup while the controller is suspended. */
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (!crg_udc->resetting)
+ crg_udc_set_pullup(crg_udc, false);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return 0;
+}
+
+static int crg_udc_resume(struct device *dev)
+{
+ struct crg_gadget_dev *crg_udc = dev_get_drvdata(dev);
+ unsigned long flags;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+ if (!crg_udc->resetting && crg_udc->gadget_driver && crg_udc->pullup_on)
+ crg_udc_set_pullup(crg_udc, true);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+
+ return 0;
+}
+
+/* Program the speed limit while the controller is stopped. */
+int crg_udc_program_speed(struct crg_gadget_dev *crg_udc)
+{
+ u32 cfg, speed;
+
+ switch (crg_udc->max_speed) {
+ case USB_SPEED_FULL:
+ speed = CRG_U3DC_CFG0_MAXSPEED_FS;
+ break;
+ case USB_SPEED_HIGH:
+ speed = CRG_U3DC_CFG0_MAXSPEED_HS;
+ break;
+ case USB_SPEED_SUPER:
+ speed = CRG_U3DC_CFG0_MAXSPEED_SS;
+ break;
+ case USB_SPEED_SUPER_PLUS:
+ speed = CRG_U3DC_CFG0_MAXSPEED_SSP;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ cfg = readl(&crg_udc->uccr->config0);
+ cfg &= ~CRG_U3DC_CFG0_MAXSPEED_MASK;
+ cfg |= FIELD_PREP(CRG_U3DC_CFG0_MAXSPEED_MASK, speed);
+ writel(cfg, &crg_udc->uccr->config0);
+
+ return 0;
+}
+
+int crg_udc_reset_data_struct(struct crg_gadget_dev *crg_udc)
+{
+ int i;
+ int ret;
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 val;
+
+ crg_udc->ep0_state = EP0_STATE_IDLE;
+
+ val = readl(&uccr->control);
+ val &= (~(CRG_U3DC_CTRL_INT_EN | CRG_U3DC_CTRL_RUN));
+ writel(val, &uccr->control);
+
+ ret = crg_udc_program_speed(crg_udc);
+ if (ret)
+ return ret;
+
+ for (i = 0; i < CRG_RING_NUM; i++) {
+ ret = crg_udc_init_event_ring(crg_udc, i);
+ if (ret)
+ return ret;
+ }
+
+ ret = crg_udc_init_device_context(crg_udc);
+ if (ret)
+ return ret;
+
+ if (!crg_udc->status_req) {
+ struct usb_request *req;
+
+ req = crg_udc_alloc_request(&crg_udc->udc_ep[0].usb_ep,
+ GFP_ATOMIC);
+ if (!req)
+ return -ENOMEM;
+
+ crg_udc->status_req =
+ container_of(req, struct crg_udc_request, usb_req);
+ }
+
+ /* other hw ops */
+ return 0;
+}
+
+static void crg_gadget_handle_interrupt(struct crg_gadget_dev *crg_udc)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ unsigned long flags = 0;
+ u32 tmp_status;
+ int ret;
+
+ spin_lock_irqsave(&crg_udc->udc_lock, flags);
+
+ tmp_status = readl(&uccr->status);
+
+ if (tmp_status & CRG_U3DC_STATUS_SYS_ERR) {
+ bool was_connected = crg_udc->connected;
+
+ dev_err(crg_udc->dev, "controller system error\n");
+ writel(CRG_U3DC_STATUS_SYS_ERR, &uccr->status);
+ /* Function drivers must see enabled endpoints on disconnect. */
+ if (was_connected)
+ crg_udc_notify_disconnect(crg_udc, &flags);
+ crg_udc->connected = 0;
+ ret = crg_udc_reset(crg_udc, &flags);
+ if (!ret)
+ ret = crg_udc_reset_data_struct(crg_udc);
+ if (!ret)
+ ret = crg_udc_init_ep0(crg_udc);
+ if (ret)
+ dev_err(crg_udc->dev,
+ "failed to recover from controller error: %d\n",
+ ret);
+ crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
+ if (!ret)
+ crg_udc->resetting = false;
+ if (!ret && crg_udc->pullup_on)
+ crg_udc_set_pullup(crg_udc, true);
+
+ /*
+ * tmp_status was captured before the reset above, which tore
+ * down and rebuilt the event rings. It no longer reflects
+ * the controller's current state, and if recovery failed the
+ * rings may not be consistent at all. Skip the rest of this
+ * interrupt's processing; any event still pending will raise
+ * a fresh interrupt once the controller is running again.
+ */
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+ return;
+ }
+
+ if (tmp_status & CRG_U3DC_STATUS_EINT) {
+ int i;
+
+ writel(CRG_U3DC_STATUS_EINT, &uccr->status);
+
+ for (i = 0; i < CRG_RING_NUM; i++) {
+ ret = crg_udc_process_event_ring(crg_udc, i, &flags);
+ if (ret)
+ dev_err(crg_udc->dev,
+ "process event ring error: %d\n", ret);
+ }
+ }
+
+ if (crg_udc->device_state == USB_STATE_RECONNECTING &&
+ crg_udc->portsc_on_reconnecting == 1 &&
+ crg_udc_is_event_rings_empty(crg_udc)) {
+ crg_udc->portsc_on_reconnecting = 0;
+ crg_udc_handle_port_status(crg_udc, &flags);
+ }
+
+ if (crg_udc->device_state == USB_STATE_RECONNECTING &&
+ crg_udc->connected == 1)
+ crg_udc_prepare_for_setup(crg_udc);
+
+ spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
+}
+
+/*
+ * Primary handler only checks whether the shared interrupt belongs to the
+ * controller. All controller work, including reset polling, runs in the
+ * threaded handler and therefore never busy-waits in hard IRQ context.
+ */
+static irqreturn_t crg_udc_common_irq(int irq, void *dev)
+{
+ struct crg_gadget_dev *crg_udc = dev;
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 status = readl(&uccr->status);
+
+ if (!(status & (CRG_U3DC_STATUS_EINT | CRG_U3DC_STATUS_SYS_ERR)))
+ return IRQ_NONE;
+
+ return IRQ_WAKE_THREAD;
+}
+
+static irqreturn_t crg_udc_irq_thread(int irq, void *dev)
+{
+ struct crg_gadget_dev *crg_udc = dev;
+
+ crg_gadget_handle_interrupt(crg_udc);
+
+ return IRQ_HANDLED;
+}
+
+static int crg_gadget_irq_init(struct platform_device *pdev,
+ struct crg_gadget_dev *crg_udc)
+{
+ int irq;
+ int ret;
+
+ irq = platform_get_irq(pdev, 0);
+ if (irq < 0)
+ return irq;
+
+ /*
+ * IRQF_ONESHOT keeps this (level-triggered) line masked until
+ * crg_udc_irq_thread() has cleared the status bits, which is the
+ * correct pairing for IRQF_SHARED with a threaded handler.
+ */
+ ret = devm_request_threaded_irq(&pdev->dev, irq, crg_udc_common_irq,
+ crg_udc_irq_thread,
+ IRQF_SHARED | IRQF_ONESHOT, "crg_udc",
+ crg_udc);
+ if (ret) {
+ dev_err(crg_udc->dev, "request of irq%d failed\n", irq);
+ } else {
+ crg_udc->irq = irq;
+ crg_udc->gadget.irq = irq;
+ }
+
+ return ret;
+}
+
+int crg_udc_reset(struct crg_gadget_dev *crg_udc, unsigned long *flags)
+{
+ struct crg_uccr __iomem *uccr = crg_udc->uccr;
+ u32 i, tmp;
+ struct crg_udc_ep *udc_ep_ptr;
+ u32 val;
+
+ /*
+ * Keep all endpoint operations gated until the caller has recreated
+ * the event rings, device context and EP0 after this reset.
+ */
+ crg_udc->resetting = true;
+ val = readl(&uccr->control);
+ val |= CRG_U3DC_CTRL_SWRST;
+ writel(val, &uccr->control);
+ spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
+ /* Poll until SWRST clears; hardware reset takes up to 10 ms */
+ if (readl_poll_timeout_atomic(&uccr->control, tmp,
+ !(tmp & CRG_U3DC_CTRL_SWRST), 10, 10000)) {
+ spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+ dev_err(crg_udc->dev, "reset error\n");
+ return -ETIMEDOUT;
+ }
+ spin_lock_irqsave(&crg_udc->udc_lock, *flags);
+
+ crg_udc_clear_portpm(crg_udc);
+
+ crg_udc->setup_status = WAIT_FOR_SETUP;
+ /* Base on Figure 9-1, default USB_STATE is attached */
+ crg_udc->device_state = USB_STATE_ATTACHED;
+ crg_udc->dev_addr = 0;
+
+ /*
+ * Mark all endpoints unavailable before request callbacks drop the
+ * lock. A callback must not enqueue on a later, not-yet-nuked EP.
+ */
+ for (i = 2; i < CRG_NUM_EP_CX; i++) {
+ udc_ep_ptr = &crg_udc->udc_ep[i];
+ udc_ep_ptr->generation++;
+ udc_ep_ptr->desc = NULL;
+ udc_ep_ptr->comp_desc = NULL;
+ udc_ep_ptr->tran_ring_full = false;
+ udc_ep_ptr->ep_state = EP_STATE_DISABLED;
+ /*
+ * This bypasses the generic usb_ep_disable() API, so the
+ * framework's own usb_ep.enabled bookkeeping needs to be
+ * cleared directly here too (crg_udc_reinit() already does
+ * this for the live bus-reset case). Otherwise a later
+ * usb_ep_enable() call for this endpoint short-circuits on
+ * the stale "already enabled" flag without ever calling our
+ * .enable callback, leaving desc/the ring never set up for
+ * the new session.
+ */
+ udc_ep_ptr->usb_ep.enabled = 0;
+ }
+ crg_udc->num_enabled_eps = 0;
+
+ udc_ep_ptr = &crg_udc->udc_ep[0];
+ udc_ep_ptr->generation++;
+ udc_ep_ptr->desc = NULL;
+ udc_ep_ptr->comp_desc = NULL;
+ udc_ep_ptr->ep_state = EP_STATE_DISABLED;
+
+ for (i = 2; i < CRG_NUM_EP_CX; i++)
+ crg_udc_ep_nuke(&crg_udc->udc_ep[i], -ESHUTDOWN);
+ crg_udc_ep_nuke(udc_ep_ptr, -ESHUTDOWN);
+
+ return 0;
+}
+
+static const struct crg_udc_priv crg_udc_data = {
+ .quirks = CRG_UDC_QUIRKS_DEFAULT,
+};
+
+static const struct of_device_id of_crg_udc_match[] = {
+ { .compatible = "axiado,ax3000-udc", .data = &crg_udc_data },
+ { },
+};
+MODULE_DEVICE_TABLE(of, of_crg_udc_match);
+
+static DEFINE_SIMPLE_DEV_PM_OPS(crg_udc_pm_ops, crg_udc_suspend,
+ crg_udc_resume);
+
+static struct platform_driver crg_udc_driver = {
+ .probe = crg_udc_probe,
+ .remove = crg_udc_remove,
+ .shutdown = crg_udc_shutdown,
+ .driver = {
+ .name = KBUILD_MODNAME,
+ .of_match_table = of_crg_udc_match,
+ .pm = pm_sleep_ptr(&crg_udc_pm_ops),
+ },
+};
+module_platform_driver(crg_udc_driver);
+
+MODULE_AUTHOR("Axiado Corporation");
+MODULE_DESCRIPTION("Axiado AX3000/AX3005 USB Device Controller");
+MODULE_LICENSE("GPL");
--
2.34.1
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [PATCH v4 6/6] usb: gadget: udc: crg: Add core driver and build integration
2026-10-30 8:10 ` [PATCH v4 6/6] usb: gadget: udc: crg: Add core driver and build integration Petar Stepanovic
@ 2026-10-30 8:46 ` sashiko-bot
2026-10-30 9:34 ` Philipp Zabel
1 sibling, 0 replies; 14+ messages in thread
From: sashiko-bot @ 2026-10-30 8:46 UTC (permalink / raw)
To: Petar Stepanovic; +Cc: linux-usb, devicetree, robh, conor+dt
Thank you for your contribution! Sashiko AI review found 10 potential issue(s) to consider:
- [High] usb: udc: crg: dequeue fails in crg_udc_issue_command() when stopped
- [High] usb: gadget: crg: controller left stopped after disconnect reset
- [High] usb: gadget: crg: deferred crg_udc_prepare_for_setup() races with SETUP
- [High] usb: gadget: crg: GFP_KERNEL allocation in crg_udc_ep_enable()
- [High] usb: gadget: crg: race conditions when crg_udc_reset() drops udc_lock
- [Medium] usb: udc: crg: stale reconnect and ep0 state across crg_udc_reset()
- [Medium] usb: gadget: udc: crg: state inconsistency across crg_udc_suspend() and resume
- [Medium] usb: gadget: udc: crg: race in crg_udc_suspend() re-enables pullup
- [Medium] usb: gadget: crg: race with IRQ thread in crg_udc_shutdown() and teardown
- [Low] usb: gadget: crg: uncapped and unvalidated max_speed in crg_udc_probe()
--
commit ff1f5c0791ff275876830e2f2710870a97b34e7c
Author: Petar Stepanovic <pstepanovic@axiado.com>
usb: gadget: udc: crg: Add core driver and build integration
This commit introduces the core probe, remove, power management, interrupt
handling, and controller reset functions for the Axiado AX3000/AX3005 USB
Device Controller driver, and integrates it into Kconfig and Makefile.
> diff --git a/drivers/usb/gadget/udc/crg/crg-udc-core.c b/drivers/usb/gadget/udc/crg/crg-udc-core.c
> new file mode 100644
> index 0000000000000..33be5c920ebee
> --- /dev/null
> +++ b/drivers/usb/gadget/udc/crg/crg-udc-core.c
> @@ -0,0 +1,723 @@
[ ... ]
> +int crg_udc_issue_command(struct crg_gadget_dev *crg_udc,
> + enum crg_cmd_type type, u32 param0, u32 param1)
> +{
> + struct crg_uccr __iomem *uccr = crg_udc->uccr;
> + u32 status;
> + u32 tmp;
> + bool running;
> +
> + running = !!(readl(&uccr->control) & CRG_U3DC_CTRL_RUN);
> + /* Only INIT_EP0 may be staged while the controller is stopped. */
> + if (!running && type != CRG_CMD_INIT_EP0)
> + return -ESHUTDOWN;
[Severity: High]
As reported by Sashiko on v3 [1], rejecting all commands other than
CRG_CMD_INIT_EP0 when CRG_U3DC_CTRL_RUN is clear causes crg_udc_ep_dequeue()
to fail with -ESHUTDOWN whenever the pullup has been disabled.
During gadget unbind (gadget_unbind_driver() -> usb_gadget_disconnect_locked()
-> crg_gadget_pullup()), soft_connect disable, or crg_udc_suspend(),
crg_udc_set_pullup(crg_udc, false) clears CRG_U3DC_CTRL_RUN in uccr->control
while enabled endpoints remain in EP_STATE_RUNNING with active requests on
udc_ep->queue:
drivers/usb/gadget/udc/crg/crg-udc-gadget.c:crg_udc_ep_dequeue() {
...
if (old_ep_state == EP_STATE_RUNNING) {
int ret = crg_udc_issue_command(crg_udc, CRG_CMD_STOP_EP,
BIT(dci), 0);
if (ret) {
spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
return ret;
}
...
}
When a function driver's disconnect() or unbind() callback (or a FunctionFS
endpoint close) calls usb_ep_dequeue() -> crg_udc_ep_dequeue() before
crg_gadget_stop() runs, crg_udc_issue_command() returns -ESHUTDOWN for
CRG_CMD_STOP_EP (and would also reject CRG_CMD_SET_TR_DQPTR in
crg_udc_epcx_update_dqptr()). Because crg_udc_ep_dequeue() aborts without
unlinking or giving back the request, a subsequent usb_ep_free_request()
frees the request while it is still linked on udc_ep->queue, leading to a
use-after-free when crg_gadget_stop() -> crg_udc_reset() -> crg_udc_ep_nuke()
later traverses udc_ep->queue.
How should crg_udc_ep_dequeue() unlink and give back queued requests when
CRG_U3DC_CTRL_RUN is already clear?
[ ... ]
> +static int crg_udc_probe(struct platform_device *pdev)
> +{
[ ... ]
> + crg_udc->gadget.ops = &crg_gadget_ops;
> + crg_udc->gadget.ep0 = &crg_udc->udc_ep[0].usb_ep;
> + INIT_LIST_HEAD(&crg_udc->gadget.ep_list);
> + /*
> + * The controller supports up to SuperSpeed. Honour the standard
> + * "maximum-speed" property to cap it; default to the hardware
> + * maximum when the property is absent.
> + */
> + crg_udc->gadget.max_speed = usb_get_maximum_speed(&pdev->dev);
> + if (crg_udc->gadget.max_speed == USB_SPEED_UNKNOWN)
> + crg_udc->gadget.max_speed = USB_SPEED_SUPER;
> + crg_udc->max_speed = crg_udc->gadget.max_speed;
[Severity: Low]
The comment, Kconfig, and DT binding state that the controller supports
High-Speed and SuperSpeed and that the maximum-speed property caps the speed
up to SuperSpeed, but crg_udc_probe() only checks for USB_SPEED_UNKNOWN.
If the DT maximum-speed property is set to super-speed-plus
(USB_SPEED_SUPER_PLUS), crg_udc->gadget.max_speed is not capped to
USB_SPEED_SUPER and crg_udc_program_speed() writes CRG_U3DC_CFG0_MAXSPEED_SSP.
Conversely, if maximum-speed (or a gadget driver's max_speed passed to
crg_udc_set_speed()) is USB_SPEED_LOW, crg_udc_program_speed() returns
-EINVAL and aborts crg_udc_reset_data_struct().
Should crg_udc_probe() and crg_udc_set_speed() clamp speeds above
USB_SPEED_SUPER (or outside the supported range) to USB_SPEED_SUPER?
[ ... ]
> +static void crg_udc_shutdown(struct platform_device *pdev)
> +{
> + struct crg_gadget_dev *crg_udc = platform_get_drvdata(pdev);
> + unsigned long flags;
> +
> + spin_lock_irqsave(&crg_udc->udc_lock, flags);
> + crg_udc->pullup_on = false;
> + crg_udc->resetting = true;
> + crg_udc->device_state = USB_STATE_ATTACHED;
> + crg_udc_set_pullup(crg_udc, false);
> + crg_udc_set_vbus_detect(crg_udc, false);
> + if (crg_udc->uccr) {
> + u32 val = readl(&crg_udc->uccr->control);
> +
> + val &= ~(CRG_U3DC_CTRL_INT_EN | CRG_U3DC_CTRL_RUN);
> + writel(val, &crg_udc->uccr->control);
> + }
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + if (crg_udc->irq >= 0)
> + synchronize_irq(crg_udc->irq);
> + cancel_work_sync(&crg_udc->udc_reset_work);
> +}
[Severity: Medium]
As reported by Sashiko on v3 [2], what happens if crg_udc_irq_thread() is
already mid-recovery in crg_gadget_handle_interrupt() (or in the disconnect
reset path of crg_udc_handle_port_status()) and has dropped udc_lock inside
crg_udc_notify_disconnect() or crg_udc_reset() when crg_udc_shutdown() runs?
Once crg_udc_shutdown() releases udc_lock and waits in synchronize_irq(), the
in-flight IRQ thread reacquires udc_lock, runs crg_udc_reset_data_struct() and
crg_udc_init_ep0(), and sets crg_udc->resetting = false, undoing the shutdown
gate.
Similarly, in crg_udc_free_resources(), uccr->control is modified without
holding udc_lock before devm_free_irq() is called, while crg_udc_irq_thread()
can still be running after crg_gadget_stop() dropped udc_lock during
crg_udc_reset(). Couldn't that race on uccr->control or with resource
teardown?
> +
> +static int crg_udc_suspend(struct device *dev)
> +{
> + struct crg_gadget_dev *crg_udc = dev_get_drvdata(dev);
> + unsigned long flags;
> +
> + /* Stop the pullup while the controller is suspended. */
> + spin_lock_irqsave(&crg_udc->udc_lock, flags);
> + if (!crg_udc->resetting)
> + crg_udc_set_pullup(crg_udc, false);
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> + return 0;
> +}
[Severity: Medium]
Can an in-flight CRG_U3DC_STATUS_SYS_ERR recovery in
crg_gadget_handle_interrupt() re-enable the pullup after crg_udc_suspend()
returns?
During system suspend, crg_udc_suspend() runs before device IRQs are disabled.
If crg_gadget_handle_interrupt() is handling CRG_U3DC_STATUS_SYS_ERR and has
dropped udc_lock in crg_udc_notify_disconnect() or crg_udc_reset(),
crg_udc_suspend() either sees crg_udc->resetting == true and skips
crg_udc_set_pullup(crg_udc, false), or clears CRG_U3DC_CTRL_RUN and returns
while crg_udc->pullup_on remains true. When the IRQ thread finishes reset
recovery, it checks:
if (!ret && crg_udc->pullup_on)
crg_udc_set_pullup(crg_udc, true);
which re-asserts CRG_U3DC_CTRL_RUN while the controller is suspended. Should
crg_udc_suspend() synchronize with the IRQ/reset path and track a suspended
state so recovery does not re-assert CRG_U3DC_CTRL_RUN?
> +
> +static int crg_udc_resume(struct device *dev)
> +{
> + struct crg_gadget_dev *crg_udc = dev_get_drvdata(dev);
> + unsigned long flags;
> +
> + spin_lock_irqsave(&crg_udc->udc_lock, flags);
> + if (!crg_udc->resetting && crg_udc->gadget_driver && crg_udc->pullup_on)
> + crg_udc_set_pullup(crg_udc, true);
> + spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
> +
> + return 0;
> +}
[Severity: Medium]
When crg_udc_suspend() clears CRG_U3DC_CTRL_RUN via
crg_udc_set_pullup(crg_udc, false) while the gadget is enumerated, the
controller halts and drops bus terminations so the host sees a disconnect,
but crg_udc_suspend() does not notify the gadget driver of disconnect or
reset crg_udc->connected, crg_udc->device_state, crg_udc->dev_addr, or
enabled endpoints.
When crg_udc_resume() re-asserts CRG_U3DC_CTRL_RUN:
1. If the cable was unplugged while suspended, CCS is already 0 when
CRG_U3DC_CTRL_RUN is set, so no CSC event is generated and the gadget
driver remains stuck in a connected/configured state with queued requests
never completing.
2. If the cable remains connected to a SuperSpeed host, link training to U0
raises CRG_U3DC_PORTSC_CSC without CRG_U3DC_PORTSC_PRC. In
crg_udc_handle_port_status(), the CSC connect branch does not call
crg_udc_reinit(), and because crg_udc->device_state is still
USB_STATE_CONFIGURED or USB_STATE_ADDRESS (>= USB_STATE_RECONNECTING),
neither crg_udc_enable_setup() nor crg_udc_prepare_for_setup() runs, the
stale dev_addr and endpoints remain active, and setaddressrequest() stalls
SET_ADDRESS in USB_STATE_CONFIGURED.
Should crg_udc_suspend() disconnect the gadget session and reset controller
state before suspending?
[ ... ]
> +static void crg_gadget_handle_interrupt(struct crg_gadget_dev *crg_udc)
> +{
> + struct crg_uccr __iomem *uccr = crg_udc->uccr;
> + unsigned long flags = 0;
> + u32 tmp_status;
> + int ret;
> +
> + spin_lock_irqsave(&crg_udc->udc_lock, flags);
> +
> + tmp_status = readl(&uccr->status);
> +
> + if (tmp_status & CRG_U3DC_STATUS_SYS_ERR) {
> + bool was_connected = crg_udc->connected;
> +
> + dev_err(crg_udc->dev, "controller system error\n");
> + writel(CRG_U3DC_STATUS_SYS_ERR, &uccr->status);
> + /* Function drivers must see enabled endpoints on disconnect. */
> + if (was_connected)
> + crg_udc_notify_disconnect(crg_udc, &flags);
> + crg_udc->connected = 0;
> + ret = crg_udc_reset(crg_udc, &flags);
> + if (!ret)
> + ret = crg_udc_reset_data_struct(crg_udc);
> + if (!ret)
> + ret = crg_udc_init_ep0(crg_udc);
> + if (ret)
> + dev_err(crg_udc->dev,
> + "failed to recover from controller error: %d\n",
> + ret);
> + crg_udc->gadget.speed = USB_SPEED_UNKNOWN;
> + if (!ret)
> + crg_udc->resetting = false;
> + if (!ret && crg_udc->pullup_on)
> + crg_udc_set_pullup(crg_udc, true);
[Severity: High]
Here in the CRG_U3DC_STATUS_SYS_ERR recovery path, after crg_udc_reset(),
crg_udc_reset_data_struct(), and crg_udc_init_ep0() complete, the driver
restores the controller's running state via crg_udc_set_pullup(crg_udc, true)
when crg_udc->pullup_on is set.
However, in the cable disconnect (!cable_connected) branch of
crg_udc_handle_port_status(), the driver runs the same crg_udc_reset(),
crg_udc_reset_data_struct(), and crg_udc_init_ep0() sequence (which clears
uccr->config1 as well as CRG_U3DC_CTRL_INT_EN and CRG_U3DC_CTRL_RUN in
uccr->control, and stages CRG_CMD_INIT_EP0 while stopped) and then returns
-ECONNRESET without calling crg_udc_set_pullup(crg_udc, true):
drivers/usb/gadget/udc/crg/crg-udc-gadget.c:crg_udc_handle_port_status() {
...
} else if (!cable_connected) {
dev_dbg(crg_udc->dev,
"cable disconnected, rst controller\n");
if (was_connected)
crg_udc_notify_disconnect(crg_udc, flags);
ret = crg_udc_reset(crg_udc, flags);
if (!ret)
ret = crg_udc_reset_data_struct(crg_udc);
if (!ret)
ret = crg_udc_init_ep0(crg_udc);
if (ret)
dev_err(crg_udc->dev,
"controller reset failed on disconnect: %d\n",
ret);
else
crg_udc->resetting = false;
crg_udc->connected = 0;
crg_udc->device_state = USB_STATE_ATTACHED;
return -ECONNRESET;
}
}
Because gadget disconnect callbacks do not re-invoke the pullup callback,
won't unplugging the USB cable leave CRG_U3DC_CTRL_RUN, CRG_U3DC_CTRL_INT_EN,
and uccr->config1 cleared so the controller never consumes CRG_CMD_INIT_EP0
or detects a subsequent cable reconnection?
[ ... ]
> + if (tmp_status & CRG_U3DC_STATUS_EINT) {
> + int i;
> +
> + writel(CRG_U3DC_STATUS_EINT, &uccr->status);
> +
> + for (i = 0; i < CRG_RING_NUM; i++) {
> + ret = crg_udc_process_event_ring(crg_udc, i, &flags);
[Severity: High]
When crg_udc_process_event_ring() dispatches a SET_CONFIGURATION or
SET_INTERFACE control transfer via crg_udc_handle_event() ->
crg_udc_handle_setup_pkt(), it drops crg_udc->udc_lock and calls
driver->setup(&crg_udc->gadget, setup_pkt).
In composite_setup(), set_config() and f->set_alt() run under
spin_lock(&cdev->lock) and call usb_ep_enable() -> crg_udc_ep_enable().
However, crg_udc_ep_enable() drops udc_lock and allocates the transfer ring
with GFP_KERNEL:
drivers/usb/gadget/udc/crg/crg-udc-gadget.c:crg_udc_ep_enable() {
...
spin_unlock_irqrestore(&crg_udc->udc_lock, flags);
new_ring = dma_alloc_coherent(crg_udc->dev, len, &new_dma,
GFP_KERNEL);
spin_lock_irqsave(&crg_udc->udc_lock, flags);
...
}
Since usb_ep_enable() is called while composite holds cdev->lock (and the
gadget API requires ep->ops->enable() to be callable in atomic context),
won't dma_alloc_coherent(..., GFP_KERNEL) sleep in atomic context?
> + if (ret)
> + dev_err(crg_udc->dev,
> + "process event ring error: %d\n", ret);
> + }
> + }
> +
> + if (crg_udc->device_state == USB_STATE_RECONNECTING &&
> + crg_udc->portsc_on_reconnecting == 1 &&
> + crg_udc_is_event_rings_empty(crg_udc)) {
> + crg_udc->portsc_on_reconnecting = 0;
> + crg_udc_handle_port_status(crg_udc, &flags);
> + }
> +
> + if (crg_udc->device_state == USB_STATE_RECONNECTING &&
> + crg_udc->connected == 1)
> + crg_udc_prepare_for_setup(crg_udc);
[Severity: High]
When a bus reset (CRG_U3DC_PORTSC_PRC) completes while crg_udc->device_state
is at least USB_STATE_DEFAULT, crg_udc_handle_port_status() calls
crg_udc_reinit(), which sets crg_udc->device_state = USB_STATE_RECONNECTING,
and sets crg_udc->connected = 1 while CRG_U3DC_CFG1_SETUP_EVENT_EN remains
enabled in uccr->config1 from the prior connection.
Because crg_udc_prepare_for_setup() is only called here after
crg_udc_process_event_ring() finishes (and returns -EBUSY if the event ring
is not empty), what happens if the host's first post-reset SETUP packet is
already in the event ring behind the port status event, or arrives before
crg_udc_prepare_for_setup() checks crg_udc_is_event_rings_empty()?
In that case, crg_udc_process_event_ring() processes TRB_TYPE_EVT_SETUP_PKT
while crg_udc->device_state is still USB_STATE_RECONNECTING:
setaddressrequest() rejects USB_REQ_SET_ADDRESS and stalls EP0 because
device_state is neither USB_STATE_DEFAULT nor USB_STATE_ADDRESS, or another
control request queues a transfer on EP0 only for crg_udc_prepare_for_setup()
to immediately stop EP0 and nuke the newly queued request with -ESHUTDOWN.
[ ... ]
> +int crg_udc_reset(struct crg_gadget_dev *crg_udc, unsigned long *flags)
> +{
> + struct crg_uccr __iomem *uccr = crg_udc->uccr;
> + u32 i, tmp;
> + struct crg_udc_ep *udc_ep_ptr;
> + u32 val;
> +
> + /*
> + * Keep all endpoint operations gated until the caller has recreated
> + * the event rings, device context and EP0 after this reset.
> + */
> + crg_udc->resetting = true;
> + val = readl(&uccr->control);
> + val |= CRG_U3DC_CTRL_SWRST;
> + writel(val, &uccr->control);
> + spin_unlock_irqrestore(&crg_udc->udc_lock, *flags);
> + /* Poll until SWRST clears; hardware reset takes up to 10 ms */
> + if (readl_poll_timeout_atomic(&uccr->control, tmp,
> + !(tmp & CRG_U3DC_CTRL_SWRST), 10, 10000)) {
> + spin_lock_irqsave(&crg_udc->udc_lock, *flags);
> + dev_err(crg_udc->dev, "reset error\n");
> + return -ETIMEDOUT;
> + }
> + spin_lock_irqsave(&crg_udc->udc_lock, *flags);
> +
> + crg_udc_clear_portpm(crg_udc);
> +
> + crg_udc->setup_status = WAIT_FOR_SETUP;
> + /* Base on Figure 9-1, default USB_STATE is attached */
> + crg_udc->device_state = USB_STATE_ATTACHED;
> + crg_udc->dev_addr = 0;
[Severity: Medium]
When crg_udc_reset() resets crg_udc->device_state to USB_STATE_ATTACHED, it
does not clear crg_udc->portsc_on_reconnecting, crg_udc->setup_fn_call_back,
or crg_udc->set_tm (nor does crg_udc_reset_data_struct()).
If a port status event sets crg_udc->portsc_on_reconnecting = 1 while
crg_udc->device_state == USB_STATE_RECONNECTING and crg_udc_reset() runs
before that flag is consumed (for example due to a cable disconnect,
CRG_U3DC_STATUS_SYS_ERR, or crg_gadget_stop()), won't the stale
portsc_on_reconnecting == 1 persist until the next connection's bus reset
sets device_state = USB_STATE_RECONNECTING, causing crg_udc_prepare_for_setup()
to fail with -EBUSY or crg_gadget_handle_interrupt() to call
crg_udc_handle_port_status() spuriously?
> +
> + /*
> + * Mark all endpoints unavailable before request callbacks drop the
> + * lock. A callback must not enqueue on a later, not-yet-nuked EP.
> + */
> + for (i = 2; i < CRG_NUM_EP_CX; i++) {
> + udc_ep_ptr = &crg_udc->udc_ep[i];
> + udc_ep_ptr->generation++;
> + udc_ep_ptr->desc = NULL;
> + udc_ep_ptr->comp_desc = NULL;
> + udc_ep_ptr->tran_ring_full = false;
> + udc_ep_ptr->ep_state = EP_STATE_DISABLED;
> + /*
> + * This bypasses the generic usb_ep_disable() API, so the
> + * framework's own usb_ep.enabled bookkeeping needs to be
> + * cleared directly here too (crg_udc_reinit() already does
> + * this for the live bus-reset case). Otherwise a later
> + * usb_ep_enable() call for this endpoint short-circuits on
> + * the stale "already enabled" flag without ever calling our
> + * .enable callback, leaving desc/the ring never set up for
> + * the new session.
> + */
> + udc_ep_ptr->usb_ep.enabled = 0;
> + }
> + crg_udc->num_enabled_eps = 0;
> +
> + udc_ep_ptr = &crg_udc->udc_ep[0];
> + udc_ep_ptr->generation++;
> + udc_ep_ptr->desc = NULL;
> + udc_ep_ptr->comp_desc = NULL;
> + udc_ep_ptr->ep_state = EP_STATE_DISABLED;
> +
> + for (i = 2; i < CRG_NUM_EP_CX; i++)
> + crg_udc_ep_nuke(&crg_udc->udc_ep[i], -ESHUTDOWN);
> + crg_udc_ep_nuke(udc_ep_ptr, -ESHUTDOWN);
[Severity: High]
As reported by Sashiko on v3 [3], dropping udc_lock inside crg_udc_reset()
(both during the 10 ms CRG_U3DC_CTRL_SWRST poll and inside crg_udc_ep_nuke()
-> crg_udc_req_done()) introduces several races:
1. Neither crg_gadget_handle_interrupt() nor crg_udc_process_event_ring()
checks crg_udc->resetting after acquiring or reacquiring udc_lock, and
udc_event->generation is not incremented until crg_udc_reset_data_struct()
-> crg_udc_init_event_ring() runs after crg_udc_reset() returns. If
crg_udc_irq_thread() acquires udc_lock while crg_udc_reset() is polling
CRG_U3DC_CTRL_SWRST or giving back requests, it can access controller
registers and process stale event ring entries mid-reset.
2. Callers of crg_udc_reset() (crg_gadget_handle_interrupt(),
crg_udc_handle_port_status(), and crg_gadget_stop()) do not serialize
against an already-running reset. If two reset paths overlap across the
udc_lock drops, one can run crg_udc_reset_data_struct() and
crg_udc_init_ep0() while the other's CRG_U3DC_CTRL_SWRST is active, or
the second crg_udc_init_ep0() can see CRG_U3DC_CMD_CTRL_ACTIVE still set
from the first staged CRG_CMD_INIT_EP0 while CRG_U3DC_CTRL_RUN is 0 and
fail with -EBUSY.
3. During the 10 ms SWRST poll window, crg_udc->resetting is true while
usb_ep.enabled is still 1 and requests remain on udc_ep[i].queue; then
crg_udc_reset() clears udc_ep_ptr->usb_ep.enabled = 0 for all endpoints
before calling crg_udc_ep_nuke() (which drops udc_lock per request). If a
function driver concurrently calls usb_ep_disable() on a higher-numbered
endpoint while an earlier endpoint is being nuked (or during the SWRST
poll), usb_ep_disable() returns immediately without draining that
endpoint's queue; when the function driver then calls
usb_ep_free_request(), the freed request remains on udc_ep[i].queue and is
dereferenced by crg_udc_ep_nuke().
Can reset execution and endpoint teardown be serialized so concurrent callers
wait for the reset and request givebacks to complete?
[1] https://lore.kernel.org/r/20261020-axiado-ax3000-usb-device-controller-v3-6-3012ecfc0a93@axiado.com
[2] https://lore.kernel.org/r/20261020-axiado-ax3000-usb-device-controller-v3-6-3012ecfc0a93@axiado.com
[3] https://lore.kernel.org/r/20261020-axiado-ax3000-usb-device-controller-v3-6-3012ecfc0a93@axiado.com
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261030-axiado-ax3000-usb-device-controller-v4-0-c5267b36a777@axiado.com?part=6
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH v4 6/6] usb: gadget: udc: crg: Add core driver and build integration
2026-10-30 8:10 ` [PATCH v4 6/6] usb: gadget: udc: crg: Add core driver and build integration Petar Stepanovic
2026-10-30 8:46 ` sashiko-bot
@ 2026-10-30 9:34 ` Philipp Zabel
1 sibling, 0 replies; 14+ messages in thread
From: Philipp Zabel @ 2026-10-30 9:34 UTC (permalink / raw)
To: Petar Stepanovic, Naveen Kumar Rajgiri Bassappa, Prasad Bolisetty,
Greg Kroah-Hartman, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Tzu-Hao Wei, Karthikeyan Mitran
Cc: linux-usb, devicetree, linux-arm-kernel, linux-kernel, openbmc,
Krutik Shah
On Fr, 2026-10-30 at 01:10 -0700, Petar Stepanovic wrote:
> This patch introduces the core probe/remove functions for the Axiado
> AX3000/AX3005 UDC driver, and wires up the directory into the build
> system via Kconfig and Makefile.
>
> Co-developed-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
> Signed-off-by: Naveen Kumar Rajgiri Bassappa <rbnaveenkumar@axiado.com>
> Co-developed-by: Krutik Shah <krutikshah@axiado.com>
> Signed-off-by: Krutik Shah <krutikshah@axiado.com>
> Signed-off-by: Petar Stepanovic <pstepanovic@axiado.com>
> ---
> drivers/usb/gadget/udc/Kconfig | 2 +
> drivers/usb/gadget/udc/Makefile | 1 +
> drivers/usb/gadget/udc/crg/Kconfig | 16 +
> drivers/usb/gadget/udc/crg/Makefile | 9 +
> drivers/usb/gadget/udc/crg/crg-udc-core.c | 723 ++++++++++++++++++++++++++++++
> 5 files changed, 751 insertions(+)
>
[...]
> diff --git a/drivers/usb/gadget/udc/crg/crg-udc-core.c b/drivers/usb/gadget/udc/crg/crg-udc-core.c
> new file mode 100644
> index 000000000000..33be5c920ebe
> --- /dev/null
> +++ b/drivers/usb/gadget/udc/crg/crg-udc-core.c
> @@ -0,0 +1,723 @@
[...]
> +static int crg_udc_init_hw(struct crg_gadget_dev *crg_udc)
> +{
> + struct device *dev = crg_udc->dev;
> + struct reset_control *rst;
> + int ret;
> +
> + crg_udc->clk = devm_clk_get_optional(dev, NULL);
> + if (IS_ERR(crg_udc->clk))
> + return dev_err_probe(dev, PTR_ERR(crg_udc->clk),
> + "failed to get clock\n");
> +
> + ret = clk_prepare_enable(crg_udc->clk);
> + if (ret)
> + return dev_err_probe(dev, ret, "failed to enable clock\n");
> +
> + ret = devm_add_action_or_reset(dev, crg_udc_clk_disable, crg_udc->clk);
> + if (ret)
> + return ret;
> +
> + rst =
> + devm_reset_control_get_optional_exclusive_deasserted(dev, NULL);
This can be put on one line.
regards
Philipp
^ permalink raw reply [flat|nested] 14+ messages in thread